Automated Alert Remediation in vRealize Operations 7.x using vRealize Orchestrator

Have you ever wished that you could automatically execute a custom workflow in response to an alert generated in vRealize Operations? In previous releases of vRealize Operations, there wasn’t an easy way of accomplishing this. While there were many actions that could be executed using the built-in VMware vSphere solution, there wasn’t an easy built-in method to execute custom actions. With the introduction of the VMware vRealize Operations Management Pack for vRealize Orchestrator, VMware has finally made this possible.

With this management pack, you can now integrate your own custom vRealize Orchestrator workflows as alert remediation actions. This allows you to configure your vRealize Operations policy to automatically execute the remediation action anytime the alert is generated. As an example, I will demonstrate one of my use cases which involves the implementation of United States Department of Defense VMware vSphere 6.0 Virtual Machine STIG - Ver 1, Rel 1 settings to any VM that vRealize Operations finds to be non-compliant. This solution allows me to ensure that my environment is compliant with the STIG requirements at all times without manually reviewing compliance reports or dashboards.

vRealize Orchestrator Workflow

First, let’s take a look at the vRealize Orchestrator workflow that I utilize to apply the STIG settings to the VMs. This workflow is based on the workflow published in a 2012 VMware Blog post titled Automatically Securing Virtual Machines Using vCenter Orchestrator by William Lamb. The workflow is very simple and takes a text file containing advanced VM settings listed as key/value pairs and applies them to a VM.
A vRealize Orchestrator package containing the complete workflow and resource file can be downloaded here: vRealize Orchestrator Package for Applying DISA STIG Settings to Virtual Machines

Next, we need to configure the vRealize Operations Management Pack for vRealize Orchestrator to discovery our workflows. This is accomplished by configuring the discovery settings of the management pack to include the vRealize Orchestrator package that contains our workflow. To modify the list of packages to discover, with the vRealize Operations interface select the Environment tab, then in the left select All Objects. Next, you will need to expand vRealize Orchestrator Adapter -> vRealize Orchestrator Adapter Instances object on the left side and select your instance of vRealize Orchestrator that you configured when you installed the management pack. On the resulting details window, select the Action menu from the top and select Configure Package Discovery.

In the resulting Configure Package Discovery window, click on the first item in the list under the Included Packages header. If necessary, expand the text box so that you can see all of the content. By default, it will include the following text:

#Contains the list of packages to be included.
#Add package details in each line with the format <packageName> to discover new packages.
#Below mentioned vCenter package is discovered by default.

com.vmware.library.vcenter
com.vmware.vrops.oob\_content

Add the name of the package to this list that contains your vRealize Orchestrator workflows. In the case of this example, I’ll be adding com.stevenbright.apply-dod-stigs-to-vm. Then click the BEGIN ACTION button to begin the workflow discovery process.

The resulting dialog will provide you with the Task ID for the discovery action. You can click on this ID to view the status of your package discovery under the Recent Tasks to confirm that it has completed successfully.

Configure the Workflow as a vRealize Operations Action

Next up, we need to relate our vRealize Orchestrator workflow as an action that can be executed against a vRealize Operations object type. In this case, we’ll be assigning the workflow as an action for the vRealize Operations Virtual Machine object type. To do this, you’ll need to find your newly imported vRealize Orchestrator workflow in vRealize Operations either by searching for it by name or browsing for it under the Environment tab. Once you have your workflow select, click on the Actions menu and select Create/Modify Workflow Action on vCenter Resources. In the resulting dialog box, select Virtual Machine for the Resource Type, and Add for the Operation. Click the BEGIN ACTION button to add the workflow action. vRealize Operations will again provide you with a dialog box that lists a Task ID for adding the action. You view the status of the task under the Recent Tasks view in vRealize Operations.

To verify that your workflow can now be executed as vRealize Operations Action, select the Alerts tab from the top of the window, then on the left side under Alert Settings, select Actions. Search through the resulting list to verify that your workflow is available as an action.

Configure the vRealize Operations Alert Recommendation

Now that we have verified that our workflow is available as an action, we need to assign this action to either an existing alert recommendation or a newly created recommendation. To do this, select Recommendations under the Alert Settings, then click on the green plus sign to create a new recommendation. Enter a string that describes what the recommendation is, then select vRealize Orchestrator Adapter from the Adapter Type list. For the Action, select your workflow from the list then click the SAVE button.

Since we are creating a new recommendation in this example, we now need to assign this recommendation to an alert. To do this, click on Alert Definitions under the Alert Settings. Either create a new alert definition or select an existing definition to edit. While I recommend creating a custom alert specific to the DoD STIG checks, for this example, I will assign our new recommendation to the existing alert Virtual Machine is violating Risk Profile 1 in VMware vSphere Security Configuration Guide for vSphere version 6.7. To accomplish this, we will select the alert from the list of alerts and click on the pencil icon at the top of the list to edit the alert definition. In the resulting dialog, we will select Add Recommendations from the left side. Filter the list of recommendations by searching for “stig”, and then drag/drop the recommendation from the left side to the Recommendations section on the right side of the Alert Definition Workspace ensuring that our recommendation is the first in the list. Click the SAVE button to update the alert definition.

Update the vRealize Operations Policy

The final step is to configure our vRealize Orchestrator policy so that the alert is configured to execute the automation. To do this, select Administration from the top of the window, then select Policies from the left side of the window. Select the policy that you would like to apply this automation to, then click the pencil icon at the top of the list to edit the policy.

In the resulting Edit Monitoring Policy dialog, select Alert/Symptom Definitions from the left side. In the Filter box at the top of the Alert Definitions list, we’ll search for “Virtual Machine is violating Risk Profile 1 in VMware vSphere Security Configuration Guide for vSphere version 6.7”. This should filter the list so that only our desired alert is listed. Notice that our action is listed under the Actionable Recommendations. To have vRealize Operations automatically execute this action, under the Automate column, change the value from Inherit to the value Local that has a green checkmark to the left of it. Click the SAVE button to save the policy change.

Now that the policy has been changed to include the automated remediation action, vRealize Operations will execute our workflow against every VM that generates the alert “Virtual Machine is violating Risk Profile 1 in VMware vSphere Security Configuration Guide for vSphere version 6.7”. vRealize Operations will only execute the automation a single time when the alert is first generated. Because of this, any existing VMs that already have the alert active will need to have the alert canceled so that vRealize Operations can regenerate the alert.

That is all there is to it. Every time your alert triggers, it will now automatically execute your vRealize Orchestrator workflow. To verify that this has occurred, go back to the Recent Tasks list in vRealize Operations by selection Administration at the top of the window, then expand History, and select Recent Tasks.

Share:

Search

Get Notified of Future Posts

Follow Me

Recent Posts