Category: Log Insight

In my previous blog I created a script to get the last backup status from NSX Manager in order to quickly check multiple NSX Managers. Today I had a need to bring the alarms raised in all of these NSX Managers into one single location, which necessitated creating an NSX Alarm Check Script.
‘But surely the NSX Management Pack would allow you to do this you?’ may ask. Unfortunately it is missing some of the alarms which gets raised on the NSX Managers such as passwords expiring for example. This one being an annoyance if you do not notice until after it’s expired and you are having LDAP issues.

Now luckily this time, we CAN use the NSX REST API to get these details, and I had a script lying around which could provide a skeleton for this. You can find that script here: NSX Backup Check Script

In order to adapt this script to use REST we need to change the Invoke-WebMethod to Invoke-RestMethod

Interrogating NSX REST API

I used the documentation from VMware {code} to find this API and how to handle the results. Luckily this is a lot more detailed than the web API. You can find the NSX API details here:

https://developer.vmware.com/apis/547/nsx-t

so we want to request /api/v1/alarms in order to return a list of all alarms on the nsx managers.

$result = Invoke-RestMethod -Uri https://$nsxmgr/api/v1/alarms -Headers $Header -Method 'GET' -UseBasicParsing

Handling the Outputs.

Running this command will give a response similar to this:

{
  "result_count": 4,
  "results": [
      {
        "id": "xxxx",
        "status": "OPEN",
        "feature_name": "manager_health",
        "event_type": "manager_cpu_usage_high",
        "feature_display_name": "Manager Health",
        "event_type_display_name": "CPU Usage High",
        "node_id": "xxxx",
        "last_reported_time": 1551994806,
        "description": |
          "The CPU usage for the manager node identified by 
           appears to be\nrising.",
        "recommended_action": |
          "Use the top command to check which processes have the most CPU
           usages, and\nthen check \/var\/log\/syslog and these processes'
           local logs to see if there\nare any outstanding errors to be
           resolved.",
        "node_resource_type": "ClusterNodeConfig",
        "severity": "WARNING",
        "entity_resource_type": "ClusterNodeConfig",
      },
      ...
  ]
}

From this output I wanted to pull out the severity, status, alarm description and the node which was impacted, so I pulled these into an array and add the items to variables.

$nsxAlarms = $result.results 
    foreach ($nsxAlarm in $nsxalarms) {
        $nsxAlarmCreated = (get-date 01.01.1970).AddSeconds([int]($nsxAlarm._create_time/1000)).ToString("yyyy/MM/dd HH:mm:ss")
        $timestamp = (Get-Date).ToString("yyyy/MM/dd HH:mm:ss")
        $nsxAlarmSeverity = $nsxAlarm.severity
        $nsxAlarmStatus = $nsxAlarm.status
        $nsxAlarmNode_display_name = $nsxAlarm.node_display_name
        $nsxAlarmDescription = $nsxAlarm.description

From here I wanted to only include any alarms which had not been marked acknowledged or resolved to avoid constantly reporting a condition which was known about.

if($nsxAlarm.status -ne "ACKNOWLEDGED" -and $nsxAlarm.status -ne "RESOLVED"){ 
    Add-Content -Path $exportpath -Value "$timestamp [$nsxAlarmSeverity] $NSXMGR - Alarm Created $nsxAlarmCreated Status $nsxAlarmStatus Affected Node $nsxAlarmNode_display_name Description  $nsxAlarmDescription"
}

It is also possible to bypass this by running the following command, however I wanted to pull in all alarms for my specific use case.

GET /api/v1/alarms?status=OPEN

As per the previous script, this was wrapped in a try catch and the catch failure tested if the host was up. A full explanation can be found on the blog about this script here: NSX Backup Check Script

.

The Final NSX Alarm Check Script

param ($nsxmgr)

$curDir = &{$MyInvocation.PSScriptRoot}
$exportpath = "$curDir\..\Logs\NSXAlarmCheck.log"
$credPath = "$curDir\$nsxmgr.cred"
$scriptName = &{$MyInvocation.ScriptName}

add-type @"
   using System.Net;
   using System.Security.Cryptography.X509Certificates;
   public class TrustAllCertsPolicy : ICertificatePolicy {
      public bool CheckValidationResult(
      ServicePoint srvPoint, X509Certificate certificate,
      WebRequest request, int certificateProblem) {
      return true;
   }
}
"@
[System.Net.ServicePointManager]::CertificatePolicy = New-Object TrustAllCertsPolicy

function catchFailure {
    $timestamp = (Get-Date).ToString("yyyy/MM/dd HH:mm:ss")
    if (Test-Connection -BufferSize 32 -Count 1 -ComputerName $nsxmgr -Quiet) {
        Add-Content -Path $exportpath -Value "$timestamp [ERROR] $NSXMGR - $_"
    }
    else {
        Add-Content -Path $exportpath -Value "$timestamp [ERROR] $NSXMGR - Host Not Found"
    }
exit
}

if (!$nsxmgr) {
    Write-Host "please provide parameter 'nsxmgr' in the format '$scriptName -nsxmgr [FQDN of NSX Manager]'"
    exit
    }

if (-Not(Test-Path -Path  $credPath)) {
    $username = Read-Host "Enter username for NSX Manager" 
    $pass = Read-Host "Enter password" -AsSecureString 
    $password = [System.Runtime.InteropServices.Marshal]::PtrToStringAuto([System.Runtime.InteropServices.Marshal]::SecureStringToBSTR($pass))
    $userpass  = $username + ":" + $password

    $bytes= [System.Text.Encoding]::UTF8.GetBytes($userpass)
    $encodedlogin=[Convert]::ToBase64String($bytes)
    
    Set-Content -Path $credPath -Value $encodedlogin
}

$encodedlogin = Get-Content -Path $credPath

$authheader = "Basic " + $encodedlogin
$header = New-Object "System.Collections.Generic.Dictionary[[String],[String]]"
$header.Add("Authorization",$authheader)

try{
    $result = Invoke-RestMethod -Uri https://$nsxmgr/api/v1/alarms -Headers $Header -Method 'GET' -UseBasicParsing

        $nsxAlarms = $result.results 
        foreach ($nsxAlarm in $nsxalarms) {
            
            $nsxAlarmCreated = (get-date 01.01.1970).AddSeconds([int]($nsxAlarm._create_time/1000)).ToString("yyyy/MM/dd HH:mm:ss")
            $timestamp = (Get-Date).ToString("yyyy/MM/dd HH:mm:ss")
            $nsxAlarmSeverity = $nsxAlarm.severity
            $nsxAlarmStatus = $nsxAlarm.status
            $nsxAlarmNode_display_name = $nsxAlarm.node_display_name
            $nsxAlarmDescription = $nsxAlarm.description

            if($nsxAlarm.status -ne "ACKNOWLEDGED" -and $nsxAlarm.status -ne "RESOLVED"){ 
                Add-Content -Path $exportpath -Value "$timestamp [$nsxAlarmSeverity] $NSXMGR - Alarm Created $nsxAlarmCreated Status $nsxAlarmStatus Affected Node $nsxAlarmNode_display_name Description  $nsxAlarmDescription"
            }
        
    }
 }
catch {catchFailure}

Overview

The final script above can be altered to be used as a skeleton for any other Invoke-RestRequest APIs as well as simply being adapted for Web API. I will be following up this post with further updates to adapt the script in order to use PowerCLI, which required a different credential store.

I was recently asked for a way to have a simple check and report on the last backup status for a global company with multiple VMware NSX managers.

For some reason their NSX Managers were not reporting the backup status via syslog to VMware vRealize Log Insight (vRLI) and even if it was, they only have one vRLI cluster per site and wanted one simple place to do their daily checks.

So let’s make an NSX backup check script, PowerShell and REST API to the rescue! … right?

So … no, you cannot get the backup status via REST API

Great.

But you can via the WebAPI!

Hurrah! Lets throw in some Invoke-WebRequest and get the data we need.

After some basic checks, I got the info I wanted – now I need to schedule it and have it run a short period after the backup window.

This part resulted in a path of trying to figure out a way to hold account passwords in a usable manner without them being written in clear-text anywhere because that’s just no good. There are a few different ways to do this, but they either tie it to one user profile and computer, or don’t work with the basic auth needed to run against NSX to get the data via webrequest. I will go into how I achieved that further down, but first, the web API to get backup status.

Interrogating NSX Web API

After some looking around, I discovered the following URL called via Invoke-WebRequest would give us the backup results:

Invoke-WebRequest -Uri https://[nsxmgr]/api/v1/cluster/backups/overview -Headers $Header 

Now the big problem with Invoke-WebRequest is that you would have assumed that it would return any response status such as 403 Forbidden. Nope!

You don’t get any helpful error catching, it either works or bombs out. Not much good for an unattended script that you want to tell you about any issues.

So the best fix I came up with was using a try and catch

try { 
    $result = Invoke-WebRequest -Uri https://...
    }
catch {catchFailure}

I then created a function to run in the event of the failure which will ping the host to see if it’s online and if it is output the error, if it isn’t output that the host is unreachable.

if (Test-Connection -BufferSize 32 -Count 1 -ComputerName $nsxmgr -Quiet) {
        <error output>
    } else { <host offline output> } exit

Job jobbed, no more bombing out with red text.

Dealing with certificates

When you run Invoke-WebRequest against an NSX manager with self signed certificates you get the error "The Underlying connection was closed: Could not establish trust relationship for the SSL/TLS secure channel"

The fix for this is to add in this code near the top of your script to resolve the error.

add-type @"
   using System.Net;
   using System.Security.Cryptography.X509Certificates;
   public class TrustAllCertsPolicy : ICertificatePolicy {
      public bool CheckValidationResult(
      ServicePoint srvPoint, X509Certificate certificate,
      WebRequest request, int certificateProblem) {
      return true;
   }
}
"@
[System.Net.ServicePointManager]::CertificatePolicy = New-Object TrustAllCertsPolicy

Password Management

Great stuff I now have a working Script, but ideally, I want it to be scheduled and unattended.

This is where I spun around for a while trying different ways to store credentials in a secure format, because passwords in plain text is uncool.

I initially was trying to use the encoded credentials modules but having little luck getting it passed as a header value, so bugged a colleague (@pauldavey_79) for some help and ideas from his many years of experience prodding APIs.

What we came up with was to take the username and password as an requested input via Read-Host and encode it in the Base 64 format required to pass via the header in Invoke-WebRequest and store that in a text file.

[System.Runtime.InteropServices.Marshal]::PtrToStringAuto([System.Runtime.InteropServices.Marshal]::SecureStringToBSTR($pass))
    $userpass  = $username + ":" + $password

    $bytes= [System.Text.Encoding]::UTF8.GetBytes($userpass)
    $encodedlogin=[Convert]::ToBase64String($bytes)
    
    Set-Content -Path $credPath -Value $encodedlogin

This worked a charm.

Handling the Outputs.

With this script I wanted to feed the output into a syslog server (vRealize Log Insight in this case) which would then send emails to the appropriate places and allow for a nice dashboard for quick whole estate checks.

In order to achieve this, I used the Add-Content command to append the data to a .log file which was monitored by the Log Insight Agent and sent off to the Log Insight Server.

if($LatestBackup.success -eq $true){ 
  Add-Content -Path $exportpath -Value "$timestamp [INFO] $NSXMGR - Last backup successful. Start time $start End time $end"
} else{ 
  Add-Content -Path $exportpath -Value "$timestamp [ERROR] $NSXMGR - Last backup failed $start $end"

This gives us a nice syslog formatted output which can be easily manipulated within Log Insight. Hurrah.

One thing to note is that the NSX WebAPI returned the start and end times in the usual unix format, so I needed to convert that to a more suitable human readable date, that was done with the line:

 $var = (get-date 01.01.1970).AddSeconds([int]($LatestBackup.end_time/1000))

I also needed to get my try-catch error collector to output the error messages in the same format so that was done as so:

Add-Content -Path $exportpath -Value "$timestamp [ERROR] $NSXMGR - $_"

Pulling all of that together we get the final script which can be used as a skeleton for any future work required. A few of them will be posted at a later date.

The Final NSX Backup Check Script

I have put all the variables at the top and the script is designed to be run in a folder and to have another separate folder with the logs. This was done in order to manage multiple scripts logging to the same location
eg:
c:\scripts\NSXBackupCheck\NSXBackupCheck.ps1
c:\scripts\Logs\NSXBackupCheck.log

param ($nsxmgr)

$curDir = &{$MyInvocation.PSScriptRoot}
$exportpath = "$curDir\..\Logs\NSXBackupCheck.log"
$credPath = "$curDir\$nsxmgr.cred"
$scriptName = &{$MyInvocation.ScriptName}

add-type @"
   using System.Net;
   using System.Security.Cryptography.X509Certificates;
   public class TrustAllCertsPolicy : ICertificatePolicy {
      public bool CheckValidationResult(
      ServicePoint srvPoint, X509Certificate certificate,
      WebRequest request, int certificateProblem) {
      return true;
   }
}
"@
[System.Net.ServicePointManager]::CertificatePolicy = New-Object TrustAllCertsPolicy

function catchFailure {
    $timestamp = (Get-Date).ToString("yyyy/MM/dd HH:mm:ss")
    if (Test-Connection -BufferSize 32 -Count 1 -ComputerName $nsxmgr -Quiet) {
        Add-Content -Path $exportpath -Value "$timestamp [ERROR] $NSXMGR - $_"
    }
    else {
        Add-Content -Path $exportpath -Value "$timestamp [ERROR] $NSXMGR - Host Not Found"
    }
exit
}

if (!$nsxmgr) {
    Write-Host "please provide parameter 'nsxmgr' in the format '$scriptName -nsxmgr [FQDN of NSX Manager]'"
    exit
    }

if (-Not(Test-Path -Path  $credPath)) {
    $username = Read-Host "Enter username for NSX Manager" 
    $pass = Read-Host "Enter password" -AsSecureString 
    $password = [System.Runtime.InteropServices.Marshal]::PtrToStringAuto([System.Runtime.InteropServices.Marshal]::SecureStringToBSTR($pass))
    $userpass  = $username + ":" + $password

    $bytes= [System.Text.Encoding]::UTF8.GetBytes($userpass)
    $encodedlogin=[Convert]::ToBase64String($bytes)
    
    Set-Content -Path $credPath -Value $encodedlogin
}

$encodedlogin = Get-Content -Path $credPath

$authheader = "Basic " + $encodedlogin
$header = New-Object "System.Collections.Generic.Dictionary[[String],[String]]"
$header.Add("Authorization",$authheader)

try{
    $result = Invoke-WebRequest -Uri https://$nsxmgr/api/v1/cluster/backups/overview -Headers $Header -UseBasicParsing
    if($result.StatusCode -eq 200) {
        $nsxbackups = $result.Content | ConvertFrom-Json
        $LatestBackup = $nsxbackups.backup_operation_history.cluster_backup_statuses
        $start = (get-date 01.01.1970).AddSeconds([int]($LatestBackup.start_time/1000))
        $end = (get-date 01.01.1970).AddSeconds([int]($LatestBackup.end_time/1000))
        $timestamp = (Get-Date).ToString("yyyy/MM/dd HH:mm:ss")
        if($LatestBackup.success -eq $true){ 
            Add-Content -Path $exportpath -Value "$timestamp [INFO] $NSXMGR - Last backup successful. Start time $start End time $end"
        } else{ 
            Add-Content -Path $exportpath -Value "$timestamp [ERROR] $NSXMGR - Last backup failed $start $end"
        }
    }
 }
catch {catchFailure}

Overview

The final script above can be altered to be used as a skeleton for any other Invoke-WebRequest APIs as well as simply being adapted for REST API. I will be following up this post with further updates to this script using RESTAPI and also an adaption to use PowerCLI which required a different credential store.

The REST API Script can be found here: NSX Alarm Check Script