Find all VMs with XHCI USB controller


Just a quick heads up about VMSA-2020-0026 and especially about (CVE-2020-4004) concerning the XHCI (USB3) Controller.

Use-after-free vulnerability in XHCI USB controller (CVE-2020-4004)

Description
VMware ESXi, Workstation, and Fusion contain a use-after-free vulnerability in the XHCI USB controller. VMware has evaluated the severity of this issue to be in the Critical severity range with a maximum CVSSv3 base score of 9.3.

Known Attack Vectors

A malicious actor with local administrative privileges on a virtual machine may exploit this issue to execute code as the virtual machine’s VMX process running on the host.

The following PowerCLI one-liner searches for all virtual machines and templates that are configured with the XHCI (USB3) controller.

Get-View -ViewType VirtualMachine -Property Name, Config.Hardware | Where-Object {$_.Config.Hardware.Device.DeviceInfo.Label -match "xhci"} | Select -ExpandProperty Name 

Just connect to your vCenter of choice via Connect-VIServer and see if your environment has VMs or templates that uses the XHCI controller and apply the patch or workaround described in the advisory when needed.

– Happy Scripting –

Update-Module VMware.PowerCLI – Unable to find repository


On one of my development machines (old school Windows 2012 R2 with PowerShell 5.1) I tried to upgrade the VMware.PowerCLI module to the latest version. But this time it was not just type the “update-module VMware.PowerCLI” command and you’re ready. Instead I received the following error:

Unable to find repository ‘https://www.powershellgallery.com/api/v2′. Use Get-PSRepository to see all available repositories.

image

I remembered this post from Przemyslaw Klys describing the same issue. Recently PowerShellGallery disabled support for TLS 1.0 and is now requiring TLS 1.2 So you have to enable TLS 1.2 support on your “old” servers.

The quick fix, if you want to enable TLS 1.2 for the current PowerShell session is to run the following command:

[Net.ServicePointManager]::SecurityProtocol = [Net.ServicePointManager]::SecurityProtocol -bor [Net.SecurityProtocolType]::Tls12

After enabling TLS 1.2 it’s time to upgrade to the latest version of the VMware.PowerCLI module.

Or you can just enable TLS 1.2 on your system by checking this guide from Microsoft.

– Happy scripting –

Network issues with VMware Tools 10.2.0 and Windows Server 2008 R2 Guest VMs


When you’re (still) running Windows 2008 R2 and are using VMware Tools 10.2.0 you might run in an issue regarding to network loss. VMware has published KB54459.

Windows Server 2008 R2 guest VM ports are exhausted after upgrading to VMware Tools 10.2.0

Details:

  • Guest virtual machine ports are exhausted after a few days.
  • Networking is lost.
  • Network connections cannot be made.

Source: https://kb.vmware.com/s/article/54459

VMware has fixed the issue with the release of VMware Tools 10.2.5.

Ports are exhausted on a guest VM after using VMware Tools 10.2.0

Guest VM ports are exhausted after using VMware Tools 10.2.0. This results in network connection failure.

This issue is resolved in this release.

Source: https://docs.vmware.com/en/VMware-Tools/10.2/rn/vmware-tools-1025-release-notes.html

But how do you know if you’re running this buggy combo? Well PowerCLI to rescue. The script below gathers all the VMs running Windows 2008 R2 and VMware Tools 10.2.0:

Import-Module VMware.VimAutomation.Core

Connect-VIServer -Server <vCenterName>

$AllVms_view = get-view -ViewType VirtualMachine -Property Name, Config, Guest

Disconnect-VIServer -Confirm:$false

$AllAffectedVMs = $Allvms_view |?{$_.Config.GuestId -eq 'windows7Server64Guest' -and $_.Config.Tools.ToolsVersion -eq '10304'}

$AllAffectedVMsInfo = @()
$AllAffectedVMs | % {

  $AllAffectedVMsInfo += [pscustomobject]@{
     Name = $_.Name
     ToolsVersion = $_.Config.Tools.ToolsVersion
     GuestOSId = $_.Config.GuestId
     GuestFullName = $_.Config.GuestFullName
  }
}

$AllAffectedVMsInfo | ft -AutoSize

Copy the script and change the <vCenterName>. The script will gather all the VMs via Get-View with the Name, Config and Guest properties only. So it’s lightning fast. Once the script has all the information the filtering will take place.  The variable $AllAffectedVms contains all the VMs with Windows 2008R2 as GuestOS and with tools version 10304. Take a look at https://packages.vmware.com/tools/versions to correlate all the different version/build numbers.

The fix is easy. Just upgrade the VMware Tools on the affected VMs.

Veeam: Bitlooker & Powershell


In this post I will share some Powershell code to enable or disable the feature BitLooker for your Veeam jobs. Since Bitlooker is a “new” feature you have to enable it on your jobs that are created before you installed Veeam v9 or higher. I my case we are running Veeam Backup for quite some years now, so we had to enable this feature on hundreds of jobs. Once again Powershell to the rescue.

Bur first things first, what is Bitlooker?

With this option enabled, Veeam Backup & Replication performs the following operations during the job session:

  1. Veeam Backup & Replication accesses the MFT file on the VM guest OS to identify deleted file blocks, and zeros out these blocks.
  2. Veeam Backup & Replication processes and transports data blocks of the VM image in the following manner:
  • If a data block of the VM image contains only the deleted file blocks, Veeam Backup & Replication does not read this data block from the source datastore.
  • If a data block of the VM image contains zeroed out blocks and other data, Veeam Backup & Replication copies this block to the target. Due to data compression, data blocks that are marked as deleted are compressed, and the size of the resulting backup or replica file reduces.

source: https://helpcenter.veeam.com/docs/backup/vsphere/dirty_blocks.html?ver=95

You can find the Exclude delete file blocks option in the job configuration wizard under Storage – Advanced – Storage. This option is enabled by default if you configured the job after you installed Veeam v9 but is disabled on jobs created before the installation or upgrade to Veeam v9. See the screenshot below:

image

Well that’s great this new option but I need to configure this setting for hundreds or maybe even more jobs. Is this possible via Powershell? On the Veeam forum you can find a sample script written by v.Eremin. This script will enable this setting right away with no interaction to the user. I wanted to create a script so I was able to see which jobs needs to be configured. So I came up with the following script:

Add-PSSnapin VeeamPSSnapIn

Connect-VBRServer -Server VEEAM-SRV01.DOMAIN.LOCAL

$BitlookerInfo = @()
$AllVbrJobs = Get-VBRJob
$AllVbrJobsCount = ($AllVbrJobs | Measure).Count
$VeeamBackupServer = (Get-VBRLocalhost | Select -ExpandProperty Name).ToUpper()
Write-Output "Checking Veeam Server: $($VeeamBackupServer) ..."
Write-Output ''
$AllVbrJobsDirtyBlocksNullingDisabled = $AllVbrJobs | ?{$_.Options.ViSourceOptions.DirtyBlocksNullingEnabled -eq $False}
$AllVbrJobsDirtyBlocksNullingDisabled | %{
    $BitlookerInfo += New-Object PSCustomObject -Property @{
        BackupServer = $VeeamBackupServer
        Name = $_.Name
        ExcludeDeletedFileBlocks = $_.Options.ViSourceOptions.DirtyBlocksNullingEnabled
    }

    foreach($job in $BitlookerInfo){
        $vbrjob = Get-VBRJob -Name $job.Name
        Write-Output "Job: $($vbrjob.Name)"
        $options = $vbrjob.GetOptions()
        $options.ViSourceOptions.DirtyBlocksNullingEnabled = $true
        $vbrjob.SetOptions($options)
    }

}

$AllBitlookerDisabledJobsCount = ($BitlookerInfo | Measure).Count
Write-Output "Total Veeam jobs: $($AllVbrJobsCount)"
Write-Output "Total Veeam jobs where Bitlooker is enabled: $($AllBitlookerDisabledJobsCount)"
Write-Output ''
$BitlookerInfo | Sort Name | ft -AutoSize

Disconnect-VBRServer

This is the output of the script:

Checking Veeam Server: VEEAM-SRV01.DOMAIN.LOCAL …

Total Veeam jobs: 2
Total Veeam jobs where Bitlooker is enabled: 1

Name              BackupServer                              ExcludeDeletedFileBlocks
—-                   ————                                      ————————
Demo-Job-01 VEEAM-SRV01.DOMAIN.LOCAL    False

You can add a for each loop to loop through the Veeam Backup & Replication servers you configured. This will look like this:

Add-PSSnapin VeeamPSSnapIn $AllVeeamServers = @("VEEAM-SRV01.DOMAIN.LOCAL","VEEAM-SRV02.DOMAIN.LOCAL","VEEAM-SRV03.DOMAIN.LOCAL","VEEAM-SRV04.DOMAIN.LOCAL") foreach($VeeamServer in $AllVeeamServers){ Connect-VBRServer -Server $VeeamServer $BitlookerInfo = @() $AllVbrJobs = Get-VBRJob $AllVbrJobsCount = ($AllVbrJobs | Measure).Count $VeeamBackupServer = (Get-VBRLocalhost | Select -ExpandProperty Name).ToUpper() Write-Output "Checking Veeam Server: $($VeeamBackupServer) ..." Write-Output '' $AllVbrJobsDirtyBlocksNullingDisabled = $AllVbrJobs | ?{$_.Options.ViSourceOptions.DirtyBlocksNullingEnabled -eq $False} $AllVbrJobsDirtyBlocksNullingDisabled | %{ $BitlookerInfo += New-Object PSCustomObject -Property @{ BackupServer = $VeeamBackupServer Name = $_.Name ExcludeDeletedFileBlocks = $_.Options.ViSourceOptions.DirtyBlocksNullingEnabled } foreach($job in $BitlookerInfo){ $vbrjob = Get-VBRJob -Name $job.Name Write-Output "Job: $($vbrjob.Name)" $options = $vbrjob.GetOptions() $options.ViSourceOptions.DirtyBlocksNullingEnabled = $true $vbrjob.SetOptions($options) } } $AllBitlookerDisabledJobsCount = ($BitlookerInfo | Measure).Count Write-Output "Total Veeam jobs: $($AllVbrJobsCount)" Write-Output "Total Veeam jobs where Bitlooker is enabled: $($AllBitlookerDisabledJobsCount)" Write-Output '' $BitlookerInfo | Sort Name | ft -AutoSize Disconnect-VBRServer }

Powershell: Veeam Cloud Connect total virtual machines report


The PowerShell script below helps you to create a simple overview of your Veeam Cloud Connect environment. The environment where this script is developed in consist of two Veeam Cloud Connect Backup servers. The first server will be used for the Cloud Connect Backup solution and the second server will be used for the Cloud Connect Replica server. To connect to these servers we use the Connect-VBRServer cmdlet. I personally love this new cmdlet. It just works like the Connect-VIServer cmdlet. The script will get all the Cloud Tenants and gathers usage information, for Backup tenants only, and will return the total number of virtual machines. At the end of the script all the information will be bundled into a html report. This report will be send to the recipients defined in the $MailReportTo variable.

Add-PSSnapin VeeamPSSnapIn
$AllVccServers = @("vccr-bs01.lab.loc","vccb-bs01.lab.loc")

$MailReportTo = @("user1@domain.com","user2@doman.com")
$MailReportSubject = "Veeam Cloud Connect - Total virtual machines report"
$MailReportFrom = "$(hostname)@domain.com"
$MailReportSmtpServer = "smtp.domain.com"

$VeeamTenantInfo = @()
foreach($VccBackupServer in $AllVccBackupServers){
    Write-Output "Connecting to Veeam Cloud Connect Backup Server $($VccBackupServer)"
    Connect-VBRServer -Server $VccBackupServer

    foreach($VBRCloudTenant in (Get-VBRCloudTenant | Sort Name)){
        
        $RepositoryQuota = $([math]::Truncate($VBRCloudTenant.Resources.RepositoryQuota / 1024))
        $UsedSpace = $([math]::Truncate($VBRCloudTenant.Resources.UsedSpace / 1024))
        $UsedSpacePercentage = $VBRCloudTenant.Resources.UsedSpacePercentage
        $Repository = $VBRCloudTenant.Resources.Repository.name
        $Leaseperiod = $VBRCloudTenant.LeaseExpirationDate


        if($Repository -eq $null -and $VBRCloudTenant.ReplicaCount -ne 0){
            $UsedSpace = 0
            $Repository = 0
            $RepositoryQuota = 0
            $UsedSpacePercentage = 0
            $Repository = "n.a."
        }
        
        if($Leaseperiod -eq $null){
            $Leaseperiod = "n.a."
        }

        $VeeamTenantInfo += New-Object PSObject -Property ([ordered]@{
            User = $VBRCloudTenant.Name
            Enabled = $VBRCloudTenant.Enabled
            VMCount = $VBRCloudTenant.VMCount
            ReplicaCount = $VBRCloudTenant.ReplicaCount
            RepositoryQuota = $RepositoryQuota
            UsedSpace = $UsedSpace
            UsedSpacePercentage = $UsedSpacePercentage
            Repository = $Repository 
            LeaseExpirationEnabled = $VBRCloudTenant.LeaseExpirationEnabled
            LeaseExpirationDate = $Leaseperiod
        })
    }
    Disconnect-VBRserver
}

$html = "<html><body><h1>Veeam Cloud Connect</h1><table border=1 cellspacing=0 cellpadding=3>"
$html += "<html><body><h2>Cloud Connect Server: $((hostname).ToUpper())</h2><table border=1 cellspacing=0 cellpadding=3>"
$html += "<html><body><h3>Usage report generated on $(Get-Date -Format g)</h3><table border=1 cellspacing=0 cellpadding=3>"
$html += "<tr>"
$html += "<th>User</th>"
$html += "<th>Enabled</th>"
$html += "<th>VMCount</th>"
$html += "<th>ReplicaCount</th>"
$html += "<th>RepositoryQuota in GB</th>"
$html += "<th>UsedSpace in GB</th>"
$html += "<th>UsedSpacePercentage</th>"
$html += "<th>Repository</th>"
$html += "<th>LeaseExpirationEnabled</th>"
$html += "<th>LeaseExpirationDate</th>"
$html += "</tr>"
foreach($veeamTenant in $VeeamTenantInfo){
    $html += "<tr>"
    $html += "<td>$($veeamTenant.User)</td>"
    $html += "<td>$($veeamTenant.Enabled)</td>"
    $html += "<td>$($veeamTenant.VMCount)</td>"
    $html += "<td>$($veeamTenant.ReplicaCount)</td>"
    $html += "<td>$($veeamTenant.RepositoryQuota)</td>"
    $html += "<td>$($veeamTenant.UsedSpace)</td>"
    $html += "<td>$($veeamTenant.UsedSpacePercentage)</td>"
    $html += "<td>$($veeamTenant.Repository)</td>"
    $html += "<td>$($veeamTenant.LeaseExpirationEnabled)</td>"
    $html += "<td>$($veeamTenant.LeaseExpirationDate)</td>"
    $html += "</tr>"
}
$html += "<tr>"
$html += "<td colspan=2><b>Totaal</b></td>"
$html += "<td>$(($VeeamTenantInfo.VMCount | Measure -Sum).sum)</td>"
$html += "<td>$(($VeeamTenantInfo.ReplicaCount | Measure -Sum).sum)</td>"
$html += "<td>$(($VeeamTenantInfo.RepositoryQuota | Measure -Sum).sum)</td>"
$html += "<td>$(($VeeamTenantInfo.UsedSpace | Measure -Sum).sum)</td>"
$html += "<td></td>"
$html += "<td></td>"
$html += "<td></td>"
$html += "<td></td>"
$html += "</tr>"
$html += "</table></body></html>"

$ScriptBaseName = ((gci ([string]$MyInvocation.InvocationName)).FullName) -replace "\.ps1$", ''
$html | Out-File "$($ScriptBaseName)_$(Get-date -Format yyyyMM).log.html"

try{
    if ($MailReportTo -eq $null){ throw [Exception]"No To address specified" }
    if ($MailReportSubject -eq $null) { throw [Exception]"No Subject specified" }
    if ($MailReportFrom -eq $null){ throw [Exception]"No From address specified" }
    if ($MailReportSmtpServer -eq $null) { throw [Exception]"No Smtp server specified" }
    Send-MailMessage -BodyAsHtml -Body $html -From $MailReportFrom -SmtpServer $MailReportSmtpServer -To $MailReportTo -Subject $MailReportSubject
    Write-Output "Mail message sent to $MailReportTo"
}
catch
{
    Write-Error "Error sending mail: $($_.Exception.Message)."
}

The report will look like this:

image

If you don’t want to use the html report. Use the code below to generate the report directly in the Powershell console.

Add-PSSnapin VeeamPSSnapIn
$AllVccServers = @("vccr-bs01.lab.loc","vccb-bs01.lab.loc")

$VeeamTenantInfo = @()
foreach($VccBackupServer in $AllVccBackupServers){
    Write-Output "Connecting to Veeam Cloud Connect Backup Server $($VccBackupServer)"
    Connect-VBRServer -Server $VccBackupServer

    foreach($VBRCloudTenant in (Get-VBRCloudTenant | Sort Name)){
        
        $RepositoryQuota = $([math]::Truncate($VBRCloudTenant.Resources.RepositoryQuota / 1024))
        $UsedSpace = $([math]::Truncate($VBRCloudTenant.Resources.UsedSpace / 1024))
        $UsedSpacePercentage = $VBRCloudTenant.Resources.UsedSpacePercentage
        $Repository = $VBRCloudTenant.Resources.Repository.name
        $Leaseperiod = $VBRCloudTenant.LeaseExpirationDate


        if($Repository -eq $null -and $VBRCloudTenant.ReplicaCount -ne 0){
            $UsedSpace = 0
            $Repository = 0
            $RepositoryQuota = 0
            $UsedSpacePercentage = 0
            $Repository = "n.a."
        }
        
        if($Leaseperiod -eq $null){
            $Leaseperiod = "n.a."
        }

        $VeeamTenantInfo += New-Object PSObject -Property ([ordered]@{
            User = $VBRCloudTenant.Name
            Enabled = $VBRCloudTenant.Enabled
            VMCount = $VBRCloudTenant.VMCount
            ReplicaCount = $VBRCloudTenant.ReplicaCount
            RepositoryQuota = $RepositoryQuota
            UsedSpace = $UsedSpace
            UsedSpacePercentage = $UsedSpacePercentage
            Repository = $Repository 
            LeaseExpirationEnabled = $VBRCloudTenant.LeaseExpirationEnabled
            LeaseExpirationDate = $Leaseperiod
        })
    }
    Disconnect-VBRserver
}
$VeeamTenantInfo

PowerCLI: Reconfigure for vSphere HA


Sometimes when you change something to the vSphere HA configuration like an Advanced Option, you have to reconfigure vSphere HA on each host inside that particular cluster. You can do that by hand via the Reconfigure for vSphere HA.. option inside the vSphere (Web) Client:

Screenshot 2016-01-22 13.11.47

Or you can use the following PowerCLI one-liner to perform this step on every host inside that cluster.

Get-Cluster <clusterName | Get-VMhost | Sort Name | %{$_.ExtensionData.ReconfigureHostForDAS()}

Just change the to the name of the cluster en open PowerCLI, connect to the vCenter server and run the one-liner.

Powershell: Change Veeam San Snapshot Options


In the case you want to use the Storage Snapshot feature with Veeam Backup, the feature will be enabled for all the Backup jobs and will be the default option to process a backup job.
Screenshot 2016-01-22 11.53.51

We only want to enable the feature for certain jobs with a special name convention like SAN-BackupJob-01. So I created a small Powershell script to disable this feature for all the jobs without the special name convention.

The Powershell script below will return all the jobs where the job name not match the “SAN” search string. The next step is to disable the “Use storage snapshots” feature for the jobs.

Script: Disable-SanIntegrationOptions.ps1:

if((Get-PSSnapin -Name VeeamPSSnapIn -ErrorAction SilentlyContinue) -eq $null){
    Add-PSSnapin "VeeamPSSnapIn"
}

foreach($vbrjob in (get-vbrjob | ?{$_.Name -notmatch "SAN"})){
    Write-Output "Changing job: $($vbrjob.name).."
    $SanIntegrataionOptions = $vbrjob.GetOptions()
    $SanIntegrataionOptions.SanIntegrationOptions.UseSanSnapshots = $false

    $vbrjob.SetOptions($SanIntegrataionOptions)
}

From now on only the jobs that need the SAN Snapshot feature, can utilise this feature and all the other jobs will use the default processing mode.

Powershell: Return Virtual Machines from PernixData FVP Blacklist


It’s a long time since my last blog post.  So it’s good to be back. In mean time we deployed PernixData FVP and it was fun to play with the Powershell module. 

You can import the PernixData FVP module installed on the PernixData FVP management server via:

Import-Module PrnxCli

This post is not about the Powershell module or how it works. You can check the following post by Byron Schaller.

In this post I want to share a Powershell script that can be used to report all the virtual machines currently added to the PernixData FVP blacklist.

The script below returns all the PernixData FVP clusters and for each cluster it will check the if there are virtual machines added to the Blacklist. If this is true, the script will return those virtual machines.

Import-Module PrnxCli Connect-PrnxServer <PRNXSERVER> $prnxVmBlacklistInfo = @() foreach ($fvpCluster in (Get-PrnxObject -Type FlashCluster | Sort-Object Name)) { Write-Host "Measuring Pernix FVP Cluster: $($fvpCluster.Name)" -ForegroundColor Yellow $BlacklistVMs = Get-PrnxObject -Cluster $fvpCluster.Name -Type "VM" | Where-Object {($_ | Get-PrnxAccelerationPolicy).CachePolicy -eq "Blacklisted"} foreach($vm in $BlacklistVMs){ $BlacklistInfo = New-Object PSObject -Property ([ordered]@{ FVPCluster = $fvpCluster.Name TotalFVPBlacklistVMs = $BlacklistVMs.Count VM = $vm.Name }) $prnxVmBlacklistInfo += $BlacklistInfo } }

 

 

 

$prnxVmBlacklistInfo

$timestamp = get-date -Format HH-mm-ss-dd-MM-yyyy $csv = $env:TEMP + $timestamp + "_PernixDataFVP_Blacklist.CSV" $prnxVmBlacklistInfo | Export-CSV -NoTypeInformation -UseCulture -Path $csv Disconnect-PrnxServer

The script exports the information gathered by the script to a CSV file inside the TEMP directory.

ssl-updater.bat: Cannot validate the Lookup Service connection: 3


Last week I was working on a change plan to update one of our vSphere environments with new SSL certificates on vSphere 5.1 installed on a Windows virtual machine. When I tried to update the SSL certificates of the vCenter Inventory service, I received the following error message:

image

In the vc-update-ssl.log log file I saw the following message:

Cannot authenticate user – Return code is InvalidCredentials 3

The first thing I thought was a typo when entering the password for the SSO Admin user. But after three times I stll got the same error. So the next step was logging into the vSphere web-client as admin@system-domain. But I was unable to logon because the associated user’s password is expired. That explains a lot.

image

Lucky me VMware has a KB article with a solution: KB2060150. The reason why the user password is expired, is because of the default password policy:

This issue occurs when the Admin@system-domain password has expired; the default is 365 days.

To resolve this issue you have to change the password policy and in particular the MAX_LIFE_SEC column in the RSA database.

Follow the steps described in KB2060150 to change the MAX_LIFE_SEC column.

To increase the MAX_LIFE_SEC column:

    1. Stop the vCenter Single Sign-on service (SSO).
    2. Log in to SQL Management Studio.
    3. Go to the RSA database.
    4. Expand Tables and highlight the dbo.IMS_AUTHN_PASSWORD_POLICY table.
    5. Right-click and select Edit Top 200 Rows.
       Scroll over to the MAX_LIFE_SEC column. The default setting is 31536000 seconds (365 days).

       Note: Select the policy that contains Password Policy for SSO system users within the NOTES  field.

    6. Increase this value (for example: 47304000 seconds = 546.5 days, 63072000 seconds = 730 days, 90000000 seconds = 1041days).
    7. Restart the vCenter Single Sign-on service.
       Once the service has started, log in the Web Client to vCenter Server with admin@system-domain:default URL is https://vCenter-server-fqdn:9443
       Navigate to Administration > Configuration.
    8. Click the Policies tab.
    9. Click Edit.
    10.Change maximum lifetime to 0 (never expire) or enter the approximate number of days corresponding to the value you set in the database, MAX_LIFE_SEC field above.
       Save your changes and exit the edit.

Note: Instead of steps 6 and 7, you can scroll to the column named PERIODIC_EXPIRE, and set that value to 0. This will prevent password expiration. You should only do this if your security policy allows non-expiring passwords.

When you succeed in step7, you can continue updating the SSL certificates with the ssl-updater.bat tool.