vCenter Cannot complete login due to an incorrect user name or password

Today I received weird request from my 2 of my peer colleagues, that they are not able to login to particular vCenter server and getting Cannot complete login due to an incorrect user name or password, when check with other users including me, were able to connect to the vCenter Server successfully without any hiccups, Also if affected users try with same username and password on other vCenter servers it works perfectly and have no trouble.


While isolating I found permissions are in place on affected vCenter server and indeed they are trying with correct username and password. This vCenter is an appliance and running version is 5.0.

When I checked messages log file under /var/log on the vCenter server (used grep command to find username for only related logs) then I found deny messages for affected users only.

cat /var/log/messages | grep userid



vpxd: pam_tally(vmware-authd:auth): user DOMAIN\user tally 9, deny 3
 
After some digging up on VMware kb I stumble upon, KB2008986, it was dictating similar symptom my colleagues were facing. Cannot login to the vCenter Server Appliance using the vSphere Client or vSphere Web Client after joining Active Directory (2008986)

As per the KB this happens because the deny 3 in the /var/log/messages file indicates that a maximum of 3 failed logins have occurred. After 3 failed logins, all subsequent log in attempts are denied. And below is the command to reset it
/sbin/pam_tally --user user@domain.com --reset

and after this my colleagues were able to connect to the vcenter successfully without any issue.

Regarding pam_tally more information can be found on http://linux.die.net/man/8/pam_tally

Powercli VMHost esxi server inventory

After successfully tweaking my VM Inventory script, I had written this VMHost inventory script, which helps me for quick info for Capacity Planning, CPU EVC mode, SSH service status and etc,

Below is the full list.

• Management IP and its VLAN
• Model
• ESXi Service Tag/ Serial No
• TotalVMs and PowerOn VMs count
• Esxi host CPU Sockets and Core per socket
• Total ESXi host CPU mhz and Logical CPU
• Esxi memory and assigned to VMs
• MAX-EVC-Key (This is helpful when setting up EVC mode on the vmware cluster)
• ESXi uptime
• Domain, syslog and dump collector settings info
• DRacIP and RackLocation (You will have to fillup onetime custom attributes (annotation))

Checkout below screenshot

 function Get-VMHostinventory {   
   foreach ($vmhost in Get-VMHost) {  
     Write-host $vmhost.Name  
  #####################################    
  ## http://kunaludapi.blogspot.com    
  ## Version: 1    
  ## Tested this script on successfully    
  ## 1) Powershell v4  
  ## 2) Powercli v5.5    
  ## 2) Windows 7   
  ## 3) vSphere 5.5 (vcenter, esxi, powercli)   
  #####################################  
     if ($vmhost.Version -ne "4.1.0") {  
       $esxcli = $vmhost | Get-EsxCli  
       $serviceTag = $esxcli.hardware.platform.get().SerialNumber  
     }  
        else {  
             $serviceTag = $vmhost.ExtensionData.summary.hardware.otheridentifyinginfo | select-object -ExpandProperty IdentifierValue -last 1  
        }  
             
     #Esxihost Management IP and vlan ID  
     $Managementinfo = $vmhost | Get-VMHostNetworkAdapter | Where-Object {$_.ManagementTrafficEnabled -eq $true}  
     $VirtualPortGroup = $vmhost | Get-VirtualPortGroup  
        $IPinfo = $Managementinfo | select-object -ExpandProperty ip  
     $ManagementPortGroup = $Managementinfo.extensiondata.spec  
     $ManagementIP = $IPinfo -join ", "  
       
     $MulitvLans = @()  
     if ($ManagementPortGroup.DistributedVirtualPort -ne $null) {  
       $vLanIDinfo = $VirtualPortGroup | Where-Object {$Managementinfo.PortGroupName -contains $_.name}  
       foreach ($MGMTVlan in $vLanIDinfo) {  
         $MulitvLans += $MGMTVlan.ExtensionData.config.DefaultPortConfig.Vlan.VlanId  
       }  
       $vLanID = $MulitvLans -join ", "  
     }  
     else {  
       $vLanIDinfo = $VirtualPortGroup | Where-Object {$ManagementPortGroup.Portgroup -contains $_.name } | Select-Object -ExpandProperty VLanId  
       foreach ($MGMTVlan in $vLanIDinfo) {  
         $MulitvLans += $MGMTVlan  
       }  
     $vLanID = $MulitvLans -join ", "  
     }  
       
     #EsxiHost CPU info  
     $HostCPU = $vmhost.ExtensionData.Summary.Hardware.NumCpuPkgs  
     $HostCPUcore = $vmhost.ExtensionData.Summary.Hardware.NumCpuCores/$HostCPU  
   
     #All Virtual Machines Info  
     $VMs = $vmhost | Get-VM   
     $PoweredOnVM = $VMs | Where-Object {$_.PowerState -eq "PoweredOn"}  
   
     #EsxiHost and VM -- CPU calculation  
     $AssignedTotalvCPU = $VMs | Measure-Object NumCpu -Sum | Select-Object -ExpandProperty sum  
     $PoweredOnvCPU = $PoweredOnVM | Measure-Object NumCpu -Sum | Select-Object -ExpandProperty sum  
     $onecoreMhz = $vmhost.CPUTotalMhz / $vmhost.NumCpu  
     $TotalPoweredOnMhz = $onecoreMhz * $PoweredOnvCPU  
       
     #EsxiHost and VM -- Memory calculation  
     $TotalMemory = [math]::round($vmhost.MemoryTotalGB)  
     $Calulatedvmmemory = $VMs | Measure-Object MemoryGB -sum | Select-Object -ExpandProperty sum  
     $TotalvmMemory = [math]::round($Calulatedvmmemory)  
     $Calulatedvmmemory = $PoweredOnVM | Measure-Object MemoryGB -sum | Select-Object -ExpandProperty sum  
     $PoweredOn_vMemory = "{0:N2}" -f $Calulatedvmmemory  
   
     #EsxiHost Domain Details  
     $domain = ($vmhost | Get-VMHostAuthentication).Domain  
   
     #Cluster and Datstore info  
     $Clusterinfo = $vmhost | Get-Cluster  
     $Clustername = $Clusterinfo.Name  
     $DataCenterinfo = Get-DataCenter -VMHost $VMHost.Name  
     $Datacentername = $DataCenterinfo.Name  
   
     #vCenterinfo  
     $vCenter = $vmhost.ExtensionData.CLient.ServiceUrl.Split('/:')[3]  
     $vcenterversion = $global:DefaultVIServers | where {$_.Name -eq $vCenter} | %{"$($_.Version) build $($_.Build)"}  
   
     #vmhost SSH service Staus  
     $SSHservice = $vmhost | Get-VMHostService | Where-object {$_.key -eq "Tsm-ssh"} | Select-Object -ExpandProperty running  
   
     #vmhost Uptime  
     $UPtime = (Get-Date) - ($vmhost.ExtensionData.Runtime.BootTime) | Select-Object -ExpandProperty days  
   
     #vmhost syslog server settings  
     if ($vmhost.Version -ne "4.1.0") {  
       $syslog = ($vmhost | Get-AdvancedSetting -Name Syslog.global.logHost).value  
     }  
     else {$syslog = "Not Supported"}  
           
       
     #vmhost Dump collector  
     $DumpCollector = $esxcli.system.coredump.network.get().NetworkServerIP  
   
     $VmHostresult = New-Object PSObject   
     $VmHostresult | add-member -MemberType NoteProperty -Name "Name" -Value $vmhost.Name  
     $VmHostresult | add-member -MemberType NoteProperty -Name "Management IP" -Value $ManagementIP  
     $VmHostresult | add-member -MemberType NoteProperty -Name "vLan ID" -Value $vlanID  
     $VmHostresult | add-member -MemberType NoteProperty -Name "PowerState" -Value $vmhost.PowerState  
     $VmHostresult | add-member -MemberType NoteProperty -Name "Manufacturer" -Value $vmhost.Manufacturer  
     $VmHostresult | add-member -MemberType NoteProperty -Name "Model" -Value $vmhost.Model  
     $VmHostresult | add-member -MemberType NoteProperty -Name "Service_Tag" -Value $serviceTag  
     $VMHostresult | add-member -MemberType NoteProperty -Name "TotalVms" -Value $VMs.count  
     $VMHostresult | add-member -MemberType NoteProperty -Name "PoweronVMs" -Value $PoweredOnvm.Count  
     $VmHostresult | add-member -MemberType NoteProperty -Name "ProcessorType" -Value $VMHost.ProcessorType  
     $VmHostresult | add-member -MemberType NoteProperty -Name "CPU_Sockets" -Value $HostCPU  
     $VmHostresult | add-member -MemberType NoteProperty -Name "CPU_core_per_socket" -Value $HostCPUcore  
     $VmHostresult | add-member -MemberType NoteProperty -Name "Logical_CPUs" -Value $vmhost.Numcpu  
     $VmHostresult | add-member -MemberType NoteProperty -Name "TotalHost_Mhz" -Value $vmhost.CPUTotalMhz  
     $VmHostresult | add-member -MemberType NoteProperty -Name "AssignedTotal_vCPUs" -Value $AssignedTotalvCPU  
     $VmHostresult | add-member -MemberType NoteProperty -Name "PoweredOn_vCPUs" -Value $PoweredOnvCPU  
     $VmHostresult | add-member -MemberType NoteProperty -Name "PoweredOn_Mhz" -Value $TotalPoweredOnMhz  
     $VmHostresult | add-member -MemberType NoteProperty -Name "Memory(GB)" -Value $TotalMemory  
     $VmHostresult | add-member -MemberType NoteProperty -Name "AssignedTotal-vMemory(GB)" -Value $TotalvmMemory  
     $VmHostresult | add-member -MemberType NoteProperty -Name "PoweredOn-vMemory(GB)" -Value $PoweredOn_vMemory  
     $VmHostresult | add-member -MemberType NoteProperty -Name "Esxi-Version" -Value $vmhost.Version  
     $VmHostresult | add-member -MemberType NoteProperty -Name "Build-Number" -Value $vmhost.Build  
     $VmHostresult | add-member -MemberType NoteProperty -Name "Domain" -Value $domain  
     $VmHostresult | add-member -MemberType NoteProperty -Name "Max-EVC-Key" -Value $vmhost.ExtensionData.Summary.MaxEVCModeKey  
     $VmHostresult | add-member -MemberType NoteProperty -Name "Cluster" -Value $ClusterName  
     $VmHostresult | add-member -MemberType NoteProperty -Name "DataCenter" -Value $DatacenterName  
     $VmHostresult | add-member -MemberType NoteProperty -Name "vCenter Server" -Value $vcenter  
     $VmHostresult | add-member -MemberType NoteProperty -Name "vCenter version" -Value $vcenterversion  
     $VMHostresult | add-member -MemberType NoteProperty -Name "Esxi-status" -Value $vmhost.ExtensionData.Summary.OverallStatus  
     $VMHostresult | add-member -MemberType NoteProperty -Name "Physical-Nics" -Value $vmhost.ExtensionData.summary.hardware.NumNics  
     $VMHostresult | add-member -MemberType NoteProperty -Name "SSH-Enabled" -Value $SSHservice  
     $VMHostresult | add-member -MemberType NoteProperty -Name "Uptime" -Value $UPtime  
     $VMHostresult | add-member -MemberType NoteProperty -Name "Syslog-Server" -Value $syslog  
     $VMHostresult | add-member -MemberType NoteProperty -Name "Dump-Collector" -Value $DumpCollector  
     $VmHostresult   
   }  
 }  
 Get-VMHostinventory   

#To write information to csv file
#Get-VMHostinventory | export-csv -path c:\vmhostlist.csv

 
Other helpful inventories 
Extra Extended VM inventory using powercli - Part 2
VMware Datastores inventory powercli

Other useful blogs
Schedule PowerShell (.PS1 file) script in Windows Task Scheduler 

Manage vCenter server appliance AD authetication from commandline

Recently I faced some issue with my LAB AD and due to this my vmware infrastructure disturbed. While troubleshooting I came across tool vCenter Appliance called domainjoin-cli located under /opt/likewise/bin folder. With this tool you can manage AD authentication settings from command line. Below are the some of the screenshots how the command works.
When you CD to the directory and run this command you will see the standard help and its parameters how to use it.


Below is the query result when computer account for vCenter appliance was deleted from AD.
Error: LW_ERROR_KRB5KDC_ERR_C_PRINCIPAL_UNKNOWN [code 0x0000a309]
client not found in Kerberos database


To correct this I manually created new computer account in AD (New Computer account SID (Password) is not matching with vCenter server). Which resulted into below error.
Error: LW_ERROR_PASSWORD_MISMATCH [code 0x00009c56]
The password is incorrect for the given username

 

And this is the one when my AD server was down
Error:LW_ERROR_DOMAIN_IS_OFFLINE [code 0x00009cb9]
The domain is offline

 
You can disjoin vCenter from AD with ./domainjoin-cli leave


This is query status after dis-joining from AD.


When in the last I rejoined it in AD, this the query status (you can use ./domainjoin-cli join command to do the same join this VC in domain.)