Symptom: You start seeing the alarm “vSphere Health detected new issues in your environment” and it won’t go away.
I wrote about this alarm with one cause: Memory Exhaustion with a Tiny deployment in my other blog. If you navigate to your vCenter appliance website (https://vcenter.company.com:5480) and see memory warnings, check the fix in that blog first.
Symptom: You recently upgraded to vCenter or vSphere ESXi 6.7 U2 (Update 2, April 2019, May 2019)
Symptom: Warning in event logs “Alarm ‘vSphere Health detected new issues in your environment’ on Datacenters changed from Green to Yellow”
Symptom: Warning in event logs: “event.vsphere.online.health.alarm.event.fullFormat (vsphere.online.health.alarm.event)
Symptom: You don’t see anything to explain the issue in the logs. Looks like a false positive?
Symptom: When you navigate to vCenter > Monitor > Health, there is no health tab.
This is the main symptom for this particular issue. Read on!
Root Cause #1: You are still using the Flash vSphere client from version 6.0 and 6.5.
You need to change the URL you are using for vSphere and vCenter: https://vCenter.company.com/ui
You can find this URL from scratch by navigating directly to your vCenter: https://vCenter.company.com and clicking the HTML5 button
You can also find it right at the top of your vSphere website – Look for a button that says “Launch vSphere Client (HTML5)”
Now that you’ve launched the HTML5 site, you will notice that it looks way different!
Root Cause #2: The latest updates for vCenter and vSphere include new checks for common issues.
The April 2019 and May 2019 release of 6.7 Update 2 include new health checks. Your vCenter will now warn you about things like problematic drivers and known memory leaks.
These checks are only visible in the HTML5 client. This is why you couldn’t find the cause of the alert before. Read on for how to find them.
These checks are also handled by the Customer Experience Improvement Program (VMware CEIP). If you are a typical business (not at high risk from cyber-attack), the CEIP program is highly recommended. If you are at risk from cyber-attack, there are ways to secure the CEIP connection so you can still use it.
How to troubleshoot the cause of vSphere Health detected new issue in VMware 6.7
When you open the HTML5 vSphere Client, you will now be able to navigate to Monitor > Health and see what is causing your health alarm.
Using the instructions above, open your HTML5 vSphere client by navigating to https://vcenter.company.com/ui
Select your vCenter object in Hosts & Clusters view. (This is the top level object in your tree)
Click the Monitor button from the middle menu.
Click the Health button from the middle-middle menu.
Identify warnings that have yellow exclamation marks next to them. These are causing your health alarm.
You can click each item to view information about them. If you select the Info tab for that problem, you will see a button for “Ask VMware” which gives additional help.
Click the RETEST button on the top-right of the window to see if the issue still exists.
How do I enable CEIP for VMware?
From the vSphere HTML5 Client, click the Menu drop-down button
Navigate to Deployment > Customer Experience Improvement Program
Click Join…
This VMware blog has a nice video of how to click through and enable CEIP if you are having trouble.
How do I fix “Enable SCAv2 for optimal hyperthreading performance”?
This is a continuation of the SPECTRE/MELTDOWN or “L1 Terminal Fault” issue that you’ve heard about.
WARNING: VMware default settings are for highest performance. If you make changes to increase security against SPECTRE / MELTDOWN, your performance may be impacted significantly! In other words, if your virtual environment is using more than 20% CPU at any given time, you should probably NOT enable these changes without a lot of research.
You probably have already applied the fix for previous versions of vSphere. The fix was to edit Advanced System Settings for each host and change the value of VMkernel.Boot.hyperthreadingMitigation = true
In 6.7 Update 2 and later, VMware added VMkernel.Boot.hyperthreadingMitigationIntraVM which defaults to true.
To enable SCAv2, you would verify that VMkernel.Boot.hyperthreadingMitigation = true and change the VMkernel.Boot.hyperthreadingMitigationIntraVM = false and reboot each host.
This setting can be reached by opening vSphere Client website (https://vcenter.company.com/ui) then select Hosts & Clusters view, then select a host. Click the Configure tab and select Advanced System Settings from the middle menu. Repeat for each host.
How do I fix “ESXi with a problematic driver for Gigabit network adapter”?
Follow the Ask VMware link on the alert to find specific information about your problematic network card.
It will open a VMware KB article and probably recommend installing an updated driver.
To update to a new driver, here are the basic steps… please use caution and common sense!
Download the VIB file from VMware
While you are at it, download the README and review it. If it has instructions, follow those.
If it is in a .zip format, unzip it and find the .vib file
Move your VMs to a different host if possible.
Put your ESXi host into maintenance mode (this procedure could cause impact to any running VMs)
Back up your ESXi host configuration if you still have any VMs on it (in other words, you can’t afford to rebuild it if something goes wrong).
Start SSH service in your host > Configuration > Security Profile menu.
Using WinSCP or another reliable SCP client, connect to your host using IP and root / (root password)
Navigate to the /tmp/ directory and upload the VIB file to that directory.
Using Putty or another reliable SSH / console client, connect to your host using IP and root / (root password)
If your VIB doesn’t say “offline bundle”, type esxcli software vib update -v \tmp\NameOfVIBFile.vib
If your VIB says “offline bundle”, type esxcli software vib update -d \tmp\NameOfVIBFile-offline_bundle.vib
Read the results.
If the the result says “Reboot required: true” , then type reboot (this will reboot your host)
Make sure to test your host with a non-critical VM before moving important VMs to it.
How do I fix “Concurrent-context attack vector vulnerability in Intel processors”?
This error is referring to the “L1 Terminal Fault” which is widely known as SPECTRE / MELTDOWN.
Basically, there is a flaw in all Intel Processors (at least as of late 2018) which allows processes running in the operating system to observe what the CPU is doing with other processes. This is a critical vulnerability for cloud hosts or any servers that allow untrusted users to access them.
L1 Terminal Fault a major concern for cloud hosting companies, not on-premises companies
For example, if you have an account on AWS, your virtual servers are running on the same physical hardware as other people’s virtual servers. If this vulnerability isn’t mitigated, then you could potentially write code to steal data from the other customers, or vice-versa.
To my knowledge, the vulnerability cannot be exploited without running a process on the system, and most of the people who run processes on servers have no need to snoop on the CPU. In other words, if all the other admins on your server work at your company, you should be fine.
What is the fix?
For now, while the physical processors have this flaw, the fix is to logically reduce the hyper-threading capability of Intel CPUs so they can’t be snooped on. This removes 5-20% of the performance capacity of the CPU.
If your VMware environment isn’t really using the CPU (peak CPU on your hosts is less than 30%), go ahead and implement the fix!
If your servers ARE using the CPU intensively (peak CPU is greater than 30%), then think hard before making a change.
To implement this fix, edit Advanced System Settings for each host and change the value of VMkernel.Boot.hyperthreadingMitigation = true , then reboot the host. Since you are already at 6.7 Update 2, your health alarm will probably change to “Enable SCAv2 for optimal hyperthreading performance” which is addressed a few sections above this one.
What if I don’t want to fix concurrent context?
Some environments cannot afford to lose the CPU performance. For example, I have a client that runs a lab environment with extremely high processing requirements. The hosts are running 70%+ CPU constantly.
So how can you remove the vSphere health warning about concurrent-context attack vector?
Event: “Alarm ‘vSphere Health detected new issues in your environment’ on Datacenters changed from Green to Yellow
Even on healthy vCenters, you will see this event appear about once a week. In my environments it lasts for about one hour (green to yellow, then yellow to green). It doesn’t appear to be an actual issue.
Selfish plug time – Need help?
I am a consultant in the Maryland/DC area in the USA. My specialties are Windows migrations (to 2016 and to Office 365 / Azure), VMware migrations, Netapp and SAN, and high availability / disaster recovery planning. If you would like help with your complex project, training, or would like a architectural review to improve your availability, please reach out! More information and contact can be found on the About page. – Amira Armond
Symptom: Your vCenter appliance was deployed using “Tiny” resource sizing of 10 GB RAM and 2 vCPU.
Symptom: Your vCenter is version 6.7
Symptom: When you open your vCenter Appliance Management website at :5480, you see an alert in the Memory area.
Symptom: You may see occasional alarms stating “Memory Exhaustion on vCenter”
Symptom: You may see this in your event log: “event.vsphere.online.health.alarm.event.fullFormat (vsphere.online.health.alarm.event)”
Symptom: You see constant alarms in vSphere Web Client stating “vSphere Health detected new issues in your environment” referencing your vCenter server.
Root cause for memory error with Tiny deployment:
The memory issue may be a bug specific to the 6.7.0 Update 2a release of vCenter 6.7 (6.7.0-13643870) April 2019, which is the first place I’ve seen it.
In general, the memory warning can be ignored as long as your vCenter is working properly. However, it is very annoying.
Root cause for constant vSphere health alarm (not memory)
VMware added new functionality in vCenter 6.7 update 2 to perform preventative health checks using the Customer Experience Improvement Program.
Almost everyone who upgrades to 6.7 will get constant warnings and alerts saying “vSphere health detected new issues in your environment” because of these new health checks.
Honestly, I recommend just ignoring this alert. I think the ultimate fix will be a patch from VMware.
If you assign 12 GB of RAM to your vCenter, the alert should go away.
Once you make the change, your vCenter appliance dashboard will go green for memory.
The steps to assign more RAM are as follows.
Easy mode: If you have purchased the right VMware licenses, you can simply edit settings on your vCenter appliance VM and assign it more RAM. If you don’t know if you have the licenses, try it anyways. Worst that happens is you will see an error in the task area.
How to assign more RAM to vCenter appliance if you don’t have the hot-pluggable virtual hardware license.
Connect to the vSphere website for the host that is running vCenter
Connect to the management website for your vCenter appliance ( https://vcenter:5480)
In the management website for vCenter, click Actions > Shutdown.
Monitor the vSphere website on your host until you see vCenter powered off.
Right-click vCenter and select Edit Settings.
Modify the Memory to 12288 MB (or 12 GB) and click OK.
Thanks for reading this article! I hope it helps you! If you have tips or feedback, please comment or send me an email so that others can benefit.
I am a consultant in the Maryland/DC area in the USA. My specialties are Windows migrations (to 2016 and to Office 365 / Azure), VMware migrations, Netapp and SAN, and high availability / disaster recovery planning. If you would like help with your complex project, training, or would like a architectural review to improve your availability, please reach out! More information and contact can be found on the About page. – Amira Armond
These screenshots show the entire process to upgrade vCenter from 6.0 to 6.7 using a Windows 10 desktop as my administrative workstation. The same steps work for vCenter 6.5 to 6.7.
Is your existing vCenter server running on Windows? This article is primarily about upgrading vCenter appliances (linux-based). If you have a Windows vCenter server, check the comments for instructions from Greg Curry. Thanks Greg!
Preparing for the vCenter appliance upgrade
First, create a snapshot for your existing vCenter server. If you have disk space available, make sure to Quiesce memory.Log on to vmware.com and navigate to the Downloads section. Pick vCenter Server from the list. (It does not matter if you choose Essentials, Standard, etc, they are all the same).Verify the version you want to use (it will default to the latest one), and click Download for VMware vCenter Server Appliance, file type = iso. Now wait for a bit for it to download.Navigate to your downloads folder when the download completes. In Windows 10, you can double-click the file to mount it to a virtual DVD drive. For non Windows workstations, use an .iso viewer or burn the image to a CD then open it on the CD.Navigate to the contents of your .iso file. On Windows 10, use Windows Explorer and look for the files inside your DVD drive. Then expand vcsa-ui-installer to open that folder.Expand the folder that matches your computer (Windows 7, 8, and 10, pick win32)(Windows 7, 8, 10) Run the installer.exe file to start the upgrade wizard.
Appliance Upgrade Stage 1
Options display. For this scenario, pick Upgrade (to upgrade an existing vCenter appliance from 6.0 or 6.5 to 6.7)The VCSA introduction screen displays.Agreement displays. You need to accept it to proceed.Enter the DNS name for your existing vCenter server. You can look this up in the address bar when you open vCenter to manage your VMware environment. Port 443 is appropriate for almost everyone, unless the IT department has customized this. When you Connect to Source, the rest of the window will display.Most environments will use administrator@vsphere.local for the SSO User name. SSO password is the password for the user name you entered above (administrator@vsphere.local). Appliance (OS) root password means the root password for your vCenter appliance website (https://vcenter01.company.com:5480). See next step to identify which ESXi host is managing your appliance.
——————
Additional info from Todd who provided this quote in the comments (Thanks Todd!)
I ran into one minor issue during the “Upgrade Stage 1: Deploy Appliance” phase. Operation would halt with “Failed to authenticate with the guest operating system using the supplied credentials.” I know they’re good as I can login to VCSA, VAMI, MOB, ESXi using them.
I changed all of the passwords to numbers and letters only with a single known good special character tacked on at the end. Et Voila! Problem solved. The rest of the upgrade proceeds without errors.
Passwords like these will not work with this utility:
Tv5$8FG#m*Djn 7xjg8sP%C#usXH8H X8xk$vR^q9ccw@WxR
———————–
In the VMware hosts & clusters view, if you select your existing vCenter server and go to the Summary tab, you can see the address of the ESXi host.You will get prompted to review the certificate. Just click Yes.Now enter the ESXi server that will host your upgraded vCenter server. Yes, we are making a new vCenter server with this procedure. The old vCenter will be powered off if the upgrade succeeds. Make sure you pick a host that has plenty of RAM and disk (at least 12 GB RAM, at least 500 GB disk) available.This is where you set the name and password for your new vCenter. The name will be displayed in VMware Hosts & Clusters. This is not the DNS name or IP address for your new vCenter. Your new vCenter will inherit the existing DNS name and IP address since we are doing an upgrade.Almost all medium and small businesses use Tiny for deployment size and Large for storage size. If you have 10 or less ESXi servers, use Tiny.Pick the datastore that will host your new vCenter server. Remember that by default it will use 850 GB thick provisioned for a Tiny deployment.
Optional: Check the “Enable Thin Disk Mode” so that your vCenter server uses less disk space (about 80 GB). This is mildly dangerous because the usage can grow over time and cause a datastore to fill up (crashing all VMs on the datastore), but for most small environments it doesn’t grow more than a few GB per year and 850 GB is WAY more than you need.
These are the network settings that your new vCenter server will use. Make sure to pick a Network that both your existing vCenter and the new vCenter can communicate on, with their IP addresses. The new vCenter will use the Temporary IP address for a few minutes during the upgrade process, then it will switch to the existing IP address of your vCenter server. Make sure no other devices are using the Temporary IP address.Ready to complete displays. When you click Finish, the deployment will start.Even across WAN links, this process takes less than an hour. Make sure that your workstation doesn’t disconnect from the work network during this process. (If it does disconnect, no problem, just re-start from the beginning.)
Errors that can occur during stage 1
If you used the wrong password for the old vCenter, the source host, or the destination host, you may get an error at this point.
If you are on an unstable VPN link, you may get an error. Try running the upgrade from a workstation or server on the LAN.
If you used DNS instead of IP addresses, try changing to IP addresses.
Temporary IP error: The upgrade will attempt to ping the IP address you chose for the temporary vCenter IP address. If it pings, the upgrade will fail. Make sure nothing is using that IP.
For information about cleaning up failed upgrade steps, see troubleshooting at the bottom of the article.
Stage 2 Upgrade VCSA
Phase 1 shows complete. The new vCenter virtual machine is deployed, but not configured. Beginning Phase 2 of the upgrade.Introduction for Stage 2 displays. Click Next.It is normal to get several warnings. Errors are bad, and will cause your upgrade to fail. Warnings are not bad. Read them and take action if appropriate.Most small and medium businesses don’t have that much data in their vCenter. In this example, there is only 3.18 GB of data to transfer. Since there is not a big difference between the sizes of each choice, I choose to move over everything. It is good to retain historical data when possible.Most businesses are fine to join the CEIP. If you are working on a secure network, then don’t check it.Ready to complete displays. You will need to check that you have backed up the source vCenter server (you did, right?). Ways to perform backups: 1) Veeam. 2) Create that snapshot at the beginning of this post. 3) Use the appliance website (https://vcenter.company.com:5480) and click the Backup button (you will need a web server or SCP host to do this). The reality is that this upgrade process doesn’t delete your original server. At the worst, it powers it off. There is very little risk of harming your original vCenter.Phase 2 proceeds. It should finish within about 30 minutes.
Possible errors during phase 2
NTP error: Make sure your source VCSA has good NTP settings which are the same as the host you are using for source and destination. See this VMware article about configuring NTP. https://kb.vmware.com/s/article/57146 If you don’t have an internal time server, then the recommended option is to point to internet NTP servers: 0.vmware.pool.ntp.org ; 1.vmware.pool.ntp.org ; 2.vmware.pool.ntp.org
Success!!
At this point, your upgrade should be successful. When the upgrade wizard completes, your old vCenter will be powered off and your new vCenter (running 6.7) will be powered on.
You may want to rename your old vCenter to something like “old_vCenter01”.
Test your ability to log onto the vCenter appliance (https://vcenter01.company.com:5480)
Test your ability to manage your virtual machines (https://vcenter01.company.com/vsphere-client)
Make sure to check your backup jobs. They should switch to using the new vCenter automatically, but double-check.
Once you are comfortable with the new vCenter and you have at least one good backup of it, you can delete your old one.
Remember that if the new vCenter doesn’t work, you can revert your changes simply by powering it off and powering the new vCenter on. You may need to log into individual hosts (https://host_ip_address) to do this if your vCenter isn’t working.
Troubleshooting:
Manually verify every password for each device (your existing vCenter administrator@vsphere.local, your existing vCenter root, the source ESXi host root, the destination ESXi host root). It is common for the vCenter root to be expired. See my other blog for easy steps to fix an expired vCenter root password.
Orphaned vCenter attempts: If your deployment fails, the new vCenter may display as orphaned. There is no impact at this time, because you will continue using your existing vCenter. I’ve heard that using VMware Workstation to connect to your vCenter will allow removal. There is a blog that describes various methods for dealing with orphaned VMs: https://www.altaro.com/vmware/how-to-deal-with-orphaned-virtual-machines/ For a more authoritative source, see VMware KB article for removal steps: https://kb.vmware.com/s/article/1003742#vmdeleted
Thanks for reading this article! I hope it helps you! If you have tips or feedback, please comment or send me an email so that others can benefit.
I am a consultant in the Maryland/DC area in the USA. My specialties are Windows migrations (to 2016 and to Office 365 / Azure), VMware migrations, Netapp and SAN, and high availability / disaster recovery planning. If you would like help with your complex project, or would like a architectural review to improve your availability, please reach out! More information and contact can be found on the About page. – Amira Armond
Your vCenter 6.5 or 6.7 server automatically expired your root password
You left the default setting to expire the root password in your vCenter Appliance
When you try to log on to the management website :5480 with root, it says the password is expired.
You know what the root password is.
How to fix:
Open console to your VCSA by logging on to the vSphere ESXi server that is hosting it
Press F2 to configure your vCenter appliance
When prompted, type in your root password (it should work, even if expired)
Note: Even though there is an option here to change your root password, it doesn’t seem to work when root is expired.
Go to troubleshooting options and enable BASH and enable SSH
Connect to your VCSA using SSH – logging on with root and your expired password will work.
Select BASH shell from the menu
type “passwd” and press Enter
If you want to keep your current root password, just enter it twice here.
If you want to use a new root password, enter it twice here.
Verify it works by opening the management website :5480 to your vCenter server and logging on.
Prevent the root password from expiring again
To prevent the root password from expiring again, log onto vCenter management website ( :5480), go to Administration menu, and change the password settings here.
If you use a long, complex password, there is no reason to automatically expire it. Long and complex means 14+ characters!
Selfish plug time (sorry!)
Thanks for reading this article! I hope it helps you! If you have tips or feedback, please comment or send me an email so that others can benefit.
I am a consultant in the Maryland/DC area in the USA. My specialties are Windows migrations (to 2016 and to Office 365 / Azure), VMware migrations, Netapp and SAN, and high availability / disaster recovery planning. If you would like help with your complex project, or would like a architectural review to improve your availability, please reach out! More information and contact can be found on the About page. – Amira Armond
When attempt to add a new vSphere host to vCenter, an error appears after entering the host credentials: “A general runtime error occurred”
You clicked YES on the certificate prompt.
You have a cluster with EVC enabled, and you are attempting to add the host to that cluster
You are using vCenter 6.0 , 6.5, or 6.7
How to add your new host to vCenter 6.0 , 6.5, or 6.7:
Instead of right-clicking your Cluster object and add host, you should right-click your Datacenter object (the item that is one level higher in your menu tree) and add host.
Right-click your Datacenter object (icon looks like two buildings)
Add Host
Enter the host name (fully qualified DNS name), or the IP address).
Click Next
Enter the credentials to the host (normally root + the password you set when you installed ESXi)
Click Next
Certificate prompt appears asking whether you want to replace the certificate for the host. Click Yes.
At this point, the wizard should proceed to the next step (Host Summary).
Click Next
VM location will display. If you have any VMs on the new host, you can select where they will be logically grouped.
Click Next.
Ready to complete displays.
Click Finish.
Root cause for “general runtime error” when adding host:
Your host has hardware incompatibilities with the EVC level selected in your cluster. Most likely, you will need to reduce the CPU generation to an older model to account for the age of your new host.
You can prove this theory by trying to move your new host to the cluster. It will probably give you an error like this:
If you want to add the host to your EVC-enabled cluster, you will need to research CPU generation levels for your server hardware, to find the common denominator.
Since you already have hosts in the cluster, you will want to create a new cluster with a compatible EVC level, and move your hosts into it one by one. Alternately, you can downgrade your current cluster by moving your hosts out, then back in. Both methods involve downtime for your VMs.
Move each of your hosts out of the cluster and to the Datacenter level
Right-click the cluster and Edit Settings, then select the “VMware EVC” tab
Enable EVC, and select the lowest (oldest) CPU generation for all of your hosts.
One by one, move your hosts into the cluster. If your host has VMs, you will need to put it into maintenance mode first (powering off the VMs during the move).
Thanks for reading this article! I hope it helps you! If you have tips or feedback, please comment or send me an email so that others can benefit.
I am a consultant in the Maryland/DC area in the USA. My specialties are Windows migrations (to 2016 and to Office 365 / Azure), VMware migrations, Netapp and SAN, and high availability / disaster recovery planning. If you would like help with your complex project, training, or would like a architectural review to improve your availability, please reach out! More information and contact can be found on the About page. – Amira Armond
Many websites will say that vCenter 6x requires DNS lookup, but these steps and screenshots prove that you can install a new vCenter 6.7 with only an IP address and no DNS.
Start vCenter install from .iso file using Windows 10
1. Download the install .iso file for your vCenter (available from the downloads area of vmware.com. On a Windows PC (I am using Windows 10), double-click the .iso file to mount it. It will show up in a virtual DVD drive.2. Open your virtual DVD drive and expand the vcsa-ui-installer folder.3. Expand the win32 folder (for a Windows PC). If you are using Linux or Mac as your desktop, select the appropriate folder.4. Run the installer.exe file as administrator (on Windows). This will start the program that allows you to deploy, upgrade, or migrate vCenters.
vCenter install stage 1
5. Initial options display. In our case, we are installing a brand new vCenter with no DNS, only IP address. So pick Install.6. Go through the introduction and end use license agreement. For the deployment type, select Embedded Platform Services Controller (this is normally the correct choice unless you have a very large, complex environment)7. Normally you will be installing a new vCenter directly onto an ESXi host. Put the IP address of your ESXi host into the “ESXi host or vCenter Server name”. Leave the port 443, enter the correct root account and password for that ESXi host. 8. Configure options for “set up appliance VM” — this gives the name for the VM on your host (and will affect naming of the datastore folders). Set the root password for the vCenter appliance here. Select deployment size – for most situations, the default of Tiny is fine unless you have more than 10 hosts. Select datastore – pick the datastore to create the VM into.9. Configure network settings: Here is the important part. Network should be a virtual port group (switch) that your desktop can communicate with. IP assignment should be static. In the FQDN field, enter the IP address of your vCenter server. In the IP address field, enter the IP address of your vCenter server. Enter the rest of the fields as appropriate for your environment. If you don’t have a DNS server at all, use your gateway or another bogus IP.10. Confirm your settings and start deployment. For me, I was able to complete Stage 1 in about 15 minutes, despite using a slow (100mbps) network.
vCenter installation stage 2
11. Stage 2 starts. If you have an NTP server, set it here, otherwise make sure your ESXi host has correct time and use it.12. Most deployments create a new SSO domain.13. Enter “vsphere.local” into SSO domain name, enter a password for administrator. Your logon username will be administrator@vsphere.local with the password you set.14. Select whether you want to participate in the Customer Experience Improvement Program (secure facilities say no). Then review your settings and Finish to start configuration.15. How long does it take to complete Stage 2, Appliance setup? For me, it took 10 minutes. If it takes more than 45 minutes, your deployment is probably failing.16. Stage 2 completes. Note that the link assumes you are using DNS. It won’t work unless you have updated your HOSTS or put an entry into your DNS server. Since we are avoiding DNS, lets just say it won’t work.
Now to connect to your new vCenter VCSA without DNS..
17. As you can see, if you don’t have DNS and just click the link, the page won’t open.18. Change the URL to the IP address of the vCenter server. Now the page displays.19. If you follow the “Launch vSphere client HTML5” link, or simply enter the URL https://vCenter_IP/vsphere-client , the vCenter logon page will display.20. Enter administrator@vsphere.local and the password you set in Step 13. Make sure you have Flash enabled on your browser.21. The vCenter page will open successfully with no DNS.22. How to access the VCSA appliance to perform management functions? Use the URL https://vCenter_IP:5480 and log on with root and the password you set in Step 8.23. Here is the management website for the VCSA
Selfish plug time (sorry!)
Thanks for reading this article! I hope it helps you! If you have tips or feedback, please comment or send me an email so that others can benefit. I am a consultant in the Maryland/DC area in the USA. My specialties are Windows migrations (to 2016 and to Office 365 / Azure), VMware migrations, Netapp and SAN, and high availability / disaster recovery planning. If you would like help with your complex project, or would like a architectural review to improve your availability, please reach out! More information and contact can be found on the About page. – Amira Armond
Tips, tricks, and missing steps for the VMWare upgrade
Having done several small / medium business upgrades from vCenter and ESXi 5.5 to 6.0 and 6.5 and 6.7, I wanted to share some best practices and lessons learned with the IT community.
About 2-4 hours during the actual upgrade process. Figure one hour to deploy the appliance (no risk, original vCenter is unchanged), then another hour and a half to replace your existing appliance (your original vCenter is shut down). VMWare is very good about this process and will perform many system checks before shutting down your original vCenter. If something goes wrong at this stage, remember you can shut down (power off) the new vCenter and then start your old vCenter and it should recover.
Don’t forget the preliminary research, system checks, and preparation is another 4 hours for a small or medium business (this includes research for vSphere upgrades).
1-2 hours of the preparation is getting the images downloaded, either to your burned DVD or directly from the VMWare website.
——————————–
What is the risk of downtime while upgrading vCenter?
Good news: For small and medium businesses, there is almost no chance of customer impact while upgrading vCenter. Just don’t plan to do other management tasks at the same time. Remember that the hosts will keep running the VMs, the storage, and the virtual networking unchanged even if they can’t reach vCenter. DRS, HA and vMotion will not be available, so you won’t have normal redundancy.
As long as you complete your upgrade in a reasonable amount of time (one evening), you should be fine. What are the odds a host crashes in that 2-4 hour period? Generally extremely low (less than 0.1%) unless you have more than 5 hosts.
If you have 100+ hosts, then your odds of a server randomly crashing during the upgrade gets closer to 5%. But in that case, you should have a team of experts on the job, so you shouldn’t need much help from this lowly blogger <grin>.
——————————–
Can I upgrade vCenter and vSphere remotely?
You can perform this entire process (vCenter and ESX host upgrades) 100% remotely. Make sure you have a plan for what to do if a host doesn’t come back after rebooting. Ideally, you have an Integrated Lights-Out (iLO) module on your servers, which will allow you to restart them or power on/off remotely.
——————————–
Research before migrating or upgrading vCenter
Make sure you have deep-dived into the existing vCenter appliance and each host before starting.
Make sure to get the following configurations:
Existing vCenter IP , verify root username and password (see next topic)
Network basics (internal domain name, subnet mask, gateway, two DNS servers)
Time provider (NTP). Make sure this matches the NTP source your hosts are using! This is a good time to standardize it across your environment.
A ‘free’ IP address on the same subnet as your existing vCenter server. Will be used for a few minutes during the upgrade.
Configure your DNS servers to have a fully-qualified name for vCenter (the old vCenter IP and your new vCenter’s name). For example: vcenter01.company.com . Make sure you can resolve it before you start the upgrade. The vCenter install (particularly new installs) will bomb out if you don’t have good DNS.
——————————–
When you upgrade to vCenter or VMWare 6x, your vSphere Client (the program) will stop working! There is no client download on the website! How can you connect?
History: vCenter and vSphere 6x are designed to be managed with your web browser, using flash. VMware did this because there are all sorts of really niche configuration options that almost no-one uses. Adding the niche options to a website is easy. Adding them to the vSphere client is not. In version 5x, VMware put only basic functionality in the vSphere client and started offering niche options in the management website ( https://hostname:9443/vsphere-client ). In version 6x, VMware got sick of updating both the website and the vSphere client, so they stopped offering the vSphere client program. Nooo!
So now that you’ve upgraded to 6x, how can you connect and manage your virtual environment?
Using Chrome, IE, or Firefox, make sure you have Flash installed. https://get.adobe.com/flashplayer/ (Personally, I use Chrome, so the rest of the instructions will match Chrome.)
Navigate to https://vcenter_server_fqdn/vsphere-client
or for a host.. https://host_ip_address_or_fqdn/vsphere-client
Didn’t connect? Make sure you have a DNS entry for your vCenter server. This is a requirement for 6x
Now you probably have a warning “To view this page ensure that Adobe Flash Player version x or greater is installed”. But you already installed Flash Player! What the heck, Chrome?
You need to enable Flash each time you open the website for the day. Look for the lock icon in the address bar. Click on it and you will see Flash listed as blocked. Change it to Allow.
You will be prompted to Reload the page. Do so. Now flash will work and you can log on. Default username for vCenter “vSphere Web Client”: administrator@vsphere.local . Default username for vSphere (individual host ESXi servers): root. In both cases, you should have been prompted to set the password during install or upgrade.
——————————–
What about opening the Console to a virtual machine?
The console still works, but it now requires you to download a separate program Here is how to open console in 6.0 6.5 and 6.7.
You need to download the “VMware Remote Console” from the VMware website. It can be reached by … Downloads > vSphere > “Drivers & Tools” tab > VMware Remote Console.
Expand VMware Remote Console then click Go To Downloads.
You will see separate installers for Windows (all Windows variants), Macintosh, and Linux systems.
You will need to log on with a VMware account – if you don’t have one already, just go ahead and register for the free account.
Download and install the remote console on your computer. No reboot necessary.
Next time you select “Open Console” from the vSphere / vCenter web client, it should prompt you to open the link with a program. Select the VMRC (Vmware Remote Console).
Your VMRC console should now open and you can interact with the desktop.
When you click Open Console in 6x, a new webpage will open. It attempts to start the VMware Remote Console program and connect it to the guest VM. It is safe to close this window as soon as VMRC is started.
In 2017 and early 2018, the remote console was terrible with jumpy mouse, mouse disappearing, or offset mouse issues. The remote console has since been updated and works very well with all the operating systems I’ve tried (CentOS, Ubuntu, RedHat, Windows Server 2012, Server 2016, etc.
Installing VMware tools on the guest VM will normally help with sync issues.
Remember, if your mouse or keyboard get ‘stuck” in the console session (normally this only happens with pre-boot environments), just press left-CTRL and left-ALT at the same time.
——————————–
Problem: Can’t log onto the old 5.5 vCenter appliance
Symptom: You are pretty sure you know the root password, but it is not working on the web GUI. ( https://vCenterIP:5480 ) It says wrong password.
Symptom: The 5.5 vCenter appliance was built more than a year ago.
Fix: Connect to vCenter using SSH
(I recommend using the Putty SSH application which is available at this link). After connecting, when you enter ‘root’ and the correct password (default is ‘vmware’ but you’ve probably changed it), you will be told that the password is expired. Change the password using the SSH dialogue and now you can log on to the web GUI.
Cause: When you build a vCenter appliance (5.5, 6.0, or 6.5 or 6.7), by default, the system expires the root account after one year. I recommend un-checking this immediately whenever you build vCenter. If you want to change the root password periodically, you can still do it, but this way your department won’t get locked out.
——————————–
How to upgrade vCenter from 5.5 to 6.0 or 6.5
I personally recommend deploying 6.5 vCenter appliances (rather than deploying vCenter onto a Windows server).
The first step to upgrading is to perform a snapshot of your vCenter virtual machine. If you quiesce guest memory and something goes wrong during the upgrade, you can revert to the snapshot and everything will be working again within 5 minutes. This truly can be depended upon. Make a reminder to remove the snapshot after everything is stable.
Stage the upgrade
Note: You will need a valid VMware account. These steps are performed on a Windows 10 administrative workstation.
Adjust the version selection as necessary. Make sure that your vCenter is the same or HIGHER version as your ESXi servers.
Find the VMware vCenter Server Appliance , file type: iso ( about 3-4 GB) and download it.
Mount the ISO by right-click, Open With, Windows Explorer.
The files will appear in your DVD drive letter in Windows Explorer.
Navigate to \vcsa-ui-installer\win32\installer.exe and start it
vCenter appliance installer will display. Click Upgrade.
Here is a helpful video that describes pre-requisites and has step-by-step instructions to perform the upgrade. Text steps continue below.
vCenter Appliance Upgrade Stage 1
Upgrade Stage 1: Deploy Appliance – Introduction displays. Click Next
Upgrade Stage 1: Deploy Appliance – License agreement displays. If you want to proceed, click Accept and Next.
Upgrade Stage 1: Deploy Appliance – Source Appliance displays. Identify your Source Appliance (your existing vCenter 5.5 server) and provide credentials to access it. The default HTTPS port 443 is correct unless your environment has changed it (this is unlikely). The SSO User Name is normally administrator@vsphere.local , but this could be any full administrator account. Appliance (OS) root password refers to the root password for the vCenter 5.5 appliance. The root account can be tested by opening website to the vCenter 5.5 server https://vCenterIP:5480 or connect via SSH to the server.
You have a choice to specify either the ESXi host or vCenter server that manages the source appliance. It is recommended to specify the ESXi host, not the vCenter server, in that box. You can specify by IP or DNS (IP is less likely to have issues).
Click Next
Upgrade Stage 1: Deploy Appliance – .Specify deployment type displays – Embedded Platform Services Controller is appropriate for environments with 1-20 vSphere ESXi servers. If you aren’t sure, and you have less than 20 hosts, go for Embedded. If you have more than that, do some research to see what is best for your environment.
Upgrade Stage 1: Deploy Appliance – Appliance deployment target displays. Enter the ESXi host IP address or DNS that you want to host the new vCenter server. This can be the same ESXi host as is already hosting (make sure you have at least 10 GB of RAM available on it), or a different one. You will need to specify the username and password for the ESXi host.
Upgrade Stage 1: Deploy Appliance – Select folder displays. You can put the new vCenter server into a logical folder in your environment.
Upgrade Stage 1: Deploy Appliance – Select Compute Resource displays. You can choose which ESXi server runs the new vCenter server.
Upgrade Stage 1: Deploy Appliance – Set up target appliance VM displays. You can assign a name to the new vCenter server and give it a root password. Take note of this username and password – you will need it to manage the new vCenter server in the future.
Upgrade Stage 1: Deploy Appliance – Select deployment size displays. Look up the number of VMs you are running in the table, and pick your vCenter size based on it. Most environments qualify for Tiny, with less than 10 hosts.
Upgrade Stage 1: Deploy Appliance – Select datastore displays. Pick a datastore that has space to hold the server size you picked in the last step (Tiny = at least 250 GB free).
Upgrade Stage 1: Deploy Appliance – Configure Network Settings displays. Pick the VM network from the list that is the same as your old vCenter server and assign a temporary IP address on the same subnet as the old vCenter server. Your admin workstation needs to be able to communicate with this IP address as well.
Upgrade Stage 1: Deploy Appliance – Ready to complete stage 1 displays. Click Finish to run the deployment. It should complete Stage 1 in about 30 minutes or less.
Click Continue to proceed to Stage 2
Note: If you get an error communicating with the new vCenter at this point, your workstation is having trouble reaching the new server over the network. Did you assign the VM to the correct virtual switch? Did you give it an IP address that can route on that subnet?
Stage 2 of the vCenter Appliance Upgrade
Upgrade Stage 2: Introduction displays. Click Next
Upgrade Stage 2: Pre-upgrade check result displays. Warnings may display, depending on how your vCenter 5.5 server is configured. Recommended resolutions will also display. You can close the results after reading through them.
Upgrade Stage 2: Select upgrade data displays. I recommend copying everything from the source to the new vCenter, but if you are worried about space, just move the configuration. The events, tasks, and performance metrics are just for audit purposes.
Upgrade Stage 2: Configure CEIP displays. Depending on your company policy, join or do not join VMware’s Customer Experience Improvement Program. Most security conscious businesses or government agencies will not join the CEIP.
Upgrade Stage 2: Ready to complete displays. Click Finish when ready.
Data transfer and setup runs – this will take up to 45 minutes normally. When finished, you can click the Close button to finish the wizard.
Now you can log on to the new vCenter and check it out! https://vCenter_IPaddress/vsphere-client
——————————–
Can I install vCenter without DNS?
YES.
Check out this blog I wrote on how I installed vCenter 6.7 without DNS.
What if you have a Windows-based vCenter appliance?
Especially if you have update manager installed? This can get ugly. The basic steps are to install the vCenter update then install the Update Manager update. But, occasionally, vCenter will start acting really buggy and the client crashes. The good news is that you can roll back to the latest snapshot of vCenter (you did snapshot it with quiesce memory before the upgrade , right?) and it will go back to normal, the bad news is that you probably will need to rebuild vCenter or call VMware if you have a problem.
——————————–
Upgrade fails, need to rebuild vCenter
If your vCenter deployment is a typical medium or small business size, re-building your vCenter from scratch is actually faster than troubleshooting it, and doesn’t involve downtime. 90% of the configuration is held on the hosts, including datastores, virtual switches, and VMs. Make sure to spend 10 minutes looking at your old vCenter and screenshot the following:
License keys
User accounts and any custom permissions
Network settings for vCenter server
NTP settings
HA and DRS settings
CPU compatiblity settings
Keep-away and custom rules settings
If you use distributed switches or other advanced items like NSX and vSAN, rebuild at your own risk. Make sure you’ve really captured the configuration steps before you proceed.
Once you have the config’s captured, just turn off your old vCenter and build a new appliance from scratch. Set up your clusters and licensing, then add each host. Everything should come in correctly. Make sure to re-connect your backup programs to the new vCenter afterward.
Hint: You can install vCenter at 6x directly. No need to install a 5.5 version.
——————————–
How to set password policy so that you don’t get locked out
One of the first things you should do is change the password policy on your vCenter so that the accounts do not expire after 90 days. If you require a password to be complex (meaning 14+ characters long), you shouldn’t need to expire it for years and years.
To view or change the policy:
Log on to vCenter https://vCenterName.domain.etc/vsphere-client with administrator@vsphere.local
From the Home menu (icon looks like a house), select Administration
Under Single Sign On, select Configuration
Select Policies, then Password Policy, then click edit.
To set passwords to never expire, change the Maximum lifetime to 0.
I also recommend increasing the maximum length to 16, reducing the character requirements to 1 each, and limiting the identical adjacent characters to 3.
Click Save to save changes. These changes are global to all _______@vsphere.local accounts.
——————————–
vSphere ESXi upgrades from 5.5 to 6.5 or 6.7
Before you upgrade your ESX hosts, take a quick read through the rest of the topics below. There are lots of good tips and things to avoid.
——————————–
How long does it take to upgrade an ESXi host?
Do you have a shared SAN with 10gbps storage networking??? It is normally about 1.5 hours per host to perform the upgrade.
Are your VMs stored on the local hard drive of the ESXi hosts? It will take # minutes to vMotion the VMs to another host, + 1 hour. In many cases, it is more efficient to power off all VMs and perform the upgrade, then power the VMs back on, if you are using local storage.
You can save time by upgrading multiple hosts at once. This of course requires either powering off the guests or having enough extra resources to host them on the other servers.
Here is the time breakdown I use:
10 minutes: vMotion all guests off the host (this assumes you have fast shared storage). Or power-off the guests.
2 minutes: Put the host into maintenance mode
5 minutes: Enable SSH in services. SSH into the host and perform pre-upgrade scripts.
30 minutes: Download and install the latest image from VMware repository (requires Internet)
15 minutes: Reboot the host and wait for the server to load back into the ESXi operating system fully.
20 minutes: Apply SPECRTRE / MELTDOWN setting and reboot again ( only if you haven’t patched in a while).
5 minutes. Disable SSH in services. Take the host out of maintenance mode.
10 minutes: vMotion a non-critical guest to the host and test functionality.
10 minutes: Finish vMotioning other guests to the host (or power on the guests).
——————————–
What is the risk of downtime for ESXi upgrades?
The first question is: Are all your hosts running the same model of hardware?
If YES… then your risk is almost nil.
If NO… your hosts are running different hardware models (such as a Dell R710 and a HP DL360 gen8 and a HP DL360 gen9), then your risk of downtime or customer impact is pretty high. Read through the topic below, “CPU Generation Compatibility Levels” before you continue.
Next question: Do you have enough RAM and disk space to run all your VMs on (HOSTS – 1) ?
Figure this out ahead of time. Many companies have a few high-resource VMs such as database servers, which take up all or most of the resources on a host. Plan how you will migrate the VMs around so that all the VMs will fit onto your other servers. Note: You CAN overallocate RAM on ESXi hosts. They will page the less-used memory to disk. This normally won’t hurt performance much as long as you don’t over-allocate too many resources. Here is a (very) technical article about overallocating memory and CPU on VMware: https://labs.vmware.com/vmtj/memory-overcommitment-in-the-esx-server
If you properly migrate your VMs off each host, putting it into maintenance mode, before upgrade, you should be good. I still recommend doing this portion after-hours so that you minimize impact from vMotion and have the maximum amount of redundancy during the workday in case one of your hosts randomly fails.
——————————–
Problem: When I try to vMotion a guest VM, the option is grayed out?
Normally this issue is because the VM is changing in some way. For example, it might be migrating somewhere already, or you just modified the hardware settings and it is still saving, or you just powered it on, etc. Another possibility is that your Veeam is performing a backup of that VM.
If Veeam backup is occurring (you can tell by looking at the current Snapshots), wait for the Veeam backup to complete, or cancel it.
If a fast task like changing settings is running, wait for it to complete.
If you can’t find the cause, try shutting down the VM and powering it back on. Sometimes that clears the issue. Obviously not the ideal answer.
——————————–
How to upgrade ESXi hosts using command line remotely
For this procedure, you WILL need Internet access from the host. If you have a decent business ISP, it should take less than 20 minutes to download the update.
1. vMotion or power-off all VMs off the host. Note: you may need to disable automatic load balancing in your cluster if the VMs are automatically migrated back.
2. Put the host into Maintenance Mode
3. Start the SSH service on the host ( Configuration > Security Profile > Services > Edit )
4. SSH to the host ( I use the Putty program which is freely available on the internet)
5. Log on as root
6. Perform this command to enable downloads from the VMware Internet repository”
esxcli network firewall ruleset set -e true -r httpClient
7. Determine which version of ESXi you want to update to.
IMPORTANT: Never update your ESXi version higher than your vCenter version!! If you do, your vCenter will disconnect the hosts!!!
For example: When I check my vCenter version in the Appliance website ( https://vcenter.fqdn.com:5480 , update tab), I see my vCenter is version “6.5.0.23000 Build Number 10964411” with a Release date of November 29, 2018.
I look up the VMware patch tableand decide I want to download ESXi 6.5 EP 11 which was released November 9, 2018 and has a slightly lower build number than my vCenter server.
Version
Release Name
Release Date
Build Number
ESXi 6.5 EP 11
ESXi650-201811001
11/09/2018
10719125
Perform this command in SSH to see what versions of ESXi are available from the VMware Internet Repository:
Review the list for the version you want. Copy the package name to your clipboard. The package name will look like ESXi-6.#.#-########-standard
When I perform the query in SSH, I see packages that match the release name:
ESXi-6.5.0-20170304101-standard VMware, Inc. PartnerSupported ESXi-6.5.0-20171201001s-standard VMware, Inc. PartnerSupported ESXi-6.5.0-20170304101-no-tools VMware, Inc. PartnerSupported ESXi-6.5.0-20181101001s-standard VMware, Inc. PartnerSupported ESXi-6.5.0-20180604001-no-tools VMware, Inc. PartnerSupported ESXi-6.5.0-20180502001-no-tools VMware, Inc. PartnerSupported ESXi-6.5.0-20171204001-no-tools VMware, Inc. PartnerSupported ESXi-6.5.0-20170304001-standard VMware, Inc. PartnerSupported ESXi-6.5.0-20181104001-no-tools VMware, Inc. PartnerSupported ESXi-6.5.0-4564106-no-tools VMware, Inc. PartnerSupported ESXi-6.5.0-20181101001s-no-tools VMware, Inc. PartnerSupported ESXi-6.5.0-20181103001-no-tools VMware, Inc. PartnerSupported ESXi-6.5.0-20180501001s-standard VMware, Inc. PartnerSupported ESXi-6.5.0-20170304001-no-tools VMware, Inc. PartnerSupported
Since I want the full package including latest VMWare tools, I pick the package named “-standard”.
8. I then run the UPDATE command, adding the package name to it.
After about 5 minutes (depending on your internet connection), you should get a successful message.
Update Result Message: The update completed successfully, but the system needs to be rebooted for the changes to be effective. Reboot Required: true VIBs Installed: VMware_bootbank_esx-base_6.5.0-2.71.10868328,………,………,………,………,………,………,………,………,………
9. Type reboot in SSH to reboot your host.
10. Wait for the reboot to finish. Generally, I just wait for vCenter to show that the host is back online.
11. SSH back into the host, logging on as root. Note: If SSH doesn’t connect, you may need to re-start the SSH service in Security Profile.
12. Disable downloads from the VMware Internet repository by running this command:
esxcli network firewall ruleset set -e false -r httpClient
13. Take the host out of Maintenance Mode
14. vMotion or power-on a non-critical guest VM to this upgraded host.
15. Test functionality of the non-critical VM (especially check the ability to ping it, which is a good all-around test of host, network, and VM)
16. vMotion or power-on the other VMs to the upgraded host.
17. Stop the SSH service on the host (Configuration > Security Profile > Services > Edit )
18. Continue upgrading other hosts.
If you need more information, here is a longer guide for upgrading ESX 5.5 to 6.x using command line by Deflounder.
This is a big concern because if you don’t avoid it in advance, you will have to shut down some VMs in order to complete the ESXi hosts upgrade.
If you have more than one model of server, deep dive on this BEFORE you start upgrading ANY hosts.
Symptoms
You have older hosts mixed in with newer servers.
After upgrading a host to vSphere 6.0 or 6.5, you cannot vMotion your VMs to it. “The virtual machine requires hardware features that are unsupported or disabled on the target host. General incompatibilities. If possible, use a cluster with Enhanced vMotion Compatibility (EVC) enabed, see KB article 1003212.“
Root Cause
vCenter 5.5 seems to handle CPU compatibility without any configuration steps. Later versions need to be configured for this before you turn up hosts.
In vCenter 6.0 and 6.5 and 6.7 , you need to set up a Cluster object and put your hosts into it in order for Enhanced vMotion Compatibility to work across CPU generations. But you can’t do this while your host has any VMs running. If you upgrade then move your VMs onto the host without setting up the cluster object, you will have issues vMotioning the VMs to other hosts later.
Fix: Set up cluster in vCenter, configure CPU compatibility levels to the lowest common denominator, and add your newest host to it early.
After upgrading vCenter, but before upgrading hosts, create a cluster object (right-click the datacenter, New Cluster)
After naming the cluster, right-click it and select Settings.
Select the “VMware EVC” tab in settings. Enable it, and select the lowest (oldest) CPU generation for all of your hosts.
Starting with your newest hosts (the highest generation), move them into this cluster after upgrade but before migrating VMs back onto them.
For example, I will upgrade ESX05 first because it has the highest CPU generation.
(I have already upgraded vCenter to 6.5)
Create a cluster object and configure CPU compatibility at the LOWEST common level
vMotion all VMs off ESX05 to older hosts.
Put ESX05 into maintenance mode in vCenter
Install ESXi 6.5 on this host
Re-add ESX05 to vCenter if needed.
Move ESX05 into the compatibility cluster while it is in maintenance mode with no VMs.
Configure and thoroughly test ESX05 (see topic below).
Take ESX05 out of maintenance mode
vMotion some VMs onto ESX05, in preparation for upgrading ESX04 (the next lower generation server) next.
If you do this right, you should be able to avoid customer impacting downtime.
Problem: Did you miss the CPU compatibility fix and already vMotioned some VMs into an upgraded, newer host?
Sorry. You are stuck with a situation where you can’t get your last host(s) into the compatibility cluster because you have live VMs on it and you can’t vMotion the VMs to any other host. This is not an acceptable stopping point because you lose cluster benefits like DRS and also can’t vMotion for regular maintenance in the future. So bite the bullet and fix it now, rather than wait until you are forced to do it in the future.
If you get stuck in this compatibility hell, then the easiest solution is to shut down all the VMs on that host then move the host into the cluster. This normally results in a downtime of about 10-15 minutes, depending on how fast your VMs boot up.
Problem: Did you upgrade the host with vCenter on it, and now you can’t migrate without shutting down vCenter?
How to fix vCenter vMotion CPU compatibility issue:
Here is the situation. You are trying to vMotion your vCenter to another host, but you get an error about CPU compatibility. Errors like “The virtual machine requires hardware features that are unsupported or disabled on the target host” “If possible, use a cluster with Enhanced vMotion Compatibility (EVC) enabled; see KB article 1003212″ ” CPUID details” “incompatibility at level 0x1 register ‘ecx'”. “FMA3 is unsupported” “MOVBE is unsupported” “RDRAND is unsupported”
The normal fix is to power off the VM momentarily, vMotion it, then power it back on. During power-on, it will use the new host’s CPU capabilities.
But with vCenter, you can’t do this, because you are using vCenter to perform the vMotion, so powering it off means you can’t communicate with both hosts at once.
Shared storage fix:
If you have shared storage (both hosts can reach the same datastore), you can use VMWare’s instructions to fix the problem. Essentially, you make sure the vCenter server is on shared storage, then you connect to each host’s vSphere website directly. Power vCenter off and remove it from inventory (no delete!!!!) on the first host. Then Add vCenter to inventory on the second host and power it on.
No shared storage? Using local disk / datastore for each host?
This is how I’ve fixed it.
Using vCenter, with vCenter powered on, I right-click the existing vCenter server and Clone it to the second host’s datastore. Note: Name the clone something you are happy to use permanently)
Wait for the clone to finish, (the clone will be powered off).
Open the vCenter management website ( https://vcenterURL:5840 ) and log on to it
In another tab, open vSphere website to the second host (that has the clone) and log on to it.
Right-click the clone vCenter and Power On
Immediately Shut Down the original vCenter using the management website.
Wait for about 10 minutes, then attempt to connect to vCenter normally. You should be working from the clone now.
Once you feel comfortable, delete the old vCenter.
Something went wrong?? Rollback instructions:
If the clone vCenter doesn’t work, just shut it down using the second host’s vSphere website.
Connect to the first host’s vSphere website and power on the original vCenter.
You can avoid 99% of the risk by testing your upgraded host thoroughly before you put production VMs onto it.
I recommend the following tests (at minimum)
Make sure you’ve got your upgraded host configured, into vCenter, and into it’s destination cluster.
Using a test VM (build one up if you don’t have one ready), test connecting to each virtual switch and pinging to-and-from other servers across the network.
Storage vMotion the test VM to each major storage system you have.
vMotion (processing) the test VM between your upgraded host and other hosts. Test both directions.
Snapshot and delete snapshot on the test VM.
And before you close this project, make sure to re-connect your backup jobs and test them.
——————————–
Good luck in your upgrade process!
(shameless self promotion) By the way, I consult on VMware, Storage Area Networks (such as Netapp, Synology, FreeNAS), high availability designs, and migrations to the cloud. I offer 30 minutes of free technical advice on these topics: if we can solve your problems in that time, great! If the problem is too big, we can start working on a plan together. Please schedule using the button below! – Amira Armond, President, Kieri Solutions
