Time sync? Yes, last week I saw some activites around time synchronization on social media and had an odd issue myself. So that triggered this blogpost with following in mind:
- A note from Michael White regarding VMware Tools synchronizing forward and backward in time.
- A tweet regarding the same VMware Tools functionality as above.
- I had an issue with my domain controller which did not synchronize with a NTP server. The result were that it relied on the local CMOS clock (not good!)
Let me start with recommending Michael Whites notes. Michael creates these weekly newsletters about things he encounters, experiences or reads. There are some really great information and tips – be sure to follow him. Read his latest newsletter here and get inspired: http://notesfrommwhite.net
In the newsletter he wrote a few lines about VMware tools does time synchronization, both forward and backwards (though not stating that he recommends it). The tweet I saw was also regarding the same subject.
So lets dig into it.
VMware Tools are a set of features, functions and drivers for the operating system. One of the function that VMware Tools provide is Time Synchronization.
When a virtual machine boots up, the VMware Tools synchronizes the time with the ESXi host. If the time is configured wrong on the ESXi host, the time on the virtual machine will also be accordingly wrong.
In a Windows Active Directory environment time sync is critical. The time skew must not be more than 5 minutes (default setting), as you will start experience authentication failures. A lot of other bad things can happen for your business applications with incorrect timestamps, so time synchronization is really critical and it need to be as accurate as possible.
So if the time is exactly the same on every VM then you are good – even if it is not showing the correct time. However, if you have physical servers, desktops etc, you will probably see time skews, so you really want to have everything set with the correct time.
Okay, so lets see what options there is.
You can configure your ESXi host to get its time from a NTP server.
Now you can be assured that your time is running correctly on your windows VM. Or can you?
If VMware tools are, for some reason, not running you might experience time skew. Time may go backwards or forward. If the time goes backward and VMware tools are started, VMware Tools corrects the time immediately. If time goes forward, and VMware Tools are started then we see a quite different behaviour. The time on the virtual machine actually just slow downs. It does NOT correct the time immediately. That means if a virtual machine is a year ahead of time it will take like forever to get back to the correct time.
So what should you do?
Well in a Active Directory environment you should configure your PDC emulator (Domain Controller FSMO Role) to get its time from a reliable NTP server. Other domain controllers gets their time from the PDC. Clients and member servers then get their time from the domain controllers. All this is done by w32time (Windows Time Service).
I made a simple diagram to illustrate this.
Hosts in Active Directory?
Your ESXi hosts can also join your Active Directory. If configured, remember that they start getting their time from Active Directory. I think it’s a great fit if you are running a bunch of windows virtual machines. You actually don’t NEED to configure anything on the virtual machines – the time syncs from the host and Windows Time Service. As the time source for them both are same, there is no issues other than it will sync from the VMware Tools and Windows Time Service periodically. Even it is not needed I do recommend that you disable the VMware Tools time sync feature and let Windows Time service handle time. If you f.ex. move your VM to a host that is not Active Directory integrated – the VM will start flipping time between VMware Tools and Windows Time. Then again – if you have ESXi hosts not joined to the domain, you should define your PDC as the NTP server.
What about my linux machines?
Its a bit different story with time sync for Linux machines. There are a great KB for this here. However, you will face the same issues with VMware Tools. So the recommendation are to use a NTP server within your Linux virtual machines.That NTP server could be your PDC.
Try to keep time keeping simple across your entire infrastructure. Time is really sensitive and you want to have it running correctly and consistently. Create a time synchronization strategy. You could add your PDC as the NTP server for your networking devices, storage etc. Add multiple NTP pools to your PDC in case of downtime of these services. It is not needed to sync with an external time source, but I would recommend it from a availability perspective. Another thing to consider is that your PDC will become a single point of failure as only one Domain Controller can have the PDC role. Create new A record in DNS and point it to your PDC. This way you can easily switch to another NTP server in case your PDC is under maintenance. However, this is your Active Directory – you probably have a recovery strategy in place for this already. From a manageability perspective it will become easier to maintain and troubleshoot – one place to troubleshoot time in your environment.
My DC and the time sync issues I experienced.
Some time ago I went through the same configuration as described above for my Domain Controller. However the outcome were not as expected.
I configured my Domain Controller to sync via NTP and gave it a valid NTP server pool. Rebooted the server and saw that time was fine and left my environment for a couple of days. When I went back to the lab environment I realized that time were 4 days behind!
I was sure that the correct parameters were given and got it verified via regedit. all seemed to look good.
But a status from the command line showed another story!
net stop w32time
net start w32time
Now that everything were reset I added the settings again.
Then I tried to get the time synchronized. But it looks like the time difference were to big. (This behaviour is specific for NTP sources – it will not be a issue for other computer objects in AD – they will sync to the correct time immediately)
So I manually edited the time on the Domain Controller to the approx correct time and tried the command again.
It finally resynced. But what was the result via commandline?
Now it was successfully synchronizing with the NTP Servers. However I never found the root cause. Be sure to verify that time is configured correctly if you are following my example.