Wed Mar 21, 2018 10:13 pm
Login Register Lost Password? Contact Us

Time sync problem on cluster

Post questions specific to installation or configuration for the HPCC Systems platform

Tue Jun 14, 2016 4:53 am Change Time Zone

We have a 10 node production cluster. It has been alive for months and over the time accumulated hundreds of sprayed data and many superfiles.

The time that has been set on every node of the cluster is not in sync with the actual time. So it requires a bit of arithmetic everytime I need to look at for how long a job has been running.

Almost on a whim, I have reset the date ONLY on master node today. It seemed to be causing some issues. Some jobs run. But some jobs seem to be blocked forever. And the time change seemed to be the cause behind it.

I could not reset the date to original as I do not have it. I could take time from one of the slave nodes and use it to set the time on master but I doubt if it solves the problem.

Few questions in this regard:
1) What is the impact of date reset on one node of the cluster?
2) Would I lose any of the work that has been done so far if I had to reconfigure the cluster?
3) Are there solutions that can ensure no loss of data?

Thanks in advance,
Posts: 4
Joined: Wed Mar 25, 2015 5:15 am

Thu Jun 16, 2016 10:19 am Change Time Zone

Having the nodes on different times may lead to all kinds of unforeseen issues.

It would be best to set up a centralized time server and have the all the nodes synchronize their time to it. Keep ALL the servers including the middleware (dali, eclccserver etc,) components in-sync with the same time for best results.

Stop all the components, sync the time all the nodes, restart components.
Posts: 5
Joined: Thu Jun 19, 2014 1:29 pm

Return to Installation

Who is online

Users browsing this forum: No registered users and 1 guest