• Introduction
• Issue
• Cause
• Solution
• Prevention

Introduction

Changing the system time backwards is a problem because it can cause Iguana log timestamps to get out of order — which means later records can have earlier timestamps.

This problem occurs whether Iguana is running or stopped at the time: If Iguana is running you will see an error immediately, if it is stopped you will see an error next time Iguana starts.

We  need to avoid writing log records during the “unsafe” period when we can get conflicting timestamps:

Issue [top]

Changing the system time backwards 20 minutes or more, causes Iguana to raise an error and stop all channels. Changing the system time backwards less than 20 minutes, causes Iguana to pause message processing for all channels until the “time change point” is reached.

Changing the system time backwards 20 minutes or more

You will see this error displayed in the Dashboard and in the Logs:

The system clock appears to have been set backwards by XXXX seconds while Iguana was running. Iguana has been halted as a precaution to prevent possible data corruption.

Changing the system time backwards less than 20 minutes

If you do this while iguana is running will not see an error displayed in the Dashboard and in the Logs. Channels will continue running but message processing will be paused until the time catches up to the “time change point” (ensuring there is no risk of out-of-order log timestamps).

However if you do this when the Iguana service/daemon is stopped then it safe. In this case you will not be to restart the Iguana service/daemon until the time catches up to the “time change point”.

Cause [top]

Setting the system time backwards:

Changing the system time backwards can cause problems because it means Iguana log timestamps can get out of order — for example if you set the time to one hour earlier, then it is possible for later messages to have a timestamp up to one hour earlier.

Iguana is designed to prevent writing log records during the “unsafe” period when time is “overlapping” and we can get conflicting timestamps:

Iguana takes the following actions to prevent this:

1. If you change the system time backwards by 20 minutes or more, Iguana will halt processing and stop all channels:
   1. Iguana will raise a fatal error:

      The system clock appears to have been set backwards by XXXX seconds while Iguana was running. Iguana has been halted as a precaution to prevent possible data corruption.

      
      

   2. If Iguana is running you will see the error immediately, if Iguana is stopped you will see the error next time Iguana starts.
2. If you set the time backwards by less than 20 minutes Iguana will temporarily stop processing messages until the time catches up:
   1. This only applies while Iguana is running.

      If you change the time when Iguana is stopped you will not be able to restart the service/daemon until until the time catches up to the “time change point” — which ensures there is no risk of out-of-order timestamps.

   2. All active channels will continue to run — with green lights.
   3. Message processing for active channels (with green lights) will be paused.
   4. Message processing for active channels will be resumed once the “time change point” is reached.

      Pausing processing until the time catches up to the “time change point” ensures there is no risk of duplicating past log timestamps.

      Warning: If you notice that channels are not processing messages after after the system time is changed backwards — do not restart the channels!

      If you do restart channels they will start processing messages immediately which will cause out-of-order timestamps.

      If this occurs your log messages will end up looking something like this:

      

      If you do find yourself in a situation with out-of-order log message timestamps please contact us at support@interfaceware.com for further help.

Setting the system time forwards:

Changing the system time forwards does not create problems because it will not cause Iguana log timestamps to get out of order — though it will create a gap in the timeline where there no timestamps are used.

Solution [top]

Luckily the solution to this problem is easy — you simply need to wait until the “time change point”, which is after the “unsafe” period when time is “overlapping” before creating more log messages.

From a user perspective this problem usually presents as one of four scenarios:

1. Time was changed backwards and you need to keep the change.
2. Time was changed backwards and you need to undo the change.
3. Time was changed forwards and you need to keep the change.
4. Time was changed forwards and you need to undo the change.

You can follow these steps to resolve those scenarios:

Time was changed backwards and you need to keep the change.

1. Stop the Iguana service (or daemon), if it is still running.
2. Wait until the until the time catches up to the “time change point”.
3. Optional precaution: You can backup (copy) today’s *.log file (the day when time travel occurred), e.g., 20141111.log.
4. Restart the Iguana service (or daemon).

   Note: Occasionally restarting the service/daemon does not work the first time (usually because you started it too soon) — in most cases starting it a second time will resolve the issue. If this still does not work you can try removing the index file and meta information files for today (the day when time travel occurred).

   If you are still having problems then please contact us at support@interfaceware.com.

Time was changed backwards and you need to undo the change.

1. Stop the Iguana service (or daemon), if it is still running.
2. Set the system time forward to the correct time.
3. If necessary wait until the until the time catches up to the “time change point”.

   This will never happen if you are simply reversing a mistake. But it could happen if you need to make a partial adjustment, for example you set time back by two hours then set it forward by one.

4. Optional precaution: You can backup (copy) today’s *.log file (the day when time travel occurred), e.g., 20141111.log.
5. Restart the Iguana service (or daemon).

   Note: Occasionally restarting the service/daemon does not work the first time (usually because you started it too soon) — in most cases starting it a second time will resolve the issue. If this still does not work you can try removing the index file and meta information files for today (the day when time travel occurred).

   If you are still having problems then please contact us at support@interfaceware.com.

Time was changed forwards and you need to keep the change.

1. No action is required.

   Though there will be a gap in the timeline where no log timestamps are written.

Time was changed forwards and you need to undo the change.

1. Stop the Iguana service (or daemon), if it is still running.
2. Set the system time backwards to the correct time.
3. If necessary wait until the until the time catches up to the “time change point”.

   This will never happen if you are simply reversing a mistake. But it could happen if you need to make a larger backward adjustment, for example you set time forward one hour then set it back by two hours (this could happen if you are adjusting manually for daylight saving — and initially you went forward when you should have gone back).

4. Optional precaution: You can backup (copy) today’s *.log file (the day when time travel occurred), e.g., 20141111.log.
5. Restart the Iguana service (or daemon).

   Note: Occasionally restarting the service/daemon does not work the first time (usually because you started it too soon) — in most cases starting it a second time will resolve the issue. If this still does not work you can try removing the index file and meta information files for today (the day when time travel occurred).

   If you are still having problems then please contact us at support@interfaceware.com.

Prevention [top]

Generally you should not change the System Time while Iguana is running!

Instead follow these steps to safely change the system time:

1. Never change the system time manually:

   Once a system is setup the only changes to system time should be for daylight saving — which iguana can handle without any problems.

   1. Allow the daylight saving time to be applied automatically.
2. Set the system time forwards:
   1. This is safe and can be done while Iguana is running.
   2. You can optionally stop and restart if you prefer to be cautious.
3. Set the system time backwards:
   1. Stop the Iguana service (or daemon).
   2. Change the system time backwards.
   3. Wait until the time catches up to the “change point”.

      For example if you set the time back one hour you need to wait for one hour, or if you set the time back 20 minutes you need to wait 20 minutes.

   4. Restart Iguana.