- Iguana Log/Queue Design
- Log file and directory specifics
- Viewing Log Statistics
- Changing the Log Directory
- Limiting Access to the Log Directory
- Log Age and Logging Performance
- Synchronous Writes and Logging Performance
- Verifying Log File Integrity Using iguana_log_verify
- Storing log indexes separately from the logs
Iguana Log/Queue Design
We often get questions about how Iguana implements queuing.
The internal logging and queuing system of Iguana is one the unique proprietary features of the platform that make Iguana stand out as a robust engine for message processing. We developed the technology specifically to give a robust, highly available and fast means of queuing messages and retrieving log information.
The queue/logging system was first introduced in Iguana 4.0. Since then the system has proven it’s performance and stability throughout thousands of deployments.
At the core Iguana stores queue and log information in *.log files. One of these files is generated per day. This minimizes the amount of disc IO on the underlying operating system. The log files are append only which means that Iguana never writes information back into the middle of these files. This is highly fault tolerant design that makes it possible to replicate and do hot back ups on these files without needing to stop Iguana. No special tools are required to do these back ups.
The log files have periodic CRC32 checksums written within them to validate the integrity of the data. This guards against the potential for disc corruption.
Querying the logs is blindingly fast because we use a full text search index implemented on the logs. This same type of algorithm that Google uses for internet searches. The logging system does generate index and meta files for these searches. If deleted, Iguana can regenerate these from the main *.log files.
Open access is provided to the logs by means of RESTful APIs which allow customers to extract data from the logs and also APIs in Java/C# etc. which sit on top of these base APIs. This makes it easy to display the log information in the context of user applications or for external archiving of the data.
If you have any additional questions please do free to ask.
Log file and directory specifics [top]
Iguana stores its log messages in files contained in the log directory. A new log file is created every day. Each log file is named <date>.log, where <date> is the date of the file in YYYYMMDD format. For example, the log file for August 13, 2008 is named 20080813.log.
Iguana also creates a log metafile every day, and stores it in the meta subdirectory of the log directory. Each metafile is named <date>_meta.xml, and contains a summary of the contents of the log file for that day. Log metafiles ensure that Iguana does not have to scan all of the log files when it starts up.
The meta subdirectory also contains a channel metafile, named channel_position_meta.xml, that keeps a count of the number of messages that have been sent by each channel.
Iguana also creates a log index every day, and stores it in the index subdirectory of the log directory. Each index is named <date>_log_index.xml, and contains a index information for the contents of the log file for that day.
Note: You should never backup the meta and index files! If you do you it can stop Iguana from functioning since the backup process can lock these files so that you will see this error.
If any index or meta files are missing or corrupted Iguana will generate them from the *.log files when it starts up.
Tip: For best results, define a separate local partition for your log directory instead of using your system partition or C drive.
For the best performance use the fastest possible disk device.
For the highest reliability enable Synchronous Writes in the Logging settings.
Warning! Do not open or edit any log files manually. If you change the log files yourself, Iguana might not be able to access them.
Viewing Log Statistics [top]
From the Logs screen, you can view log statistics. These statistics provide information on the size of the log files, their names, and the date on which each file was created.
To view the log statistics:
- Click the Logs tab to display the log messages.
- Hover over the Log Usage link at the lower left of the screen to view the name of the log directory and a summary of the log usage statistics:
- If background text indexing is being performed, its progress is displayed in the Log Usage progress bar:
Note: The progress bar gives a visual indication of how much of the available disk space has been used. - Click the Log Usage link, for detailed log usage statistics:
- If background text indexing is being performed, its status is displayed in the Log Usage window:
Changing the Log Directory [top]
You can change the directory in which Iguana stores its logging information.
Warning! to change the log directory, you must first stop the Iguana service.
Warning! Using NFS-mounted disks for Iguana’s logs can cause errors. For shared storage, use a SAN.
To change the log directory:
- Pick a time when your Iguana server is not being heavily used.
- Log out of Iguana.
- Stop the Iguana service.
- Use a text editor to
- Update the log directory in the Iguana configuration file IguanaConfiguration.xml:
- Use a text editor to open the config file: <configuration-directory>/IguanaConfigurationRepo/IguanaConfiguration.xml
- Change the log_directory entry to the location of the directory in which you want Iguana to write its log files.
Note: The Iguana service user must have read and write permission for this directory, or Iguana will not run. - Save and close IguanaConfiguration.xml.
- Make a backup copy of the log directory.
- Move or copy the files from the old log directory to the new log directory.
Note: If you copy the files you can use the original copy as your backup. - Restart the Iguana service and log back into Iguana.
Limiting Access to the Log Directory [top]
If you are using Iguana to transmit messages containing confidential patient data, you can modify the permissions on the log directory to ensure that the files containing log messages cannot be accessed by people who are not authorized to view them. This will also help for HIPAA compliance.
Ideally access to the Iguana log files should be restricted to the Iguana Windows Service user (or Daemon user in Linux etc.). All access to the logs should be through the Iguana logs interface. Also the system user that runs your backup software will need read access to the log files.
Note: The only exception is the need for administrator access in rare cases of log file corruption in the case of a server crash. In this case you should always contact us at support@interfaceware.com so we can help.
- Log into your Linux system as a user that is authorized to change permissions (such as root).
- Set your current directory to be the directory in which Iguana is installed.
- Run the following command:
chmod 700 logs - This will disable read, write and execute access for everyone except the owner of the Iguana log directory.
Note: To change the directory owner, use the chown and chgrp commands. See the man pages of your Linux distribution for specific instructions.
To limit access to the log directory when Iguana is running on Windows:
- In Windows Explorer, locate the folder in which Iguana is installed (for example, C:\Program Files\iNTERFACEWARE\Iguana).
- Right-click the logs directory. From the popup menu that appears, select Properties.
- Click the Security tab to display the security options for the logs directory:
- Click Edit.
- From the top pane of the Security window, select a user for which you want to limit access. (If the user is not listed in this pane, click Add to add the user to the list.):
- In the lower pane of the Properties window, in the Deny column, select the permissions that you want to allow or disallow for that user.
- Click Apply to apply your changes.
- Repeat the above three steps for all users for which you want to limit permissions.
- When you are finished, click OK to close the Properties window.
Note: For maximum security, deny access to all users except:
- The user under which the Iguana server is running;
- Any other users that absolutely have to access the directory, such as the user that runs system backups.
Log Age and Logging Performance [top]
In Iguana, you can specify the number of days that log files are to be kept before being deleted. When you are specifying this number, you should consider the following factors:
- Keeping log files for a longer period of time means that there is more data available for access.
- If the log files are large, it may take longer for Iguana to start up.
If your log directory has one or more missing/corrupted log index or metafiles, Iguana will need to recreate those files when it is started. When Iguana needs to recreate its log indexes and metafiles, it will take longer to start up.
Note: On one of our test systems – a 2GHz Intel iMac running Windows in a virtual machine – the log index and metafiles for 20GB of data were re-created in 15 minutes.
Subsequent restarts of the Iguana server did not take a noticeable amount of time. This will likely be the normal behavior for most users.
For more information on log metafiles and the other files that are contained in the Iguana log directory, see Log file and directory specifics.
For details on how to specify the number of days that log files are to be kept, see the Logging settings.
Synchronous Writes and Logging Performance [top]
In Iguana, you can specify that frequently written files, such as log files, are to be synchronized immediately after each write operation. The details of how this is performed depend on the operating system that you are using:
- If you are using Windows, Mac or Linux, the fsync operation is invoked whenever the log output buffer and corresponding index files are flushed.
- If you are using Mac OS X, fsync and the pragma synchronous setting are set in the same way as in Windows, Mac and Linux. In addition, the F_FULLFSYNC file control operation is used, which ensures that all buffered data is flushed to disk. In the Mac environment, F_FULLFSYNC is the only way to ensure reliable and safe writing of files to disk. Unfortunately, this is significantly slower than asynchronous writes.
Note: files that are not updated frequently, such as log metafiles, are always written synchronously.
Tip: If your files are being synchronized after each write operation and your Iguana channels have a high volume of activity, it is best to store the Iguana log files and database files on a separate physical device from the operating system and other applications. This ensures that the synchronize operations do not adversely affect the performance of your operating system or of the other applications that you are running.
For details on how to change the Iguana log directory, see Changing the Log Directory.
Verifying Log File Integrity Using iguana_log_verify [top]
Iguana stores its log messages in files contained in its log directory. You can use the iguana_log_verify command line utility to verify that log files in this directory have not been corrupted. This is useful if you are planning to back up your log files or migrate them from one server to another.
Note: How could log files get corrupted?
Iguana’s logging system has a fail-safe design that makes it highly fault-tolerant. However, because hard drives are physical devices, there is always a possibility that data stored on disk may have become corrupted. For this reason, Iguana’s logging system makes extensive use of checksum information to ensure that any data corruption that might occur results in minimal data loss.
Because of the fragile nature of hard drive disk storage, it is important to implement a backup strategy to safeguard your data.
Tip: The location of the log directory is included in the log statistics for the Iguana server. See Viewing Log Statistics for information on how to view log statistics.
iguana_log_verify is included in your installation of Iguana. It is located in the directory in which Iguana is installed (for example, C:\Program Files\iNTERFACEWARE\Iguana).
The syntax for iguana_log_verify is:
iguana_log_verify <parameters>
The following table describes the parameters that can be provided with iguana_log_verify.
| Parameter | Description |
|---|---|
| –all | Verify all of the log files in the specified directory. |
| –file <filename> | Verify the log file specified by <filename>. |
| –log_dir <directory> | The directory containing the Iguana log files. If this parameter is not specified, the current working directory is assumed. |
| –no_progress | Normally, iguana_log_verify provides continuously updated progress information as it scans the log files that it is verifying. This parameter disables this progress information, which is useful if the output from iguana_log_verify is being redirected to a log file. |
| –verbose | Output additional information for each of the log files verified. |
Note: When running iguana_log_verify, you must supply either –all or –file. If both parameters are provided, –all is assumed.
The following is an example of what you see when you use iguana_log_verify to verify all of the files in the Iguana log directory:
D:\iNTERFACEWARE\Iguanalogs>iguana_log_verify --all iguana_log_verify 1.0.1 Directory : D:\iNTERFACEWARE\Iguana\logs Scanning 20081230.log...100% Scanning 20090102.log...100% Scanning 20090105.log...100% Scanning 20090106.log...100% Scanning 20090107.log...100% Scanning 20090108.log...100% Scanning 20090109.log...100% Scanning 20090112.log...100% Scanning 20090113.log...100% Scanning 20090114.log...100% Scanning 20090115.log...100% Scanning 20090116.log...100% Scanning 20090119.log...100% Scanning 20090120.log...100% Scanning 20090121.log...100% Scanning 20090122.log...100% Scanning 20090123.log...100% Scanning 20090126.log...100% Scanning 20090127.log...100% Scanning 20090128.log...100% Scanning 20090129.log...100% Scanning 20090130.log...100% Scanning 20090202.log...100% Scanning 20090203.log...100% Scanning 20090204.log...100% Scanning 20090205.log...100% Scanning 20090206.log...100% Scanning 20090209.log...100% Scanning 20090210.log...100% ---SUMMARY--- Directory : D:\iNTERFACEWARE\Iguana\logs Total : 29 files Valid : 29 files
If you specify the –verbose option, you will see detailed information about the file or files verified:
D:\iNTERFACEWARE\Iguana\logs>iguana_log_verify --verbose --file 20090116.log
iguana_log_verify 1.0.1
File : D:\iNTERFACEWARE\Iguana\logs\20090116.log
Scanning 20090116.log...100%
---DETAILS---
Directory : D:\iNTERFACEWARE\Iguana\logs
20090116.log : OK
# of Messages : 139803
# of 1MB Pages : 18
Total Message Size : 16561684
Avg Message Size : 119
Avg Messages/1 MB Page: 7767
---SUMMARY---
Directory : D:\iNTERFACEWARE\Iguana\logs
Total : 1 files
Valid : 1 files
Storing log indexes separately from the logs [top]
This feature was introduced in Iguana version 6.1.3.
Some of our customers have had corrupted log indexes on NFS drives. This requires Iguana to be restarted so the indexes can be rebuilt — this can take some time if you have a large amount of log files.
We use SQLite for our log indexes and it can have corruption problems with indexes on network drives — particularly NFS.
The solution has been to separate the indexes from the log files and to store them on the local drive.
See how to store log indexes separately from the logs FAQ for instructions on how to do this.