Iguana Log/Queue Design

We often get questions about how Iguana implements queuing.

The internal logging and queuing system of Iguana is one the unique proprietary features of the platform that make Iguana stand out as a robust engine for message processing. We developed the technology specifically to give a robust, highly available and fast means of queuing messages and retrieving log information.

The queue/logging system was first introduced about half a decade ago in Iguana 4.0. Since then the system has proven it’s performance and stability throughout thousands of deployments.

At the core Iguana stores queue and log information in *.log files. One of these files is generated per day. This minimizes the amount of disc IO on the underlying operating system. The log files are append only which means that Iguana never writes information back into the middle of these files. This is highly fault tolerant design that makes it possible to replicate and do hot back ups on these files without needing to stop Iguana. No special tools are required to do these back ups.

The log files have periodic CRC32 checksums written within them to validate the integrity of the data. This guards against the potential for disc corruption.

Querying the logs is blindingly fast because we use a full text search index implemented on the logs. This same type of algorithm that Google uses for internet searches. The logging system does generate index and meta files for these searches. If deleted, Iguana can regenerate these from the main *.log files.

Open access is provided to the logs by means of RESTful APIs which allow customers to extract data from the logs and also APIs in Java/C# etc. which sit on top of these base APIs. This makes it easy to display the log information in the context of user applications or for external archiving of the data.

If you have any additional questions please do free to ask.