How to Troubleshoot Iguana Memory Leak

Introduction

Occasionally, we receive Iguana memory leak support inquiries. Here are some of the strategies that can help you to isolate and troubleshoot.

To determine whether there is a memory leak issue, try to observe:

  • Iguana Instance’s Memory  (found on the dashboard)
  • Iguana Crash due to memory leak  (found in the crash dump files)
  • Slow performance on Iguana  (observe slow performance)

And ask the following questions:

  • Has this memory usage resulted in Iguana crashing?
  • How many channels are on this Iguana instance, and how many are running concurrently?
  • What kind of workflows are these channels implementing?

    For example, are they processing HL7 messages through LLP, or reading files from a directory?

  • Has the Iguana service been restarted recently, and if so, has the problem still persisted?
  •  What operating system is Iguana running on, and what is the exact version of Iguana

Cause [top]

  • Memory usage should not be increasing:

    If you look at Memory graphs, from resource Monitor, the lines should be flat, i.e., memory use is not growing over time. If the graphs are angled upwards, or even if they show an upward-trending sawtooth pattern, it would suggest a leaky script.

  • Do you see in your code any global variables that should be declared local variables?

    Unintentional global variables may cause memory leaks, leading to performance issues.

  • Also, was there any code changes made immediately before the issue was noticed?
  • Are there any sheduled or frequent tasks that may not close connections correctly?

    Are there any scheduled tasks running, that are not closing properly or using database connections that are not being closed. These can be the culprit for these kind of memory leaks.

Solution [top]

Most of time memory leak issues happen on a single or a few channels and sometimes is difficult or impossible to recreate memory leak issues on DEV or TEST due to environment specific variables or message volume.

Here are the suggested troubleshooting techniques that you can use:

 

  • Create backup Iguana instances on the same production server (PROD for example)
  • Identify the leaking channel: Turn on and off individual channels on the Iguana Backup instance which channel is causing the memory leak issue:

    Often the issue will be a single channel — when this channel is turned off the memory leak will stop. This method can help to identify which channel (or channels) is causing the leak.

  • Check the global variable in order to make sure that there is no calls that store information in Iguana
  • Observe the setmetatable lua function to make sure that you absolutely need that piece of code
  • Third party lua module or functions that save and store information on Iguana
  • Non persistent database connections can cause a memory leak:

    This problem can occur if there are lot of number of DB connections being opened and closed. If there are a large number of connection the the server may not getting enough time to close the previous connection before a new connection request is being made.

    Note: A persistent database connection is when there is a single connection that is re-used for all database queries. This is implemented by opening a single DB connection in a translator channel. The connection is initialized outside the main() function — so it only runs once when the channel is started, and is closed when the channel is stopped. All queries in the channel will use the same connection. This is very efficient as it reduces database connections/disconnections to the absolute minimum.

    A non-persistent connection is opened and closed inside main() or another function and runs every time a message is processed (or potentially several times per message if several queries are required to process each message). This is much less efficient as every query requires a database connection/disconnection.

    • Check the code to see if the database connections are persistent or not.
    • If DB connections are not persistent then you can modify the code to use a persistent connection:

      Refer to db conn query{}: Reading from a database in Database Fundamentals that shows an example which uses a persistent connection declared outside the main() function.

Tagged: