How to fix the “too many open sockets” error

Introduction

Sometimes our clients will get a “too many open sockets” error on very busy servers. This error occurs when you try to open more than 2048 socket connections for an instance of Iguana.

This error has several variations but will always contain the string “too many open sockets (FD_SETSIZE=2048 reached)“.

If you have any questions please contact us at support@interfaceware.com.

Issue [top]

The “too many open sockets” error occurs when you try to open more than 2048 socket connections for an instance of Iguana.

Solution [top]

  1. Check how many open sockets you have:
    1. Run the socket diagnostic command:
      http://localhost:<Iguana port number>/socket_diagnostic.html

      For example using the default 6543 port:

      http://localhost:6543/socket_diagnostic.html
    2. The results will look something like this:
      socket diagnostic

      Note: In this case the number of ports is low and will not be causing any issues.

      Our support staff will usually request screenshot of this result – as it can be very helpful for diagnosing this error.

  2. Reduce the socket timeout setting:

    The default socket timeout for Iguana is 120 seconds, for high transactions systems this will need to be reduced – the minimum setting is one second. For systems with extremely high transaction volumes there may be more transactions than a single Iguana instance can process – in this case you can split the processing between several Iguana servers.

    1. Adjust the IGUANA_WEB_SOCKET_IDLE_TIMEOUT_SECONDS system variable to a lower value (default is 120 seconds) – you can set it as low as one second.

      Warning: Often we recommend managing system variables using the Iguana Environment Variables setting – but this will not work in this case.

      This is because the environment variable IGUANA_WEB_SOCKET_IDLE_TIMEOUT_SECONDS needs to be created as a system-wide environment variable at the OS level (rather than the Iguana environment variable level), since the Iguana web server starts up before variables are loaded from the IguanaEnv.txt file.

    2. Set the IGUANA_WEB_SOCKET_IDLE_TIMEOUT_SECONDS system variable using the operating system.

      This procedure will vary depending on the operating system you are using. For Windows you will use the System Properties > Environment Variables settings. For Linux you need to update the appropriate user login/startup file, which can vary depending on your Linux setup, so it is best to discuss this with your Linux administrator.

      Note: Be aware that if you create a (global) system variable as opposed to a user variable (for your Iguana logon user) that it will affect all instances of Iguana on the machine.

      1. Windows: Open the the System Properties > Environment Variables settings:

        This will create a (global) system variable as opposed to a user variable (for your Iguana logon user) that it will affect all instances of Iguana on the machine.

        1. Search for Variables in the start menu.
        2. Choose Edit the system environment variables.
        3. Click the Environment Variables button.
          environment variables
        4. Create a new IGUANA_WEB_SOCKET_IDLE_TIMEOUT_SECONDS System Variable.

          You will have to experiment to find the appropriate timeout value – you can set it as low as one second.

          environment variable

      2. Alternatively: If you have created a Windows user for the Iguana Service then you can logon to Windows with this user and create a user variable instead.

        This has the advantage that you will only be setting the socket timeout for the Iguana Server(s) using this login – and other Iguana Servers using a different login (like LocalSystem) will not be affected.

  3. Check that the code does not allow the number of socket connections to continually increase:

    This is basically checking for “connection leaks” in the the same way as you would check code for memory or other resource leaks – this is particularly important for long running programs as any “leak” will eventually reach the system limit

    1. Make sure connections are closed as soon as possible.
    2. Set limits on the maximum number of connections.
  4. You can also review the Socket management section of Tuning a large scale Iguana install – this article was originally written for Iguana 5, but the principles remain the same for Iguana 6.
  5. In the unlikely situation where you actually need more than 2048 sockets you can split the processing between multiple Iguana servers.

Leave A Comment?