Introduction
Sometimes our clients will get a “too many open files” error on busy servers. This error occurs when a large number of files, databases or web service connections are open – this usually occurs because connections are not closed promptly.
The maximum number of files you can open for a Windows Iguana Server is 512 and 1024 on Linux or Mac – if you attempt to exceed these numbers you will this error.
Note: This page is about the mindset of limiting the number of connections that you use – rather than specific techniques. The specific techniques may vary depending on what you need to do in each case – but the principle of keeping connections to a minimum remains the same.
Issue [top]
Too many file, database or web service connections are open at one time.
Solution [top]
- Inspect the code to ensure that it closes connections immediately it has finished using them.
- Check that the code opens the minimum number of connections required:
- If you are processing a large number of files you can open one or a group of several at a time rather than all of them. Then you can close the first group before processing the next group.
- You might set an upper limit on connections and not open more when the limit is reached. Then when the number of connections drops you can open more.
- In particular you need to ensure that persistent connections are kept to a minimum:
Persistent connections (particularly to databases) can be very efficient – and it is possible reduce the number of connections by using a small number of persistent connections. The key thing with persistent connections is to ensure that the number cannot keep increasing.
- Persistent connections to resources that you are re-using all the time make good sense – they are particularly useful for databases
- This page explains about persistent database connections: https://help.interfaceware.com/v6/database-connection
- Make sure that you are not continuing to open extra or duplicate persistent connections unintentionally (i.e., logic error that opens multiple database connections).
- Check that the code does not allow the number of connections to continually increase:
This is basically checking for “connection leaks” in the the same way as you would check code for memory or other resource leaks – this is particularly important for long running programs as any “leak” will eventually reach the system limit.
- Make sure connections are closed as soon as possible.
- Set limits on the maximum number of connections.
- In the unlikely situation where you actually need more files open than the limits allow you can split the processing between several Iguana servers.
Note: It may be possible to increase the number of file handles per server on Windows, please contact us at support@interfaceware.com to discuss this.
Warning: Experienced Linux admins will know that it is possible to change the number of files (sockets) available for a process. Do not increase this number above 1024 as this will cause problems for Iguana.
Iguana uses the select library on Linux and Mac – select has a maximum of 1024 files (sockets).
Prevention [top]
There are various ways to prevent this issue:
- Open the minimum number of connections.
- Close connections immediately when you are finished using them.
- Set limits on the maximum number of connections that you allow.
- Ensure that your code does not allow a continually increasing number of connections.
- If you need to open an extremely large number of files you can split the processing across multiple Iguana instances.