Background
A customer reported an Iguana message error condition in the following test scenario: 17 channels and 2,500 messages queued per channel (42,500 messages in total). The channel used the ‘net.http’ module to act as a web client, and to communicate with a web server hosting a C# based web service. The customer reported at approximately 16,500 messages, Iguana began to report errors. Netstat output showed a similar number of listings for web service’s IP and port. The one-to-one correlation between messages and sockets meant that the cURL was not reusing the socket connection. After a period of time, an operating-system-imposed limit constrained Iguana, and it reported an error condition.The Iguana Translator’s ‘net.http’ module is an HTTP 1.1 compliant client that uses the cURL library for its implementation. Different combinations of HTTP verbs and headers in HTTP negotiation can have a dramatic effect on effective message throughput. In practice, the Iguana Translator uses a standard cURL library and can reuse socket connections. In fact, in baseline tests, the Translator has demonstrated 1,000+ messages per second per channel.
Two approaches were tested for effects on speed and Iguana service under load. One focused on closing HTTP sockets as quickly as possible, while the other focused on reusing HTTP socket connections.
Approach #1
After the socket client and server processes are both closed, the open sockets linger for an operating system configured time (default is 2 minutes). This is reported by netstat until the operating system cleans them up. The operating system can be configured to clean up the sockets more quickly by lowering the TcpTimedWaitDelay registry entry.
Approach #1 is a quick fix to improve service under load. It updates a registry setting to close idle sockets quicker, which frees up system resources. This quick fix will provide a scalability improvement, but not a complete solution. The issue of Iguana reporting a socket error may be resolved with this setting, or at the very least, the socket resource exhaustion symptom may appear less frequently.
Approach #2This approach is preferred for both speed and improved service under load. Test results show that Iguana cURL libraries support socket reuse. Setting the ‘Connection=Keep-Alive’ header in both the HTTP client request and the HTTP server response is integral for socket reuse. Online documentation for cURL and web servers both suggest that this is the default behaviour. To run high message loads between an Iguana client and a web server it is recommended that HTTP socket connections be reused.
Evaluating approaches [top]
We considered two approaches for improving the effective message throughput of Iguana when acting as a web client. Approach #1 was an attempt to close sockets quickly and avoid reaching operating-system-imposed limits on the number of open sockets possible. Approach #2 concentrated on making sure socket reuse was in effect. The Iguana HTTP client ran on Mac OSX. The HTTP server was Apache 2.2.25, running on Windows. Five variations of web client scenarios are described below:
1. Close sockets quickly using ‘Connection=close’
The HTTP client header ‘Connection = close’ was set in the Iguana Translator script.
When the test was run, Iguana queued up 30,000 messages to send to the web server. Iguana started to report errors at ~ 16,000 messages. Netstat output showed 16,000 sockets in a TIME_WAIT state. After about 2 minutes, netstat output showed that all sockets had been closed.
The results show that Iguana works within the operating-system-imposed handle limit, but it can’t continue once that limit is reached. The operating system cleans up closed sockets after a 2-minute time delay.
2. Close sockets quickly using ‘Connection=close’ and ‘TcpTimedWaitDelay=30’
The HTTP client header ‘Connection = close’ was set in the Iguana Translator script. We also set the registry entry ‘TcpTimedWaitDelay’ to 30 seconds, and the machine was re-booted.
When the test was run, Iguana queued up 1,000 messages to send to the web server. The messages were delivered to the web server in under 3 seconds. The Iguana process and the web server process were both exited. Netstat reported approximately 1000 sockets in a TIMED_WAIT. After 30 seconds, netstat reported 2 sockets open. No errors were reported. The results show that cleaning up closed sockets is the responsibility of the operating system, and that the timing can be configured.
Note that a second registry entry (‘MaxUserPort’) can also be increased in support of having a large number of connections to a server.
3. Reuse sockets by using ‘Connection=Keep-Alive’
The HTTP client header ‘Connection = Keep-Alive’ was set in the Iguana Translator script. 100,000 messages were sent and results recorded. The sending of 100,000 messages was iterated until there were 10 sets of results. The time to completely send one batch and note the socket count was observed on the Windows machine by watching the Apache access.log, and by using netstat.
| Iteration # | Time to Send 100,000 Messages (in Seconds) | Total Number of Sockets at End of Iteration (netsat report) |
|---|---|---|
| 1 | 69 | 330 |
| 2 | 79 | 357 |
| 3 | 80 | 399 |
| 4 | 85 | 355 |
| 5 | 94 | 318 |
| 6 | 101 | 291 |
| 7 | 110 | 300 |
| 8 | 114 | 241 |
| 9 | 123 | 220 |
| 10 | 129 | 244 |
| Average | 98 | 306 |
- Average “Messages per Second” were calculated as 100,000 messages over 98 seconds = 1020 messages per second
- 100,000 messages over 306 sockets = 327 messages per socket
- Netstat reported that on average, 306 sockets were open. No errors were reported.
These results show that Iguana could send 100,000 messages without interruption at very high message throughput, and that more than one message could be sent on a single socket.
4. Reuse sockets by using ‘Connection=Keep-Alive’ and running two channels
The HTTP client header ‘Connection = Keep-Alive’ was set in the Iguana Translator script. Two identical channels were run concurrently, sending 250,000 messages through each channel.
| Interval # | Elapsed Time Recorded At Every 100,000 messages sent Interval | Total Number of Sockets at End of Interval (nestat report) |
|---|---|---|
| 1 | 50 | 616 |
| 2 | 109 | 536 |
| 3 | 150 | 793 |
| 4 | 194 | 744 |
| 5 | 240 | 609 |
| Average | n/a | 660 |
- 500,000 messages were sent in 240 seconds
- “Messages per Second” were calculated as 500,000 messages over 240 seconds = 2,083 messages per second
- Netstat reported that on average, 660 sockets were open
These results show the following:
- Iguana could send 500,000 messages without interruption at very high message throughput
- More than one message could be sent on a single socket
- Channels could operate concurrently for improved message throughput
5. Reuse Sockets by using ‘Connection=Keep-Alive’ and by using POST instead of GET
The HTTP client header ‘Connection = Keep-Alive’ was set in the Iguana Translator script.
The HTTP POST verb was used instead of GET, and 100,000 POST Requests were sent.
| Iteration # | Time Interval for Iteration in Seconds | Netstat Reported Total Number of Sockets at End of Interation |
|---|---|---|
| 1 | 70 | 448 |
- “Messages per second” calculated as 100,000 messages over 70 seconds = 1,428 messages per second
- Netstat reported that the number of sockets in use peaked at 448
These results show that GET and POST had a similar timing.
Conclusion
Both the Iguana web client and the third party web server participating in an HTTP 1.1 conversation have to be configured to reuse sockets with ‘Connection=Keep-Alive’ to achieve high performance under heavy message load.
Sandbox Environment [top]
Hardware
Scenarios were run entirely on the following hardware:
- Model Name: MacBook Pro
- Model Identifier: MacBookPro8,2
- Processor Name: Intel Core i7
- Processor Speed: 2.5 GHz
- Number of Processors: 1
- Total Number of Cores: 4
- L2 Cache (per Core): 256 KB
- L3 Cache: 8 MB
- Memory: 8 GB
- Boot ROM Version: MBP81.0047.B27
Operating System
- XP 64-bit SP2 in Parallels 8 on Mac
- OS X 10.8.3
Software
- Iguana 6.0-snapshot
- Apache 2.2.25 (Win32)
Iguana Channel Setup
- Source component = From Translator
- Destination component = To Translator*
*This component makes the HTTP GET and POST requests to the web server.
Example Headers
request header
[‘Keep-Alive’]=’timeout=5, max=10000′ [‘Connection’]=’Keep-Alive’response header
[‘Connection’]=’Keep-Alive’ [‘Date’]=’Tue, 03 Sep 2013 01:02:09 GMT’ [‘Content-Type’]=’text/plain’ [‘ETag’]=’\”40000000376e6-5e-4e56cb57702bb\”‘ [‘Last-Modified’]=’Mon, 02 Sep 2013 20:49:30 GMT’ [‘Keep-Alive’]=’timeout=5, max=99′ [‘Content-Length’]=’94’ [‘Accept-Ranges’]=’bytes’ [‘Server’]=’Apache/2.2.25 (Win32)’- The ‘timeout’ header parameter indicates the time that a connection will be allowed to remain idle before it is closed
- The ‘max’ header parameter indicates the maximum number of requests that will be permitted before the connection is closed
- ‘Connection=Keep-Alive’ is the critical header attribute sent by both client and server to ensure that the conversation reused the socket
- ‘Content-Length’ should also be provided and correctly set