Pipe-lining by batching messages: If you are running a high volume data center one topic you should be interested in is what I call pipe-lining, or batching of data from several (or many) messages together into a single transaction.

Iguana processes messages extremely fast, the bottleneck is actually saving the data to database, FTP, HTTP(s), disk files etc.

For now let’s just consider databases, but remember that the same principles apply when saving data to other targets.

if you think about the underlying mechanics of transactions in databases, the bottleneck is always the time needed to physically sync data down to the disc storage. So, with databases, if you insert 100 rows in one transaction, you’ll get dramatically better performance than if you were to do 100 individual transactions.

Relating this to HL7: If the nature of your data set is that you are always inserting data, this method is very good candidate for optimization. This technique will work extremely well if you are processing large batches containing 100s of HL7 transactions.

Then can you consider an algorithm something like this:

1. Use a From Translator component and Lua file handling APIs to break the incoming files into chunks of 100 messages and enqueue these blocks.
2. In a To Translator component break the chunks of 100 into individual messages.
3. Parse each message and map out the data into a single set of tables as you obtain from db.tables{}
4. Then do one call to conn:merge{} which commits all data mapped from the 100 messages in a single database transaction.

This has the potential to give you a much faster throughput for a large amount of batched HL7 data. The performance gains can be dramatic as writing 100 records as opposed to writing a single record is very little extra overhead for a database engine, so saving the data may not be 100 times faster but it certainly will be many times faster!

If you need more information please contact us at support@interfaceware.com.