• Introduction
• Task
• Implementation
• How it works
• More information

Introduction

This article demonstrates a technique where you can resubmit messages in bulk, while continuing to process incoming messages.

There are three “standard” ways to resubmit messages that we explain in Reprocessing and Resubmitting Log Messages:

1. Choose an individual message to be re-sent:

   This method allows you resubmit a one message at a time. This is excellent for one or two processing errors — but it takes a long time if you need to re-send more than a few messages.

2. Reposition the queue to start from a specified message:

   This method allows you to “go back to an earlier point in time” and re-send messages that were previously processed. This has the disadvantage that all the messages front the selected time are processed — there is no way of excluding some messages that you do not want to reprocess.

3. Export messages and resubmit then using the HL7 Simulator:

   This method is the most flexible. You simply export the messages you need to re-sent into a file an resubmit them using the HL7 Simulator. This means you can use the filter the logs before exporting to exclude messages you do not want to resend — and you can edit the exported message file to modify or remove messages. However this has three disadvantages: First you can only resend messages to a channel with an LLP Listener source, you need to edit the LLP Listener to listen to the HL7 Simulator, and you have to (temporarily) stop process incoming messages.

The method we demonstrate here is really a variation on the third method but using channels instead of using the simulator. It takes a little bit of work to set it up, but once it is working it makes it much easier to resubmit messages.

Using a single source channel means you need stop processing incoming messages while you are resubmitting messages.

If you split the input and processing into separate channels, then you can add a second input channel to resubmit messages from a file. This allows you to resubmit messages while continuing to process the incoming messages.

Some examples of when you may need to resubmit messages:

• When a downstream application is offline for a period of time:

  This can occur if your channel is setup to skip messages and continue processing (rather than stopping the channel), see Trap errors with pcall() for more information.

• There is an intermittent network issue:

  In this case we recommend using the retry module to resend messages for unreliable connections. This will help to minimize the number of messages that need to be resubmitted.

• When a downstream application rejects messages it receives:

  In this case you would need to fix the messages before resending them. For example you may get NACK responses for HL7 messages.

• Messages with invalid information need to be resent:

  You may realize that incorrect data was included in messages and need to resubmit to make corrections. Exactly how this would work will depend on the circumstances

Task [top]

How to resubmit a subset of messages that are being sent to a destination system without disrupting the current messages that are being sent.

Implementation [top]

Our example shows how to split a single channel into an input and an output channel. In this case the original channel is processing HL7 messages using a From LLP source and a To LLP destination. The same method can be used for any channel that queues messages.

First we list the general steps, and then the steps for a simple example that processes HL7 messages.

General steps:

1. Choose the “original” channel that you need to resend messages to.
2. Split the original channel into input and an output channels:

   The input channel simply queues the messages for processing by the output channel. Any processing done by the original channel must be done by the output channel (no processing is done by the input channel).

   1. Create the input channel:
      • Use the same Source Component as the original channel.
      • Use a To Channel destination component.
   2. Create the output channel:
      • Use a From Channel source component.
      • Use the same Destination Component as the original channel.
   3. Setup the output channel:
      • Configure the From Channel source component to receive messages from the input channel.
      • Move all processing done by the original channel to the Filter component of the output channel:
        

        If the code in the “original” channel was all contained in the Filter component — then copying that code to into the Filter component of the output channel is all that is required. If however there was also code in the source component (like Iguana custom ACKs) then you will need to copy this also — feel free to contact us at support@interfaceware.com if you need help with this.

3. Create the File Input channel:

   The file input channel allows you to resubmit messages from a file to the output channel. The output channel simply queues the resubmitted messages for processing by the output channel.

   Tip: If you need to do more sophisticated logic around file handling you can use a From Translator component instead of a File Input. This would enable you to do things like: Only processing files with specific names (rather than only relying on extensions), or perhaps processing different file types (extensions) differently.

   However in most cases using a File Input component should be sufficient.

   1. Create the File Input channel:
      • Use a From File source component.
      • Use a To Channel destination component.
   2. Setup the From File source component for the File Input channel:
      • Choose the input directory.
      • Choose the file type(s) to process.
      • Choose the action to perform for processed files (move or delete).

        Note: We strongly recommend moving the processed files rather than deleting them.

   3. Configure the the From Channel source component for the output channel to receive messages from the File Input channel:
4. Export the messages that you want to resubmit from the logs:

   You can take advantage of log search and log regex search functionality to find the messages that you want to resubmit, then you can then export them to a text file. To extract messages associated with error messages you can click on Related Messages. If you are moving the files after processing you will need to ensure that the names of the exported files are unique.

   Note: The log export creates a unique file name in the format log_export_yyyymmdd_hhmmss, like log_export_20190220_201659.txt — you can use this or customize it (though we suggest keeping the time component for uniqueness).

5. Place the exported message file in the source directory for your File Input channel:

   Once you have your text file with all of the messages that you want to resubmit, place the file in the source directory for your From File component of the File Input Channel.

6. Start your File Input Channel to process the messages from the text file.
7. Once the text file has been processed, you can turn off the File Input Channel:

   Once the processing has completed the input file will either deleted or moved to a “processed” folder — this is how you will know the processing is finished.

8. Observe the receiving channel to verify that it has received and processed the resubmitted messages and that no problems occured:

   We suggest you compare log entries for the three different channel configurations:  First the original single channel, then compare to the equivalent input + output channel, and finally compare to the resubmit configuration of file input + output channels. Finally you need to check the content of the resubmitted messages to ensure that everything is correct.

Example:

1. Choose the “original” channel that you need to resend messages to:

   In this case we will create our own example channel called LLP Single Channel. This channel uses one the most common channel configurations: An LLP Input source component, some simple Filter component processing, and an LLP Client component.

   1. Create a new channel called LLP Single Channel.
   2. Choose the channel components:
      • Source: LLP Listener
      • Destination: LLP Client
   3. Enable the Filter component:
      • Check the Use Filter checkbox
      • Leave other settings as default
   4. Import the code for the Filter component:

      If the code in the “original” channel was all contained in the Filter component — then copying that code to into the Filter component of the output channel is all that is required. If however there was also code in the source component (like Iguana custom ACKs) then you will need to copy this also — feel free to contact us at support@interfaceware.com if you need help with this.

      1. Click the Edit Script… link in the Filter properties.
      2. Import this project zip file: LLPsinglechannel-Filter.
2. Split the original channel into input and an output channels:

   The input channel simply queues the messages for processing by the output channel. Any processing done by the original channel must be done by the output channel (no processing is done by the input channel).

   1. Create the input channel:
      • Channel Name: “LLP input channel”
      • Choose the channel components:
        • Source: LLP Listener (the same source component as the original channel)
        • Destination: To Channel
   2. Create the output channel:
      • Channel Name: “LLP output channel”
      • Choose the channel components:
        • Source: From Channel
        • Destination: To Client (the same destination component as the original channel)
   3. Setup the output channel:
      1. Configure the From Channel source component to receive messages from the input channel:
         1. Edit the Source properties for the LLP output channel.
         2. Choose the LLP input channel from the Add a source channel droplist:
            
         3. Save the changes.
         4. The result should look like this:
            
      2. Import the code for the Filter component:

         This moves all the processing done by the original channel to the Filter component of the output channel.

         1. Click the Edit Script… link in the Filter properties.
         2. Import this project zip file: LLPsinglechannel-Filter.
3. Create the File Input channel:

   The file input channel allows you to resubmit messages from a file to the output channel. The output channel simply queues the resubmitted messages for processing by the output channel.

   1. Create the File Input  channel:
      • Channel Name: “LLP file input channel”
      • Choose the channel components:
        • Source: From File
        • Destination: To Channel
   2. Setup the From File source component for the File Input channel:

      We chose to create a directory reprocess_hl7 in the users home directory — you will need to create this directory for the example code to work. The home directory is home directory of the user you are logged on as, it can also be found by checking the value of the Iguana HOME environment variable. We will use the ${HOME}/<directory> syntax in the file directory fields of the From File component.

      1. Create the input directories:
         1. Create the reprocess_hl7 directory in you home directory.
         2. Create the processed directory within the reprocess_hl7 directory.
      2. Choose the file type(s) to process: In this case we will use txt (.txt extension).
      3. Choose the action to perform for processed files (move or delete): In this case we will move the files.
      4. Set the properties for the the From File source component (to reflect the choices above):
         • Source directory: ${HOME}/iguana/reprocess_hl7/
         • Extension of files to read: txt
         • Full directory path to move error files into: ${HOME}/iguana/reprocess_hl7/processed/
         • Full directory path to move processed files into: ${HOME}/iguana/reprocess_hl7/processed/
      5. The result will look similar to this (notice how Iguana resolves the ${HOME} variable):
         
   3. Configure the the From Channel source component for the output channel to receive messages from the File Input channel:
      1. Edit the Source properties for the LLP output channel.
      2. Choose the LLP file input channel from the Add a source channel droplist.
      3. Save the changes.
      4. The result should look like this:
         
4. Export the messages that you want to resubmit from the logs:

   You can take advantage of log search and log regex search functionality to find the messages that you want to resubmit, then you can then export them to a text file. To extract messages associated with error messages you can click on Related Messages. If you are moving the files after processing you will need to ensure that the names of the exported files are unique.

   Note: The log export creates a unique file name in the format log_export_yyyymmdd_hhmmss.txt, like log_export_20190220_201659.txt — you can use this or customize it (though we suggest keeping the date and time component for uniqueness).

5. Place the exported message file in the source directory for your File Input channel:

   Once you have your text file with all of the messages that you want to resubmit, place the file in the source directory for your From File component of the File Input Channel.

   Copy the file to the input directory (we kept the default name), the result looks like this on a Mac:
   

6. Start your File Input channel to process the messages from the text file.
7. Once the text file has been processed, you can turn off the File Input Channel:

   Once the processing has completed the input file will either deleted or moved to a “processed” folder — this is how you will know the processing is finished.

   As we chose to move the file to a processed directory, the result looks like this:
   

8. Observe the receiving channel to verify that it has received and processed the resubmitted messages and that no problems occurred:

   We suggest you compare log entries for the three different channel configurations:  First the original single channel, then compare to the equivalent input + output channel, and finally compare to the resubmit configuration of file input + output channels. Finally you need to check the content of the resubmitted messages to ensure that everything is correct.

   1. There are five log entries for each successfully processed message with the LLP single channel:
      • The incoming processed message
      • The ACK returned for the incoming message
      • The message read into the Filter component
      • The message output from the Filter component
      • The ACK returned from the down-line LLP Client
        
   2. There are also five equivalent log entries for each processed message using the LLP input channel + LLP output channel:
      • LLP input channel: The incoming processed message
      • LLP input channel: The ACK returned for the incoming message
      • LLP output channel: The message read into the Filter component
      • LLP output channel: The message output from the Filter component
      • LLP output channel: The ACK returned from the down-line LLP Client
        
   3. There are four log entries for each processed message using the LLP file input channel + LLP output channel:
      • LLP file input channel: The incoming processed message
      • LLP output channel: The message read into the Filter component
      • LLP output channel: The message output from the Filter component
      • LLP output channel: The ACK returned from the down-line LLP Client
        

        Note: The first ACK is missing as the resubmitted message is read from a file not received from an LLP source.

   4. Check the resubmitted messages to ensure that everything is working correctly:
      • Checking for NACKs from resubmitted messages
      • Visual checks to ensure that the content of the output messages being sent are correct
      • Visual checks to ensure that the content of resubmitted messages read from file are correct
      • Count the number of messages the messages being resubmitted
      • Any other checks that you wish to do

How it works [top]

There are issues when you are using a single channel interface because you need stop processing incoming messages while you are resubmitting messages — and you may need to (temporarily) reconfigure the interface channel to accept the resubmitted messages (perhaps the resumitted LLP messages could come from a different port).

For example a From LLP source component can only accept one incoming LLP connection at a time. This means you have to stop your incoming LLP message stream first in order to resubmit messages (either using the HL7 Simulator, or a resubmit channel). Other similar issues can occur with different source components.

If you split the input and processing into separate channels, then you can add a second input channel to resubmit messages from a file. This allows you to resubmit messages while continuing to process the incoming messages.

Although our example was for From LLP to To LLP channels, this could easily be modified to work with other workflows, such as channels with From/To Translator components etc.

More information [top]

• Reprocessing and Resubmitting Log Messages
• Accessing the Logs From Other Applications