Handling broken MSH segments

Please be aware that the code in this article is intended to give you ideas, it is not production ready. If you need help with similar problems please contact us at support@interfaceware.com.

My question now is whether we could automatically detect the off by one problem? Perhaps a little routine which could parse the MSH segment looking for where the event codes are. Could we get the problem solved in an almost plug and play manner?

This problem that was brought to our attention by one of our customers. He’s encountered a number of off by one problems. One was a counter party that just didn’t know how to count fields. He guessed they knew MSH-9 was message type, so counted backwards but didn’t know that MSH-1=”|”. That led them to MSH-1=”MSH” and PV1-1=”PV1″, so patient class would be after the first separator. The end result being that the MSH segment was correct, and all the other segments off by one. The mis-understanding by the counter party sounds like something that could easily happen in practice. It’s a subtle problem which was compounded in this case with the implementation putting data into random fields (a mapping issue that we do not address here). Off by one issues are complex and the code below does not automatically solve them all, instead it demonstrates an approach to fixing the issues. This is an HL7 message with a valid MSH segment:

MSH|^~&||Lab|Main HIS|St. Micheals|20110213144932||ADT^A03|9B38584D9903051F0D2B52CC0148965775D2D23FE4C51BE060B33B6ED27DA820|P|2.6
EVN||20110213144532||||20110213145902|
PID||4525285^^^ADT1||Smith^Tracy||19980210|F||Martian|86 Yonge St.^^ST. LOUIS^MO^51460|||||||10-346-6|284-517-569|
NK1||Smith^Gary|Second Cousin|
PV1||E||||||5101^Garland^Mary^F^^DR|||||||||||1318095^^^ADT1|||||||||||||||||||||||||20110213144956|
OBX|||WT^WEIGHT||102|pounds|
OBX|||HT^HEIGHT||32|cm|

But sometimes, a vendor sends a message with an extra field at the beginning of every segment, after the name:

MSH|^~&|||Lab|Main HIS|St. Micheals|20110213144932||ADT^A03|9B38584D9903051F0D2B52CC0148965775D2D23FE4C51BE060B33B6ED27DA820|P|2.6
EVN|||20110213144532||||20110213145902|
PID|||4525285^^^ADT1||Smith^Tracy||19980210|F||Martian|86 Yonge St.^^ST. LOUIS^MO^51460|||||||10-346-6|284-517-569|
NK1|||Smith^Gary|Second Cousin|
PV1|||E||||||5101^Garland^Mary^F^^DR|||||||||||1318095^^^ADT1|||||||||||||||||||||||||20110213144956|
OBX||||WT^WEIGHT||102|pounds|
OBX||||HT^HEIGHT||32|cm|

Or one less field:

MSH|^~&|Lab|Main HIS|St. Micheals|20110213144932||ADT^A03|9B38584D9903051F0D2B52CC0148965775D2D23FE4C51BE060B33B6ED27DA820|P|2.6
EVN|20110213144532||||20110213145902|
PID|4525285^^^ADT1||Smith^Tracy||19980210|F||Martian|86 Yonge St.^^ST. LOUIS^MO^51460|||||||10-346-6|284-517-569|
NK1|Smith^Gary|Second Cousin|
PV1|E||||||5101^Garland^Mary^F^^DR|||||||||||1318095^^^ADT1|||||||||||||||||||||||||20110213144956|
OBX||WT^WEIGHT||102|pounds|
OBX||HT^HEIGHT||32|cm|

It would be nice if we could detect this, and either add or remove the field. Below is a module that will do just that. We added some code to doctor a single message so we could show the fixes in a one picture. You may find it easier to just change the messages manually. Then you can lose the kludgy msgAddField() and msgRemoveField() and simplify the code like this: Cut and paste the code from below. This code goes in main:

require("oneoff")
require('stringutil')

local function trace(a,b,c,d) return end

function main(Data)

   -- A normal message will not be changed
   local msg = oneoff.adjust(Data)  
   trace(msg)
   assert(msg == Data)

   -- Add a  field - then fix it
   msg = msgAddField(Data)
   trace(msg)
   msg = oneoff.adjust(msg)
   trace(msg)
   assert(msg == Data)

   -- Remove a field - then fix it
   -- Only removes a blank 3rd field
   msg = msgRemoveField(Data)
   trace(msg)
   msg = oneoff.adjust(msg)
   trace(msg)
   assert(msg == Data)
end

function msgAddField(Msg)
   Msg = oneoff.genMsg(oneoff.forEachSegment(oneoff.parse(Msg),oneoff.addField))
   return Msg
end

function msgRemoveField(Msg)
   local T = oneoff.parse(Msg)
   if T.segments[1][3]=='' then -- only remove a blank 3rd field
      Msg = oneoff.genMsg(oneoff.forEachSegment(oneoff.parse(Msg),oneoff.removeField))
   end
   return Msg
end

Or the simple version:

require('oneoff')
require('stringutil')

local function trace(a,b,c,d) return end

function main(Data)

   -- Fix message
   local msg = oneoff.adjust(Data)  
   trace(msg)
   assert(msg == Data)

end

This goes in the oneoff module:

require 'stringutil'

oneoff = {}

local function trace(a,b,c,d) return end

-- iterates through segments, applies the 
-- operation to the segment
function oneoff.forEachSegment(Parsed, Operation)
   for i,v in pairs(Parsed.segments) do Operation(Parsed,i,v) end
   return Parsed
end

-- Takes the output from parse and reconstructs the
-- hl7 message
function oneoff.genMsg(Parsed)
   local output = ''
   trace(#Parsed.segments)
   for i,Segment in pairs(Parsed.segments) do 
      if i < #Parsed.segments then
         output = output..table.concat(Segment,Parsed.delimiter)..'r'
      else
         output = output..table.concat(Segment,Parsed.delimiter)..Parsed.termination
      end
   end
   return output
end

-- a segment operation that will remove the extra pipe character
-- after the segment name
function oneoff.removeField(Parsed,i,Fields)
   --assumes that first segment is MSH
   local RemoveIndex = 2
   -- skip MSH delimiter definition
   if (i == 1) then RemoveIndex = 3 end
   table.remove(Fields,RemoveIndex)
end

-- a segment operation that will add an extra pipe character
-- after the segment name
function oneoff.addField(Parsed,i,Fields)
   local InsertIndex = 2
   -- skip MSH delimiter definition
   if (i == 1) then InsertIndex = 3 end
   table.insert(Fields,InsertIndex,'')
end

-- parses into segments and fields
-- return a table with the discovered delimiter and the 
-- segments+fields
function oneoff.parse( Data )
   local Delimiter = Data:sub(4,4);   
   local Segments =  string.split(Data,'r')
   local T = ''
   -- get Data terminator - append in genMsg()
   local i = -1
   while Data:sub(i,i)=='r' or Data:sub(i,i)=='n' do
      T = Data:sub(i,i)..T
      i=i-1
   end
   trace(T)
   -- remove any segments that are just white space
   local SegmentIndex = 1
   while SegmentIndex <= #Segments do
      if (Segments[SegmentIndex]:find('^%s*$')) then
         table.remove(Segments,SegmentIndex)
      else     
         SegmentIndex = SegmentIndex + 1
      end
   end
   --parse out the fields
   for SegmentIndex,Segment in pairs(Segments) do
      Segments[SegmentIndex] = string.split(Segment,Delimiter)
   end
   return { segments=Segments, delimiter=Delimiter, termination=T }
end

-- This function determines if a message should remove a field
-- or add a field after every segment name, by checking 
-- to see if the HL7 version number is in the right spot
-- and returning an adjustment operation
function oneoff.detectOneOffSegments( Parsed )
   local Fields = Parsed.segments[1];
   local VersionIndex = 12;
   local VerPattern = '2%.%d-'
   if (#Fields >= VersionIndex-1 and Fields[VersionIndex-1]:find(VerPattern)) then
         return oneoff.addField
   elseif (#Fields >= VersionIndex and Fields[VersionIndex]:find(VerPattern)) then
      return nil -- nothing to do
   elseif (#Fields >= VersionIndex+1 and Fields[VersionIndex+1]:find(VerPattern)) then
      return oneoff.removeField
   end   
   error('Cannot find version number')
end

-- A function that detects and applies the one off adjustment
-- as necessary
function oneoff.adjust(Input) 
   local Parsed = oneoff.parse(Input)
   local Operation = oneoff.detectOneOffSegments(Parsed)
   if (Operation) then
--      trace(tostring(Operation))
      return oneoff.genMsg(oneoff.forEachSegment(Parsed,Operation))
   end
   -- else nothing to do, return input
   return Input   
end

return oneoff

Also you need string.split() function in stringutil.lua, if your copy of stringtuil.lua doesn’t have string.split(), you can download the latest version from our code server.