This topic contains 27 replies, has 8 voices, and was last updated by  Eliot Muir 2 years, 1 month ago.

New sample data editor

  • So Jonathan Marshall has been working hard on a complete redo of the sample data editor in Iguana. It’s coming together quite nicely.

    There are a lot of finer points with editors – everyone tends to have a different pattern in which they use their keyboard so I suggested to Jonathan that we just have a go at putting a beta of the editor up live on our website so that people can click on it and give it a bang around and give him some early feedback.

    There are some features that haven’t been implemented that we thinking of implementing and it’s a nice opportunity to give some feedback on the whole thing. Try it out here:

    http://help.interfaceware.com/beta/editor/main.html

    Not to forget the EDI (X12) users.
    Would be nice to have Sample Data Editor being able to display X12 segments in individual lines, but without injecting CR and/or LF into the data.

    For example:

    This TA1 acknowledgment

    ISA✽00✽..........✽01✽SECRET....✽ZZ✽SUBMITTERS.ID..✽ZZ✽ RECEIVERS.ID...✽ 930602✽ 1253✽ U✽ 00401✽ 000000905✽ 1✽ T✽ :~TA1✽000000905✽940101✽0100✽A✽000~IEA✽1✽000000905~

    could be shown in editor like (note 0x7E instead of CR or LF):

    ISA✽00✽..........✽01✽SECRET....✽ZZ✽SUBMITTERS.ID..✽ZZ✽ RECEIVERS.ID...✽ 930602✽ 1253✽ U✽ 00401✽ 000000905✽ 1✽ T✽ :0x7E
    TA1✽000000905✽940101✽0100✽A✽0000x7E
    IEA✽1✽0000009050x7E

    Advantages? It is in human readable form without changing actual data by artificially added CR/LF, which would have corrupt data integrity in this case.

    Any invisible characters not at the end of the line are represented as tokens “x0D”. The byte is preserved and saved on submit. It’s currently only working with bytes in the range of if(Code <= "1F" || (Code >= "7F" && Code <= "FF")). This is going to be improved to include all characters less than 0xFF so that if the character is printable it will just add the character, not a token.

    If you open the console and click save you can see what is submitted and that the bytes are preserved. Firefox shows a nice little glyph for non-printable characters, if you look closely you can even see it’s hex number on it.

    Please continue to add any more comments. There are a lot of things left to catch, improve, and implement.

    One thing we’ve been debating on internally is whether the Convert button is needed. Personally I haven’t run into a situation where I had needed to do a bulk replace of newlines in the sample data editor.

    My gut feeling with the design is always to be quite restrained in terms of holding back on features just the from the perspective that if a feature only benefits 0.01% of users but there other ways to achieve that goal that in aggregate having less complexity in GUI is of greater benefit.

    Another discussion is whether to add in a search(&replace) field up in the top right – preferably it should be consistent in look and feel and behavior to the main code editor in the translator.

    (after all if the sample data editor fulfils 99% of uses cases and the occasion time a power user copy pastes the data into their editor of choice for the 1% outliers then this is a decent design tradeoff)

    Jonathan, I see what you say about current range of bytes, but what I suggested is to be able to display 7E as token equivalent to LF when 7E is used for segments terminator in data context, even that it is legitimate printable character. All this without affecting actual data piped to Translator for processing. We can take it to internal discussion if I don’t explain myself.

    0x7E is the squiggle character right? ~

    So basically have an X12 mode which breaks the lines at the segment terminator – yes that makes sense. It’s a good idea.

    A x7E will not be shown as an LF. LF is a (somewhat) well known representation for x0D, same as say, using a special TAB-like token for x09. Also, messages are split as lines, then lines are parsed for bytes. If/when X12 support is added it will be smart enough to know that X12 lines end with x7E, split on those, and could be represented with a x7E token or maybe EOL, which ever is preferred.

    Also, there will be an internal document for detailed discussion.

    yes, to break visually at segment terminator. Just keep in mind that segment terminator won’t be a ~ (7E)in necessary, it is allowed to be something else. At any rate it will be the 106-th character from the begining of x12 interchange envelope

    Yes, currently I have the Hl7 delimiters working dynamically basic on characters 3 through 6. It’s only working on message load right now. Paste is not implemented yet. Here is a screenshot of it working on my local version.

    Attachments:
    You must be logged in to view attached files.

    Eliot, I agree about the Convert button not being necessary. Currently, I do that kind of bulk change in Notepad++. Its RE find/replace makes it really powerful for that.

    I do really like the invisibles being shown.

    For HL7 specifically, it would be nice to have something that helps identify a field location. Counting the “|” to get to OBR-32 gets tiresome and error prone. Most of what I use the sample editor for is to insert or change a value in a field/component.

    I don’t know if the best solution is replicating the segment view below this, or a tool-tip that gives the location (i.e. “OBR-32.2”) when the pointer hovers over the place in the message, or something else.

    Mark, i really like the ” a tool-tip that gives the location (i.e. “OBR-32.2″) when the pointer hovers over the place in the message”. Even better than to display by Segments/fields,etc …

    This can’t work because, like the Translator, VMDs would be required. Currently only the Translator with VMDs can understand Hl7 messages to that extent.

    Actually this might be doable if we can figure out how to expose the Translator’s environment to the sample data editor.

    I think the idea that would work that Jonathan and I have discussed is to make it possible to invoke the sample data editor in drop down panel within the translator environment itself.

    So it would mean you open up the relevant dialogs in the translator to focus in on say the OBR-32.2 fields and edit the sample data in real time and have the dialogs update. It has big benefits versus trying to do something like drop a message browser into a panel on the screen for the sample data editor itself.

    With this approach you could have a complex piece of sample data like a HTTP message with a HL7 payload and still have the utility of being able to open up the relevant focused dialog of the part of message you want to edit. Jonathan is going to have a go and see if he can make it work – it would be a really cool improvement.

    Thoughts?

    To re-phrase it another way what I like about this solution is that it means less ‘tree geometry’ management to use it since if you had more traditional browser panel below the editor there is a lot more clicking and expanding and scrolling of the tree to the right spot.

    Jonathan I like the direction this is going in. I have been playing around with the editor and one issue I noted was when I added a HL7 message with a long base 64 string. The pipe delimiters at the end of the base64 were not recognised by the editor (as in colour highlighted). Also adding additional pipes at the end of this string did not place them in the expected location where my cursor was pointing. I don’t know if this is due to the length of the string or if it is the base64 itself. If you need any further details or a sample of the message I supplied let me know.

    Hi Guys,

    First of all, can I say this looks great! As it is, this would really make life easier for us.

    I was reading back through messages there and I saw you were discussing exposing the translator environment to the sample data editor. If you did this, then it would open up a lot more possibilities for manipulating the data in different ways. How about an anonymize button for HL7 for example, which could replace the patient name, episode number, address, phone etc. with test data in one click?

    For XMl, you could remove all X tags. In the example given, you could remove all <author> tags. This would be useful if you have a large XML and you know a tag is breaking something; it would save removing every one manually.

    The option of exporting the message to a text could be useful too. This way, I would never have to leave Iguana when manually editing messages and saving them for re-submission for example.

    I hope this is of some use!
    Alan, Slainte Healthcare.

    Good spot on the base64 problem Colin – if you could email in the example that would be awesome.

    With the integration into the translator the idea was just to make it a drop down in the translator itself to make it easier to use.

    With anonymizing data the slight challenge is that it’s hard to know what sample data is in there and if we put it in anonymizing code that kinda of works but not quite then it’s a bit of exposure in terms of liability.

    On the positive side though did you know that the sample data is stored in a SQLlite table named after the GUID of the translator? This means it’s completely possible to write scripts in Lua to open sample data up and whip through and anonymize it.

    Thanks for the feedback. Any is appreciated. It’s helpful to know how the sample data editor is used today and what might be useful features.

    For the base64 string, paste isn’t yet implemented so it’s likely caused by that. It could become problematic if the encoded data is unusually large (and on a single line), but that could affect any editor. If it was important to test with this size of a payload, you could save the data in a file and then load it from the Translator and insert into the message this way.

    I like the convert button and would have used it many times when pasting sample data from editors that used the “wrong” line endings. Obviously this can be solved in other ways (ie Notepad++), however I would prefer the simplicity and convenience of having it right there in the editor.

    The other option is to add the search box, and make sure it is able to do search and replace on the line ending.

    How about a hex view for the line endings? So you can choose to view as character or hex. Not a biggie but nice if you want to see what is going on (or are just are more used to thinking hex).

    Tooltip VS tree view.

    I like the tree view as it keeps consistency with the editor (and users are already used to it), also it is very useful when exploring data.

    I also like the tooltip idea that Mark suggests, and it could be quicker than the treeview – so long as the tooltips are responsive (open/change quickly) and no too sensitive (meaning that the actual tip area is large enough to “find” easily with the mouse pointer).

    Personally I would like to see both implemented, but I think the treeview is more important.

    Don’t think it would make sense for us to compete with notepad++ 🙂

    Having said that I think Jonathan’s well on his way to building out a pretty substantial feature that will really improve day to day life within the translator.

    I’ve attached a screenshot which shows his current prototype. In the process he’s upgraded the translator to use the latest version of code mirror which was a major project – it’s yielding some really nice subtle improvements in terms of how copy pasting works inside of the editor – there were a few things I didn’t find perfect which all work really nicely in this newer code mirror version.

    Attachments:
    You must be logged in to view attached files.

    Don’t think it would make sense for us to compete with notepad++ 🙂

    Having said that I think Jonathan’s well on his way to building out a pretty substantial feature that will really improve day to day life within the translator.

    I’ve attached a screenshot which shows his current prototype. In the process he’s upgraded the translator to use the latest version of code mirror which was a major project – it’s yielding some really nice subtle improvements in terms of how copy pasting works inside of the editor – there were a few things I didn’t find perfect which all work really nicely in this newer code mirror version.

    Attachments:
    You must be logged in to view attached files.

    Incidentally the big problem with making the sample data editor itself substantially more ‘grammar aware’ is that it’s really arbitrary what goes into the sample data – if you have a script for instance that takes and ID and executes a SQL query off that ID or you do something like:

    function main(Data)
    Data = “Foo”

    then there isn’t any clear relationship between what is in the sample data tank and what ‘grammar’ to associate with it.

    However having the editor right in the the translator as a dialog means that as a user you can manipulate the usual tree dialogs that are natively built into the translator to give visibility into the parts of the sample data you are editing so you should be able to see your changes in real time. As I said I think this is really moving along the lines to provide a substantial improvement in usability of the translator which should add up to thousands of hours saved by hundreds of people globally ever year – it’s good to know we’ve working on something that can make a real difference.

    Nice work!

    Will the Sample Data selector buttons (currently present in the button bar/upper left corner of the translator window) step forward and backward through the sample message queue as before? And with the new design, update the message editor in the process?

    Given that the message editor will be much more front-and-center and therefore experimentation with modifications to the data much easier, have you given any thought to auto-saving (with, perhaps, a checkbox to enable/disable the feature) changes to the current message when moving to a different message?

    Jeff Drumm ◊ VP and COO ◊ HICG, LLC. ◊ http://www.hicgrp.com

    Nice work!
    Will the Sample Data selector buttons (currently present in the button bar/upper left corner of the translator window) step forward and backward through the sample message queue as before? And with the new design, update the message editor in the process?

    Yes definitely – as of today Jonathan hasn’t got it hooked up like that yet. The auto-saving thing will come into focus after the first cut is done. There are a few modes of doing that – there is the way the general translator works with marking a line dirty (yellow block on the left) and then it saves/re-executes the data in the translator as you move off the line.

    Should also be possible to have a ‘full screen’ button to expand to a full screen view of the sample data like the current one. But first Jonathan has to incrementally get a few more things working – it’s hard to see the flow until those things are in place…

You must be logged in to reply to this topic.