Iguana application to import and export channels.

This is something we have been working on internally and tried out with a few customers. I had a urgent request to make this available now so I am putting this up, as is, as now.

It does have a few rough edges.

Firstly:

1. You’ll need to install Iguana 5.6.5
2. You will need the command line utility ‘zip’ installed on your system.
3. You will probably need to edit the script to put the location of the directory to export and import channels to

Not to worry – happy to help people that have an interest in trying out this utility and also chat through what some of applications this thing has.

The beauty of it is not that it’s an out of the box GUI – although it does have a dashboard GUI but that there is an example Lua library which could allow you to script out operations importing and exporting channels.

The API can work on remote Iguana servers too so you can potentially move channels from one server to another remotely.

Attachments:

You must be logged in to view attached files.

To install it you will need to do at least the following:

1. Make sure you have Iguana 5.6.4 or above installed.
2. Make a new channel – From HTTPS –> To Channel
3. Make the From HTTPS component a From Translator component
4. Give it a base URL something like /channelmanager/
5. Import the zipped translator above
6. You’ll probably need to edit cm.app to set cm.config.channelExportPath – use / instead of \ characters for directory separators.

Check to see if ‘zip’ is installed on your system. You can do this by typing ‘zip’ at the command line and you’ll see something like this:

D:\community\iguana-web-apps\other>zip
Copyright (c) 1990-2008 Info-ZIP – Type ‘zip “-L”‘ for software license.
Zip 3.0 (July 5th 2008). Usage:
zip [-options] [-b path] [-t mmddyyyy] [-n suffixes] [zipfile list] [-xi list]
The default action is to add or replace zipfile entries from list, which
can include the special name – to compress standard input.
If zipfile and list are omitted, zip compresses stdin to stdout.
-f freshen: only changed files -u update: only changed or new files
-d delete entries in zipfile -m move into zipfile (delete OS files)
-r recurse into directories -j junk (don’t record) directory names
-0 store only -l convert LF to CR LF (-ll CR LF to LF)
-1 compress faster -9 compress better
-q quiet operation -v verbose operation/print version info
-c add one-line comments -z add zipfile comment
-@ read names from stdin -o make zipfile as old as latest entry
-x exclude the following names -i include only the following names
-F fix zipfile (-FF try harder) -D do not add directory entries
-A adjust self-extracting exe -J junk zipfile prefix (unzipsfx)
-T test zipfile integrity -X eXclude eXtra file attributes
-y store symbolic links as the link instead of the referenced file
-e encrypt -n don’t compress these suffixes
-h2 show more help

If you don’t have it. Then get it from http://www.info-zip.org/

Here is the zip.exe to be used with current version of this project. Until porting the code over to work with the in memory zip will be completed. Place the zip file in location specified in PATH variable, so that script could access it easily.

trying to attach the zip of zip.exe

Attachments:

You must be logged in to view attached files.

This is new version of the channel manager which doens’t require zip.exe. This is because it uses the new filter.zip.deflate and filter.zip.inflate methods that were introduced in Iguana 5.6.5.

Attachments:

You must be logged in to view attached files.

And this latest version uses standard LF endings lines – both on Windows and POSIX.

Attachments:

You must be logged in to view attached files.

Next iteration – this converts \r\n line endings to \n on all platforms and only writes out files which have changed.

Attachments:

You must be logged in to view attached files.

So the next snafu this version solves is that:

1. It only writes out files which have changed.
2. It does chmod a+w on all files which it writes out – to make sure Iguana can write to these files – useful if you are switching from a Mac to a Windows machine.

Attachments:

You must be logged in to view attached files.

Looks like another little patch is needed to make that code work on windows since os.fs.chmod does not exist in the windows version of Iguana.

Attachments:

You must be logged in to view attached files.

Hello Eliot,
Thanks for a terrific tool. It will be very useful.

I did hit one small snag however. I started with exporting channels but one channel failed with an error. Since the export file name is based on the channel name, any channels that contain an illegal file name character will not work. In my case, it was a colon “:” that was the culprit. I added a function in file.lua to escape the invalid characters in the path (attached) and this may help others if they hit a similar issue.

Thanks again.

Attachments:

You must be logged in to view attached files.

Ah thanks for that patch – I’ll see if we get that merged into the repo tomorrow. Excuse me for all the links below – I’m in documentation mode preparing for the masters class on Friday but I think you’ll find this material of interest.

I was thinking about reaching out to you to see if you’d mind if we put your work on FHIR into the repo? I guess it many ways it doesn’t really need to go into our official repo since you pretty much just put your own repo out there and use the channel manager to export it – we’re just getting our heads around using GIT effectively:

http://help.interfaceware.com/kb/regression-testing-round-ii-ding-ding

We made another release of the channel manager yesterday in preparation for the masters class on Friday. This version eliminates the step of having hack your username and password into the application – it’s gets you to login through the web api so it makes it little easier to install:

http://help.interfaceware.com/kb/channelmanager

In the next couple of iterations we should get it so we keep the repo directory in a configuration file and after that we’d be in good shape to add a button that could auto-update the channel manager itself.

There are awful lot of things that one can do with this thing. Some of more obvious applications are:

http://help.interfaceware.com/kb/high-availability-and-promoting-interfaces-from-development-to-test-to-production

But more importantly this stuff is like lego. You can put the pieces together to more and more outlandish things. For instance if you take the core library in the channel manager:

http://help.interfaceware.com/kb/the-anatomy-of-an-iguana-app/6

It basically gives you a library that makes it convenient to export and import channels. Now if you combine that with a global dashboard that operates through firewalls:

http://help.interfaceware.com/kb/monitoring-remote-iguana-instances

Now it’s possible to have the remote Iguana’s accept commands from the central Iguana instance. So put that together with the channel API and you have the pieces that can push updated channels out to remote Iguana instances. Bit scary you say? Yep I’d agree unless you have the infra-structure required to properly unit test those interfaces, for that you need good regression testing:

http://help.interfaceware.com/kb/regression-testing-round-ii-ding-ding

That’s a just a beginning of what we’re doing with regression testing.

Friday in the office will a lot of fun. We’re doing the second installment of our masters class. Basically the development team gets a day off to go and hack Iguana apps with our customers that are coming for the day. It’s bit of crazy experiment but I think we’ll have a lot of fun with it:

http://help.interfaceware.com/kb/9262

The by products of what we’ve produced already have value – the next few months should produce some creative anarchy but we’ll see a lot of useful things drop out very quickly.

All good fun!

Anyways… basically this approach to building these Apps in this manner offer a means to implement ideas I have had for years but aren’t practical to deliver in more traditional models.

Regression testing for instance is a pretty hairy problem that is highly dependent how someone has implemented their interfaces.

Hi Eliot,
Please use anything from my work on FHIR interfaces as you see fit. I’m happy to share it.

There was certainly some interesting reading in the links. I’m still trying to get me head around some of it. The Channel Manager did raise an interesting question around source control for me, and that was managing foreign code. I’m sure that this will be a familiar scenario for you but I’d be interested in your thoughts.

The problem – as soon as I installed the Channel Manager all my other channels broke. Over time, I had extended node.lua, stringutil.lua and others with my own library of functions and the shared modular structure of Iguana had made these functions available to all my channels. As soon as I installed Channel Manager, the default modules overwrote my customised ones 🙁

Fortunately, fossil had copies – although finding the most recent update across quite a few channels was a challenge – and I soon up and running again. I’ve now moved all my custom code to its own module so hopefully this won’t happen again unless that module names gets used by another channel I import.

How do you think this scenario could be best managed?

Thanks.

Yep that’s a very common problem. stringutils.lua and node.lua are the modules that get just about everybody when it comes to sharing code.

The fact that every Iguana has built in source control does ameliorate the issue a little bit – since at least it’s always possible to roll back to a previous copy of your code and fix what was broken.

I’ve seen larger projects which have involved an onshore team working with an offshore team and they would run a lot of this type of problem.

The solution which everyone comes to eventually is strict conventions about what files you can alter and what you can’t. Since we use stringutils.lua and node.lua it’s probably best not to add to those particular files.

If you examine the structure of the repo you’ll see we’re using a namespacing convention:

These are application specific files for the bed monitor:

https://github.com/interfaceware/iguana-web-apps/tree/master/shared/bedmonitor

and these are application specific files for the Channel Manager

https://github.com/interfaceware/iguana-web-apps/tree/master/shared/cm

We run into the same problem with our wiki in that it’s basically very hard to maintain a lot of code with fragments and not end up with conflicts based on common shared modules.

So what is the solution…

• Naming and code layout conventions
• The flexibility of the project structure in the translator could be improved to make it possible to have modules specific and local to one translator instance and have many shared directories – although this really just shifts the problem – it’s still possible to for users to clobber themselves with any module which is shared between projects
• Automated regression testing.

The last point is the one I have a strong interest in pushing forward. Once you have automated regression testing in place then if you do have a change to a common module that breaks stuff it should show up immediately. Regression testing is a hairy problem since it requires quite tight knit collaboration with our users who are close to the real problems and us as the interface engine vendor. That is is what makes all this kind of thing very interesting:

http://help.interfaceware.com/kb/regression-testing-round-ii-ding-ding

It would give an early warning the moment a check in is done which breaks something.

The other thing is making the import and export process more explicit so it can tell you what files are being imported and how they differ from current local copies. It’s one of the ideas for the evolution of the next couple of iterations of the channel manager is to make it possible to do that. The information is all there, it wouldn’t be a big deal to expose it:

http://help.interfaceware.com/kb/9262

See my comment that start with:

It’s would be nice to know what files are changed with both importing and exporting translator instances. The information is…

Hi Garry,

Eliot and I have been having a bit of back-and-forth on the Channel Manager with regard to issues presented by the ability to name a channel just about anything you’d like. Your file.lua code resolves most of the major issues, but one thing that isn’t is when a path separator (‘/’ or ‘\’) is used as part of the channel name.

My suggestion would be to uri-encode the channel name before creating the xml file and resulting subdirectory in the export path. The downside is that this:

File: -> ASTM TCP/IP

comes out looking like this:

File+-%3E+ASTM+TCP%2FIP.xml

That doesn’t present an issue for me, but if there’s a better way to do it I’d like to hear about it.

Jeff Drumm ◊ VP and COO ◊ HICG, LLC. ◊ http://www.hicgrp.com

One could have pre-process with something like (I haven’t tried so the calls are probably wrong):

Name = Name:gsub(” “, “YGIUYGT2332ihsd34534534”);
Name = filter.uri.enc(Name)
Name = Name:gsub(“YGIUYGT2332ihsd34534534″, ” “);

That would keep spaces as spaces

Or maybe just convert + to ” ” after URI encoding. I guess that would work too…

Eliot Muir wrote:

One could have pre-process with something like (I haven’t tried so the calls are probably wrong):
Name = Name:gsub(” “, “YGIUYGT2332ihsd34534534″);<br> Name = filter.uri.enc(Name)<br> Name = Name:gsub(“YGIUYGT2332ihsd34534534″, ” “);
That would keep spaces as spaces
Or maybe just convert + to ” ” after URI encoding. I guess that would work too…

I’ve always felt that spaces in filenames just made my life more difficult, especially in the command-line world of Linux/Unix. Having to backwhack every space isn’t really that much fun 🙂

As far as converting + to ” ” after encoding is concerned, that doesn’t appear to be an issue when decoding the xml filenames to build the import list in the Channel Manager. I sort of expected it to . . .

Jeff Drumm ◊ VP and COO ◊ HICG, LLC. ◊ http://www.hicgrp.com

Yes… bit of tabbing in the Git bash shell… 🙂

Just in process of refactoring the Javascript in the monitor application to follow the same structure as the bed monitor and the channel manager.

Seems like we could probably document how we use the datatable better – it’s a pretty key component.

Just in process of refactoring the Javascript in the monitor application to follow the same structure as the bed monitor and the channel manager.

Seems like we could probably document how we use the datatable better – it’s a pretty key component.

It’s one of the big key factors in making these apps useful is being to make it accessible to many people to see how the components clip together and keeping the modular, understandable and documented…

After playing with the Channel Manager and working with Git and Bitbucket as remote repositories, I’ve put together some thoughts on things that I’d find useful in the Channel Manager:

1. It would be great to have the destination directory for exports on the Iguana host selectable on export
2. An option to download to the browser’s host as zip file would also be a welcome addition
3. I think I’d prefer that channel exports have all configuration information stored within the individual channel directory, including the xml file currently found in the export path root. This would allow each channel to be more easily treated as its own repository in GitHub/Bitbucket. The other/shared directories present a challenge, though, as they’re common across all channels; it might make sense for each of those to be source controlled as well.
4. It would be handy to have Multiple/all channels selectable (via checkbox) for batch export and import
5. The number of channels shown per page should stick once selected, or be a configurable option

Just some thoughts . . . I may try adding some of these myself.

Jeff Drumm ◊ VP and COO ◊ HICG, LLC. ◊ http://www.hicgrp.com

Yes for the export directory what I’d like to see is the ability to:

• Have a list of destinations
• All those destinations should be configurable – i.e. you shouldn’t have to edit the code of the channel manager to put it in
• It should start to be possible not just to support dumb directories but actually have some binding in there for specific source code control systems

Adding an option to download an exported channel as a zip file would be pretty trivial. lib.webserver would need some tweaking to allow more than just JSON actions since you’d need to have an action which could return a zip. There are already APIs filter.zip which give the ability to make a zip file in memory. The iguana.channel module serves up the data in a Lua table before another routine writes the files to disc – so making it so that it instead invokes a zip operation shouldn’t involve a lot of code.

I think the exporter is a chance to re-address how Iguana’s translator repo should actually be structured. We’re getting by partitioning the ‘shared’ and ‘other’ with a sub-directory which has ‘app’ areas with a separate area per app. You can see how we’re partitioning things here currently:

https://github.com/interfaceware/iguana-web-apps/tree/master/other/app (other dir)

https://github.com/interfaceware/iguana-web-apps/tree/master/shared/cm (shared dir)

I think we haven’t got the convention for the application specific Lua modules in the shared directory quite right.

What we could do in the short term is create a little config file in the projects which could be used to shift files over into we’re we’d like them to be in the GIT repository. One we have it right, then we have a blue print for how to redo the repository structure in Iguana itself.

Multiple selection on channels would definitely be a good idea. Not the first person to say that – we just need to look up the documentation on jQuery datatables to see how to do that – I’m sure it’s trivial thing to expose that functionality and then the iguana.channel library would be workable to do that.

If the import screen was switched over to a data table it could also allow multiple imports which would be cool. To get started on jQuery datatables start here:

http://help.interfaceware.com/kb/the-anatomy-of-an-iguana-app/7

Storing the number of desired channels to show could be serialized in two ways – you can do it browser side using local storage – that makes it dependent on the browser you are using. (it would be a 2-3 line change). Or one could maintain a configuration file on the server/lua side which would keep this information on a per user basis.

Either is quite do-able.

Monday is a public holiday here in Canada – we celebrate our allegiance to Queen Vic
🙂 I think we’ll be catching up on work things – after masters day events there is always a of work todo…

Some of the things I’ve encountered as I’ve started attempting to manage source code control externally . . . in my case using git and bitbucket.org.

• Treating the CM export directory as a “working copy” is problematic when attempting to keep multiple environments in sync, even if the intent is that they be exact replicas. The channel configuration XML files get merge markup due to differing UUIDs, which means they have to be edited before import. I think I remember Eliot mentioning something about this, with a potential solution of modifying CM to export without UUIDs.
• Once you’ve performed a “git pull” to get the updated code to the local export/import directory, you can’t refresh the code for an existing channel with an import; the channel needs to be deleted in the dashboard and re-imported (and because of that the internal Iguana version history is lost, but how many source code control systems do you really need? :)).
• Still struggling with what git needs to make merge resolutions less painful. Some of the tutorials mention that the insertion of comments helps, but I need to better understand the specifics.

The journey continues . . .

Jeff Drumm ◊ VP and COO ◊ HICG, LLC. ◊ http://www.hicgrp.com

Yes I think we should remove the GUIDs (UUIDS) before saving the channel XML fragments – the channel manager checks what it is exporting and only saves files which have changed. We would see a lot less spurious changes in the repo that way.

Generally when I commit to a repo one discipline I try and teach my staff is to check every diff and make sure you really intend to make that change. You can revert a file with:

git checkout

We do need to add the ability to update a channel.

Fossil history isn’t very important when you are using an external repo. I don’t bother with meaningful commit comments when using GIT as the external repo.

Not sure about git merge resolutions. In general with any kind of code the best thing is to write code in a manner which doesn’t create the need for merges. That usually means keeping each module short and resisting the temptation to lump everything and the kitchen sink into a long modules. Less merge conflicts is one of the many benefits of that discipline.

I think we are going to take the bull by the horns and reform the project structure of the translator to add the capacity to have each translator instance have it’s own non shared files.

It will take a couple of passes. The first pass will be to replace the tree control we use currently so that we can have directories represented by nodes which can be collapsed and expanded. Once that is through we’ll add in a new directory structure that is part of the main area.

It be a huge improvement – we’ve tried in the past to overhaul the project manager structure in the translator – we ran into issues with it being too much to bite off all at once. Taking it in pieces makes it easier.

Woohoo!!! 😀

Jeff Drumm ◊ VP and COO ◊ HICG, LLC. ◊ http://www.hicgrp.com

Alrighty!

There is a new fun version of the channel manager at:

http://help.interfaceware.com/kb/channelmanager

This one is easier to use since it has a configuration file which allows you to configure multiple directories for importing/exporting channels. So from here on upgrading the channel manager won’t involve editing it’s source code and you can support exporting to multiple places.

The Add Channel screen has been spruced up a little using a data table that gives descriptions of the channels.

Internally the code makes more use of CSS styling (although with some ugly colours) and more of the presentation logic is in the Javascript side – the back end code for populating the dashboard is lot simpler and the HTML is simpler – more styling logic is now in CSS.

Enjoy the weekend!

Just uploaded a new version. This one has a few little refinements:

1. GUIDs in the XML channel configuration fragments are zeroed out
2. It’s optional to export sample data

That results in a lot less churn in the ‘differences’ which is a nice improvement. The new version will also offer to replace existing channels which is also convenient.

I haven’t patched in the name fix yet – been mainly focused on things things which were blocking getting things prepared for the Californian road trip next week.

One little nice module that came out of the last round of changes:

http://help.interfaceware.com/forums/topic/easier-way-to-create-json-with-json-serialize

Uploaded the next iteration of the channel manager. This supports naming the repository directories and also has the functionality to ask if you want to replace an existing channel.

Just uploading it for the San Diego work shop tomorrow.

So more interesting changes in the pipe line – Kevin is working on this and adding the ability to export and import multiple channels at once. He’s also getting the channel manager to list the files which are being changed and showing differences.

Had a question come up from a customer asking whether it would be possible to support more colors for the dashboard lights for the API call in Iguana that allows Lua scripts to set channel colors – currently only grey, red, yellow and green are supported.

We’re planning a big overhaul of Iguana’s GUI after the user conference (Zack and Art showed me a really cool prototype of what we could do) so it’s something we’ll be working into that dashboard along with the ability to set more custom content.

But if one wanted to tweak the channel manager to provide this then there is an easy path to do so – if Kevin get’s time we might have him do it.

This is how you’d go about doing it. Firstly one would need to store a bit more information in the channel manager application about channel status – or perhaps a complete custom column. Then have a web service call that an Iguana channel can call to set the status – wrap it in a nice handy Lua helper function.

Then go and tweak the back end code that populates the back end dashboard:
https://github.com/interfaceware/iguana-web-apps/blob/master/shared/cm/app/listChannels.lua
then tweak this code:
https://github.com/interfaceware/iguana-web-apps/blob/master/other/app/cm/dashboard.js
to render the extra status information in the data table.

The way to do so would be to modify the output of the div tags you can see there like:

To have some new statuses.

Then it would be about editing the CSS file:
https://github.com/interfaceware/iguana-web-apps/blob/master/other/app/cm/style.css

and adding a rule like some of the existing ones that tell the browser how to style the status icon:

div.chan-on {
width:10px;
height:10px;
border-radius:50px;
background:linear-gradient(to bottom, #a6e182, #54c600);
border: 2px solid #FFFFFF;
margin: 0px auto;
}

If Kevin has the time we might have him go and try this out.

Just wrote this little tutorial about how to reverse engineer Iguana apps, it focuses on the Channel Manager:

http://help.interfaceware.com/kb/tricks-for-pulling-apart-iguana-apps

Thank you to Rick Thiessen who contributed a nice little patch to the basic auth module in Iguana:

https://github.com/interfaceware/iguana-web-apps/commit/0a1ad0a4b5a4f9fe0ff232a5d714ed28fd391ac7

It’s a problem I had noticed with a couple of people but hadn’t thought through a good solution – this looks like a good solution. (my solution was hacking in ‘localhost’ into the code).

I am running into an error just trying to setup the repository/add a channel. It is stating that the dir doesn’t exist and i have tried both on the linux server itself that this Iguana instance is running on and my local PC dir resulting in the same error for each. I have verified that the dir exists, houses the files from Github, and has full rwx permissions. What next?

The repository directory \opt\iNTERFACEWARE-Iguana\iguana-web-apps\ does not exist.

What user id is your Iguana instance running under?

admin

Try out the following code within a blank translator instance on your Linux machine:

for FileName in os.fs.glob('\\opt\\iNTERFACEWARE-Iguana\\iguana-web-apps\\') do
trace(FileName)
end
for FileName in os.fs.glob('/opt/iNTERFACEWARE-Iguana/iguana-web-apps/') do
trace(FileName)
end

This code should give some insight into whether or not the Iguana instance running on your Linux box has permissions to see the directory. I notice you gave the file permission path with backward \ characters – normally Linux file paths are with forward / characters. Give it a try and see what you find from that.

In a blank instance I get the error:
bad argument #1 to ‘?’ (pattern expected)

(I am using the forward / character but the error sends it back with backward \ characters listed)

On the linux server I have made sure the iguana-web-apps folder has full rwx for all users (chmod 777).

Sounds like an issue with file user rights. I think you should raise a standard support ticket with the first goal of showing a simple translator Lua script to read a directory.

Is Channel manager still supported?

The newest Channel manager return an error on iguana ver. 5.6.20

I’m trying to run it, but when I follow instructions from tutorial above (after step 4) Iguana script editor return me an error:

Error in lib.webserver on line 252 (Input data is incomplete.)

Stack Trace:
ServeRequest(lib.webserver:252)
serveRequest(lib.webserver:270)
main (main:14)

Sounds like you need to comment out a line in the top of the script which pulls the Javascript etc. resources from one’s sandbox rather than the packaged code for running the translator. Or maybe you just need to import an HTTP transaction to see what is going on. There isn’t all that much information you have provided.

You’ll need to be confident enough to dig in and use the source to figure out what is happening with it.

With Iguana 6 the channel manager will be superseded. Iguana 6.0 comes with built in tools to import and export channels to external GIT repositories directly and gives much better visibility on what is changed, supports restarted affected channels etc.

Incidentally you didn’t respond to Lev or myself when you emailed the same question…?

One frequent problem that people run into with the channel manager is an error like this:

{
“error”: “…A6EE6EE48DB8701D5DC88695860\\shared\\lib\\webserver.lua:230: attempt to index local ‘Func’ (a nil value)”
}

This is usually caused by putting the channel manager in under one user ID and trying to access it from another.

The problem can be fixed by commenting out the line that says:

test=’admin’

In the main lua module for the channel manager – see:

https://github.com/interfaceware/iguana-web-apps/blob/master/Channel%20Manager_http/main.lua

The channel manager proved to be very popular amongst our users. The good news is that with Iguana 6 the need for this application goes away. Iguana 6 has built functionality that exceeds what the channel manager does in terms of making it clear what is changed with an import etc.

Thanks for that. I missed Lev email, sorry.
I will follow the instructions.