checking if a file is completely written

Hi,

Is there any way I can check if a file is stable to read/pick/move to another location. Or how to prevent Iguana/translator code to not to pick the file if it is being written by some other application at the same time. I know we can rename or move to temp location before reading it but even from getting the file from temp location we need to check if that file is stable for not.

Thank you in advance

One simple and sure method, to avoid numerous applications to use same file simultaneously, is to create a lock file. Then check if lock exists before attempting to operate on file. Having said so, this method adds computing cycles and is forcing sequential processing.

Hi Lev,

I don’t have any control on incoming/source file so even to change its extension to .lock, how can i check the file is stable?

Lua doesn’t support kernel-enforced file locks, so that’s out.

If the file supplier can write the file to a temporary name, then rename it to its final name when it’s completely written, you can be guaranteed of having the complete file.

Otherwise, your only real option is to develop a mechanism for tracking the state of the file between invocations of the translator. If the file’s metadata returned by os.fs.stat() does not change between translator invocations (and the translator poll time is long enough), you will have some level of confidence that the file is stable and completely written.

The process would be something like this:

1. Check to see if a file is present
2. Query the SQLite database to determine if the file and associated metadata is already present
3. If the file is in the database, compare the current metadata to the recorded metadata
4. If the metadata has changed, update the record with the new metadata
5. If the metadata has not changed, the file is very likely complete and can be processed
6. If the file is not present in the table, create a new record with the current metadata and return so that it can be checked at the next poll interval
7. Once the file is processed, delete the associated record from the tracking table.

Jeff Drumm ◊ VP and COO ◊ HICG, LLC. ◊ http://www.hicgrp.com

Because this forum software constantly eats my posts when I go back to edit them, I’m forced to make a couple of corrections to my previous post as a comment rather than fixing the post itself.

The original post is not the one you see above. That one was apparently sidelined to a moderation queue or simply deleted after I attempted to edit it.

The solution above presumes that you would create a mechanism to track the file’s state across invocations of the translator. My suggestion is a SQLite database, included with and well-supported by Iguana. I neglected to mention that you would need to create that database/table as a first step.

The last step should also have indicated that the processed file is deleted, moved or renamed so that it does not remain as a candidate for processing after its associated record is deleted from the SQLite table.

Jeff Drumm ◊ VP and COO ◊ HICG, LLC. ◊ http://www.hicgrp.com

Hi Jeff,

Thank you so much for providing a detailed solution for this requirement. I was also thinking for another approach where I will check file size in two-interval and if the size is same, I would consider that file as a stable file.

Thank you so much once again.

As always, you are awesome.

There are some scenarios where the file’s reported size is not a guarantee of completion/stability; there are some file operations that pre-allocate the storage space and then write the file contents.

Using multiple criteria as provided in os.fs.stat() may be a better option.

Jeff Drumm ◊ VP and COO ◊ HICG, LLC. ◊ http://www.hicgrp.com

Yes, you are right. I should use os.fs.stat() .

Thank you, Jeff.