charrold303 charrold303 - 2 months ago 8
Javascript Question

Python file.close() and with() behavior in high frequency loops

I am working with a Raspberry PI and outputting a value from a sound sensor using a python script. In order to display this, I use an HTML page on my PI that calls a javascript include which is simply a single line that defines a value that will be used to change what the Google gauge displays. All of this is pretty simple and straightforward, and works just fine for my needs. (Added bonus that I actually understand how it works.)

The .js file is written out by my python script which does a bunch of other stuff too. This all actually works exactly as I want it to, and the values that get written to the file are accurate and do appear correctly on the gauge if I run it as a single loop and not in a continuous "monitoring service" mode where it runs until interrupted.

Here's where the issue starts I think: the loop that measures sound levels runs ~ 30k times in 5 seconds. In my python code I use:

with open(web_file, 'w') as f_output:
f_output.write("var int_level = " + str(per_detected))
f_output.close()


I am doing this because I THINK I need to do the close each time because of the loop (I know the "with" is an implicit close, but because I reopen the file every time it seemed better to close it by force to be sure), BUT I think it also may be the issue. (also I am using mode 'w' on purpose so I can reset the file each time as it is only a single line - seemed faster and less computationally expensive than replacing a value)

The symptom is that the Google gauge HTML page refreshes once every 5 seconds to load the new value, BUT the gauge itself only renders maybe once every 10 refreshes, but this is totally random, obviously indicating that there is no value in the .js file to be returned. I take this to mean it is probably getting lucky once in a few tries and hitting the file before the close, OR is it just too fast and this is a stupid way to do it?

As an additional note, when I cat level.js (<- the include file) from the command line on the PI, after "killing" my python code, it is empty.

I have all the code (including the HTML page) at:

https://www.GitHub.com/ChrisHarrold/Pi-Projects

if you wish to review for greater detail. It is commented 6-ways till Sunday so it should be trivial to figure out what I did and why.

Answer

The .close() is definitely not needed. At issue is your browser reading the file while it is still open anyway and finding it in a truncated (so empty) state from time to time. And you can never close the file fast enough to prevent this.

What you should do instead is write the file to a different filename, then rename the file to replace the old file. This is an atomic operation (meaning the Operating System guarantees that any other access to the file will either get the old, or the new file, but never just a part of both).

This is as simple as:

with open(web_file + '.new', 'w') as f_output:
    f_output.write("var int_level = " + str(per_detected))

os.rename(web_file + '.new', web_file)

The above code writes out to a different filename, one ending in .new, and only when writing has completed (and the file is closed), is it moved into place to replace the old version of the file.