user1610950 user1610950 - 2 months ago 12
Python Question

Pausing a long running program

I have a question about pausing / resuming a long running program. I'll use python in this example but it could be for any programming language really.

Let's say I want to sum up all the numbers to a billion for example

results = []

for i in range(1000000000):
results.append(i + i*2)

with open('./sum.txt', 'w') as outfile:
for r in results:
output.write(r+'\n')
outfile.close()


that's a very simple program and in the end I would like to write the values out to a file.

Let's say I start this script and it's taking hours and I want to shut down my computer. Now I could wait but what are some ways that I could stop this process while keeping the data in tact?

I've though about opening the file before the loop and inside the loop, instead of appending to the results list, I could just write [append] to the opened file but that seems like still if I quit the program, the file never closes and the data could get corrupted. Even if I just write to the file in the loop then directly close the file after the write operation, couldn't that still cause data corruption or something?

What are some ways that I could tackle a problem like this. Where I'd like to be able to stop a program and later, restart the program and it could continue from where it left off?

[edit]
I don't see why all the requests to close this question or down votes.

If you have a long running script that does most of the work on the inner part of a loop.


@MisterMiyagi
Can you please specify what your actual question is?
Creating safe-points inside the loop is the way to go if you want to
preserve the loop's state. Whether file I/O is safe against corruption
doesn't depend on your application so much as operating/file system.
In Linux, it's safe to write to a temporary file, then rename it;
files are also closed automatically when a program exits.


I was more interested in a safe place inside the loop. What are some recommended ways to do that in a program that's structured like the one above.

Answer

Create a condition for the loop to either continue or hold

A possible solution would be in the script, periodically checking for a "trigger" file to exist. In the example below, the loop checks once in hundred cycles, but you could make it 500 for example.

If and as long as the file exists, the loop will hold, checking again every two seconds.
While this works fine, you will have to accept the fact that it slightly slows down the loop, depending on n; larger n will decrease the effect. ANY solution inside the loop will however have at least some effect on the loop.

Although this is on python, the concept should be possible in any language, and on any os.

The print functions are only to show that it works of course:)

The edited loop:

import os
import time

results = []
# check every n- cycles for the triggere file to exist
n = 100

for i in range(1000000000):
    if i%n == 0:
        # checking for the trigger file to exist
        while os.path.exists("test123"):
            time.sleep(2)
            print("wait")   
        else:
            print("go on")
    else:
         results.append(i + i*2)
    print(i)

with open('./sum.txt', 'w') as outfile:
    for r in results:
        output.write(r+'\n')
outfile.close()

The result, if I create the trigger file and remove it again:

4096
4097
4098
4099
wait
wait
wait
wait
wait
wait
go on
4100
4101
4102