Bhishan Poudel Bhishan Poudel - 5 months ago 10
Python Question

Better way to "copy only the comments from one file" and "prepend it into another file" using python

Basically I want to copy comments from one file and add it to the another data.

The file

'data_with_comments.txt'
can be obtained from pastebin:
http://pastebin.com/Tixij2yG

And it looks like this:

# coating file for detector A/R
# column 1 is the angle of incidence (degrees)
# column 2 is the wavelength (microns)
# column 3 is the transmission probability
# column 4 is the reflection probability
14.2000 0.300000 8.00000e-05 0.999920
14.2000 0.301000 4.00000e-05 0.999960
14.2000 0.302000 2.00000e-05 0.999980
14.2000 0.303000 2.00000e-05 0.999980
14.2000 0.304000 2.00000e-05 0.999980
14.2000 0.305000 3.00000e-05 0.999970
14.2000 0.306000 5.00000e-05 0.999950


Now, i have another datafile
'test.txt'
which looks like this:

300.0 1.53345164121e-32
300.1 1.53345164121e-32
300.2 1.53345164121e-32
300.3 1.53345164121e-32
300.4 1.53345164121e-32
300.5 1.53345164121e-32


Required output:

# coating file for detector A/R
# column 1 is the angle of incidence (degrees)
# column 2 is the wavelength (microns)
# column 3 is the transmission probability
# column 4 is the reflection probability
300.0 1.53345164121e-32
300.1 1.53345164121e-32
300.2 1.53345164121e-32
300.3 1.53345164121e-32
300.4 1.53345164121e-32


One way to do this is:

#!/usr/bin/env python
# -*- coding: utf-8 -*-
# Author : Bhishan Poudel
# Date : Jun 18, 2016


# Imports
from __future__ import print_function
import fileinput


# read in comments from the file
infile = 'data_with_comments.txt'
comments = []
with open(infile, 'r') as fi:
for line in fi.readlines():
if line.startswith('#'):
comments.append(line)

# reverse the list
comments = comments[::-1]
print(comments[0])
#==============================================================================


# preprepend a list to a file
filename = 'test.txt'

for i in range(len(comments)):
with file(filename, 'r') as original: data = original.read()
with file(filename, 'w') as modified: modified.write(comments[i] + data)


In this method we have to open the file many times and it is not efficient when the data file is very large.

Is there any better way of doing this?

Related links are following:

Appending a list to the top of Pandas DataFrame output

Prepend line to beginning of a file

Python f.write() at beginning of file?

How can I add a new line of text at top of a file?

Prepend a line to an existing file in Python

Answer

You already have a great answer using a temporary directory but it is also common to just create a temporary file in the same directory as the target file. On systems where tmp is a separate mount point, you avoid an additional copy of the data when renaming the temporary file. Notice that there is no intermediate list of comments which is significant if the comment list is large.

import os
import shutil

infile = 'data_with_comments.txt'
filename = 'test.txt'

tmpfile = filename + '.tmp'

try:
    # write wanted data to tempfile
    with open(tmpfile, 'w') as out_fp:
        # prepend comments from infle
        with open(infile) as in_fp:
            out_fp.writelines(filter(lambda l: l.startswith('#'), in_fp))
        # then add filename
        with open(filename) as in2_fp:
            shutil.copyfileobj(in2_fp, out_fp)
    # get rid of original data
    os.remove(filename)
    # replace with new data
    os.rename(tmpfile, filename)
finally:
    # cleanup on error
    if os.path.exists(tmpfile):
        os.remove(tmpfile)
Comments