Huanian Zhang Huanian Zhang - 2 days ago 4
Python Question

Numpy savetxt do not handle 18 ditgits long int correctly

I need to save some long int to text file. I use Numpy savetxt to do it as below:

np.savetxt('test.txt', np.c_[objID,z,e,PA,n],fmt='%i %1.4f %1.4f %1.4f %1.4f')


I tried either
%i
or
%f
for
objID
, but the item in the
text.txt
is not correct. I list a few
objID
from the source as below:

587722952230174996
587722952230175035
587722952230175138


However, the same item in the
test.txt
becomes:

587722952230174976
587722952230174976
587722952230175104


Anyone knows how to fix that? Thanks.

Answer

I don't have numpy installed in Py2, but I can recreate your numbers with plain Python operations:

>>> ll=[587722952230174996,587722952230175035,587722952230175138]
>>> ll
[587722952230174996L, 587722952230175035L, 587722952230175138L]
>>> '%.0f, %.0f, %.0f'%tuple(ll)
'587722952230174976, 587722952230174976, 587722952230175104'
>>> '%i, %i, %i'%tuple(ll)
'587722952230174996, 587722952230175035, 587722952230175138'

The problem display is consistent with rounding to float, but %i formatting handles them without problem.

With numpy on py3

In [243]: '%i, %i, %i'%tuple(np.array(ll))
Out[243]: '587722952230174996, 587722952230175035, 587722952230175138'

In [245]: '%i, %i, %i'%tuple(np.array(ll, np.float))
Out[245]: '587722952230174976, 587722952230174976, 587722952230175104'

np.array(ll) has dtype int64.

==================

My guess is that:

`np.c_[objID,z,e,PA,n]`

is concatenating the values, and creating a float64 array. The other numbers appear to be floats, so even if objID is long, the whole thing is float. So the %i format still sees the float 'rounding'.

What's the shape and type of objID',z', etc? You may have to join them in a structured array if you want to keep the full long value.

In [249]: z=[.1,.2,.3]
In [250]: np.c_[ll,z]
Out[250]: 
array([[  5.87722952e+17,   1.00000000e-01],
       [  5.87722952e+17,   2.00000000e-01],
       [  5.87722952e+17,   3.00000000e-01]])

In [255]: np.savetxt('test.csv', data,fmt='%i, %f')
In [256]: cat test.csv
587722952230174976, 0.100000
587722952230174976, 0.200000
587722952230175104, 0.300000

Taking the structured array route:

In [274]: ds = np.empty((3,),dtype=[('f0',np.int64),('f1',np.float64)])
In [275]: ds['f0']=ll
In [276]: ds['f1']=z
In [277]: ds
Out[277]: 
array([(587722952230174996, 0.1), (587722952230175035, 0.2),
       (587722952230175138, 0.3)], 
      dtype=[('f0', '<i8'), ('f1', '<f8')])
In [278]: np.savetxt('test.csv', ds,fmt='%i,   %f')
In [279]: cat test.csv
587722952230174996,   0.100000
587722952230175035,   0.200000
587722952230175138,   0.300000
Comments