I have a pickle file which upon unpickling throws an
UnpicklingError: invalid load key, 'A'.
with open('hist_vs_years2.pkl', 'ab') as hist_pkl:
pickle.dump(hist, hist_pkl, -1)
hist_vs_samples2 = 
more_values = True
with open('hist_vs_years2.pkl', 'rb') as hist_vs_samples_pkl:
while more_values == True:
more_values = False
File "C:\Anaconda2\lib\pickle.py", line 1384, in load
File "C:\Anaconda2\lib\pickle.py", line 864, in load
When storing multiple objects (by repeated
dump, not from containers) Pickle will store objects sequentially in pickle files, so if an object is broken it can be removed without corrupting the others.
In principle, the pickle format is pseudo-documented in
pickle.py. For most cases, the opcodes at the beginning of the module are sufficient to piece together what is happening. Basically, pickle files are an instruction on how to build objects.
How readable a pickle file is depends on its pickle format - 0 is doable, everything above is difficult. Whether you can fix or must delete depends entirely on this. What's consistent is that each individual pickle ends with a dot (
.). For example,
b'\x80\x04\x95\x05\x00\x00\x00\x00\x00\x00\x00\x8c\x01a\x94.' both are the character '"a"', but in protocol 0 and 4.
The simplest form of recovery is to count the number of objects you can load:
with open('/my/pickle.pkl', 'rb') as pkl_source: idx = 1 while True: pickle.load(pkl_source) print(idx) idx += 1
Then open the pickle file, skip as many objects and remove everything up to the next