If I've got an iterable containing strings, is there a simple way to turn it into a stream? I want to do something like this:
output = tarfile.TarFile(…)
stream = iterable_to_stream(make_file())
Here's my streaming iterator an experimental branch of urllib3 supporting streaming chunked request via iterables:
class IterStreamer(object): """ File-like streaming iterator. """ def __init__(self, generator): self.generator = generator self.iterator = iter(generator) self.leftover = '' def __len__(self): return self.generator.__len__() def __iter__(self): return self.iterator def next(self): return self.iterator.next() def read(self, size): data = self.leftover count = len(self.leftover) if count < size: try: while count < size: chunk = self.next() data += chunk count += len(chunk) except StopIteration: pass self.leftover = data[size:] return data[:size]
Source with context: https://github.com/shazow/urllib3/blob/filepost-stream/urllib3/filepost.py#L23
Related unit tests: https://github.com/shazow/urllib3/blob/filepost-stream/test/test_filepost.py#L9
Alas this code hasn't made it into the stable branch yet as sizeless chunked requests are poorly supported, but it should be a good foundation for what you're trying to do. See the source link for examples showing how it can be used.