I'm using the Python
request does take an iterator or generator as
data argument, the details are described in Chunk-Encoded Requests. The transfer encoding needs to be chunked in this case because the data size is not known beforehand.
Here is a very simle example that uses a
queue.Queue and can be used as a file-like object for writing:
import requests import queue import threading class WriteableQueue(queue.Queue): def write(self, data): # An empty string would be interpreted as EOF by the receiving server if data: self.put(data) def __iter__(self): return iter(self.get, None) def close(self): self.put(None) # quesize can be limited in case producing is faster then streaming q = WriteableQueue(100) def post_request(iterable): r = requests.post("http://httpbin.org/post", data=iterable) print(r.text) threading.Thread(target=post_request, args=(q,)).start() # pass the queue to the serializer that writes to it ... q.write(b'1...') q.write(b'2...') # closing ends the request q.close()