I have the following Python code where I collect data from standard input into a list and run syntaxnet on it. The data is in the form of json objects from which I will extract the text field and feed it to syntaxnet.
data = 
for line in sys.stdin:
run_syntaxnet(data) ##This is a function##
I would gather the data into chunks and process those chunks when they get "large":
LARGE_DATA = 10 data =  for line in sys.stdin: data.append(line) if len(data) > LARGE_DATA: run_syntaxnet(data) data =  run_syntaxnet(data)