I'm using mysql connector of python3 to query from a mysql database. I have so many data that if I collect the recent 7 days' data, the size of the query will be over 10GB and therefore forced my python script to be killed. I think usually we can stream the result however I don't find a way to stream the query result in mysql conncetor. Is there any other way that I can solve the problem?
@Anthony Kong's comment is correct. To solve this issue, we can do
fetchmany function from https://dev.mysql.com/doc/connector-python/en/connector-python-api-mysqlcursor-fetchmany.html
After digging into the code a little bit, I found that the
fetchmany function calls many
fetchone to achieve "fetchmany". So I decide to use fetchone at last. Also fetchone comes with an example on the document https://dev.mysql.com/doc/connector-python/en/connector-python-api-mysqlcursor-fetchone.html