George George - 1 month ago 18
Python Question

arbitrary gql filters and sorts without huge index.yaml

I've written a tiny app on Google App Engine that lets users upload files which have about 10 or so string and numeric fields associated with them. I store the files and these associated fields in an ndb model. I then allow users to filter and sort through these files, using arbitrary fields for sorting and arbitrary fields or collections of fields for filtering. However, whenever I run a sort/filter combination on my app that I didn't run on the dev_appserver before uploading, I get a NeedIndexError along with a suggested index, which seems to be unique for every combination of sort and filter fields. I tried running through every combination of sort/filter field on the appserver, generating a large index.yaml file, but at some point the app stopped loading altogether (I wasn't monitoring whether this was a gradual slowdown or a sudden breaking).

My questions are as follows. Is this typical behavior for the GAE datastore, and if not what parts of my code would be relevant for troubleshooting this? If this is typical behavior, is there an alternative to the datastore on GAE that would let me do what I want?

Answer

It seems like Google Cloud SQL would do what I need, but since I'm trying not to spend any money on this project and GCS doesn't have a free unlimited tier, I've resorted to querying by my filter and then sorting the results myself.