Rob Rob - 17 days ago 5
Python Question

how to speed up mongoengine queries

If I have two objects like:

class User(Document):
name = StringField()
following = ListField(ReferenceField('User'))
meta = {
'indexes': [
'following',
]
}

class Media(Document):
owner = ReferenceField('User')
url = StringField()
is_hidden = BooleanField()
posted_date = Date

meta = {
'indexes': [
'owner',
'posted_date',
'is_hidden',
]
}


and when I want to look up the following condition where it is not hidden and its owner is someone I am following and it was recently posted, I have a query like this:

user = User.objects.first()
Media.objects(Q(owner__in=user.following) &
Q(is_hidden=False) &
Q(posted_date__gte=dt.now()-dt.timedelta(days=3))


which is not scaling and is becoming much slower. What can I do to speed up the performance of these types of complex queries?

Answer

1) Use User.objects.get(id=user_id) instead of first(). Tough I am not really sure if that will make a difference, I think yes and that is a find operation where MongoDB will return a cursor and mongoengine iterates to the first document. Instead, get() is doing a findOne and returns only 1 document. If I mistake, someone should correct me please.

2) Use a compound index (not multiple single index) since your query is using multiple fields (example here also):

 meta = {
            'indexes': [
                ('owner', 'posted_date', 'is_hidden',)
            ]
        }

3) Limit returning data to only the fields you need, project your fields, using only()

Use explain() on your query to styudy it and improve it in oder to reach a covered query.

Comments