reedb89 reedb89 - 1 year ago 217
Java Question

Spring data Sort Operation Exceeds Max Size

I am pretty new Spring and Mongodb and am having an issue with pulling data from my MongoDB. I am trying to get a fairly large amount of data and am receiving the following exception:

...
Query query = new Query();
query.with(new Sort(Sort.Direction.DESC, "vin"));
Criteria c = new Criteria().andOperator(Criteria.where("updateTime").gt(startDate),
Criteria.where("updateTime").lte(endDate));

query.addCriteria(c);

return this.mongoOperations.find(query, VehicleStatus.class);



  • Executor error: OperationFailed: Sort operation used more than the maximum 33554432 bytes of RAM. Add an index, or specify a smaller limit.; nested exception is com.mongodb.MongoException: Executor error: OperationFailed: Sort operation used more than the maximum 33554432 bytes of RAM. Add an index, or specify a smaller limit.



After reading further it seemed as if using an aggregation was a good idea because i could allow disk usage for temporary storage. I thought allowing disk space would fix this issue, perhaps i am using it incorrectly? However i am now receiving the following error: aggregation result exceeds maximum document size (16MB)" , "code" : 16389

...
MatchOperation matchStage = Aggregation.match(new Criteria().andOperator(Criteria.where("updateTime").gt(startDate),
Criteria.where("updateTime").lte(endDate)));
SortOperation sort = Aggregation.sort(Direction.DESC, "vin");

Aggregation agg = Aggregation.newAggregation(VehicleStatus.class, matchStage, sort)
.withOptions(Aggregation.newAggregationOptions().allowDiskUse(true).build());

List<VehicleStatus> vs = this.mongoOperations.aggregate(agg, "vehicleStatus", VehicleStatus.class).getMappedResults();
return vs;


I really am not sure how to handle this. Both queries work if they are within the given size limit. If anyone has any tips or advice it would be greatly appreciated. If you have any questions or need any additional code let me know!

Thanky you

Answer Source

The result of the aggregate pipeline exceeds 16MB and since the maximum allowable size of a MongoDB document is 16MB, this exception is thrown.

So, you have two options:

  • Reduce the size of your output, one possibility here might be to use the Spring Data MongoDB’s limit() method to limit the size of the output
  • Use MongoDB’s $out operator to pipe the aggregation results to a collection

You can adapt your code to use $out via Spring’s Mongo DB’s facade like this (note: I haven’t verified this code but the important point is that you must provide an AggregationOperation like the one I have coded below):

Aggregation agg = Aggregation.newAggregation(VehicleStatus.class, matchStage, sort, new    AggregationOperation() {
  @Override
  public DBObject toDBObject(AggregationOperationContext context) {
    return new BasicDBObject("$out", “outputCollection”);
  }
}).withOptions(Aggregation.newAggregationOptions().allowDiskUse(true).build());
Recommended from our users: Dynamic Network Monitoring from WhatsUp Gold from IPSwitch. Free Download