James111 James111 - 2 months ago 21
Java Question

How can I count the number of times a field occurred with spring elasticsearch?

I'm running a cronjob that runs every X minutes to find out what the most popular tags are (hashtags). I'm storing user activities in Elasticsearch with Fluentd, so say a user searches for a term, that would be stored in elasticsearch like so:

{ activity: "search", user: X, searchTerm: "XYZ"}


I've got one an activity for tags. Each time a user searches for a tag, it gets inserted as an activity. This way I can find out what tags a user likes as well as find out the most popular tags overall.

Here is the structure of my ES:

{
"_index":"user_activity",
"_type":"user_activity",
"_id":"AVcokbsXKR86Bn8FzoFU",
"_score":1.0,
"_source":{
"user":{
"userId":4,
"name":"Another one",
},
"activity":"CREATE",
"date":1473854418419,
"article":{
{user who posted article...}
... Article data (title, etc...)
// Tags associated to this article
"tags":[
{
"tagId":23,
"tagName":"randomTagName"
}
],
},
}
},
{
"_index":"user_activity",
"_type":"user_activity",
"_id":"AVcomLEnKR86Bn8FzoFu",
"_score":1.0,
"_source":{
"user":{
"userId":1,
"name":"MEATTTT DAMOENNNN",
},
"activity":"TAG_SEARCH",
"date":1473854873951,
"tag":"photos"
}
}
... There can be different types of activities


As you can see there are different activity types. I'm trying to query ES and have it return the total amount of searches for each unique tag. If you look at the TAG_SEARCH object, you can see it has a field tag. I'm trying to find the total amount of TAG_SEARCH activities for the unique tag value!

Here is what I've tried using the
NativeSearchQueryBuilder
.

SearchQuery searchQuery = new NativeSearchQueryBuilder()
.withQuery(matchAllQuery())
.withSearchType(COUNT)
.withIndices("user_activity").withTypes("user_activity")
.addAggregation(terms("activity").field("activity"))


What this does is return the total amount of documents for each unique activity value. So the result for the above JSON would be:

"buckets" : [ {
"key" : "create",
"doc_count" : 1
}, {
"key" : "tag_search",
"doc_count" : 1
}
]


But what I'm trying to get is:

"buckets" : [ {
"key" : "tag",
"value": "TagNameGoesHere",
"doc_count" : 4
}, {
"key" : "tag",
"value": "AnotherTagNameGoesHere",
"doc_count" : 10
}
]


How can I find out the total number of unique tag: X values where the activity: TAG_SEARCH?

I'd expect the SearchQuery to look something like this:

SearchQuery searchQuery = new NativeSearchQueryBuilder()
.withQuery(matchAllQuery())
.withSearchType(COUNT)
.withIndices("user_activity").withTypes("user_activity")
.addAggregation(terms("activity").value("TAG_SEARCH"))
.addAggregation(terms("tag")) // Count no docs per tag

Val Val
Answer

You simply need a query on the activity field and then an aggregation on the tag field.

SearchQuery searchQuery = new NativeSearchQueryBuilder()
        .withQuery(QueryBuilders.matchQuery("activity", "TAG_SEARCH"))
        .withSearchType(COUNT)
        .withIndices("user_activity")
        .withTypes("user_activity")
        .addAggregation(AggregationBuilders.terms("tags").field("tag"));

The corresponding raw JSON query would be this one

POST user_activity/user_activity/_search?search_type=count
{
  "query": {
    "match": {
      "activity": "TAG_SEARCH"
    }
  },
  "aggs": {
    "tags": {
      "terms": {
        "field": "tag"
      }
    }
  }
}