micthaworm micthaworm - 18 days ago 4
JSON Question

Counting how many times a value occurs in a json file

how do i read count how many times each "type" and "key" occurs in a json file with data like below

{"method":"GET","UDID":"26:90:A4:46392970","dataset":"deCarta","production":true,"type":"REVERSEGEOCODE","path":"/v1/04track12netics2015/reverseGeocode/-28.45818,24.39608.xml?returnSpeedLimit=true","key":"04track12netics2015","cost":1,"vendorCost":{"dataProvider":1,"trafficProvider":0},"roundTripTime":5,"nonCompQuery":0,"level":"request","message":"Processing Request","timestamp":"2016-08-12T23:59:52.975Z"}
{"method":"GET","UDID":"26:90:A4:46392915","dataset":"deCarta","production":true,"type":"REVERSEGEOCODE","path":"/v1/04track12netics2015/reverseGeocode/-25.96607,28.14167.xml?returnSpeedLimit=true","key":"04track12netics2015","cost":1,"vendorCost":{"dataProvider":1,"trafficProvider":0},"roundTripTime":96,"nonCompQuery":0,"level":"request","message":"Processing Request","timestamp":"2016-08-12T23:59:53.011Z"}
{"method":"GET","UDID":"26:90:A4:46393163","dataset":"deCarta","production":true,"type":"REVERSEGEOCODE","path":"/v1/04track12netics2015/reverseGeocode/-28.45818,24.39608.xml?returnSpeedLimit=true","key":"04track12netics2015","cost":1,"vendorCost":{"dataProvider":1,"trafficProvider":0},"roundTripTime":7,"nonCompQuery":0,"level":"request","message":"Processing Request","timestamp":"2016-08-12T23:59:53.170Z"}
{"method":"GET","UDID":"26:90:A4:81986956","dataset":"tomtom-world","production":true,"type":"TILE","path":"/v1/fpOgtLY1ZF22m3va4FLkU52tsLmpaNyo/tile/1185/1176/11.png?settings=russian-river&res=mdpi","key":"fpOgtLY1ZF22m3va4FLkU52tsLmpaNyo","cost":1,"vendorCost":{"dataProvider":0.03125,"trafficProvider":0},"roundTripTime":438,"nonCompQuery":0,"level":"request","message":"Processing Request","timestamp":"2016-08-12T06:06:27.394Z"}
{"method":"GET","UDID":"26:90:A4:81987014","dataset":"tomtom-world","production":true,"type":"TILE","path":"/v1/fpOgtLY1ZF22m3va4FLkU52tsLmpaNyo/tile/1185/1177/11.png?settings=russian-river&res=mdpi","key":"fpOgtLY1ZF22m3va4FLkU52tsLmpaNyo","cost":1,"vendorCost":{"dataProvider":0.03125,"trafficProvider":0},"roundTripTime":390,"nonCompQuery":0,"level":"request","message":"Processing Request","timestamp":"2016-08-12T06:06:27.404Z"}
{"method":"GET","UDID":"26:90:A4:81986884","dataset":"tomtom-world","production":true,"type":"TILE","path":"/v1/fpOgtLY1ZF22m3va4FLkU52tsLmpaNyo/tile/1183/1175/11.png?settings=russian-river&res=mdpi","key":"fpOgtLY1ZF22m3va4FLkU52tsLmpaNyo","cost":1,"vendorCost":{"dataProvider":0.03125,"trafficProvider":0},"roundTripTime":533,"nonCompQuery":0,"level":"request","message":"Processing Request","timestamp":"2016-08-12T06:06:27.417Z"}
{"method":"GET","UDID":"26:90:A4:41422853","dataset":"TomTom","production":true,"type":"SEARCH","path":"/v1/fpOgtLY1ZF22m3va4FLkU52tsLmpaNyo/search/0A%20Schoonder%20Stree,.json?countrySet=ZA","key":"fpOgtLY1ZF22m3va4FLkU52tsLmpaNyo","cost":1,"vendorCost":{"dataProvider":1,"trafficProvider":0},"roundTripTime":824,"nonCompQuery":0,"level":"request","message":"Processing Request","timestamp":"2016-08-12T22:37:03.677Z"}
{"method":"GET","UDID":"26:90:A4:88284444","dataset":"deCarta","production":true,"type":"REVERSEGEOCODE","path":"/v1/fpOgtLY1ZF22m3va4FLkU52tsLmpaNyo/reverseGeocode/-33.92456,18.54727.json","key":"fpOgtLY1ZF22m3va4FLkU52tsLmpaNyo","cost":1,"vendorCost":{"dataProvider":1,"trafficProvider":0},"roundTripTime":170,"nonCompQuery":0,"level":"request","message":"Processing Request","timestamp":"2016-08-12T07:51:24.615Z"}
{"method":"GET","UDID":"26:90:A4:10992447","dataset":"tomtom-world","production":true,"type":"TILE","path":"/v1/CxIQlYBhwykcIxtYwrrbltCDiJ4xUxfN/tile/18063/19670/15.png?transparent=true&settings=russian-river&res=hdpi","key":"CxIQlYBhwykcIxtYwrrbltCDiJ4xUxfN","cost":1,"vendorCost":{"dataProvider":0.03125,"trafficProvider":0},"roundTripTime":407,"nonCompQuery":0,"level":"request","message":"Processing Request","timestamp":"2016-08-21T20:23:12.854Z"}
{"method":"GET","UDID":"26:90:A4:10992826","dataset":"tomtom-world","production":true,"type":"TILE","path":"/v1/CxIQlYBhwykcIxtYwrrbltCDiJ4xUxfN/tile/18060/19670/15.png?transparent=true&settings=russian-river&res=hdpi","key":"CxIQlYBhwykcIxtYwrrbltCDiJ4xUxfN","cost":1,"vendorCost":{"dataProvider":0.03125,"trafficProvider":0},"roundTripTime":219,"nonCompQuery":0,"level":"request","message":"Processing Request","timestamp":"2016-08-21T20:23:13.046Z"}
{"method":"GET","UDID":"26:90:A4:46392970","dataset":"deCarta","production":true,"type":"REVERSEGEOCODE","path":"/v1/04track12netics2015/reverseGeocode/-28.45818,24.39608.xml?returnSpeedLimit=true","key":"04track12netics2015","cost":1,"vendorCost":{"dataProvider":1,"trafficProvider":0},"roundTripTime":5,"nonCompQuery":0,"level":"request","message":"Processing Request","timestamp":"2016-08-12T23:59:52.975Z"}
{"method":"GET","UDID":"26:90:A4:46392915","dataset":"deCarta","production":true,"type":"REVERSEGEOCODE","path":"/v1/04track12netics2015/reverseGeocode/-25.96607,28.14167.xml?returnSpeedLimit=true","key":"04track12netics2015","cost":1,"vendorCost":{"dataProvider":1,"trafficProvider":0},"roundTripTime":96,"nonCompQuery":0,"level":"request","message":"Processing Request","timestamp":"2016-08-12T23:59:53.011Z"}
{"method":"GET","UDID":"26:90:A4:81986949","dataset":"tomtom-world","production":true,"type":"TILE","path":"/v1/fpOgtLY1ZF22m3va4FLkU52tsLmpaNyo/tile/1182/1175/11.png?settings=russian-river&res=mdpi","key":"fpOgtLY1ZF22m3va4FLkU52tsLmpaNyo","cost":1,"vendorCost":{"dataProvider":0.03125,"trafficProvider":0},"roundTripTime":472,"nonCompQuery":0,"level":"request","message":"Processing Request","timestamp":"2016-08-12T06:06:27.421Z"}
{"method":"GET","UDID":"26:90:A4:10992826","dataset":"tomtom-world","production":true,"type":"TILE","path":"/v1/CxIQlYBhwykcIxtYwrrbltCDiJ4xUxfN/tile/18060/19670/15.png?transparent=true&settings=russian-river&res=hdpi","key":"CxIQlYBhwykcIxtYwrrbltCDiJ4xUxfN","cost":1,"vendorCost":{"dataProvider":0.03125,"trafficProvider":0},"roundTripTime":219,"nonCompQuery":0,"level":"request","message":"Processing Request","timestamp":"2016-08-21T20:23:13.046Z"}

Answer

I'm not sure if you want separate counts for types and keys, or if you want the counts for each combination, so this does both:

from collections import Counter
import json

with open('logs.txt') as infile:
    data = [json.loads(line) for line in infile]

for keys in [['type'], ['key'], ['type', 'key']]:
    print(Counter(tuple(row[key] for key in keys) for row in data))

Output:

Counter({(u'TILE',): 7, (u'REVERSEGEOCODE',): 6, (u'SEARCH',): 1})
Counter({(u'fpOgtLY1ZF22m3va4FLkU52tsLmpaNyo',): 6, (u'04track12netics2015',): 5, (u'CxIQlYBhwykcIxtYwrrbltCDiJ4xUxfN',): 3})
Counter({(u'REVERSEGEOCODE', u'04track12netics2015'): 5, (u'TILE', u'fpOgtLY1ZF22m3va4FLkU52tsLmpaNyo'): 4, (u'TILE', u'CxIQlYBhwykcIxtYwrrbltCDiJ4xUxfN'): 3, (u'SEARCH', u'fpOgtLY1ZF22m3va4FLkU52tsLmpaNyo'): 1, (u'REVERSEGEOCODE', u'fpOgtLY1ZF22m3va4FLkU52tsLmpaNyo'): 1})
Comments