ethrbunny ethrbunny - 5 months ago 64
Java Question

kafka ktable - rocksdb access via java

Ive been reading about ktables this morning in hopes of implementing a rolling-window key-value store. I can see where the latest release of kafka seems to hint that this is possible but I'm more wondering about accessing the key-value data from 'outside' applications.

Say I implement a kstreams application and it's consuming log data (or it's ilk) from a topic and merrily windowing, aggregating and producing to another topic. Now I want to look at this key-value data from some other process. The docs hint that the data is stored by rocksdb. Can I read this from 'outside' as a call to said db? Or is this data only available as a virtual construct to kstreams applications?

Answer

RocksDBStore shows how to access the RocksDB store from Java. Basically:

RocksDB db = RocksDB.open(options, dir.getAbsolutePath());

and then you can get() and put() and remove(). You can configure where KafkaStreams stores the RocksDB state, so that's basically all you need to know to work with the store outside of the KafkaStreams.

With regard to windows—depending on how your windowing is configured, multiple RocksDBStore called Segments are created are each window segment. You can access each of these stores as mentioned above.

Comments