Amnor Amnor - 1 month ago 13
Java Question

How can I connect Storm and D3.js using Redis and Flask?

I have my Storm testing topology done, and before I created a d3 script on an Html code, that readed the data from a text file. I want it now to read the data directly from a Storm topology (a bolt maybe?) But I have no clue of how to do it. I'm using Horton Works Sandbox for the testing, Any help would be apprecieated.

Thanks in advance!

I've found a storm package for redis that I'm trying to use now. It allows you to set a bolt for writting on redis, and I've set the node already. My problem now is that eclipse can't find the imports of the java code and the ones on the pom.xml.I've downloaded the package. My current java bolt and imports are:

package Storm.practice.Storm.Prova;
import backtype.storm.Config;
import backtype.storm.LocalCluster;
import backtype.storm.StormSubmitter;
import backtype.storm.task.OutputCollector;
import backtype.storm.task.TopologyContext;
import backtype.storm.testing.TestWordSpout;
import backtype.storm.topology.OutputFieldsDeclarer;
import backtype.storm.topology.TopologyBuilder;
import backtype.storm.topology.base.BaseRichBolt;
import backtype.storm.tuple.Fields;
import backtype.storm.tuple.Tuple;
import backtype.storm.tuple.Values;
import backtype.storm.utils.Utils;
import backtype.storm.spout.SpoutOutputCollector;
import backtype.storm.topology.base.BaseRichSpout;
import java.util.Map;
import java.io.BufferedReader;
import java.io.FileReader;
import java.io.IOException;
import java.util.concurrent.atomic.AtomicLong;
import storm.external.*;// error from here
import storm.external.storm-redis.org.apache.storm.redis.common.config.JedisClusterConfig;
import org.apache.storm.redis.common.config.JedisPoolConfig;
import org.apache.storm.redis.common.mapper.RedisDataTypeDescription;
import org.apache.storm.redis.common.mapper.RedisStoreMapper;
import redis.clients.jedis.JedisCommands;//to here


..........
class MortsStoreMapper implements RedisStoreMapper {
private RedisDataTypeDescription description;
private final String hashKey = "wordCount";

public WordCountStoreMapper() {
description = new RedisDataTypeDescription(
RedisDataTypeDescription.RedisDataType.HASH, hashKey);
}

@Override
public RedisDataTypeDescription getDataTypeDescription() {
return description;
}

@Override
public String getKeyFromTuple(ITuple tuple) {
return tuple.getStringByField("word");
}

@Override
public String getValueFromTuple(ITuple tuple) {
return tuple.getStringByField("count");
}
}


And my pom.xml:

<project xmlns="http://maven.apache.org/POM/4.0.0" xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
xsi:schemaLocation="http://maven.apache.org/POM/4.0.0 http://maven.apache.org/xsd/maven-4.0.0.xsd">
<modelVersion>4.0.0</modelVersion>

<groupId>Storm.practice</groupId>
<artifactId>Storm.Prova</artifactId>
<version>0.0.1-SNAPSHOT</version>
<packaging>jar</packaging>

<name>Storm.Prova</name>
<url>http://maven.apache.org</url>

<properties>
<project.build.sourceEncoding>UTF-8</project.build.sourceEncoding>
</properties>

<dependencies>
<dependency>
<groupId>org.apache.storm</groupId>
<artifactId>storm-core</artifactId>
<version>0.9.1-incubating</version>
</dependency>
<dependency> #error from here...
<groupId>org.apache.storm</groupId>
<artifactId>storm-redis</artifactId>
<version>{0.9.1-incubating}</version>
<type>jar</type>
</dependency>#... to here
<dependency>
<groupId>junit</groupId>
<artifactId>junit</artifactId>
<version>3.8.1</version>
<scope>test</scope>
</dependency>
</dependencies> <build>
<plugins>
<plugin>
<groupId>org.codehaus.mojo</groupId>
<artifactId>exec-maven-plugin</artifactId>
<version>1.2.1</version>
<executions>
<execution>
<goals>
<goal>java</goal>
</goals>
</execution>
</executions>
<configuration>
<mainClass>Storm.practice.Storm.Prova.ProvaTopology</mainClass>
</configuration>
</plugin>
</plugins>
</build>


</project>


The errors are that Eclipse can't find the dependences and the packages

Answer

I now it's a bit late, almost a year, but I was reviewing my account, and saw this question.

I finally used Redis, interfaced wit Jedis, that I import as a Maven artifact. Once this was working and I was able to see the results with the Redis Monitor via telnet, I created a simple Node.js code, launched it, and the data was arriving to the client, hence to d3. I needed Socket.io and Redis.js to achieve this, but is working now.

If someone need some details, please, ask me and I will help you happily.