Szymon Stepniak Szymon Stepniak - 3 months ago 340
Groovy Question

Spring Boot + Eureka Server + Hystrix with Turbine: empty turbine.stream

I'm trying to run Spring Boot (with Spring Cloud) + Eureka Server + Hystrix Dashboard and Turbine stream, but I run into a problem I couldn't find any solution so far. I use Spring Boot 1.2.1.RELEASE and Spring Cloud 1.0.0.RC2. Here is what I have:

The first instance is running Eureka server and Hystrix dashboard:

@Configuration
@EnableAutoConfiguration
@EnableEurekaServer
@EnableHystrixDashboard
@EnableDiscoveryClient
class Application {

public static void main(String[] args) {
SpringApplication.run Application, args
}
}


Here you can find
build.gradle
for that instance - https://gist.github.com/wololock/570272ad7cf2d14a4d3c

Eureka server is running ok, I can see registered instances on eureka server dashboard, I can also use
LoadBalancer
to get the URL to registered instance using its id. So far everything is ok.

I have a few instances that are run with
@EnableHystrix
annotation and use
@HystrixCommand
to define which methods have to be monitored by Hystrix. When I pass URL to hystrix.stream of single instance to Hystrix dashboard, I can see it running with no problem.

I have also separate Turbine server, not complicated one:

@EnableAutoConfiguration
@EnableTurbine
@Configuration
@EnableDiscoveryClient
class Application {

public static void main(String[] args) {
SpringApplication.run Application, args
}
}


Here you can find
build.gradle
for Turbine server instance - https://gist.github.com/wololock/ff0d855b8a890232851e

It uses very simple configuration, build mostly on the one provided by sample turbine app - https://github.com/spring-cloud-samples/turbine

info:
component: Turbine

endpoints:
restart:
enabled: true
shutdown:
enabled: true

turbine:
appConfig: pdf-creator-service

InstanceDiscovery:
impl: io.spring.platform.netflix.turbine.EurekaInstanceDiscovery

server:
port: 8989

management:
port: 8990

eureka:
instance:
leaseRenewalIntervalInSeconds: 10
client:
serviceUrl:
defaultZone: ${vcap.services.${PREFIX:}eureka.credentials.uri:http://user:password@localhost:8761}/eureka/


After running those instances in order:


  1. eureka server

  2. turbine server

  3. discover client instance,



I have 2nd and 3rd instance registered in eureka server, turbine server log says, that there is one instance up:

[2015-02-06 12:35:04.162] boot - 20495 INFO [Timer-0] --- EurekaInstanceDiscovery: Fetching instance list for apps: [pdf-creator-service]
[2015-02-06 12:35:04.162] boot - 20495 INFO [Timer-0] --- EurekaInstanceDiscovery: Fetching instances for app: pdf-creator-service
[2015-02-06 12:35:04.162] boot - 20495 INFO [Timer-0] --- EurekaInstanceDiscovery: Received instance list for app: pdf-creator-service = 1
[2015-02-06 12:35:04.162] boot - 20495 INFO [Timer-0] --- InstanceObservable: Retrieved hosts from InstanceDiscovery: 1
[2015-02-06 12:35:04.162] boot - 20495 INFO [Timer-0] --- InstanceObservable: Found hosts that have been previously terminated: 0
[2015-02-06 12:35:04.162] boot - 20495 INFO [Timer-0] --- InstanceObservable: Hosts up:1, hosts down: 0
[2015-02-06 12:36:04.162] boot - 20495 INFO [Timer-0] --- EurekaInstanceDiscovery: Fetching instance list for apps: [pdf-creator-service]
[2015-02-06 12:36:04.162] boot - 20495 INFO [Timer-0] --- EurekaInstanceDiscovery: Fetching instances for app: pdf-creator-service
[2015-02-06 12:36:04.162] boot - 20495 INFO [Timer-0] --- EurekaInstanceDiscovery: Received instance list for app: pdf-creator-service = 1
[2015-02-06 12:36:04.162] boot - 20495 INFO [Timer-0] --- InstanceObservable: Retrieved hosts from InstanceDiscovery: 1
[2015-02-06 12:36:04.162] boot - 20495 INFO [Timer-0] --- InstanceObservable: Found hosts that have been previously terminated: 0
[2015-02-06 12:36:04.162] boot - 20495 INFO [Timer-0] --- InstanceObservable: Hosts up:1, hosts down: 0


Calling hystrix.stream from a single instance works, e.g.
curl http://localhost:8885/hystrix.stream
returns:

data: {"type":"HystrixCommand","name":"post","group":"PdfController","currentTime":1423223614259,"isCircuitBreakerOpen":false,"errorPercentage":0,"errorCount":0,"requestCount":0,"rollingCountCollapsedRequests":0,"rollingCountExceptionsThrown":0,"rollingCountFailure":0,"rollingCountFallbackFailure":0,"rollingCountFallbackRejection":0,"rollingCountFallbackSuccess":0,"rollingCountResponsesFromCache":0,"rollingCountSemaphoreRejected":0,"rollingCountShortCircuited":0,"rollingCountSuccess":0,"rollingCountThreadPoolRejected":0,"rollingCountTimeout":0,"currentConcurrentExecutionCount":0,"latencyExecute_mean":0,"latencyExecute":{"0":0,"25":0,"50":0,"75":0,"90":0,"95":0,"99":0,"99.5":0,"100":0},"latencyTotal_mean":0,"latencyTotal":{"0":0,"25":0,"50":0,"75":0,"90":0,"95":0,"99":0,"99.5":0,"100":0},"propertyValue_circuitBreakerRequestVolumeThreshold":20,"propertyValue_circuitBreakerSleepWindowInMilliseconds":5000,"propertyValue_circuitBreakerErrorThresholdPercentage":50,"propertyValue_circuitBreakerForceOpen":false,"propertyValue_circuitBreakerForceClosed":false,"propertyValue_circuitBreakerEnabled":true,"propertyValue_executionIsolationStrategy":"THREAD","propertyValue_executionIsolationThreadTimeoutInMilliseconds":8000,"propertyValue_executionIsolationThreadInterruptOnTimeout":true,"propertyValue_executionIsolationThreadPoolKeyOverride":null,"propertyValue_executionIsolationSemaphoreMaxConcurrentRequests":10,"propertyValue_fallbackIsolationSemaphoreMaxConcurrentRequests":10,"propertyValue_metricsRollingStatisticalWindowInMilliseconds":10000,"propertyValue_requestCacheEnabled":true,"propertyValue_requestLogEnabled":true,"reportingHosts":1}

data: {"type":"HystrixCommand","name":"generate","group":"WkHtmlToPdfGenerator","currentTime":1423223614259,"isCircuitBreakerOpen":false,"errorPercentage":0,"errorCount":0,"requestCount":0,"rollingCountCollapsedRequests":0,"rollingCountExceptionsThrown":0,"rollingCountFailure":0,"rollingCountFallbackFailure":0,"rollingCountFallbackRejection":0,"rollingCountFallbackSuccess":0,"rollingCountResponsesFromCache":0,"rollingCountSemaphoreRejected":0,"rollingCountShortCircuited":0,"rollingCountSuccess":0,"rollingCountThreadPoolRejected":0,"rollingCountTimeout":0,"currentConcurrentExecutionCount":0,"latencyExecute_mean":0,"latencyExecute":{"0":0,"25":0,"50":0,"75":0,"90":0,"95":0,"99":0,"99.5":0,"100":0},"latencyTotal_mean":0,"latencyTotal":{"0":0,"25":0,"50":0,"75":0,"90":0,"95":0,"99":0,"99.5":0,"100":0},"propertyValue_circuitBreakerRequestVolumeThreshold":20,"propertyValue_circuitBreakerSleepWindowInMilliseconds":5000,"propertyValue_circuitBreakerErrorThresholdPercentage":50,"propertyValue_circuitBreakerForceOpen":false,"propertyValue_circuitBreakerForceClosed":false,"propertyValue_circuitBreakerEnabled":true,"propertyValue_executionIsolationStrategy":"THREAD","propertyValue_executionIsolationThreadTimeoutInMilliseconds":8000,"propertyValue_executionIsolationThreadInterruptOnTimeout":true,"propertyValue_executionIsolationThreadPoolKeyOverride":null,"propertyValue_executionIsolationSemaphoreMaxConcurrentRequests":10,"propertyValue_fallbackIsolationSemaphoreMaxConcurrentRequests":10,"propertyValue_metricsRollingStatisticalWindowInMilliseconds":10000,"propertyValue_requestCacheEnabled":true,"propertyValue_requestLogEnabled":true,"reportingHosts":1}

data: {"type":"HystrixThreadPool","name":"PdfController","currentTime":1423223614259,"currentActiveCount":0,"currentCompletedTaskCount":4,"currentCorePoolSize":10,"currentLargestPoolSize":4,"currentMaximumPoolSize":10,"currentPoolSize":4,"currentQueueSize":0,"currentTaskCount":4,"rollingCountThreadsExecuted":0,"rollingMaxActiveThreads":0,"propertyValue_queueSizeRejectionThreshold":5,"propertyValue_metricsRollingStatisticalWindowInMilliseconds":10000,"reportingHosts":1}


But when I attach turbine.stream to hystrix dashboard, I get nothing. Logs say:

[2015-02-06 12:42:48.922] boot - 24816 INFO [Timer-0] --- EurekaInstanceDiscovery: Received instance list for app: pdf-creator-service = 1
[2015-02-06 12:42:48.922] boot - 24816 INFO [Timer-0] --- InstanceObservable: Retrieved hosts from InstanceDiscovery: 1
[2015-02-06 12:42:48.922] boot - 24816 INFO [Timer-0] --- InstanceObservable: Found hosts that have been previously terminated: 0
[2015-02-06 12:42:48.922] boot - 24816 INFO [Timer-0] --- InstanceObservable: Hosts up:1, hosts down: 0
[2015-02-06 12:43:26.237] boot - 24816 INFO [XNIO-2 task-4] --- TurbineStreamServlet: FilterCriteria: []
[2015-02-06 12:43:26.237] boot - 24816 INFO [XNIO-2 task-4] --- TurbineStreamServlet: StatsType filters: []
[2015-02-06 12:43:26.237] boot - 24816 INFO [XNIO-2 task-4] --- TurbineStreamingConnection: Relevance config: []
[2015-02-06 12:43:26.237] boot - 24816 INFO [XNIO-2 task-4] --- TurbineStreamingConnection: Relevance metrics config: {}
[2015-02-06 12:43:26.237] boot - 24816 INFO [XNIO-2 task-4] --- ClusterMonitor: Registering event handler for cluster monitor: StreamingHandler_f1308dda-58c5-47a5-b1e2-5a0bea32226b
[2015-02-06 12:43:26.237] boot - 24816 INFO [XNIO-2 task-4] --- TurbineDataDispatcher:

Just added and starting handler tuple: StreamingHandler_f1308dda-58c5-47a5-b1e2-5a0bea32226b
[2015-02-06 12:43:26.238] boot - 24816 INFO [XNIO-2 task-4] --- AggDataFromCluster: Per handler dispacher started for: StreamingHandler_f1308dda-58c5-47a5-b1e2-5a0bea32226b
[2015-02-06 12:43:26.238] boot - 24816 INFO [XNIO-2 task-4] --- ClusterMonitor: All event handlers for cluster monitor: [StreamingHandler_637572ab-acda-4bf4-81cd-6a658adb73eb, StreamingHandler_f1308dda-58c5-47a5-b1e2-5a0bea32226b, StaticListener_For_Aggregator, StreamingHandler_5ec12ee8-3fcd-4a6f-9006-d2a6ecc309d0, StreamingHandler_72d7b9e2-ad98-42a0-9ac3-abe4aa57cc7a]
[2015-02-06 12:43:26.238] boot - 24816 INFO [XNIO-2 task-4] --- ClusterMonitor: Starting up the cluster monitor for default_agg


If I do
curl http://localhost:8989/turbine.stream
I get only:

: ping
data: {"reportingHostsLast10Seconds":0,"name":"meta","type":"meta","timestamp":1423223006935}

: ping
data: {"reportingHostsLast10Seconds":0,"name":"meta","type":"meta","timestamp":1423223010935}

: ping
data: {"reportingHostsLast10Seconds":0,"name":"meta","type":"meta","timestamp":1423223013936}

: ping
data: {"reportingHostsLast10Seconds":0,"name":"meta","type":"meta","timestamp":1423223017936}

: ping
data: {"reportingHostsLast10Seconds":0,"name":"meta","type":"meta","timestamp":1423223020937}

: ping
: ping
data: {"reportingHostsLast10Seconds":0,"name":"meta","type":"meta","timestamp":1423223024937}

: ping
data: {"reportingHostsLast10Seconds":0,"name":"meta","type":"meta","timestamp":1423223028938}

: ping
data: {"reportingHostsLast10Seconds":0,"name":"meta","type":"meta","timestamp":1423223032938}

: ping
: ping
data: {"reportingHostsLast10Seconds":0,"name":"meta","type":"meta","timestamp":1423223036938}

: ping
data: {"reportingHostsLast10Seconds":0,"name":"meta","type":"meta","timestamp":1423223039939}

: ping
data: {"reportingHostsLast10Seconds":0,"name":"meta","type":"meta","timestamp":1423223043939}

: ping
data: {"reportingHostsLast10Seconds":0,"name":"meta","type":"meta","timestamp":1423223046940}

: ping
data: {"reportingHostsLast10Seconds":0,"name":"meta","type":"meta","timestamp":1423223050940}

: ping
: ping
data: {"reportingHostsLast10Seconds":0,"name":"meta","type":"meta","timestamp":1423223054941}


My question is: do I missed something in setting it up and running? Previously I tried to manage this by having eureka and turbine server in single instance, but in that case turbine couldn't even find the registered application in eureka using proper application name. I made a progress after splitting eureka and turbine, but it still does not work correctly.

I will be grateful for any suggestion. If you need more information, just let me know, I might miss something important.

UPDATE 20150209



Following Dave's suggestion I applied small changes in
application.yml
file of
turbine-server
. Now the file contains only:

info:
component: Turbine

turbine:
appConfig: pdf-creator-service
clusterNameExpression: 'default'

server:
port: 8989

management:
port: 8990

eureka:
instance:
leaseRenewalIntervalInSeconds: 10
client:
serviceUrl:
defaultZone: ${vcap.services.${PREFIX:}eureka.credentials.uri:http://user:password@localhost:8761}/eureka/


But it doesn't make turbine.stream working. After turbine-server gets aware of registered client in eureka server, it fails with given exception:

[2015-02-09 21:25:03.516] boot - 4808 INFO [Timer-0] --- EurekaInstanceDiscovery: Fetching instance list for apps: [pdf-creator-service]
[2015-02-09 21:25:03.516] boot - 4808 INFO [Timer-0] --- EurekaInstanceDiscovery: Fetching instances for app: pdf-creator-service
[2015-02-09 21:25:03.516] boot - 4808 INFO [Timer-0] --- EurekaInstanceDiscovery: Received instance list for app: pdf-creator-service = 1
[2015-02-09 21:25:03.520] boot - 4808 ERROR [Timer-0] --- EurekaInstanceDiscovery: Failed to fetch instances for app: pdf-creator-service, retrying once more
org.springframework.expression.spel.SpelEvaluationException: EL1008E:(pos 0): Property or field 'default' cannot be found on object of type 'com.netflix.appinfo.InstanceInfo' - maybe not public?
at org.springframework.expression.spel.ast.PropertyOrFieldReference.readProperty(PropertyOrFieldReference.java:226)
at org.springframework.expression.spel.ast.PropertyOrFieldReference.getValueInternal(PropertyOrFieldReference.java:93)
at org.springframework.expression.spel.ast.PropertyOrFieldReference.getValueInternal(PropertyOrFieldReference.java:81)
at org.springframework.expression.spel.ast.SpelNodeImpl.getValue(SpelNodeImpl.java:120)
at org.springframework.expression.spel.standard.SpelExpression.getValue(SpelExpression.java:242)
at org.springframework.cloud.netflix.turbine.EurekaInstanceDiscovery.getClusterName(EurekaInstanceDiscovery.java:183)
at org.springframework.cloud.netflix.turbine.EurekaInstanceDiscovery.marshallInstanceInfo(EurekaInstanceDiscovery.java:141)
at org.springframework.cloud.netflix.turbine.EurekaInstanceDiscovery.getInstancesForApp(EurekaInstanceDiscovery.java:123)
at org.springframework.cloud.netflix.turbine.EurekaInstanceDiscovery.getInstanceList(EurekaInstanceDiscovery.java:88)
at com.netflix.turbine.discovery.InstanceObservable.getInstanceList(InstanceObservable.java:327)
at com.netflix.turbine.discovery.InstanceObservable.access$500(InstanceObservable.java:66)
at com.netflix.turbine.discovery.InstanceObservable$1.run(InstanceObservable.java:258)
at java.util.TimerThread.mainLoop(Timer.java:555)
at java.util.TimerThread.run(Timer.java:505)


I tried writting
'default'
as well as
default
, but the result is unfortunately the same.

Solution



Thanks Dave Syer for finding the proper solution. Basically what did the trick was adding:

turbine:
clusterNameExpression: new String("default")


to
application.yml
file in turbine server application instance. It might look weird, I didn't believe it will work, but it does. Now when I call my hystrix client application I get the proper information in
hystrix.stream
that is served by this application and in
turbine.stream
of Turbine server as well. My current
application.yml
in turbine server looks as follows:

info:
component: Turbine

turbine:
clusterNameExpression: new String("default")
appConfig: pdf-creator-service

server:
port: 8989

management:
port: 8990

eureka:
instance:
leaseRenewalIntervalInSeconds: 10
client:
serviceUrl:
defaultZone: ${vcap.services.${PREFIX:}eureka.credentials.uri:http://user:password@localhost:8761}/eureka/


turbine.appConfig
holds the information about the hystrix clients (by their IDs). To add another client to your turbine server you will have to simply put another instance id, coma separated with the previous one. And that's all folks :)

Answer Source

It works for me if I add some configuration for the cluster, e.g.

turbine:
  appConfig: customers,stores
  clusterNameExpression: new String('default')

Turbine has to know how to construct the "cluster" name (an aggregation key for sets of applications). The default is to use the appname, so if you don't set the clusterNameExpression you need to use a query param in the stream URL, e.g. /turbine.stream?cluster=CUSTOMERS (uppercased appname).