markvgti markvgti - 25 days ago 8
Java Question

Random REST API calls fail after upgrading Jersey version

Recently had to upgrade version of Jersey from 2.5.1 (reason explained later) to 2.7 and ever since the upgrade, every time I upload code to the server, random API calls start failing with the following stacktrace:

org.glassfish.jersey.internal.Errors logErrors: The following warnings have been detected: WARNING: HK2 service reification failed for [com.example.rest.LeaderboardApi] with an exception:
MultiException stack 1 of 2
java.lang.NullPointerException
at org.jvnet.hk2.internal.Utilities.hasInjectAnnotation(Utilities.java:1484)
at org.jvnet.hk2.internal.Utilities.findInitializerMethods(Utilities.java:1425)
at org.jvnet.hk2.internal.DefaultClassAnalyzer.getInitializerMethods(DefaultClassAnalyzer.java:102)
at org.glassfish.jersey.internal.inject.JerseyClassAnalyzer.getInitializerMethods(JerseyClassAnalyzer.java:239)
at org.jvnet.hk2.internal.Utilities.getInitMethods(Utilities.java:243)
at org.jvnet.hk2.internal.ClazzCreator.initialize(ClazzCreator.java:163)
at org.jvnet.hk2.internal.ClazzCreator.initialize(ClazzCreator.java:200)
at org.jvnet.hk2.internal.SystemDescriptor.internalReify(SystemDescriptor.java:649)
at org.jvnet.hk2.internal.SystemDescriptor.reify(SystemDescriptor.java:604)
at org.jvnet.hk2.internal.ServiceLocatorImpl.reifyDescriptor(ServiceLocatorImpl.java:405)
at org.jvnet.hk2.internal.ServiceLocatorImpl.narrow(ServiceLocatorImpl.java:2046)
at org.jvnet.hk2.internal.ServiceLocatorImpl.access$700(ServiceLocatorImpl.java:116)
at org.jvnet.hk2.internal.ServiceLocatorImpl$6.compute(ServiceLocatorImpl.java:1011)
at org.jvnet.hk2.internal.ServiceLocatorImpl$6.compute(ServiceLocatorImpl.java:1006)
at org.glassfish.hk2.utilities.cache.LRUHybridCache$OriginThreadAwareFuture$1.call(LRUHybridCache.java:115)
at org.glassfish.hk2.utilities.cache.LRUHybridCache$OriginThreadAwareFuture$1.call(LRUHybridCache.java:111)
at java.util.concurrent.FutureTask.run(FutureTask.java:260)
at org.glassfish.hk2.utilities.cache.LRUHybridCache$OriginThreadAwareFuture.run(LRUHybridCache.java:173)
at org.glassfish.hk2.utilities.cache.LRUHybridCache.compute(LRUHybridCache.java:292)
at org.jvnet.hk2.internal.ServiceLocatorImpl.internalGetDescriptor(ServiceLocatorImpl.java:1093)
at org.jvnet.hk2.internal.ServiceLocatorImpl.getService(ServiceLocatorImpl.java:616)
at org.jvnet.hk2.internal.ServiceLocatorImpl.getService(ServiceLocatorImpl.java:606)
at org.glassfish.jersey.internal.inject.Injections.getOrCreate(Injections.java:173)
at org.glassfish.jersey.server.model.MethodHandler$ClassBasedMethodHandler.getInstance(MethodHandler.java:185)
at org.glassfish.jersey.server.internal.routing.PushMethodHandlerRouter.apply(PushMethodHandlerRouter.java:74)
at org.glassfish.jersey.server.internal.routing.RoutingStage._apply(RoutingStage.java:112)
at org.glassfish.jersey.server.internal.routing.RoutingStage._apply(RoutingStage.java:115)
at org.glassfish.jersey.server.internal.routing.RoutingStage._apply(RoutingStage.java:115)
at org.glassfish.jersey.server.internal.routing.RoutingStage._apply(RoutingStage.java:115)
at org.glassfish.jersey.server.internal.routing.RoutingStage.apply(RoutingStage.java:94)
at org.glassfish.jersey.server.internal.routing.RoutingStage.apply(RoutingStage.java:63)
at org.glassfish.jersey.process.internal.Stages.process(Stages.java:197)
at org.glassfish.jersey.server.ServerRuntime$1.run(ServerRuntime.java:261)
at org.glassfish.jersey.internal.Errors$1.call(Errors.java:271)
at org.glassfish.jersey.internal.Errors$1.call(Errors.java:267)
at org.glassfish.jersey.internal.Errors.process(Errors.java:315)
at org.glassfish.jersey.internal.Errors.process(Errors.java:297)
at org.glassfish.jersey.internal.Errors.process(Errors.java:267)
at org.glassfish.jersey.process.internal.RequestScope.runInScope(RequestScope.java:297)
at org.glassfish.jersey.server.ServerRuntime.process(ServerRuntime.java:252)
at org.glassfish.jersey.server.ApplicationHandler.handle(ApplicationHandler.java:1023)
at org.glassfish.jersey.servlet.WebComponent.service(WebComponent.java:372)
at org.glassfish.jersey.servlet.ServletContainer.service(ServletContainer.java:382)
at org.glassfish.jersey.servlet.ServletContainer.doFilter(ServletContainer.java:535)
at org.glassfish.jersey.servlet.ServletContainer.doFilter(ServletContainer.java:483)
at com.example.config.JerseyFilter.doFilter(JerseyFilter.java:27)
at org.glassfish.jersey.servlet.ServletContainer.doFilter(ServletContainer.java:420)
at org.mortbay.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1157)
at org.apache.shiro.web.servlet.ProxiedFilterChain.doFilter(ProxiedFilterChain.java:61)
at org.apache.shiro.web.servlet.AdviceFilter.executeChain(AdviceFilter.java:108)
at org.apache.shiro.web.servlet.AdviceFilter.doFilterInternal(AdviceFilter.java:137)
at org.apache.shiro.web.servlet.OncePerRequestFilter.doFilter(OncePerRequestFilter.java:125)
at org.apache.shiro.web.servlet.ProxiedFilterChain.doFilter(ProxiedFilterChain.java:66)
at org.apache.shiro.web.servlet.AbstractShiroFilter.executeChain(AbstractShiroFilter.java:449)
at org.apache.shiro.web.servlet.AbstractShiroFilter$1.call(AbstractShiroFilter.java:365)
at org.apache.shiro.subject.support.SubjectCallable.doCall(SubjectCallable.java:90)
at org.apache.shiro.subject.support.SubjectCallable.call(SubjectCallable.java:83)
at org.apache.shiro.subject.support.DelegatingSubject.execute(DelegatingSubject.java:383)
at org.apache.shiro.web.servlet.AbstractShiroFilter.doFilterInternal(AbstractShiroFilter.java:362)
at org.apache.shiro.web.servlet.OncePerRequestFilter.doFilter(OncePerRequestFilter.java:125)
at org.mortbay.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1157)
at com.googlecode.objectify.ObjectifyFilter.doFilter(ObjectifyFilter.java:48)
at org.mortbay.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1157)
at com.google.apphosting.utils.servlet.ParseBlobUploadFilter.doFilter(ParseBlobUploadFilter.java:125)
at org.mortbay.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1157)
at com.google.apphosting.runtime.jetty.SaveSessionFilter.doFilter(SaveSessionFilter.java:37)
at org.mortbay.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1157)
at com.google.apphosting.utils.servlet.JdbcMySqlConnectionCleanupFilter.doFilter(JdbcMySqlConnectionCleanupFilter.java:60)
at org.mortbay.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1157)
at com.google.apphosting.utils.servlet.TransactionCleanupFilter.doFilter(TransactionCleanupFilter.java:48)
at org.mortbay.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1157)
at org.mortbay.jetty.servlet.ServletHandler.handle(ServletHandler.java:388)
at org.mortbay.jetty.security.SecurityHandler.handle(SecurityHandler.java:216)
at org.mortbay.jetty.servlet.SessionHandler.handle(SessionHandler.java:182)
at org.mortbay.jetty.handler.ContextHandler.handle(ContextHandler.java:765)
at org.mortbay.jetty.webapp.WebAppContext.handle(WebAppContext.java:418)
at com.google.apphosting.runtime.jetty.AppVersionHandlerMap.handle(AppVersionHandlerMap.java:257)
at org.mortbay.jetty.handler.HandlerWrapper.handle(HandlerWrapper.java:152)
at org.mortbay.jetty.Server.handle(Server.java:326)
at org.mortbay.jetty.HttpConnection.handleRequest(HttpConnection.java:542)
at org.mortbay.jetty.HttpConnection$RequestHandler.headerComplete(HttpConnection.java:923)
at com.google.apphosting.runtime.jetty.RpcRequestParser.parseAvailable(RpcRequestParser.java:76)
at org.mortbay.jetty.HttpConnection.handle(HttpConnection.java:404)
at com.google.apphosting.runtime.jetty.JettyServletEngineAdapter.serviceRequest(JettyServletEngineAdapter.java:146)
at com.google.apphosting.runtime.JavaRuntime$RequestRunnable.dispatchServletRequest(JavaRuntime.java:657)
at com.google.apphosting.runtime.JavaRuntime$RequestRunnable.dispatchRequest(JavaRuntime.java:619)
at com.google.apphosting.runtime.JavaRuntime$RequestRunnable.run(JavaRuntime.java:589)
at com.google.tracing.TraceContext$TraceContextRunnable.runInContext(TraceContext.java:453)
at com.google.tracing.TraceContext$TraceContextRunnable$1.run(TraceContext.java:460)
at com.google.tracing.CurrentContext.runInContext(CurrentContext.java:293)
at com.google.tracing.TraceContext$AbstractTraceContextCallback.runInInheritedContextNoUnref(TraceContext.java:319)
at com.google.tracing.TraceContext$AbstractTraceContextCallback.runInInheritedContext(TraceContext.java:311)
at com.google.tracing.TraceContext$TraceContextRunnable.run(TraceContext.java:457)
at com.google.apphosting.runtime.ThreadGroupPool$PoolEntry.run(ThreadGroupPool.java:263)
at java.lang.Thread.run(Thread.java:745)

MultiException stack 2 of 2
java.lang.IllegalArgumentException: Errors were discovered while reifying SystemDescriptor(
implementation=com.example.rest.LeaderboardApi
contracts={com.example.rest.LeaderboardApi}
scope=org.glassfish.jersey.process.internal.RequestScoped
qualifiers={}
descriptorType=CLASS
descriptorVisibility=NORMAL
metadata=
rank=0
loader=null
proxiable=null
proxyForSameScope=null
analysisName=null
id=156
locatorId=0
identityHashCode=1367521629
reified=false)
at org.jvnet.hk2.internal.SystemDescriptor.reify(SystemDescriptor.java:615)
at org.jvnet.hk2.internal.ServiceLocatorImpl.reifyDescriptor(ServiceLocatorImpl.java:405)
at org.jvnet.hk2.internal.ServiceLocatorImpl.narrow(ServiceLocatorImpl.java:2046)
at org.jvnet.hk2.internal.ServiceLocatorImpl.access$700(ServiceLocatorImpl.java:116)
at org.jvnet.hk2.internal.ServiceLocatorImpl$6.compute(ServiceLocatorImpl.java:1011)
at org.jvnet.hk2.internal.ServiceLocatorImpl$6.compute(ServiceLocatorImpl.java:1006)
at org.glassfish.hk2.utilities.cache.LRUHybridCache$OriginThreadAwareFuture$1.call(LRUHybridCache.java:115)
at org.glassfish.hk2.utilities.cache.LRUHybridCache$OriginThreadAwareFuture$1.call(LRUHybridCache.java:111)
at java.util.concurrent.FutureTask.run(FutureTask.java:260)
at org.glassfish.hk2.utilities.cache.LRUHybridCache$OriginThreadAwareFuture.run(LRUHybridCache.java:173)
at org.glassfish.hk2.utilities.cache.LRUHybridCache.compute(LRUHybridCache.java:292)
at org.jvnet.hk2.internal.ServiceLocatorImpl.internalGetDescriptor(ServiceLocatorImpl.java:1093)
at org.jvnet.hk2.internal.ServiceLocatorImpl.getService(ServiceLocatorImpl.java:616)
at org.jvnet.hk2.internal.ServiceLocatorImpl.getService(ServiceLocatorImpl.java:606)
at org.glassfish.jersey.internal.inject.Injections.getOrCreate(Injections.java:173)
at org.glassfish.jersey.server.model.MethodHandler$ClassBasedMethodHandler.getInstance(MethodHandler.java:185)
at org.glassfish.jersey.server.internal.routing.PushMethodHandlerRouter.apply(PushMethodHandlerRouter.java:74)
at org.glassfish.jersey.server.internal.routing.RoutingStage._apply(RoutingStage.java:112)
at org.glassfish.jersey.server.internal.routing.RoutingStage._apply(RoutingStage.java:115)
at org.glassfish.jersey.server.internal.routing.RoutingStage._apply(RoutingStage.java:115)
at org.glassfish.jersey.server.internal.routing.RoutingStage._apply(RoutingStage.java:115)
at org.glassfish.jersey.server.internal.routing.RoutingStage.apply(RoutingStage.java:94)
at org.glassfish.jersey.server.internal.routing.RoutingStage.apply(RoutingStage.java:63)
at org.glassfish.jersey.process.internal.Stages.process(Stages.java:197)
at org.glassfish.jersey.server.ServerRuntime$1.run(ServerRuntime.java:261)
at org.glassfish.jersey.internal.Errors$1.call(Errors.java:271)
at org.glassfish.jersey.internal.Errors$1.call(Errors.java:267)
at org.glassfish.jersey.internal.Errors.process(Errors.java:315)
at org.glassfish.jersey.internal.Errors.process(Errors.java:297)
at org.glassfish.jersey.internal.Errors.process(Errors.java:267)
at org.glassfish.jersey.process.internal.RequestScope.runInScope(RequestScope.java:297)
at org.glassfish.jersey.server.ServerRuntime.process(ServerRuntime.java:252)
at org.glassfish.jersey.server.ApplicationHandler.handle(ApplicationHandler.java:1023)
at org.glassfish.jersey.servlet.WebComponent.service(WebComponent.java:372)
at org.glassfish.jersey.servlet.ServletContainer.service(ServletContainer.java:382)
at org.glassfish.jersey.servlet.ServletContainer.doFilter(ServletContainer.java:535)
at org.glassfish.jersey.servlet.ServletContainer.doFilter(ServletContainer.java:483)
at com.example.config.JerseyFilter.doFilter(JerseyFilter.java:27)
at org.glassfish.jersey.servlet.ServletContainer.doFilter(ServletContainer.java:420)
at org.mortbay.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1157)
at org.apache.shiro.web.servlet.ProxiedFilterChain.doFilter(ProxiedFilterChain.java:61)
at org.apache.shiro.web.servlet.AdviceFilter.executeChain(AdviceFilter.java:108)
at org.apache.shiro.web.servlet.AdviceFilter.doFilterInternal(AdviceFilter.java:137)
at org.apache.shiro.web.servlet.OncePerRequestFilter.doFilter(OncePerRequestFilter.java:125)
at org.apache.shiro.web.servlet.ProxiedFilterChain.doFilter(ProxiedFilterChain.java:66)
at org.apache.shiro.web.servlet.AbstractShiroFilter.executeChain(AbstractShiroFilter.java:449)
at org.apache.shiro.web.servlet.AbstractShiroFilter$1.call(AbstractShiroFilter.java:365)
at org.apache.shiro.subject.support.SubjectCallable.doCall(SubjectCallable.java:90)
at org.apache.shiro.subject.support.SubjectCallable.call(SubjectCallable.java:83)
at org.apache.shiro.subject.support.DelegatingSubject.execute(DelegatingSubject.java:383)
at org.apache.shiro.web.servlet.AbstractShiroFilter.doFilterInternal(AbstractShiroFilter.java:362)
at org.apache.shiro.web.servlet.OncePerRequestFilter.doFilter(OncePerRequestFilter.java:125)
at org.mortbay.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1157)
at com.googlecode.objectify.ObjectifyFilter.doFilter(ObjectifyFilter.java:48)
at org.mortbay.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1157)
at com.google.apphosting.utils.servlet.ParseBlobUploadFilter.doFilter(ParseBlobUploadFilter.java:125)
at org.mortbay.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1157)
at com.google.apphosting.runtime.jetty.SaveSessionFilter.doFilter(SaveSessionFilter.java:37)
at org.mortbay.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1157)
at com.google.apphosting.utils.servlet.JdbcMySqlConnectionCleanupFilter.doFilter(JdbcMySqlConnectionCleanupFilter.java:60)
at org.mortbay.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1157)
at com.google.apphosting.utils.servlet.TransactionCleanupFilter.doFilter(TransactionCleanupFilter.java:48)
at org.mortbay.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1157)
at org.mortbay.jetty.servlet.ServletHandler.handle(ServletHandler.java:388)
at org.mortbay.jetty.security.SecurityHandler.handle(SecurityHandler.java:216)
at org.mortbay.jetty.servlet.SessionHandler.handle(SessionHandler.java:182)
at org.mortbay.jetty.handler.ContextHandler.handle(ContextHandler.java:765)
at org.mortbay.jetty.webapp.WebAppContext.handle(WebAppContext.java:418)
at com.google.apphosting.runtime.jetty.AppVersionHandlerMap.handle(AppVersionHandlerMap.java:257)
at org.mortbay.jetty.handler.HandlerWrapper.handle(HandlerWrapper.java:152)
at org.mortbay.jetty.Server.handle(Server.java:326)
at org.mortbay.jetty.HttpConnection.handleRequest(HttpConnection.java:542)
at org.mortbay.jetty.HttpConnection$RequestHandler.headerComplete(HttpConnection.java:923)
at com.google.apphosting.runtime.jetty.RpcRequestParser.parseAvailable(RpcRequestParser.java:76)
at org.mortbay.jetty.HttpConnection.handle(HttpConnection.java:404)
at com.google.apphosting.runtime.jetty.JettyServletEngineAdapter.serviceRequest(JettyServletEngineAdapter.java:146)
at com.google.apphosting.runtime.JavaRuntime$RequestRunnable.dispatchServletRequest(JavaRuntime.java:657)
at com.google.apphosting.runtime.JavaRuntime$RequestRunnable.dispatchRequest(JavaRuntime.java:619)
at com.google.apphosting.runtime.JavaRuntime$RequestRunnable.run(JavaRuntime.java:589)
at com.google.tracing.TraceContext$TraceContextRunnable.runInContext(TraceContext.java:453)
at com.google.tracing.TraceContext$TraceContextRunnable$1.run(TraceContext.java:460)
at com.google.tracing.CurrentContext.runInContext(CurrentContext.java:293)
at com.google.tracing.TraceContext$AbstractTraceContextCallback.runInInheritedContextNoUnref(TraceContext.java:319)
at com.google.tracing.TraceContext$AbstractTraceContextCallback.runInInheritedContext(TraceContext.java:311)
at com.google.tracing.TraceContext$TraceContextRunnable.run(TraceContext.java:457)
at com.google.apphosting.runtime.ThreadGroupPool$PoolEntry.run(ThreadGroupPool.java:263)
at java.lang.Thread.run(Thread.java:745)


Every time I do a clean build and re-upload, a new set of methods start failing (and the previously-failing ones start working) without any code change!

After trying 2.7, I switched to 2.25.1 and that opened up a whole another can of worms, so I tried with 2.5.2 (instead of the original 2.5.1) and it had the same problems as 2.5.1. I switched to 2.8, and it had the same random issues as with 2.7.

Current status is that with either version 2.7 or 2.8, a small (at least seemingly) random subset of the REST API methods fails with messages similar to the one above.

I would like to find out why this is and fix it.

Method Followed While Switching Jersey Versions



I would clean out the
/WEB-INF/lib
folder, change the
jersey.version
variable in
pom.xml
to the new version. Eclipse would then fetch all the necessary libs and re-build code. Running
mvn appengine:update
would re-populate the
/WEB-INF/lib
folder and package everything up.

A few times I did a clean (full) build to ensure that no artefacts from different versions would get mixed.

Reason for Upgrading from 2.5.1



Had to add a method (with the following signature):

public Response importChildren(@Context ContainerRequestContext reqCtx,
@PathParam(ID) String id, @FormDataParam("file") InputStream csvStream,
@FormDataParam("test") String testStr)


that would take an
InputStream
of a CSV file and process that. Because of that had to add
jersey-media-multipart
. Used with version 2.5.x (haven't tried with 2.6), this starts giving an error because it tries to write the
InputStream
to a temporary file (writing to disk is prohibited in Google App Engine (Standard), which is what I am using).

With versions >= 2.7 this tendency to write a temporary file can be avoided by adding file
jersey-multipart-config.properties
with the content
jersey.config.multipart.bufferThreshold = -1


A Desperate Workaround



If nothing else works, I could move the CSV processing code out of Jersey-mediated methods and into a simple servlet. Then I don't need
jersey-media-multipart
and can move back to using version 2.5.1, where all the code was working reliably.

Answer Source

This one

java.lang.NullPointerException
    at org.jvnet.hk2.internal.Utilities.hasInjectAnnotation(Utilities.java:1484)

is a known bug of interoperation between Jersey and particular underlying HK2 kernel version, which comes with Jersey 2.7 as transitive dependency

We've switched to HK2 2.3.0-b05 in 2.9-SNAPSHOT. After we release 2.9 the problem should be gone.

Looks like the closest Jersey version which has that bug fixed is 2.9. But if you'd prefer to stay with 2.7 please find below workaround.

To force usage of hk2-api 2.3.0-b05 version through Maven:

<dependency>
    <groupId>org.glassfish.jersey.core</groupId>
    <artifactId>jersey-common</artifactId>
    <version>2.7</version>
    <exclusions>
        <exclusion>
            <groupId>org.glassfish.hk2</groupId>
            <artifactId>hk2-api</artifactId>
            <!-- this is to exclude 2.2.0 version -->
        </exclusion>
    </exclusions>
</dependency>
<dependency>
    <groupId>org.glassfish.hk2</groupId>
    <artifactId>hk2-api</artifactId>
    <version>2.3.0-b05</version>
</dependency>

That exclusion exercise may be required for some of the rest Jersey artifacts in your pom.xml, just check the mvn dependency:tree ... accordingly