CptS CptS - 1 month ago 10
Java Question

Classloader leak because of invalid SSL certificate in SolrJ, HttpClient, JVM or my application?

I've analyzed a classloader leak the last few days in a large application and I've worked out the problem.

My application uses SolrJ which will be initialilzed through a

@Bean
-Method:

@Bean(destroyMethod = "close")
public SolrClient solrClient() {
return new HttpSolrClient(SOLR_URL);
}


SolrJ (
org.apache.solr:solr-solrj:5.4.1
) uses the Apache HttpClient (
org.apache.httpcomponents:httpclient:4.4.1
). The HttpClient initializes the SSL context by using normal java classes like
javax.net.ssl.SSLSocketFactory
.
In this way java loads the trustManager and analyzes all trusted certificates. If there is an error the certificate (an instance of
sun.security.x509.X509CertImpl
) is stored in a list and gets enriched by the thrown exception.
This exception is swallowed and my application remains unaware.

As far as I can see, the SSL Context is in the System / Root Classloader, my Application is in the dedicated
WebappClassLoader
and this is the problem because now there is an
IOException
inside of the SSL context which contains references in the stacktrace, backtrace and so on to classes in my application.

But now I don't know where this came from. Is it the SolrJ client, the Apache HttpClient, Java itself (the JVM) or is it my application?

I have made a small application to reproduce the problem which you can find here: https://github.com/CptS/solrj-classloader-leak
This also contains a Workaround (a shutdown hook which removes the references which leads to the classloader leak).

If you disable the shutdown hook (e.g. by commenting it out) and start a clean Tomcat (see "Environment to reproduce" below) you can reproduce it by following this steps:


  1. deploy the war of the demo project (A)

  2. reload it (B)

  3. reload it again (C)

  4. Trigger GC (D)

  5. undeploy

  6. Trigger GC (E)

  7. See that the metaspace gets not completly cleaned up (F)



enter image description here

I've create a heap dump and the shortest path to GC looks like this:

enter image description here

This was the same as in my large application.
The mentioned Workaround (a little bit inspired by https://github.com/mjiderhamn/classloader-leak-prevention, but this unfortunately doesn't solve my problem) searches by using reflection for these
unparseableExtensions
and removes the Exception stored in the
why
field through this way:
SSLContextImpl.DefaultSSLContext#defaultImpl
->
SSLContextImpl#trustManager
->
X509TrustManager#trustedCerts
->
X509CertImpl#info
->
X509CertInfo#extensions
->
CertificateExtensions#unparseableExtensions
->
UnparseableExtension#why


By doing this i've got the stacktrace of the exception if it helps someone:

java.io.IOException: No data available in passed DER encoded value.
at sun.security.x509.GeneralNames.<init>(GeneralNames.java:61)
at sun.security.x509.IssuerAlternativeNameExtension.<init>(IssuerAlternativeNameExtension.java:136)
at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method)
at sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:62)
at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45)
at java.lang.reflect.Constructor.newInstance(Constructor.java:423)
at sun.security.x509.CertificateExtensions.parseExtension(CertificateExtensions.java:113)
at sun.security.x509.CertificateExtensions.init(CertificateExtensions.java:88)
at sun.security.x509.CertificateExtensions.<init>(CertificateExtensions.java:78)
at sun.security.x509.X509CertInfo.parse(X509CertInfo.java:702)
at sun.security.x509.X509CertInfo.<init>(X509CertInfo.java:167)
at sun.security.x509.X509CertImpl.parse(X509CertImpl.java:1804)
at sun.security.x509.X509CertImpl.<init>(X509CertImpl.java:195)
at sun.security.provider.X509Factory.engineGenerateCertificate(X509Factory.java:100)
at java.security.cert.CertificateFactory.generateCertificate(CertificateFactory.java:339)
at sun.security.provider.JavaKeyStore.engineLoad(JavaKeyStore.java:755)
at sun.security.provider.JavaKeyStore$JKS.engineLoad(JavaKeyStore.java:56)
at sun.security.provider.KeyStoreDelegator.engineLoad(KeyStoreDelegator.java:224)
at sun.security.provider.JavaKeyStore$DualFormatJKS.engineLoad(JavaKeyStore.java:70)
at java.security.KeyStore.load(KeyStore.java:1445)
at sun.security.ssl.TrustManagerFactoryImpl.getCacertsKeyStore(TrustManagerFactoryImpl.java:226)
at sun.security.ssl.SSLContextImpl$DefaultSSLContext.getDefaultTrustManager(SSLContextImpl.java:767)
at sun.security.ssl.SSLContextImpl$DefaultSSLContext.<init>(SSLContextImpl.java:733)
at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method)
at sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:62)
at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45)
at java.lang.reflect.Constructor.newInstance(Constructor.java:423)
at java.security.Provider$Service.newInstance(Provider.java:1595)
at sun.security.jca.GetInstance.getInstance(GetInstance.java:236)
at sun.security.jca.GetInstance.getInstance(GetInstance.java:164)
at javax.net.ssl.SSLContext.getInstance(SSLContext.java:156)
at javax.net.ssl.SSLContext.getDefault(SSLContext.java:96)
at javax.net.ssl.SSLSocketFactory.getDefault(SSLSocketFactory.java:122)
at org.apache.http.conn.ssl.SSLSocketFactory.getSystemSocketFactory(SSLSocketFactory.java:190)
at org.apache.http.impl.conn.SchemeRegistryFactory.createSystemDefault(SchemeRegistryFactory.java:85)
at org.apache.http.impl.client.SystemDefaultHttpClient.createClientConnectionManager(SystemDefaultHttpClient.java:121)
at org.apache.http.impl.client.AbstractHttpClient.getConnectionManager(AbstractHttpClient.java:484)
at org.apache.solr.client.solrj.impl.HttpClientUtil.setMaxConnections(HttpClientUtil.java:234)
at org.apache.solr.client.solrj.impl.HttpClientConfigurer.configure(HttpClientConfigurer.java:40)
at org.apache.solr.client.solrj.impl.HttpClientUtil.configureClient(HttpClientUtil.java:149)
at org.apache.solr.client.solrj.impl.HttpClientUtil.createClient(HttpClientUtil.java:125)
at org.apache.solr.client.solrj.impl.HttpSolrClient.<init>(HttpSolrClient.java:189)
at org.apache.solr.client.solrj.impl.HttpSolrClient.<init>(HttpSolrClient.java:162)
at de.test.spring.SolrJConfig.solrClient(SolrJConfig.java:20)
at de.test.spring.SolrJConfig$$EnhancerBySpringCGLIB$$dbd4362f.CGLIB$solrClient$0(<generated>)
at de.test.spring.SolrJConfig$$EnhancerBySpringCGLIB$$dbd4362f$$FastClassBySpringCGLIB$$8e7566a6.invoke(<generated>)
at org.springframework.cglib.proxy.MethodProxy.invokeSuper(MethodProxy.java:228)
at org.springframework.context.annotation.ConfigurationClassEnhancer$BeanMethodInterceptor.intercept(ConfigurationClassEnhancer.java:309)
at de.test.spring.SolrJConfig$$EnhancerBySpringCGLIB$$dbd4362f.solrClient(<generated>)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:498)
at org.springframework.beans.factory.support.SimpleInstantiationStrategy.instantiate(SimpleInstantiationStrategy.java:162)
at org.springframework.beans.factory.support.ConstructorResolver.instantiateUsingFactoryMethod(ConstructorResolver.java:588)
at org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.instantiateUsingFactoryMethod(AbstractAutowireCapableBeanFactory.java:1119)
at org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.createBeanInstance(AbstractAutowireCapableBeanFactory.java:1014)
at org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.doCreateBean(AbstractAutowireCapableBeanFactory.java:504)
at org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.createBean(AbstractAutowireCapableBeanFactory.java:476)
at org.springframework.beans.factory.support.AbstractBeanFactory$1.getObject(AbstractBeanFactory.java:303)
at org.springframework.beans.factory.support.DefaultSingletonBeanRegistry.getSingleton(DefaultSingletonBeanRegistry.java:230)
at org.springframework.beans.factory.support.AbstractBeanFactory.doGetBean(AbstractBeanFactory.java:299)
at org.springframework.beans.factory.support.AbstractBeanFactory.getBean(AbstractBeanFactory.java:194)
at org.springframework.beans.factory.support.DefaultListableBeanFactory.preInstantiateSingletons(DefaultListableBeanFactory.java:755)
at org.springframework.context.support.AbstractApplicationContext.finishBeanFactoryInitialization(AbstractApplicationContext.java:757)
at org.springframework.context.support.AbstractApplicationContext.refresh(AbstractApplicationContext.java:480)
at de.test.WicketApplication.init(WicketApplication.java:32)
at org.apache.wicket.Application.initApplication(Application.java:950)
at org.apache.wicket.protocol.http.WicketFilter.init(WicketFilter.java:429)
at org.apache.wicket.protocol.http.WicketFilter.init(WicketFilter.java:353)
at org.apache.catalina.core.ApplicationFilterConfig.initFilter(ApplicationFilterConfig.java:279)
at org.apache.catalina.core.ApplicationFilterConfig.getFilter(ApplicationFilterConfig.java:260)
at org.apache.catalina.core.ApplicationFilterConfig.<init>(ApplicationFilterConfig.java:105)
at org.apache.catalina.core.StandardContext.filterStart(StandardContext.java:4640)
at org.apache.catalina.core.StandardContext.startInternal(StandardContext.java:5247)
at org.apache.catalina.util.LifecycleBase.start(LifecycleBase.java:150)
at org.apache.catalina.core.ContainerBase.addChildInternal(ContainerBase.java:724)
at org.apache.catalina.core.ContainerBase.addChild(ContainerBase.java:700)
at org.apache.catalina.core.StandardHost.addChild(StandardHost.java:714)
at org.apache.catalina.startup.HostConfig.deployWAR(HostConfig.java:919)
at org.apache.catalina.startup.HostConfig$DeployWar.run(HostConfig.java:1703)
at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
at java.util.concurrent.FutureTask.run(FutureTask.java:266)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
at java.lang.Thread.run(Thread.java:745)


My workaround solves the problem for now, but of course this is just a workaround.

I want to know and maybe someone can answer one or more of my question:


  1. Is this a "bug" in SolrJ, HttpClient, Java or my application?

  2. If it is my application, what I am doing wrong?

  3. If it is not my application, is it a known problem? I can't find any information about this. (Where) should I create a bug ticket?

  4. Why is there a "invalid" certificate? (BTW: Maybe the leak would also be solve if I remove this certificate from the trust store... I haven't tested but I think a invalid or damaged certificate should never lead to a classloader leak...)

  5. Has anyone some more information on this? I can't belive that I'm the only one who detect this behaviour (except it is my appication... see my question 2).



At last but not least, my Environment to reproduce:


  • Tomcat Version: Apache Tomcat/8.0.14 (Debian)

  • JVM Version: 1.8.0_91-b14

  • JVM Vendor: Oracle Corporation

  • OS Name: Linux

  • OS Version: 3.16.0-4-amd64

  • Architecture: amd64


Answer

It's a bug in java, the bug ticket is here: http://bugs.java.com/bugdatabase/view_bug.do?bug_id=8168069

Many thanks to mjiderhamn (on GitHub). He is the developer of the great classloader-leak-prevention library and has now included a preventor for this issue (version 2.1.0).