Fix deadlock when loading global configuration on startup #449

nikita-tkachenko-datadog · 2024-09-17T11:02:46Z

Requirements for Contributing to this repository

Fill out the template below. Any pull request that does not include enough information to be reviewed in a timely manner may be closed at the maintainers' discretion.
The pull request must only fix one issue at the time.
The pull request must update the test suite to demonstrate the changed functionality.
After you create the pull request, all status checks must be pass before a maintainer reviews your contribution. For more details, please see CONTRIBUTING.

What does this PR do?

The plugin creates an instance of DatadogClient on startup.
It happens during plugin config initialization.
This PR moves config initialization to a later stage: to wait until the system config is initialized.
This is needed to prevent a deadlock: if not all extensions are loaded by the time global configuration is initialized, it is possible to hit the deadlock shown below:

Found one Java-level deadlock:
=============================
"Handling HEAD / from 172.18.0.5 : Jetty (winstone)-17":
  waiting to lock monitor 0x0000fffe7800eff0 (object 0x00000000898f9758, a hudson.ExtensionList$Lock),
  which is held by "GitSCM.onLoaded"

"GitSCM.onLoaded":
  waiting for ownable synchronizer 0x000000008c1223d0, (a java.util.concurrent.locks.ReentrantLock$NonfairSync),
  which is held by "DatadogGlobalConfiguration.onStartup"

"DatadogGlobalConfiguration.onStartup":
  waiting to lock monitor 0x0000fffe7800eff0 (object 0x00000000898f9758, a hudson.ExtensionList$Lock),
  which is held by "GitSCM.onLoaded"

Java stack information for the threads listed above:
===================================================
"Handling HEAD / from 172.18.0.5 : Jetty (winstone)-17":
	at hudson.ExtensionList.ensureLoaded(ExtensionList.java:319)
	- waiting to lock <0x00000000898f9758> (a hudson.ExtensionList$Lock)
	at hudson.ExtensionList.iterator(ExtensionList.java:172)
	at jenkins.util.HttpServletFilter$1.doFilter(HttpServletFilter.java:70)
	at hudson.util.PluginServletFilter$1.doFilter(PluginServletFilter.java:160)
	at hudson.util.PluginServletFilter.doFilter(PluginServletFilter.java:166)
	at org.eclipse.jetty.ee8.servlet.FilterHolder.doFilter(FilterHolder.java:171)
	at org.eclipse.jetty.ee8.servlet.ServletHandler$Chain.doFilter(ServletHandler.java:1348)
	at jenkins.ErrorAttributeFilter.doFilter(ErrorAttributeFilter.java:29)
	at org.eclipse.jetty.ee8.servlet.FilterHolder.doFilter(FilterHolder.java:171)
	at org.eclipse.jetty.ee8.servlet.ServletHandler$Chain.doFilter(ServletHandler.java:1348)
	at hudson.security.csrf.CrumbFilter.doFilter(CrumbFilter.java:160)
	at org.eclipse.jetty.ee8.servlet.FilterHolder.doFilter(FilterHolder.java:171)
	at org.eclipse.jetty.ee8.servlet.ServletHandler$Chain.doFilter(ServletHandler.java:1348)
	at hudson.security.ChainedServletFilter$1.doFilter(ChainedServletFilter.java:94)
	at jenkins.security.AcegiSecurityExceptionFilter.doFilter(AcegiSecurityExceptionFilter.java:52)
	at hudson.security.ChainedServletFilter$1.doFilter(ChainedServletFilter.java:99)
	at hudson.security.UnwrapSecurityExceptionFilter.doFilter(UnwrapSecurityExceptionFilter.java:54)
	at hudson.security.ChainedServletFilter$1.doFilter(ChainedServletFilter.java:99)
	at org.springframework.security.web.access.ExceptionTranslationFilter.doFilter(ExceptionTranslationFilter.java:126)
	at org.springframework.security.web.access.ExceptionTranslationFilter.doFilter(ExceptionTranslationFilter.java:120)
	at hudson.security.ChainedServletFilter$1.doFilter(ChainedServletFilter.java:99)
	at org.springframework.security.web.authentication.AnonymousAuthenticationFilter.doFilter(AnonymousAuthenticationFilter.java:100)
	at hudson.security.ChainedServletFilter$1.doFilter(ChainedServletFilter.java:99)
	at org.springframework.security.web.authentication.rememberme.RememberMeAuthenticationFilter.doFilter(RememberMeAuthenticationFilter.java:145)
	at org.springframework.security.web.authentication.rememberme.RememberMeAuthenticationFilter.doFilter(RememberMeAuthenticationFilter.java:101)
	at hudson.security.ChainedServletFilter$1.doFilter(ChainedServletFilter.java:99)
	at org.springframework.security.web.authentication.AbstractAuthenticationProcessingFilter.doFilter(AbstractAuthenticationProcessingFilter.java:227)
	at org.springframework.security.web.authentication.AbstractAuthenticationProcessingFilter.doFilter(AbstractAuthenticationProcessingFilter.java:221)
	at hudson.security.ChainedServletFilter$1.doFilter(ChainedServletFilter.java:99)
	at jenkins.security.BasicHeaderProcessor.doFilter(BasicHeaderProcessor.java:97)
	at hudson.security.ChainedServletFilter$1.doFilter(ChainedServletFilter.java:99)
	at org.springframework.security.web.context.SecurityContextPersistenceFilter.doFilter(SecurityContextPersistenceFilter.java:117)
	at org.springframework.security.web.context.SecurityContextPersistenceFilter.doFilter(SecurityContextPersistenceFilter.java:87)
	at hudson.security.HttpSessionContextIntegrationFilter2.doFilter(HttpSessionContextIntegrationFilter2.java:63)
	at hudson.security.ChainedServletFilter$1.doFilter(ChainedServletFilter.java:99)
	at hudson.security.ChainedServletFilter.doFilter(ChainedServletFilter.java:111)
	at hudson.security.HudsonFilter.doFilter(HudsonFilter.java:172)
	at org.eclipse.jetty.ee8.servlet.FilterHolder.doFilter(FilterHolder.java:171)
	at org.eclipse.jetty.ee8.servlet.ServletHandler$Chain.doFilter(ServletHandler.java:1348)
	at org.kohsuke.stapler.UncaughtExceptionFilter.doFilter(UncaughtExceptionFilter.java:27)
	at org.eclipse.jetty.ee8.servlet.FilterHolder.doFilter(FilterHolder.java:171)
	at org.eclipse.jetty.ee8.servlet.ServletHandler$Chain.doFilter(ServletHandler.java:1348)
	at hudson.util.CharacterEncodingFilter.doFilter(CharacterEncodingFilter.java:86)
	at org.eclipse.jetty.ee8.servlet.FilterHolder.doFilter(FilterHolder.java:171)
	at org.eclipse.jetty.ee8.servlet.ServletHandler$Chain.doFilter(ServletHandler.java:1348)
	at org.kohsuke.stapler.DiagnosticThreadNameFilter.doFilter(DiagnosticThreadNameFilter.java:31)
	at org.eclipse.jetty.ee8.servlet.FilterHolder.doFilter(FilterHolder.java:171)
	at org.eclipse.jetty.ee8.servlet.ServletHandler$Chain.doFilter(ServletHandler.java:1348)
	at jenkins.security.SuspiciousRequestFilter.doFilter(SuspiciousRequestFilter.java:38)
	at org.eclipse.jetty.ee8.servlet.FilterHolder.doFilter(FilterHolder.java:171)
	at org.eclipse.jetty.ee8.servlet.ServletHandler$Chain.doFilter(ServletHandler.java:1348)
	at org.eclipse.jetty.ee8.servlet.ServletHandler.doHandle(ServletHandler.java:454)
	at org.eclipse.jetty.ee8.nested.ScopedHandler.handle(ScopedHandler.java:119)
	at org.eclipse.jetty.ee8.security.SecurityHandler.handle(SecurityHandler.java:478)
	at org.eclipse.jetty.ee8.nested.HandlerWrapper.handle(HandlerWrapper.java:108)
	at org.eclipse.jetty.ee8.nested.ScopedHandler.nextHandle(ScopedHandler.java:183)
	at org.eclipse.jetty.ee8.nested.SessionHandler.doHandle(SessionHandler.java:516)
	at org.eclipse.jetty.ee8.nested.ScopedHandler.nextHandle(ScopedHandler.java:181)
	at org.eclipse.jetty.ee8.nested.ContextHandler.doHandle(ContextHandler.java:878)
	at org.eclipse.jetty.ee8.nested.ScopedHandler.nextScope(ScopedHandler.java:152)
	at org.eclipse.jetty.ee8.servlet.ServletHandler.doScope(ServletHandler.java:423)
	at org.eclipse.jetty.ee8.nested.ScopedHandler.nextScope(ScopedHandler.java:150)
	at org.eclipse.jetty.ee8.nested.SessionHandler.doScope(SessionHandler.java:500)
	at org.eclipse.jetty.ee8.nested.ScopedHandler.nextScope(ScopedHandler.java:150)
	at org.eclipse.jetty.ee8.nested.ContextHandler.doScope(ContextHandler.java:823)
	at org.eclipse.jetty.ee8.nested.ScopedHandler.handle(ScopedHandler.java:117)
	at org.eclipse.jetty.ee8.nested.ContextHandler.handle(ContextHandler.java:1421)
	at org.eclipse.jetty.ee8.nested.HttpChannel$RequestDispatchable.dispatch(HttpChannel.java:1294)
	at org.eclipse.jetty.ee8.nested.HttpChannel.dispatch(HttpChannel.java:624)
	at org.eclipse.jetty.ee8.nested.HttpChannel.handle(HttpChannel.java:456)
	at org.eclipse.jetty.ee8.nested.ContextHandler$CoreContextHandler$CoreToNestedHandler.handle(ContextHandler.java:2365)
	at org.eclipse.jetty.server.handler.ContextHandler.handle(ContextHandler.java:1060)
	at org.eclipse.jetty.server.handler.gzip.GzipHandler.handle(GzipHandler.java:549)
	at org.eclipse.jetty.server.Server.handle(Server.java:181)
	at org.eclipse.jetty.server.internal.HttpChannelState$HandlerInvoker.run(HttpChannelState.java:648)
	at org.eclipse.jetty.server.internal.HttpConnection.onFillable(HttpConnection.java:403)
	at org.eclipse.jetty.io.AbstractConnection$ReadCallback.succeeded(AbstractConnection.java:322)
	at org.eclipse.jetty.io.FillInterest.fillable(FillInterest.java:99)
	at org.eclipse.jetty.io.SelectableChannelEndPoint$1.run(SelectableChannelEndPoint.java:53)
	at org.eclipse.jetty.util.thread.strategy.AdaptiveExecutionStrategy.runTask(AdaptiveExecutionStrategy.java:478)
	at org.eclipse.jetty.util.thread.strategy.AdaptiveExecutionStrategy.consumeTask(AdaptiveExecutionStrategy.java:441)
	at org.eclipse.jetty.util.thread.strategy.AdaptiveExecutionStrategy.tryProduce(AdaptiveExecutionStrategy.java:293)
	at org.eclipse.jetty.util.thread.strategy.AdaptiveExecutionStrategy.produce(AdaptiveExecutionStrategy.java:195)
	at org.eclipse.jetty.io.ManagedSelector$$Lambda$290/0x0000000100322550.run(Unknown Source)
	at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:979)
	at org.eclipse.jetty.util.thread.QueuedThreadPool$Runner.doRunJob(QueuedThreadPool.java:1209)
	at org.eclipse.jetty.util.thread.QueuedThreadPool$Runner.run(QueuedThreadPool.java:1164)
	at java.lang.Thread.run([email protected]/Unknown Source)
"GitSCM.onLoaded":
	at jdk.internal.misc.Unsafe.park([email protected]/Native Method)
	- parking to wait for  <0x000000008c1223d0> (a java.util.concurrent.locks.ReentrantLock$NonfairSync)
	at java.util.concurrent.locks.LockSupport.park([email protected]/Unknown Source)
	at java.util.concurrent.locks.AbstractQueuedSynchronizer.acquire([email protected]/Unknown Source)
	at java.util.concurrent.locks.AbstractQueuedSynchronizer.acquire([email protected]/Unknown Source)
	at java.util.concurrent.locks.ReentrantLock$Sync.lock([email protected]/Unknown Source)
	at java.util.concurrent.locks.ReentrantLock.lock([email protected]/Unknown Source)
	at com.google.inject.internal.CycleDetectingLock$CycleDetectingLockFactory$ReentrantCycleDetectingLock.lockOrDetectPotentialLocksCycle(CycleDetectingLock.java:170)
	at com.google.inject.internal.SingletonScope$1.get(SingletonScope.java:160)
	at hudson.ExtensionFinder$GuiceFinder$FaultTolerantScope$1.get(ExtensionFinder.java:448)
	at com.google.inject.internal.InternalFactoryToProviderAdapter.get(InternalFactoryToProviderAdapter.java:45)
	at com.google.inject.internal.InjectorImpl$1.get(InjectorImpl.java:1148)
	at hudson.ExtensionFinder$GuiceFinder._find(ExtensionFinder.java:406)
	at hudson.ExtensionFinder$GuiceFinder.find(ExtensionFinder.java:397)
	at hudson.ClassicPluginStrategy.findComponents(ClassicPluginStrategy.java:353)
	at hudson.ExtensionList.load(ExtensionList.java:384)
	at hudson.ExtensionList.ensureLoaded(ExtensionList.java:320)
	- locked <0x00000000898f9758> (a hudson.ExtensionList$Lock)
	at hudson.ExtensionList.iterator(ExtensionList.java:172)
	at jenkins.model.Jenkins.getDescriptorByType(Jenkins.java:1660)
	at hudson.plugins.git.GitSCM.onLoaded(GitSCM.java:2164)
	at jdk.internal.reflect.NativeMethodAccessorImpl.invoke0([email protected]/Native Method)
	at jdk.internal.reflect.NativeMethodAccessorImpl.invoke([email protected]/Unknown Source)
	at jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke([email protected]/Unknown Source)
	at java.lang.reflect.Method.invoke([email protected]/Unknown Source)
	at hudson.init.TaskMethodFinder.invoke(TaskMethodFinder.java:109)
	at hudson.init.TaskMethodFinder$TaskImpl.run(TaskMethodFinder.java:185)
	at org.jvnet.hudson.reactor.Reactor.runTask(Reactor.java:305)
	at jenkins.model.Jenkins$5.runTask(Jenkins.java:1176)
	at org.jvnet.hudson.reactor.Reactor$2.run(Reactor.java:221)
	at org.jvnet.hudson.reactor.Reactor$Node.run(Reactor.java:120)
	at jenkins.security.ImpersonatingExecutorService$1.run(ImpersonatingExecutorService.java:68)
	at java.util.concurrent.ThreadPoolExecutor.runWorker([email protected]/Unknown Source)
	at java.util.concurrent.ThreadPoolExecutor$Worker.run([email protected]/Unknown Source)
	at java.lang.Thread.run([email protected]/Unknown Source)
"DatadogGlobalConfiguration.onStartup":
	at hudson.ExtensionList.ensureLoaded(ExtensionList.java:319)
	- waiting to lock <0x00000000898f9758> (a hudson.ExtensionList$Lock)
	at hudson.ExtensionList.size(ExtensionList.java:194)
	at hudson.ExtensionList.lookupSingleton(ExtensionList.java:456)
	at hudson.diagnosis.OldDataMonitor.get(OldDataMonitor.java:93)
	at hudson.diagnosis.OldDataMonitor.report(OldDataMonitor.java:230)
	at hudson.util.RobustReflectionConverter.doUnmarshal(RobustReflectionConverter.java:401)
	at hudson.util.RobustReflectionConverter.unmarshal(RobustReflectionConverter.java:289)
	at com.thoughtworks.xstream.core.TreeUnmarshaller.convert(TreeUnmarshaller.java:74)
	at com.thoughtworks.xstream.core.AbstractReferenceUnmarshaller.convert(AbstractReferenceUnmarshaller.java:72)
	at com.thoughtworks.xstream.core.TreeUnmarshaller.convertAnother(TreeUnmarshaller.java:68)
	at com.thoughtworks.xstream.core.TreeUnmarshaller.convertAnother(TreeUnmarshaller.java:52)
	at com.thoughtworks.xstream.core.TreeUnmarshaller.start(TreeUnmarshaller.java:136)
	at com.thoughtworks.xstream.core.AbstractTreeMarshallingStrategy.unmarshal(AbstractTreeMarshallingStrategy.java:32)
	at com.thoughtworks.xstream.XStream.unmarshal(XStream.java:1464)
	at hudson.util.XStream2.unmarshal(XStream2.java:230)
	at hudson.util.XStream2.unmarshal(XStream2.java:201)
	at com.thoughtworks.xstream.XStream.unmarshal(XStream.java:1441)
	at hudson.XmlFile.unmarshal(XmlFile.java:196)
	at hudson.XmlFile.unmarshal(XmlFile.java:179)
	at hudson.model.Descriptor.load(Descriptor.java:937)
	- locked <0x000000008c11a568> (a org.datadog.jenkins.plugins.datadog.DatadogGlobalConfiguration)
	at org.datadog.jenkins.plugins.datadog.DatadogGlobalConfiguration.<init>(DatadogGlobalConfiguration.java:146)
	at org.datadog.jenkins.plugins.datadog.DatadogGlobalConfiguration$$FastClassByGuice$$3cd46085.GUICE$TRAMPOLINE(<generated>)
	at org.datadog.jenkins.plugins.datadog.DatadogGlobalConfiguration$$FastClassByGuice$$3cd46085.apply(<generated>)
	at com.google.inject.internal.DefaultConstructionProxyFactory$FastClassProxy.newInstance(DefaultConstructionProxyFactory.java:82)
	at com.google.inject.internal.ConstructorInjector.provision(ConstructorInjector.java:114)
	at com.google.inject.internal.ConstructorInjector.access$000(ConstructorInjector.java:33)
	at com.google.inject.internal.ConstructorInjector$1.call(ConstructorInjector.java:98)
	at com.google.inject.internal.ProvisionListenerStackCallback$Provision.provision(ProvisionListenerStackCallback.java:109)
	at hudson.ExtensionFinder$GuiceFinder$SezpozModule.onProvision(ExtensionFinder.java:613)
	at com.google.inject.internal.ProvisionListenerStackCallback$Provision.provision(ProvisionListenerStackCallback.java:117)
	at com.google.inject.internal.ProvisionListenerStackCallback.provision(ProvisionListenerStackCallback.java:66)
	at com.google.inject.internal.ConstructorInjector.construct(ConstructorInjector.java:93)
	at com.google.inject.internal.ConstructorBindingImpl$Factory.get(ConstructorBindingImpl.java:300)
	at com.google.inject.internal.ProviderToInternalFactoryAdapter.get(ProviderToInternalFactoryAdapter.java:40)
	at com.google.inject.internal.SingletonScope$1.get(SingletonScope.java:169)
	at hudson.ExtensionFinder$GuiceFinder$FaultTolerantScope$1.get(ExtensionFinder.java:448)
	at com.google.inject.internal.InternalFactoryToProviderAdapter.get(InternalFactoryToProviderAdapter.java:45)
	at com.google.inject.internal.InjectorImpl$1.get(InjectorImpl.java:1148)
	at com.google.inject.internal.InjectorImpl.getInstance(InjectorImpl.java:1186)
	at jenkins.ProxyInjector.getInstance(ProxyInjector.java:113)
	at hudson.init.TaskMethodFinder.lookUp(TaskMethodFinder.java:129)
	at hudson.init.TaskMethodFinder.invoke(TaskMethodFinder.java:110)
	at hudson.init.TaskMethodFinder$TaskImpl.run(TaskMethodFinder.java:185)
	at org.jvnet.hudson.reactor.Reactor.runTask(Reactor.java:305)
	at jenkins.model.Jenkins$5.runTask(Jenkins.java:1176)
	at org.jvnet.hudson.reactor.Reactor$2.run(Reactor.java:221)
	at org.jvnet.hudson.reactor.Reactor$Node.run(Reactor.java:120)
	at jenkins.security.ImpersonatingExecutorService$1.run(ImpersonatingExecutorService.java:68)
	at java.util.concurrent.ThreadPoolExecutor.runWorker([email protected]/Unknown Source)
	at java.util.concurrent.ThreadPoolExecutor$Worker.run([email protected]/Unknown Source)
	at java.lang.Thread.run([email protected]/Unknown Source)

Description of the Change

Alternate Designs

Possible Drawbacks

Verification Process

Additional Notes

Release Notes

Review checklist (to be filled by reviewers)

Feature or bug fix MUST have appropriate tests (unit, integration, etc...)
PR title must be written as a CHANGELOG entry (see why)
Files changes must correspond to the primary purpose of the PR as described in the title (small unrelated changes should have their own PR)
PR must have one changelog/ label attached. If applicable it should have the backward-incompatible label attached.
PR should not have do-not-merge/ label attached.
If Applicable, issue must have kind/ and severity/ labels attached at least.

drodriguezhdez

Nice one!

Fix deadlock when loading global configuration on startup

2b8eb28

nikita-tkachenko-datadog added the changelog/Fixed Fixed features results into a bug fix version bump label Sep 17, 2024

nikita-tkachenko-datadog marked this pull request as ready for review September 18, 2024 09:12

drodriguezhdez approved these changes Sep 18, 2024

View reviewed changes

nikita-tkachenko-datadog merged commit f8d01f9 into master Sep 18, 2024
19 checks passed

nikita-tkachenko-datadog deleted the nikita-tkachenko/fix-startup-deadlock branch September 18, 2024 12:23

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Fix deadlock when loading global configuration on startup #449

Fix deadlock when loading global configuration on startup #449

nikita-tkachenko-datadog commented Sep 17, 2024

drodriguezhdez left a comment

Fix deadlock when loading global configuration on startup #449

Fix deadlock when loading global configuration on startup #449

Conversation

nikita-tkachenko-datadog commented Sep 17, 2024

Requirements for Contributing to this repository

What does this PR do?

Description of the Change

Alternate Designs

Possible Drawbacks

Verification Process

Additional Notes

Release Notes

Review checklist (to be filled by reviewers)

drodriguezhdez left a comment

Choose a reason for hiding this comment