昨天晚上,一台服务器(JBoss 5.1GA,Java(TM)SE Runtime Environment(build 1.6.0_20-b02))
在VMWare的 Linux VM中运行的Java HotSpot(TM)64位服务器虚拟机)突然开始使用100%的cpu.该应用程序是一个运行Seam的相当典型的J2EE业务应用程序,没有什么特别的.当时的负载极低.
在VMWare的 Linux VM中运行的Java HotSpot(TM)64位服务器虚拟机)突然开始使用100%的cpu.该应用程序是一个运行Seam的相当典型的J2EE业务应用程序,没有什么特别的.当时的负载极低.
我设法得到一个线程转储,在进程被杀死并重新启动之前(应用程序必须可用). JStack在其死锁检测中有例外;除了大多数线程都被阻塞.
这是JVM的错误?我以前没看过.重新启动后,一切正常,就像在应用程序在该服务器上使用的几个月(定期更新)一样.
感谢任何建议.
稍微编辑的线程转储(适合SO的职位大小限制)在下面.我删除了几个相同的线程(从JBoss / Seam内的各种线程池),只剩下堆栈跟踪的顶线.
编辑:全线程转储在http://pastie.org/1083984
[~]$jstack -F 13553 Attaching to process ID 13553,please wait... Debugger attached successfully. Server compiler detected. JVM version is 11.3-b02 Deadlock Detection: java.lang.NullPointerException at sun.jvm.hotspot.oops.InstanceKlass.computeSubtypeOf(InstanceKlass.java:426) at sun.jvm.hotspot.oops.Klass.isSubtypeOf(Klass.java:137) at sun.jvm.hotspot.oops.Oop.isA(Oop.java:92) at sun.jvm.hotspot.runtime.DeadlockDetector.print(DeadlockDetector.java:93) at sun.jvm.hotspot.runtime.DeadlockDetector.print(DeadlockDetector.java:39) at sun.jvm.hotspot.tools.StackTrace.run(StackTrace.java:52) at sun.jvm.hotspot.tools.StackTrace.run(StackTrace.java:45) at sun.jvm.hotspot.tools.JStack.run(JStack.java:60) at sun.jvm.hotspot.tools.Tool.start(Tool.java:221) at sun.jvm.hotspot.tools.JStack.main(JStack.java:86) at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25) at java.lang.reflect.Method.invoke(Method.java:597) at sun.tools.jstack.JStack.runJStackTool(JStack.java:118) at sun.tools.jstack.JStack.main(JStack.java:84) Can't print deadlocks:null Thread 26208: (state = BLOCKED) Thread 25250: (state = BLOCKED) - sun.misc.Unsafe.park(boolean,long) @bci=0 (Interpreted frame) - java.util.concurrent.locks.LockSupport.park(java.lang.Object) @bci=14,line=158 (Interpreted frame) Thread 25249: (state = BLOCKED) - sun.misc.Unsafe.park(boolean,line=158 (Interpreted frame) Thread 21240: (state = BLOCKED) - java.lang.Object.wait(long) @bci=0 (Interpreted frame) - java.lang.Object.wait() @bci=2,line=485 (Interpreted frame) - org.apache.tomcat.util.net.JIoEndpoint$Worker.await() @bci=8,line=416 (Interpreted frame) Thread 21140: (state = BLOCKED) - java.lang.Object.wait(long) @bci=0 (Interpreted frame) - java.lang.Object.wait() @bci=2,line=416 (Interpreted frame) Thread 21139: (state = BLOCKED) - java.lang.Object.wait(long) @bci=0 (Interpreted frame) - java.lang.Object.wait() @bci=2,line=416 (Interpreted frame) Thread 21138: (state = BLOCKED) - java.lang.Object.wait(long) @bci=0 (Interpreted frame) - java.lang.Object.wait() @bci=2,line=416 (Interpreted frame) Thread 19380: (state = BLOCKED) - sun.misc.Unsafe.park(boolean,line=158 (Interpreted frame) Thread 19377: (state = BLOCKED) - sun.misc.Unsafe.park(boolean,line=158 (Interpreted frame) Thread 19361: (state = BLOCKED) - sun.misc.Unsafe.park(boolean,long) @bci=0 (Interpreted frame) Thread 19343: (state = BLOCKED) - sun.misc.Unsafe.park(boolean,line=158 (Interpreted frame) Thread 19317: (state = BLOCKED) - sun.misc.Unsafe.park(boolean,line=158 (Interpreted frame) Thread 18995: (state = BLOCKED) - java.lang.Object.wait(long) @bci=0 (Interpreted frame) - java.lang.Object.wait() @bci=2,line=485 (Interpreted frame) Thread 18986: (state = BLOCKED) - java.lang.Object.wait(long) @bci=0 (Interpreted frame) - java.lang.Object.wait() @bci=2,line=485 (Interpreted frame) Thread 17659: (state = BLOCKED) - sun.misc.Unsafe.park(boolean,line=158 (Interpreted frame) Thread 17658: (state = BLOCKED) - sun.misc.Unsafe.park(boolean,line=158 (Interpreted frame) Thread 17653: (state = BLOCKED) - sun.misc.Unsafe.park(boolean,line=158 (Interpreted frame) Thread 17631: (state = BLOCKED) - sun.misc.Unsafe.park(boolean,line=158 (Interpreted frame) Thread 14977: (state = BLOCKED) - sun.misc.Unsafe.park(boolean,line=158 (Interpreted frame) Thread 14796: (state = BLOCKED) - sun.misc.Unsafe.park(boolean,line=158 (Interpreted frame) Thread 14609: (state = BLOCKED) - sun.misc.Unsafe.park(boolean,line=158 (Interpreted frame) Thread 14437: (state = BLOCKED) - sun.misc.Unsafe.park(boolean,line=158 (Interpreted frame) Thread 14248: (state = BLOCKED) - sun.misc.Unsafe.park(boolean,line=158 (Interpreted frame) Thread 14232: (state = BLOCKED) - java.util.Arrays.copyOf(char[],int) @bci=1,line=2882 (Compiled frame) - java.lang.AbstractStringBuilder.expandCapacity(int) @bci=33,line=100 (Compiled frame) Thread 14040: (state = IN_NATIVE) - java.net.PlainSocketImpl.socketAccept(java.net.SocketImpl) @bci=0 (Interpreted frame) - java.net.PlainSocketImpl.accept(java.net.SocketImpl) @bci=7,line=384 (Interpreted frame) Thread 13858: (state = BLOCKED) - java.lang.Object.wait(long) @bci=0 (Interpreted frame) - java.lang.ref.ReferenceQueue.remove(long) @bci=44,line=116 (Compiled frame) Thread 13815: (state = BLOCKED) - java.lang.Object.wait(long) @bci=0 (Interpreted frame) - java.lang.Object.wait() @bci=2,line=416 (Interpreted frame) Thread 13814: (state = IN_NATIVE) - java.net.PlainSocketImpl.socketAccept(java.net.SocketImpl) @bci=0 (Interpreted frame) Thread 13813: (state = BLOCKED) - java.lang.Object.wait(long) @bci=0 (Interpreted frame) - java.lang.Object.wait() @bci=2,line=416 (Interpreted frame) Thread 13812: (state = BLOCKED) Error occurred during stack walking: sun.jvm.hotspot.utilities.AssertionFailure: range check at sun.jvm.hotspot.utilities.Assert.that(Assert.java:32) at sun.jvm.hotspot.runtime.RegisterMap.<init>(RegisterMap.java:109) at sun.jvm.hotspot.runtime.x86.X86RegisterMap.<init>(X86RegisterMap.java:39) at sun.jvm.hotspot.runtime.x86.X86RegisterMap.clone(X86RegisterMap.java:43) at sun.jvm.hotspot.runtime.VFrame.<init>(VFrame.java:37) at sun.jvm.hotspot.runtime.JavaVFrame.<init>(JavaVFrame.java:45) at sun.jvm.hotspot.runtime.CompiledVFrame.<init>(CompiledVFrame.java:43) at sun.jvm.hotspot.runtime.VFrame.newVFrame(VFrame.java:77) at sun.jvm.hotspot.runtime.VFrame.sender(VFrame.java:129) at sun.jvm.hotspot.runtime.VFrame.javaSender(VFrame.java:146) at sun.jvm.hotspot.runtime.JavaThread.getLastJavaVFrameDbg(JavaThread.java:231) at sun.jvm.hotspot.tools.StackTrace.run(StackTrace.java:76) at sun.jvm.hotspot.tools.StackTrace.run(StackTrace.java:45) at sun.jvm.hotspot.tools.JStack.run(JStack.java:60) at sun.jvm.hotspot.tools.Tool.start(Tool.java:221) at sun.jvm.hotspot.tools.JStack.main(JStack.java:86) at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25) at java.lang.reflect.Method.invoke(Method.java:597) at sun.tools.jstack.JStack.runJStackTool(JStack.java:118) at sun.tools.jstack.JStack.main(JStack.java:84) Thread 13811: (state = IN_NATIVE) - java.net.PlainSocketImpl.socketAccept(java.net.SocketImpl) @bci=0 (Interpreted frame) Thread 13810: (state = IN_NATIVE) - java.net.PlainSocketImpl.socketAccept(java.net.SocketImpl) @bci=0 (Interpreted frame) Thread 13809: (state = BLOCKED) - java.lang.Object.wait(long) @bci=0 (Interpreted frame) - java.lang.Object.wait() @bci=2,line=485 (Interpreted frame) - org.jboss.jms.client.container.ClientConsumer.getMessage(long) @bci=71,line=866 (Interpreted frame) Thread 13808: (state = BLOCKED) - java.lang.Object.wait(long) @bci=0 (Interpreted frame) - java.lang.Object.wait() @bci=2,line=866 (Interpreted frame) Thread 13807: (state = BLOCKED) - java.lang.Object.wait(long) @bci=0 (Interpreted frame) - java.lang.Object.wait() @bci=2,line=485 (Interpreted frame) - org.jboss.bootstrap.AbstractServerImpl$LifeThread.run() @bci=11,line=866 (Interpreted frame) Thread 13806: (state = BLOCKED) - sun.misc.Unsafe.park(boolean,line=158 (Interpreted frame) Thread 13805: (state = BLOCKED) - sun.misc.Unsafe.park(boolean,line=158 (Interpreted frame) Thread 13804: (state = BLOCKED) - java.lang.Object.wait(long) @bci=0 (Interpreted frame) - java.util.TimerThread.mainLoop() @bci=201,line=509 (Interpreted frame) Thread 13803: (state = BLOCKED) - java.lang.Object.wait(long) @bci=0 (Interpreted frame) - java.util.TimerThread.mainLoop() @bci=201,line=509 (Interpreted frame) Thread 13802: (state = BLOCKED) - java.util.Arrays.copyOfRange(char[],int,int) @bci=40,line=3209 (Compiled frame) - java.lang.String.<init>(char[],int) @bci=65,line=216 (Compiled frame) - java.lang.StringBuffer.toString() @bci=13,line=585 (Compiled frame) - org.apache.log4j.PatternLayout.format(org.apache.log4j.spi.LoggingEvent) @bci=68,line=506 (Compiled frame) Thread 13784: (state = BLOCKED) - java.lang.Object.wait(long) @bci=0 (Interpreted frame) - java.lang.ref.ReferenceQueue.remove(long) @bci=44,line=116 (Compiled frame) - org.jboss.el.util.ReferenceCache$ReferenceQueueRunner.run() @bci=1,line=159 (Interpreted frame) Thread 13753: (state = BLOCKED) - java.lang.Thread.sleep(long) @bci=0 (Compiled frame; information may be imprecise) - org.jboss.ejb3.cache.simple.SimpleStatefulCache$SessionTimeoutTask.block() @bci=11,line=210 (Interpreted frame) Thread 13752: (state = BLOCKED) - java.lang.Object.wait(long) @bci=0 (Interpreted frame) - org.jboss.resource.connectionmanager.IdleRemover$IdleRemoverRunnable.run() @bci=31,line=167 (Interpreted frame) Thread 13749: (state = BLOCKED) - java.lang.Object.wait(long) @bci=0 (Interpreted frame) - java.lang.Object.wait() @bci=2,line=485 (Interpreted frame) - org.jboss.remoting.transport.socket.ServerThread.run() @bci=209,line=284 (Interpreted frame) Thread 13734: (state = BLOCKED) - java.lang.Object.wait(long) @bci=0 (Interpreted frame) - java.util.TimerThread.mainLoop() @bci=201,line=509 (Interpreted frame) Thread 13701: (state = BLOCKED) - sun.misc.Unsafe.park(boolean,line=158 (Interpreted frame) Thread 13700: (state = BLOCKED) - java.lang.Thread.sleep(long) @bci=0 (Compiled frame; information may be imprecise) - org.jboss.console.plugins.AOPLister$RefreshPoller.run() @bci=41,line=898 (Interpreted frame) Thread 13699: (state = BLOCKED) - java.lang.Thread.sleep(long) @bci=0 (Compiled frame; information may be imprecise) - org.apache.catalina.core.ContainerBase$ContainerBackgroundProcessor.run() @bci=22,line=1590 (Interpreted frame) - java.lang.Thread.run() @bci=11,line=619 (Interpreted frame) Thread 13698: (state = IN_NATIVE) - java.net.PlainSocketImpl.socketAccept(java.net.SocketImpl) @bci=0 (Interpreted frame) Thread 13678: (state = BLOCKED) - sun.misc.Unsafe.park(boolean,long) @bci=0 (Interpreted frame) - java.util.concurrent.locks.LockSupport.parkNanos(java.lang.Object,long) @bci=20,line=198 (Interpreted frame) Thread 13554: (state = BLOCKED) Thread 13560: (state = BLOCKED) Thread 13559: (state = BLOCKED) - java.lang.Object.wait(long) @bci=0 (Interpreted frame) - java.lang.ref.ReferenceQueue.remove(long) @bci=44,line=116 (Compiled frame) - java.lang.ref.ReferenceQueue.remove() @bci=2,line=132 (Compiled frame) - java.lang.ref.Finalizer$FinalizerThread.run() @bci=3,line=159 (Compiled frame) Thread 13558: (state = BLOCKED) - java.lang.Object.wait(long) @bci=0 (Interpreted frame) - java.lang.Object.wait() @bci=2,line=485 (Compiled frame) - java.lang.ref.Reference$ReferenceHandler.run() @bci=46,line=116 (Compiled frame)