Finding the top cpu consumer threads with jconsole

1) Download this jar file


2) Open the jconsole using the plugin


ssunel@javabender:$ /usr/java/bin/jconsole -pluginpath /home/ssunel/Downloads/topthreads.jar

Finding top consumer Java Threads with Unix TOP command and JStack

1)ssunel@VTEKPSSUNEL:~$ top -c -b -H -n1 -d1 | head -n 20 | grep 'java\|CPU'

  PID USER      PR  NI  VIRT  RES  SHR S %CPU %MEM    TIME+  COMMAND                                                                                    
28763 ssunel    20   0 2893m 821m  10m R  100 10.3   0:38.97 /usr/java/bin/java
28518 ssunel    20   0 2893m 821m  10m S    2 10.3   0:00.30 /usr/java/bin/java
28667 ssunel    20   0 2893m 821m  10m S    2 10.3   0:00.07 /usr/java/bin/java
28733 ssunel    20   0 2893m 821m  10m S    2 10.3   0:00.11 /usr/java/bin/java 

2)Go to : here or use any other converter & convert decimal thread-id(s) [28763  -  28518  -  28667  -  28733 ] to hexedecimal.

Java thread identifiers will be something like that : 

pid    :  nid

28763  : 0x705b

28518 : 0x6f66

28667 : 0x6ffb

28733 : 0x703d

3)ssunel@VTEKPSSUNEL:~$ jstack <pid>

4) Search for the nativeId(s) in the thread dump : nid=0x705b or nid=0x6f66 or nid=0x6ffb or nid=0x703d

Example: 

"SocketListener" daemon prio=10 tid=0x0f474c00 nid=0x705b runnable [0x0386e000]
   java.lang.Thread.State: RUNNABLE
    at java.lang.Throwable.fillInStackTrace(Native Method)
    - locked <0x299364f4> (a java.nio.channels.NotYetBoundException)
    at java.lang.Throwable.<init>(Throwable.java:181)
    at java.lang.Exception.<init>(Exception.java:29)
    at java.lang.RuntimeException.<init>(RuntimeException.java:32)
    at java.lang.IllegalStateException.<init>(IllegalStateException.java:27)
    at java.nio.channels.NotYetBoundException.<init>(NotYetBoundException.java:30)
    at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:139)
    - locked <0x6567a198> (a java.lang.Object)
    at com.mycompany.smppgwsocket.smpp34server.Smpp34ServerSocketListener.run(Smpp34ServerSocketListener.java:08)

How to display JVM HEAP and PERMGEN using JMAP

ssunel@javabender:/space/bin$ jmap -heap 4028
Attaching to process ID 4028, please wait...
Debugger attached successfully.
Server compiler detected.
JVM version is 20.4-b02

using thread-local object allocation.
Parallel GC with 4 thread(s)

Heap Configuration:
   MinHeapFreeRatio = 40
   MaxHeapFreeRatio = 70
   MaxHeapSize      = 1073741824 (1024.0MB)
   NewSize          = 1048576 (1.0MB)
   MaxNewSize       = 4294901760 (4095.9375MB)
   OldSize          = 4194304 (4.0MB)
   NewRatio         = 2
   SurvivorRatio    = 8
   PermSize         = 16777216 (16.0MB)
   MaxPermSize      = 67108864 (64.0MB)

Heap Usage:
PS Young Generation
Eden Space:
   capacity = 31326208 (29.875MB)
   used     = 9114008 (8.691795349121094MB)
   free     = 22212200 (21.183204650878906MB)
   29.09387564559362% used
From Space:
   capacity = 10944512 (10.4375MB)
   used     = 10937848 (10.431144714355469MB)
   free     = 6664 (0.00635528564453125MB)
   99.93911103574102% used
To Space:
   capacity = 13828096 (13.1875MB)
   used     = 0 (0.0MB)
   free     = 13828096 (13.1875MB)
   0.0% used
PS Old Generation
   capacity = 54329344 (51.8125MB)
   used     = 30716608 (29.29364013671875MB)
   free     = 23612736 (22.51885986328125MB)
   56.5377855473462% used
PS Perm Generation
   capacity = 39321600 (37.5MB)
   used     = 22346712 (21.311485290527344MB)
   free     = 16974888 (16.188514709472656MB)
   56.83062744140625% used


ssunel@javabender:/space/$ jmap -permstat 4028
Attaching to process ID 4028, please wait...
Debugger attached successfully.
Server compiler detected.
JVM version is 20.4-b02
11313 intern Strings occupying 886336 bytes.
finding class loader instances ..Finding object size using Printezis bits and skipping over...
Finding object size using Printezis bits and skipping over...
Finding object size using Printezis bits and skipping over...
Finding object size using Printezis bits and skipping over...
Finding object size using Printezis bits and skipping over...
done.
computing per loader stat ..done.
please wait.. computing liveness......................................liveness analysis may be inaccurate ...
class_loader classes bytes parent_loader alive? type

<bootstrap> 2784 12492344  null   live <internal>
0xa1655e20 1 1784  null   dead sun/reflect/DelegatingClassLoader@0x6f7a9c30
0x73a52dc8 6 18072 0x7447ada0 live sun/reflect/misc/MethodUtil@0x6fce5a10
0xa1682da8 1 1184 0x7447ada0 dead sun/reflect/DelegatingClassLoader@0x6f7a9c30
0x74406470 1 1184 0x7447ada0 dead sun/reflect/DelegatingClassLoader@0x6f7a9c30
0xa16830d8 1 1184 0x7447ada0 dead sun/reflect/DelegatingClassLoader@0x6f7a9c30
0x73b660b8 1 1184 0x7447ada0 dead sun/reflect/DelegatingClassLoader@0x6f7a9c30
0x7440c5b8 1 1184 0x7447ada0 dead sun/reflect/DelegatingClassLoader@0x6f7a9c30
0x73b66160 1 1184 0x7447ada0 dead sun/reflect/DelegatingClassLoader@0x6f7a9c30
0x74406790 1 1184 0x7447ada0 dead sun/reflect/DelegatingClassLoader@0x6f7a9c30
0x73b66128 1 1216 0x73a52dc8 dead sun/reflect/DelegatingClassLoader@0x6f7a9c30
0x74b86370 1 1184  null   dead sun/reflect/DelegatingClassLoader@0x6f7a9c30
0x74405e30 1 1184 0x7447ada0 dead sun/reflect/DelegatingClassLoader@0x6f7a9c30
0xa15be7c0 1 1992  null   dead sun/reflect/DelegatingClassLoader@0x6f7a9c30
0x73b660f0 1 1184  null   dead sun/reflect/DelegatingClassLoader@0x6f7a9c30
0x73a52e60 1 1824  null   dead sun/reflect/DelegatingClassLoader@0x6f7a9c30
0x7447ade8 85 405928  null   live sun/misc/Launcher$ExtClassLoader@0x6f89e7b0
0x73a52e98 1 1992  null   dead sun/reflect/DelegatingClassLoader@0x6f7a9c30
0x7447ada0 2498 8514496 0x7447ade8 live sun/misc/Launcher$AppClassLoader@0x6f8ead78
0xa1683ad0 1 1808  null   dead sun/reflect/DelegatingClassLoader@0x6f7a9c30
0x73b8f2e0 0 0 0x7447ada0 dead java/util/ResourceBundle$RBClassLoader@0x6fa09738
0xa16834a8 1 1184  null   dead sun/reflect/DelegatingClassLoader@0x6f7a9c30
0x73a52e08 1 1784  null   dead sun/reflect/DelegatingClassLoader@0x6f7a9c30
0xa15beae0 1 2008  null   dead sun/reflect/DelegatingClassLoader@0x6f7a9c30
0x74406150 1 1184 0x7447ada0 dead sun/reflect/DelegatingClassLoader@0x6f7a9c30
0x74407ad0 1 1184 0x7447ada0 dead sun/reflect/DelegatingClassLoader@0x6f7a9c30
0xa16837c8 1 1184  null   dead sun/reflect/DelegatingClassLoader@0x6f7a9c30

total = 27 5395 21461824    N/A     alive=4, dead=23    N/A  

Attaching jdb debugger to a Core File or Hung Process

Sample Usage :jdb -connect sun.jvm.hotspot.jdi.SAPIDAttachingConnector:pid=20441

When this Connector attaches to a process it does so in read-only mode; that is the debugger can examine threads and the running application but it cannot change anything.

Alternative debuggers and descriptions

Connector : Description
SA Core Attaching Connector : This Connector can be used by a debugger to debug a core file.
SA PID Attaching Connector : This Connector can be used by a debugger to debug a process.
SA Debug Server Attaching Connector :  This Connector can be used by a debugger to debug a process or core file on a machine other than the machine upon which the debugger is running.

ssunel@javabender:/space$ jdb -connect sun.jvm.hotspot.jdi.SAPIDAttachingConnector:pid=20441
Initializing jdb...
> threads
Group system:
(java.lang.ref.Reference$ReferenceHandler)0x23 Reference Handler
unknown
(java.lang.ref.Finalizer$FinalizerThread)0x22
Finalizer unknown
(java.lang.Thread)0x21 Signal Dispatcher running
(java.lang.Thread)0x24 main running
[... more lines removed here to reduce output ...]
Group main:
(java.lang.Thread)0x1
TP-Monitor
unknown
(com.ecyrd.jspwiki.PageManager$LockReaper)0x0 Thread-33
sleeping
> thread 0x24
main[1] where
[1] java.net.PlainSocketImpl.socketAccept (native method)
[2] java.net.PlainSocketImpl.socketAccept (native method)
[3] java.net.PlainSocketImpl.accept (PlainSocketImpl.java:384)
[4] java.net.ServerSocket.implAccept (ServerSocket.java:450)
[5] java.net.ServerSocket.accept (ServerSocket.java:421)
[6] org.apache.catalina.core.StandardServer.await (StandardServer.java:552)
[7] org.apache.catalina.startup.Catalina.await (Catalina.java:634)
[8] org.apache.catalina.startup.Catalina.start (Catalina.java:596)
[9] sun.reflect.NativeMethodAccessorImpl.invoke0 (native method)
[10] sun.reflect.NativeMethodAccessorImpl.invoke
(NativeMethodAccessorImpl.java:39)
[11] sun.reflect.DelegatingMethodAccessorImpl.invoke
(DelegatingMethodAccessorImpl.java:25)
[12] java.lang.reflect.Method.invoke (Method.java:585)
[13] org.apache.catalina.startup.Bootstrap.start (Bootstrap.java:295)
[14] org.apache.catalina.startup.Bootstrap.main (Bootstrap.java:392)
main[1] up 13
main[14] where
[14] org.apache.catalina.startup.Bootstrap.main (Bootstrap.java:392)
47
main[14] locals
Method arguments:
args = instance of java.lang.String[1] (id=40)
Local variables:
command = "start"
> help
** command list **
connectors                -- list available connectors and transports in this VM

run [class [args]]        -- start execution of application's main class

threads [threadgroup]     -- list threads
thread <thread id>        -- set default thread
suspend [thread id(s)]    -- suspend threads (default: all)
resume [thread id(s)]     -- resume threads (default: all)
where [<thread id> | all] -- dump a thread's stack
wherei [<thread id> | all]-- dump a thread's stack, with pc info
up [n frames]             -- move up a thread's stack
down [n frames]           -- move down a thread's stack
kill <thread id> <expr>   -- kill a thread with the given exception object
interrupt <thread id>     -- interrupt a thread

print <expr>              -- print value of expression
dump <expr>               -- print all object information
eval <expr>               -- evaluate expression (same as print)
set <lvalue> = <expr>     -- assign new value to field/variable/array element
locals                    -- print all local variables in current stack frame

classes                   -- list currently known classes
class <class id>          -- show details of named class
methods <class id>        -- list a class's methods
fields <class id>         -- list a class's fields

threadgroups              -- list threadgroups
threadgroup <name>        -- set current threadgroup

stop in <class id>.<method>[(argument_type,...)]
                          -- set a breakpoint in a method
stop at <class id>:<line> -- set a breakpoint at a line
clear <class id>.<method>[(argument_type,...)]
                          -- clear a breakpoint in a method
clear <class id>:<line>   -- clear a breakpoint at a line
clear                     -- list breakpoints
catch [uncaught|caught|all] <class id>|<class pattern>
                          -- break when specified exception occurs
ignore [uncaught|caught|all] <class id>|<class pattern>
                          -- cancel 'catch' for the specified exception
watch [access|all] <class id>.<field name>
                          -- watch access/modifications to a field
unwatch [access|all] <class id>.<field name>
                          -- discontinue watching access/modifications to a field
trace [go] methods [thread]
                          -- trace method entries and exits.
                          -- All threads are suspended unless 'go' is specified
trace [go] method exit | exits [thread]
                          -- trace the current method's exit, or all methods' exits
                          -- All threads are suspended unless 'go' is specified
untrace [methods]         -- stop tracing method entrys and/or exits
step                      -- execute current line
step up                   -- execute until the current method returns to its caller
stepi                     -- execute current instruction
next                      -- step one line (step OVER calls)
cont                      -- continue execution from breakpoint

list [line number|method] -- print source code
use (or sourcepath) [source file path]
                          -- display or change the source path
exclude [<class pattern>, ... | "none"]
                          -- do not report step or method events for specified classes
classpath                 -- print classpath info from target VM

monitor <command>         -- execute command each time the program stops
monitor                   -- list monitors
unmonitor <monitor#>      -- delete a monitor
read <filename>           -- read and execute a command file

lock <expr>               -- print lock info for an object
threadlocks [thread id]   -- print lock info for a thread

pop                       -- pop the stack through and including the current frame
reenter                   -- same as pop, but current frame is reentered
redefine <class id> <class file name>
                          -- redefine the code for a class

disablegc <expr>          -- prevent garbage collection of an object
enablegc <expr>           -- permit garbage collection of an object

!!                        -- repeat last command
<n> <command>             -- repeat command n times
# <command>               -- discard (no-op)
help (or ?)               -- list commands
version                   -- print version information
exit (or quit)            -- exit debugger

<class id>: a full class name with package qualifiers
<class pattern>: a class name with a leading or trailing wildcard ('*')
<thread id>: thread number as reported in the 'threads' command
<expr>: a Java(tm) Programming Language expression.
Most common syntax is supported.




Attaching a Core File 

Usage : jdb -connect sun.jvm.hotspot.jdi.SACoreAttachingConnector:javaExecutable=$JAVA_HOME/bin/java,core=core.20441

Attaching to a Remote Machine

Usage  :jdb -connect sun.jvm.hotspot.jdi.SADebugServerAttachingConnector:debugServerName=machine1