Attaching jdb debugger to a Core File or Hung Process

Sample Usage :jdb -connect sun.jvm.hotspot.jdi.SAPIDAttachingConnector:pid=20441

When this Connector attaches to a process it does so in read-only mode; that is the debugger can examine threads and the running application but it cannot change anything.

Alternative debuggers and descriptions

Connector : Description
SA Core Attaching Connector : This Connector can be used by a debugger to debug a core file.
SA PID Attaching Connector : This Connector can be used by a debugger to debug a process.
SA Debug Server Attaching Connector :  This Connector can be used by a debugger to debug a process or core file on a machine other than the machine upon which the debugger is running.

ssunel@javabender:/space$ jdb -connect sun.jvm.hotspot.jdi.SAPIDAttachingConnector:pid=20441
Initializing jdb...
> threads
Group system:
(java.lang.ref.Reference$ReferenceHandler)0x23 Reference Handler
unknown
(java.lang.ref.Finalizer$FinalizerThread)0x22
Finalizer unknown
(java.lang.Thread)0x21 Signal Dispatcher running
(java.lang.Thread)0x24 main running
[... more lines removed here to reduce output ...]
Group main:
(java.lang.Thread)0x1
TP-Monitor
unknown
(com.ecyrd.jspwiki.PageManager$LockReaper)0x0 Thread-33
sleeping
> thread 0x24
main[1] where
[1] java.net.PlainSocketImpl.socketAccept (native method)
[2] java.net.PlainSocketImpl.socketAccept (native method)
[3] java.net.PlainSocketImpl.accept (PlainSocketImpl.java:384)
[4] java.net.ServerSocket.implAccept (ServerSocket.java:450)
[5] java.net.ServerSocket.accept (ServerSocket.java:421)
[6] org.apache.catalina.core.StandardServer.await (StandardServer.java:552)
[7] org.apache.catalina.startup.Catalina.await (Catalina.java:634)
[8] org.apache.catalina.startup.Catalina.start (Catalina.java:596)
[9] sun.reflect.NativeMethodAccessorImpl.invoke0 (native method)
[10] sun.reflect.NativeMethodAccessorImpl.invoke
(NativeMethodAccessorImpl.java:39)
[11] sun.reflect.DelegatingMethodAccessorImpl.invoke
(DelegatingMethodAccessorImpl.java:25)
[12] java.lang.reflect.Method.invoke (Method.java:585)
[13] org.apache.catalina.startup.Bootstrap.start (Bootstrap.java:295)
[14] org.apache.catalina.startup.Bootstrap.main (Bootstrap.java:392)
main[1] up 13
main[14] where
[14] org.apache.catalina.startup.Bootstrap.main (Bootstrap.java:392)
47
main[14] locals
Method arguments:
args = instance of java.lang.String[1] (id=40)
Local variables:
command = "start"
> help
** command list **
connectors                -- list available connectors and transports in this VM

run [class [args]]        -- start execution of application's main class

threads [threadgroup]     -- list threads
thread <thread id>        -- set default thread
suspend [thread id(s)]    -- suspend threads (default: all)
resume [thread id(s)]     -- resume threads (default: all)
where [<thread id> | all] -- dump a thread's stack
wherei [<thread id> | all]-- dump a thread's stack, with pc info
up [n frames]             -- move up a thread's stack
down [n frames]           -- move down a thread's stack
kill <thread id> <expr>   -- kill a thread with the given exception object
interrupt <thread id>     -- interrupt a thread

print <expr>              -- print value of expression
dump <expr>               -- print all object information
eval <expr>               -- evaluate expression (same as print)
set <lvalue> = <expr>     -- assign new value to field/variable/array element
locals                    -- print all local variables in current stack frame

classes                   -- list currently known classes
class <class id>          -- show details of named class
methods <class id>        -- list a class's methods
fields <class id>         -- list a class's fields

threadgroups              -- list threadgroups
threadgroup <name>        -- set current threadgroup

stop in <class id>.<method>[(argument_type,...)]
                          -- set a breakpoint in a method
stop at <class id>:<line> -- set a breakpoint at a line
clear <class id>.<method>[(argument_type,...)]
                          -- clear a breakpoint in a method
clear <class id>:<line>   -- clear a breakpoint at a line
clear                     -- list breakpoints
catch [uncaught|caught|all] <class id>|<class pattern>
                          -- break when specified exception occurs
ignore [uncaught|caught|all] <class id>|<class pattern>
                          -- cancel 'catch' for the specified exception
watch [access|all] <class id>.<field name>
                          -- watch access/modifications to a field
unwatch [access|all] <class id>.<field name>
                          -- discontinue watching access/modifications to a field
trace [go] methods [thread]
                          -- trace method entries and exits.
                          -- All threads are suspended unless 'go' is specified
trace [go] method exit | exits [thread]
                          -- trace the current method's exit, or all methods' exits
                          -- All threads are suspended unless 'go' is specified
untrace [methods]         -- stop tracing method entrys and/or exits
step                      -- execute current line
step up                   -- execute until the current method returns to its caller
stepi                     -- execute current instruction
next                      -- step one line (step OVER calls)
cont                      -- continue execution from breakpoint

list [line number|method] -- print source code
use (or sourcepath) [source file path]
                          -- display or change the source path
exclude [<class pattern>, ... | "none"]
                          -- do not report step or method events for specified classes
classpath                 -- print classpath info from target VM

monitor <command>         -- execute command each time the program stops
monitor                   -- list monitors
unmonitor <monitor#>      -- delete a monitor
read <filename>           -- read and execute a command file

lock <expr>               -- print lock info for an object
threadlocks [thread id]   -- print lock info for a thread

pop                       -- pop the stack through and including the current frame
reenter                   -- same as pop, but current frame is reentered
redefine <class id> <class file name>
                          -- redefine the code for a class

disablegc <expr>          -- prevent garbage collection of an object
enablegc <expr>           -- permit garbage collection of an object

!!                        -- repeat last command
<n> <command>             -- repeat command n times
# <command>               -- discard (no-op)
help (or ?)               -- list commands
version                   -- print version information
exit (or quit)            -- exit debugger

<class id>: a full class name with package qualifiers
<class pattern>: a class name with a leading or trailing wildcard ('*')
<thread id>: thread number as reported in the 'threads' command
<expr>: a Java(tm) Programming Language expression.
Most common syntax is supported.




Attaching a Core File 

Usage : jdb -connect sun.jvm.hotspot.jdi.SACoreAttachingConnector:javaExecutable=$JAVA_HOME/bin/java,core=core.20441

Attaching to a Remote Machine

Usage  :jdb -connect sun.jvm.hotspot.jdi.SADebugServerAttachingConnector:debugServerName=machine1

Jgroups LockService Example

import org.jgroups.*;
import org.jgroups.blocks.locking.LockService;

import java.util.concurrent.atomic.AtomicBoolean;
import java.util.concurrent.locks.Lock;

public class TheOne {

    private volatile AtomicBoolean becomeMaster = new AtomicBoolean (false);
    private JChannel channel;
    private LockService lock_service;
    private Thread acquiringThread;
    private Thread statusPrinter;

    public static void main(String[] args) throws Exception {
        Thread.currentThread().setName("MyMainThread");
        TheOne master = new TheOne();
        master.channel = new JChannel("/home/ssunel/udp.xml");
        master.channel.connect("lock-cluster");
        master.lock_service = new LockService(master.channel);
        master.startAcquiringThread();
        master.startStatusPrinterThread();
    }

    public void startAcquiringThread() {
        acquiringThread = new Thread() {
            @Override
            public void run() {
                try {
                    Thread.currentThread().setName("MyAcquiringThread");                   
                    getLock();
                } catch (Exception e) {
                    e.printStackTrace();
                }
            }
        };
        acquiringThread.setDaemon(true);
        acquiringThread.start();
       
    }

    private void getLock() throws Exception{
        Lock lock = lock_service.getLock("mylock");
        lock.lock();
        becomeMaster.set(true);
        System.out.println(" -- I GOT THE LOCK !!!");
    }

    private void startStatusPrinterThread() {
        statusPrinter = new Thread(){
            @Override
            public void run() {
                Thread.currentThread().setName("MyStatusPrinterThread");                   
               
                while (true) {
                    try {
                        System.out.println("becomeMaster ? -> { " + becomeMaster + " }");
                        System.out.println("thread state ? -> { " + acquiringThread.getState() + " }");
                        sleep(2000L);
                    } catch (InterruptedException e) {
                        e.printStackTrace();
                    }
                }
            }
        };
        statusPrinter.setDaemon(true);
        statusPrinter.start();
    }
}

Content of UDP.XML
open the jgroups.3.1-final.jar and edit the udp.xml inside it .
Add CENTRAL_LOCK tag 

Note : DistributedLockManager is Depracated in jgroups.3.1-final release .For older release example refer to http://javabender.blogspot.com/2010/10/solving-leader-election-problem-in.html


udp.xml content will be something like this :

<!--
  Default stack using IP multicasting. It is similar to the "udp"
  stack in stacks.xml, but doesn't use streaming state transfer and flushing
  author: Bela Ban
-->

<config xmlns="urn:org:jgroups"
        xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
        xsi:schemaLocation="urn:org:jgroups http://www.jgroups.org/schema/JGroups-3.0.xsd">
    <UDP
        mcast_port="${jgroups.udp.mcast_port:45588}"
         tos="8"
         ucast_recv_buf_size="20M"
         ucast_send_buf_size="640K"
         mcast_recv_buf_size="25M"
         mcast_send_buf_size="640K"
         loopback="true"
         level="WARN"
         log_discard_msgs="false"
         discard_incompatible_packets="true"
         max_bundle_size="64K"
         max_bundle_timeout="30"
         ip_ttl="${jgroups.udp.ip_ttl:8}"
         enable_bundling="true"
         enable_diagnostics="true"
         thread_naming_pattern="cl"

         timer_type="new"
         timer.min_threads="4"
         timer.max_threads="10"
         timer.keep_alive_time="3000"
         timer.queue_max_size="500"

         thread_pool.enabled="true"
         thread_pool.min_threads="2"
         thread_pool.max_threads="8"
         thread_pool.keep_alive_time="5000"
         thread_pool.queue_enabled="true"
         thread_pool.queue_max_size="10000"
         thread_pool.rejection_policy="discard"

         oob_thread_pool.enabled="true"
         oob_thread_pool.min_threads="1"
         oob_thread_pool.max_threads="8"
         oob_thread_pool.keep_alive_time="5000"
         oob_thread_pool.queue_enabled="false"
         oob_thread_pool.queue_max_size="100"
         oob_thread_pool.rejection_policy="Run"/>

    <PING timeout="2000"
            num_initial_members="3"/>
    <MERGE2 max_interval="30000"
            min_interval="10000"/>
    <FD_SOCK/>
    <FD_ALL/>
    <VERIFY_SUSPECT timeout="1500"  />
    <BARRIER />
    <pbcast.NAKACK exponential_backoff="300"
                   xmit_stagger_timeout="200"
                   use_mcast_xmit="false"
                   discard_delivered_msgs="true"/>
    <UNICAST />
    <pbcast.STABLE stability_delay="1000" desired_avg_gossip="50000"
                   max_bytes="4M"/>
    <pbcast.GMS print_local_addr="true" join_timeout="3000"
                view_bundling="true"/>
    <UFC max_credits="2M"
         min_threshold="0.4"/>
    <MFC max_credits="2M"
         min_threshold="0.4"/>
    <FRAG2 frag_size="60K"  />
    <pbcast.STATE_TRANSFER />
    <CENTRAL_LOCK />
    <!-- pbcast.FLUSH  /-->
</config>

How to @Consume MediaType.APPLICATION_JSON with Jersey

Sit Back And get REST...

 Blahh Blahh Blahh and the Code !







    @GET
    @Path(value = "/json")
    @Produces(MediaType.APPLICATION_JSON)
    public JSON getJSON() {
        JSON json = new JSON();
        json.setName("json");
        json.setSurname("gson");
        json.setAge(32);
        json.setHasCar(true);
        json.setMarried(true);
        json.setSalary(123L);

        List childs = new ArrayList();
        childs.add("serkan");
        childs.add("volkan");
        childs.add("aybars");
        json.setChilds(childs);

        return json;
    }

    @PUT
    @Path(value = "/pson")
    @Consumes( { MediaType.APPLICATION_JSON })   
    @Produces( { MediaType.TEXT_PLAIN })   
    public String consumeJSONObject (JSON  pson) {
        System.out.println("Consumed json object is : " + pson);
        return pson.toString();
    }



import java.util.List;

import javax.xml.bind.annotation.XmlRootElement;

@XmlRootElement
public class JSON {

    private String name;
    private String surname;
    private Integer age;
    private long salary;
    private boolean isMarried;
    private boolean hasCar;
    private List childs;
    
    public JSON() {
    }

    public String getName() {
        return name;
    }

    public void setName(String name) {
        this.name = name;
    }

    public String getSurname() {
        return surname;
    }

    public void setSurname(String surname) {
        this.surname = surname;
    }

    public Integer getAge() {
        return age;
    }

    public void setAge(Integer age) {
        this.age = age;
    }

    public long getSalary() {
        return salary;
    }

    public void setSalary(long salary) {
        this.salary = salary;
    }

    public boolean isMarried() {
        return isMarried;
    }

    public void setMarried(boolean isMarried) {
        this.isMarried = isMarried;
    }

    public boolean isHasCar() {
        return hasCar;
    }

    public void setHasCar(boolean hasCar) {
        this.hasCar = hasCar;
    }

    public List getChilds() {
        return childs;
    }

    public void setChilds(List childs) {
        this.childs = childs;
    }

    @Override
    public String toString() {
        return "JSON [name=" + name + ", surname=" + surname + ", age=" + age
                + ", salary=" + salary + ", isMarried=" + isMarried
                + ", hasCar=" + hasCar + ", childs=" + childs + "]";
    }
    
} 

Json data in soapui : {"name":"json","surname":"gson","married":true,"age":32,"salary":123,"hasCar":true,"childs":["serkan","volkan","aybars"]}

note : Using Jersey 1.8 ! (really important)

Line number information for SLF4J and LOG4J log messages

Briefly;

Line number information should be put into the log messages by processing classes and enhancing the byte code.


#Ant task definition and usage can be like below. Feel free to customize according to your requirements


<taskdefname="enhanceLogs"classname="your.lovely.company.LoggerEnhancement"classpathref="lib.ant.path"/>

<enhanceLogs classesDir="${classesDir}"loggerJarsDirectory="${logging.jars.dir}"/>

import java.io.File;
import java.io.FileNotFoundException;
import java.util.ArrayList;
import java.util.Arrays;
import java.util.Collections;
import java.util.List;

import javassist.CannotCompileException;
import javassist.ClassPool;
import javassist.CtClass;
import javassist.expr.ExprEditor;
import javassist.expr.MethodCall;

import org.apache.tools.ant.BuildException;
import org.apache.tools.ant.Task;

public class LoggerEnhancement extends Task {
    public String classesDir;
    public String loggerJarsDirectory;

    public String getLoggerJarsDirectory() {
        returnthis.loggerJarsDirectory;
    }

    public voidsetLoggerJarsDirectory(final String loggerJarsDirectory) {
        this.loggerJarsDirectory= loggerJarsDirectory;
    }

    public String getClassesDir() {
        returnthis.classesDir;
    }

    public voidsetClassesDir(final String classesDir) {
        this.classesDir= classesDir;
    }

    @Override
    public voidexecute() throws BuildException {
        try{
            this.enhanceClasses();
        } catch(final Exception e) {
            e.printStackTrace();
            throw new BuildException(e);
        }
    }

    private voidenhanceClasses() throws FileNotFoundException, Exception {
        finalList classFiles = this.getClassFiles(this.classesDir);
        for(final File file : classFiles) {
            String qualifiedClassName = file.toString().substring(this.classesDir.length() + 1, file.toString().length() - 6);
            qualifiedClassName = qualifiedClassName.replaceAll("\\\\", ".");
            this.enhanceClassFile(qualifiedClassName);
        }
    }

    private voidenhanceClassFile(final String className) throws Exception {
        finalClassPool pool = ClassPool.getDefault();
        pool.appendClassPath(this.classesDir);
        pool.appendClassPath(this.loggerJarsDirectory+ "\\" + "slf4j-api-1.6.1.jar");
        pool.appendClassPath(this.loggerJarsDirectory+ "\\" + "org.apache.log4j_1.2.15.v201005080500.jar");
        finalCtClass compiledClass = pool.get(className);
        if(compiledClass == null) {
            System.err.println("Class " + className + " not found");
        } else{
            final LoggerEnhancementExpEditor exp = new LoggerEnhancementExpEditor();
            exp.setClaz(compiledClass);
            compiledClass.instrument(exp);
            compiledClass.writeFile(this.classesDir);
        }
    }

    private List getClassFiles(final String classesDir) throws FileNotFoundException {
        finalList result = new ArrayList();

        finalFile startingDirectory = new File(classesDir);
        finalList files = this.getFileListing(startingDirectory);
        System.out.println(" -- Processing classes under directory:" + classesDir);
        for(final File file : files) {
            if (file.isFile() && file.getName().endsWith(".class")) {
                result.add(file);
            }
        }
        returnresult;
    }

    private List getFileListing(final File aStartingDir) throws FileNotFoundException {
        this.validateDirectory(aStartingDir);
        finalList result = this.getFileListingNoSort(aStartingDir);
        Collections.sort(result);
        returnresult;
    }

    private List getFileListingNoSort(final File aStartingDir) throws FileNotFoundException {
        finalList result = new ArrayList();
        finalFile[] filesAndDirs = aStartingDir.listFiles();
        finalList filesDirs = Arrays.asList(filesAndDirs);
        for(final File file : filesDirs) {
            result.add(file);
            if (!file.isFile()) {
                final List deeperList = this.getFileListingNoSort(file);
                result.addAll(deeperList);
            }
        }
        returnresult;
    }

    private voidvalidateDirectory(final File aDirectory) throws FileNotFoundException {
        if(aDirectory == null) {
            throw new IllegalArgumentException("Directory should not be null.");
        }
        if(!aDirectory.exists()) {
            throw new FileNotFoundException("Directory not exist: " + aDirectory.getAbsolutePath());
        }
        if(!aDirectory.isDirectory()) {
            throw new IllegalArgumentException("Is not a directory: " + aDirectory);
        }
        if(!aDirectory.canRead()) {
            throw new IllegalArgumentException("Directory cannot be read: " + aDirectory);
        }
    }

    static classLoggerEnhancementExpEditor extends ExprEditor {
        privateCtClass claz;

        publicCtClass getClaz() {
            return this.claz;
        }

        publicvoid setClaz(final CtClass claz) {
            this.claz= claz;
        }

        @Override
        publicvoid edit(final MethodCall m) throws CannotCompileException {

            if (this.isLoggerMethodCall(m)) {
                try {
                    System.out.println("      -- Enhancing log line: "
                                       + this.getClaz().getName()
                                       + "."
                                       + m.where().getName()
                                       + "("
                                       + this.getClaz().getSimpleName()
                                       + ".java:"
                                       + m.getLineNumber()
                                       + ")");

                    m.replace("{ $1 = $1 +\", "
                              + this.getClaz().getName()
                              + "."
                              + m.where().getName()
                              + "("
                              + this.getClaz().getSimpleName()
                              + ".java:"
                              + m.getLineNumber()
                              + ") "
                              + "\"; $_ = $proceed($$); }");
                } catch (final Exception e) {
                    throw new CannotCompileException(e);
                }
            }
        }

        privateboolean isLoggerMethodCall(final MethodCall m) {
            return this.isSlf4jLoggerMethod(m) || this.isLog4jLoggerMethod(m);
        }

        privateboolean isSlf4jLoggerMethod(final MethodCall m) {
            return m.getClassName().equals("org.slf4j.Logger") && this.isALogMethod(m);
        }

        privateboolean isLog4jLoggerMethod(final MethodCall m) {
            return m.getClassName().equals("org.apache.log4j.Logger") && this.isALogMethod(m);
        }

        privateboolean isALogMethod(final MethodCall m) {
            boolean result = false;

            final String[] slf4LogLevels = { "trace", "debug", "info", "warn", "error", "fatal" };
            final String methodName = m.getMethodName();
            for (final String logLevel : slf4LogLevels) {
                if (methodName.equals(logLevel)) {
                    result = true;
                }
            }
            return result;
        }

    }

}

LOG4J Usage

<appender name="errorAppender" class="org.apache.log4j.RollingFileAppender">
            <param name="File" value="logs/errors.log" />
            <param name="Append" value="true" />
            <param name="MaxFileSize" value="10000KB" />
            <param name="MaxBackupIndex" value="50" />
            <layout class="org.apache.log4j.PatternLayout">
            <param name="ConversionPattern" value="%d %-5p [%-16t]%m [%l : %t] %n" />
            </layout>
<filter class="org.apache.log4j.varia.LevelRangeFilter">
            <param name="LevelMin" value="ERROR" />
</filter>
</appender>
<appender name="ASYNC_errorAppender" class="org.apache.log4j.AsyncAppender">
            <param name="BufferSize" value="1000"/>
            <param name="Blocking" value="true"/>
            <param name="LocationInfo" value="false"/>
            <appender-ref ref="errorAppender"/>
</appender>

<logger name="com.mycompany.product.core" additivity="false"<
            <level value="DEBUG" /<
            <appender-ref ref="ASYNC_coreappender" /<
            <appender-ref ref="ASYNC_errorAppender" /<
            <appender-ref ref="ASYNC_ConsoleAppender" /<
</logger<


"LocationInfo" parameter is set to FALSE because of line number calculation's performance impact.You can set it to TRUE when you needed it. Log4j will take necessary actions on the fly. (Nevertheless i don't undertsand why this calculation needed to to be done on my thread! )

"Blocking" parameter is set to TRUE. Normally I prefer to set it true but this is my error log so i don't want to miss a error log if it happens.

"BufferSize" param is set to 1000 means :
     1) if you set blocking param to false : you can miss some logs when your buffer is full.
     2)if you set blocking param to true : your threads can be suspended while trying to log something while buffer is full. I don't prefer this kind of usage in production mode if my log entries not so critical like callDetailRecords/errors.

I will not write about why/when you need to use async logger. I want to finalize this blog entry ASAP before the blogger.com is blocked county wide  (Turkey : A Middle East Country)

Take Care..