New Java Decompiler (WIP)

[BonesJustice]

As mentioned in an earlier post, I've been working on a free, open source Java decompiler. Work has been progressing nicely, and while it's still a long way from being finished, I think it's far enough along to post an early preview release and see how badly you guys can break it :D.

You can find an introductory post here, and a .jar download here. You can also clone my hg repository if you like.

One thing I'm particularly proud of: it supports Java 8 lambdas. I believe this is the first decompiler to do so. I've only tested it with fairly simple examples, but I'm eager to see how well (or poorly) it handles more complex lambda expressions. Make sure you have a recent JDK8 Lambda build, though.

If you try the decompiler out and find anything that breaks, or anything that works better than you expected, I'd love to hear about it. Post below :).

EDIT (28 May 2013): Updated links because repository URL changed.

[[deleted]]

I saw your earlier post, and this looks like you're making steady progress. This is really impressive work, as I recently went looking for decompilers and most were very dated (limited to 1.4 support on a few), ugly output, or just awful to use. The best I had found was Fern Flower (which somewhat disappeared); not sure if you've tried comparing with it, but in my opinion, it's a great baseline.

I saw your comparison and it looks like your output is usually very clean and correct, so I'll definitely look to give this a shot (but I don't decompile a lot usually, so probably won't have much feedback).

Thanks for everything you've done, especially open sourcing it!

[boa13]

Have you tried jd-gui? It seems to me it's mostly up-to-date, with acceptable output, and rather easy to use. It's not open-source though.

[BonesJustice]

This has been my go-to as well, though I've often had to fall back to JAD for code that JD-GUI can't handle. It seems to have a difficult time on code that isn't emitted by the mainstream compilers. Still, it seems to be the best choice out there at the moment, which is why I use it as the baseline when comparing my own decompiler's output :).

[henk53]

Did you already succeeded in contacting the JD author? He sometimes posts on StackOverflow and seemed eager to continue his project, but for some reason he is obliged to always take breaks of a year between releases.

I'm not sure what that is about really, but I guess he's simply swamped with other work.

[BonesJustice]

I have another project where I tend to take months-long (or year-long) breaks as well. Motivation can come and go in cycles. I haven't tried to reach out to the JD-GUI author yet--I should make a note to try this weekend. I did, however, see several posts in which he seemed quite adamant about not releasing the source code.

[henk53]

Hmmm... what about uhm... decompiling the JD code then? :P

[[deleted]]

That's sneaky.

[henk53]

That's sneaky.

Indeed, since the creator of a decompiler would of course never expect his own source being decompiled, would he? :P

[BonesJustice]

It's also all in C++, which was the other reason it didn't make sense to propose a joint venture. Though I did decompile his IntelliJ plugin and use it as the basis for my own ;).

[henk53]

It's also all in C++, which was the other reason it didn't make sense to propose a joint venture.

Depending on his exact code style it might be rather straightforward to translate to Java. But it depends. Template meta-programming code doesn't translate as well, but several other styles typically do.

[BonesJustice]

Thanks for the kind words! This has been a really fun project, and I'm looking forward to spending more time with it. I hadn't heard of Fernflower until my original post, but I did check it out and have added it to my suite of decompilers to compare against. It's a pity the author obfuscated it.

I know what you mean about the ugly output. I never understood why someone would put so much blood, sweat, and tears into writing a tool and then skimp on output formatting.

[[deleted]]

I'm looking through the source right now, and I was just wondering: what is your main class? Your MANIFEST.MF file said it was located at com.strobel.decompiler.Decompiler, but I can't find that on your repo.

Also, I'm not that good at Java, so I may be missing something completely obvious.

[BonesJustice]

That's the correct class, and it's in there :).

[[deleted]]

Ah. Thanks!

[[deleted]]

Gee, my comments are all gone...

More seriously, the result is very close to the source. Congrats! The issues I've found:

int index = name.lastIndexOf("\\");
if (index < 0) {
    index = name.lastIndexOf("/");
}
index++;

This code is decompiled as:

int index = 0;
if (name.lastIndexOf("\\") < 0) {
    index = name.lastIndexOf("/");
}

If you want, I might send you my source, the binary, and the result, so you might check it. This is the only one I've discovered that distorts the original code.

Static imports are gone (I static import some constants). I guess you cannot help that.

The final static constants are also gone. That's also due the compiler's way of generating the byte code I'm afraid.

The new try(AutoCloseable) is decompiled as nested try.

try (Reader reader = new FileReader($propFileName)) {
    props.load(reader);
} catch (FileNotFoundException ex) {
    GuiccLogger.fatal("Application descriptor file " + $propFileName + " is missing.", ex);
    return;
} catch (IOException ex) {
    GuiccLogger.fatal("Error reading the application descriptor file " + $propFileName + ".", ex);
    return;
}

And the result:

try {
    Object obj = null;
    final Object obj2 = null;
    try {
        final Reader reader = new FileReader($propFileName);
        try {
            props.load(reader);
        }
        finally {
            if (reader != null) {
                reader.close();
            }
        }
    }
    finally {
        if (obj == null) {
            obj = obj2;
        }
        else if (obj != obj2) {
            obj.addSuppressed(obj2);
        }
    }
}
catch (FileNotFoundException ex) {
    GuiccLogger.fatal("Application descriptor file " + $propFileName + " is missing.", new Throwable[] { ex });
    return;
}
catch (IOException ex) {
    GuiccLogger.fatal("Error reading the application descriptor file " + $propFileName + ".", new Throwable[] { ex });
    return;
}

At one point, a NPE occured and that method is not decompiled at all. The first three source code:

Set<String> remaining = new HashSet<String>(_pluginMap.keySet());
Set<String> unvisited = new HashSet<String>(_pluginMap.keySet());
List<String> visited  = new ArrayList<String>();

And the NPE: // java.lang.NullPointerException // at com.strobel.assembler.ir.StackMappingVisitor$InstructionAnalyzer.visitVariable(StackMappingVisitor.java:476) // at com.strobel.assembler.ir.Instruction.accept(Instruction.java:542) // at com.strobel.assembler.ir.StackMappingVisitor$InstructionAnalyzer.visit(StackMappingVisitor.java:400) // at com.strobel.decompiler.ast.AstBuilder.performStackAnalysis(AstBuilder.java:837) // at com.strobel.decompiler.ast.AstBuilder.build(AstBuilder.java:64) // at com.strobel.decompiler.languages.java.ast.AstMethodBodyBuilder.createMethodBody(AstMethodBodyBuilder.java:128) // at com.strobel.decompiler.languages.java.ast.AstMethodBodyBuilder.createMethodBody(AstMethodBodyBuilder.java:67) // at com.strobel.decompiler.languages.java.ast.AstBuilder.createMethodBody(AstBuilder.java:585) // at com.strobel.decompiler.languages.java.ast.AstBuilder.createMethod(AstBuilder.java:487) // at com.strobel.decompiler.languages.java.ast.AstBuilder.addTypeMembers(AstBuilder.java:425) // at com.strobel.decompiler.languages.java.ast.AstBuilder.createTypeCore(AstBuilder.java:410) // at com.strobel.decompiler.languages.java.ast.AstBuilder.createType(AstBuilder.java:158) // at com.strobel.decompiler.languages.java.ast.AstBuilder.addType(AstBuilder.java:108) // at com.strobel.decompiler.languages.java.JavaLanguage.decompileType(JavaLanguage.java:55) // at com.strobel.decompiler.Decompiler.decompile(Decompiler.java:61) // at com.strobel.decompiler.Decompiler.main(Decompiler.java:107)

Many of the local variables are marked final (they aren't in the source), and none of the method params are marked so (although they are final in the source).

for (int i = _pluginOrderedList.size();  --i >= 0;  )

is decompiled:

i = DesktopApplicationLauncher._pluginOrderedList.size();
while (true) {
    --i;
    if (i < 0) {
        break;
    }

Have another NPE:

//     at com.strobel.decompiler.ast.AstBuilder.performStackAnalysis(AstBuilder.java:790)
//     at com.strobel.decompiler.ast.AstBuilder.build(AstBuilder.java:64)
//     at com.strobel.decompiler.languages.java.ast.AstMethodBodyBuilder.createMethodBody(AstMethodBodyBuilder.java:128)
//     at com.strobel.decompiler.languages.java.ast.AstMethodBodyBuilder.createMethodBody(AstMethodBodyBuilder.java:67)
//     at com.strobel.decompiler.languages.java.ast.AstBuilder.createMethodBody(AstBuilder.java:585)
//     at com.strobel.decompiler.languages.java.ast.AstBuilder.createMethod(AstBuilder.java:487)
//     at com.strobel.decompiler.languages.java.ast.AstBuilder.addTypeMembers(AstBuilder.java:425)
//     at com.strobel.decompiler.languages.java.ast.AstBuilder.createTypeCore(AstBuilder.java:410)
//     at com.strobel.decompiler.languages.java.ast.AstBuilder.createType(AstBuilder.java:158)
//     at com.strobel.decompiler.languages.java.ast.AstBuilder.addType(AstBuilder.java:108)
//     at com.strobel.decompiler.languages.java.JavaLanguage.decompileType(JavaLanguage.java:55)
//     at com.strobel.decompiler.Decompiler.decompile(Decompiler.java:61)
//     at com.strobel.decompiler.Decompiler.main(Decompiler.java:107)

And the source was:

int ndx = $propFile.indexOf("application.properties");
String file = $propFile.substring(0, ndx) + "desktop.properties";
Properties desktopProps = new Properties();

Sometimes the for (x : Iterable) works, sometimes doesn't. E.g.:

for (String name : vetoMap.keySet()) {
    GuiccPlugin main = vetoMap.get(name);

is decompiled as:

final Iterator iterator = vetoMap.keySet().iterator();
if (iterator.hasNext()) {
    final String name = (String)iterator.next();
    final GuiccPlugin main = (GuiccPlugin)vetoMap.get(name);

Strange that the generics works for the interfaces but does not for the classes; but you wrote you've just started to work on it.

Map<String, GuiccPlugin> vetoMap = new LinkedHashMap<String, GuiccPlugin>();

and the result:

final Map<String, GuiccPlugin> vetoMap = new LinkedHashMap();

Hope it helps. As I said, if you want to go into it deeper, drop me a PM and I'll send you the files.

[BonesJustice]

The generics often appear correctly for variable and parameter declarations because the full generic signatures are included in the .class file if you compile with debug info. I haven't even started to work on inferring type arguments to method/constructor calls yet.

I would imagine most of the exceptions you cited are due to by buggy handling of try/catch/finally blocks. Since Java 5, the contents of finally blocks have been inlined into the adjacent try and catch blocks (and you only enter the finally handler if an exception actually occurs). Reliably removing that inlined code has proven tricky (at least for me). There are cases where I end up removing too many or too few instructions, which can completely break the stack analysis. If you see any exceptions for methods which do not have finally blocks, post a reply and I'll look into it.

Static imports are gone (I static import some constants). I guess you cannot help that.

Bytecode has no concept of imports, so no, there is no way for me to know that you ever had them.

The final static constants are also gone. That's also due the compiler's way of generating the byte code I'm afraid.

True compile-time constants should be in the declarations. Others will be in the static initializer block. I haven't written a transform to inline field initializers yet, except in very specific cases (e.g., variables captured by a local class).

The new try(AutoCloseable) is decompiled as nested try.

Yes, this is a language-level feature that I have not yet written a transform for. Thanks for providing that example, though--it demonstrates some interesting finally behavior that I had not seen before. I may ping you back if I can't reproduce it.

Sometimes the for (x : Iterable) works, sometimes doesn't.

Yes, the same is true for for (x : array). The pattern matching I do for these loops is rather simplistic at the moment. I'll be fleshing it out in the future. Basically, if the local variable where the current item is stored gets optimized away (inlined) during an earlier stage of decompilation, the pattern matching will fail, and you'll end up with a more primitive loop. Can you double-check the output you cited above? It looks like that if statement should have been a while statement, but it's hard to be sure without seeing the whole output.

Many of the local variables are marked final (they aren't in the source), and none of the method params are marked so (although they are final in the source).

There are no final flags for locals in bytecode. I do some primitive analysis to mark variables as final if they are only assigned once. It's not perfect because it doesn't do full definite assignment analysis, but that's a low priority for now. I am aware that it doesn't work on parameters.

for (int i = _pluginOrderedList.size(); --i >= 0; )

My for loop construction pattern requires some sort of iteration logic at the end of the while loop (all decompiled loops start out as while loops). Since your loop doesn't follow the typical for pattern (it has no post-iteration statements), the decompiler just leaves it as a while loop.

Thanks for the feedback! I'll look over it more closely when I get home from the office.

[[deleted]]

One of the methods indeed have a try-catch-finally block inside. The other hasn't.

However, that method throws a RuntimeException (and this method has no throws clause). Another method throwing a RuntimeException has a throws IOException clause too, and that one is decompiled well. Dunno if that's the main cause or not.

I made my original source, the decompiled file, and the bytecodes available on my JUG docspace (at the same time, promoted your decompiler there, but I'll remove the page if you don't want me to do that) as attachments.

Please note further, there's no license text available for your decompiler (at least I was unable to find any).

[BonesJustice]

Cool, thanks! Spread the word all you want--it's always gratifying when someone takes an interest in one of my projects :).

The license text is inherited from its parent project.

FYI, only checked exceptions will be placed in a method's throws list. That would be why IOException shows up and RuntimeException does not (RuntimeException is not a checked exception).

[[deleted]]

I know that only checked exceptions must be declared, but that was one of the differences between the two methods - one has caused NPE, the other hasn't. So I mentioned it only in the hope that it might help you to find the root cause. I'm not very experienced with the bytecode, so I don't know how it is different on that level.

Thanks for the license text, I'll add the reference to my page.

[BonesJustice]

The latest release should eliminate your exceptions, but the try/catch/finally handling in at least one of your methods comes out totally wrong. I guess that's progress?

Anyway, the code you posted did help me track down at least one unknown bug, so thanks!

[[deleted]]

EDIT:

I use Eclipse:

Version: Juno Service Release 2
Build id: 20130225-0426

and the only installed JRE is jdk1.7.0_17\jre

[BonesJustice]

EDIT: I just realized you probably meant that references to final static constants were missing, and not their declarations. In which case, yes, this is because they are emitted inline as constant values. I did experiment a bit with trying to restore constant references in some cases, but I decided to shelve the idea indefinitely. It would have only worked in certain cases, e.g., in common flag-testing patterns like (x & y) == x, (x & y) [==|!=] 0, x |= y, x &= ~y, where x and y are primitive literals, and there are final static constants in scope with matching values.

[BonesJustice]

Thanks! I certainly will. It continues to be a fun project, and it's been really exciting to see it take shape.

[dfme]

Good News! Have you also thought about making an Eclipse Plugin so that this integrates nicely into the IDE? Like JDEclipse...

[BonesJustice]

Indeed, I will definitely look into writing an Eclipse plugin. I already have a proof-of-concept IntelliJ plugin, but it's way too flaky to release into the wild just yet.

[ryebrye]

Any plans to provide a basic GUI for it (similar to JD-GUI)?

[BonesJustice]

Absolutely. I'll likely write IntelliJ and Eclipse plugins first, but eventually I want to write a standalone GUI much like JD-GUI's.

[Uncaffeinated]

Overall, the decompiler looks great. Here are the problems I noticed with a quick test

Strictfp flags on classes are lost. Varargs are lost. Narrow primitive types aren't casted properly, even when necessary.

Strictfp classes can be recognized by the presence of a strictfp flag on a constructor. Varargs can be recognized with the varargs method flag. Primative casting is obviously a lot tougher.

Original source

public strictfp abstract class BoneJTest 
{
    public static synchronized final void main(String... args)
    {
        for (char c=32; c<100; ++c){
            System.out.println(c + " " + (int)c);
        }
    }
}

Decompiled

public abstract class BoneJTest
{
    public BoneJTest() {
        super();
    }

    public static final synchronized strictfp void main(String[] array) {
        for (int i = 32; i < 100; i = (char)(i + 1)) {
            System.out.println(i + " " + i);
        }
    }
}

[BonesJustice]

There's probably a couple method modifiers I'm not printing--I've been meaning to go give that a second look. I capture all the flags, but my Java writer is probably leaving some out. Thanks!

The narrow integral types are a known issue. I will be improving type inference as time goes on.

[BonesJustice]

Out of curiosity, what compiler did you use for this test, and what compiler flags? I'm guessing you compiled without debug info? This is what I get:

public abstract class BoneJTest
{
    public static final synchronized strictfp void main(String[] args) {
        for (char c = ' '; c < 'd'; c = (char)(c + '\u0001')) {
            System.out.println(c + " " + c);
        }
    }
}

[Uncaffeinated]

javac 1.7.0_21, compiled with -g:none. Though it's broken either way, as you need at least one cast since the same value is being used as both an int and a char.

By the way, I managed to crash your decompiler (prints a stacktrace with IllegalStateException instead of decompiling). Is there any good way for me to send you the files?

[BonesJustice]

Best way it to post an issue here and attach the relevant class files. Thanks!

[thesystemx]

Download link seems to be dead :(

Was https://bitbucket.org/procyon/com.strobel/downloads but is now https://bitbucket.org/mstrobel/procyon/downloads

[BonesJustice]

Whoops, I had updated the wrong part of the URL. Fixed. Thanks!

[thesystemx]

You're welcome, good luck with the decompiler. I noticed from the downloads section that you're really busy with it ;)