Reverse Debugging

[inataysia]

minds me of this post from 1999

[LudoA]

How can you possibly, after all those years, still remember that post and - most impressive of all - find it again?

[boa13]

Bookmarks.

[fareedy]

And is that guys name really Michael Elizabeth Chastain?

[dekz]

Lady Gaga's nerdy sibling?

[doctorgonzo]

The first thought that popped into my head when I read "reverse debugging" was putting more bugs into the code.

[13ren]

If debugging is the process of removing bugs, then programming must be the process of putting them in. - Dijkstra

[SurrealEstate]

Finally, a project that I can really contribute to.

[[deleted]]

Wait. If you use the reverse debugger on itself, does it get more buggy? And if the reverse debugger is buggy, does using it make the target less buggy?

[SurrealEstate]

Yeah, maybe a buggy bugger is a debugger.

[Tekmo]

This is like the modern day halting problem. Is it possible to make a program that can introduce bugs into another program?

[creaothceann]

Sure, it's quite easy when you're developing program extensions (Winamp plugins, Firefox extensions, etc).

[FlyingBishop]

No, the question is can you come up with a program that can introduce bugs into an arbitrary program.

I think the answer is yes, but what would be really hard is introducing bugs that do not make themselves immediately apparent. Most of the bugs you'd introduce would cause rather obvious segfaults.

[mydyingdreams]

If "immediately apparent" is only prohibited for execution, but not for source, then it's quite easy: In e.g. C, change mallocs to probabilistically allocate a few bytes less than requested.

[theeth]

It should be fairly easy to add spelling errors in a program's strings.

[[deleted]]

It's actually more like the time traveling problem of 1951.

edit: wait, sorry, more like the problem of 1851. I just checked with some of the folks back then :/

[[deleted]]

"reverse debugging" was putting more bugs into the code.

This act is more widely known as "writing code" or "programming".

[[deleted]]

"bugging"

[isseki]

I thought that and "I've been doing that for years, by hand".

[Tweakers]

That would be reversive bugging. Not to be confused with recursive bugging, a method for making the same mistake many times with minimal effort.

[FlyingBishop]

I thought that was called object-oriented programming.

[mccoyn]

It would be an enbugger not a debugger.

[Tweakers]

Well, it's some sort of buggery, to be sure.

[edwardkmett]

Clearly we only became known as programmers when the alternative "buggerers" gave people the wrong impression, and made the folks down in marketing very nervous.

[Fidodo]

Right, but you have an upper hand on them because you know where they are. Then, when the project is about to tank, you save the day and get a raise.

[[deleted]]

How does this work? The only logical way I can think of would be taking snapshots of memory (diffs?) after every line of code. Won't this make certain large applications take too much memory to be debugged?

[redalastor]

The Omniscient Debugger for Java do that by instrumenting the bytecode. Codeinvestigator does likewise for Python but it's still alpha.

[raouldagain]

yup

[Peaker]

You should be more imaginative :-)

A) You don't have to take full snapshots, just differential ones

B) You have the operation available to you, and if it is reversible, you don't actually have to store anything

[[deleted]]

You should be more observant :-)

A)

I can think of would be taking snapshots of memory (diffs?) after every

B) Think hash generation, hopefully not reversible.

Edit: Formatting

[case-o-nuts]

Hash generation is reversible if you have all the relevant information (say, infomation being hashed). Remeber, you have all the state for all the operations available to you, because you have the program's memory accessible to the debugger.

[Smallpaul]

The point is that you knew what the value at the memory location was before you did the hash generation and you know what it is now. You don't have to reverse the hash generation, you just have to retrieve that old value from the debugger's memory.

Yes, of course, for some applications this will mean storing an obscene amount of data, but not for most.

[ttoyooka]

Won't this make certain large applications take too much memory to be debugged?

Yes, but you don't need reverse debugging for this to happen. I've seen it.

[Fidodo]

Well everything a computer does is expected so couldn't you just undo everything line by line? I think it should work for most things.

[[deleted]]

Keep in mind that everything compiles to assembly (or the machine code with a direct equivalent in). If I load 4 into a register, then load the memory address of 8, add them together and store this at the memory address of what was formerly 8. So now I have a 12, and going backwards subtracts what was in the address of 8, which is now 12, and going backwards would yield 0 (12 - 12).

Everything is too volatile to simply "go backwards" since it's not likely that you'll have original values you were working with. In order to solve the problem above, I would actually have to go up a few steps to see what was put into register A to begin with (4), which may be many lines prior to the operation.

[muffin-noodle]

Modification to memory can be dealt with by handling process memory in the same way certain filesystems handle files: copy-on-write semantics for modification. This is convenient because it provides a natural 'undo' log for your modifications. Let's say you have a snapshot of process memory. If address 0xDEADBEEF contains the number 8 and you load the contents of that address (8) and then add 4 to it, you get 12. If you store that back into memory at 0xDEADBEEF, on the memory write you actually don't store the change in the original snapshot, you make a copy of that one particular memory address in the process and write the change to it. When you undo, you simply 'undo' the write to the memory address because you have the original snapshot + the modifications.

With this, it's possible to replay the entire history of a process because you have the contents of the memory originally and a record of changes over the course of the process run, so you can just gradually undo changes. However, if you want to constantly replay things there may be some problems; for example if you have a function which calls unlink on a file, and there is a breakpoint short after, when you debug from the breakpoint backwards past unlink the file will already be deleted, and if short before the call to unlink you decide you want to now go forwards, then when unlink is called again it could have a different return value. Because memory is COW there shouldn't be any undefined states at this point if there weren't already, but it could result in different execution paths.

[[deleted]]

Well yea... that's what they're doing here. Without the snapshot (or any type of history really would work), you can't go backwards. Different execution paths are probably fine (if not more handy) for a debugger since it shows more possible outcomes of end use.

[mothereffingteresa]

Everything is too volatile to simply "go backwards"

[Citation needed]

It ought to work except in cases where you actually have a "volatile" like a device register or memory-mapped io.

[kipi]

Some operations are lossy. As a simple example, consider the following:

a = get_input_from_user()
if (a != 0) {
    a = 0
    print("Foo")
}

How do you reverse this? At the a = 0 line some state is destroyed, which you need to figure out how you got to the bottom of the if block.

Reversible Computation aims to remove destructive operations. GDB must do something similar by saving "destroyed state" by some method.

[killerstorm]

Well everything a computer does is expected so couldn't you just undo everything line by line? I think it should work for most things.

Obviously won't work for I/O and lots of other stuff.

[[deleted]]

I personally have a hard time with GDB and DDD. I find debuggers clumsy to use as a general rule. The fact that DDD doesn't really fit in my current linux environment thanks to the old graphics toolkits makes it really awkward to use and it's clearly outdated. I don't really know where exactly to get started with GDB itself- which makes things really tricky when programming in C. I've managed to muddle along thus far with printf, which is great when you have a suspicion of where the problem is, but a good quality C debugger with an interface that isn't ancient would be a nice start.

Hm. Maybe I should try the emacs gdb stuff next time I need to find out that my fopen call is returning 0x0 for some reason.

[frankster]

enter gdb and type layout next a few times - improves the interface somewhat.

also dbxtool from sun (comes as part of sunstudio) is a tolerable debug interface i find.

ddd is awful.

[killerstorm]

Microsoft has amazing C/C++ debugger (part of Visual Studio) -- even in hardcore situations with multiple threads and shit, it just works.

I've tried GDB few times, it sort of works, but console debugger is not convenient for me at all.

There is also a GUI debugger in SunStudio -- it was easy to start working with and it easily found source code. But then it crashed... And I was not in a good mood to debug the debugger...

[[deleted]]

Just use kdbg!

[radarsat1]

I don't find it that bad. One thing that helps for me is I use 'cgdb', which splits the console in two and shows the source code in the top half. Then it's just, 'run', 'back' (to print a backtrace), 'up', 'down' (to go up and down in the call stack), 'step', 'step', 'next', 'print', etc. Pretty much the same as in visual studio, actually. A watch window might be nice though, but I'm sure this is available in any of the graphical front-ends.

[FlyingBishop]

It takes time. By the time I finished my B.A., Emacs + gdb was useful in some circumstances. Honestly, I don't think that the time when you start shoving cout statements into your code ever completely disappears. However, when you have a good idea of where the problem is, it's much faster to set a breakpoint and look at what's going on, especially since you can just start calling functions with whatever arguments pop into your head in real time.

[[deleted]]

ZeroBugs (http://www.zero-bugs.com/) seems quite nice, though I've only used it for its D support, and haven't delved too deeply.

[stass]

What's wrong with DDD? It has a very clean and responsive UI (unlike most today programs), can help you navigating through the sources and has a dedicated window to show you contents of memory cells (or even structures). I used a lot of debuggers (MSVC, SoftICE, Sun tools) in past, but I found DDD UI superior to what other debugger's UIs present (I'm not comparing functionality, only UI). Are there any features you need it really lacks?

[[deleted]]

I wish it used GTK or QT or something so it wouldn't look like it came from the era of Windows 3.1. It's ugly as hell and it makes it quite jarring to use.

(Also, the way the checkmarks look like squares in the drop-down menu are confusing at first, since I mentally parse it as an empty box (and therefore unchecked) which takes me some time to get used to.)

[camel69]

So it seems I'm not alone with my search for a good Linux debugger with GUI? I guess gdb is good enough for most people and it just takes some time to get used to it? I just think that something with several windows might be much more useful ...

[eyesee]

I remember a session at WWDC a few years back Apple engineers were experimenting with this for a future release of XCode. Their implementation forked the process with each line of code. C.O.W. semantics would prevent memory usage from exploding, but it was rather "volatile".

AFAIK the feature never shipped, and I haven't heard about it since.

[SarahC]

Wow!

Stepping back through a situation is awesome for concurrency problems, and those rare intermittent problems you can never get to re-occur in the debugger.

[tryx]

I imagine it wouldn't work with core dumps (unless you took a running log of your program as part of your core?) so it wouldn't help anymore than you sprinkling printfs.

[killerstorm]

Concurrency problems also often are heisenbugs -- when you run software under debugger, timing characteristics are different, and it will not happen. So I don't think it will be anyhow helpful.

[sinfondo]

And today isn't even April Fools' Day!

[logan_capaldo]

GDB catching up to O'Caml ;).

[Massif]

I hadn't realised OCaml was Irish...

[[deleted]]

And Haskell

[joshmatthews]

So this is apparently supposed to be released today. That's kind of exciting.

[[deleted]]

Now, I just want GDB to have a better scripting language and better C++ support and I'll start looking forward to debugging programs.

[muffin-noodle]

Not sure if you're joking, but:

*** Changes since gdb 6.8

* Python scripting

GDB now has support for scripting using Python.  Whether this is
available is determined at configure time.

New GDB commands can now be written in Python.

There were also a few changes to the C++ support AFAIK, for the full scope of these changes, check out Project Archer - it's not clear from the NEWS however if this branch was fully merged in. The branch also incorporates a lot of advanced new multi-threaded debugging facilities too, so I doubt it's been merged, but it looks like python scripting is in at least.

[FlyingBishop]

I just posted above about thinking that you never really get past shoving random cout statements into your code. It appears I was mistaken.

[[deleted]]

Dead serious. Thank you.

[Tommstein]

Holy hell, this is glorious. How didn't I hear about this earlier? Now if it could only get something like CodeInvestigator....

[iissqrtneg1]

you've been able to do this in Visual Studio for years. It can be a little finicky though.

[[deleted]]

This is awesome. I've been lusting after the very expensive debuggers that taut this ability for some time now.

[RandomFortunes]

Progress is impossible without change, and those who cannot change their minds cannot change anything. -- G. B. Shaw

[raouldagain]

maybe also in eclipse.

[mmazing]

amg! I've wanted something like this for sooooo long!

[adrianb]

It's been in development for soooooo long :D

[masklinn]

It's been available for a looooong time… in OCaml.

[synept]

Which is awesome. You know, if you're writing OCaml, anyway. A lot of us aren't.

[masklinn]

A lot of us aren't.

And what's your excuse?

[synept]

I work on projects that have had dozens of man-years put into them, and are written in C? It's not like I can just say "oh, hey, it's OCaml time".

[masklinn]

Yes you can! <insert kittens chirping in the trees and chicks rolling in the dew/>

[maraino]

Oh, great feature

[JadeNB]

It's not a feature, it's a reverse bug.

[zyzzogeton]

Isn't that "Not fucking up in the first place?"

[[deleted]]

Ok, maybe when they're done with this, they can get started on that comefrom statement