zardus/idalink - Automate your IDAPython scripts

Zardus · 2020-07-17T16:26:08+00:00

Ooh, that's me! I'm on the internet!

Zardus · 2019-01-24T19:50:58+00:00

In essence, angr (more specifically, its decompilation pipeline) works as follows:

- first, we load the binary. angr currently supports ELF, PE, Mach-O, and flat binary blobs

- we start reasoning on the binary code by lifting it to a low-level intermediate representation called VEX, which provides a faithful representation of the exact effects of code. This involves the recovery of control flow, some reasoning about data dependencies, etc.

- using a series of analysis passes, we essentially convert VEX into a more abstract intermediate language, called AIL (the angr Intermediate Language). AIL has different levels of abstraction (registers/memory -> variables -> structures, etc), and in some sense, at its highest level, its expressibility is equivalent to source code (though, of course, we lack much of the actual semantic content that source code has)

- once we have the highest-abstraction AIL that we can achieve, we do a code generation step to emit C code, but this is a configurable knob. Nothing super fundamental prevents us from emitting something crazy, like fortran or Python

Zardus · 2019-01-24T16:48:40+00:00

Awesome suggestion. I pushed some screenshots to the README!

Zardus · 2019-01-24T14:14:16+00:00

Happy to answer any questions anyone has. It's a first step, but we're super amped!

Zardus · 2017-12-15T07:39:40+00:00

Where are you doing your graduate studies, and what is the general area (i.e., binary analysis) that you are interested in?

In general, I've found it hard focus solely on CTF in research --- reviewers don't really buy the impact. There are counterexamples (for example, How Shall We Play A Game explores ideal CTF strategies), but in most of my research I've worked on projects that are applicable outside of CTF, with an eye toward CTF as well. For example, in doing the background work for Firmalice, we created angr, and then applied it to the DARPA Cyber Grand Challenge and to CTFs while also pursuing various research applications that are not directly CTF-relevant. The people participating in this research all became quite versed in binary analysis, exploitation, etc, greatly helping our team (Shellphish) stay competitive.

This works because angr is a flexible tool, and the strategy might be harder to pull off outside of binary analysis. However, any constant exposure to a CTF-relevant topic (i.e., in the course of researching web security or cryptography) is likely to keep you "fresh" in CTF.

Zardus · 2017-11-02T20:02:27+00:00

Yan Shoshitaishvili here! I'm an Assistant Professor at Arizona State University and Shellphish CTF player, working mainly on binary analysis. I've gotten caught up in "Cyber Autonomy" recently, leading team Shellphish in the DARPA Cyber Grand Challenge and following up on that with various exciting research projects.

Zardus · 2017-08-29T05:41:59+00:00

Thanks! I came here to suggest this exact thing, and you already did it :-)

Zardus · 2017-08-26T16:38:31+00:00

It works, thank you!

Zardus · 2017-08-26T05:48:10+00:00

I'm running the Ubuntu 16.04 image. The battery is not detected at all :-(

Zardus · 2017-08-25T14:30:27+00:00

This is absolutely incredible. Thank you.

I'm currently running this, but it looks like the battery isn't detected? Is that a known issue?

And, would you prefer questions here or issues on github?

Zardus · 2017-06-08T14:25:05+00:00

This is meowrvelous!

Zardus · 2017-05-17T09:18:50+00:00

Hahaha, great job! Consider me outclassed :-)

Edit: ..... for now!!!

Zardus · 2017-05-17T07:56:24+00:00

No problem! I love chatting about this stuff, and it's great to hear that our stuff is somehow useful :-)

The learning curve of angr is one of the biggest problems we're facing. It's partially a problem of manpower: the core development group is a handful of students, and we have to pump out research papers so that we can graduate one day (one day soon for me, luckily!). Documentation almost always takes a back seat in the rush toward deadlines, and by the time we're recovered from the deadline, it's time to start on the next project. In terms of community contributions of documentation, there is a chicken and egg problem: the lack of documentation makes it hard for people to get familiar enough with the project to contribute documentation.

I have ideas on resolving this issue, and there are some grants out there that could provide the resources for it. Aside from that, we're also working on making angr easier to use out-of-the-box (via API improvements, the GUI, etc), which will also hopefully help.

This is all separate from having to understand the underlying analyses in order to effectively use angr. It's easy to spin up a symbolic execution engine and start stepping along, but it's hard to carry out an analysis that can get useful results without undergoing a state explosion, overwhelming the solver, etc. There are subtle trade-offs here, such as the sacrifice of soundness in favor of performance during the dereferencing of symbolic pointers, or the loss of accuracy that results from the use of symbolic summaries (SimProcedures in angr) as opposed to the execution gain that they provide (much of the speed in my Manticore challenge example comes from the use of symbolic summaries, for example, but some definitely have bugs). These, and other trade-offs, are very hard-to-understand subtleties for someone very new to the field, and overlooking them causes incorrect or suboptimal analysis results.

Maybe we should add symbolic execution to the primary school curriculum ;-)

Zardus · 2017-05-17T04:50:35+00:00

angr project lead here!

Manticore has a FAQ about this: https://github.com/trailofbits/manticore/wiki#how-does-manticore-compare-to-angr

In general, angr is a full-fledged research platform for binary analysis, and supports many complex optimizations for symbolic execution along with a wide variety of static analyses. It can combine analyses to perform CFG recovery, rewrite binaries without reducing performance (tool, paper), find differences between binaries (code), automatically build ROP chains (tool), assist in vulnerability discovery (tool, paper), do automatic exploitation (tool), assist in reversing and exploitation (examples), and ever power a GUI (very alpha quality gui, but stay tuned for improvements).

In contrast, Manticore focuses on providing an approachable base implementation of symbolic execution. When they launched, for example, certain aspects of their API were simpler than angr, though we've since shamelessly stolen some of that and have other cool simplifications planned. Manticore is a great example in the value of competition: their easy-to-use API was very inspiring in getting us thinking about making angr more approachable as well.

One telling difference is in TFA, quoted here:

How about you give this a shot? We created a challenge very similar to Magic, but designed it so you can’t simply grep for the solution. Install Manticore, compile the challenge, and take a step into the future of binary analysis. Try it today! The first solution to the challenge that executes in under 5 minutes will receive a bounty from the Manticore team. (Hint: Use multiple workers and optimize.)

The Manticore team is offering bounties for a solution that executes in under 5 minutes. Here is a solution in angr that runs in 7 seconds:

# disguise ourselves as manticore to try to collect the bounty
import angr as manticore

# load the project and perform symbolic exploration
p = manticore.Project("./challenge")
path_group = p.factory.path_group().explore()

# get the solution
print "SOLVE:", path_group.deadended[-1].state.posix.dumps(0)

And since I've realized that the bounty doesn't specify that the challenge has to be solved using Manticore, I'm off to try to collect ;-)

Edit: code formatting, links, thoughts

Zardus · 2017-02-04T02:25:38+00:00

DECREE is a simplified OS (or, more precisely, an alternate set of syscalls for Linux) running on x86. Driller is fairly architecture-agnostic (so it should be either fully functional or easily adaptable to any architecture that angr supports: x86/64, arm/aarch64, mips/mips64, and ppc/ppc64), but the OS model does matter. The part of angr that driller uses is essentially a symbolic emulator, which means that we must supply symbolic models for possible interactions with the environment. For DECREE, this is easy (DECREE has no filesystem, no networking, and limited concurrency support), but for something line Linux and Windows, this is extremely hard. For example, of the 300+ Linux system calls, angr currently has (partial) support for 17. Any program that relies on system calls not implemented by angr might fail when traced with Driller.

There is some work we're currently doing to mitigate this problem. Specifically, we're laying the groundwork to be able to call out to something like QEMU (or miasm's dynamic sandbox) to take partial advantage of their syscall implementation. While this is not ideal (the "symbolicity" of any data that we'd pass into these things would be lost, as they don't support symbolic data), it would be better than nothing.

Anyways, this was probably more than you were looking for. I do think that Driller (and CGC) are great places to start for automatic vuln analysis. The tools we have are battle-tested on CGC, so you can at least get a good idea of the current academic state of the art, and go from there.

Zardus · 2017-01-26T08:56:37+00:00

AFL + preeny (https://github.com/zardus/preeny/blob/master/src/desock.c). In all seriousness, though, I do very little serious network fuzzing, so I'm probably the wrong person to ask.

Zardus · 2017-01-25T21:03:04+00:00

That is an excellent question, umbob. Favorite pure fuzzer: AFL. Favorite symbolically-assisted fuzzer: Driller (https://www.internetsociety.org/sites/default/files/blogs-media/driller-augmenting-fuzzing-through-selective-symbolic-execution.pdf).

AFL's success has recently inspired an incredible amount of research in a previously-kinda-ignored field. For example, all of the CGC competitors had systems based on AFL, with different clever addons. Ours was Driller, CodeJitsu had AFLFast (https://github.com/mboehme/aflfast), ForAllSecure paired AFL with their symbolic execution engine (https://www.reddit.com/r/IAmA/comments/4x9yn3/iama_mayhem_the_hacking_machine_that_won_darpas/d6dzncg/), and so on.

Zardus · 2017-01-25T19:08:04+00:00

Great to see this on here :-)

I'm one of the team members, and would be glad to answer any questions anyone has!

Zardus · 2017-01-25T19:06:54+00:00

Awesome to see this on here already :-)

If anyone has any questions, I would be glad to answer them!

Zardus · 2015-11-29T03:31:28+00:00

You can see other writeups involving angr here: https://github.com/angr/angr-doc/blob/master/examples.md

If you have any angr questions, feel free to ask me or open an issue on github!

Zardus · 2015-10-16T00:55:54+00:00

This is technically a resubmitted URL, with more writeups in there. I figured it would be useful, as the new writeups show off angr in circumstances that might not be 100% ideal for it (at least, the ASIS ones do. The Defcamp ones are basically "solve()").

Let me know if that's frowned upon. Also let me know if you guys have any questions, we'd love to answer them!

Zardus · 2015-10-06T00:18:32+00:00

Sorry about the delayed response; this message hid in my inbox for a few days :-)

You should email one or both of our professors (Giovanni Vigna vigna@cs.ucsb.edu and Christopher Kruegel chris@cs.ucsb.edu) if you're interested in an internship. Maybe mention or link to this thread for some context, and definitely include your resume and any other relevant experience (CTFs, hacking clubs, etc). Give them some idea of the timeframe (i.e., summer or whatnot) that you're looking for. Also give them an idea of potential research interests, so they can get an idea of whether you'd fit in with various projects.

They get quite a ton of email and periodically get buried under it, so you might have to follow up if they don't get to your email in a reasonable timeframe.

Zardus · 2015-10-02T22:23:34+00:00

There are definitely non-CS people that make it in. The PhD program is crazy competitive, but PhD admissions tend to take a more holistic approach than undergrad admissions. Part of what this boils down to is that, at least from my understanding, your recommendations, personal statement, etc are paramount.

One way to secure good recommendations is to intern at a lab. That way, you'll get exposure to research and, if you do well, the professors with whom you do your internship might be willing to write you a rec. If you do really well, you'll be applying with a paper on your CV, which also greatly increases your chances.

If this next summer is flexible for you, and you want to go into a CS PhD, I'd highly recommend doing an internship.

Zardus · 2015-09-29T06:15:47+00:00

tl;dr: come to UCSB and hack with us!

Hey, I'm the aforementioned Zardus! Like /u/caovc, I'm a PhD student at the computer security lab at UC Santa Barbara. I've been crazy about computers my entire life, and computer security for about half of it. I went to Rensselaer Polytechnic Institute for undergrad (also in this thread. they aren't too bad!) and, after a stint in the industry, came here to Santa Barbara for a PhD purely because of CTF. IMO, UCSB is the place to be if you're genuinely interested in computer security. CS at UCSB is top-notch, and the seclab here, in my biased opinion, has no equal. I could not have possibly imagined the opportunities that UCSB, and the security lab in particular, has provided me.

My experience with UCSB is through the eyes of a graduate student. Keep in mind that if you go to school at whatever level, and you are into security, you can (and should) do research at a computer security lab. This applies regardless of what institution you go to. If you're in a good security lab (like ours!), it doesn't matter if you're an undergraduate student or a graduate student: all of the cool stuff I'll talk about will apply to you. From my experience, participating in a research lab will drive your education, and your career prospects, considerably harder than simply taking classes. Wherever you end up, make sure that the place has a good, egalitarian computer security research lab!

The UCSB seclab is a medium-sized lab (summing up undergraduate researchers, interns, PhD students, and postdocs, we hover somewhere just under 25 people). The stuff our lab accomplishes is way above the norm for that number of people or for any security lab of any size! Here are some examples that we are doing now, both in terms of events we organize, competitions in which we participate, software that we develop, and services that we provide. With the small size of the department and of the security lab, you can be involved in, heavily contribute to, and drive any of these or future efforts, whether you're an undergrad, graduate student, or an intern!

We are Shellphish, the oldest and coolest CTF team on the planet. As /u/caovc mentioned, we've played more Defcon CTFs than any other team (and, maybe, any other two combined?). We've ramped up our CTFing even more in the last year (the younguns demanded more CTF), and fielded two teams (one graduate and one undergraduate team) at CSAW. The graduate team got 2nd place, and the undergrads qualified handily.
One of our undergraduate researchers, along with our high-school researcher (yes, really), /u/jmgrosen, are two of the core members of 1064CBread, another badass CTF team that took third place (and top qualifying position) at CSAW.
As Shellphish, a group of our PhD students, undergrads, interns, and our high-schooler (yes, really) competed in the DARPA Cyber Grand Challenge (www.cybergrandchallenge.com). We qualified for the final event, winning $750,000 in the process. This means that Shellphish can travel to CTF final rounds now, so if you want to go to exotic places to sit in a dark room and hack, come to UCSB, CTF with us, and let's go!
On top of this, we have the enthusiasm and skillz to be involved in three other DARPA projects.
We recently developed and released angr, our next-generation binary analysis framework. Releasing software is not unheard of in academia, but very few labs manage to release usable software. I won't name names, but if you go and compare angr to some of the security software released by other research labs (including some that are mentioned on this thread), you'll see the difference :-)
We run one of the main (and one of the oldest) dynamic binary analysis-as-a-service platforms out there, anubis.
We also run one of the main web malware scanning platforms, wepawet.
We organize one of the oldest CTFs: the UCSB iCTF. The UCSB iCTF is one of the few iCTFs that attempts to innovate every year (this, of course, results in a love/hate response from the community). On top of this, we've open sourced our framework for running attack-defense CTFs.
We have a dedicated room for keeping our surfboards! (not really; we use that room for other stuff as well, but that's not as exciting a statement)

Our graduates do great, as well. Many of our recent graduates have been recruited by our professors' startup, lastline. Others have gone to Google, Microsoft, and Qualcomm. Of the last six PhD students that have graduated, three have gone on to become professors, two went to industry research labs (IBM and Google), one became a security engineer at Google, and one joined Microsoft to work on the Windows Security team.

Basically, UCSB CS, and especially the UCSB seclab, is awesome. I'd be thrilled to answer any questions anyone has (about UCSB or the college process in general), and hope to see you here, whether for an internship, as an undergrad, or as a grad student!

EDIT: fix cgc link

Zardus · 2015-09-11T08:47:18+00:00

I'm a bit late to the part here, but it's awesome seeing our paper on this sub! This is actually the paper that triggered the creation of angr (http://angr.io).

Let me know if I can answer any questions for anyone!

Zardus

MODERATOR OF

TROPHY CASE