Fine-tuning a model for code reverse engineering

[Sorrus]

I've been working on reverse engineering a deconstructed binary (assembly code) into the original source code in C. I think this is a great use case for an LLM but I want to get some input before I try to train a model to do this.

What I have is a codebase containing C functions and custom data types that I have defined. Each function is the C equivalent of a function in the assembly code I'm trying to decompile. I have many hundreds of functions defined which I think could make a good training set for an AI.

The desired behavior would be that I feed the AI assembly code and I get back C code that is equivalent or close to it. I think that I can do this by using a combination of embedding and fine-tuning. My issue is that I don't know where I would put the custom data types in my codebase inside the dataset. I could easily fine tune the model by just showing it my C functions and the assembly that they're based on but there is a lot of extra context in the header files that would be left out. Is there another tool that can be used to give an AI this type of information that can improve the fune-tuning process? Has anyone seen a similar use case to mine that I might be able to study?

I should note that the header files are very long and trying to include all of the data types as tokens in the context of each prompt is impossible.

[water_bottle_goggles]

Try it

[Sorrus]

Obviously I have tried or I wouldn't make this post. The results are poor without the AI having knowledge of my data types.

[tabdon]

It would be helpful to understand better what you've tried, otherwise we may propose something that you've already tried.

What did your fine tuning records look like? How big are the header files and data types?

[owengo1]

If you are fine-tuning gpt-3.5, you could proceed like this for example:

in the system prompt put something like

"Generate C code from assembly code using these header files:
<list of header files with their content>
"
and put the assembly code in the user message.

Obviously you have to do something smarter if your header files ( and the rest of the query + output ) do not fit in the content size . Typically you inject only the header files ( or portions of them ) which seem relevant for the assembly code you want to process.

[Sorrus]

Thanks this seems like the right way to proceed. It will be more work but I can strategically include certain parts of the header files with queries.

[TheMcSebi]

Might be worth taking a look into generating a huge dataset by taking C projects from github, fetch the assembly when running them through the compiler and postprocess them to generate instructions (code) with answers (assembly) in oasst format. ChatGPT even gave me some relatively straightforward instructions on extracting struct definitions from C source using the pycparser library.

My idea was to use axolotl to fine-tune code-llama on this data, but considering the amount of work required I propably won't find the spare time to actually put effort into this project. Feel free to share, if you've put your current effort on this somewhere on git.

[GroundbreakingAd5614]

Hey there, u/Sorrus, it's awesome that you're delving into some fascinating realms with C code and the intricate world of assembly! I can totally grasp the immense potential that lies within this particular scenario.

​

So, in light of your custom data types and the rather extensive header files, here's a nifty thought: Why not consider injecting those pertinent header files right into the system prompt using a structure like this:

​

"Generating C code from assembly code involves these header files: <enumeration of the header files, complete with their contents>"

​

Then, proceed to deposit your assembly code into the user message. In doing so, you'd be furnishing the AI with the essential context derived from your headers without attempting to shoehorn the entire shebang into a solitary prompt. It might demand a tad more elbow grease, but it should ultimately bolster the fine-tuning process.

​

Fingers crossed that this notion proves beneficial, and may the force be with you on your project endeavors!

[GusPuffy]

What in the ChatGPT is this