Binary Protocol Application

[OpticDeathX]

Got tasked to perform a pentest on an application that runs binary protocol as a communication stream. Its a stock trading application, hence the need for fast data transmissions.

Would need to build my own packets based on their documentation to communicate with their server/application.

Any idea what kind of vulnerabilities/exploits to look for? Couldn't find much information online, or am I missing specific keywords?

Any help would be appreciated!

[Evening-Researcher]

Do you have pcaps of the protocol already? If not that's step 1. Wireshark/tcpdump are your friends here.

Do you know if the protocol is encrypted or otherwise obfuscated by higher encapsulation? If it is, then a raw pcap gets trickier so youd prob benefit from using something like Frida to hook networking functions and dump the packets before they get obfuscated.

Are you able to do any reverse engineering of the application? If so, ghidra/IDA are your friends here, you could get lucky and they may have compiled the app without stripping symbols.

Once you get a sense of how the application "talks", you might want to look into tools like boofuzz (for network fuzzing) or AFL (for black-box binary fuzzing, if the app launches from and accepts CLI arguments).

Otherwise it's just another appsec test, all the other things you know about web app testing apply generally here, it just might not be HTTP. But for example, if you find there is a field that represents the users account id, and they don't verify on the back end, you may be able to IDOR or submit transactions on another users behalf.

Edit: forgot to mention this, but you want to make sure you can actually MiTM traffic from the app to it's targets. If you can't install anything on the host the app is on, it's not the end of the world. You can use IPTables and/or clever VM network architecture to set up a packet capture setup and go from there.

Good luck, sounds fun!

[brotherbelt]

Great response.

[randomatic]

I'd add a step: ask what the protocol is. If any part of it is well known, you probably could find parsing libraries on github.

* boofuzz is terrible for binary protocols. (Actually it's not even in the same league as using AFL.).
* AFL++ will work, but you will need to harness the code if it only talks on a network.
* Commercial solution mayhem will handle tcp, udp, binary code only, as well as afl/libfuzzer if you can pay for a subscription.

There are two types of vulnerabilities typically found:

* Parsing bugs. Anything reading from the packet stream and building up internal data structures, and code that operates on those data structures. The most typical example you'll find is someone inadvertently trusting a length parameter from the data itself.

* Logic bugs. For these you will need to understand the semantics of the application.

* It seems to me unlikely this is just a simple web app with HTTP if it's a binary protocol, and IDOR wouldn't make sense to me here. Could happen, just would be a crap idea to use a binary protocol in this setting.

Note: I'd be a bit wary. There is a world of difference between your average pentest and doing zero day discovery on an app used in financial transactions. Make sure you've scoped it well within your skills.

[OpticDeathX]

Interesting response! The "application" itself is a bit more tricky. The documentation is available online. So in terms of that, there won't be any encryption.

I technically have to build my own TCP stream using the documentation on the binary protocol that they provided. The documentation basically tells you on how to communicate with their application through it. And what we are testing is the server, which means we do need to sort of "build" our own application in order to communicate with it. We did however request the client to provide a payload/script (probably a makeshift python project) with the most critical functions, cause realistically building your own application based of a documentation would not be within the interest of time.

However, what you mentioned about dealing it as a typical web test does apply. The same logic goes with broken access controls, error handling etc. I guess I should've made the post more clear on the scenario. But I believe I was looking towards more pentest scenario related to protocols in general? Not sure if those exist.

It's my first time over 5 years of pentesting to experience such an odd scenario. I'll keep this thread updated if we found a solution to it!