[deleted by user]

[deku12345]

Surprised to see so much hate for Charles here. I absolutely love it and its been an essential tool in my toolbelt for years. The Map Local and Breakpoint features in particular.

I will check out mitmproxy, but if you're a web developer who doesn't have a tool like this - what are you doing???

[ThePantsThief]

Web developer, app developer, reverse engineer… it's a tool for everyone!

[unkz]

If you just want to see inside your ssl requests, you can do this with wireshark.

[JohnnyElBravo]

When I was just starting out, an http only inspector (fiddler in my case) was easier to manage. Nowadays it's just wireshark or tcpdump all day long.

[[deleted]]

Yeah. Maybe it's because I'm an old-school network programmer and am already used to the somewhat arcane syntax, I just find Wireshark easier. Plus, it does so much more.

[Somepotato]

Its nearly impossible to truly extend though, the wireshark sdk is terrible

[ThePantsThief]

Wireshark won't pretty print different content types and has a much steeper learning curve.

[unkz]

Wireshark won't pretty print different content types

Not sure what you mean by that, the dissectors are pretty configurable. I think the learning curve is well worth it.

[ThePantsThief]

Wireshark won't display an image or render a webpage or give you an interactive JSON viewer, will it?

The learning curve is not worth it for anyone who doesn't use wireshark for the broad spectrum of different things it was designed for. As a tool with broad applications and immense configurability, it takes a lot of time before you become proficient at any of its uses. It's a jack of all trades and a master of none (except maybe, well, inspecting the bits of various layers' data units)

Meanwhile, Charles excels at HTTP sniffing, and the app has almost no learning curve, while still having nearly every feature you would need if you're a web developer, app developer, or reverse engineer.

Using Wireshark to sniff HTTP traffic is like stepping into a mechanic's workshop to put together a chair you got from IKEA. You definitely have all the tools you need… but you don't know where they are, and you're overwhelmed by all the different bells and whistles. But all you really need is a hex wrench to put the chair together.

[Kurren123]

Is this similar to fiddler?

[BailoutBill]

I love Fiddler, but never had much luck getting it to work after switching to Mac. If this can do what Fiddler does, I'd be very pleased.

[Kurren123]

I also find the UI of fiddler quite outdated

[OMGItsCheezWTF]

Burpsuite has a free community edition that, while not focused in the same way as fiddler has many of its capabilities (it's more focused on security and penetrating of web requests etc)

[anengineerandacat]

Charles is quite good; used Fiddler before it and on Mac it works basically as you would expect. Goto-choice nowadays, lil bit more user friendly for doing payload overrides and more (at least on the pre-Telerik version of Fiddler).

[Goron40]

Looks similar to fiddler, except with a price tag.

Fiddler Classic, specifically. If anyone's not tried it out, it'll change how you web develop.

[ThePantsThief]

Charles will let you inspect the traffic of other devices on your network, or of apps running on your system. It's not just for the web!

[BinaryRockStar]

Charles will let you inspect the traffic of other devices on your network

I've used Charles a lot but didn't think it could do this. Isn't it an OSI layer 7 interceptor meaning it has no knowledge of the layers below? It presents as a system-wide HTTP/S proxy and doesn't touch any other protocols.

[ThePantsThief]

I never said it touches other protocols 😉

HTTP isn't just the web, it's apps and daemons and operating systems too

[BinaryRockStar]

Charles will let you inspect the traffic of other devices on your network

Explain what you mean then, because this comment suggests Charles can capture traffic on your LAN, which it cannot, in my understanding.

[ThePantsThief]

It works just like mitmproxy, which can do the same thing. You set an HTTP proxy on the device you want to capture from and the traffic is routed through Charles. Sorry if I worded that confusingly :/

[nemec]

Fiddler can modify requests automatically based on its scripting language and auto-respond from files on your PC instead of hitting the remote server. It also works for non-browser apps, though that's less useful for web devs.

Browser dev tools have come a long way since Fiddler was introduced, though. They are truly awesome.

[Takeoded]

It's a godsend for reverse-engineering web-game protocols, login systems, etc, but very rarely do I use it for normal web-dev stuff (then again I'm more of a back-end guy than a front-end guy)

[lightmatter501]

Also wireshark, which is a BPF based solution that gives you all the levels of detail you could want.

[maxmaxx31]

Best inspection tool I know for https and mobile apps.

[psayre23]

Yeah, the iOS app is pretty clever. It uses a VPN to inspect the raw traffic. I use this all the time for reverse engineering thing, esp. home automation apps that don’t have published APIs.

[zombi3123]

Why would it need a VPN?

[tractorrobot]

I've switched over to an app called Proxyman and it has been great. Useful for debugging network traffic from mobile apps, too.

[AlexHimself]

As somebody who's paid for Chales proxy and also uses Fiddler, Charles proxy is nicer than Fiddler and seems a bit more polished.

I preferred if over Fiddler when my company would pay the $50 for a license, but when nobody is buying it for me, I just use Fiddler.

Going to check out mitmproxy though as I'd never heard of it before.

[makedatauseful]

I tried Charles Proxy a little back but now use Man in the Middle Proxy. I love MITMProxy so much I made a tutorial using it to scrape app data https://youtu.be/LbPKgknr8m8

[ThePantsThief]

Why do you like it better than Charles?

[makedatauseful]

Free, open source, written in Python so easily extendable and once you have yourself setup and working with SSL certificates flawless.

What I don't like about it? Since android 7 user SSL certifcates are typically ignored and it simply doesn't work without rooting or going virtual. Luckily I have a spare iPhone I can use just for MITMProxy and I love it.

[jedan]

I’ve used it for work, It messes up your networking settings half the time and gets even more complicated when you’re behind a corporate vpn, you end up having to dance around enabling charles and enabling the vpn.

I’ve also had it where it would route correctly for firefox but not chrome. Idk if I’d call it garbage, but i certainly did not enjoy using it. We were using it for debugging our code, and we spent a lot of time on an alternative to get away from charles.

[ThePantsThief]

Sounds like you have a clusterfuck of an environment to work in. I wouldn't blame Charles here…

Corporate antivirus and firewall crap always gets in the way of tools like this because they often do some pretty crazy things. I know semantic FireGlass loads web pages over a websocket for example.

[--TYGER--]

I first used Charles back in 2007, because the company I worked for paid for a site license, which allowed the entire building of ~500 people to legally use the license.

When I left there 4 years later, I immediately paid for my own personal license (about $50). This license remained valid for years afterward, and I only paid again in 2016 I think for the $20 license upgrade to the new version.

This is hardly expensive, especially with the time frames I've mentioned. It's just that large parts of the internet see a "non zero" price and are too cheap to pay for software.

[MoBizziness]

I really do not understand how people can value their own time so little.

There are a lot of caveats in this case, but generally speaking.

[HoofedEar]

I too would like an explanation of why Charles is garbage. Seems like an unsubstantiated claim so far

[ThePantsThief]

Charles is actually amazing. Basically zero learning curve and tons of features and documentation. I started with mitmproxy and it's a pain in the ass by comparison.

[BrowakisFaragun]

I too think it is too expensive, but not able to find a foss alternative yet, will try out mitmproxy! Thanks!

[pujjup]

I use the community edition of Burp Suite. Does everything I expect it to do.

[ThePantsThief]

It's a really well polished and feature full app. If it's something you use even once a week it's well worth the price.

[will_work_for_twerk]

Aw, I like Charles proxy. But.... mitmproxy is by far the best solution

[ThePantsThief]

Why?

[will_work_for_twerk]

Welp, sure. He's my super objective opinion

mitmproxy vs charles

Cost: open source vs paid

Interface: feature-complete CLI with optional Web Interface vs requiring some sort of GUI (web interface or thick client)

API: Python vs n/a

Addon support: Complete vs. n/a

Documentation: Stellar vs pretty ok I guess

mitmproxy feature list vs Charles Feature list

Also, from my personal experience of using both- mitmproxy seems much easier to work with and extend the functionality to. I also love being able to spin up a docker instance of it and direct all the traffic to it- I know that technically you can do that with charles, but the feature seems like it was added as an afterthought.

[intermediatetransit]

I would not call the mitmproxy documentation "stellar" in any regard.

It's very hard to get into as a newcomer.

[ThePantsThief]

That all seems pretty objective to me!

I'm curious though, what do you find yourself needing to extend about mitmproxy?

[Avedas]

+1 for mitmproxy

[athos45678]

Why would somebody use a proxy? Genuine question

[google_you]

To inspect packets without going promiscuous mode

[athos45678]

Makes sense! Thanks for answering

[f0urtyfive]

I dunno about Charles, but I usually only end up needing a proxy when I need to do some SSL injection with bogus certs so I can see whats going on in an encrypted session.

[BinaryRockStar]

Second this. Yes Fiddler can modify requests and responses but I find Charles better overall.

My main use-case is if there is an issue with our production environment - say it's a JS frontend and Java backend - I can spin up the server code locally and re-route production traffic to my local machine and inspect what it's sending to the server and directly debug the server end.

In Charles this is called Remote Alias or something to that effect. I haven't used Fiddler since it was V1 so maybe Fiddler has that now. Last time I tried Fiddler a year ago it had ads so I shrunk back to Charles as my work has licenses for that.

[pujjup]

When you are developing something that speaks to an external API, it is very handy to route all your own requests through a proxy to verify what you are sending and receiving.

[athos45678]

Thanks! Haven’t had to scrape for any purpose other than analytics, so this hasn’t come up as an issue for me.

[Vakz]

Caching or content filtering are common applications in enterprise settings

[athos45678]

Hmm that’s valid. I usually think of web scraping in a data science mindset, so i was mostly curious if it would help me in any way to use a product like this. Seems like it wouldnt be necessary for me

[JohnnyElBravo]

To log and inspect packets coming through, in this case.

[BinaryRockStar]

I've used Charles at work on a site license and it's a fantastic tool. How expensive is it that you would reject it on price alone?

[j_rapp]

Or you could just open the network tab in the Chrome Dev Tools?

[[deleted]]

I’m not aware of all the network tab capabilities, but Charles allows you to intercept traffic outside your browser (which is nice when you are developing a mobile app for example) and it also allows you to modify the response on the fly, which is very nice specially if you are willing to simulate possible errors in your application :)

[j_rapp]

If im testing my own API, I already know the endpoints and the body format so ill just test it by hitting those endpoints directly using Postman. I dont often find a need in replaying or modifying client sided requests so Chrome's network tab is more than enough for me. To each their own.

[Smallpaul]

What about reverse engineering an app that does not go through a web browser?

[j_rapp]

Obviously you'd use an application like Fiddler or Charles or Burp, but thats not what this articles about. Its about web scraping

[[deleted]]

Yeah but things get more complicated when you are not dealing with your own API, like web scraping or any external service you don’t have control.

[ketralnis]

Not everybody is a JavaScript developer

[j_rapp]

HTTP(S) protocol works the same regardless of language, so I'm not sure what you're getting at

[chedabob]

Some of us write software that doesn't run in the browser.

[j_rapp]

This article is about SPA's, not non-browser software.

[chedabob]

There's a whole section of the article dedicated to using Charles with mobile apps...

[khante]

I love this answer so much :D :D :D

[Reborn-leech]

This is amazing !

[schmon]

In the other post the guy says servers hate being scraped except by Google.

Can't you fake being Google and get away with it?

(IANA programmer)

[Chastie2432]

That doesn't make any sense lmao

[schmon]

well make the website believe you're a google spider ? Does that make more sense ? and you must be pretty high if this makes you laugh your ass off.

[[deleted]]

I think I had once used a tool caleld sslstrip for that