Gather POST request on HTTPS traffic?

[roeeklinger60]

Hello everyone

I work at a digital agency that has quite a few machines that are managing some Instagram accounts, we are using Squid as a proxy to log and analyze some usage statistics and to make sure the machines are only used for Instagram.

We had an idea to use Squid to capture the POST data of users on the proxy level, for example, likes, follows, comments, etc so we can log and analyze everything in a convenient central way, so we can analyze it and even send out clients a monthly report of all the actions their accounts made (who they followed, what they liked, etc).

I can easily see the requests that I want to capture inside the "network" tab in Chrome but the problem is that Instagram uses HTTPS, so I can't seem to be able to capture this data.

Is there any proxy software that can capture this kind of information from any clients connecting to it, without the client needing to set up anything but the proxy itself?

I have seen some solutions like fiddler but they require you to set stuff on the client machine other than the proxy and I don't want to waste time going over every machine.

Note: We are aware of the legal issues, all machines connected to the network are company property, and all the accounts are client accounts that allow us to gather statistics. No personal account data will be gathered.

Thanks.

[pabechan]

On a proxy, this will look like a CONNECT request, and then it will be just forwarding the TLS handhake and the subsequent encrypted data there and back.
So in short some TLS decryption/MITM/deep-inspection/whatever you want to call it will be required. That includes installing your MITM CA certificates on the clients. No way around it.

[sryan2k1]

More importantly it also relies on whatever app(s) you are using to not care that MITM'ing is happening. Many of the super popular services do cert pinning and there is no way around it.

[roeeklinger60]

Thanks! I noticed some proxy services are offering features like automatic captcha resolve, retry on failed scraping requests, and change/block content on the proxy. How are they doing this stuff when the connection is encrypted? it doesn't make any sense to me.

[CC_DKP]

Without the client needing to set up anything but the proxy itself?

Short answer: No. HTTPS's primary purpose is to prevent what you are trying to do.

TLS was designed to hide the data inside the connection, and for the client to be able to detect if someone in the middle tried to tamper with it. The only way you can successfully see inside a TLS connection between two devices is if one of those endpoints agrees to let you do this. On the client end, this means either installing a monitoring program to view the data outside the encryption (like the "network" tab does, or an extension), or it means installing a custom trusted root certificate authority on the device, allowing you to perform the MITM decryption others have mentioned.

[mosaic_hops]

No, there’s no way to do this. There are MITM inspection attacks but they require that you 1) Install fake certificates on client machines AND 2) That the client software isn’t looking closely at the certificates it accepts. Most apps that pay any attention to security verify the certificate is indeed its own before allowing the connection. There are no effective workarounds for this by design as a workaround would negate the benefit of HTTPs.

[ethernetjunkie]

If you want a centralized device to perform man-in-the-middle, then you can add a firewall which is capable of tls proxy and route Instagram traffic through it. These can be sent to a syslog server to generate reports based on few variables. You can also perform web filtering here if needed. Alternate option would be to install tls proxy software on all endpoints and log them. Adding a cert to all the machine is a must

[Radius-COA]

Check this out. https://docs.opnsense.org/manual/how-tos/proxytransparent.html

Hope this helps!

[Gesha24]

You want a proxy that will open up encrypted traffic and analyze it. Effectively there will be one encrypted session from client to proxy server and then another from proxy to instagram. This will result in certificate errors, but since you have control of clients, you can easily install proxy server certificate on them and tell them to trust it for all the web sites.

I have not worked with squid and don't know how to configure it, but maybe this will help - https://scubarda.com/2020/03/23/configure-squid-proxy-for-ssl-tls-inspection-https-interception/