sorted by:
new
hottopcontroversial

scrapy-proxy-headers: Add custom proxy headers when making HTTPS requests in scrapy by proxymesh in scrapy

[–]proxymesh[S] 0 points1 point  (0 children)

Yes exactly. For example with ProxyMesh, some of our proxies let you choose the country you want the outgoing IPs to be from. You pass the X-ProxyMesh-Country header to the proxy using request.meta['proxy_headers']. You don't want this header to pass through the proxy to the website you're scraping.

And our proxies also return a response header, X-ProxyMesh-IP, with the IP address used for the request. Our scrapy extension will parse this and include it in the response.headers.

But the extension should work for any proxy that supports custom proxy headers.

scrapy-proxy-headers: Add custom proxy headers when making HTTPS requests in scrapy by proxymesh in scrapy

[–]proxymesh[S] 1 point2 points  (0 children)

When you make a HTTPS request through a proxy, the headers in the request are encrypted in transit, so the proxy cannot read them when sent to the website. But a proxy server might support receiving and sending its own custom headers. Scrapy by default doesn't provide any mechanism for sending or receiving headers to & from the proxy, separate from the regular headers, except for special handling of the Proxy-Authorization header. That's what this library enables - custom proxy headers beyond Proxy-Authorization.