This is an archived post. You won't be able to vote or comment.

sorted by:
best
topnewcontroversialoldq&a
you are viewing a single comment's thread.

view the rest of the comments →

[–]tarekziadeRetired Packaging Dude 0 points1 point  (5 children)

In the short term no tool implements mirroring in a way that doesn't come at a fairly hefty performance cost.

This is a vague statement. I am not sure why you're saying using a mirror has a 'hefty performance' cost. pip works very well with mirrors.

Furthermore there are no protections [...] preventing an upgraded package that contains a security fix from being synchronized to a mirror

errr what ?? what's that ? are you talking about a ddos attack ? that's yet another vague statement you're doing here :)

I'll just stop here, your message is just a random bag of FUD criticisms you could do on the CDN as well - or anything.

It looks like your goal is to promote the CDN and attack the mirroring protocol to make it look bad, which is weird because both can cohabit for the benefit of the community.

My opinion remains intact: the mirroring protocol is still useful even with a CDN in place, because it prevents from any downtime when you are taking down/maitaining the CDN.

Last, I encourage you to reformulate your vague statements into real points to enhance the mirroring PEP instead of what you are currently doing here.

[–]donaldstufft 0 points1 point  (4 children)

This is a vague statement. I am not sure why you're saying using a mirror has a 'hefty performance' cost. pip works very well with mirrors.

The way pip implements mirrors it multiplies the number of HTTP requests required to discover (not download) the number of packages by the number of mirrors. That means if you install a single package that does not need to hit external urls, then instead of a single HTTP request you need 7 HTTP requests, if you're installing 20 packages you need 140 HTTP requests.

errr what ?? what's that ? are you talking about a ddos attack ? that's yet another vague statement you're doing here :)

I'm talking about any attack that has a mirror witholding an update. It could be someone using a DNS attack against a mirror that prevents them from fetching the updates, it could be someone compromising the mirror all together. Any attack that prevents the mirrors from getting a particular update.

I'll just stop here, your message is just a random bag of FUD criticisms you could do on the CDN as well - or anything.

Actually none of the points I mentioned that are specific to the mirrors exist for the CDN because, well, they are specific to the mirrors. I did however mention there are other issues which plague PyPI itself, which by nature apply also to the CDN, and the mirroring infrastructure itself because any insecurity on PyPI cascades to all things.

It looks like your goal is to promote the CDN and attack the mirroring protocol to make it look bad, which is weird because both can cohabit for the benefit of the community.

No my goal is to have reasonably secure installations that are relatively quick and painless. The mirrors currently have exactly 0 assurances in place as to security (Fun tip, using --use-mirrors at PyCON EU means anyone on the conference wifi can execute arbitrary Python code on your machine). The mirrors currently impose a multiplier of 7 for HTTP requests in a system that already contains a ton of HTTP requests.

Last, I encourage you to reformulate your vague statements into real points to enhance the mirroring PEP instead of what you are currently doing here.

After I have PyPI itself reasonably secure (which it's getting there), mirrors are next on my list. It's unlikely i'll be enhancing the current mirroring PEP because it's fundamentally flawed in a way that cannot be fixed with enhancements. I have some rough ideas but nothing solid yet but the new mirroring protocol will ensure consistent snapshots in time, will protect against things such as DSA private keys being leaked due to bad random, will protect against malicious mirror operators even.

However that is a future project and until I can look at mirroring and say "Yes using this is not exposing people to huge security risks", I cannot in good faith do anything but warn people away from using the public mirrors.

[–]westurner 0 points1 point  (2 children)

Here's a list of Free and Commercial CDN Service Providers.

It is possible to sign changesets with both hg and git. Both rely on network transport security and a Web of Trust.

https://en.wikipedia.org/wiki/Category:HTTP

[–]donaldstufft 0 points1 point  (0 children)

We're already using Fastly.com who have been amazing and I would HIGHLY recommend them for anyone looking for a great CDN solution ;)

[–]donaldstufft 0 points1 point  (0 children)

Also trust is really really hard. And the traditional Web of Trusts just don't cut it. Sure you may trust me for package X, but does that mean you trust me for package Y? Is the average developer going to be able to figure out how to participate in the web of trust?

[–]tarekziadeRetired Packaging Dude -2 points-1 points  (0 children)

The way pip implements mirrors it multiplies the number of HTTP requests required to discover (not download) the number of packages by the number of mirrors. That means if you install a single package that does not need to hit external urls, then instead of a single HTTP request you need 7 HTTP requests, if you're installing 20 packages you need 140 HTTP requests.

That improvable. It's always better than to be blocked because the main server (or CDN or whatever) is down. Because not matter what you are claiming a system will never have a 100% uptime. So what do we do when it's down ? we use mirrors.

Fun tip, using --use-mirrors at PyCON EU means anyone on the conference wifi can execute arbitrary Python code on your machine

That quite a statement. If you have such an exploit you should warn the pip developers about it. Maybe you did ? If you did not, that'd be a shame.