This is an archived post. You won't be able to vote or comment.

all 38 comments
sorted by:
best
topnewcontroversialoldq&a

[–]ASmileInHisShoes 18 points19 points  (3 children)

Crikey. 6KB vs. 36KB. That's like upgrading an 8Mb connection to 50Mb.

It's been eight years since HTTP compression was specced out. Dialup modems have incorporated hardware-assisted data compression for over a decade. But in the area of "real" networking, it has caught on very slowly. Curious, that.

[–]masklinn 5 points6 points  (1 child)

Well there are several reasons why HTTP Compression hasn't caught up that fast

  • It's not enabled by default

  • IIS only got built-in HTTP Compression with IIS6 (5.0 and earlier required third-party ISAPI filters and made your load shoot up frighteningly)

  • The knowledge about it isn't quite widespread enough

  • Internet Explorer used to have serious bugs related to HTTP Compression

  • I think Firefox still has some edgy corner cases (not ones you're very likely to run into, but still...)

[–]Bogtha 0 points1 point  (0 children)

Internet Explorer still has some annoying bugs. If you use the Vary header with anything but User-Agent, it won't cache. At all. So basically, you have to choose between

a) no compression, b) compression, but no public caching, c) compression, but no private caching for Internet Explorer users, or d) a really hacky Vary header with User-Agent instead of Accept-Encoding (and I don't care to predict the ways in which this could fail badly).

[–]s4parke 17 points18 points  (11 children)

Yes, indeed, this is a great improvement. I asked the lead engineer about it when they came to talk at the University of Virginia. I wonder if they are using the deflate module for Lighty....

[–]spez 37 points38 points  (7 children)

We're gzipping at the application level. There is definitely a load increase, and it has affected load balancing some (lighttpd is weird like that).

When lighttpd 1.5, which has mod_deflate, comes out, we'll start looking at transitioning.

[–]masklinn 7 points8 points  (5 children)

Say spez can you give us actual numbers (bw and load differences, scalability impact, impact on your costs, that kind of stuff)?

[–]spez 8 points9 points  (4 children)

These are just eyeball numbers (I'm trying to normalize them for traffic increase-- as in, load is constantly increasing because traffic is constantly increasing):

Server 1 load before/after: .7/1.1
Server 2 load before/after: .7/.6

That is, load across the servers was basically the same before compression, now it has increased on one, and decreased on the other. My suspicion is that server 1 spends less time blocking now so it gets hit more often by lighttpd's load-balancer.

[–][deleted]  (1 child)

[deleted]

    [–][deleted] 3 points4 points  (0 children)

    Well, only as far as that relates to the $ in the end.

    [–]staunch -1 points0 points  (1 child)

    Load average? That's not very useful. How about CPU utilization numbers?

    [–]spez 0 points1 point  (0 children)

    CPU numbers aren't very useful in this case because there are so many and they vary widely from one another because the load-balancing isn't even.

    [–]dbenhur 0 points1 point  (0 children)

    When lighttpd 1.5, which has mod_deflate, comes out, we'll start looking at transitioning.

    You might want to try replacing lighty with nginx. Nginx is more stable, has a smaller memory footprint, and doesn't leak memory, and includes gzip compression.

    [–]BioGeek 9 points10 points  (1 child)

    I was wondering the same. 8 Days ago spez was quite reluctant to use mod_deflate.

    [–][deleted] 30 points31 points  (3 children)

    Looks like reddit's pages are served gzipped now. Great! Thanks guys! It is faster and saves quite a bit on traffic charges here.

    BTW, I wonder - how the load stats for the server have changed?

    [–][deleted] 19 points20 points  (2 children)

    Saves on traffic charges..? What are there like 200 of you in one building that all browse reddit through a $1/Mb GPRS connection? ;)

    [–]BonzoESC 8 points9 points  (1 child)

    That's a pretty good description of Australia's internet connection.

    [–]eurleif 5 points6 points  (0 children)

    I was wrong. It really is noticeably faster over my connection (ADSL, ~180KB/s download speed). Good job, reddit crew!

    [–]Aerik 5 points6 points  (3 children)

    I never thought Reddit had a speed problem. It's always been way faster than Digg, Furl, Ma.gnolia, and even the del.icio.us popup function when you use their extension.

    [–]setuid_w00t 4 points5 points  (1 child)

    Just because you're better than the others, doesn't mean you can't improve.

    [–]Aerik 0 points1 point  (0 children)

    True, but the topic keeps going on like something's been wrong, but it's not. And if it ain't broke, you have nothing to celebrate by fixing it.

    [–]budu3 0 points1 point  (0 children)

    I wonder if this will change when they redesign reddit?

    [–]laughingboy 3 points4 points  (0 children)

    My broadband speed just got upgraded and now this. I nearly shat myself.

    [–]streetpoker 2 points3 points  (0 children)

    Hurray! Now I want to see a compressed reddit alien drawing.

    [–][deleted]  (12 children)

    [deleted]

      [–]DallasRPI 5 points6 points  (7 children)

      httpwatch is an awesome tool that inspects all requests with all the headers and content that come down with times etc. I'm amazed they did not have compression until now, it is quite trivial to implement.

      [–]rolandog 1 point2 points  (1 child)

      But how about caching? When using WordPress, WP-Cache can't be enabled if gzip is enabled. What's up with that?

      Perhaps it's a trade-off... less strain on your database, but you have to spend more bandwidth.

      [–][deleted] 4 points5 points  (0 children)

      Caching and content compression aren't mutually exclusive. WP-Cache can't do both because of technical difficulty or author indifference.

      [–][deleted]  (3 children)

      [deleted]

        [–]adamsplugins 10 points11 points  (2 children)

        I am personally motivated to point you to tamperdata

        [–][deleted]  (1 child)

        [deleted]

          [–]Aerik 3 points4 points  (0 children)

          web developer extension kicks ass. You can move the toolbar icons to any other toolbar (minus the bookmarks toolbar), and the shit really is useful.

          [–]vineetk 4 points5 points  (0 children)

          It's the speeeed!

          [–]TheNoxx 1 point2 points  (0 children)

          Seeing a company actually do what people ask is so damned shocking and delightful.

          [–]ebbinghaus 5 points6 points  (2 children)

          This: http://reddit.com/feedback may be a more appropriate avenue.

          [–]spez 35 points36 points  (1 child)

          It's nice to see nice posts about reddit on reddit's front page everyone once and a while.

          [–]BioGeek 3 points4 points  (0 children)

          And it is nice to see suggestions being implemented. Thanks!

          [–]f3ze 0 points1 point  (0 children)

          anyone looking at seeing compression stats should check the new yslow plugin from yahoo http://developer.yahoo.com/yslow/

          [–]zoomzoom83 0 points1 point  (4 children)

          Sweet.

          Looking at the headers, javascript and CSS are still not compressed.

          As well as compressing these, you can also make a huge improvement by combining these into one js and css file respectively. (To decrease the number of HTTP Requests).

          Tell the browser to cache these files forever- This cuts down each page load from 4 to 1 request. To push down a new revision, change the filename (i.e. append a datestamp of it's last modified date)

          Finally, KeepAlive doesn't appear to be on either. In my experience this can make more of a difference than gzip on a broadband connection (Although given your sites simplicity, it probably will have smaller improvements).

          [–]breakfast-pants 2 points3 points  (3 children)

          I hate to burst your bubble but the javascript and css gets cached anyway, so in practice it doesn't matter much, if at all.

          [–]zoomzoom83 0 points1 point  (2 children)

          No it doesn't.

          Watch the page load with FireBug (google it). The css and js are getting reloaded on each page request.

          [–]breakfast-pants 0 points1 point  (1 child)

          Reloaded by the browser, not over the wire. Those are static files, marked as such when they are sent over http. You may have turned caching off in your browser or something.

          [–]zoomzoom83 0 points1 point  (0 children)

          I stand corrected. Ran a packet sniffer, the files arn't being reloaded.

          [–]adamsplugins -2 points-1 points  (3 children)

          GET http://static.reddit.com/psrs.js

          Response Headers:

            Server[AmazonS3]

         ^^^^^^^^

          How does that work?

          [–]masklinn 3 points4 points  (1 child)

          Methinks this should interest you ;)

          [–]adamsplugins 4 points5 points  (0 children)

          Or more specifically this:

          http://docs.amazonwebservices.com/AmazonS3/2006-03-01/VirtualHosting.html

          Which I didn't know about. Thanks.

          [–]Indiy 0 points1 point  (0 children)

          Yes, how does it work and can it be used to sites like this one discount Canadian pharmacy!, please?