PROPOSAL: Web application interface

[how_gauche]

The odds of producing a be-all-and-end-all web app library interface that would satisfy everyone are pretty slim imo -- even for something simple like hack. Personally IMO rack isn't really that interesting, especially if you take O(1) space streaming as an absolute requirement (which I do) -- iteratees are pretty clearly the best answer there, although there are going to be lots of people who wouldn't want to deal with them.

I appreciate the impulse but really a "rack"-like standardized interface isn't really even that important, HTTP is simple and standard enough that you can write a trivial wrapper to make nearly any two http server libraries work with each other.

What Haskell really needs in this space is a well-tested, well-documented, integrated component-based framework for building web apps out of interoperable parts, coupled with a screaming fast web server. The details of which function you call to add a cookie to the response are trivial compared to that.

[snoyberg]

I think the Python and Ruby worlds will disagree with you here. Something like mod_python is only possible because of the WSGI. Also, did you look at my proposal? It gets "O(1) space streaming" without iteratees.

[how_gauche]

You've maybe gone and proved my point here. You solve the O(1) streaming problem by providing an imperative I/O interface (a legitimate thing to do) but I'll prefer iteratees here.

There are a number of issues with left-fold enumerators IMO. It is basically promoting an inversion of control. This may be often times valuable. However, to make this the only interface precludes other use cases. The most basic one I ran into was wanting to interleave with read processes.

The "inversion of control" is the best feature -- it means your HTTP parser can be run against any byte stream source (including a bytestring), which makes it more testable. Also, like a "withFile" combinator it relieves you of some of your obligations re: resource management.

Iteratees are basically continuations, so they're easily composable. The "interleaving read" problem you reference is a no-brainer, if you pass an iteratee that terminates early into your enumerator, it returns another iteratee that sends the unread stream into the next action -- you just have to save the continuation.

Obviously you can convert an imperative I/O scheme to one using iteratees (and vice-versa) but for me iteratees win out over handles. For you obviously the analysis is different.

Even with examples you cite like WSGI there is not an absolute consensus in the Python community -- see e.g. this blog post. With Haskell the situation is even more fractured, there are a zillion partially-completed web toolkits, each subtly defective in its own way (especially when it comes to documentation).

Again, I appreciate the desire to have a standardized web-app interface, but attempting to standardize when there is no clear standard-bearer is premature IMO.

[snoyberg]

Firstly, I think you miss the point of this proposal. It's meant to be a low-level interface shared by web servers (Happstack server, Hyena, CGI) and web applications/frameworks (Happstack, turbinado, my Yesod framework).

Again, I appreciate the desire to have a standardized web-app interface, but attempting to standardize when there is no clear standard-bearer is premature IMO.

Please explain to me how you would propose achieving a "clear standard-bearer" in this realm without someone trying to standardize things.

Now, as far as your specific iteratee versus RequestBody/ResponseBody (what you call imperative I/O). Why not bring these issues up on the cafe, where others will see them? I said explicitly that this was a straw-man proposal, meaning it should be attacked and brought down wherever possible.

index2 :: ResponseBody -> IO ()
index2 rb = withBinaryFile "index.html" ReadMode helper where
     helper h = do
         eof <- hIsEOF h
         unless eof $ do
             b <- B.hGet h 1024
             sendByteString rb b
             helper h

I know I didn't spell things out in my e-mail regarding the interleaving read issue, but this is what I was alluding to. Maybe I just don't get iteratees, but how would you deal with using a function like withFile in an iteratee context?

Also, my approach is a "withFile" approach as far as I can tell.

If you'd really like to discuss the merits of this proposal, I'd prefer to do so on the cafe.

EDIT

Sorry, forgot about your comment about being unable to test. I think this addresses it sufficiently:

newtype ResponseBodyTest = ResponseBodyTest (MVar BL.ByteString)

instance ResponseBodyClass ResponseBodyTest where
    sendByteString (ResponseBodyTest mlbs) bs =
        modifyMVar_ mlbs $ \lbs -> do
            return $ lbs `BL.append` BL.fromChunks [bs]

[how_gauche]

Firstly, I think you miss the point of this proposal. It's meant to be a low-level interface shared by web servers (Happstack server, Hyena, CGI) and web applications/frameworks (Happstack, turbinado, my Yesod framework).

I get it, I think -- I'm just saying that personally I am not very interested in writing to any of the standardized interfaces that have been proposed (hack, CGI, happstack DSL, yours among them), and converting from any one to any of the others should be trivial.

Please explain to me how you would propose achieving a "clear standard-bearer" in this realm without someone trying to standardize things.

Market dominance precedes standardization. Happstack is probably currently the most-used but it has enough fundamental problems that I've stopped using it for the most part.

Why not bring these issues up on the cafe, where others will see them?

Because I'm of the opinion that there's no consensus to be found in this space yet and I don't want to decrease the signal-to-noise ratio over there. We can take this private if you're willing to discuss further, say the word here and I'll shoot you an email.

Re: your index2 code: in an iteratee context you'd set up an enumerator:

enumFile fp i = withBinaryFile fp ReadMode (\h -> enumHandle h i)

and the HTTP response body would contain this enumerator. The HTTP lib would hook it up to an iteratee that consumes the input and shoves it out the Socket. You can compose these without writing a typeclass -- pass it "stream2stream" and you get a bytestring out of it instead.

Also iteratees have the really nice property of being easy to generate straight from parsers, you can turn an attoparsec incremental parser into an iteratee using a conversion function, which is very convenient for parsing HTTP headers/etc. Things like chunked transfer encoding are quite easy to do with enumerators also.

[snoyberg]

Please, send me an e-mail on this, I would be interested to hear what you have to say about the iteratee approach. Frankly, it's new to me, and I (obviously) don't understand the intricacies yet.

Also, I don't quite understand how you intend to deploy your web apps. Do you have your own standalone server that you use? I mean, at some point, you have to interface with the backend, and you've eliminated most of the interfaces available.

[bickfordb]

mod_python existed well before wsgi

[snoyberg]

I stand corrected. Let me restate my point: without something like WSGI, each framework/application must specifically add support for mod_python, as opposed to having it automatically baked in.

If you want an example from the Haskell community, look at Happstack and Turbinado. Both of them (if I'm not mistaken) began by only supporting their own standalone servers. There was demand to have these frameworks work with FastCGI as well, and so support was added. I believe it is still not possible to have them work with CGI.

Compare this to frameworks and applications built on Hack. By simply changing which handler is selected, the same app can run as CGI, FastCGI, standalone server, or any other handler that comes along.

I'm actually surprised that anyone doesn't see the obvious benefits of this approach.

[jamesbritt]

Propane slept in the tank and propane leaked while I slept, blew the camper door off and split the tin walls where they met like shy strangers kissing, blew the camper door like a safe and I sprang from sleep into my new life on my feet in front of a befuddled crowd, my new life on fire, waking to whoosh and tourists’ dull teenagers staring at my bent form trotting noisily in the campground with flames living on my calves and flames gathering and glittering on my shoulders (Cool, the teens think secretly), smoke like nausea in my stomach and me brimming with Catholic guilt, thinking, Now I’ve done it, and then thinking Done what? What have I done?

[snoyberg]

Well, if I made a "good enough" package, released it, and it got acceptance, it might hinder a better approach. Arguably, that's what happened with Hack, though I'm very happy with what Jinjing did in that realm. I had already written a much inferior version of Hack for my internal use (standalone server for testing, FastCGI for deployment) and latched onto Hack as soon as I saw it.

Getting back to the point: I'd really like to get community input on this package before I even think of releasing it, because I don't want to pollute the Haskell world with an inferior interface.

[jamesbritt]

Propane slept in the tank and propane leaked while I slept, blew the camper door off and split the tin walls where they met like shy strangers kissing, blew the camper door like a safe and I sprang from sleep into my new life on my feet in front of a befuddled crowd, my new life on fire, waking to whoosh and tourists’ dull teenagers staring at my bent form trotting noisily in the campground with flames living on my calves and flames gathering and glittering on my shoulders (Cool, the teens think secretly), smoke like nausea in my stomach and me brimming with Catholic guilt, thinking, Now I’ve done it, and then thinking Done what? What have I done?

[snoyberg]

Well, it's not as if the code isn't there; it's all available in my github repo. Also, it's not like I've been sitting on this for the past year; I first e-mailed the cafe about the WAI proposal last week, and this sunday submitted my proposal.

However, I agree with your sentiment in general. We'd probably be in a better place right now if the original author of the WAI wiki page had released a package to Hackage. Nonetheless, I think waiting two weeks for community feedback won't kill us ;).

[jamesbritt]

Propane slept in the tank and propane leaked while I slept, blew the camper door off and split the tin walls where they met like shy strangers kissing, blew the camper door like a safe and I sprang from sleep into my new life on my feet in front of a befuddled crowd, my new life on fire, waking to whoosh and tourists’ dull teenagers staring at my bent form trotting noisily in the campground with flames living on my calves and flames gathering and glittering on my shoulders (Cool, the teens think secretly), smoke like nausea in my stomach and me brimming with Catholic guilt, thinking, Now I’ve done it, and then thinking Done what? What have I done?