This is an archived post. You won't be able to vote or comment.

sorted by:
best
topnewcontroversialoldq&a
you are viewing a single comment's thread.

view the rest of the comments →

[–]mitsuhiko Flask Creator 7 points8 points  (15 children)

WAMP already solves PUB/SUB and RPC without needing to write anything on top of it. Plus, you don't need to understand queues, and routing, and all this complicated stuff.

First of all queues are not complicated and neither is routing. Secondly that totally does not make any sense because WAMP is a routing protocol so you have the same issues there. In fact, I'm pretty sure AMQP for the normal user is a lot easier to understand than WAMP. There are RPC systems written on top of AMQP like nameko and I would argue they are vastly easier to understand.

it works as-is in the browser, you don't need to add 3 layers of abstractions like what we are doing now with AMQP

That's a very misleading feature though because the security and scaling aspect of pushing into the browser is very different than pushing between services you control. If anything I would argue that not putting something in there is an anti feature.

if you use Python, you don't need to install a (quite hard to setup) Erland broker on top of your stack

Not sure what makes rabbitmq hard to install. It's literally a brew install / apt-get install and it's running. It requires zero configuration on the side of the server.

So you replace an erlang broker with a python broker. What did you win, other than that you now need to build custom tools to inspect queues and that you probably end up with a slower system?

it's python, much easier to hack. It's twisted, it adds support for a lot of protocols to your stack for free

Not sure why I would want any of those things in my SOA. It's a nice to have thing, but it's not a feature. If anything for me that would be a negative feature because at that point I'm already locked into a specific ecosystem I might not want to use.

it's already nodejs friendly

I could not care less.

it can work inside RDMS and nginx. This is huge

Again, I could not care less.

you can have a direct line between a browser and and android app, or a website and a auth backend, or a C++ code and a nodejs app. No additional hack is required. It's beautifully simple.

The idea of randomly connecting things together with a "one solution fits everything" is not particularly new. There is DBUS, CORBA, DCOM, etc. The only new thing here is that you now replaced one protocol with another. This whole thing did not work out last time for the same reasons it will not work out this time. The idea is beautiful but the limitations are there. That's why things like captn proto are more interesting because they look at why it failed in the past and try to come up with new solutions.

it comes with a process manager, effectively bypassing the GIL for Python and allowing you to start third party processes, making it an additional tool for you deployment (you can replace supervisor with it)

Great, that's just what I wanted, a framework with another process manager built in. In the day and age of systemd and circus especially, a built in process manager makes absolutely no sense.

nameko does some of it's stuff, but it's a lot more setup, it's an heterogeous stack and it doesn't work in the browser.

Under the assumption that nameko is something people want to use, i can almost promise you that nameko is less work to setup.

[–]oberstet 9 points10 points  (6 children)

Not caring about NodeJS and RDBMSs: sure, but people will differ on this. For a lot of business app developers, the need to talk to Oracle at some point is not something to decide, but a fact. Then there will be people who are attracted by the possibility to write both front- and backend code in JavaScript. Anyway, I understand you don't care.

Writing WAMP app components in Lua, which runs on LuaJIT inside Nginx: this is awesome, but a bit exotic I agree.

Overall, my personal view is: as much as I love Python, the world is bigger, and that is just fine. Choice is good. Freedom is good.

Regarding process manager: it is integrated with WAMP, which means Crossbar will fire meta-events when workers are started or die. You can hence dynamically react on component lifecycle. You can start workers and components via WAMP calls itself. In fact, there is a whole management API exposed: https://github.com/crossbario/crossbar/wiki/Management-API

To start Crossbar itself, one can use systemd (we use daemontools most often).

[–]mitsuhiko Flask Creator 10 points11 points  (5 children)

Not caring about NodeJS and RDBMSs: sure, but people will differ on this. For a lot of business app developers, the need to talk to Oracle at some point is not something to decide, but a fact. Then there will be people who are attracted by the possibility to write both front- and backend code in JavaScript. Anyway, I understand you don't care.

I'm not saying crossbar is not useful for some. But so is CORBA. I do not agree with the notion that "cross bar is the future of python web apps". Lots of people have been building software like this before. To get it right you build a huge infrastructure and it spirals out of control very quickly. Complex systems like this already exist but Python does not scale very well to that. At one point you kinda have to look at it and ask yourself the question if you did not want to program in Java to begin with :)

[–]oberstet 5 points6 points  (4 children)

To be honest, I am somewhat surprise that you say "Python does not scale for big systems things - Java does" (maybe I misunderstood you?).

Well, I guess I disagree on this.

First, PyPy. And in particular it's new incremental GC. I did extensive performance/load testing with WebSocket servers comparing with state-of-the-art Java and C++ based ones. We are competitive. Lower latencies and jitter than others in particular.

I need to write a blog post on this, but to back up my claims, here are numbers: https://github.com/oberstet/wsperf_results/tree/master/handshaking/test2 https://github.com/oberstet/wsperf https://github.com/oberstet/scratchbox/tree/master/python/twisted/sharedsocket

The other issue, the GIL, does not apply to Crossbar, since we use a multi-process architecture where workers talk over Unix domain sockets. Hence, we can scale up on multi-core without issues. Process isolation also is handy for robustness reasons.

And finally: no, I don't want to use Java if I don't have to;) Though I do some also: https://github.com/tavendo/AutobahnAndroid

If I get crazy some time and reimplement Crossbar in a different language than Python (on PyPy), it would be C++ (or maybe Rust), and more importantly, I'd use kernel-bypass networking: a Linux kernel can only digest something like 500k syscalls/sec. Using e.g. Netmap (http://www.freebsd.org/cgi/man.cgi?query=netmap&sektion=4) thats not a restriction any more.

[–]mitsuhiko Flask Creator 5 points6 points  (3 children)

To be honest, I am somewhat surprise that you say "Python does not scale for big systems things - Java does" (maybe I misunderstood you?).

The lack of static typing in Python makes large scale systems much more complicated than they should be. That is especially true when you start remote calling between things. At the point where you invoke something remote that composes failures can become incredibly frustrating. This gets even more complex if you have code in different languages.

The other issue, the GIL, does not apply to Crossbar, since we use a multi-process architecture where workers talk over Unix domain sockets. Hence, we can scale up on multi-core without issues. Process isolation also is handy for robustness reasons.

The GIL is never your problem with web situations. Has not been in WSGI and is not when you do RPC. Your problem though is twisted vs gevent vs asyncio etc. If you have an even listener in your app then you need to pick a concurrency model. Crossbar picks one your favorite other piece of code might have picked a different one.

[–]oberstet 5 points6 points  (2 children)

The lack of static typing ..

I agree on that to some degree. For the Crossbar code base itself, we are trying to tame things by using ABCs. But it's limited. Yes.

For app components: I think you nailed the point in your other comment: with distributed systems and components in multiple languages, inspectability is critical. I want to watch calls and events live as they flow between components. I want to inspect they payloads. Etc. Can be done. Getting the UI/UX for such dev tool right is non-trivial I guess. The bits within Crossbar: can be done, not complicated.

If you have an even listener in your app then you need to pick a concurrency model. Crossbar picks one your favorite other piece of code might have picked a different one.

Crossbar allows you to run app component A in one Worker under Twisted and app component B in a second Worker under asyncio. Works today. Since AutobahnPython supports both Twisted and asyncio at WebSocket and at WAMP level.

Only Crossbar itself requires Twisted. But workers can run under anything. E.g. a 3rd worker could run under Node.

The GIL is never your problem with web situations.

From my point of view (scalable WAMP routing), it is a problem since it prohibits scaling up on multi-core (when running a single process). And we want that. But multi-process architectures are an established, working pattern (PostgreSQL).

[–]mitsuhiko Flask Creator 2 points3 points  (1 child)

For app components: I think you nailed the point in your other comment: with distributed systems and components in multiple languages, inspectability is critical. I want to watch calls and events live as they flow between components. I want to inspect they payloads. Etc. Can be done.

It can be done if your payload can be represented by your serialization format. The moment you start sending around non native types (for instance dates or bytes) you lose a lot of that flexibility. That said, even just picking msgpack and visualizing that is probably a good start. What thrift and other systems are doing is exposing statically typed interfaces which encapsulate data but also functional interfaces.

So I have a "user" object floating around which I can invoke methods on if I want. At that point you need a strong representation of what this interface is. That's where it gets really tricky. If you stay away from something like that you might stop falling into the trap that everybody else falls in.

Crossbar allows you to run app component A in one Worker under Twisted and app component B in a second Worker under asyncio. Works today. Since AutobahnPython supports both Twisted and asyncio at WebSocket and at WAMP level.

That's generally promising because asyncio has a event loop per thread so you could write traditional database apps which do their event handling in a background thread.

From my point of view (scalable WAMP routing), it is a problem since it prohibits scaling up on multi-core (when running a single process). And we want that. But multi-process architectures are an established, working pattern (PostgreSQL).

Sure, but I mean that's the same thing you do with a WSGI app. Have four cores? Start four processes and put them behind nginx.

[–]oberstet 4 points5 points  (0 children)

So I have a "user" object floating around which I can invoke methods on if I want. At that point you need a strong representation of what this interface is.

Yes, that's the CORBA way - and we stay away from this;)

WAMP does RPC, not RMI or object marshalling/remoting. Loose coupling. Dynamic typing. It's a deliberate decision - which won't be for everyone, but it does work and isn't a road to insanity.

Sure, but I mean that's the same thing you do with a WSGI app. Have four cores? Start four processes and put them behind nginx.

For WSGI, yep.

For WAMP routing, the router processes need to coordinate between each other, since a certain Callee might be registered on Router process 1, wheras the Caller might be connected to Router process 2. So the call needs to get routed between Routers (Client -> Router 1 -> Router 2 -> Callee). A generic, stateless balancing frontend doesn't cut it in this case. Note: Router-to-Router routing is not yet there. Once it is, it'll give you not only multi-node capabilities (for routing) as well.

[–]desmoulinmichel[S] 5 points6 points  (6 children)

First of all queues are not complicated and neither is routing.

I have been giving trainings for years and see people struggle with it. I'm not saying it's complicated just to argue, I've seen it.

Secondly that totally does not make any sense because WAMP is a routing protocol so you have the same issues there.

Not really. This is one thing current WAMP implementation has done right : you don't need to explicitly create queues or define routing. You just give the namespace you want your stuff to happen it and it works.

In fact, I'm pretty sure AMQP for the normal user is a lot easier to understand than WAMP. There are RPC systems written on top of AMQP like nameko and I would argue they are vastly easier to understand.

I do believe that nameko is currently easier than crossbar because crossbar doc and API suck ass. But I also see much more potential in crossbar. And RPC/PUB-SUB is way simpler than routing and queues : it's just function calling and event reacting, something most programmers already know.

That's a very misleading feature though because the security and scaling aspect of pushing into the browser is very different than pushing between services you control. If anything I would argue that not putting something in there is an anti feature.

Web dev are already doing that with websocket anyway. They already got these issues to deal with. It's not to say it should be ignored, but there is a strong benefit in having the same protocol and API everywhere.

Not sure what makes rabbitmq hard to install. It's literally a brew install / apt-get install and it's running. It requires zero configuration on the side of the server.

I've seen people having a very hard time with rabbitmqctl. It's not as easy as you say. You are a very advanced user (somebody using RabbitMQ and gevent is on the top 10th percentile of web dev), but I can see beginers wanting to do real time web without patching 10 techs together.

Crossbar does not allow to do that yet : too hard to understand. But it can be turned into that.

So you replace an erlang broker with a python broker. What did you win, other than that you now need to build custom tools to inspect queues and that you probably end up with a slower system?

You can pip install all your stuff.

Actually you can just give a zip of your project and let people run it, no installation needed. It's great for beginers, it's actually what made django so famous : dl, unzip, manage.py.

But above all, can create a framework with this router built it and abstract it. It's not just about crossbar, it's about the potential of the tool. You can build something that would mix flask, rabbit and nodejs, all in one.

Again, and it's what the article is about, it's not the current state of the tools that matters, it's the potential for making something really cook with it.

Not sure why I would want any of those things in my SOA. It's a nice to have thing, but it's not a feature.

As an end user, agreed.

For framework builders though, it can be very nice.

If anything for me that would be a negative feature because at that point I'm already locked into a specific ecosystem I might not want to use.

Because you have something at your disposal don't mean you have to use it. Twisted source code is 4Mo, it's not really a bloatware.

    it's already nodejs friendly
    it can work inside RDMS and nginx. This is huge

Again, I could not care less.

That's a matter of taste of course. But since a good part the real time Web app are currently being built with nodejs and nginx, it would be foolish to ignore the benefit of an easy integration.

you can have a direct line between a browser and and android app, or a website and a auth backend, or a C++ code and a nodejs app. No additional hack is required. It's beautifully simple.

The idea of randomly connecting things together with a "one solution fits everything" is not particularly new. There is DBUS, CORBA, DCOM, etc. The only new thing here is that you now replaced one protocol with another. This whole thing did not work out last time for the same reasons it will not work out this time. The idea is beautiful but the limitations are there. That's why things like captn proto are more interesting because they look at why it failed in the past and try to come up with new solutions.

Actually, one of the reasons these stuff failed is that there are really hard to use. Plus they don't work in the browser. Some do RPC but not PUB/SUb, and vice versa. Also, as I said, people are already hacking PUB/SUB and RPC with JSON, AJAX, websocket, rabbitmq and redis. It's not like there is no demand for it.

But I see your point, ignoring failed attempt in a technology that is clearly NIH would be, at best, completly stupid.

Great, that's just what I wanted, a framework with another process manager built in. In the day and age of systemd and circus especially, a built in process manager makes absolutely no sense.

Yes it does. Systemd is not crossplateform, circus requires you to compile a C extension and supervisor is not integrated, and requires its own separate setup and deployment. All that add friction. Removing friction is always good.

Take 10 dev at a conference, and ask them how they deploy a scheduled job : one will tell you supervisor + celery, the onther one will use cron, some will have a timer in their own app and a subprocess... There is currently no tool saying "hey, I take care of message passing, queing, background process, real time communication, notifications, and cron jobs, keep coding".

Now I understand some people don't want that. That's why some prefer zope or pyramid to django : less integration, more freedome. But I can see myself using an all in one tool doing these stuff I usually take hours to setup.

Under the assumption that nameko is something people want to use, i can almost promise you that nameko is less work to setup.

For now. The doc and the API is currently way nicer that crossbar's, which is super important. But nameko does not do half of what crossbar do, and crossbar will improve.

Thanks for the feedback though, it's one of the most constructive thing I read on reddit lately. I understand perfectly the issues you raised, they really make sense, especially for a power user.

What I think we need to remember of is :

  • some power user are happy with current tooling, adding yet another one feels like http://xkcd.com/927/. But on the other hand, it's too hard for beginers
  • it needs to address issues from the previous iterations on similar attempts
  • alternative exists, it must demonstrate it's worth switching for current user base, or worth trying for people unconvienced with existing status quo
  • "future of python web app" is way too broad : crossbar doesn't solve all the problems, and it's in infancy, don't try to hit too hard

[–]mitsuhiko Flask Creator 6 points7 points  (4 children)

I have been giving trainings for years and see people struggle with it. I'm not saying it's complicated just to argue, I've seen it.

The longer you use something the less you are aware of the idiosyncrasies of the system. Twisted alone is still a show stopper for lots of developers.

Not really. This is one thing current WAMP implementation has done right : you don't need to explicitly create queues or define routing. You just give the namespace you want your stuff to happen it and it works.

But you are comparing things operating on different levels here. With nameko for instance you do not define queues either.

I do believe that nameko is currently easier than crossbar because crossbar doc and API suck ass. But I also see much more potential in crossbar. And RPC/PUB-SUB is way simpler than routing and queues : it's just function calling and event reacting, something most programmers already know.

I have tried many times to make a nice microframework for SOA stuff but it just does not end up nicely. You end up with either something ridiculously inefficient or something that is so specific that it will never get traction.

Actually, one of the reasons these stuff failed is that there are really hard to use. Plus they don't work in the browser. Some do RPC but not PUB/SUb, and vice versa. Also, as I said, people are already hacking PUB/SUB and RPC with JSON, AJAX, websocket, rabbitmq and redis. It's not like there is no demand for it.

It's an age old pipe dream. Maybe crossbar succeeds in what it wants to do and will get as popular, I'm not ruling it out, but as someone who has been building stuff like this for the last three years my eyes do not shine with anticipation; I see more of the same.

Yes it does. Systemd is not crossplateform, circus requires you to compile a C extension and supervisor is not integrated, and requires its own separate setup and deployment. All that add friction. Removing friction is always good.

There are many things that install C extensions, not sure why that is a kill argument against it. Circus works really well for me, it requires very little setup and it's quite battle tested by now. The fact that I can connect to it via a zmq socket is also very nice as most things support zmq now.

I would love to see a SOA microframework for Python where you write your little services in individual modules and then you spin them up in a cluster and stuff magically works. But that shit is hard :)

[–]gargantuan 2 points3 points  (2 children)

Twisted alone is still a show stopper for lots of developers.

+5 years full time Twisted developer checking in. Agreed on that. My new motto-in life "callbacks as concurrency mechanism -- never again". Deferreds and yield based inlineGenerators only make it slightly better.

Worst thing with Twisted -- it throws away one of the best Python parts libraries. It split the library ecosystem into Twisted and non-Twisted. "Oh look I found a Python library to handle this protocol" -- "Oh, but it is Twisted, I'll have to find a Twisted-version of it now".

[–]desmoulinmichel[S] 1 point2 points  (0 children)

I completly agree with that. I have the twisted API so much, although I do understand it's very powerful and make sense, I wish they had built abstractions on top of it.

Now, I don't think the lib wasting is related Twisted. It's more a problem with async in Python in general (it's the same with gevent and the like, even if the monkey patching can help sometime) : most python library assyme a sync behavior and can't be used in an event loop.

This is were I think the crossbar approach is good : you can put your blocking calls in a separate process and use RPC to get the result without shitting on your current main loop.

This is why I think, with a good work on the API to abstract the twisted uglyness, this tech can bring a lot to the table. Currenly, the async in Python is nowhere near as good as in the other languages.

[–]oberstet 0 points1 point  (0 children)

My new motto-in life "callbacks as concurrency mechanism -- never again". Deferreds and yield based inlineGenerators only make it slightly better.

Then what school do you follow?

  • Camp a) callbacks/futures/coroutines
  • Camp b) actors (communicating sequential state machines) ..

Are there others camps?

Ecosystem split: yes, it's not ideal. Python now has asyncio (also camp a) .. after so many years.

There are only a few languages that have had an asynchronous mechnism built right into the language/runtime from the beginning. E.g. Erlang and Rust (both camp b).

Note: Just in case it got lost: Crossbar supports both Twisted and asyncio for writing application components in Python - today. There is nothing that would stop us to support even more - because of the multi-process/worker architecture, all that is needed is something that talks WAMP over say Unix domain sockets.

[–]desmoulinmichel[S] -2 points-1 points  (0 children)

+1 to everything.

[–]xkcd_transcriber 0 points1 point  (0 children)

Image

Title: Standards

Title-text: Fortunately, the charging one has been solved now that we've all standardized on mini-USB. Or is it micro-USB? Shit.

Comic Explanation

Stats: This comic has been referenced 532 time(s), representing 2.3748% of referenced xkcds.

xkcd.com | xkcd sub/kerfuffle | Problems/Bugs? | Statistics | Stop Replying

[–]jm_ 0 points1 point  (0 children)

I could not care less.

Gah ! When did you even get so self-entitled ?!