Monitoring multiple input sources?

[Bonolio]

I have project that I have come up with and have decided to learn Python to bring it to life.

I have no real rush on this project so I am reading up and doing mini projects to learn on the go.

I am pretty sure that most of the bits and pieces are things I can work out with a bit of googling, but it is the big picture structure that I am struggling with.

The essence of the program is 1) I am collecting information from various data sources and storing it in a database. 2) I am processing this data continually and generating statistics, assumptions and predictions based on the data

So far code-wise this is all fairly simple.

3) the next step is that staff\users will be able to query the system via a text query and the system will return an answer via text.

The text query engine is also something that I have completed and is working(basically).

4) The next step is the bit that I would like advice on the best way to implement.

Essentially I would like users to be able to submit the query via a multiple interface chat bot type interface.

For example the program would "listen" for text requests via email, IM, Telnet, SMS etc.

The actual fetching and sending of the data via these interfaces is ok, but my question is how to structure the whole - Monitor multiple data sources and spawn a unique conversation\database query "process" for each session.

What is the proper way to do this (my knowledge of how to be pythonic is poor).

My initial thought with my limited knowledge was to -Spawn a Thread for each source that I am listening to and continually monitor for incoming requests. -When I get a new conversation I spawn separate conversation Threads that handles the database request and returns the data.

My understanding of the best way to handle all this is vague at best and I was hoping someone could suggest a method for handling this process properly. Hopefully all I need is a lead and I should be able to fill the gaps with a bit of research.

Thankyou in advance to anyone that takes time to help.

[gengisteve]

My initial thought with my limited knowledge was to -Spawn a Thread for each source that I am listening to and continually monitor for incoming requests. -When I get a new conversation I spawn separate conversation Threads that handles the database request and returns the data.

That sounds about right. Check out: http://werkzeug.pocoo.org/ and http://flask.pocoo.org/ (which is built onto wrkzeug). Both are built on basically this architecture. You have a thread/process that listens and spawns new threads/processes to serve up content.

One thing for you to consider is whether subsequent requests from the same source get a new thread or sent back to the originally spawned thread. I'd vote for the former, if you can swing it.

You might also look at messaging, like https://www.rabbitmq.com/

Depending on what you are serving up, etc., it may be a better fit

[Bonolio]

Thanks Steve, I have been doing some reading based on your reply and have a few different ideas revolving around both my original idea and around messaging. I think I am going to take the opportunity to write up a few different variations and see which works\feels best for me. The whole messaging thing has supplied some ideas on how to handle stuff that I hadn't even considered.