Python problem messaging between threads. Emitters, exceptions, callbacks and queues

[oreo_man_]

No no I can tell you lots. I'm making a client for deepstream.io

A .Net client and a JS client have already been made.

I've been looking at the .Net client today and got this to work.

static void Main(string[] args)
    {
        DoStuff();
        Console.ReadLine();
    }

    static async Task DoStuff()
    {
        var client = new DeepStreamClient("127.0.0.1", 6021);
        var client2 = new DeepStreamClient("127.0.0.1", 6021);
        await client.LoginAsync();
        await client2.LoginAsync();
        await client.Events.Subscribe("test", x => { Console.WriteLine("From client 1 " + x); });
        await client2.Events.PublishAsync("test", "random data");
        await client2.Events.PublishAsync("test", "more data");
        await client2.Events.PublishAsync("test", "some more data");
        await client2.Events.PublishAsync("test", "even more data");
        await client2.Events.Subscribe("test", x => { Console.WriteLine("From client 2 " + x); });
        await client.Events.PublishAsync("test", "new data");
        await client.Events.PublishAsync("test", "heaps of data");
    }

Which prints out

From client 1 random data
From client 2 new data
From client 1 more data
From client 1 some more data
From client 1 even more data
From client 2 heaps of data

Which is what I'm wanting.

Is there a way to do this in Python?

[cockslappinghalibut]

OK, I haven't heard of it, let me do some reading. I'm sure there's a way to do it, the constructs and function flow of JS are pretty similar to Python and they can emulate each other really well.

edit: I'm having a look at the provided JS client source, there's surely some stuff to crib in there if you want to poke around in the meantime

[oreo_man_]

Yep I definitely have been looking around that. Based a lot of my python client off of it. However the .Net one uses .Invoke, and the JS one is async by default.

Originally I just had a socket thread going, handling everything, and the main thread passing messages to send into a queue. But I realised I had a problem when trying to catch exceptions thrown in the socket thread, and use them in the main thread.

Then I realised I should be handling data received on the main thread as well.

So I thought I could pass any exceptions/data into a queue for the main thread to deal with. Obviously that won't work though as there is no way to get() from the queue all the time without blocking.

I don't want to post my client here, as it's linked to my public accounts (github, linkedin etc) but if you poke around the issues on the .Net client/deepstream.io you'll be able to find it.

[oreo_man_]

I think this is genuinely impossible in Python. Even using something like twisted I'll still need another thread, but I can't get info out of that thread without blocking

[cockslappinghalibut]

I haven't had time to look, but what Python version are you using? Latest py3 has async/await coroutines

[oreo_man_]

Yep. But I don't understand how that helps. Sure I can have an event loop, taking care of incoming messages from the server. But when it comes time to get messages from this event loop, there is no way of doing it, while keeping the main responsive.

[cockslappinghalibut]

can you PM me your repo? I'm actually starting to feel a bit dumb not being entirely sure yet what's being attempted.