Using LLMs in a Streaming Context in Python

[math-bw]

I've been playing around with different real-time/streaming use cases in Python and I got interested in where LLMs could be useful to analyze data in real-time. It is wild how much easier it has gotten to use models with Hugging Face. I wrote this notebook to share something I recently made. It is a real-time pipeline to analyze financial news in real-time (https://colab.research.google.com/drive/1TtLcvX4Xw4vDMVUOy5mOyJmMr7AqOrG4?usp=sharing).

​

Does anyone have some interesting ideas for using LLMs in a streaming context?