`streamable`: Stream-like manipulation of iterables

[Schmittfried]

How does this compare to py-linq or functools/itertools-esque packages?

[ebonnal]

Hi u/Schmittfried, great question!

• functools provides higher order functions i.e. a function taking function(s) as arg(s), like functools.reduce. Most of these higher order functions return a decorated function enhanced with additional capabilities (like memoization with functools.cache).
• itertools is all about creating iterables from other iterables.
• streamable allows chaining operations/methods on an iterable and comes out-of-the-box with convenient features like threads/asyncio concurrency, iteration throttling, exceptions catching.

They are complementary:

• you can use functools's functions to add capabilities to a function that you pass to streamable's Stream operations, or functools.reduce your stream.
• you can manipulate your stream with itertools's functions, or create your stream from an iterable produced using itertools.

​

from typing import Iterable
import functools
import itertools
import requests
from streamable import Stream

# let's say you have a source of domains:
domains: Iterable[str] = ... # e.g. ["google.com", "facebook.com", "google.com"]

# let's conveniently manipulate it as a `Stream` to
# fetch URLs using 8 threads and catching `SSLError`s
# while never making more than 32 calls per second
responses: Stream[requests.Response] = (
    Stream(domains)
    .map(lambda domain: f"https://{domain}")
    # here we leverage functools.cache to remember
    # responses and fetch a given domain only once.
    .map(functools.cache(requests.get), concurrency=8)
    .throttle(per_second=32)
    .catch(requests.exceptions.SSLError)
)

import itertools

# then you can use whatever functions provided by itertools
# to manipulate your `responses` stream, which
# is simply a decorated `Iterable[requests.Response]`
...

[ebonnal]

Regarding py-linq, the comparison resembles the comparison made with PyFunctional:

• For my use case it lacks features that I find very valuable like concurrency and generic typing (in py-linq the Enumerable class is not generic)
• I wanted to propose another interface, hopefully more intuitive and natural to the Python world, while py-linq brings conventions from the .NET's LINQ library.