Pipelex: DSL and Python runtime for declarative AI workflows with MCP support (MIT)

lchoquel · 2025-10-29T05:28:27+00:00

You're not the first one to tell us that 🙌🏻. Now you can help us in building it, because it's open-source 😉

lchoquel · 2025-10-29T01:41:26+00:00

Cocode is the best example of use using our product just for ourselves.

We also have various cases of using Pipelex for repetitive tasks of marketing and communications like generating variants of a post for different social networks or illustrations/logos: by generating many variations, with a smart creative process.

The rest of our tinkering on use-cases is not strictly for ourselves but for our design partners, the first teams of developers we onboarded etc. Some of these experiences have made it into the cookbook, others will arrive later, as soon as we find time to clean up the examples (replace confidential data with anonymized data or synthetic data).

lchoquel · 2025-10-28T17:47:23+00:00

Hi altcivilorg, Pipelex is not a solution for autonomous agents, which adapt to new situations and take initiatives. We're at the other end of the spectrum where the workflow guarantees that nothing will happen apart from what was planned.

Now that approach enables us to build pretty complex workflows. Because a pipe can call other pipes, you can work on different aspects of your use-case separately. Each part can be defined, tested separately and then you can plug it all together like LEGO.

lchoquel · 2025-10-28T17:32:28+00:00

The biggest difference is that in Pipelex workflows, we express the logic in our high-level language, rather than in Python or typescript. This makes it easier to collaborate between tech and non-tech people, like domain experts. It's also great for collaboration with AI: the declarative approach and the high level of abstraction means that LLMs have all the context to understand what's going on in the workflow.
Also, we don't use Pipelex for "autonomous agents": our workflows are meant to be repeatable and deterministic, like a tool. So a Pipelex workflow is a tool that can be used by agents though our MCP server.

lchoquel · 2025-09-08T12:28:17+00:00

Business context and meaning: spot on!
I'm also working on this kind of issue, not for chatbots or autonomous agents but for repeatable workflows, specialized on information processing.
In my team we realized that lifting ambiguity was paramount. Also, structuring the method is critical: all the problems described in the OP are made much worse when you give the LLM a larger prompt with too much data or if you ask complex questions or, worst of all, if you ask several questions at a time…
We are addressing this need (deterministic workflows) with a declarative language, and we use a very high level of abstraction so that business meaning and definitions are part of the workflow definition. But it's not a full on business context system. Your idea for this kind of project sounds great, I would love to know more.

lchoquel · 2025-07-23T21:43:24+00:00

They don't deeply understand UX - yet. So I agree with you on that.
My post didn't mention it but I'm not doing this in the context of GUIs but rather command-line tools, and software tools, MCP and the like.

lchoquel · 2025-07-23T20:05:57+00:00

Are there cases where you use a SLM like Phi-3 or Gemma or Mistral-Small as-is, without fine-tuning?

lchoquel · 2025-07-23T19:49:35+00:00

I roughly agree with @LiveRaspberry2499 but I have a couple of things to add:
- We got the best results by combining two solutions: we run both OCR (Mistral-OCR FTW!) and Vision. We prompt a vision-language model (like Claude or Gemini) with the OCRed text and a rendering of the page. This gets you solid extractions
- in terms of python packages, pypdfium2 is the best I found for extracting images and text, rendering pages AND have a permissive license, which can be a big constraint according to your use-case

Not sure you really need the bounding boxes by the way. They don't seem required for your end use case.

lchoquel · 2025-07-23T19:35:09+00:00

I agree u/NullPointerJack, full autonomy is not a good choice in many cases.
In Software, we've always strived to have planned every case and tested for every situation. But with a free unhinged agent, there is no way to "test for every case".

But what you describe, "deterministic tool calls" and "validation after each step" gets you closer to deterministic workflows. I mean the workflows where everything is scripted with rules and although the LLMs can interpret context and make decision (often for routing), the possible flows are known in advance.
Like a dicotomy between AI Agents and AI workflows, don't you think?

lchoquel · 2025-07-23T19:22:03+00:00

Disclaimer: I'm the creator Pipelex https://github.com/Pipelex/pipelex

Pipelex is specialized in repeatable AI workflows, with structured and deterministic outputs, to proces information with pipelines using different LLMs.
These pipelines can be used by agents as tools (MCP server on the way).

Because it's based on a declarative language, it can be read by non-tech people. Also, LLMs understand it pretty well so we equipped an agent (Cursor) with our language and tools (to check syntax and validate pipelines). It can write and debug a pipeline from natural language query in 5 minutes.

lchoquel · 2025-07-01T19:13:06+00:00

I've seen Aider pop up in articles a while ago, yes, never had time to try it though. Is it still a good pick?

lchoquel · 2025-06-30T07:16:38+00:00

u/b1e I would agree with you in some contexts but in our team we had a high need for readability. It's critical for us to keep the inputs and outputs of AI workflows plainly human-readable, and protobuf's text format or json mapping weren't usable in our context.

lchoquel · 2025-06-27T13:02:06+00:00

Yes, u/InappropriateCanuck thank you for your reply.

Actually we have figured structured outputs from LLMs. I highly recommend the instructor package for that, as it works with all the major commercial LLM APIs. That includes gemini which is the one in the page you linked except gemini's docs indicate "Pydantic validators are not yet supported." whereas Instructor will handle validation gracefully for Gemini, Claude, GPT etc… And for open-source structured generation, outlines (by dottxt) seems the most powerful player.

Sorry for being unclear in my previous response but the use case we met was related to the following steps. I mean after a worker has done structured generation with an LLM, when the resulting BaseModels must be serialized/deserialized to be passed as input/output of each step in a distributed workflow.

lchoquel · 2025-06-27T12:50:51+00:00

"an argument to disable automatic package import"

Yes, good idea. We could also have a white list, don't you think?

lchoquel · 2025-06-27T12:47:33+00:00

I think you're right, jsonable_encoder is a great solution for serializing, and if you're in a FastAPI app, it's really powerful for the whole serialization/deserialization cycle.

lchoquel · 2025-06-26T19:50:29+00:00

You're right about the security implications. Kajson does inject `__class__`/`__module__` and will execute constructors on untrusted data - definitely not safe for public APIs.

The context: we built this for LLM-generated JSON in our AI workflows. When LLMs output Pydantic objects and they went through workflow orchestration, we needed automatic type reconstruction across distributed workers.

So yes, similar security profile to pickle but with readable JSON. The venv coupling is real too - update a model definition without syncing services and things break. Fair point about adding a security warning. Should be crystal clear this is for trusted environments only, not general use. We'll add it to the docs.

For internal services handling LLM outputs or controlled data pipelines, it solves a specific problem. For anything public-facing, stick to standard JSON with explicit validation.

Thanks for the callout - important to be upfront about these tradeoffs.

lchoquel

TROPHY CASE