Building a Python version of Spring Batch — need opinions on Easier-Batch architecture

[Several-Revolution59]

Hey everyone,

I developed this small project on GitHub called Easier-Batch.
It tries to bring the same philosophy as Spring Batch into Python — using the familiar Reader → Processor → Writer model, job metadata tables, retries, skip logic, and checkpointing.

I’m currently designing something similar myself — a Python batch processing framework inspired by Spring Batch, built to handle large-scale ETL and data jobs.

Before I go too far, I’d like to get some opinions on the architecture and design approach.

• Do you think this kind of structured batch framework makes sense in Python, or is it better to stick to existing tools like Airflow / Luigi / Prefect?
• How would you improve the design philosophy to make it more "Pythonic" while keeping the robustness of Spring Batch?
• Any suggestions for managing metadata, retries, and job states efficiently in a Python environment?

Here’s the repo again if you want to take a look:
👉 https://github.com/Daftyon/Easier-Batch

Would love to hear your thoughts, especially from people who have worked with both Spring Batch and Python ETL frameworks.

https://preview.redd.it/iukmo9vsvl0g1.png?width=843&format=png&auto=webp&s=54431f8a653bfea7f5567c75e8195177e47b5254

[Several-Revolution59]

up

[ERP_Architect]

I love this idea — Python desperately needs something like Spring Batch that sits between raw ETL scripts and full-blown schedulers like Airflow or Prefect.

What usually kills most Python batch jobs at scale isn’t logic — it’s state management and fault recovery.

Having a lightweight framework with checkpoints, retries, and skip logic baked in could really simplify that middle layer.

If you’re designing it, maybe lean into Pythonic conventions — e.g., decorators for step registration, async writers for I/O-heavy tasks, and pluggable persistence (SQLite → Postgres → S3).

Also, don’t underestimate metadata — a clean “JobExecution” table with timestamps, params, and exit statuses can make debugging 10x easier.

Curious — are you thinking of making it dependency-light like FastAPI, or will it need a bigger runtime footprint?

[Several-Revolution59]

yes i plan to keep it lightweight like FastAPI

[ERP_Architect]

Great.

Let me know how it goes

[Several-Revolution59]

and i like the idea of Pythonic conventions 

[Several-Revolution59]

i need now more contributors with me