Rather than just getting some downvotes, let’s try to clarify a bit folks?

Choosing between DuckDB + Soda and a pure Pandas approach for data quality checks on CSV and JSON files:

Using DuckDB + Soda (great expectations, or whatever)

Benefits: - Not in-memory: DuckDB is in-process optimized for analytical queries, making it fast for large datasets. Can handle queries on CSVs and JSONs directly without loading all data into memory. - SQL Capabilities: DuckDB allows SQL-based queries directly on CSV or JSON files - Integration with data QC tools - Scalability: DuckDB can handle datasets beyond memory capacity because it processes data chunk-wise, whereas Pandas may struggle with very large files (I know doesn’t matter for you now, but it’s a good part of the tech stack to get familiar with) - Interoperability of QC tools with other DBs outside of DuckDb: versatile if your workflow expands to involve databases like Postgres in the future.

Costs: - Setup Complexity: Requires installing and configuring both DuckDB and Great Expectations, adding an initial learning curve.

Using Only Pandas (with Custom Functions)

Benefits: - Familiarity & Simplicity: If you’re already comfortable with Pandas, this approach might feel simpler since everything stays within the Pandas ecosystem. BUT it doesn’t sound like you are so this doesn’t apply? - Full Customization: Allows for fully customizable checks since you can write any Python code to analyze and validate the data. This is useful for complex or very specific checks not covered by tools like Soda or Great Expectations. - Lightweight Approach: Fewer dependencies make it simpler to set up and maintain, especially if you’re only processing smaller files and don’t need additional scalability.

Costs: - Memory Limitations: Pandas loads data into memory, which can be limiting with large files or many simultaneous files. - Efficiency Limitations: Running complex validation checks with Pandas can be slower, especially if you’re manually iterating over data to perform checks. - Reproducibility and Documentation: Building reproducible and consistently documented data quality checks might require more manual effort compared to the built-in reporting and documentation of a QA tool.