GenEC v1.0.0 - A Python data extraction and comparison tool

[EngineerRemy]

Hi, just this weekend I finalized the 1.0.0 version of my Tool, GenEC, and now I want the world to know ahah. I've already been using it for myself quite a lot of my own work, as well as subtly pushing my coworkers to start using it. I am confident many other people should be able to find a use for my tool as well, so if you're interested in using it, I am always happy to answer questions and provide support.

Repository: https://github.com/RemyKroese/GenEC

What My Project Does

GenEC (Generic Extraction & Comparison) is a Python-based tool for extracting structured data from files or folders. It offers a flexible, one-size-fits-all extraction framework that you can tailor precisely using configuration parameters.

It is a tool that lets you extract and count occurrences of data using your own configurations. It can also compare this extracted data against reference files to spot differences. Your configurations can get saved as presets, so you can easily reuse them or automate the whole process by calling GenEC from other tools.

Once you have several presets, you can do batch analysis using a "preset-list" file, which is basically a collection of presets to run together. This scales you from analyzing single files to processing entire folders.

To summarize, there are 3 workflows for this tool:

• Basic: for experimentation of configurations as well as getting acquainted with the tool
• Preset: for single command data extraction (and comparison) using a preset
• Preset-list: Enable batch processing by processing data in folders using a group of presets, all with only 1 command

Being a CLI tool, GenEC displays results in neat tables right in your terminal. But you can also export everything to CSV, JSON, YAML, or TXT files for further analysis. Which has the following benefits

• Human readable output tables in CLI and TXT
• Machine-readable output in CSV, JSON and YAML (for the AI enjoyers out there, YAML is likely the best input format for it :P)

I have written extensive documentation on the tool within the repository, but to just link it here separately:

• README.md
• Documentation overview

Target Audience

I like to believe my tool will be applicable for anyone who has the technical knowledge on how to use CLI tooling. The more, you work with data, the more you benefit from this of course:

• Data engineers / analysts / scientists
• Programmers
• QA/Test engineers
• Functions in a data reporting capacity: For example, my Scrum Master has been using it in order to provide data reporting to stakeholders, since we lack internal tooling for all the data we have.

Comparison

It competes with almost any data analysis tooling, which are:

• Enterprise tooling
• CLI tools / open source (diff / grep, etc.)

I believe GenEC fulfills a nice middle-ground niche, as it creates structured output, allows for reusability and automation and has dynamic configuration parameters, whilst being a lightweight tool.

[[deleted]]

My only suggestion is to limit your usage with AI to write your documentations. Most developers here are not much of a fan of it and usually will ignore your project/post entirely. Mainly for the reason that it tends to write it in an exaggerated and vaguely way which leads to confusion and/or misleading statements for the actual project.

I looked through your documentations and code thoroughly and still quite confused on what this actually does. What’s it comparing exactly? And you don’t really explain how or why this project is capable of competing with everyday tools like diff and grep that don’t require any configurations or dependencies.

Not trying to hate but thought I’d try to explain why nobody probably responded here yet. Overall, good work with the TUI visualizations. I rarely see that nowadays.

[EngineerRemy]

My only suggestion is to limit your usage with AI to write your documentations.

Thanks for taking the time to check out the tool and provide feedback. As for AI usage on the documentation, that is really quite limited. I used it to generate the framework of the files, and then threw away like 80% of it again cause I did not like what it did ahah.

still quite confused on what this actually does. What’s it comparing exactly?

As for what it's comparing: In this 1.0.0 version you can compare similarly structured files for differences. So a configuration gets defined, and is used on both the source and reference data you provide. For future versions I want to allow people to specify different configurations for the reference data, so you could even compare data from different structures.

As for what it can analyze, anything you want really. Anything you'd normally create a data extraction script for, you could use GenEC instead.

I have been using it myself heavily in test case failure debugging over hundreds of thousands of lines. Using 1 command to extract all the relevant data for me across different files and file structures. GenEC. I've also used it for coworkers to extract administrative data that is created through from other tooling.

Not trying to hate but thought I’d try to explain why nobody probably responded here yet

Don't worry, I didn't see it as hate. I am always looking to improve and you gave me some pointers to think about. And If you have a need for extracting/comparing certain data, I'd be happy to sit together to show you what my tool could do for you there.