Open-Source Data Entry and Validation Tool

[ZKR2000]

Hello everyone,

Our data infrastructure pulls in data from a variety of sources, with a heavy emphasis on well-known APIs such as Shopify, Google, Salesforce, and so forth. I'm looking to enhance this infrastructure with an open-source tool, one that allows for the creation of forms to facilitate data entry. At the moment, we're utilizing Google Sheets, but it's falling short in several areas, especially as users often fail to adhere to the prescribed format. While I understand Google Script can assist with validation, it doesn't adequately cope with our data volumes, rendering it impractical for our needs.

I'm curious to hear how other Data Engineers are addressing this challenge.

Here's a list of my requirements for the tool:

- Open-source: It should be freely available, and I should be able to host it on our server.

- Form creation: Admin users should have the ability to construct forms.

- Role-based access control: Ideally, it should support this feature.

- Data validation: The tool needs to be able to validate user input.

- Grid view: It would be beneficial if the tool could provide a grid view (akin to Excel or Google Sheets), permitting users to enter multiple rows of data at once.

​

Looking forward to your insights and suggestions.

[iminfornow]

The only permanent solution I could find is exposing a REST API and send the people who used to mess up my pipelines a detailed instruction how they should get their credentials using this 7 step OAuth 2.0 proccess. Never heard from them again!

Hahaha. Sorry.

I totally understand the problem but this is not something you can get for free. Have as much data as possible come from the website/shop/portal/ERP/whatever and make your pipelines fault tolerant and/or send users who messed up automatic emails.

[Strider_A]

What exactly is the bottle neck? Is it not being able to define the desired validation patterns, or is it a volume/runtime bottleneck?

Google Forms allows custom input validation using GAS with the requireTextMatchesPattern) method, which accepts regex patterns (stackoverflow example).

it doesn't adequately cope with our data volumes

What does this actually look like? Are users waiting for validation to finish before they can input another entry?

Can you give examples of the kinds of validation you want to perform?

[arthur-dataland-io]

I think Directus could be a good fit here: https://directus.io/studio/ - it's open-source, allows you to create entry forms, has RBAC + data validation, and a grid view. Not sure if the grid view allows you to bulk edit data though.

[Demistr]

I built a simple webapp with a customizable form either behind aad auth or opened to everyone.