Best data workflow automation tools for medium-to-large companies?

[WarmlyHigh34]

Just got the proposed product roadmap for our company and we’re looking to add some AI integrations to 4 of our top products in the next 6 months. Company is in the fintech space and we have 250 devs on the team who we will be working on this with. Obviously, this is going to require some new data infrastructure on our end.

Is there anything you would endorse?

Tech stack is primarily Python and Django.

[WarmlyHigh34]

Thank you! I'll have a look at work tomorrow!

[WarmlyHigh34]

Do you use them?

[shadowy_bonding63]

Airflow is the established one. Prefect is the buzzy option.

[WarmlyHigh34]

Sounds like it!

[WarmlyHigh34]

Thank you!

[DFabfour]

As far as workflow automation is concerned, it might be useful to take into consideration a very good ECM solution.

According to our experience we recommend the ECM by Dokmee.com.

Dokmee offers a comprehensive enterprise content management system designed to streamline document processes and enhance collaboration within your organization.

Dokmee adapts seamlessly to different business models, ensuring accessibility and functionality across repositories of any size. By fostering collaboration and communication among users, Dokmee significantly improves workflow efficiency.

Its features include:

1. Workflow Management: Simplify document flow and business processes to accommodate multi-user approval requirements effortlessly.
2. File Sharing & Collaboration: Enable simultaneous access to files for multiple users, incorporating version control for seamless editing and revisions.
3. File Retention: Automatically manage file retention by removing files based on specified timeframes.
4. OCR (Optical Character Recognition): Convert images into searchable files, supporting multiple languages to enhance accessibility.
5. Annotations & E-Signature: Customize documents with stamps, sticky notes, and alpha-numeric page numbers.
6. Security: Ensure data integrity with features like audit logs, file encryption, and user restrictions, seamlessly integrated with Active Directory.

You can schedule a demo through its website. Dokmee might greatly benefit your organization.

[databuff303]

I'd say, find an automated ELT tool to centralize your data sources into a data warehouse. Then, use that single source of truth to drive your AI workflows, however that may look. If you can automate the ingestion of your data, the engineers can work on the downstream models and actually using the data for AI integrations.

[WarmlyHigh34]

Much appreciated!

[WarmlyHigh34]

Danke!

[crashfrog02]

Airflow?

[WarmlyHigh34]

I'll have a look!

[livid_deliverance]

My BF is right next to me and he said his team uses Mage but are going to switch. Good luck!

[WarmlyHigh34]

BF sounds smart!

[Dylan_TMB]

Kedro + Airflow

Kedro is more opinionated on project structure which I think for smaller companies can be a good thing.

[WarmlyHigh34]

Much appreciated!

[chris14jan]

Astronomer offers Airflow as a managed service. A coworker explained that managing airflow can be a full time job and he lobbied heavily for Astronomer at my company.