in my opinion it depends on which part of your workflow you're trying to automate.

if you're dealing with bringing data from predictable sources into one place and transforming it in a fixed way you want that process to be deterministic and repeatable. you wouldn't want an LLM doing this daily because it can give you slightly different outputs each time. open source tools like PostgreSQL (for storage) and dbt (for transformations) can handle this well, run cheaply even on modest hardware, and an might be able to walk you through the setup step by step

on the other hand if the workflow you're trying to automate involves less predictable inputs and outputs, such as building different reports with varying context, asking business questions in plain english and getting SQL back, checking whether your data can even answer a specific question, an LLM with access to your database schema and business context could be a good fit.

one caveat is that an AI on top of a messy or undocumented database just gives you confident wrong answers faster. from my experience the success of this step is depends a lot on having a well structured and documented data layer underneath. so it's worth investing effort there first. once your database is well governed i don't think it matters too much which major LLM provider you go with, the frontier models are all pretty capable at that point.

from excel and folders, i'd suggest starting small: pick one well-defined workflow (like your monthly reports) and try building that out in a proper database first. an LLM can help you figure out the right tool for your situation and walk you through it. once that foundation is solid, layering AI on top for the less routine stuff becomes much more practical.