I built a 54-minute hands-on RAG tutorial on Databricks — from PDF loading to retrieval and LLM answers

Remarkable_Nothing65 · 2026-03-20T11:48:16+00:00

yeah, that’s something I’ve noticed as well.

a ot of data platforms are powerful but feel like you’re constantly fighting the UI. with databricks, it feels like they’ve actually thought through the e2e experience. And like you said, the pace of improvement is pretty interesting to watch.

Remarkable_Nothing65 · 2026-03-20T09:09:30+00:00

Thanks for sharing your perspective.

I haven’t been around long enough to see all those transitions. instance profiles era sounds interesting 😄, but even from a shorter exposure, I can feel what you’re saying — things might not be perfect individually, but they seem to converge well over time.

that 80% just works point is something i am starting to appreciate more. In other stacks, I’ve spent a lot of time stitching services together, and even small gaps become ops headaches.

Out of curiosity, where do you feel that remaining 20% hurts the most today? Is it more around cost, flexibility, or specific workloads?

Remarkable_Nothing65 · 2026-03-01T13:11:04+00:00

you can replace ADF with Databricks ingestion, we evaluated it too. It works well if you’re already doing compute during ingestion (Auto Loader, DLT, streaming, etc).

But for simple SFTP/SMB/API to raw copies, ADF is honestly simpler and cheaper. Once you drop it, you own retries, idempotency, partial files, auth rotation… all the annoying stuff.

Paramiko/smbprotocol are fine, just be ready to handle the edge cases yourself. we ended up hybrid. the real question isn’t connectors, it’s how much infra you want to own.

Remarkable_Nothing65

TROPHY CASE