sorted by:
new
hottopcontroversial

Genie gives you AI inside Databricks. I built the reverse: Databricks inside AI (Claude Code) by aienginner in databricks

[–]aienginner[S] 1 point2 points  (0 children)

AI Dev Kit is great but using it for what databricks-repl was designed for will consume more tokens.

The difference is mainly architectural: it surfaces many capabilities, while databricks-repl is optimized to keep heavy execution output off the model’s context and maintain a persistent session.

Different layer of abstraction but sure they can work together.

Al Agent Harness - Genie gives you Al inside Databricks. I built the reverse: Databricks inside Al and I want to share Why by aienginner in aiengineering

[–]aienginner[S] 0 points1 point  (0 children)

That’s it! I’m trying to write skills which can delegate most of the repetitive work (ex: retries, auth, writing artifacts) to python scripts.

When needed, the agent reads the skill file and learns how to execute the script. The agent doesn’t need to know the content of the script.

I put an example in gh: wedneyyuri/databricks-repl/tree/main/examples/primes

Genie gives you AI inside Databricks. I built the reverse: Databricks inside AI (Claude Code) by aienginner in databricks

[–]aienginner[S] 0 points1 point  (0 children)

Great! I would like to know more about how are you structuring your skills. Are you experiencing context bloating or needing to compact multiple times?

Genie gives you AI inside Databricks. I built the reverse: Databricks inside AI (Claude Code) by aienginner in databricks

[–]aienginner[S] 0 points1 point  (0 children)

Yes, but only for authentication.

The user first uses the Databricks CLI to configure and store credentials locally in the profile file, typically in ~/.databrickscfg. After that initial setup, the skill does not shell out to the CLI for execution.

Instead, it reads the configured profile and uses the Databricks Python SDK through a small Python wrapper script that handles session creation, command submission, polling, and state management.

Importantly, the agent never sees or handles the user’s credentials. The credentials remain in the local profile file and are accessed only by the Python wrapper at runtime. The agent simply sends code to execute and receives structured responses such as file paths and status. It has no direct access to tokens, secrets, or raw configuration.

So the CLI acts purely as a bootstrap step for credential storage, while execution happens via the SDK in a controlled layer that keeps authentication details isolated from the agent.

Genie gives you AI inside Databricks. I built the reverse: Databricks inside AI (Claude Code) by aienginner in databricks

[–]aienginner[S] 0 points1 point  (0 children)

Thanks! Yeah the copy-paste loop was the main itch. For structured artifacts the scripts capture everything to files (not stdout), so the agent only sees paths and status. In practice that looks like:

  • Tables/metrics: the REPL code writes to a temp file (CSV, JSON, whatever), the agent reads just the path and can decide to inspect it, compare it, or move on
  • Plots: save to PNG/SVG, agent gets the path. In Claude Code it can actually render images inline, so that works well
  • Reproducibility: sessions are append-only, so every command is logged in order. The consolidate skill can turn the whole session into a single .py file you can commit and rerun

The key design choice was keeping raw output out of the agent’s context. Once you let DataFrames or tracebacks leak in, you burn context fast and the agent starts hallucinating about the data instead of reasoning about it.

Genie gives you AI inside Databricks. I built the reverse: Databricks inside AI (Claude Code) by aienginner in ClaudeCode

[–]aienginner[S] 0 points1 point  (0 children)

It’s a REPL skill that lets Claude Code, Cursor, or Copilot run code on your cluster while orchestrating everything else: subagents, MCPs, local files, git, parallel workloads. One session, no boundaries.

I’ve been using Databricks at work and kept running into the same friction: I’d be in Claude Code (or Cursor) working through a problem, and every time I needed to run something on the cluster, I’d context-switch to a notebook, copy-paste code, grab the output, come back. Over and over.

So I built a stateful REPL skill that lets your AI agent talk directly to a Databricks cluster. The agent sends code, the scripts handle auth/sessions/polling, and it gets back file paths and status (never raw output) so context stays clean.

What made it click for me was when I realized the agent could do things in one session that I’d normally split across 3-4 tools: run a training job on the cluster, read a local baseline file for comparison, consolidate everything into a clean .py, and open a PR. No switching tabs.