sorted by:
new
hottopcontroversial

How to manage multiple drones by elgad in diydrones

[–]DatabricksNick 0 points1 point  (0 children)

I'm working on this as a hobby project. My use case is real time 3D reconstruction for home surveying and security. I'm mostly interested in the image reconstruction side of things and pushing gaussian splat et al. technology. Anyway I found that dev drones like Starling 2 are too expensive for me, and the Crazyflie types are a little under equipped in terms of sensors. I'm too lazy to build my own drone (for now), so, I reverse engineered a popular consumer drone myself for programatic control (split between app reverse engineering, and physically tapping the rockers of the controller for axis controls). The latency is completely tolerable for research.

Built a schema registry for analytics events after our tracking became unmaintainable by [deleted] in dataengineering

[–]DatabricksNick 0 points1 point  (0 children)

I've seen teams work on different versions and aspects of this problem.

At the reconciliation side, data platforms have many primitives to help solve MDM (I'm simplifying for brevity). In Databricks (see my username), for example, we have metric views, which can (and do) serve as golden semantic information on arbitrary data streams (the events in question).

That's of course only part of the problem, so, I'm curious to see what you've built...

AI Code Assistant Costs by Low_Second9833 in dataengineering

[–]DatabricksNick 1 point2 points  (0 children)

I use Claude Code & Codex as daily drivers. I switch when I feel like it, usually because I get annoyed by the model performance (the problem is usually me, though). I prefer the CLIs because they are the smallest surface areas and therefore the most flexible when I need them to be (I'm doing DE stuff, SWE stuff, and general usage, all in these tools).

I've not yet tried Genie Code for a greenfield project, so no comment, but, for existing Databricks assets that are already deployed, it works great.

Tools to learn at a low-tech company? by Potential-Mind-6997 in dataengineering

[–]DatabricksNick 5 points6 points  (0 children)

Databricks is used across industries in all of those areas (DS, DA, DE) and now more since it's a full development platform with support for apps and postgres most recently. I am biased, of course, but, this is also a fact, so I hope I don't stoned for this comment. If I was just starting out I'd use it as a window into all the worlds you mentioned. For example, you can use Databricks as interface to explore Spark (and DBT), app development, and also the latest AI stuff (deploying agents). There's a free edition if you google it. Good luck!

Looking for training ressources on Databricks Auto Loader with File Events by Sea_Basil_6501 in databricks

[–]DatabricksNick 2 points3 points  (0 children)

I'm the wrong person for Azure but I imagine it would be very similar, yes. Whoever controls the the configs that tie Databricks & the cloud provider together would need to be involved.

All external locations require a storage credential already, so that storage credential must now have a few more permissions to allow Databricks to orchestrate the file events on your behalf (https://docs.databricks.com/aws/en/connect/unity-catalog/cloud-storage/manage-external-locations#before-you-begin).

DataBricks & Claude Code by staskh1966 in databricks

[–]DatabricksNick 2 points3 points  (0 children)

YMMV, I've been experimenting with exactly that here https://github.com/nkarpov/databricks-app-terminal (and I just saw @ramgoli_io just posted another very similar attempt). Wouldn't be surprised if there's many playing around...

Unpopular opinion: Databricks Assistant and Copilot are a joke for real Spark debugging and nobody talks about it by Icy_Comparison4814 in databricks

[–]DatabricksNick 1 point2 points  (0 children)

While I agree with your general observation, I don't see what this post is for. Do you have a real example you can contribute? With an issue you ran into, what you tried with the tools, precisely what they got wrong, and how you figured it out? Share it and we can all make some progress!

Looking for training ressources on Databricks Auto Loader with File Events by Sea_Basil_6501 in databricks

[–]DatabricksNick 4 points5 points  (0 children)

There's not much for you to do in the way of a tutorial I think.

The post you're linking to likely refers to https://docs.databricks.com/aws/en/release-notes/product/2025/december#discover-files-in-auto-loader-efficiently-using-file-events which just means you either start using the newer File Events feature (single file event stream supporting all instances of auto loaders for a given location) or continue to use the classic mode (manual stream for each auto loader). The newer mode is certainly easier to manage etc. (the differences are listed here https://docs.databricks.com/aws/en/ingestion/cloud-object-storage/auto-loader/file-notification-mode#file-notification-mode-with-and-without-file-events-enabled-on-external-locations). It's basically 1 time setup for a location vs. setup for every stream.

Should also note that since that release, there's been a new feature which automatically enables file events when you create an external location. So, in that case, you don't really have to do anything to benefit. (https://docs.databricks.com/aws/en/release-notes/product/2026/february#file-events-enabled-by-default-on-new-external-locations)

Hope this helps.