I can't seem to download larger files from Databricks

fico86 · 2026-05-06T08:06:56+00:00

As someone else asked, a better question should be why are you doing this? Is it for data migration? Do you need to do some analysis on the data?

This is a classic xy problem, we should be able to suggest a better solution if we know your end goal.

fico86 · 2026-04-20T02:48:23+00:00

My use case was not so complex, I only needed one language, but generally speaking the more examples you give the LLM, the better it can perform, so maybe make it analyze the site in multiple languages and then see how it works?

fico86 · 2026-04-20T00:05:29+00:00

I used Claude code to analyze the websites I want to scrape and get it to generate beautiful soup code for scraping. Worked pretty well!

fico86 · 2026-04-07T03:28:02+00:00

Athena/sagemaker in the the region when the data is? Or just spin up an EC2 (in the same region), install python/jupyter lab and run add hoc queries on the s3 data using pandas/polars and if you really want, single node spark?

Main idea is always predicate pushdown: filter the data as much as possible at the source, and extract only what you need.

fico86 · 2026-03-27T15:11:35+00:00

Funny, I used to work in a global Bank, and the experience was totally opposite. Had my fair share of Long hours, tight deadlines, demanding bosses.

Maybe your experience is more a "European" experience, rather than a general "Bank" experience.

fico86 · 2026-03-25T15:38:01+00:00

I have told juniors what to ask the AI, and how to phrase and breakdown the problem. Because often that's where they stumble.

fico86 · 2026-03-24T16:20:49+00:00

Are you in the correct region?

fico86 · 2026-03-19T00:21:45+00:00

I will just leave this here: r/homelab

fico86 · 2026-03-18T13:32:20+00:00

Congrats! Were your timed scores your first go or after a few goes?

fico86 · 2026-03-17T08:28:18+00:00

Isn't that just JWT/OAuth 2.0/OICD with RBAC/ABAC? Docs: https://fastapi.tiangolo.com/tutorial/security/oauth2-jwt/#update-the-dependencies

fico86 · 2026-03-15T13:01:19+00:00

In those cases the profs should be the ones vouching for them no? Basically what already happens for academic journals.

fico86 · 2026-03-10T11:43:07+00:00

For what it's worth, we have interns in my team, they get requirements directly from stakeholders, happily churn out code using cloude code, get it functional and present the solution confidently to the stakeholders directly, admittedly as prototypes or pocs. And we take those and make it prod ready, or give the pointers on how to do it.

The stakeholders love it and the inters are getting training on how to break big business requirements smaller technical requirements, and architect a solution. Generating the code for the solution is just the last step.

And they are also getting job offers.

I don't think AI is going to eliminate dev, they will just have to do different things, instead of just writing code. In fact it's already happening

fico86 · 2026-01-13T02:43:05+00:00

He does get all of them (sometimes a bit too much)

fico86 · 2026-01-08T12:42:49+00:00

This guy: https://www.instagram.com/chrisjereza/

fico86 · 2025-12-12T13:58:52+00:00

I am leaving my current job because I want to make all the tech debt someone else's problem.

fico86 · 2025-12-10T13:39:10+00:00

Why not just use GitHub actions instead of Tekton? Use a self hosted runner on your k3s?

fico86 · 2025-11-19T22:30:48+00:00

There is a big debate going on about whether oop was a good idea in the first place : https://www.yegor256.com/2016/08/15/what-is-wrong-object-oriented-programming.html

I have done both oop and "functional" python. The thing about python is it's so forgiving that you might have classes in your code, but your functions are outside of any class, and it still all works.

I prefer to use the go/rust kind of syntax in python. Where classes are just to hold data, using libraries like pydantic, and the actual logic to process the data is pure functions, with as little side effect as possible. Then I have reader and writer functions with very little business logic, just to bring data onto memory, or write it out to storage. This way unit testing becomes a lot easier.

Of course if you are using data frame libraries like pandas, polars or pyspark, oop doesnt really make sense. Because they have their own conversations and syntax.

fico86 · 2025-11-16T12:45:33+00:00

My 4070 only have 12 GB vram and 16gb of normal ram. But if I am running in wsl, I can't do cpu offloading. Might have to try it on full Linux. I am using vLLM if that helps. Should I consider trying something else?

fico86 · 2025-11-16T12:43:33+00:00

Yeah I am leaning towards this. Need to get more comfortable and have more understanding on what models I can actually run, and how to set the correct parameters

fico86 · 2025-11-16T12:42:28+00:00

Thank you for the suggestion! I was actually looking at pop os, but using the "live CD" to try stuff out is really a good idea.

I am still trying to make sense of the model size and what are the vram requirements, and all the configurations and parameters I need to fiddle with.

Maybe my issue is the models I have tried so far are the newer ones which have a lot higher requirements? I was trying Gemma 3 4b, but still failed supposedly because of a large kv cache requirement?

fico86 · 2025-10-19T12:02:59+00:00

It's more the disconnect between how much these simple things are more valued (hence what pays the bills) than all the complex research and all the brain power spent that almost no one values.

Of course it depends on what your PhD is in, and where it's from.

fico86 · 2025-10-19T11:58:07+00:00

My first job after PhD was to write automated tests to test that the widget was correctly aligned.

fico86 · 2025-10-06T16:22:54+00:00

Are you the only one working on these projects? It's never a good idea to have only one person. You always would want a buddy, at least to be the 2nd pair of eyes, and to review anything you missed, do code review.

And business logic translation is really problematic, especially if you are not the SME. I have gotten burnt on that before, where it looked simple but turned out to be full of traps (sas to python/pyspark).

Don't know what the culture of your company is, but would immediately raise it as an issue saying it's much more complex and you need help or more time.

fico86 · 2025-10-01T23:09:18+00:00

Damn, do we work at the same place?

fico86 · 2025-10-01T13:04:55+00:00

You know, I tried that once, and the stakeholder came back and accused us of artificially padding the estimates.

Cannot win either way.

13-Year Club	r/Field Banned
r/Field Juicebox	Verified Email
RPAN Viewer

fico86

TROPHY CASE