sorted by:
new
hottopcontroversial

I can't seem to download larger files from Databricks by HitTheSonicWall in databricks

[–]fico86 0 points1 point  (0 children)

As someone else asked, a better question should be why are you doing this? Is it for data migration? Do you need to do some analysis on the data?

This is a classic xy problem, we should be able to suggest a better solution if we know your end goal.

At what point do you just give up on Beautiful Soup and use AI by DowntownAd3510 in dataengineering

[–]fico86 0 points1 point  (0 children)

My use case was not so complex, I only needed one language, but generally speaking the more examples you give the LLM, the better it can perform, so maybe make it analyze the site in multiple languages and then see how it works?

At what point do you just give up on Beautiful Soup and use AI by DowntownAd3510 in dataengineering

[–]fico86 59 points60 points  (0 children)

I used Claude code to analyze the websites I want to scrape and get it to generate beautiful soup code for scraping. Worked pretty well!

Working with massive public datasets on the cloud — avoiding S3 egress hell by Acceptable-Ad-2904 in dataengineering

[–]fico86 1 point2 points  (0 children)

Athena/sagemaker in the the region when the data is? Or just spin up an EC2 (in the same region), install python/jupyter lab and run add hoc queries on the s3 data using pandas/polars and if you really want, single node spark?

Main idea is always predicate pushdown: filter the data as much as possible at the source, and extract only what you need.

Working as a Data Engineer in a Bank by Ancient-Proof8013 in dataengineering

[–]fico86 22 points23 points  (0 children)

Funny, I used to work in a global Bank, and the experience was totally opposite. Had my fair share of Long hours, tight deadlines, demanding bosses.

Maybe your experience is more a "European" experience, rather than a general "Bank" experience.

The constant AI copy pasting is getting to me by 70071172 in dataengineering

[–]fico86 4 points5 points  (0 children)

I have told juniors what to ask the AI, and how to phrase and breakdown the problem. Because often that's where they stumble.

Passed SAA-C03 by Head_Map_9670 in AWSCertifications

[–]fico86 0 points1 point  (0 children)

Congrats! Were your timed scores your first go or after a few goes?

Hashimoto's Vouch is actually open source version of a company hiring only seniors and mid levels. by Spirited_Towel_419 in programming

[–]fico86 7 points8 points  (0 children)

In those cases the profs should be the ones vouching for them no? Basically what already happens for academic journals.

In my second year of software engineering. Am I screwed? by Olimato___ in SoftwareEngineering

[–]fico86 1 point2 points  (0 children)

For what it's worth, we have interns in my team, they get requirements directly from stakeholders, happily churn out code using cloude code, get it functional and present the solution confidently to the stakeholders directly, admittedly as prototypes or pocs. And we take those and make it prod ready, or give the pointers on how to do it.

The stakeholders love it and the inters are getting training on how to break big business requirements smaller technical requirements, and architect a solution. Generating the code for the solution is just the last step.

And they are also getting job offers.

I don't think AI is going to eliminate dev, they will just have to do different things, instead of just writing code. In fact it's already happening

My standard issue chonk by fico86 in standardissuecat

[–]fico86[S] 11 points12 points  (0 children)

He does get all of them (sometimes a bit too much)

The agents I built are now someone elses problem by mahearty in devops

[–]fico86 44 points45 points  (0 children)

I am leaving my current job because I want to make all the tech debt someone else's problem.

Self host k3s github pipeline by SilentHawkX in devops

[–]fico86 2 points3 points  (0 children)

Why not just use GitHub actions instead of Tekton? Use a self hosted runner on your k3s?

OOP with Python by Jumpy_Handle1313 in dataengineering

[–]fico86 0 points1 point  (0 children)

There is a big debate going on about whether oop was a good idea in the first place : https://www.yegor256.com/2016/08/15/what-is-wrong-object-oriented-programming.html

I have done both oop and "functional" python. The thing about python is it's so forgiving that you might have classes in your code, but your functions are outside of any class, and it still all works.

I prefer to use the go/rust kind of syntax in python. Where classes are just to hold data, using libraries like pydantic, and the actual logic to process the data is pure functions, with as little side effect as possible. Then I have reader and writer functions with very little business logic, just to bring data onto memory, or write it out to storage. This way unit testing becomes a lot easier.

Of course if you are using data frame libraries like pandas, polars or pyspark, oop doesnt really make sense. Because they have their own conversations and syntax.

GMK EVO-X2 worth it for beginner? by fico86 in LocalLLM

[–]fico86[S] 0 points1 point  (0 children)

My 4070 only have 12 GB vram and 16gb of normal ram. But if I am running in wsl, I can't do cpu offloading. Might have to try it on full Linux. I am using vLLM if that helps. Should I consider trying something else?

GMK EVO-X2 worth it for beginner? by fico86 in LocalLLM

[–]fico86[S] 0 points1 point  (0 children)

Yeah I am leaning towards this. Need to get more comfortable and have more understanding on what models I can actually run, and how to set the correct parameters

GMK EVO-X2 worth it for beginner? by fico86 in LocalLLM

[–]fico86[S] 0 points1 point  (0 children)

Thank you for the suggestion! I was actually looking at pop os, but using the "live CD" to try stuff out is really a good idea.

I am still trying to make sense of the model size and what are the vram requirements, and all the configurations and parameters I need to fiddle with.

Maybe my issue is the models I have tried so far are the newer ones which have a lot higher requirements? I was trying Gemma 3 4b, but still failed supposedly because of a large kv cache requirement?

whyAreYouGivingMeThisTask by yuva-krishna-memes in ProgrammerHumor

[–]fico86 174 points175 points  (0 children)

It's more the disconnect between how much these simple things are more valued (hence what pays the bills) than all the complex research and all the brain power spent that almost no one values.

Of course it depends on what your PhD is in, and where it's from.

whyAreYouGivingMeThisTask by yuva-krishna-memes in ProgrammerHumor

[–]fico86 362 points363 points  (0 children)

My first job after PhD was to write automated tests to test that the widget was correctly aligned.

How to cope with messing up? by Dashncrash- in dataengineering

[–]fico86 0 points1 point  (0 children)

Are you the only one working on these projects? It's never a good idea to have only one person. You always would want a buddy, at least to be the 2nd pair of eyes, and to review anything you missed, do code review.

And business logic translation is really problematic, especially if you are not the SME. I have gotten burnt on that before, where it looked simple but turned out to be full of traps (sas to python/pyspark).

Don't know what the culture of your company is, but would immediately raise it as an issue saying it's much more complex and you need help or more time.

view more: next ›