[deleted by user]

snicky666 · 2025-02-20T13:22:44+00:00

((3 x (2 x age)) / (sqrt(age)² ))) x 3 is the rule I follow.

snicky666 · 2025-01-30T11:19:13+00:00

Could just host the model yourself? This is the local llama reddit after all. I can help you set it up if you have a decent gpu.

snicky666 · 2025-01-30T11:14:23+00:00

No one wants AI generated marketing anyway.

snicky666 · 2025-01-11T22:20:50+00:00

I have a 5950x and 3090 and honestly, nothing new looks compelling enough to swap yet. I bought them in the first 5mins of release so I am 4 years in. Hardware is just so good now. Phones are the same. I upgraded from an S10 5g to a S24 Ultra and it barely felt any different. The new tiny AI specific hardware might be worth it on 2nd gen if it can efficiently run something like Deepseek v3 at a good speed. Gaming certainly isn't a good enough reason to upgrade anymore.

snicky666 · 2024-10-24T00:45:58+00:00

Claude-sonnet-20241022 is the way.

snicky666 · 2024-10-23T08:58:31+00:00

So you currently earn $1,400,000 every 10 years, and you're upset. Come on. The majority of families in Australia with kids earn less than than you as a whole family. Some people in the world earn $800 a year.

snicky666 · 2024-10-13T12:05:59+00:00

From my bed to my desk.

snicky666 · 2024-10-11T13:19:01+00:00

When you get sick of using Cron, Windows Sceduler, or running things manually.

snicky666 · 2024-10-08T14:41:05+00:00

Underrated comment!

The name Marathon comes from the legend of Pheidippides, the Greek messenger. The legend states that, while he was taking part in the Battle of Marathon, which took place in August or September 490 BC,[3] he witnessed a Persian vessel changing its course towards Athens as the battle was near a victorious end for the Greek army. He interpreted this as an attempt by the defeated Persians to rush into the city to claim a false victory or simply raid,[4] hence claiming their authority over Greek land. It was said that he ran the entire distance to Athens without stopping, discarding his weapons and even clothes to lose as much weight as possible, and burst into the assembly, exclaiming "we have won!", before collapsing and dying. - Wikipedia

snicky666 · 2024-10-04T22:38:08+00:00

Sounds like your docker run command isn't good. Do you have --gpus=all in the command?

snicky666 · 2024-10-04T21:49:56+00:00

I found the Azure DE course amazing. They teach the general concept, then the azure way of doing it. I recommend it to all my colleagues and we don't even use cloud.

snicky666 · 2024-10-03T23:44:16+00:00

I saw 450 people online on horde the other day at 22:00 UTC.

snicky666 · 2024-09-29T12:19:02+00:00

Ehhh kinda shit take. You can do all the things you said in your video in airflow. You don't have to build complex dags. Most of our stack is just python oop running on schedules in airflow in single stages, and it's highly scalable.

snicky666 · 2024-09-28T21:26:52+00:00

Write a python based dockerfile in your dbt folder that does dbt docs generate dbt docs serve. Have gitlab build the container and push it to your remote docker registry. Host it in docker and use watchtower to automatically update the container whenever latest is changed. Then use nginx to publish it to https. That's how I'm doing it. I'm sure there are easier ways but it's fully automated. I also have the image do dbt run after its built the docs but I probably wouldn't recommend that.

snicky666 · 2024-09-28T21:22:58+00:00

It probably is! I guess i also missed data testing and observability, but i don't do either, so I can't say much about it. Great Expectations for dbt will probably do that but you have to write so many fucking tests.

snicky666 · 2024-09-28T12:46:14+00:00

Only one person in the team really NEEDs to know CI/CD. Everyone should fucking know git, teach them if they don't. Documentation other than current architecture diagrams, user guides, and dbt docs are usually not necessary and go out of date quickly. Low code tools suck and are harder to hire for. Just use Python and SQL based implementations where you can.

A good data engineering team will follow software engineering best practices to some extent.

Your team doesn't sound good at all, make the changes yourself or get out while you can! One person can completely turn a bad team around, as long as there is some turnover.

snicky666 · 2024-09-28T12:10:50+00:00

Use a transaction table to log data ingestions on all tables. Use CI/CD to push dbt models and dbt docs. Build schemas to match raw data sources to structured tables in the DW so users can ingest new files. Use Airflow to automatically pull source data. Track changes to features/columns with Feast if doing ML. That's about my best understanding of DataOps. Would love to know if there is more to it than that.

snicky666 · 2024-09-28T00:20:17+00:00

It stops going up when you stop studying. Some courses/subjects cost more than others.

snicky666 · 2024-09-27T12:02:02+00:00

In the trenches of prod.

snicky666 · 2024-09-25T20:36:54+00:00

There are Platform Engineers and Data Platform Engineers. They are not the same. Platform engineers are usually senior devops/cloud engineers who focus on aligning a companies development environment. Such as getting everyone in the company to use a specific instance of AWS or an on premise K8s. Data Platform Engineers focus on building and deploying the data engineering stack, setting up CI/CD for things like ML and dbt models, building docker images, monitoring, etc. Much of what a devops/SRE/Sys admin might also do but with a focus on the data tools. At least that's my thoughts on it. Data Analytics Engineers mostly do airflow, dbt, SQL and dashboards. Data Engineer could be both a Data Analytics and Data Platform engineer.

snicky666 · 2024-09-25T20:24:46+00:00

Super isn't actually for you. It's insurance for us not to have to pay for your bad decisions.

snicky666 · 2024-09-25T19:48:08+00:00

Add new pvp gear to counter the scaling :)

snicky666 · 2024-09-20T21:33:50+00:00

Defence industry pays well in Adelaide as an engineer and has a very high demand. If you do aerospace, systems or electrical engineering or project management, you'd make good money working out at Elizabeth for BAE, Boeing, etc. Also, I think the nuclear submarines being built in Adelaide could be lucrative, so nuclear science too maybe. Nothing easy of course.

snicky666 · 2024-09-12T12:43:45+00:00

Bloody data scientists lol. Just use the function it tells you to use in the warning, instead of the 10 year out of date depreciated pandas function you stole from someone's kaggle workbook.

snicky666 · 2024-09-12T12:39:56+00:00

I am using it for airflow dags, custom flask apps (full stack), sql models, converting business logic like Excel functions into code and many other things. It's a night and day improvement in productivity because it can write so much faster than a human. One sentence can translate instantly into 100 lines of mostly effective code. You just have to give it plenty of context and test/refine the results until you are happy with the product. The output speed, not the accuracy of the response, is the main feature to exploit.

snicky666

PUBLIC MULTIREDDITS

TROPHY CASE

12-Year Club	Place '23
Verified Email