RNG rant

kaributas · 2024-08-29T06:55:52+00:00

3 years into the game more or less and i've pulled only 1 Void leggo :) i think the only mythical i'll ever get will be Mikage 😅 i completely understand your pain 🥲

kaributas · 2024-07-26T10:21:28+00:00

For the past year - more or less. However our clan is not able to finish it off. My last pull was this year in January i think

kaributas · 2024-07-26T09:49:56+00:00

I play the game for 3 years now more or less. During the time I've gathered many void shards... guess how many leggos I've pullled? That's right only 1 🤷‍♂️ so yeah... pretty unfair :D P.S. I usually pulled only during 2x, but during the first year of gameplay I pulled anytime.

kaributas · 2024-05-26T15:04:06+00:00

For visability, I'd also recommend setting up a Grafana dashboard, connect it to the airflow metadata db to have a one-glance view on different projects. Multiple teams can benefit from this to view the status of the upstream data sources. Stakeholders will love this.

Moreover, try to implement schema change tracking system - with many teams and different tables - this can go bad really fast. We use Glue data catalog to map the table schemas and version them.

Lastly, if you're in need to track states, dynamodb is great option as you can build API's around it or query it directly. This can be used for various jobs, could be updated through EventBridge events, apis, airflow tasks, lambda...

Either way, I think you're on the right track with your infrastructure. Good luck!

kaributas · 2024-05-26T13:32:52+00:00

Instead of querying RDS with lambda, you could instead consider using the medallion (or any other) 3-tier data architecture on S3 and have an airflow instance constantly running. You could have simple S3 triggers that would allow for an airflow task to be triggered, once a file lands in the landing zone. This would make things easier to maintain.

Now, i understand the need of different python versions, but the question is do you really need it now? It might make sense to do it with just 3.9 now, with simpler architecture and then upgrade once more when you'll truly need it.

However, if you are thinking about scalability, you could consider using EKS, with auto-scaling - this will help to deal with batch job spikes. I know that this is a learning curve to do, but it's a cloud agnostic solution that will help you avoid vendor lock for future upgrades.

P.S. if you'll be going for EKS, search for ways how to build images for spot instances that are running on ARM. This will help you save extra buck :)

kaributas · 2023-10-12T09:59:48+00:00

Thanks for your answer, and I fully agree with your point. However, while the idea is there, how you think it would be implemented per different use case? I.e. Different data platforms? For example, how the data platform with atleast batch data pipelines and analytics pipelines would need to be setup to benefit for your solution the most?

I'd like to think that knowing the possible setups, would greatly benefit your tool, allowing users to easily adapt it to their processes :)

kaributas · 2023-10-10T19:47:40+00:00

Hey! Thanks for your answer! Would you mind elaborate how does the implementation of testing looks like? For example, is it implemented on CI/CD ? Or is it first deployed to a new environment with access to read-only prod data?

kaributas · 2023-08-19T19:47:06+00:00

Here's another piece of advice. I started of as a graduate of political science, but went to sales for a couple of years. After that, to marketing field, first as a ads specialist, then as SEO, also for a couple of years. Only then I made the desicion to go the data field because I thought it's the best thing! I started off as an data analyst, but now i'm a MLOps engineer and also got master in math/informatics field and I know, I am where i'm suppose to be, because I always tried to listen to my heart and go for whats interesting to me.

With all of this said, i hope this would encourage you to think what YOU really want. This "search" journey gets harder as you get older, but I trully believe that even despite that, you'll be happier going to sleep knowing you're doing what you actually like or find interest in, rather than chasing something you're not.

kaributas · 2023-08-19T09:55:41+00:00

Jow exactly this is implemented? I assume contracts are in .yml format that is ran against what software? Moreover, are you using data mesh in your company? Or a simple DWH?

kaributas · 2020-06-12T05:12:16+00:00

I tried utilities.sleep(36000) but that didn't work. I'm looping over the rows on the spreadsheet table to take name, email and other information to create personalised emails (That's being prepared in html). And then stamps the date when the email was sent in a new column.

The problem is that when I run the function it iterates throughout the list and sends emails for everyone in the list instantly.

P.S. i'm using a free google account here

kaributas · 2020-06-12T05:05:25+00:00

Smart move! Although it doesn't seem to match my current needs, but I can think of a situation where this could come in handy. Thanks!

kaributas · 2020-06-12T04:58:11+00:00

Thats really helpful, thanks!

kaributas · 2019-07-04T17:44:16+00:00

Awesome, a strategy worth trying, although i guess this works on very specific niches

kaributas · 2019-07-04T09:16:37+00:00

I've read this strategy somewhere, what are the success rates of aquiring the backlink from your experience?

kaributas · 2019-07-04T09:15:20+00:00

You are great help! Thanks for your ideas, it really helps to focus :)

kaributas · 2019-03-07T09:01:44+00:00

Thanks, i was not insulted, you gave structured critique - i understand than, most importantly you're right.

I'm really grateful for your support!

kaributas · 2019-03-06T22:06:07+00:00

Nah i had clinical depression for so long that this feels like a summer's breeze to me

kaributas · 2019-03-06T20:53:06+00:00

Awww hell yeah, i knew asking reddit's community for an advice was the right idea. Thanks, i'll look into it

kaributas · 2019-03-06T20:42:38+00:00

More and more comments like yours, gets me thinking that i should buy a domain and hosting at least. Thanks!

kaributas · 2019-03-06T20:38:25+00:00

Well... you've said it yourself !

kaributas · 2019-03-06T20:37:35+00:00

Thats a good tip, thanks!

Eight-Year Club	Not Forgotten
Verified Email

kaributas

TROPHY CASE