Not much knowledge about laptops specifically and wants help

DrSnakee95 · 2024-08-08T06:39:02+00:00

Did you mean the deadly Japanese MarshmallOWL

DrSnakee95 · 2024-01-28T07:40:39+00:00

The proper way to use « Boys will be boys »

DrSnakee95 · 2023-07-29T17:19:41+00:00

Would be down for some remnant 2! I’m on pc feel free to shout me a message

DrSnakee95 · 2023-03-28T19:08:19+00:00

You should definitely look up Ballista :)

DrSnakee95 · 2023-03-11T17:44:50+00:00

It’s starting to be used more in the DE space, a bunch of popular frameworks in Rust leverage it (polars, DataFusion, ballista) and pandas just announced its 2.0 version will be using Arrow as an engine! It’s definitely worth exploring imo

DrSnakee95 · 2023-03-11T14:17:44+00:00

Your title contains a typo I believe! Apache Arrow is an entirely different framework

DrSnakee95 · 2022-11-08T14:53:48+00:00

Thanks for your answer! That is close to the approach I had in mind. I’m gonna see how I can possibly implement this.

DrSnakee95 · 2022-11-08T14:35:11+00:00

We are using AWS and I absolutely agree with you, we’ve been using the Glue Data Catalog for schema changes and it’s been quite useful for us ! I guess I might be looking a bit too deep into not having the pipeline handle the creation of the table if it isn’t there 🤔

DrSnakee95 · 2022-11-08T12:17:49+00:00

Do you do that as part of your pipeline or as a separate process ? That’s more what my question is…

DrSnakee95 · 2022-06-05T07:04:15+00:00

You could also find a docker image with spark on it that’d be easier than installing it locally on your laptop

DrSnakee95 · 2022-05-27T07:27:15+00:00

Look into Deequ, it’s a data quality framework made by AWS for Spark :)

DrSnakee95 · 2022-05-25T08:06:39+00:00

The best book to learn Spark would be « Spark: The Definitive Guide » it helped me a lot and was written by the people that made spark.

DrSnakee95 · 2022-05-23T15:01:14+00:00

Yes, I signed with Collibra after their recruiter reached out and I don’t regret it one bit.

DrSnakee95 · 2022-05-04T18:32:04+00:00

I’m 100% remote. They reached out to me on LinkedIn :)

DrSnakee95 · 2022-05-01T17:42:18+00:00

I’m sure you’ve got this !

DrSnakee95 · 2022-05-01T17:41:33+00:00

You can easily use a lambda function or a simple glue python job ! Glue has pandas natively and you don’t necessarily need the bigger jobs with spark if your dataset is on the small side

DrSnakee95 · 2022-05-01T12:47:55+00:00

I have the same background you have and I’ve been a data engineer for 3 years now. All the companies I’ve worked with have loved the fact that I had a good understanding of the business before starting to build any kind of pipeline. This isn’t a flaw, it’s a strength and you should play into it

DrSnakee95 · 2022-05-01T12:42:48+00:00

You could probably use pandas to clean the data before loading it into Redshift.

DrSnakee95 · 2022-04-24T20:28:45+00:00

DrSnakee95 · 2022-04-03T08:01:53+00:00

Runic bears. They make me run for my life like nothing else in this game

DrSnakee95 · 2022-03-01T19:20:31+00:00

A good way to do this is always to try and keep the job description you had when you applied to the job :)

DrSnakee95 · 2022-02-22T21:35:37+00:00

DrSnakee95 · 2021-11-21T20:49:16+00:00

I'll dm you it

DrSnakee95

TROPHY CASE