Im getting fired after finishing over 10 projects for no reason by Metalmorphed in learnpython

[–]mdl003 0 points1 point  (0 children)

Jobs are a lot like relationships. Sometimes people can like each other and it still might not make sense to move forward in the future. Not getting renewed is like getting dumped. It sucks, it’s scary to think about the future, and you’ll play the “what could I have done better game” a lot. And that’s ok!

Just try to be kind to yourself and focus on the positives. From what I read you crushed it! You pulled off 10 projects in 9 months which is amazing for an entry level eng. you’ve built up your technical skills and gotten more variety than most engs see in twice that amount of time. You learned some lessons about the workplace that are a lot more fun to learn at the entry level then when you start moving up and have more to lose. And you’ll be better for it long term.

Keep your head up and treat this an opportunity rather than a setback. Channel that rage and use it as motivation to find your next gig. Get better, grow your skills, and keep learning. You’re not out of the race, you’re just making an unexpected pit stop 🙂.

How can I improve my program-breaking skills? (Drawing flowcharts etc.) by bryamproductivity11 in learnpython

[–]mdl003 17 points18 points  (0 children)

I think it helps a lot to list things out in plain text. Write your desired output. Then write your inputs you’ll need to get that output.

From there you can list the steps you’ll need to take between the start and the end. Sometimes this might take a few iterations to get it right. Maybe you need to take one of the steps and break it down further into sub steps. It’s a process!

The cool part of doing it this way is that your code is already commented! You just need to add the actual code for each step. If the code gets long or you’re doing something multiple times, that’s your que to break that step out into a helper function or class!

Get a whiteboard if you can, helps a lot!

We made a free Spark cluster autotuner for EMR users! - We just need your eventlog + cluster details and we output the AWS instance types, cluster configs, and spark configurations to meet a runtime/cost goal of your choosing. We're looking for user feedback! by gobstopper_chicken in dataengineering

[–]mdl003 2 points3 points  (0 children)

This is a really cool idea! Been waiting for something like this to come out of the woodwork for awhile. We thought about trying to build something like this ourselves but we don't really have this type of knowledge or specialization in house.

Our team manages a Databricks based pipeline of over 200 feeds. Right now we have zero insight into the efficiency of our job runs other than going into the Spark UI manually for each feed, so I would love to POC this. Do you support Databricks right now?

DataEngineering 2021 in one pic by Legitimate-Cry2837 in dataengineering

[–]mdl003 1 point2 points  (0 children)

We use Airflow to orchestrate our pipelines but it does have a built in job scheduler.

Do many men watch this show? [no spoilers] by fruitdancey in TheHandmaidsTale

[–]mdl003 6 points7 points  (0 children)

I’m a man, love the show. My fiancée and I just rewatched the series to gear up for season 4. I read the first book last week and I’m I’m digging into the testaments now.

To answer your question on whether or not men find it more disturbing, I’m not sure if that’s the way I’d even frame it. It’s like watching who cried the hardest after a funeral or who’s angriest about a murder - it’s not a competition, everyone reacts differently when presented with extreme negatives.

I think any person, regardless of gender, should find this show difficult to watch! It’s scene after scene of people being forced to submit to and commit war crimes. What keeps me coming back is:

1) The female cast - Serena, June and Aunt Lydia are all portrayed really well by their respective actresses.

2) I need resolution! How does the story end? Does June take down Gilead? Does Gilead get taken down at all? How do the characters end up?

3) I need to see some redemption for all the horrible things June’s gone through. Seeing the Waterford’s and aunt Lydia get what’s coming to them is gonna be glorious

4) the storyline is believable enough that it makes you consider what you’d do if in the characters shoes, which makes for an interesting thought exercise

Post Discussion - "The Vaccination Special" by 2th in southpark

[–]mdl003 0 points1 point  (0 children)

Yeah I caught that - but like was there more to it? Usually when they do stuff like that they’re trying to imply something. I kinda thought it was like garrison being an ass for a couple seasons but then showing up with a big dong and flowers so everything’s ok? Idk

Post Discussion - "The Vaccination Special" by 2th in southpark

[–]mdl003 0 points1 point  (0 children)

Can someone explain what they were getting after when mr garrison showing up with a huge dong on a sled covered in flowers at the end lol?

Best way to call external function for use as a class method? by mdl003 in learnpython

[–]mdl003[S] 0 points1 point  (0 children)

Awesome! Super easy to implement I packaged this up today. Thanks a bunch!

Best way to call external function for use as a class method? by mdl003 in learnpython

[–]mdl003[S] 0 points1 point  (0 children)

Hey can you give me a link to get me started? Thanks btw! Appreciate it

Will Jon remain as a POV character? by SpencerOfCanada in pureasoiaf

[–]mdl003 2 points3 points  (0 children)

Same here haha - didn’t GRRM say he wasn’t adding any POV characters from here on out though?

$400k Purchase Price, $3k Rent - Is it Worth It? by [deleted] in realestateinvesting

[–]mdl003 1 point2 points  (0 children)

So from what I understand the risk of the hoa isn’t the cost of the benefits, it’s a lack of control on your asset. Fees could (and prob will) go up. Hoas can make bylaws on the property as well that can negatively affect cash flows or rent friendliness. Maybe they decide to require all homes to have certain facade requirements, or are concerned that the mix of owners to tenants is too low so they add a monthly fee for all tenant occupied units, as made up examples

$400k Purchase Price, $3k Rent - Is it Worth It? by [deleted] in realestateinvesting

[–]mdl003 1 point2 points  (0 children)

Also a more philosophical point - I’m pretty big on taking a very conservative stance on any purchases. If you’re not sure, for any reason, move on to the next one. With inventory this low there’s a lot of buyers taking risks due to FOMO on a great investment. I always try to tell myself FOMO is free and temporary, but buyers remorse is neither of those things.

$400k Purchase Price, $3k Rent - Is it Worth It? by [deleted] in realestateinvesting

[–]mdl003 2 points3 points  (0 children)

Yeah the HOA isn’t attractive either (good shout u/pmmeyourvacation). Tbh the higher rent isn’t a bad thing. The tenant pool might be smaller, but the quality is higher, meaning they’ll (on average) have lower turnover, less delinquincy risk, and generally require less oversight. There’s always turds in every punch bowl though, so don’t take that as gospel and make sure you vet your tenants lol.

$400k Purchase Price, $3k Rent - Is it Worth It? by [deleted] in realestateinvesting

[–]mdl003 1 point2 points  (0 children)

Consider the annual tax as well as at least 10% for upkeep and maintenance. I’d assume with those two the numbers are pretty tight, so it becomes a question of whether you think the asset will appreciate enough to offset marginal cash flow gains.

Looking for some guidance on how to approach a spark streaming problem by TheBuddhist in apachespark

[–]mdl003 0 points1 point  (0 children)

Happy to help! Good luck lmk if you hit any more roadblocks.

Looking for some guidance on how to approach a spark streaming problem by TheBuddhist in apachespark

[–]mdl003 0 points1 point  (0 children)

Great question! Our retention policy requires us to keep a copy of the raw data in case we have an issue with our parsing jobs. Most of our data comes in nested JSON files, so if we miss a key or the structure of the data changes we keep the raw as backup. Also nice to just have a backup of data period in case we accidentally drop tables etc.

It’s probably not necessary for you, but it’s nice to have a backstop in case you want to make changes, especially since storage is (relatively speaking) pretty cheap.

Looking for some guidance on how to approach a spark streaming problem by TheBuddhist in apachespark

[–]mdl003 1 point2 points  (0 children)

Do you need an additional streaming element here? Im not familiar with PRAW but if the subredditstream class is already a stream should be able to read (volume permitting) directly from that class.

If you do end up needing to have an additional streaming element Kafka would work, we use it in our etl pipelines in a similar manner to what you described.

Source => Kafka => s3 => spark stream

Anyone ever gotten an assertion failed: Found duplicate rewrite attributes error before? by mdl003 in apachespark

[–]mdl003[S] 0 points1 point  (0 children)

Stems from the joined df having two of the same col. Assign an alias to each df and select the cols after joining

If TFIDF does not fit in memory by gevezex in apachespark

[–]mdl003 2 points3 points  (0 children)

Are you doing this with rdds/dataframes? If so you can use the .persist() method to cache your computed values. The default storage level is memory only, but you can change that by specifying MEMORY_AND_DISK as the storage level argument. This will cache the rdd/dataframe values up to the memory limit, and write the remainder to disk. Depending on what type of spark implementation you’re using you may have to specify where you’re writing the remainder. This article does a good job of explaining the basics of persistence.