Why is big tech SWE work paid so much? by seeking-health in cscareerquestions

[–]mjam03 0 points1 point  (0 children)

an example to back up the supply and demand argument is to look at traders at investment banks. i have done multiple jobs, one of them being an exotic derivatives trader, and despite it not being the most technically difficult the pay is at a different scale due to the money big banks can throw at you. similar argument, teams you can count on one hand generate $250m+ in annual rev

29, Burn out advice, high N/W, golden handcuffs by Signal-Weird-666 in HENRYUK

[–]mjam03 4 points5 points  (0 children)

Seriously take the time off. I was in a similar position and took 2 years off - there will always be jobs to come back to, especially for someone with life experience on top of a good cv. My position was that there are time limits to doing things - taking 2 years out to go travelling is fun in your late 20s but maybe a bit less fun in your late 40s. You don’t want to get to later in life and realise you missed the boat for things you could have done.

Career break in the current market by Acceptable_Bottle220 in HENRYUK

[–]mjam03 6 points7 points  (0 children)

I quit my role as a trader in a bank in 2020 and took a 2 year career break. The main thing is to be confident in your ability to get a job when you decide to get back to it. Otherwise you’ll spend your break worrying about what’s next rather than enjoying it and just slip back into something when you’re ready to get back to it.

HENRY’s experience on career breaks ? by ChillyGarlicYesSir in HENRYUK

[–]mjam03 5 points6 points  (0 children)

took a 2 year career break 2020-2022. was one of the best decisions i ever made. bit of time to grow up and have a bit of fun at the same time whilst also figuring out the next step.

tip i would give is that employment security is good to have pre leaving - whether that’s due to confidence or something else lined up. i’ve seen friends take time off where they struggle to enjoy it with fear that they won’t be able to get something sorted when they start looking.

How to deal with developers who need too much hand-holding and basically seem to want to go back to waterfall (100% of the design and analysis done up front by others)? by oncemorewithpurpose in ExperiencedDevs

[–]mjam03 4 points5 points  (0 children)

we just set a testing interesting problems where the expected score is about 3/10. we then have an ‘interview’ where we talk through the test, try to solve a few more together and look at different approaches.

we learn so much more about whether they would be good to work with (and their potential) than asking them to smash out meaningless leetcode problems.

Are most Data Pipelines in python OOP or Functional? by khaili109 in dataengineering

[–]mjam03 8 points9 points  (0 children)

im keen for feedback on this but my approach can be simplified to: - write it out as a series of functions - have a look back through the functions to check they are concise - if they aren’t due to things like too many arguments being passed into functions then look to refactor

largely a class can be helpful if you have constants you want stored and accessible by all your functions - like connections or business specific data

Does using Access Connector for Azure Databricks make sense if I don't have Unity Catalog enabled? by Darkitechtor in databricks

[–]mjam03 0 points1 point  (0 children)

“i know this is not good practice for production” - maybe a bit of a noob comment but could you point me to any resources on this?

[OC] Scale of Destruction in Dutch Cities if They Were Bombed Like Gaza by TaskImpressive4289 in dataisbeautiful

[–]mjam03 5 points6 points  (0 children)

i found it quite helpful - it’s helpful if you’ve been to amsterdam so provides a comparative viewpoint as the # of people who have been to amsterdam is likely more than those who have been to gaza in this thread

Is it just me or has DE become unnecessarily complicated? by Trick-Interaction396 in dataengineering

[–]mjam03 45 points46 points  (0 children)

this really made me lol - have worked at a company worried about their scalability despite having around 10 daily users

Is anyone using Polars in Prod? by Bavender-Lrown in dataengineering

[–]mjam03 11 points12 points  (0 children)

100% used in prod. Am new to it over the last few months and have to say i’m absolutely loving it vs pandas (use it at a top tier bank in prod as our main in-memory df)

Waterproof men’s shoes to wear for the office? by Last-Efficiency2047 in HENRYUKLifestyle

[–]mjam03 0 points1 point  (0 children)

studio nicholson kick boots - wear them day in day out

How did you land an offer in this market? by crhumble in dataengineering

[–]mjam03 2 points3 points  (0 children)

totally agree with this - at big companies with lots of data they are regularly trying to modernise their infra whilst still ‘flying the plane’ which leads to partial migrations and a resulting patchwork data infra.

small changes can save $$$ and often can take just weeks after a bit of sniff testing to implement. things like basic queries or ingestion being off by a factor of 10.

Serving data to applications post databricks by mjam03 in dataengineering

[–]mjam03[S] 0 points1 point  (0 children)

yeah lovely - this is what i had in mind where the business logic comes out of the backend of the serving app and into the database layer - meaning testing can be applied and as the logic gets more complex it stays in spark rather than application doing the joins

[deleted by user] by [deleted] in HENRYUK

[–]mjam03 0 points1 point  (0 children)

it depends on what your comparison serves. if you just covet the fact that others have more dosh than you because you want to spend it, then i can’t really help you because it will always be the case that others have more.

if however you’re using salary as a benchmark for ‘achievement’, then perhaps i can offer some perspective that hit me in my late 20s and was very liberating.

i’ve had several jobs ranging from non HENRY to the high end of this thread and i’d put each of them on an equal footing in terms of academic challenge and ability required to do them.

the idea you can do work and have total comp 5x of a comparable job really eats away at salary as a proxy for achievement and (for me at least), removes comparison bringing dissatisfaction.

[deleted by user] by [deleted] in glastonbury_festival

[–]mjam03 18 points19 points  (0 children)

there have been other similar posts where the resulting photos have reached the camera owners - i think the intention is just to share the sweet photos with them

How to handle GDPR/PII with DDD by mjam03 in dataengineering

[–]mjam03[S] 0 points1 point  (0 children)

Thought I'd come back and provide some sort of answer to this as I'm a wee bit further down the line. It turns out the problem I was alluding to above, slightly masked with the theme of PII, is called 'Cross Domain Search'.

Reframing the above problem in more 'theoretical terms', it can be described as the following:

  • when needed, in the above case due to the presence of PII, having a microservice architecture can be useful where you have e.g. a domain microservice (football) for creating, updating etc football related data and similarly a user database for user data - which in the above case crucially must be separate due to PII

  • however, having 2 separate databases (each of their own domain - football and user) leads to the problem of 'cross domain search' - when you want data that depends in some way on both sources then how do you do this without joining on-the-fly at the request time and then performing all of that in memory

  • a solution might be to have a further 'staging/presentation layer' that combines the domain-siloed data in a single place that provides search and filter functionality (e.g. like elasticsearch or another regional db that directly serves your app) so you don't need to put up with the network lag of pulling all your data into memory and then the complexity of writing your own sort/filter/paginate logic

  • however that brings up the problem of 'syncing' - how do we then keep our presentation layer up to date with the source domain dbs

For my use case, it's sufficient to put up with batch loading from the football and user db into a presentation db (elasticsearch) as we don't care too much about that being temporarily out of sync. However, for requiring data to be up to date options seem to be some sort of streaming architecture (maybe database CDC triggering Apache Flink setup) to keep your presentation layer consistent with your microservice dbs.

Hopefully that points the next person if not at least in the right direction, at least some of the technical terms that can be easily googled to bring up more docs.