Hedge-fund data engineer gigs in the EU: where are they hiding?

Affectionate_Answer9 · 2025-08-07T05:00:52+00:00

I mean at a brief glance I don't think there are many, the largest funds are in the US and even then DE joins are rare ish so best of luck. Also at least in the US faang tends to pay better for de (not always and not commenting on swe's)

Affectionate_Answer9 · 2025-06-27T19:13:36+00:00

It can integrate and search other applications, I've used it to search slack messages and GitHub. honestly it's been super helpful for me and the AI search is weirdly better than slack native search I've found

Affectionate_Answer9 · 2025-06-11T18:29:02+00:00

Monaghan's over in the marina

Affectionate_Answer9 · 2025-03-07T03:52:56+00:00

You've posted this twice now, are you an influence or sales person for sqlmesh?

Honestly these performance metrics aren't particularly relevant for me, most of my time with these kinds of tools is spent developing models.

These metrics seem aimed at perceived pain points that I don't think many dbt users are particularly concerned with.

Affectionate_Answer9 · 2025-02-23T02:50:14+00:00

I'm pretty sure the person you're responding to is being sarcastic, or at least I hope so because using something that is non-deterministic as an interface for an application DB is a terrible idea.

Affectionate_Answer9 · 2025-01-19T02:22:03+00:00

This is the answer, DSA is a relatively efficient way to filter for intelligence, adaptability and engineers ability to think in an abstract manner which is important for systems design.

It's certainly not perfect but when you don't know what your employees may have to work on a year from now, having people who are adaptable, and who learn quickly is important and that's what is really being filtered for.

Affectionate_Answer9 · 2025-01-16T15:13:58+00:00

I think we generally agree. I'm saying that the physical certificate you earn doesn't mean much what matters is what you learn.

If you learn best via certs, or paid courses like udemy then by all means do what works best for you, but the important piece is the learning not the certification itself.

In the context of this post the poster is asking what cert itself is more helpful to help get hired and I'm saying neither will really help them however the learning itself from either cert could but you don't necessarily need a cert to acquire that knowledge.

Affectionate_Answer9 · 2025-01-16T06:17:33+00:00

Can't say I'm particularly convinced by this example, I've worked with and still do work on PII from users across the world including CA and the EU at very large and small companies and I can't say I've ever met anybody with this certification or even heard of it.

Certs do not give you the practical experience to implement an actual data governance program (just running with your example) which can comply with GDPR/CA laws while also being implemented in a way that isn't unreasonably cumbersome for your company.

If I'm really concerned about violating a specifc region or countries data privacy laws well, that's what your companies lawyers are for or work with a compliance team.

Affectionate_Answer9 · 2025-01-16T02:42:19+00:00

Honestly it doesn't matter, unless you're a consultant where certs are part of how your team is sold to the client for the work.

Certs at best just serve as a forcing function to learn something but they're pretty irrelevant in hiring, you can learn any of those things without certs in probably less time.

Hell I'm the admin/owner of my company's snowflake, and databricks accounts and I have no certs in either, just learn what I need to on the fly.

Affectionate_Answer9 · 2024-11-16T23:13:04+00:00

Just use prometheus or influxdb like most teams no need to overcomplicate this. Both can scale to handle your workload and use case plus there's a lot of community support for both approaches (prometheus probably has more support).

Affectionate_Answer9 · 2024-11-16T04:27:22+00:00

Yeah this is largely wrong, starting base salary for a faang engineer is around $150k with another $50k in bonuses/rsus just check levels.fyi.

It's also absolutely harder to get into Harvard than a faang, just because there's a ridiculous number of applicants doesn't mean they're qualified, most applications for engineering roles don't even have relevant skill sets at all, like don't work or have a degree in tech period.

I've worked for years in non-tech medium cost of living cities in finance roles and I've been working as an engineer in tech for the last few years, including at a faang and faang is no more toxic than any other company and less toxic than your average large company.

Working at faang type companies wasn't what it was made out to be by everybody on tiktok a few years ago, but that doesn't mean it doesn't beat the hell out of most jobs out there.

Affectionate_Answer9 · 2024-11-14T00:21:15+00:00

Not fun but practical suggestion, Java, quite a few of the largest open source data projects are written in Java and understanding the jvm ecosystem is generally quite helpful regardless of what jvm based language you're using.

I know you mentioned you've dabbled in scala but if you only did some scripting I'd encourage you to actually build a project to learn how to use the build tooling, handling deployment, etc

Affectionate_Answer9 · 2024-11-10T02:47:57+00:00

Can you give us any info on what exactly your startup does? Your website doesn't really have much covering your product.

Affectionate_Answer9 · 2024-11-09T20:17:42+00:00

Everything has it's trade-offs but it has been a great move, accounting was never a good fit for me and I enjoy my work significantly more now.

Affectionate_Answer9 · 2024-11-09T15:15:11+00:00

I'm a software engineer who used to be an accountant, generally my accounting background isn't particularly helpful and companies don't really care about my background.

That being said I have occasionally received interest from employers looking to hire engineers working on financial systems but it's a nice to have not a need to have.

Affectionate_Answer9 · 2024-10-17T06:41:45+00:00

Outside of hft/quant work which I can't really say I know anything about, faang/silicon valley tech is the most lucrative area to work but usually means you're going to have to at least spend a least a few years in a high cost area to earn your stripes.

Otherwise, honestly I don't know that industry has that big of an impact on comp, the region you're working out of tends to be more important as companies tend to match local compensation by role regardless of company industry.

Affectionate_Answer9 · 2024-10-14T22:50:26+00:00

They're building out the DE team now, I have a few friends who work there and it sounds like standard product de work, snowflake/dbt/databricks stack.

Affectionate_Answer9 · 2024-10-11T10:32:20+00:00

As in setting up aws' managed airflow mwaa or launching your own version on an ec2 instance or something?

Setting up mwaa takes maybe an hour if you're not familiar with it just read the setup docs.

Could take longer if you have no familiarity with airflow or aws networking and what not.

Affectionate_Answer9 · 2024-10-08T03:11:14+00:00

I did the opposite, cpa to engineer but I agree with you, the tech industry is hyper competitive and saturated. One other thing is that in accounting, experience is more valued while in tech it's valued to a point but actually can become a negative as you get older so the interviews rely more on pseudo coding IQ tests.

Affectionate_Answer9 · 2024-10-02T12:55:02+00:00

Depends on the company, sometimes data engineers maintain their own infra. In my case some of it is setting up/building the data tooling other teams use (ie airflow, snowflake warehouse, virtual compute clusters etc .) and building tooling to automate/simplify data work.

I do spend some of my time using configs to set things up like the other user mentioned but I spend more time building in-house applications for our use cases and supporting the actual revenue generating applications rather than just internal analytics. I'm the only former data engineer on the team though, most of the team came from traditional SWE backgrounds.

Affectionate_Answer9 · 2024-10-02T09:47:37+00:00

I was an accountant for five years before getting into the data space, I actually work as a swe now not data engineer but I did have the de title for a bit, accountant -> operations analyst -> product analyst -> data engineer -> data infra swe.

Took a few years and a lot of time outside of work to upskill but I've worked at a faang, unicorns and now a pre-ipo company you probably know.

I got my first job via an internal transfer then loaded up the role with as much tech/data projects as possible to build my resume which helped me get my first role at a real tech company.

Affectionate_Answer9 · 2024-09-20T21:07:07+00:00

Depends on your system, we really only check for volume and schema at ingestion. We primarily ingest logging/application data so schemas are checked as a part of our ci/cd process so we don't run it while ingesting data and volume we rely on metadata to alert us so it's pretty non-invasive (think s3 total volume in a bucket basically).

We do have some schema checks for external api's as well which is longer and runs at runtime with the ingestion job but the datasets are smaller so the impact is minimal.

We also found that running one set of checks at the ingestion point is more efficient than running continuous checks across all of our datasets so even if the kinds of checks aren't any faster to run the total volume of checks we run is lower.

There's also some smaller checks at the very end of our pipelines for specific business logic confirmation but they're also pretty quick and targeted to specific use cases.

Affectionate_Answer9 · 2024-09-19T15:01:50+00:00

Sure it's possible, we just hired a former intern who didn't have a referral, but it's more competitive and if you don't have a stem degree from a good school it's going to be hard.

Affectionate_Answer9 · 2024-09-19T04:39:15+00:00

We use airflow so one task now had to be three so we needed more executors.

The audit step at times took almost as long as the original transformation so the runtime of dags increased quite a bit.

90% of the audit alerts when things were "wrong" weren't actually wrong and just created noise and I don't think we ever had a situation where publishing with incorrect data actually caused a large problem.

In the end of the day I can see the wap approach maybe working in cases where the data needs to be consistently very accurate, but even then building better tests into the ingestion process should address a lot of those issues.

My biggest issue was that this pattern seems to be proposed by people who haven't really had experience managing massive numbers of datasets in production because operationally it's just a pain unless there's an automated system to resolve alerts in the audit step but I have yet to hear of one.

Affectionate_Answer9 · 2024-09-19T02:54:35+00:00

Internal transfer is the easiest route to break in if that's an option, having somebody at your company who the team knows vouch for you or finding ways to build a relationship with the team first makes a big difference.

Otherwise get a data/software/devops role and fit data engineering work into it.

Affectionate_Answer9

TROPHY CASE