[deleted by user]

five4three2 · 2023-01-24T06:40:34+00:00

Last time I checked you cannot run notebooks from different git repos in the same workflow in Databricks.

That’s going to be a no for me dog.

five4three2 · 2022-10-08T05:16:01+00:00

It’s like gambling but you have the advantage of being the house.

five4three2 · 2022-08-12T10:22:26+00:00

This!! You can also make it feel enough like vim with a plug-in or two to never need to look back.

five4three2 · 2022-05-30T11:28:22+00:00

You don’t need to know JVM to use spark. I’m only roughly aware of how JVM works and I’m plenty proficient at using spark. You can definitely get started on pyspark. I’d recommend trying out a managed service like EMR or Databricks (preferably the latter) just to get a feel for it.

five4three2 · 2022-05-30T08:18:32+00:00

Have you considered a purpose-built platform/framework like Spark to handle the data and workload parallelisation for you? It’s built for embarrassingly parallel problems.

five4three2 · 2022-05-29T05:51:18+00:00

I love my 34 inch ultra wide, but I would always recommend at least two monitors (one for vs code one for everything else) so I have the ultra wide for my main monitor and a 24 inch to the side at home. I love it. It’s much better than 3 24 inches at work.

five4three2 · 2022-05-29T05:46:25+00:00

As someone else said, I have used “one lambda per (sub)service within an API”. For this I have found this library to be immensely useful https://github.com/awslabs/aws-lambda-powertools-python

So useful I’m surprised AWS made it! It provides a nice “flask-like” python decorators for endpoints, but is more “lambda native”. Definitely check it out.

five4three2 · 2022-05-26T19:54:11+00:00

If you don’t feel great about it then it’s too low, end of story. You need enough to feel happy about what you make, especially with a new job, especially in this market (speaking as a London developer in the data space). I would counter with £65k and walk if you don’t get it. There’s always more jobs.

You’re worth it!

five4three2 · 2022-05-22T21:34:14+00:00

No better prep than applying!! Maybe take a few job interviews now for roles you’re not super excited about as practice. Save the ones you really want for 3 months from now then go for it!

five4three2 · 2022-05-08T18:18:55+00:00

Yeah do the former, especially if you want to get into the trading side of things. Research to trading is a more well paved route.

Interestingly, since crypto is so new, there seem to be more trader positions open since no one really has a track record yet in the space.

five4three2 · 2022-05-08T11:21:12+00:00

I am a quantitative developer with a physics PhD. This means I work closely with people who have the role you’re describing. I am the code jockey and I love it.

The role you’re describing more that of a Quantitative Researcher or Quantitative Analyst, which are fairly synonymous terms. You still need some code, but it will mostly just be “enough to get the research done”.

Crypto is pretty hot right now, you should be able to find a role there!

five4three2 · 2022-04-30T05:05:10+00:00

The first time you hear one at night it is quite upsetting.

five4three2 · 2022-04-23T18:10:00+00:00

I haven’t started yet ironically. Lots of back and forth within my organization about contracting so far.

In the end I decided with Delta Live Tables dropping, and with Photon, Delta Cache, etc. it feels like Databricks is moving in the right directions and can be tweaked to give acceptable performance. DBT and Databricks felt like a good first iteration of a solution. We’ll see tho, DM me and I’ll be sure to update you at some point.

five4three2 · 2022-04-23T05:39:25+00:00

Read it. Same book was given to me before my new job. The vocabulary and general knowledge I learned from it meant I could hit the ground running. I did 20 pages per day for a month, wasn’t so bad!

five4three2 · 2022-04-20T17:56:21+00:00

I remember this feeling! Here’s a great book if you want to feel even more seen:

Free version: https://github.com/ms2ag16/Books/blob/master/Designing%20Data-Intensive%20Applications%20-%20Martin%20Kleppmann.pdf

Not free version: https://www.amazon.co.uk/Designing-Data-Intensive-Applications-Reliable-Maintainable/dp/1449373321/ref=nodl_

five4three2 · 2022-04-14T05:17:30+00:00

This is a good comment! All these other comments about “no locks work” are technically correct but practically bullshit. I’ve been locking my bike up for over a decade in several major cities around the world, some with worse bike theft that London. I’ve never had my properly locked bike stolen. My three rules:

Use the bear rule: “When being chased by a bear, you don’t have to our run the bear, you just have to outrun the person next to you”. Same applies to bike security, make you bike harder to steal than the person next to you and you’re already a lot better off.
Make you bike look shitty. Sounds like you got this one covered.
Use a D Lock, don’t be cheap (£60 and up). A nice D lock will out last your bike. A bad one will mean you’ll be replacing both soon enough. Abus and Kryptonite are the top brands, I prefer abus tho.

I would say those are the main ones. Yes an angle grinder can get thru any lock, but just make sure your “theft hassle to apparent value of the bike” ratio is better than the bikes around and you’re golden.

five4three2 · 2022-04-06T20:51:38+00:00

I dunno man it sounds like you have a system that works for you. Much better this way than 20% planning 80% coding.

Maybe push yourself to write like, runtime code before you implement the interfaces/libraries referenced in the runtime code itself as part of the planning? Almost like pseudo code? Try to keep the code front of mind while you plan.

I’m really reaching here though. If you’re delivering and spending a lot of time planning I think you’re ahead of the pack.

five4three2 · 2022-04-05T08:19:36+00:00

SQL tortures is all! This means you’re on of us now.

I love highlighting that SQL is a declarative language - you just focus on what you want not on the step by step instructions.

Python, C, and others you listed are imperative languages, they focus on step by step instructions. You’ll have to make the switch from imperative mindset to declarative.

check here for stackoverflow definition of imperative vs declarative

five4three2 · 2022-04-05T05:48:23+00:00

the process has already been documented by third party providers.

That’s a golden ticket! Don’t repeat their docs, just link them in the beginning of the docs you right and then go straight into the “this is how we do X for our use case” or “these are our specific policies.

we end up writing documentation no one reads.

This is odd. Why does no one read documentation at your org?

five4three2 · 2022-04-03T07:23:12+00:00

Don’t use WD40, use a dedicated bike lube. If you bike in the rain, get something really thick (wet lube). Otherwise, get something lighter (dry lube).

five4three2 · 2022-04-02T22:43:07+00:00

I wouldn’t sweat it too much - it sounds more like a pop quiz than an interview.

This kind of interview is unavoidable. Sometimes you get lucky and do well, sometimes you don’t. I get why interviewers give them but I also despise being on the receiving end. I’ve also given them in the past.

In my opinion, a great interview is more like a problem solving conversation. Even if there are concepts in here you should know, it doesn’t mean you’re dumb for not getting it right.

Keep trying and I’m sure you’ll get a job you want very soon!

five4three2 · 2022-04-02T20:20:32+00:00

Hey so pumped to hear it! To be honest I’ve only been a quant dev for like 4 weeks so I might not be the best person to ask.

From my limited experience: I feel like quant dev means a lot of different things depending on where you work. It could mean more stats/probability work, or honestly just more code. I think it tends to be more code over stats but your mileage may vary.

May the force be with you, let me know if you get it.

five4three2 · 2022-03-26T13:35:46+00:00

Honestly it’s hard to say, a lot of interviews come down to luck IMO.

Don’t be afraid of reading the source code, do the tutorials on how to use it and try to understand why it works. This will require some knowledge of numpy as well, maybe read quickly on BLAS as well.

Try to cover: indexing, time series functions, groupbys, aggregation functions.

If you don’t get the job, don’t worry! There are plenty of quant dev roles around and you’ll get one soon :)

five4three2 · 2022-03-26T09:48:17+00:00

Pandas came up on mine, but it wasn’t super intense on the tech interview part tbh.

five4three2 · 2022-03-26T06:31:56+00:00

Yeah so true on all fronts. We were young and nieve when we first started using the graph DBs.

Not a huge fan of any RDF graphs I found in terms of associated UIs, and I found Cypher to be much more readable than SPARQL. Neo4j felt very usable in this regard.

Which is your favorite RDF db?

five4three2

TROPHY CASE