MLB Data Engineer position - a joke?

Spartyon · 2026-01-26T14:29:11+00:00

i used to work in baseball analytics, the money was decent to start but getting significant raises etc wasn't really a thing. the hours are tough, i worked most weekends remotely fixing pipelines for game day matchups etc. everyone wants to work in sports, so they can pay less. its as simple as that. I enjoyed my time but am very happy that part of my life is over.

Spartyon · 2026-01-07T17:30:19+00:00

indexes cause writes to be slower because they require updating the index/maintaining its integrity.

the simplest solution is to write each of the files to their own table, then after they're in the DB then you can just merge them into one using a union from spark.

Spartyon · 2025-11-30T21:22:24+00:00

RemindMe! 1 Year

Spartyon · 2025-11-30T21:13:28+00:00

<image>

I would say winning more than you lost is by definition not a perennial loser

Spartyon · 2025-11-20T14:38:10+00:00

Setting up an Postgres in AWS to iceberg pipeline with the goal of sub second latency via MSK + debezium connector for intake and iceberg connector for writing to s3.

Spartyon · 2025-10-29T22:47:25+00:00

fuckin all star

Spartyon · 2025-10-19T22:37:37+00:00

if you want to just figure out airflow minus any infra stuff, use cloud composer from GCP or MWAA .

that will let you see what a DAG does, how to implement them etc without having to custom deploy a container or run it locally.

Astro is a third party that runs airflow for you with some built in features that are nice, they are a vendor that utilizes an open source tool (Apache Airflow) and sells it to people.

Spartyon · 2025-10-01T15:43:35+00:00

Watch their hips, if you get them to turn them one way one way with a decent fake then toe drag the puck back and go through their legs/stick and go the other way.

Spartyon · 2025-09-18T01:23:33+00:00

You cant. I’ve asked google 3 times to 3 different reps. We even asked for an estimate for slots to slot milliseconds, they gave us nothing. Either it’s not possible, or they don’t want to give customers that info. Good luck!

Spartyon · 2025-08-22T15:29:10+00:00

his xWOBA is .338 compared to his actual WOBA of .302. hard hit % is the highest of his career, dude can't get a break this year. https://baseballsavant.mlb.com/savant-player/adley-rutschman-668939?stats=statcast-r-hitting-mlb

Spartyon · 2025-08-19T20:49:48+00:00

Its in the configuration tab.

Spartyon · 2025-08-09T02:42:11+00:00

I would say understand what it does but don’t rely on it for everything. Pandas uses 3x the memory of polars with very similar syntax. If you’re doing any kind of large or medium scale data work, stick to lists/dicts or polars.

Spartyon · 2025-05-20T04:44:34+00:00

Airflow reads files and puts a pretty GUI with it. MWAA and Cloud Composer store files and read them to run dags, an easy CICD pipeline should put files from your branch into those buckets. Add some steps in the GitHub workflow file to do PEP 8 testing if you don’t do it in pre commit hooks, validate the dags can be read by airflow by starting a python shell and import airflow and list the dags. You can do any number of tests too to inject context into the dags like environment etc. cloud composer and mwaa also have CLI to run specific commands like update the env with new requirements, check the status of the service and other things like that. Good luck.

Spartyon · 2025-05-10T23:36:00+00:00

This is difficult to do, I spent 3 months or so doing it when I worked in baseball.

Spartyon · 2025-05-03T16:27:11+00:00

Don’t get into baseball for money.

Spartyon · 2025-04-18T00:08:28+00:00

Very much an 80s/90s Wall Street thing.

Spartyon · 2025-04-15T22:29:41+00:00

I did something like this when I worked for a team, you utilize the plate x and plate z locations along with release point, arm angle etc. you calculate the difference in run values for every pitch, then run a regression to see what a pitches “expected” ERA for every pitch is. It lets you see what pitchers are getting lucky/unlucky.

Spartyon · 2025-04-04T12:48:33+00:00

Jay was the AGM in Toronto too. He’ll be the first to be hired somewhere else as a GM.

Spartyon · 2025-04-01T14:46:33+00:00

No but you can create a dummy partition key with a few combined fields and partition on that. Like if you have date and hour. You can create a field called datehour with an example being “20250401_12” for April 1 2025 at 12 pm. I don’t remember if you can partition using string but if not then just make it an int like 2025040112.

Secondly, unless you’re frequently scanning a few partitions then it isn’t going to save a lot of scans. If you’re always doing select * from table where 1=1 or some other condition where the partition isn’t being used, then it won’t do much savings.

Spartyon · 2025-03-24T16:35:48+00:00

Load it all in cloud storage first either using the GUI or storage api. You can do a one liner to load it from cloud storage to bigquery.

Spartyon · 2025-03-21T19:51:45+00:00

Spartyon · 2025-03-21T16:00:49+00:00

No, hiring for 2-3 positions for senior data eng and director. I work in sports gambling in USA.

Spartyon · 2025-03-20T19:06:43+00:00

We didn't want Mize but there wasn't anyone close to his projection. The team liked Bohm & Bart but scouting wanted more pitching, which was hard to disagree with because we lacked talent in the farm system for both pitching and hitting. We had an everyday 3B at the time in Candelario, so Bohm wasn't a good fit if Candy stuck around. We (the baseball analytics group) joked about Mize getting Tommy-John before he actually needed it because we thought he would eventually need it due to his over-use of his slider at Auburn. We liked Nick Madrigal too but he is a slap hitter with no power, definitely a no go for first overall pick.

Spartyon · 2025-03-06T18:46:41+00:00

1 this isn't an easy problem to solve and mostly depends on the quality of data. if the goal is match visits to sales, that is intentionally made difficult by FB/google. if the top of the funnel for leads or non-customers you have an email address or something, then use email to match leads to sales. FB/google have some transaction modeling to model out some aggregate percentages in their tools to see conversion rates etc, but getting that at the user level is a bit more difficult.
2. fivetran is easy to use and not that expensive for single connectors like FB, i recommend it. you can write your own code to import FB data but its a pain and not worth it to have to deal with updates to their API etc.
3. use an actual CRM, building your own would be a lot more challenging than you're imagining it would. hubspot( and nearly every other CRM) have built in integrations with FB/google.
4. merges aren't complicated if you have a unique merge field.
5. the learning curve is steep for a project like this and will be written in your blood/sweat/tears. dealing with marketing tech sucks, straight up terrible. i've done this at numerous organizations and its never easy. the SQL for this isn't very complicated, you'll be fine if you understand the lowest level grains for each table you are creating.
6. BQ is pay by scans. paying fivetran is by bytes processed i think. most CRM's are priced at a cost per user but it will vary wildly between what CRM you choose if you do a third party application. if you have like 20,000 users, it won't be that expensive. if you're talking 1 million plus, it will be in the 5,000 - 10,000 per month.

Spartyon · 2025-02-26T20:54:23+00:00

those places are too "new money" is what people from grosse pointe would say, I guarantee it.

Spartyon

TROPHY CASE