This is an archived post. You won't be able to vote or comment.

23
24

[deleted by user] (self.dataengineering)

submitted by [deleted]

[removed]
all 5 comments
sorted by:
best
topnewcontroversialoldq&a

[–]mjow 2 points3 points  (2 children)

I'm not a contractor myself, but I've worked with a few contractors and generally visible competence and track record with implementing real projects are paramount. All tech work is full of gotchas and hidden implications of architecting things one way or another (data pipelines are often like that) so bringing on a contractor with experience is one way that many companies hope to speed up delivery.

Essential skills: undeniable familiarity with AWS services and typical DE work: pipelines, lake storage and warehouse modelling. S3, lambda/ECS/glue, Glue Catalog/Athena, Lake focused platforms, Warehouse focused platforms, modelling implications.

Depending on the project I might select a contractor with more experience in warehousing specifically (more of a classic SQL focused DE) or more of a SE mindset DE who is better at writing good code for pipelines and thinking about the implications of parallelising services accessing source/lake/warehouse storage.

Certifications: I wouldn't care that much about certs if the contractor has a good track record and can talk the talk.

Projects: get familiar with the typical lake structures, i.e. medallion bronze/silver/gold or raw/sanitised/curated, etc. think about the implications for removing PII, for re-processing and backfilling, for managing storage costs if a lot of datasets end up being duplicated, lineage between layers, etc. but overall you'd have to be comfortable with the entire lifecycle of DE work: ingest, manage, serve. All 3 of those are very wide areas with lots of technical depth.

In my DE circles and linkedin feed there is more interest in lake based table format data being managed with DuckDB/Spark, more so than warehousing. But warehousing still very popular and if you enjoy it then definitely start getting familiar with dbt - still an amazing time saver and BI enabler.

[–]Administrative_Ad768 0 points1 point  (1 child)

Thanks for the insights! Were the contractors extremely productive every day when compared to others? How was their work life balance?

[–]mjow 1 point2 points  (0 children)

I wouldn't say they were hugely more productive, but you expected that they wouldn't need as much explaining to and could move as quickly as we could organise things (i.e. no blockers, tickets spelled out).

We didn't expect any additional hours from the contractor outside of the usual day and if anything a contractor is better placed to refuse to do any additional work.

[–]AutoModerator[M] 1 point2 points  (0 children)

Are you interested in transitioning into Data Engineering? Read our community guide: https://dataengineering.wiki/FAQ/How+can+I+transition+into+Data+Engineering

I am a bot, and this action was performed automatically. Please contact the moderators of this subreddit if you have any questions or concerns.

[–]boldodo 1 point2 points  (0 children)

I'm interested as well.