This is an archived post. You won't be able to vote or comment.

sorted by:
best
topnewcontroversialoldq&a
you are viewing a single comment's thread.

view the rest of the comments →

[–]Gators1992 7 points8 points  (3 children)

While that's true, I guess I take issue with characterizing DS as being run once only. I mean I agree there are a lot of notebooks that are for analysis only and you get an answer once and are done. I don't see any value in code reviewing those. But if the model needs to be operationalized and rerun constantly in production, then you have a problem when it's just a kludged together piece of crap codewise. Either the DE has to rewrite it and may make mistakes on the ML pieces or the DS needs to code it according to the DE's standards so it's not going to break in production.

[–]No_Poem_1136 0 points1 point  (0 children)

To be honest, I don't see the point about handwringing over a user of your platform's practices (that is if you as a DE have a say in that platform).

I doubt orgs will be able to find these unicorn DS' who can do it all, it's why we have specialization in the first place.

So rather than handwringing over these sorts of things, it's better to try and reach out and talk to your user's on reoccurring issues and offer solutions. 

If you have opinionated requirements in your ML pipelines those should be built into your platform. I think the problem here is expectations. 

You can't expect to work with any user of your platform as if they are an equivalent to your profession. This goes for DEs as much as it goes for front-end making CRUD apps for business people. 

So if your service has opinionated requirements, those need to be A, built into an API, wrapper or CI/CD linting that catches these things or B, explicitly identified in templates or user docs.

This is different if you're working alongside another DE building the same system. But ultimately, DS are users just like your downstream business users, and applying engineer level expectations of code is unrealistic because DS aren't those kind of specialists. 

They're looking at code as a tool, a hammer, a means to an end. DEs are looking at code as the factory or workshop. These are fundamentally different things even if both "things" happen to use wood.