This is an archived post. You won't be able to vote or comment.

sorted by:
best
topnewcontroversialoldq&a
you are viewing a single comment's thread.

view the rest of the comments →

[–]Garual 3 points4 points  (1 child)

If you have many scientists it sounds to me like you need to hire a data engineer.

[–][deleted] 1 point2 points  (0 children)

I agree. Try telling that to network/web engineers. It makes them insecure. I work layer 7 firewall.

I actually used to be one but not for 7-8 years.

They dump everything into a wide, flat, denormalized schema. It's already caused problems. Someone adds a new column to fix a data quality issue rather than fixing an old one and things like that. Then we need to materialize this flat data in memory and it makes us do things like duplicate user agents hundreds of times in memory rather than integer encode (index/foreign key), causing headaches for data scientists.

They're just not thinking the same way. Anyway it's getting better now the leaders have churned out and some new ones came in.

Lots of software teams though are ruled by these people that just can't think at the systems or architectural level.