Creating a Data Engineering Culture | Big Data Institute

[@pfrederick3927]

after years of doing this, i find this talk to be the most realistic view i have heard in the industry

[@vmramirez85]

I come every 2 to 3 months to this presentation. Thanks for clarifying the differences between data eng vs data sci. There's so much confusion in big organizations in these roles, and sometimes the same person (due to lack of resources) has to deal with both roles and expected to excel (at both!!). Thanks, Jesse.

[@sadeshsewnath6298]

Excellent presentation! Very helpful to make a transition from a Data Warehouse mindset to Big Data solutions.

[@1infiniteloop244]

Those distinctions only apply to big companies. In a small company, one person is supposed to do it all

[@q6nop]

I'm curious as to how he came up with the ratio of 2-5 data engineers per data scientist.

[@jessetanderson]

Experience and there's a paper from Google stating a similar thing. There is more effort/time that goes into the creation of the data pipeline than goes into the model creation and feature engineering.

[@q6nop]

Hi Jesse, thanks for responding. Can you please post a link to the google paper ?

[@jessetanderson]

@@q6nop Sorry I couldn't find it quickly.

[@jessetanderson]

@@q6nop Found it papers.nips.cc/paper/5656-hidden-technical-debt-in-machine-learning-systems.pdf

[@giriisindahouse]

A suspicious mute at 38:06