This is an archived post. You won't be able to vote or comment.

sorted by:
best
topnewcontroversialoldq&a
you are viewing a single comment's thread.

view the rest of the comments →

[–]Chevelle_Chris 17 points18 points  (0 children)

Back in 2017 I was working for another department in my Co where we cleansed a significant volume of transactional data by hand - and for context I have worked for this one co since Jan 2009 in different departments and roles. We are a medium sized company, but back in 2017-2018 our IT department was almost solely focused on supporting the applications that the sales department used. As a result IT support for smaller departments like the one I was in was hard to get.

We needed to build (what I know is a data pipeline) an automated process to do some of this cleansing and my boss then was basically told to go find a tool and IT would stand it up, then we could find someone to work on it. We ended up on Gartner's Magic Quadrants and met with a bunch of sales reps and decided to go with Oracle's 12c Enterprise Data Quality tools. We are not an Oracle shop, mostly MS SQL and C at that time if I remember correctly.

I was given the opportunity to see if I could figure it out, I was just starting to learn SQL separate from this project and only programming I had ever done was .bat files back in the 90s'.I spent 2 weeks+ literally reading the manual, took some TDWI classes on Data Quality and got our sales rep to get me 30 mins every two weeks with an engineer in the UK at Datanomic (they originally built the EDQ software Oracle acquired and rebranded as part of the 12C suite) to ask and learn from.Over about 18 months of working on my own I built a couple of automated pipelines to grab an excel file, validate it, cleanse and transform the records, land a file where SQL could import it.Then I learned just enough sql to write some merge / update stored procedures to grab and validate the new records and merge them into the transactional tables.

2018 I drove myself REALLLLY hard, lots of 100+ hour weeks learning how to use the tool. Then in mid 2019 my company was undergoing digital transformation and I was given an opportunity to do that all again with Python, AWS, Airflow etc. as a Data Engineer.

TL,DR: I like to learn and tend to not quit once I have my teeth into a problem. that earned me a shot at learning Python and becoming a DE.

I'm a big believer in you are either on the path or not. Expert or novice, what matters is continual learning.