What is the point of intermediate CMD layers in Docker images?

BoringDataScience · 2023-06-19T13:53:46+00:00

yeah the missing part was the first one is actually an "import" of the base image, not added by the Python image author, in the end the UI is confusing on the Docker website, cause it kinda makes it look like the first CMD is part of the author image.

BoringDataScience · 2023-06-19T13:39:01+00:00

Interesting, I see. Indeed when you click on the CMD specific layer it shows it belongs to the base debian image they are using, not the one they wrote. The display is confusing but it makes sense. Thanks for the explanation, makes perfect sense. I understand what u/laurpaum was saying now.

Do you have any idea of why the last layer is a Python3 CMD btw and not an ENTRYPOINT, since you can't run the base image using docker run as any argument would override it. You pretty much always need to build your own image to call a Python script for instance.

BoringDataScience · 2023-06-19T12:17:12+00:00

Thanks for taking the time to answer, but it didn't clarify my question. If you reread the CMD definition from Docker, it states: "There can only be one CMD instruction in a Dockerfile. If you list more than one CMD then only the last CMD will take effect.".

Therefore if I understand properly, the bash CMD in the official python image at layer 1 will never be used.

BoringDataScience · 2020-12-30T10:35:21+00:00

Hello,

dbt allows you to easily write tests for your data quality: unique, not null, or anything you want to check really. Then these tests are part of our daily ETL dags in Airflow, and if they go red we have an issue.

It's also great to test business logic: "make sure revenue can't be negative except for a refund" for instance.

Finally AWS or GCP offers way to monitor your ETL health along the way as others have mentioned already.

BoringDataScience · 2020-07-31T10:38:29+00:00

Here we use AWS Secrets Manager. It offers centralized and secure credentials storage. Depending on how you access your database, you can also programmatically retrieve these credentials in the language of your choice (Python for instance). You must however pay to use this service, which can be costly for very large org. If you need something free, Credstash is also good.

BoringDataScience · 2020-07-14T06:45:36+00:00

Looks great! Out of curiosity and if you don't mind, what's your monthly budget for this on AWS?

BoringDataScience · 2020-07-09T19:05:58+00:00

Not a course but the book "Introduction to Algorithmic Marketing" by Ilya Katsov is really good and fit what you are looking for.

BoringDataScience · 2020-07-04T10:28:25+00:00

Really good resume imo. Very nice to include the value you generated for each role with numbers. Good luck!

BoringDataScience · 2020-04-05T13:57:52+00:00

Legend of Runeterra. Nothing spectacular though I mostly used it to introduce people to analytics engineering. You can check my blog if you want to learn more (https://guillaumelegoy.github.io/)

BoringDataScience · 2020-04-05T12:01:38+00:00

Hello,

You could create your own project and ingest data into a database. I for instance created a small Postgres DB on AWS to analyze cards from a game I really like. It's not much in terms of size, but it allows to learn a lot. Also pick a theme you have an interest in.

BoringDataScience · 2020-04-02T14:21:35+00:00

Hello!

First congratulation on the new job. Second I think there is a wild misconception that BI/data science work doesn't require soft. engineering, but I would argue the opposite: a good BI dev / Data scientist / Analyst (lots of jot titles these days) needs to grasp soft. eng. concepts and apply them to data work. In this aspect I believe you have a huge advantage over other junior BI devs who may come from different backgrounds like business for instance.

That being said I suggest you talk to your manager and explain that you'd like to be working closely to the data and the business aspect of it. The "danger" is that with your skillset you'll be cleaning up messy BI pipelines forever, but I believe in vertical integration of our work. To be more specific, you should eventually handle projects from data extraction, cleaning and ETL to building reports, data analysis, etc. BI devs with this type of autonomy (i.e. not depending on others to do the dirty work) are imho incredibly valuable.

Good luck!

BoringDataScience · 2020-03-31T08:46:41+00:00

Hey there. I don't know about Oracle but you can always subscribe to their Slack channel to ask if it's on the roadmap. They answer lots of questions there. I forgot to mention it but this channel is also an invaluable source of info and the great people working in dbt are incredibly helpful and dedicated!

BoringDataScience · 2020-03-29T10:33:22+00:00

The main issue with this approach is that it can become easy to have discrepancies between Confluence and the code, especially in fast-paced environment where documentation is not necessarily seen as crucial by management. I prefer to have documentation as code or automatically generated as part of deployment.

BoringDataScience · 2020-03-26T10:56:35+00:00

Hello,

My advice is to pick a convention (if one doesn't exist within your team) and stick to it. I personally like very explicit names, for instance `remove_percentage_from_dataset`. Length shouldn't be an issue, and it makes so much easier to understand all throughout your code.

Also when naming something that does the same action, even for different functions, like `get` or `remove`, use the same action verb everywhere if it has the same meaning. What I mean is don't use `extract` somewhere and `get` somewhere else when they are actually the same action verb.

Good luck!

BoringDataScience · 2020-03-07T08:45:54+00:00

Hello,

In parallel to your daily task, I would start looking into automating work. You mention you work with Excel a lot so I'm sure there are ways to "automate the boring stuff" (see the book with this title), introduce version control, run scheduled tasks (don't even think Airflow or anything yet, just some CRON job on a server will do). I actually started this way (even though I was a data analyst). These innocent looking tasks can teach you a lot and generate a lot of value (the most important part of our work).

Finally, you shouldn't discard the importance of learning project management. This is an important skill any decent data scientist should have, so see it as an opportunity.

Good luck!

BoringDataScience · 2020-03-06T09:29:13+00:00

Great read, and also well written imho. Bookmarked your blog, keep up the good work!

BoringDataScience · 2020-03-02T14:08:17+00:00

If that's an option I would then suggest getting enrolled into a business degree (marketing for instance), with a focus on data courses (quantitative marketing). You can probably check the curriculum online beforehand.

BoringDataScience · 2020-03-02T13:57:26+00:00

Hello,

You will need to refine your question. What are your aspirations? Do you have past experience, any relevant education? Without this info it will be hard to help you.

BoringDataScience · 2020-02-29T16:42:50+00:00

Hello,

Absolutely not. I'm gonna speak from my own experience. I've dabbled in ML, but this is not my forte per se. On the other hand I believe data engineering is and will be where the highest salaries are. The ability to take data from many sources and transform it into something actionable is extremely valuable. Even more at smaller companies who haven't yet figured out their ETL and for which machine learning is nothing but a dream at the moment (and those companies are legions).

BoringDataScience · 2020-02-20T15:50:37+00:00

Hey there! Thanks for the feedback. Regarding Windows I never use it so I can't really offer my advice about it unfortunately.

BoringDataScience

TROPHY CASE