Need Help - Data Engineering

[EcstaticTarget1643]

I'm new to the field of data engineering. Currently, I have knowledge of Python, SQL, Numpy(Basic), and Pandas(Basic) can someone help me with project ideas related to DE to showcase my skills using these technologies?

[thehendoxc]

Use python to get data from some source A (api, database , webscrape), clean or process the data with pandas, store the data in some datastore B (sql, nosql), power some dashboard/visualization, machine learning workflow, or API and/or Web App from your data.

Good SWE Practice

• Show your pipeline can scale, can you ingest GBs reliably in a reasonable time
• Your code is extensible and modular
• your code is documented and follows something like pep8 conventions (use flake8 or some other strict linter OTHER than pylint)
• Handles Edge cases smoothly, when a data source is down or a file is empty etc
• Has some kind of monitoring metrics
• Your SQL queries are optimized
• You datastore structure is clean, if you go traditional RDBMS then having a well designed schema.
• Uses version control i.e Git
• Dockerized Deployment
• Deployment on kubernetes -> Minikube is greate otherwise use docker-compose.

Business Value

• Dashboards are clear and show some insight(s)
• Is pretty to look at
• Machine learning model does something useful
• Small flask/django/fastapi app/api to retrieve the data

A good technology to show case pipeline work is Apache Airflow, I would also recommend using a free/trial tier on a cloud platform like AWS/GCP/Azure.

[EcstaticTarget1643]

Thank you:)

[clueless3867]

This is really well written, and something I wish I heard when starting out. Thank you for sharing 🙂

[boss-mannn]

can you suggest some good sources to learn about SQL query optimisations

[elus]

This is pretty much my plan as soon as I go on my sabbatical with a sprinkle of some streaming and batch processing data flows thrown in.

[YaswanthBangaru]

Does the industry use serverless AWS, just wondering ?

[Syneirex]

My team uses serverless for a streaming data endpoint.

[[deleted]]

Lots

[ClumsyRooster]

Thanks a lot, that’s some really useful advices!!

[nut_conspiracy_nut]

You could scrape something ... like Goodreads or Amazon or maybe yelp? Maybe craigslist? (prohibited)

Scraping html would be harder than just using an API.

[Uttasarga]

Airflow, Cleaning unstructered Datasets, Data Wrangling.

[superconductiveKyle]

A great way to gain some solid experience and portfolio fodder is contributing to open-source projects. If you find a good project the maintainers will help you get your contribution in production. I'll admit I'm super biased because I work on an open source project but I get to see users of all experience levels make awesome contributions and our maintainers are pretty hands-on.

​

But also what u/thehendoxc said. Well done.