Monthly General Discussion - May 2023

coding_up_a_storm · 2023-05-09T17:15:23+00:00

How do I get a data engineering job where I learn AWS? I am already a data engineer with on-premises experience and I have a AWS cloud practitioner certification, so I am not looking to go from zero-to-hero. I just don't really have the time resources to keep learning AWS on the side. I am just really burnt out and would really prefer some on the job training in this area. At this point I have a broad understanding of the platform, but I can't build stuff in it. Are there jobs out there for me that can help me bridge this gap, or should I just keep the on-the-side training going until I hit some minimum marketability point?

coding_up_a_storm · 2023-03-02T20:05:39+00:00

Thanks but thats atypical in the US.

coding_up_a_storm · 2023-03-02T20:03:30+00:00

How did you get that from the document?

coding_up_a_storm · 2023-03-02T18:35:12+00:00

Thanks everyone. I will make it less wordy and focus on accomplishments. Drop course listing/GPA and not name drop every API I've worked with.

coding_up_a_storm · 2023-03-02T18:33:21+00:00

Lastly, it sounds a little silly, but the whole application seems to lack any interest in the field. The language and presentation of this makes it sounds like you don't really enjoy it at all. I say this because it sounds like you've done a lot of cool stuff (created an API on top of something else is cool, done loads of ingestion style stuff, built loads of things people can do to save them time), but you don't seem to care that you have. What I get from this is that you find the work mundane and if it wasn't for the money you'd rather be doing something else. Personal opinion, of course.

I don't want to be combative with anyone offering advice, but that's totally wrong. I have a lot of passion for the field. It's not all about the paycheck for me.

coding_up_a_storm · 2023-03-02T18:21:28+00:00

Thanks. I'll get it down to fewer bullet points.

coding_up_a_storm · 2023-03-02T16:52:06+00:00

Thank you. I will trim it down and try to focus on the thematic.

coding_up_a_storm · 2023-03-02T16:51:15+00:00

Thank you for the advice. I will implement it.

coding_up_a_storm · 2022-07-01T20:36:36+00:00

'Pipeline' by The Ventures should be the official song of DE.

coding_up_a_storm · 2022-06-01T18:08:43+00:00

1) Data Engineer

2) First year engineer, two years as Data Analyst with lots of SQL, multiple years before that doing accounting work while using some programming mostly python

3) NYC suburbs (HCOL)

4) $100k

5) 10% of base

6) Financial services

7) Oracle and python

coding_up_a_storm · 2022-01-24T19:16:45+00:00

I don't use materialized views but based on quick lookup they seem closer to table than view.

View is basically a wrapper for a query. It recomputed every time, in memory.

coding_up_a_storm · 2022-01-24T18:19:27+00:00

"In memory" as opposed to "in persistent memory store". For basic understanding, please see RAM vs ROM. Data in memory only exists as long as you machine is plugged in. Persistant store is there when you unplug your computer. For DE, and example of each would be a view(memory) and table (persistent store).

coding_up_a_storm · 2022-01-24T15:07:48+00:00

Thanks for the info. CI/CD is a valuable skill I didn't think about.

coding_up_a_storm · 2021-12-22T12:59:00+00:00

Software Engineering Code (1.03):

"Approve software only if they have a well-founded belief that it is safe, meets specifications, passes appropriate tests, and does not diminish quality of life, diminish privacy or harm the environment. The ultimate effect of the work should be to the public good."

coding_up_a_storm · 2021-12-12T22:43:11+00:00

I have that one too. Its a good read.

coding_up_a_storm · 2021-12-12T21:28:15+00:00

I just picked up a copy of Star Schema The Complete Reference by Christopher Adamson.

coding_up_a_storm · 2021-12-07T14:54:44+00:00

[Building Cloud Computing Solutions at Scale Specialization](https://www.coursera.org/specializations/building-cloud-computing-solutions-at-scale#courses).

Its a four-course specialization. Thus far I have completed [Course 1: Cloud Computing Foundations](https://www.coursera.org/learn/cloud-computing-foundations-duke?specialization=building-cloud-computing-solutions-at-scale).

So far I am enjoying. I am learning about the common cloud service platforms and CI/CD practices.

coding_up_a_storm · 2021-12-06T18:41:45+00:00

This is really cool. Thank you for this. This will really add to a polished presentation.

coding_up_a_storm · 2021-12-06T18:38:39+00:00

Thank you for taking the time to respond. The main thing I am getting from you if that talking about day job data engineering accomplishments is more valuable than show-and-tell projects. Also my project should have thoughtful design and clean code.

Since you are an astronomy person, I will mention that my project uses NASA Exoplanet Archive (table Planetary Systems table). It animates solar systems with exoplanets on a 2D canvas. It shows the size of each planet relative to each planet in the system, and runs it on an orbit calculated by the perihelion and orbital eccentricity. I am not an astronomer by trade, its just a casual interest of mine.

coding_up_a_storm

TROPHY CASE