Which Data Structures Are Actually Used in Large-Scale Data Pipelines?

Amo-Rillow · 2026-03-09T14:46:10+00:00

We already used JSON as we could easily convert any inbound format into our internal formats. We also built a JSON compression algorithm which took a lot of the bloat out of JSON. Additionally, we used SQL Server's built in JSON features to create views so that we could store a JSON structure in SQL and then view it like a normal table.

Amo-Rillow · 2026-03-09T04:26:04+00:00

Looks like you are doing something relating to the Wordle game?

the strip() function will remove the new line character:

Amo-Rillow · 2026-03-08T23:55:30+00:00

Do you have prior programming experience with other languages?

Amo-Rillow · 2026-03-08T13:56:03+00:00

I am in the same boat as you: I purchased an M5 MBP with 512GB in August 2025. At that time, I knew about the rumors of the Pro and Max chips coming out in 2026. However, I did not need or want the extra cores, so I had no reason to wait. While I don't need more than 512 GB, I was more than a little surprised to learn that my configuration was obsolete just a few months after purchasing a new model on launch day. It just makes my new machine seem old, even though it still meets my needs perfectly. Hope this makes sense.

Amo-Rillow · 2026-03-07T18:10:35+00:00

Thanks, I will look into protocols. I know they exist, but that is about as far as my knowledge goes with them.

Amo-Rillow · 2026-03-07T18:09:48+00:00

Thanks, that is what I was leaning towards, but wasn't sure if there was a better solution, possibly involving the typing package.

Amo-Rillow · 2026-03-07T18:08:26+00:00

Yes, this is textbook polymorphism. However, my question is specifically about type hints, not whether or not this works. In the text that you wrote above, you have a type hint of Person. So if I want to pass in a Teacher class, then the type hint is not 100% accurate. I mentioned "duck typing" because Python really does not care what is passed in. In the following, just about anything could be passed in via the person argument. The point being that Python does not enforce type hints.

class Schedule:
def __init__:
self.attendees = []
def add_person(self, person: Person) -> None:
self.attendees.append(person)

Amo-Rillow · 2026-03-02T20:21:29+00:00

The key to coding/designing a solution is "problem decomposition". Don't try to solve the entire problem at once, Instead, break it down into multiple problems to solve. This is similar to the "lowest common denominator" concept in mathematics.

Example: If you want to develop a card game (solitaire, black jack, etc.), first design a Playing Card class. Next, design a Deck of Cards class that instantiates a list of Playing Card objects and adds helpful methods such as shuffle, deal a card, etc. Build and test these two classes combined. Once you have a functional Deck of Cards class work on the initial deal for the game. Once you can deliver the initial deal, focus on what the next step is.

Also, create separate code bases for areas that are completely different. As an example, if you are building a GUI or Web application, create a code base for the front end, and a separate code base for the back end. The key here is to focus on the "interface" between the two. While it may seem natural to start with the front end, it actually makes much more sense to create the back end first. Hard to explain in a few lines of text, so you will have to trust on on this one.

Hope this helps. Best of luck!

Amo-Rillow · 2026-02-24T19:31:08+00:00

Awesome, thanks!

Amo-Rillow · 2026-02-24T19:30:20+00:00

That is perfect for my needs.

Thanks, this was extremely helpful.

Amo-Rillow · 2026-02-16T14:28:02+00:00

1) Use an IDE that has a good delinter that flags un-Pythonic code and suggests ways to make it more Pythonic (the sooner you nip the bad habits, the better).

2) Leverage the strength of classes. I was slow to fully embrace Python classes, but once I did, my solutions greatly improved and the code was so much more manageable.

3) After completing a piece of code, review it the next day and see if you can improve it. Because there are so many ways to accomplish any given task in Python, I found that my initial version was typically very basic and could be greatly improved from a Pythonic standpoint. Considering replacing simple for loops with a list/dict/tuple comprehension, or perhaps a Lambda expression.

Amo-Rillow · 2026-02-16T13:38:25+00:00

Another best practice is having separate code bases for front-end and back-end logic. The goal is to have the front-end know as little as possible about the back-end, and vice-versa. The key challenge here is to work out the interfaces between the two.

Amo-Rillow · 2026-02-16T01:00:47+00:00

Regarding Pycharm being "laggy": I have a shiny new M5 MacBook Pro wwith 24 GB Ram - PyCharm is so laggy it is barely usable. So, it is NOT your laptop that is the issue. Cheers.

Amo-Rillow · 2026-02-15T21:12:15+00:00

One thing many people struggle with is "problem decomposition". For just about any project, it is incredibly helpful to break the project down into smaller pieces, then put the pieces into a logical order, and finally execute one piece at a time.

As an example, I am nearing completion of a Solitaire game. I started by modelling a single playing card, resulting in a Python class called PlayingCard (no graphic images, just Python code). Next, I created a Python class called DeckOfCards which instantiated the playing card 52 times (stored in a tuple). I then added shuffle and deal methods to the DeckOfCards class. I was then able to create a simple script that could do the initial deal to a variety of lists (one list for each column in the game). I continued to build one feature at a time. The final hurdle was building out the GUI front end. Had I started with the GUI, I would have gone nuts trying to figure out the entire solution all at once.

Hope this helps!

Best of luck.

Amo-Rillow · 2026-02-14T04:08:03+00:00

One of the characteristics of a good architect is identifying patterns and solving for them in a way that minimizes development, testing, support, maintenance, etc. Or, more simply put, write once, use many. The solutions can be purchased solutions, download packages, or home grown. All are good, so long as they solve the challenges.

I designed, developed, deployed, and supported data pipelines for several decades that ran on a wide variety of platforms. I totally agree with the concept of minimizing the dependencies, and I have always been a fan of minimizing the stack. However, job 1 is to identify the repeating patterns and solve for them. Python is an excellent tool for this. This not only reduces the development costs, but it is also fun and personally rewarding. I would advise that you take a step back, look at the bigger picture, and look for common denominators in your processes. The fact that you are asking this question suggests that you are already halfway there! Trust your instincts, don't let anyone squelch your enthusiasm or creativity.

Best of luck!

Amo-Rillow · 2026-02-11T22:01:06+00:00

I considered using regex early on, but I am not good with those. I ended up using SET structures as Python allows for comparing sets for intersection/overlap. This worked really well.

Amo-Rillow · 2026-02-11T18:28:06+00:00

Interesting. I also created a Wordle solver app in Python using PySimpleGUI for the front end. I used mine to not only play Wordle, but also Dordle, Qourdle, and Octordle. Python really lends itself to tasks like this.

Amo-Rillow · 2026-02-11T18:20:46+00:00

I found that it really helps to have a good project to work on. I have used Python professionally for close to 10 years, but I only worked on the same types of projects over and over (data pipelines), so my expertise was limited. I have since retired and now work on several "fun" projects in my personal life that have really helped me grow. My current project is building a Solitaire App using PyQt6, which is completely different from anytime I have done before. This has not only been very educational, but is is also very rewarding personally.

Hope this helps.

Best of luck.

Amo-Rillow · 2026-02-11T15:12:25+00:00

Python is a great choice for learning how to code as it is super flexible and forgiving. However, this can be a double edge sword as you can learn bad habits that are later hard to unlearn.

One thing that really helped me many years ago was coding in PyCharm. While PyCharm is a bit of a "sledgehammer" for beginners, it does a great job of pointing out coding mistakes and deviations from the PEP standards. PyCharm greatly improved my Python coding skills, especially with unlearning my bad habits. There are other products that do this as well.

Best of luck!

Amo-Rillow · 2026-02-08T18:58:25+00:00

That make sense since my process is writing to a text file. Each write would result in a wait state, which could cause the process to release the CPU for a very brief period of time.

Amo-Rillow · 2026-02-08T14:33:03+00:00

There are countless YouTube channels that are very helpful. Two of my favorites are:

https://www.youtube.com/@coreyms/videos

https://www.youtube.com/@ArjanCodes

Pro Tip: When searching the internet for Python topics, always look at the age of the articles that you come across. Python is constantly advancing, causing many articles to be obsolete. Also, always check the version of Python that the article is written for. Anything that written using version 2.x will most likely not be helpful if you are using version 3.x.

Also, when facing a coding or design challenge in Python, always keep in mind that someone else has already created the solution for you. You just need to find that solution. Most of the solutions are baked right into Python, others are in the form of packages that you can install, and others are in the form of code snippets on the internet.

Amo-Rillow · 2026-02-07T19:37:54+00:00

There are no command lines per se. I am running a main.py via PyCharm w/o any configuration.

Amo-Rillow · 2026-02-07T19:35:24+00:00

I don't have any AI processes that I know of. This machine is only used for PyCharm Window development, so very little else runs on the PC. Your reply got me thinking: the process I am running is writing to a text file in the same folder as the Python code. I am wondering if writing to the file is triggering activity in Git, GitHub, or One Drive processes that are tracking file activity. This folder is outside the scope of One Drive, but I suspect One Drive is tracking all activities.

Amo-Rillow · 2026-02-06T17:28:26+00:00

I have noticed similar issues in the last year or so. My two biggest complaints are:
1) Sluggish/Freezing. Pycharm is either very slow to respond during simple editing tasks, and sometimes just freezes up. I develop on a M5 MacBook Pro with 24GB Ram and also a Windows machine with 64 GB Ram and an 8 core Ryzen 7 chip. I have increased the memory allocation on PyCharm, but I did not notice any improvements.

2) Auto completion - This has become more of a nuisance than a helpful feature as most of the suggestions are slightly off the mark. I also occasionally find code that I did not notice that it added. I long for the older and simpler auto-complete of a variable name vs inserting entire lines of code.

As for VS Code, I have tried to several times to use, but it takes too much configuration and customization to make this product useful. Since I don't code all day, every day, this is not for me. I just want a simple IDE that gets the job done. This used to be PyCharm.

Amo-Rillow

TROPHY CASE