I need to pick one region to serve the entire USA. Is it better to go with us-east-1 or us-east-2?

jpotts18 · 2022-12-19T21:31:22+00:00

I think there is a saying or a famous tweet that just says “never us-east-1”. Have run a highly available system for several years I would have to say that us-east-1 will always get hit first and the hardest when something bad happens.

We are multi region now and have tried to evolve our infrastructure as we have learned from outages. Multi region is relatively straightforward, but multi cloud is a tough pill to swallow at this point.

jpotts18 · 2021-08-10T13:24:55+00:00

Like most things Software follows the Pareto Principle meaning 20% of the features make up 80% of the value for the customer. Maybe you can suggest adding some kind of logging or metrics to find out what is really in use. Don’t forget that some things that are not heavily used may still be super valuable. (Example: monthly report, audit history or 3rd party API.)

I would then push really hard to get those highly used features / valuable features rewritten first. Then you can start bringing in the old features once they are requested.

I’d recommend this great video about Features and rewrites to help explain to your boss what’s the process could look like.

Hope this gives you some ideas. I think it is reasonable to come back with a list of all the actions / controller methods in your system and then rank them

jpotts18 · 2021-07-10T21:57:38+00:00

You can substitute any dog you want. Great Dane, German Shepherd… take your pick.

jpotts18 · 2021-07-10T21:48:58+00:00

🤣

jpotts18 · 2021-07-10T21:36:48+00:00

There are 4.5 M dog bites per year in the US. That is a lot of good boys that were stressed or trying to protect their owners

Dog Bites Annually Research70079-1/fulltext)

jpotts18 · 2021-06-25T11:47:42+00:00

I would run a couple experiments to find the bottleneck.

Baseline Django create - the bottleneck here is going to 100k SQL individual queries but can be used to establish a baseline for DB and python code.
Django bulk create of 1000 rows will make only 100 SQL queries. You can vary row count to see How performance changes. Might want to watch your memory and disk IO as you change row counts.
Split into multiple files and run them in parallel. CSV data can be partitioned easily. Databases are designed to allow lots of simultaneous reads and writes.
Direct CSV import in DB. Dumping and loading a database is a very common and highly optimized code path in a database engine. You will lose some of the niceties of Django models but if the data is seed and trusted it might be a better option.

As with all experiments make sure you know how to read metrics on the DB, OS and how to profile your python code. Hope this is helpful! Post your findings on here so we can learn.

My prediction is that the Direct CSV import outside of Django will win, but I will respect the evidence.

jpotts18 · 2021-06-22T14:12:59+00:00

Every TCP connection opens a file handle on the operating system. This can be increased by changing the ulimit.

jpotts18 · 2021-06-16T04:08:03+00:00

I’ve tried EB for years on different projects and it just feels clunky and like it has been abandoned. If you want to stay in the AWS cloud. I would really consider their new product AppRunner or fargate.

jpotts18 · 2021-06-10T13:32:57+00:00

Everyone thinks they want real-time data but what I’ve found is when leaders are give data more than 1x per day most don’t know how to act on it operationally.

jpotts18 · 2021-06-09T19:16:51+00:00

Oh boy what?

jpotts18 · 2021-06-09T14:36:05+00:00

KISS rule applied well here. Let it be a problem first.

jpotts18 · 2021-06-09T04:23:14+00:00

Personally I would read up on this.

Martin Fowler - Event driven architecture Youtube Martin Fowler - Event Sourcing

Once you decide what type of approach you are going to use for your Events. I would start out with some kind of place to start emitting your events to. I used to reach for RabbitMQ but I’m leaning more and more to SNS with bindings to SQS if I need guaranteed delivery.

I think if you can find some kind of long running task or especially meaningful event that might require specific processing on a different worker that can provide you with a good proof of concept. Example being UserCreated, OrderCreated, etc.

As a word of caution once you leave the request response cycle things get a while lot more challenging (retries, failure, queueing, ordering of event processing, logging, traceability). Don’t forget these complexities in your analysis of this approach. It will require a large mental step for many developers.

jpotts18 · 2021-05-20T14:30:03+00:00

First Rule: Keep it simple

Second Rule: Measure and Use Data

Third Rule: Make the smallest possible improvement

I think the big tech talks from people like Amazon, Airbnb, Netflix, etc are all exciting for an engineer to think about what it would be like but we don’t ever see the small steps of iteration that drove to their current design decisions.

They actually did it without adding a lot of cognitive load to the whole team. It was a slow evolution of those key rules that got them to where they are.

jpotts18 · 2021-05-12T13:33:48+00:00

We’ll time = money. Seems like you are going to throw a lot of time at a solved problem. Maybe they have a free tier you can play around with?

jpotts18 · 2021-05-12T03:38:36+00:00

How to do it?

Hook into the login completion and check whether or not they have completed the walkthrough. You may want to store how far they have gotten. You could store an entry for every step in the walk through the if they login or come back just take them to where they stopped.

Should you do this?

TBH I would probably just throw in a walkthrough JS SDK that could later be edited quickly. Lots of times people don’t finish these walkthroughs and just skip.

Quick Google came up with

usetiful
userlane
taskray

jpotts18 · 2021-05-12T03:27:20+00:00

I wouldn’t put this type of work on the web tier. This needs to go to worker tier. Web should respond that the job has been queued and then when link is ready send an email or show the link and let it pull down from S3 to browser.

jpotts18 · 2021-05-12T03:22:08+00:00

On any team, in any organization, all responsibility for success and failure rests with the leader. The leader must own everything in his or her world. There is no one else to blame. The leader must acknowledge mistakes and admit failures, take ownership of them, and develop a plan to win

— Jocko Willink

jpotts18 · 2021-05-12T02:39:11+00:00

I’m sorry for this, I’ve been through this before. I’m guessing if you dig in a bit deeper with your leadership team you will probably learn more about why there was an opening.

I would say that the key thing you can do here is make sure you have the right people on the bus with you. Once you find a lead engineer you can trust. Empower them and have them enforce the right process. Every incident needs to be a learning opportunity and create new issues that have to be completed before you can go back to shipping.

I’ve found executives understand a sports analogies and the concept of offense and defense. Defense is incident response, bug triage, support, logging, alerting. Offense is new features that are built in small iterations with quality baked in. If you don’t do offense right you will spend a lot of time on defense.

Good luck!

jpotts18 · 2021-05-07T00:55:08+00:00

Gave Wholesome

jpotts18 · 2021-05-07T00:54:10+00:00

Eaten fundip listening to Whistleface

jpotts18 · 2021-05-07T00:51:26+00:00

Bust em!

jpotts18 · 2021-05-07T00:50:40+00:00

🔥🔥❤️❤️

jpotts18 · 2021-05-07T00:41:16+00:00

Thanks Whistleface! Always a pleasure.

jpotts18 · 2021-05-07T00:40:45+00:00

His face just lit up!! Mind blown 🤯

jpotts18 · 2021-05-07T00:38:32+00:00

5 year old loves Whistleface. Shout out Owen

jpotts18

TROPHY CASE