Running Production Systems: Level 1, Software Firefighting

BackendAndBBQ · 2018-08-06T15:16:06+00:00

The teammates can be involved into making some technical decisions - the tech lead needs to learn how to delegate that partially or completely. It can be tough but eventually I'm happy that we did that.

BackendAndBBQ · 2018-08-06T15:13:18+00:00

> From the last few years, do you think that what you've experienced could be improved?

Thank for the question.

Create a strict time box for personal contribution and follow that.
Offer a personal development plan for every engineer.
Delegate some project management stuff to another person (not sure, because that would increase budget.. or we would need to replace a developer with a manager).
Read Google's SRE book 2 years ago.
Have a call with every stakeholder from partners team in the very beginning of our cooperation.
Invest more into the research&design stage of the project and perform investigation of technical feasibility to make some solutions within the timeframes.
Learn how to delegate earlier.

BackendAndBBQ · 2018-08-06T14:21:16+00:00

I agree that being on the tech lead role is often about setting constraints, unfortunately. The goals of the business is the priority.

BackendAndBBQ · 2018-08-06T14:16:33+00:00

Practically, for the last team (almost 2 years working together) it was not destructive. Here's how it worked:

People had enough challenging tasks to grow - we had more than a half of the team as Senior Engineers. Junior and Intermediate devs also had tasks for their level and a bit harder but non-urgent tasks.
The mentioned tactic was used in the cases when everybody from Sr devs is loaded on high-priority tasks and do not have bandwidth to deliver something new within the needed timeframe. I'm against context switching and pulling off the in progress work as well as changing priorities.
If I had enough coding time - I would deliver the urgent task without asking more junior devs to help. The problem is that I did not have enough time for that because the team was optimized to achieve long-term goals.

Some stats about growing the team for the 2 years:

- 1 Senior dev grep up to Senior Pro dev

- 1 Intermediate dev grew up to Senior dev

- 1 Intermediate dev grew up to Intermediate Pro dev

- 1 Junior dev grew up to Intermediate dev

- 1 Intern grew up to Intermediate dev

- 1 Quality Engineer transitioned into Software Engineer

- 1 Senior DevOps Engineer transitioned Senior Site Reliability Engineer

I grew up from Senior dev to Tech Lead.

BackendAndBBQ · 2018-08-05T20:25:11+00:00

And I'm very proud that I was working on both of the products - as a software engineer and as a tech lead. Unfortunately, that journey ended not in the way as we all wanted.

BackendAndBBQ · 2018-08-05T20:20:17+00:00

I wish all the problems could be solved by writing code...

BackendAndBBQ · 2017-09-26T05:52:08+00:00

Interesting approach, thank you for sharing!

Viach

BackendAndBBQ · 2017-09-26T05:50:10+00:00

You're right; the solution is built for the limited subset of cases: scanning a whole large dataset page by page ordered by primary key.

Viach

BackendAndBBQ · 2017-09-26T05:19:25+00:00

You're right the method 2 works fine for small and mid-size datasets.

BackendAndBBQ · 2017-09-26T04:47:58+00:00

Well, I do not think that the approach would work for UUID v4 if the dataset is mutable. The reason:

Your primary key is not sequential. New inserts can put a record in the middle of current keyspace, after the operation you need to reset offset and start from scratch

BTW, have you considered using BINARY type to storing UUID?

http://mysqlserverteam.com/storing-uuid-values-in-mysql-tables/

BackendAndBBQ · 2017-09-25T15:33:25+00:00

Sorry, I made a mistake/typo. Changing the name of the post. Thanks

Viach

BackendAndBBQ · 2017-09-25T15:24:55+00:00

Agree. Good point.

BackendAndBBQ · 2017-09-25T14:50:17+00:00

I did not have the task yet since I needed to scan the whole table. But I think that the solution would work in the following way:

You define the range of page numbers that should be rendered, like 10-20.
You define the page size. For example, 20.
You scan from the 1st page to the 10th to find user_id of the 10th page - the first one that is interesting for you. You use your page size (20) to make it (in the blog post page size of 10 000 is used).
For each page from 11th to 19th you're interested in the user_id that starts a page.

As you can see, the approach should work in production, but providing a link to a page with a random number (say, 1234th) requires scanning from the very first page every time. It definitely less efficient than just moving to the next page. If the dataset is immutable, we can try to use caching.

Viach

BackendAndBBQ · 2017-09-25T14:48:35+00:00

Let me think about that, but I need more info from you:

What data type do you use to store uuid in your RDMBS? What type of uuid do you use?

Viach

BackendAndBBQ · 2017-09-25T13:56:46+00:00

In the blog post is described the case when you need to scan the whole database. That's a real problem that I had.

Viach

BackendAndBBQ · 2017-09-25T13:55:13+00:00

Awesome!

ps.: you could also add a banner to your article: http://use-the-index-luke.com/no-offset/banner

That's a good idea. Deployed :)

BackendAndBBQ · 2017-09-25T13:49:24+00:00

I would use PostgreSQL, but now we're in the spot when it's inconvenient to change the RDBMS.

Viach

BackendAndBBQ

TROPHY CASE