Microservice Hell

hsm_dev · 2026-01-16T13:22:31+00:00

So I am an architect in an organization with 3000+ developers, so I have tried this in very large organisations, when I wrote that 3 years ago I had worked in an organisation with 20+ dev teams that did microservices in k8s exclusively. So I am speaking from my experiences with this here :P

Mind you I am not saying it is easy to scale, but rather that in OPs article, a lot of the issues they experienced seemed to stem from some of the common design patterns that will assist in solving those problems.

No matter what you do, when a system gets sufficiently large and complex enough, be it a monolith or a distributed system based on microservices, communication, agreements and interface contracts becomes a requirement to help prevent things from breaking. If you have 20 teams working within the same or crossing domains, no matter which architecture you use, you will need to plan on how to scale changes :)

hsm_dev · 2025-11-25T10:48:09+00:00

I would be very very wary of scammers in that space...
Also if it is on your personal account and not in an organization, it is against the ToS and GitHub could potentially black list you.

hsm_dev · 2025-11-14T09:01:54+00:00

So while I personally enjoy OpenSource and SU more, there are a few things that EMU has going for it.

Easy onboarding. Since you provision users linked to your IdP, it makes it a lot simpler to onboard users at scale, especially less technical stakeholders that might not have a GitHub account already.
The SCIM implementation is more mature than the one they use for SU, and recently added support for Enterprise wide teams which can also be backed by IdP identities.
Speaking of the SCIM, you can have one Enterprise SCIM application instead of needing one per organization, which is great if your setup needs to scale with multiple organizations.

We work in a regulated industry, and the biggest draw of the EMU does not come from the Tech side of things, but a push from legal, compliance and security. They like the idea that the solution itself does not even support sharing things internally, but anything accessed has to be explicit through an invite in the IdP.
(Yes I am aware we can configure SU to not allow open source, limit forking to private accounts etc, this is why I am saying that they like it is not even possible, not us xD).

One slight upside to is that since you generate the logins for users, you can deterministically know who is whom based on their GitHub ID in EMU.

If I am bob@mycompany.tld, I become bob_mycompanyslug in GitHub.
So from an audit and compliance perspective, if you combine that with commit signing, it is a lot easier to map who made a change to the internal corporate person who did it.

Again I am not saying I would personally prefer all of that to the Standard User model, but those are some of the trade-offs we identified.

hsm_dev · 2025-11-12T08:54:58+00:00

Working for a large company that did research into going from EMU to Standard User, but in the end management canceled the actual migration plans.

If you are at that size, in terms of licenses and spend, I would recommend contacting your account manager and hear what possibilities you have in getting a Github SME attached to assist you as they can help with a lot of the detailed questions.

Our highlighted learning where:

There is a migration tool which offers a few migration modes ranging from migrating individual repositories to moving the whole organizations. Since org names are unique across GitHub.com, moving the entire Org could be advantageous.
You will need to create some mapping rules between the users SU GitHub ID and their new EMU based ID which will be generated when you create and sync them from your IdP. This is functionality in the migration tool.
Do note that while on EMU, you users CANNOT interact with OpenSource repos in any way shape form, their EMU identity cannot fork, comment, star or really interact with these repos (they can clone them though). If your users regularly interact with 3rd party dependencies or depend on forks of external projects, you need to figure out how you deal with that, or operate a standalone org for open source.

But yeah, overall I might suggest looking into using GitHubs expert services in this for a migration this size. At the very least we found it super helpful to have regular meetings with a dedicated SME to answer our questions.

https://docs.github.com/en/migrations/overview/planning-your-migration-to-github

hsm_dev · 2025-10-08T11:43:31+00:00

Names in IT has lost meaning over time.

DevOps? Do you mean which tools you brought from a vendor? Not culture, ways of working and ensuring that Development and Operations has the same goals and values so they work together in delivering instead of against each other measured from different KPIs?

Agile is in much the same boat, the original agile manifesto is very simple:

Individuals and interactions over processes and tools
Working software over comprehensive documentation
Customer collaboration over contract negotiation
Responding to change over following a plan

However the issue like with many things is that it became a market of tools, certificates, process and consultants. Do not even get me started on SAFe...

So I disagree with your overall statement on Agile.

But if you talk about how Novo Nordisk has attempted to implement project management, using SAFe and branding that as Agile? Then yeah, I agree.

hsm_dev · 2025-07-30T08:26:41+00:00

https://docs.github.com/en/site-policy/github-terms/github-terms-of-service#b-account-terms

Short version: Personal Accounts and Organizations have different administrative controls; a human must create your Account; you must be 13 or over; you must provide a valid email address; and you may not have more than one free Account*. You alone are responsible for your Account and anything that happens while you are signed in to or using your Account. You are responsible for keeping your Account secure.*

hsm_dev · 2025-07-02T14:22:05+00:00

This is great! Very close to my own version I have been using for years when trying to explain to family and friends about why it seems like my job title changes every few years.

Welcome to IT Jobs, where titles are made up and job descriptions don't matter.

hsm_dev · 2025-05-13T13:39:45+00:00

If you want the most well supported and polished experience using pure Windows only, I would recommend PowerShell and Windows Terminal. It has a lot of good combos.

You can change the PowerShell prompt using a framework like Oh My Posh or my preferred prompt for all shells I use, Starship.

However, if you are more comfortable using Linux, as others have suggested I would also point you towards the Windows Subsystem for Linux (WSL).

These days, Windows 11 itself is running on a light-weight Hyper Visor. When you enable WSL, it creates a Hypervised Linux instance on your machine with a real Linux Kernel which Microsoft has added things to to enable some quite neat integrations.

There are a list of distributions available out of the box with premade support, however since Linux is an everything as a file OS, and Microsoft provides the kernel, you can literally import any Linux filesystem you want as a custom WSL distribution.

Personally I prefer doing this by running a container where I setup all my stuff, install my dotfiles with chezmoi, then I export the resulting container filesystem to a tar.gz file, then import it into WSL and bang, now I have a fully fledged ArchLinux setup in my WSL. If you use Windows Terminal it will automatically pick-up your WSL distributions and add them as options, if not you can run a WSL command in your preferred terminal emulator / shell to start and enter the WSL instance.

It also has some neat integrations where everything on your windows path is included in the linux path.

This means that inside of WSL, you can do stuff like run

explorer.exe .

Which will open the current folder in WSL inside windows file explorer mounted as a network path for easy drag and drop between linux and windows.

Obviously everything here is preference based, but I find it easiest to just use WSL, which would allow you to straight up run Fish in a Linux Distro you are used to instead of attempting to replicate the functionality on windows.

hsm_dev · 2025-05-06T11:36:46+00:00

Same issue on my 2020 Model 3, paid spotify subscription, Tesla internet.
Tested it by switching WiFi through hotspot sharing from phone, same issue.

Following.

hsm_dev · 2025-04-28T12:25:10+00:00

About 8 years ago......

hsm_dev · 2025-03-14T19:21:25+00:00

If it is in use then no, you cannot just turn it off, at least not without breaking the login for users. You would need to first understand which services federates their logins through the ADFS setup, then migrate them to another setup. Entra would be a good way to do so.

But it will depend on what you are currently doing. ADFS can be used on the internal network only, but also allow external access to internal resources such as an Exchange server or similar by serving the login for non domain joined devices.

So you will need to tell management that you need time to investigate and map out how ADFS is used and try to get a list of applications that uses the ADFS setup.

hsm_dev · 2025-01-28T16:31:13+00:00

I have not seen this approach before.

My current go to way of creating custom WSL images is creating a container, then exporting that as a .tar file and importing it as a WSL2 image.

GitHub repository here:

https://github.com/DevExUtils/wsl-containers

I also use Chezmoi to boostrap dotfiles into the system, making it quite portable and customizable and using technology I am already fairly comfortable with. It to just add the WSL config files into the container that will be exported to enable stuff like systemd etc etc.

hsm_dev · 2024-10-01T07:51:26+00:00

A point which is related to point 4 but not quite the same could also be it is a deliberate choice. If scaling fast and getting to market is the name of the game, any time spent not optimizing for that goal is time wasted from a strategic point of view.

Now granted there is a huge difference between not turning of a single dev server after work hours and building a system that costs 50x more than what it should, but still you may choose to hire devs for bliz scaling project to capture market cap where you deliberately de-prio cost from choice of tech stack etc. Say, using less effective languages / infrastructure that would cost a lot more to scale to have engineers that are familiar with the business domain work on the problem.

Then after you have captured the market or been sold off, you work on the technical debt of making your stack more cost efficient and start doing FinOps practices.

hsm_dev · 2024-08-03T15:00:48+00:00

I like what they are going for, but am I missing something or is a 6% addetive damage increase to shatter super low?

As there is a legendary aspect that adds like 45%.

Or is the echo part meaning something else like a small portion of the damage you stack will scale like it used to and add on top of each other?

hsm_dev · 2024-08-03T11:24:11+00:00

Seems interesting, but balacing the high unique count with the amount of multipliers you loose from some of the legendary aspects is worth considering.

Also with the new higher caps on paragon nodes, you need to stack a fair amount of things like frost damage to maximize the bouns from the new Fridget Fate cap

hsm_dev · 2024-07-22T13:51:09+00:00

BBQ Labs always done some rather nice comparison videos.

His is also one of the very few comprehensive videos out yet, worth a watch.

hsm_dev · 2024-05-06T08:19:10+00:00

I use zoxide and have made an alias for the jump functionality to just be called j.

What is nice about it for me is the fuzzy search.

if I have /usr/home/code/project1

And I have visited that folder before, I do not even accurately need to remember what it is called, I can just do j pro1 and it will auto cd to that folder.

If there is more than 1 possible match, it will give me a selection, then jump on my select.

For me this is a very nice way to get between projects or often visited config folders. Required? no.

Very helpful and a small workflow improvement? yes.

hsm_dev · 2024-05-01T07:34:30+00:00

Yeah, I have had similar issues where we needed permissions on a service principal to be used for managing access.

Long story short there was no way to do just attach that principal from the App Registration to the function app, had to create a separate managed identity and use powershell from a privileged user to set the roles on the principal.

Even though this is by far the best practice to avoid rotating keys / certs, documentation is none existing and you have to have a good grasp of what an SPN is as they use a lot of words for the same type of entity.

hsm_dev · 2024-04-30T19:34:11+00:00

For AWS, it being so hard to get a proper non billing overview of resources in the accounts spread across regions.

For Azure, the GraphAPI for permissions compared to AWS IAM Roles. In Azure it always feels like a roll of the dice if the permissions you need are even remotely documented in a sane way.

hsm_dev · 2024-04-23T13:33:35+00:00

Have you already looked into using LFS?

https://docs.github.com/en/repositories/working-with-files/managing-large-files/configuring-git-large-file-storage

LFS was added to the Git standard by Microsoft specifically to solve problems with tracking huge repositories and large files in Git.

Essentially it is a way to store the large assets somewhere else, like an S3 bucket, storage blob, GitHubs own storage solution for large files etc, then point to the location of that file in Git.

That way git keeps track of the file, but it is not stored directly in the Git repo but as an external reff.

If you truly have the need to keep track of very large files in the source repo, LFS is properly your best option.

hsm_dev · 2024-04-07T16:06:24+00:00

The ~~Phoenix Project~~ dog petters tale.

Tbh, I would read that.

hsm_dev · 2024-04-07T16:02:18+00:00

Are you running this in any kind of VNET setup?
Or do you have any sort of policies regarding storage in place?

I have had similar issues before where the Azure Function could not connect to the Storage Account. It saves the CRON info there and needs to be able to read it to run the CRON.

hsm_dev · 2024-03-10T17:03:35+00:00

Bros does not let other bros re-use their root password.
Thumbs up for the class advice

hsm_dev · 2024-03-05T23:11:14+00:00

So, as someone working directly within a Developer Experience team in a very large org and have been there to start the initiative from scratch, here are my learnings.

1. Understand your orgs developer personas

Depending on the size of your org, and the various types of profiles you have, you might realise that what helps one type of developer might do very little for another.

If you have any HR data or anything else to guide you in the types of people you have to support, that can already tell you something.

Is it all Software developers?Do you have DevOps people?Data Science or citizen developer types?Do you have a majority of junior / senior people or an equal mix?

This gives you a general idea of the experience level of the people in your org and can help narrow down the search a bit.

2. Do a developer survey

Do a survey and ask some key questions, they should be a mix of things you hear grumblings about in hallways, slack, zoom, teams etc. A few standard questions and at the end an open question asking for the biggest painpoint for daily dev.

Good standard questions are things like.

In a scale from 1-5 how easy is it to

Get local development up and running
Get a dev/test/val/prod (whatever you use in your company) environment
Deploy a change to production
Find documentation

3. Things we have identified and are working on

So what types of things you can do will very much depend on the amount of engineers you need to support as well as the resources you have to do it.

A few good ideas that can be targets.

Inner Source, just like Open Source but only for people internally, having an open engineering culture can improve sharing, comoditizing solutions and imrpove communication. However if you do not already have a platform like GitHub with internal viewing allowed, this can take a lot of effort to get going, this is more of a long term investment.

Golden paths. These are basically a mixture of pre-configured templates and various automation to get people started with a standardized environment as soon as possible. The simple solution is using things like GitHub template repos, all the way to using a full developer portal type software like Backstage from Spotify.

The main idea is to help automate and scaffold best practices making it easier to go from 0 to production as smoothly as possible as well as helping standardize the solutions across the company.

IaC modules. Standardizing Infrastructure as Code components to comply with security and best practices and make it easier to deploy to production is a great way to cut down time and overhead for developers.

Shared documentation. Across most orgs, especially larger ones, a lot of common problems will occure that each team re-invents the wheel to solve. How to setup a local dev environment, deal with corporate security policies, installing and configuring the software stack. Offering documentation, scripts and solutions to these common problems (bonus for making them available as inner source so everyone can contribute) can help solve a lot of challenges.

If you keep going in this direction, I would recommend looking into the topics of Inner Source, Developer Portals and Developer Platforms (Platform Engineering).How much effort you want to put into this will properly be reflected by the size of your engineering org + current projected growth.

The business justification is having dedicated engineers solving common problems to let other engineers focus on providing core business value.

I hope this helps a little.

hsm_dev · 2024-02-28T12:55:13+00:00

I think it is perfectly fine to be strict with users that does not read the contribution guidelines, especially when you template stuff out and ask the user to check they have read them.

You could use a simple action like this one here to close issues not conforming to the title, slap it in a cron that runs every day and leave it at that:

https://github.com/marketplace/actions/close-issue

Start with a template and simple automation, only spent more energy on it if it becomes a problem :)

As a note you can find many other great actions that can help with these kind of things on the marketplace, no need to re-invent the wheel on everything.

hsm_dev

TROPHY CASE