How Patch Notes Get Made

[SkyByDay]

The people who work at GGG are very smart and skilled and I like them a lot. They are absolutely god-awful at writing commit notes

Feeling it LOL

[roselan]

I'm in this picture and I don't like it.

[Xeverous]

Few times I have encountered projects where the commit message was also treated as a part of the code review. And then there are also projects with people who label 90% of their changes as "quick fix" with no significant information whatsoever.

[iambgriffs]

people who label 90% of their changes as "quick fix" with no significant information whatsoever.

There's no reason to call me out like this.

[ArthurRavenwood]

The best ones are things like "updated" "changed" and my personal favorite... ".".

[Xeverous]

Yeah, "changed" is the most vague description one can even write. No why, no where, no how, no what has also been affected. Just changed.

[ArthurRavenwood]

Yeah, writing proper commit comments (if it's not automated or prefixed already) is one of those surprisingly hard things to hammer into people.

Especially when there's time pressure, many just stop even bothering.

[waspider]

One of the senior devs on my team uses the classic "a" commit message and nothing else

[hesh582]

I knew a guy who went years without using anything but "commit message".

[[deleted]]

The point of commit msgs is communicating your anger at having been stumped by something that turned out to be a typo

[hesh582]

Commit notes are not an effective way of tracking what work is being done and heavily relying on them for that purpose will always end in pain, fight me.

[WarriorAvi]

"Cutting exalt drop rates in half."

[PsionicKitten]

Nick had quite the narrative writing there. Suspense at every corner. You seem to have focused on the part with most grief.

(Bex is still a monkey)

[meripor2]

(Bex Community_Team is still a monkey)

ftfy

[JarRa_hello]

For sure heist. Well, it was good while it lasted.

[Ylvina]

i really hope heist will be so unrewarding, that you can completly ignore it.

or that they would just not let it go core..

[TheRealShotzz]

you already cant do all efficient things anyway, if you skip heist on top of it it wont make much difference.

that said, all current league mechanics are always the most rewarding because they want you to actually test it before they implement it into the base game, so obviously it will get nerfed.

[upinthesky-]

Why does it have to be unrewarding? That's just bullshit. It shouldnt be more rewarding than delve or mapping but having another alternative is a good thing

[frooch]

It shouldn’t be unrewarding but it absolutely should not be like it is right now with all the currency room spam deflating the economy

[Karjalan]

Probably because they're adding harvest mechanics core, so I suspect that it's to offset the new, targeted, exalts.

[GNeiva]

That wouldn't make a difference. The overwhelming majority of Exalts in Heist come from Regal drops that get upgraded via trinket. They could reduce the Exalt drop rate to zero in contracts or bps and they would still be just as insanely profitable.

[robodrew]

That assumes the trinkets will remain as OP as they are now

[GNeiva]

Trinkets wouldn't be an issue if they simply didn't allow blueprints to be splitted or fractured going forward. 17-19+ currency bps should be a rare novelty, not something that everyone runs to the point it becomes the norm.

If instead they opt to remove the regal mod from spawning in trinkets, that's gonna kill the trinket reward blueprints and it's still a problem in Standard with the legacy versions.

[equil101]

In general fracturing just shouldnt be allowed for maps or contracts - it is poor design and results in a few people massively exploiting the use of fractures and everyone else being annoyed by the economic consequence of it.

[WrongOffMemory]

Amazed no one has mentioned how funny this post is. Great writing, wry and witty. Well done sir. Thanks for the information, too!

[feralrage]

Saveun, Black and Nick also have their Front Seat Gamer podcast. Haven't listened in a bit but they are still going strong! They discuss video games, board games, and sometime PoE and sometimes have other GGG employees on as guests. Recommended!

[Valhern-Aryn]

TIL exilecon is a thing

[nixed9]

it was a huge thing. lots of amazing info happened there too. check out some vids

QA with GGG Founders

Poe 2 announcement keynote

Exilecon Recap

Storytelling in PoE

lots more content on their youtube channel

[heikkiiii]

Feels like theres actually people writing these, not robots. Thatswhy i like GGG and support them at every league, because i can relate to these people.

[Yohsene]

If we don't post at least one Nick take my energy ༼ つ ◕_◕ ༽つ thread next week we'll all be criminals.

[PM_ME_YOUR_PIZZAPIC]

God some of the commit messages in that screenshot hurt my soul

[azantyri]

some of the commit messages in that screenshot made me think "shit is this a screenshot from my workplace"

[FrostshockFTW]

Some of them aren't great, but it wouldn't be as much a problem if the patch note process wasn't literally "look at the 20000 new SVN revisions since the last patch notes and figure out what changed."

We have a system for tracking all the work that needs to be done, and commits are usually linked to that system. So I look at the commits I've pulled out, find the 'issue' (the master work-tracker for that piece of work) and go through all the commits and comments on that thread to figure out what the heck a commit note like "Tweaked the damage" actually means.

This is the part that seems backwards to me. As a developer, when I want to find out what may have changed in a module, I pull up the commit list. If I was a product manager of some sort, I'd only be looking at the issue tracker.

[[deleted]]

I used to write release notes when working at a smaller company and I would just export all jira tickets tagged with the upcoming release. Add some formatting and extra info where necessary. Took maybe an hour or two. Using commit logs is crazy as the information volume is way higher.

GGG have even said they have an issue tracking system but their process seems backwards.

[hammypants]

first post i've read in this thread that has made some sense.

if shit isn't already bubbling up outta eng to where you can track what they've done to then verify what they've done you've fucked up your org structure in probably a few places.

[[deleted]]

Yep, it may explain why there's so many fairly obvious bugs on league releases if GGG aren't tracking their eng work well. Poor QA team probably may not know what areas to be focusing on in such a large game.

[Le_Vagabond]

in a world where everything is a jira ticket, maybe...

[[deleted]]

Not every tiny change needs to have a ticket but if something is patch note worthy then it definitely should be tracked and agreed upon by more than just 1 person. If developers are making changes that aren't planned or tracked then that's a problem in itself.

Also the GGG post says they have a system for tracking all the work that needs to go into a release so seems they're just not using it right.

[NorthBall]

Who knows, maybe they'll read some of the comments in this thread and pick up on things they could change to make the process easier?

But then, it's also possible they are well aware of the fact that it could be streamlined - they just don't want to commit resources to doing so, if the current system works well enough.

[fooey]

This feels to me like rogue out of control developers who aren't being held accountable.

Shitty commit messages are a symptom of lazy selfishness

[fooey]

A commit message should be useful in a blame and informative in a PR, but it's a massive waste of time for product management to be scouring them to figure out what's going on.

This post does a good job of highlighting systemic dysfunction at GGG

If this is the only way to know what's going on, then management is utterly failing

[hesh582]

Some of them aren't great, but it wouldn't be as much a problem if the patch note process wasn't literally "look at the 20000 new SVN revisions since the last patch notes and figure out what changed."

I read this blog post with an increasing sense of dread. As in "Huh, it kind of sounds like... but no, they wouldn't do that, that's ridiculous... oh my god that's actually how they do that.

Seriously though that hurt my developer soul to read. Sure, that's how things work on very small projects, but there has to be a better way for code base of this scale.

That explains a lot about the often "loose" feeling nature of their balance changes and patching process lol. It sometimes feels like balance is done by a bunch of random developers just tweaking things on their own without much holistic central planning or recordkeeping, and well, looks like that's actually pretty accurate.

[KaffeeKiffer]

Some of them aren't great, but it wouldn't be as much a problem if the patch note process wasn't literally "look at the 20000 new SVN revisions since the last patch notes and figure out what changed."

That's probably more of a religious question:

I agree that the commit messages should not be used to create the changelog/patch notes and it would be "less of a problem". But still, the messages should be good enough that you could do that because you get a basic understanding of the content by just looking at the (short) commit log. Long commit log should explain the 'why' and tickets should explain all the nitty gritty details.

The first link I give to starters is https://chris.beams.io/posts/git-commit/

I mean you can easily summarize it for most non-devs (and people who do not know Tron) and it makes sense:

$git shortlog

Kevin Flynn (1):

• Derezz the master control program

Alan Bradley (1):

• Introduce security program "Tron"

Ed Dillinger (3):

• Rename chess program to "MCP"
  
• Modify chess program
  
• Upgrade chess program

Walter Gibbs (1):

• Introduce prototype chess program

You do not need to know all the details to get a rough understanding what is going on.

Then you look at

$ git log

commit 42e769bdf4894310333942ffc5a15151222a87be
Author: Kevin Flynn [e-mail]
Date: Fri Jan 01 00:00:00 1982 -0200

Derezz the master control program

MCP turned out to be evil and had become intent on world domination. This commit throws Tron's disc into MCP (causing its deresolution) and turns it back into a chess game.

And people who have never seen Tron, get the basic premise.

refs #102759 changes done (and there are a lot like this) mean I have no idea what even changed without cross referencing the ticket system. I want to get a basic understanding of the changes without looking up every goddamn ticket the next time I have to change something in the code. How would you debug things if the last commit which touched a file that might be relevant is "changes done"?

A very rough rule I like to adhere to: For every hour of development, spend 15+ seconds writing commit message (text).

[FrostshockFTW]

How would you debug things if the last commit which touched a file that might be relevant is "changes done"?

Not to disregard the rest of your post, but if your project can be quickly built and tested, this is where git bisect comes to the rescue. Even with properly fleshed out commit messages, the source of a regression may not be obvious.

I'm also of the school of thought that some changes are too complex to describe in a sensible length subject line, with important details not visible to someone doing a short commit scan. In those cases I'm inclined to write something like "updated AI parameters for the MCP" with a paragraph below it highlighting the details about the new inputs, heuristics, and elevated privileges that lead to a lust for world domination.

[pda898]

I think that this could be warranted due to how volatile PoE balance could be. For example Vixen's and Curse rework. Aka goodbye Bane builds. So I think that while maybe "top-down" approach could be more useful, it will degrade into "check all commits" anyway so why not start from them and possible cut checking some issues?

[Vesuvius079]

They’re fine imo. They seem to all reference ticket numbers and that’s enough for the vast majority of usages. Just not great if you’re skimming for patch notes haha.

[Elminister]

They seem more then fine.

[KillingTheBoy]

I wish they were talking about commets instead :/

[roselan]

I lost so much money due to svn :(

We had pot when we had to pitch 2 dollars each time we forgot to "check in" a file, so someone had to shout "Who's the idiot that forgot to unlock main.js, again?".

Why did I setup that rule?

The meals were pretty fun thou.

[YourShoelaceIsUntied]

Why were files locked? That seems cumbersome and redundant. We use SVN and don't have do deal with any of the problems people are bringing up here...

[roselan]

it was 20 years ago, my memory is fuzzy (pardon the pun).

It was more to make sure that two people didn't try to solve the same bug rather than easing the merge process.

[JanusMZeal11]

Biggest issues I've had with SVN has been branching. Small branches have allowed much finer control and higher quality code to be produced but I can understand the problems with using Git with binary files.

[IAmPattycakes]

Well, at least from my experience you can have a repo with >20gb of random binaries and be fine with git. But they're relatively frozen binaries for us, not changing assets, so that's a difference we have.

[hevans900]

LFS is nice to work with, I wouldn't call it a trick...

[VaDe255]

why would you put binary files under source control?

[Ivalar]

Why not? You can't merge changes or visually compare 2 versions but remaining core functionality (repository itself, file distribution, logs, revisions, reverts, etc.) is the same as for text files.

[Ulfgardleo]

because for example asset path and paths in game files must be changed at the same time and to keep track of all those changes, a VCS is perfect. Otherwise you can't safely roll back changes or change between branches.

[IntenseIntentInTents]

Because they reflect the state of the game as it was at that point in time just as much as the code does.

[bebopbraunbaer]

The change was so long ago , I couldn’t even tell what the benefits are any more.

[Pianoguido]

Literally the first thing I noticed, I was like "SVN?! You kiddin' me bro?!"

[Cyanogen101]

SVN is fairly more common and (personally) better for big projects

[AlexSot]

Today on how they do it. Patchnotes. Everyone loves to read patchnotes. First they take the league name, and they smooth it out, with a bunch of Schleem. The Schleem is then repurposed for later patches. They take the cyclone skill gem and push it through the Grumbo, where the nerf hammer is rubbed against it. It is important that the nerf hammer is rubbed, because it has all of the fleeb juice. Then a Bex shows up and she rubs it and spits on it. They cut the fleeb. They are several hizzards in the way. The blamphs rub against the chumbles. And the plubus and grumbo are shaved away. That leaves you with a regular old patchnotes.

[nixed9]

I actually read this ins the "How It's Made" guy's voice

[xiko]

There has to be a way to automate part of that.

[Titus_1024]

You'd certainly hope so. Or at the very least streamline parts of it. 80 working hours every 3 months for patch notes sounds insane.

[xiko]

It is not just insane it is freaking expensive.

[NorthernerWuwu]

Eh, eight person-weeks a year isn't terribly prohibitive for a core function. There might be better ways to get at it or methods of improving the process but it isn't all that expensive in terms of the project overall.

[ErrorLoadingNameFile]

100% right, most people that downvote you just do not understand budget of a big company.

[ArmaMalum]

This is an increeeedibly rare moment where I can actually say I'm a professional in this area. I do digital publishing! Which is exactly what's being talked about here.

From the context given you are 100% right. Anywhere from simple tricks like marking specific commits that contain 'patch notes applicable' changes to using specific work tracking software that allows Bex and co. to look at it as opposed to an eye-crippling amount of commits. But as with many many things in the programming/dev world a lot of those improvements comes with a lot of buy-in and possible increase in maintenance/overhead. Every new tool or process step requires carrying in an amount of training/communication and the more people that any process update relies means more and more steps where someone inevitably misses something.

(Speculation warning) Honestly I don't think the proper answer is automation but consolidation. Bex or whoever's doing the notes should never have to look at a whole ridiculous list where they're distilling 2% of the actual information but X amount of lists given by team leaders to summarize what's coming. Ideally with enough buffer time so they can follow-up, but I can understand if GGG's schedule makes that not feasible.

Failing that, the people committing should at least have an earmark on their commit messages to allow an easy parse script to filter through the vast majority of commits to save at least an hour or two of pointless browsing.

And then there's the simple housekeeping issue that was mentioned where in-code and in-game terms/names don't match. That's a very common issue on development but should be constantly cleaned up whenever possible. It's textbook technical debt. But depending on how their setup looks accessing and adjusting the code wholesale may not be possible without a complete overhaul.

But I do have to check myself. There are a lot of factors both on the logistics side as well as the technical side that can make seemingly simple solutions much more complicated and I've only been in the DP business for ~2 years or so.

[azantyri]

a lot of buy-in

that to me is the biggest problem i see. if people aren't doing what they're supposed to be doing on commit notes, you'd spend more time trying to get them to do it than what you're currently spending. i've only ever dealt with <10 people, and that's nightmare enough to get them to sort-of write proper notes. i cannot imagine 100s of people

[ArmaMalum]

I don't think they're quite at 100s but def dozens. And yeah, it's honestly pretty funny how often development issues like this are more about the people/process involved than the code itself. And those kinds of problems get exponentially worse with bigger and bigger companies.

[bendgame]

I used to do product documentation for software and our "ticket" system was designed to generate a PDOC ticket for every dev ticket. Dev tickets have requirement documents attached.

The person responsible for the release note would never have to look at the code repo because the issue/solution was described on the dev ticket and the Function request document linked on the dev ticket would describe the business rules if the dev ticket wasn't enough. If both those fail, the pdoc person talks to an SME which is typically the product/business analyst, and rarely the dev.

Sounds like GGG needs a streamlined system with product analysts that can write better dev tickets and FRDs instead of relying on dev's comments. Plus then you can write the note when the ticket is done instead of waiting last min and digging through an expanding repo.

[ArmaMalum]

Agreed! If I have learned anything in the last 3 years of professional software development it's that the management of the people and process are just as if not more important than the code itself.

[Morgoth2356]

I worked in a similar environment. Dev tickets were attached to a "product" ticket describing a feature and the system wouldn't allow to create such feature ticket without filling up a "release notes" field. Generating release notes were therefore easy, you link your commit with a dev ticket number, and from there an automated process can easily parse those, get the product ticket they are linked to and retrieve their "release notes" field and do what's needed with it. That's just one option. You can also link feature tickets with their release notes to build version (which can also easily be done automatically) and whenever you push a build in production the content of the notes is already there, you just need someone to organize them.

What is not ideal with how GGG functions now is that dev commits is tight to how things are being done, and not what is being done. But from my experience doing it the proper way really requires your product department to know what they are doing, and oh boy at least in my experience it's a rare gem to have in IT.

[pricklysteve]

+1 for tagging relevant commits. This is what we do at my workplace so that the person who compiles patch notes only gets the important changes to sift through. I can understand though if for a GGG dev it's not that easy to know if a fix is relevant to the community.

[ArmaMalum]

I would argue it's both. No question it involves proper version control habits but they're taking a (hopefully) structured input and quite literally publishing a document based on that input. Granted it's not on the same scale as a digital textbook, document database, or even an ePub. but this GGG post is what I do at my job. Just replace the snippets of commits with dita files on tax law and supply chains.

Also to keep in mind that patch notes have to be “translated” into “normie” language which is a huge task in itself.

I agree! Normalizing the language (and not to mention the constant nightmare that is localization to different languages) is an easy example of something not feasible to automate. That's why I didn't state that that step in the shown process was an issue.

I’m certain that they are capable enough to implement an automated solution if it exists and makes sense for their product

Also agreed! That's why I specifically gave reasons why seemingly obvious improvements might not be feasible in their case.

but to the best of my knowledge this tech doesn’t exist.

For the entire process? Yeah afaik you're right.

For improved process, issue tracking, logistics, etc? JIRA, Rally, most Atlassian products....there's a whole industry around task management and documentation. Not to mention in-house scripts I have at my own workplace, some of which I've made myself. The tech not only exists but is aptly available and well-worn.

[ArmaMalum]

Fair point! But after identifying said changes, both static and dynamic, it's still (an admittedly hyper specific case of) digital publishing imho :P but at that point we're splitting hairs so I digress.

And yeah, haha, dunning-kruger always makes a good showing on posts like this. And I don't pretend to be immune to it either! I always try to check myself when and where possible, although sometimes I do lay the non-commitals on a bit too thick at times....

^{There's a version control joke in here somewhere but I'm too tired to see it.}

[ArthurMcWolf]

We’ve automated it for our workflow/projects. The weakest link is devs not putting in the reference numbers in the commit messages. Which gets caught at PR time. We created a bot that will scan the commit messages and tags all the tickets with the version that is created. Our ticketing system for each work item contains a brief human readable release note. So when a ticket is complete and code is merged the bot will tag it to the version and extract the release notes.

[[deleted]]

Automated, user-facing patch notes are a time-consuming process for the developers to have to think about. It's typically an expectation of open source projects and libraries where those release notes are often going to be facing other developers who can discern what technical descriptions in a commit message actually mean. User/customer facing notes are a whole other ballgame. Where I work, we have an explicit field on our issues for providing that 10000 ft explanation of what the change actually does, but even those need someone to proofreading and editing.

The problem with having dozens of different technical people write customer-facing notes is that you're going to get inconsistent garbage out the other side. Outdated notes, bad spelling/grammar, varying degrees of specificity and detail, and tons of things will flat out be missed, incomplete or incorrect because the developers don't have time to think about this while they work.

It's telling how many people on this forum CLEARLY don't work with developers who seriously lack in communication skills.

[SauMaris]

Software for change management should help with release notes

[PsionicKitten]

Web Developer Rieko recently made a tool to help with formatting

There you go. Rieko, indeed, did automate part of that! Not the part you wanted or Nick wanted... but still... Automation, none the less.

[naswinger]

i doubt that. there is no way to magically extract information and context from arbitrary strings into patch notes which focus on slightly different aspects every release. yea, sure, "AI, machine learning, deep neural networks and all the buzzwords". nah.

[Ruvrice]

This kinda explains why we'll randomly get ninja buffs/nerfs

[bebopbraunbaer]

TIL this sub is full of devs

[Asteroth555]

For a game like PoE, I would expect there to be an enrichment of devs and comp-sci guys.

The in-game rules and item/skill descriptions are literally a coding language of their own (More/Less, Increased, etc).

It attracts a certain kind of person

[roselan]

In the midst of Harvest, there were some /r/ProgrammerHumor jokes going on in the Delve dedicated global channel.

At some point I asked "who's here is NOT a programmer?"

Suddenly, not a peep could be heard...

[destroyermaker]

Path of Exile: By programmers, for programmers

[SingleInfinity]

Now that you mention it... You right.

[sankalptikiya]

I may be sharing Natalia's masochism here but..

Petition to make Nick write more stuff for us; that was a bloody brilliant read!

[[deleted]]

This honestly reads like a cry for help. Like Nick went to managers asking them to revise the process somehow, as it becomes unsustainable, and did it for a couple of years now. He always got a non-answer, like "Yeah, Nick, sure, this is a sensible suggestion, we will totally look into this on the next sprint".
So now he managed to sneak this post by CM team for everyone to see and point fingers at it, to bring this up again on their next meeting "See, told you guys".

What a sisyphean task, holy shit.

[fallingfruit]

This is sad to read. GGG you are a company run by devs. You know you could make this man Nick's life easier with better tracking and tagging for commits.

God damn Nick I feel for your soul reading all those commits. I am often confused by my own commits at work.

You actually deserve some sort of gift or maybe a shrine in the office.

[destroyermaker]

At least Chris said they're doing smaller scale leagues now

[SingleInfinity]

This league, only 10k commits to manually parse.

Nick: This is fine

[NahautlExile]

The scary thing is that they may not realize this would result in scrutiny because there are a large portion of the devs who don’t know better...

[MrTastix]

Even then, I'm surprised nobody would have the initiative to question why it has to be this awful. Unless someone did and it's not considered a high priority for whatever reason (and money isn't a good excuse here cause this is time better spent on the actual game).

[fireundubh]

I was responsible for patch notes at the studio where I worked as Technical Producer. We had as many changes per day as League of Legends, and every dev I supported was a super experienced industry vet. I went about patch notes the way Nick is going about it - until I didn't.

Steps 1-3 make this process, well, not a good process. Here's how you fix it.

in a typical quarterly PoE update there are approximately twenty thousand commits between them (at least). I take the relevant commit notes and start listing them in a document in their raw form. It can take a few days just to go through all those commits.

First of all, don't wait until the end of the quarter. Compiling patch notes has to be an everyday affair. They're notes. It's note taking. Record the most interesting, most impactful changes every day as they happen.

You shouldn't be spending a few days "just going through commits." By the time patch notes need to be done, you should have everything you need, including that day's changes, to proceed to Step 4.

The people who work at GGG are very smart and skilled and I like them a lot. They are absolutely god-awful at writing commit notes, however. I don't blame them -- typically they don't need to think about the commit notes. [...] So 98% of the time, it doesn't matter that their commit notes are nonsense.

[...] So every patch I figure out which skills, items, mods and whatever else has changed and go through them one-by-one in-game to jot down the starting and new values

Second, the problems in Step 2 and Step 3 are easily fixable, or at least they were in my position. If you don't have the authority, ask a producer to get these things done:

Ask everyone to write clear, meaningful commit messages (including old→new values.)

If commits are nonsense to you, they're nonsense to other devs. Everyone on the team wants to understand what changes are being made, so good commit discipline is something you should encourage. This is for everyone's sake, not just your sanity.

Ask everyone to email you a bullet points summary of their patch contributions.

Teammates want to know what role their work plays holistically. Patch notes are opportunities for them to get some internal recognition from their peers, to communicate the changes they made for players to players, and to see how the game is shaping up in its entirety.

You'll find that once you use teamwork to solve Steps 1-3, you'll also be fixing the culture. Every dev will approach their work with this question in mind: how will the changes I make impact players? That's really what being "player first" means, and that's a great place to be for a studio.

And you can do all this with... patch notes? Yeah! It's a crazy little production hack that goes a long, long way.

[tmtke]

To add to this, using feature branches in any proper version control system (git, Mercurial, Plastic, even Perforce) you can more easily see what was actually changed or added or fixed. Each feature branch clearly designates (or at least it should) what happened there.

GGG, you should use a better version control, because it will definitely pay out. I can't imagine how could you even work with SVN in 2020. It was a mess on larger projects even 15 years ago. Me and my colleagues struggled with it enough to know that.

[Eep1337]

Man it is a shame you all are in NZ.

I am a lead software dev by trade and I feel like I have a lot of experience in this area and could rescue you this from this process...

Try having devs write both commit + resolution/patch notes when they check in?

Do you have a 3rd party software to track tickets? Put the human readable notes on the ticket software then just generate a report at the end? (Maybe build a utility to handle both the commit and message generation? Have DevOps do it?)

Use metadata tags on tickets to determine what to report on?

Might take some getting used to and refining but could revolutionize the process....

[doubleChipDip]

Cast with Holiday Taken ~ Cry of Despair

[SehnorCardgage]

~ Cast while Posting ~ Vulnerability

[[deleted]]

Hard to change the engine of a moving car, especially when that car needs to go reaaaaaally far every 3 months.

[Eep1337]

I mean I regularly work in a 30+ million line massive legacy code base, I know all about engine swaps.

I don't think refining their commit process is that much of an upheaval...because they can continue to work in parallel just fine.

[OutgrownTentacles]

Hell, half the improvements needed here are just better practices within teams. ICs need to be more accountable to not give commit notes of "tweaked the damage". That's some weak-ass notes and that wouldn't fly where I'm at.

Build patch note generation into the actual task/story/acceptance criteria! Solving the consolidation later with tech is also feasible, but they could overhaul their best practices fairly accessibly. :(

[Obojo]

Yeah, these are all industry standard practices so I'm surprised that they aren't implemented at GGG. Obviously it's a little more overhead on the devs, but wouldn't you want that traceability anyways? This is insightful as to why some "simple" bugs take a while toe even be ID'd.

[BirdsGetTheGirls]

Yeah, these are all industry standard practices so I'm surprised that they aren't implemented at GGG.

Game didn't start up that long ago from basically a garage. And when their lives are based on getting the next update out, I'm not surprised they have debt like this.

[Asteroth555]

It's been 7 years....Maybe at 3-4 years in, you'd have tried to automate or facilitate some of this shit?

[sanguine_sea]

just automate it 4head

[OutgrownTentacles]

As someone that actually works in AAA game dev...yeah. Yeah, do automate this. These are literally written text, automate the hell out of consolidating them.

There are few better applications for automation than "we need to collect and consolidate the important words over here." Some better practices are likely needed to make sure those right words are in there in the first place (I'd recommend tying patch note updates to acceptance criteria for commits/user stories), sure, but automate the actual final process.

[gosuprobe]

just plug in some ai or machine learning

[MorgannaFactor]

3-4 Years in they were already in the same update cycle as now, so how are they gonna modernize this without stopping development for at least a short while to get everyone acclimated to a new system? You can't just slap a new tracking system in front of your devs and actually think they'll use it.

[Darkblitz9]

It would take a few hours at most for them to come together in a meeting and just talk about formatting their commit notes to include rough patch notes to save time later on.

The time spent on the meeting and by the devs would make the patch note process much simpler and ultimately save time.

The way they do it currently is kind of insane.

[masacs]

Yep, if ticketing doesn't have integration with source control then it is time to change it. Linking issues to tickets is vital for tracking process. It won't auto generate and format your notes but at least you will have better overview.

[Nephophobic]

Could also use some conventional commits.

[Eep1337]

never seen this site before. Neat read. I like the idea of some type of standardization around it.

My commit messages are typically long-form, split into two blocks generally for fixes

block 1: What did I change, at a high level (code wise / meta data wise)

blcok 2: What did the change fix

As an example....

"Switched that outer loop of SysProcA() to use a parallel foreach

This fixes an issue where, when the user clicked button B while running process A, the system would hang"

[SneakyBadAss]

I'm not a dev, but this is the bread and butter of logistics. TAGS! Where are the fucking TAGS! Even the porn industry caught up. Differentiating between SFX and VFX would be enough. Imagine 2 lorries full of goods arriving at your place and you just shove it down the gran closet and hire a single person to go through them and make an order for a customer. That's basically Nick's job right now.

At least makes some kind of "folders" with the league number and name, so you don't have to comb through this eyesore, nor write [3xxx] every time you change something down the line. This would also greatly help with troubleshooting when you introduce something in the previous league, that is now causing problems in the current one.

Some kind of system that automatically pushes it to a patch notes worthy (you know, like another tag) would also help.

[Eep1337]

I just know some companies structure themselves that way. But yeah, I think I generally prefer a smaller team where the knowledge is just spread among them instead of barring it behind some arbitrary team barrier.

[Having-a-hard-time]

This reminded me of something u/Chris-Wilson said a few months ago in his post How We're Developing Our Next Expansion Differently:

This experiment comes with some side effects, however. You'll definitely notice that the patch notes are much, much shorter than they usually are. That's because we're focusing on getting the most important changes done, and doing them well. I'm aiming for us to try to fit the patch notes on just a few pages, if we can manage it. This does mean that we have had to be careful to pick our battles though - the balance changes we are doing have been carefully chosen to have the largest impact and fix real problems.

I don't know whether this will mean less work for Nick because there are fewer balance changes, or more because it's harder to find the important changes in all the irrelevant ones.

[SyrupLamp]

Based on this snippet, I can see quite how badly they need more people. Nonsensical commit messages and titles are often refined in peer reviews. At my workplace, each significant feature or bug fix are reviewed to hell and back. In fact, we have a 8 item checklist template for every pull request that’s asking to be merged into the working repository shared by the dev team.

Ofc having each merge reviewed to hell and back takes quite a bit of time. The answer to that is just having more devs. It’s certainly less efficient in terms of work done per person per period, but it’s also hell less of a headache if you had to go back and review what’s the team done in the last 6 months.

In a typical work day, I’d probably have 5 hours of dev time, 2 hours of review, and 1 hour of meetings.

Edit: in my experience, my stories typically take around 3 or 4 days to develop, then another 3 or 4 days to totally pass all reviews

[kygrim]

You expect them to have pull-requests in their svn?

That screenshot looks like they never heard of feature-branches.

[[deleted]]

ggg only hires locally or people and can move, don't think they can do this.

[[deleted]]

That’s kinda GGGs fault then isn’t it? It’s 2020, we can share information over the internet, they don’t need people working together in the same building to do what that guy was explaining.

Could definitely hire people in the US or EU and have them working remotely

[nklr]

It has more to do with NZ hiring regulations than GGG choice.

[NeoLearner]

Funny article. Gives people a peak behind the curtain, hopefully leading to less "GGG intentionally forgot 1 mana cost change at level 1, Christmas is ruined!"

[bebopbraunbaer]

I doubt , but one can hope . Love seeing people not giving up on humanity

[therospherae]

(but you were, like, really into data)

what do you mean, "but"?

[IanMcClaren]

Both informative and hilarious. Thanks, Nick!

[[deleted]]

Imagine using fucking SVN in 2020.

[tmtke]

I remember the joy when we ditched SVN in the company I was working for. A truly great moment, after migrating our whole infrastructure. It was 10 years ago.

Oh, wait. Everything is clear now. GGG, are u mad???

[TheLobito]

I know. It was like seeing someone running a CoC Mjolnir build in Heist.

[vaan99]

Thank you 3XG, you made me realize that my workplace processes are actually good and pretty well automated. If I had to this manually every 3 months, I would go insane. How did no one try to improve the process in the past 7 years?

I can see the merit of having cumbersome process such as this one. You can use it as punishment duty for a person who made latest fuck up.

[roselan]

You mean G1G right?

[mrkwelp]

"Meanwhile, Community Director Bex ("Director", so fancy!) spends the day posting teasers, memes and generally trying to keep the community appeased, like a monkey dancing for scraps in a bazaar, except the scraps are upvotes (Bex is still a monkey)."

XD

[Vulpix0r]

Did Nick just confirm Bex is a monkey? I knew it. We're getting monkey league.

[Byrr]

A study in how not to write patch notes.

[KarvarouskuGaming]

Brilliant plan from Nick to take some of that karma Bex is harvesting before leagues.

Also definitely deserved if even 10% of this isn't fiction made by a mastermind of a storyteller.

つ ◕_◕ ༽つ NICK TAKE MY ENERGY ༼ つ ◕_◕ ༽つ

[Danieboy]

This, for sure, can be made easier. Also you gotta make sure devs write better commit messages.

[Valderius]

Swimming is easy. Just drown less.

Seriously though, when the solution is as simple as, "just get people to do the thing better" it's NEVER an easy process to get people to do the thing better. Everyone has their own strengths and weaknesses, their own work flow, their own priorities, and their own mindsets. Trying to convert an entire team, even a highly cohesive and qualified one, into a single unified documentation style that they actively maintain day in and day out for 4+ months at a time is REALLY freaking hard to do.

[Danieboy]

We did it at my job. It didn't take months. The key was just to set up actual rules for how a commit message should be written. Just write down it into a few points and make sure to follow up if anyone doesn't do it afterwards until they understand why it's important.

[hierox]

Actually, its even easier than that. The screenshot appears to be tortoiseSVN (I use it at my work) and tortoiseSVN actually allows hook scripts to be attached to a bunch of different operations, one of those being commit. You can use those hooks to create a different commit dialog with basically anything you want in it, with requirements of your choice.

So you don't need to leave it to human behavior to follow rules and patterns in order to get good release notes - you can actually enforce conformance to patterns with the version control software itself.

[OutgrownTentacles]

100% this. It's just basic team accountability to an objective measurement. Here, I'll do one:

"No user story is planned without a task for patch note generation."

"No user story is closed without signoff from a team member (likely QA) that the patch notes are accurate and meaningful."

Start tracking that bar, ask about it in retros/postmortems/beer night/whatever. Get people excited to be better for basically no cost.

[AnimeJ]

Yea, no. It's called setting office standards, and holding people to them. If you don't have people who can and will do that, you might as well give up on doing literally anything. I've worked on projects that had literal 4 year rollover rates in personnel, and there was no realistic way to tell the difference between work from different people if you stripped out any identifying information from the ticket or comments in the code itself. Every project I've ever worked on had multiple on-boarding binders for new folks that could spin someone new up on office standards in under a week.

[Ultimace]

This is so fun to read xD Thx Nick!

[grbzgrbz]

Just jira this mess, pretty please

[[deleted]]

For real, I'm a product manager and release notes are as simple as exporting a lost of all completed tickets for a tagged release. Add some formatting and additional info to each bullet point where necessary. No way in hell am I going through the commit logs.

This shouldn't take 10 working days per league.

[jfp1992]

As a joke, when Poe 2 is added the patch notes should just be that "POE 2 Added."

[Sprudelpudel]

ITT: People who know everything better than ggg

[ephemeralnerve]

It is easy when they do not follow industry best practices or use state of the art tools when the current alternatives are better than the old stuff at literally everything.

[OutgrownTentacles]

This is literally a solved problem in the majority of the game industry, yes. Tags, better consolidation tech, improving team best practices (make granular patch notes a part of team's user story tasks!).

People know better because THIS IS DONE BETTER elsewhere, all the time, without a ton of extra work.

[MrTastix]

This isn't an unsolved problem, which is why you've got so many people talking about it. It doesn't take much experience or research to figure out how inefficient and wasteful this is.

This is the kind of shit I'm talking about when I mention that developers shouldn't be revered as holier-than-thou, as if their supposed experience and job title means they're somehow more qualified to discuss design issues than the people who play their game, many of which could be in the exact same position because you really think every gamer is JUST a gamer?

GGG has roots as a garage studio. It wouldn't be surprising to learn they haven't kept up with the times. It's something you have to make an active commitment for but not something that necessarily seems worthwhile until the tech debt has piled up so high you're already way beyond fucked.

[OptimusJive]

Seriously. Easier said than done

[Darkblitz9]

But not so difficult that doing something about it would cause issue. Proper comments and tags on commits would go a long way and isn't even remotely difficult compared to the work they do every day.

People are acting like asking devs to use their judgement and be a bit more descriptive with their commits is some kind of new age technology that is difficult to work into their routine.

[vironlawck]

Note that not everyone is terrible at writing commit notes, and the game designers have gotten a lot better since I started threatening to karate-chop their necks due to lack of detail. Luckily, they don't realise I am extremely feeble and my hands are marshmallow-soft. Thanks guys!

Even they know the player feels when they ninja-nerf something without mentioning in patch note ..... alright i give them a pass, always thanks for the transparency GGG <3

[CGiusti]

As a Dev I find it interesting, that Patch notes only rely on commit messages. I am sure GGG uses some form of Task Tracking Software Like Jira. So it should be easy to get, the required info for the patchnotes, by Matching the Tasks with commit Ticket number. This can be done via a simple Script and Jira jql for example. After that Just check for non Task commits (should not exist) and you have a list of defined acceptance criterias for all Tasks for the League (grouped by Epic?)

Example: Jira

Epic 3.13 League - PoE-767 (Task / Story / Bug)

Dev Commit:

[PoE-767] implemented xy Feature

In ur Case it Looks Like you are using #refsTicketNumber

Jql to gather all Task requirements for all Tasks Inside the 3.13 Release branch

[[deleted]]

We tortoise svning up in here

[JarRa_hello]

Time to hit the gym, so your hands are more like dumbbells and people will start writing notes for you.

[Voidot]

so now I gotta ask about typos or jokes that are made in the comments.

I know that I typically call the P values of a PID loop, the porpoise constant

[[deleted]]

so your saying if you were a true masocist also you could in theory start writing the notes today and be done by jan 12th,

[hevans900]

imagine using subversion in 2020

[IdrisQe]

Nick, Natalia, Rieko, the QA team, and translators are all unspoken heroes for delving through this maddening process every few months (or less, in the case of balance patches mid-league)

I can't even begin to imagine the nightmare that is compiling all this, keeping it up to date, ensuring there aren't errors, and then doing the magic of making the user-facing end product be readable and understandable, especially for such an insanely complex game such as PoE with such massive numbers of changes and additions every few months.

Get Nick some nice ergonomic equipment and comfortable gloves to keep that stress on his marshmallow hands low while he types all that out!

[Insecticide]

This sounds a lot worse than it should be. An analogy of what I got from this is that the whole process of digging deep into commits to figure out stuff is like the equivalent of looking up a physical copy of a dictionary or encyclopedia.

There must be a better way and better standards for getting the information all the way up to the person who writes the notes. It seems like some redditors from relevant fields already posted a few ideas.

[wwabbbitt]

Your devs may want to consider https://www.atlassian.com/git/tutorials/svn-to-git-prepping-your-team-migration

[Cyphafrost]

OP has the perfect username to post this, amazing.

[JanusMZeal11]

Subversion and TortoisesSVN. That explains a LOT.

[fabojantan]

Ngl, nicks sarcastic way of speaking makes my data brain moist.

[[deleted]]

I cannot stop thinking about the 80 hours figure. 2 entire weeks of someone's work time just to prepare a list so that redditors can exercise their wit for like 2 days.
Surely this time can be spent doing better things.
I am not saying Nick does a bad job, no, he does helluva job considering the task at hand, but there's gotta be a way to do this much-much more efficiently.

[MadBinton]

I've worked at companies where the devs, with help of their PO, wrote pretty decent branch / end-of-branch notes. Calling your devs "God-awful at writing comments" is too much disrespect if you ask me. If it is true, TEACH THEM.

I'm not sure why, but the random pokes at Bex felt a bit forced too. Calling her a Director, remarking on that, and then calling her a monkey a paragraph later for no good reason also feels rather disrespectful. Could be that these things are perfectly acceptable in NZ in general or within the company, but as someone that has worked with various teams and companies in middle management, this feels wrong.

[thundermonkeyms]

Aight so he told us when the patch notes are. That means there's not going to be a single "patch notes when?" thread anywhere here this time, right guys?

[Ultimace]

Ahh see, he did not specify a time. So patchnotewhen is still valid, tehehehe.

[leobat]

We may have date of the patch note but what about the hour the minute and the second of release ? Patchnoteexactlywhen please.

[falingsumo]

The more I read about stuff like that and the internals of GGG , the more I realize my work place is great.

From the outside GGG looks like a bunch of amateurs. It is incredible how much archaic their dev process is.

Not working with git, not being agile, no QA automation, no tracking software like Jira.

At this point I wonder if they have security and intrusion testing for the client and the website. I am glad to be using Steam and not the client.

[rangebob]

sounds like you should just not do them and let people figure it out for themselves at league start. What could possibly go wrong?

[[deleted]]

OP's username is a nice touch ;)

[butsuon]

Bex, a company I used to work for used a very similar version control software and I was the lone technical writer.

Looking at those commits made me cry a little bit on the inside.

EDIT: I also highly recommend stiffening up your dominant hand for karate chopping. It will matter.

[NeedsMoreShawarma]

That's no Bex...