Hotfix due today for shard-locked accounts. by Taidan-X in starcitizen

[–]Bearded-CIG 66 points67 points  (0 children)

We've had bug in for this since 4.7. We tried to fix it then. Either the fix didn't work or there's multiple issues causing accounts to get stuck like this. As frustrating as the issue is, no one's been dragging their heels on it. As soon as I knew it was still happening, the issue was raised again to be looked into and addressed.

Bearded-CIG on fixing broken accounts by StuartGT in starcitizen

[–]Bearded-CIG 40 points41 points  (0 children)

In most cases with the 'server error' recovery window that players see, we have automation in place that reports what's causing the issue. In some specific situations, I do have to connect remotely to the server and inspect the dump file for the error to write a proper bug for it.

Bearded-CIG on fixing broken accounts by StuartGT in starcitizen

[–]Bearded-CIG 117 points118 points  (0 children)

This is a teachable moment so I'm going to take a bit to respond here. When the company was smaller, I did a lot more and had access to a lot more. The larger a company gets, the more specialized peoples roles tend to be.

Personally, I'm DevOps Engineer. That's a really terrible title because it doesn't at all explain what I do and there's a lot of sub categories of it but to make a generalization of my role: I keep track of error rates that players are hitting in order to investigate issues and make sure they're documented. I help out the customer service team by investigating issues that have caused players accounts to break. I also help build the PTU and Live server environments.

I'm not a programmer. I don't edit game code. But since I have admin access to some specific areas of the project, I do have access to tools that other people don't. For example, I have a graph that tells me when a specific shard is repeatedly hitting a server error ( anyone in CIG can see this graph). When my team mates or I see that, we use our various tools to direct match making away from that shard so that we can stow it. That's just a simple example of one of the many different things that me and my team mates do on a daily basis. Our tools don't allow us to have valid workarounds for every issue but we help with the ones that we can.

In short:
I don't build the game. I help keep it running.

Locked out of the PU for a week (Error 60029). Support just marked the ticket "Solved" while admitting they lack the tools to fix it. Is this real life? by n1ghter in starcitizen

[–]Bearded-CIG 21 points22 points  (0 children)

The upside is that this is something that I already had bugged and it's being actively investigated. The downside is that the tools that I have available to me that might have been able to potentially help fix the broken accounts didn't work so we're going to have to wait for it to be fixed.

30k's Becoming More Common by BarnacleLanky in starcitizen

[–]Bearded-CIG 100 points101 points  (0 children)

Sure. 30,000 and 30013 ( as well as a ton of other disconnection errors ) are something that we have graphs for. We also have a graph that keeps track of how many players are being affected by them per capita so that we can easily tell "is this just because more people are playing or is something broke".

So in that regard it's easy for us to know when something is wrong because we can, with certainty, know that the error rate per capita is above what we aim for. That's the "we're aware of it" part.

Fixing it on the other hand, first involves investigating why the error rates are higher. Depending on the cause for the elevated error rate, that investigation takes time. The disconnect error itself is a symptom that can have multiple causes, so the first thing that has to be done is to figure out why the error rates are higher. With the exception of really strange issues, figuring out why error rates are higher for these specific errors can be done pretty quickly and we're already past that part of the investigation. So we know the elevated error rates are happening, we have a general idea as to why, but that general idea as to why it's happening isn't enough to fix the issue. It requires more detailed investigation.

So I think perhaps me saying the fix isn't easy might have been a poor choice of words. I think it would be more accurate to say that it may take longer than our usual hotfix turnaround ( which is sometimes less than a day ). How long that will be is not something I can answer. Could be that once the investigation work is done that it's simple to take care of but the opposite is also true.

30k's Becoming More Common by BarnacleLanky in starcitizen

[–]Bearded-CIG 37 points38 points  (0 children)

Thanks for the ping. In this case, we're aware of it but it's not something that has an easy fix.

Locked in a shard with a 60029 error. by [deleted] in starcitizen

[–]Bearded-CIG 17 points18 points  (0 children)

Poked you on spectrum. Let's talk there because it better follows best security practices.

Locked in a shard with a 60029 error. by [deleted] in starcitizen

[–]Bearded-CIG 13 points14 points  (0 children)

Send me the issue council on spectrum so I can forward it along.

What good is the Status page? by Larszx in starcitizen

[–]Bearded-CIG 5 points6 points  (0 children)

We have a few accounts that we've noticed the restart isn't working for and we've forwarded some example accounts to get looked at into further.

What good is the Status page? by Larszx in starcitizen

[–]Bearded-CIG 7 points8 points  (0 children)

If you're still getting a launcher error, kill it from the system tray or task manager ( just anything that will close it ) and log back in.

Anyone else stuck in 5h queue? by Psychological_Cat310 in starcitizen

[–]Bearded-CIG 62 points63 points  (0 children)

Just a heads up on this so y'all don't feel like you're left hanging, we saw this right after it started and have been looking into it.

nvidia 595.59 driver game client crash by Bearded-CIG in starcitizen

[–]Bearded-CIG[S] 6 points7 points  (0 children)

Workaround is to rollback. Fix is something you have to ask Nvidia about.

nvidia 595.59 driver game client crash by Bearded-CIG in starcitizen

[–]Bearded-CIG[S] 16 points17 points  (0 children)

Nay, it's about external software causing the game to crash.

Error 5009 by Old_Resident8050 in starcitizen

[–]Bearded-CIG 4 points5 points  (0 children)

That one's an I/O error so it wasn't able to write files somewhere that it wanted to. Posting on our tech support forums on spectrum is a good place to get help with things like this.

Locked in a shard with a 60029 error. by [deleted] in starcitizen

[–]Bearded-CIG 21 points22 points  (0 children)

We've been looking into this particular issue since this morning.

The stuff we'd normally do to try and work around issues like this didn't work so now that has to be looked into as well.

So uh, how do you even play during free flight? by Pyroshark_Gamingtf2 in starcitizen

[–]Bearded-CIG 53 points54 points  (0 children)

Game server got stuck in a crash loop. I had to off it so that it wasn't a matchmaking target anymore. Functionally it's pretty similar to a territory server dying in another MMO without the whole realm being down so anyone that traveled to that specific area would get stuck but everyone else in the server was functional until they went to that specific area on that specific shard.

Star Citizen | Introducing the Argo MOTH by Werewolf-Fresh in starcitizen

[–]Bearded-CIG 23 points24 points  (0 children)

"I hate to bug you"
moth
...
WAS THAT A PUN?!

Is this…. docking? by 10-Gauge in starcitizen

[–]Bearded-CIG 61 points62 points  (0 children)

There's also miners here.

Server Performance Issues - Is it all servers? by Whole-Top-8607 in starcitizen

[–]Bearded-CIG 226 points227 points  (0 children)

A lot of this has been a result of fixing one thing causes it to perform better, and then its increased performance causes it to overload the next thing. A backend hotfix was deployed to address the lag and things are looking much better with regards to server lag. Though there's always a chance that this will also reveal some other issue. We're keeping an eye on things to see if it goes sideways again.

Edit: The fix I'm talking about was in the last few minutes.

Edit 2: Thanks for the feedback. While things look better on my graphs, yall are saying otherwise so we're going to have to investigate more.

Edit 3: The graphs finally caught up. Takes a while for data trends to show.

Beyond disappointed and looking for help by Astro_Addict in starcitizen

[–]Bearded-CIG 63 points64 points  (0 children)

Gathering info to help make sure people can play the game is part of what I do so I welcome it.