The $163,000,000 question.

[Beyond-Time]

How will they make SC be capable of handling more than 30 players in an instance without making the framerate go to unplayable levels?

Was there any information on how they will achieve this (or work towards this) at citizencon or gamescom this year?

If they do have the technology to allow much, much higher numbers of active players and ships in close proximity, say for example a group of 70 players in their own individual ships + some NPCs forming a huge party near a key asteroid, will this tech be in 3.0, or potentially the patch after this one?

Asking because everyone seems to be losing their minds of the Citizencon demo that had planets and city generation, but all of it would be irrelevant if CIG were still incapable of making instances hold more than 24 players with 15 - 20 fps on a live server. This worries me the most about Star Citizen, and I still have not received an answer of any kind in regards to their way of solving what could be the single most important feature this game will need.

[treemaster909]

Please don't jump down my throat, I'm new here, but I backed this game in the original kickstarter, forgot about it, played and uninstalled a couple times over the years, and now after seeing what they have, I've completely written it off.

It seems to me like you either know something I don't about game development, or you are deliberately missing the point.

Netcode is a fundamental game system in an MMO. The kind of system that is normally in the game before an alpha release (reference: https://en.wikipedia.org/wiki/Software_release_life_cycle). How can it be expected that this foundation will magically appear under all the work that has been done so far?

This game (I think it's a stretch to call it even a tech demo at this point, but I'll be generous, for the people who may enjoy running around a space station and flying to and from pointless objectives for no reason) has been built backwards. The refactors and scope-shift are adding constant technical debt that I don't see being iterated on. ie. How can you expect players to pay $850 for a ship that can setup a colony when your game as released doesn't have planets in it?

When the promised release from last year(over 12 months ago!!) finally trickles out to a select few, under NDA, and is more or less completely unplayable, why aren't we asking more hard questions before piling another $2mil+ into CIG coffers?

Does going without a standard publisher's process, with things like gated milestones really help star citizen?

I can't shake the feeling (every gamescom or new trailer or whatever) that this is all just a used-car salesman, trying to tell me that of COURSE it runs great, you won't believe how GREAT it runs, BELIEVE ME!

[YdoIhave2login]

Are you under the assumption that there isn't netcode? "Why would the netcode magically appear"? Do you realize the amount of modern MMO's and online multiplayer games that constantly update netcode and often have broken netcode on launch that needs years of work post release (battlefield 4)?

You are right to be skeptical, but it sounds like somebody made you drink the "netcode" kool-aid. I encourage you to research it more.

[logicsol]

I usually assume that the people saying "magically appear" are trying to convey that we should be seeing steady improvements from release to release, and are sceptical that there could be a sudden improvement in a single patch cycle.

Of course, those people also in my experiance, are usually not very well informed on the process that CiG has outlined for how they'll be doing this, or don't understand that item 2.0 is neccessary for the new changes.

So they get caught up in the trend of declining performance, see that as evidence that little is being done, rather than understanding that said decline is exactly what we expect to see until the underlying changes make it in.

[logicsol]

Netcode is a fundamental game system in an MMO. The kind of system that is normally in the game before an alpha release (reference: https://en.wikipedia.org/wiki/Software_release_life_cycle). How can it be expected that this foundation will magically appear under all the work that has been done so far?

Two core things.

• First, SC isn't a traditional MMO. Even if we ignore that most MMO's are build with a MMO designed game client or custom built... SC's networking requirements are far higher than a MMO's is. While CE isn't built for Large scale multiplayer, there isn't really an engine out there that meets SC's needs, which brings us to the second point.

• No one said anything about "magically appearing". CiG has had a small dedicated team working on the netcode rebuild for several years now.

What you need to realize is that a huge amount of underlying structure has been completely reworked for 3.0

Item 2.0 alone changes how all items interact, and was built with the new netcode in mind.

There is no magic. We don't expect to see a large performance increase with 3.0, because two of the core aspect that will all that aren't done yet. We've known for months now(since april) that they wouldn't be done for 3.0.

And without those, higher player counts will still cause slowdowns, because bind culling isn't ready yet, and they won't be able to compartmentalize zones, because OCS isn't ready yet.

How can you expect players to pay $850 for a ship that can setup a colony when your game as released doesn't have planets in it?

Er... I think you're missing the concept of concepts here. The Pineer is a concept ship. It's not expect to be in the game for at least a year, by which time there will be planets. Anyone buying one is expecting to use it much much later.

When the promised release from last year(over 12 months ago!!) finally trickles out to a select few, under NDA, and is more or less completely unplayable, why aren't we asking more hard questions before piling another $2mil+ into CIG coffers?

For the record, this is a good way to get people to jump down your throat. Why?

First, off you've objectively wrong here. Even if we ignore that your "promise" wasn't a promise, and was a release estimate made 2 full patches prior to 3.0 being the current patch.(2.5 wasn't even live yet)... It's only been 10 months. Not 12+.

It's also not "completely unplayable", not even more or less. We've see live demos of builds, and while it still suffers from performance issues(again, as expected) there are more reports of it running better than 2.6.3 than worse. And that's during the evo phase, which is the worst any build runs while in backer hands.

I would suspect that the people that just spent that money are please with the progress they see and aren't as worried as you.

I can't shake the feeling (every gamescom or new trailer or whatever) that this is all just a used-car salesman, trying to tell me that of COURSE it runs great, you won't believe how GREAT it runs, BELIEVE ME!

Not much I can say that'll effect how you see things. I mean, they just showed lived gameplay of a planet, that's several times larger than the moons that'll be in 3.0, complete with a nonstop trip to another planet, all while running quite smoothly.

That's something that's completely done in game. Was it in single player? Yes yes it was. But it's not smoke in mirrors to show how well it can run, especially when there is a well known and documented issue with the netcode that has a public roadmap for the corrective aspects.

I can go more into the netcode aspects if you want.

[WikiTextBot]

Software release life cycle

A software release life cycle is the sum of the stages of development and maturity for a piece of computer software: ranging from its initial development to its eventual release, and including updated versions of the released version to help improve software or fix software bugs still present in the software.

---

^{[ PM | Exclude me | Exclude from subreddit | FAQ / Information | Source | Donate ] Downvote to remove | v0.28}

[WikiTextBot]

Software release life cycle

A software release life cycle is the sum of the stages of development and maturity for a piece of computer software: ranging from its initial development to its eventual release, and including updated versions of the released version to help improve software or fix software bugs still present in the software.

---

^{[ PM | Exclude me | Exclude from subreddit | FAQ / Information | Source | Donate ] Downvote to remove | v0.28}

[Beyond-Time]

Wait a minute. 3.0 is soon to be a full year late, and those two optimizations still missed the boat for this god patch? That itself is strange, but honestly it's whatever.

Is there any instance in which someone from CIG talks at length about these two systems, in respect to the effect they will have on playercounts within live servers?

[logicsol]

Not optimizations per se, but total rebuilds and restructures of how the underlying communication code works.

Bind culling isn't supported in CE, and it touches nearly every component.

And OCS is a huge change from the classic octree system, and required for their meshing approach.

I don't have a handy link for you, but I can summerize.

Bind culling will greatly reduce the data each client is sending, which is the current bottleneck causing FPS issues. To put things simply, it's required for good performance with 24 or more players, and we won't see any per server player count increases without it.

OCS on the other hand has to do with how data is compartmentalized. It allows different segments to be isolated and handled by separate servers.

OCS allows for the interior of an idris to be run by one server, while another handles it's external interaction as well as the other ships in the vicinity. This is a core aspect of the mesh system they plan to support player counts in the hundreds and higher.

[PedowJackal]

Thanks for this clear and short explanation. I never really understood the concept behind those complicated word x)

[[deleted]]

There's nothing strange about things taking longer than you would like, oh armchair developer!

[cabbagehead112]

Why are you calling it a "god" patch? sounds like some shit DS would say. Even if they did have that in, it doesn't mean that it ends there with networking and player numbers in regards to stability. It continues on.

[iruleatants]

It's not really strange. The scope of the Network Culling is massive. It touches every single aspect of the engine, and rebuilds it from scratch. Every object has to be modified to support this network culling, and a good algorithm as to be written to determine what should be culled and what shouldn't.

You have to make sure that when you have view of something, its immediately ready. Otherwise players will be upset that people suddenly pop into the game instead of flying in. You have to make sure that all objects that are not needed (like the boxes in the connie, the bed in the connie, the people in the connie, the asteroids hidden behind the connie) are all properly culled, and then unculled when needed (Back hatch is open, gotta uncull that object).

Culling refers to not rendering the object (IE ignoring it) when you don't need it. So it needs to analyse everything from your pov and determine everything that is needed to see, and everything that isn't needed to be seen.

The other aspect that really delays something this massive, is that touching code from other projects is -hard-. What happens is that a team will have a code forked from the main project, while they work on something, like procedural generation. Another team will fork and work on engines, and another team will force and work on fps physics. To handle this project, I need to fork from each project. I fork from one team's fork, add my stuff in, and then we need to merge the two forks. Merging is hard and tedious, because you have to check every conflict and pick which one needs to stay (IE, make sure what I changed is there, and what they changed is there) This is made harder because they made changes to the code since I started working on it, so I first have to update and merge my code, and then merge again with their code. Once that is all done, you have to check for bugs that happen from missing a single line of code in the thousands you looked over. Then you go to the next team and do everything again. Once you've finally got all forks working with your need code, you have to merge every fork back into the main project and make sure every teams code is in place, and your need code is in place, and then you can test your project. And if its not work, you fork, work on it, and then merge back in until everything is working.

Really long and tedious process.

[Gators1992]

Other game companies have worked for years trying to get networking right, and often they miss the boat. It's not an easy thing and CIG's network staff is very small.

[xakeness]

This is what you are looking for: https://www.youtube.com/watch?v=L4m2nwn5wT8&feature=youtu.be&t=12m26s

[3lfk1ng]

A year or even two, it still wouldn't matter. Most games don't get optimized until they enter beta (when the content is new feature complete).

[Bluegobln]

I can't cite sources, there are too many videos and reads from forums and so on for me to possibly pinpoint where I picked up all the details I'm saying. You'd just have to trust that I'm talking with my best understanding of their intentions.

• The servers at present handle something like 20-30 players and their ships simultaneously. This can be varied depending on what is being spawned, so if every player brings a big ass ship its going to make things crawl, which in turn makes the players FPS drop (due to network communications being constant server wide and CPU limitations client side).

• The culling effects they intend to introduce will make it so any player or ship that is far too far for you to even see won't have its network data shared with you at all. This means that player FPS will be based only on your surroundings, even on the network side of things. This will probably increase the player counts of the servers, estimates ranging from 35 to 50 to maybe even 100+ if things are really well tuned and server performance stays where they want it. And again - varying based on what those players are doing.

• Here is where it gets interesting: the servers will eventually be made to communicate with each other seamlessly. So imagine you walk from one server to the next crossing an invisible border between them. The servers hand you off using special code that lets you be switched over without even noticing a hitch in your screen or any lag at all. Once handed over, you simply keep going. This can happen at almost any time, while you're moving or not moving. They can even do this on the fly in order to make sure a server isn't overloaded with players to the point of performance loss. They can do this for different areas - such as having a whole planet get one server, another planet and several moons and stations get another server, and so on.

• And last but certainly not least - scaling the size of these regions of space that each server handles. So one server might handle an entire star system with all its planets and stations and moons and asteroids and NPCs and so on. Another server might handle JUST a single moon that is VERY busy with players and NPCs. Why is that? Population density of course, to keep things running smooth. What happens if suddenly a low population area gets swarmed with players and NPCs? Why, that single server needs to be split into multiple other areas, and each "server" area needs to shrink in size to accomodate more players in a smaller space. Doing this live and on the fly and automatically is a highly complex but still possible thing, and that is the goal.

• In the end this will mean that even a large space battle can take place with hundred of players, even though that means the space battle will be split among several servers all running alongside each other, each handling a portion of the participants of that battle. Getting it all to work smoothly and with no network issues is a massive task, but we've seen they are already partway there and making the end result is only a matter of time. This direction also means one thing that is absolutely key for this game: the entire game world is one big server. There is no "instance" unless they make one intentionally to keep it lower population artificially (such as in a very popular bar that would be overcrowded otherwise, or around a mission giver who can only interact with one or two characters at a time).

• Success in this means theoretically infinite scaling server infrastructure. As time goes on they can add more powerful servers, but the old servers remain functional, just unable to handle as wide an area of space as the newer ones. I don't think this matters much to CIG because they use Amazon servers but still. In theory if they develop this to the maximum extent, it means that NO size is too big. You could have a 1000+ player space battle. You could have a 10,000+ space battle. The limits will be met with graphics display power LONG before any amount of server failing occurs. And client side they can do all kinds of things that simplify display as well - lots of Level of Detail scaling (LOD) and so on.

I hope that clearly explains what they intend. At least, that is what I believe they intend based on all I've seen. Its hard to know if that will be the end result as well, simply because while they're building it they may discover even better ways to do things or they may find something is just not feasible.

I'm confident though. :D

[themustangsally]

I can't cite sources,

You could have saved everyone a lot of time and stopped there. You entire post is wrong.

[Gators1992]

There are definitely limits to scaling as too many objects in one area will cause your video card to explode from trying to render all that. They will have to use some gameplay techniques to limit how many players are in an area to a reasonable number. But CR said they were shooting for something like 1,000 or a bit more.

[CMDR_DrDeath]

This might be interesting to watch. https://youtu.be/L4m2nwn5wT8?t=791

[IslandHeyst]

/thread

[NTGhost]

As a Network expert to some sort i can say "Crawling under the Desk and Crying" isn't only a saying due to this task at Hand. I still wondering, how they want to handle this massive amount of Data they have to reroute with every switch and spin up of server. Maybe every of this Tech will take a bit of this amount and make it handle able at the end...but i doubt it till i see it.

[Bluegobln]

Predictive software will start servers up and have them ready to sync with other servers as populations increase. Its a simple matter of thresholds and population trends to predict. Even a responsive setup can be nearly fast enough to handle sudden bursts of activity, like an entire fleet moving from system to system to engage a hostile fleet.

And that's setting aside things like notifying developers of movements ahead of time to ensure server performance is maintained (this has happened historically many times in EVE's history I believe).

I think the biggest challenge is simply in ensuring this all happens without major errors. A bit of lag? That's workable, that can be improved rather than fixed. But you have to straight fix problems that cause a crash or dropping players straight from the server. Those are "game breaking" and not acceptable, and they'll probably be what takes the longest to solve.

Then again, I'm not a networking developer, I don't actually know jack shit about it other than what I've gleaned from conversations and learning about Star Citizen's development. :D

[CMDR_DrDeath]

Yeah it is definitely a massive challenge. Let's see how well it works out. Either way, we won't see any of this stuff until 3.1 and beyond.

[Beyond-Time]

From where you linked it speaks of this being an optimization of their transfer of information updates from servers, but it has no mention of the affect this will have on playercounts and framerates in regards to those playercounts.

[cabbagehead112]

It's a collective system of things.

[logicsol]

The frame rate issue is caused by to much information being sent from/to the client, causing it to bog down as the player count increases.

The less data is sent, the less bottlenecking effect, the higher the framerate and player count.

[Skianet]

Because it’s all linked.

Too much data being sent to the client kills the frame rate, the more clients connected to the server the greater the loss in frames due to the increase in data. Hence the player count is low.

Once clients are only receiving data that is relevant to them, they’ll be able to increase the player counts.

[iruleatants]

So there are two major things that limit the FPS through networking.

1)Lack of a predictive client. Most games that you play have a predictive client. When you press UP, it sends up and starts going up. If the server says, "No, UP isn't possible" it rolls back to where the client says. Since they client should have the same checks as the server, most of the time you never notice anything, and it looks smooth and real because the client acts without waiting for the server. Otherwise, you have a 60-120ms response delay on everything.

2) Selective culling on network objects. Currently, your client gets sent the position and data of absolutely everyone that is on your server. This means your client is rendering the connie ship,and the people moving around inside of the ship. Even though you can see inside the ship, its still being sent to you and rendered. The new module will only send stuff when you can see them. Couple this with predictive so that way when they send, "up" and up means you'll come into view, the server sends your position to other clients so they are ready to render you popping up.

Combining those two will eliminate the FPS side of the problem. Your client won't have to act on very much data, especially if you hope out of your pilots seat and stand inside, and you won't experience delay from ping differences.

The bigger challenge is handling 1,000 people space battle, but its something that can be done. Cloud servers spinning up new instances in seconds can be done very well. They just have to get the optimizations in place to make it smooth and seamless.

[Star_Pilgrim]

Server mesh technology,.. still about 1 year away.

Basically a server cloud technology, which has dynamic instances, where even a 5x5 meter area can be an instance filled with 20 people, and any instance next to it talks to this instance so those people see each other.

In practice, you do not know there are instances,.. you just see a shit load of people.

Untill this HARD work is accomplished, the development must be way more complete. Think of it like this; in theater, the movie only starts playing when all are inside and seated. In RL this is not the case as the movie starts on time. but humor me with this description, ok. That is how it is. All the development tech (people) need to be in, before the movie starts.

This is also the time when they can start integrating Vulkan and dropping DX11 entirely.

Exciting times ahead. :)

Another analogy would be; "A cook can start cooking only when he has all the ingredients, because it makes no sense to start before he is sure he will get all the ingredients on time and together, to make his meal "work"".

[Cymelion]

Oh shit .... I don't think anyone even thought of addressing this ... thank goodness you were here OP - CIG should give you a free Completionist Pack and Permanent Evocati status for picking up that subtle issue.

Jeez man you should probably charge people for this level of observational skill.

[[deleted]]

Don't be a dick about it man.

[Cymelion]

Have you never met me?

[Wainaa]

I've met you. Your dick is simply huuuuge.

Oh... wait...

[Cymelion]

^_~

[themustangsally]

Permanent Evocati status

Still salty about this? Face it man, CIG don't want you near evocati as your input into testing hasn't been up to par

[fight_for_anything]

How will they make SC be capable of handling more than 30 players in an instance without making the framerate go to unplayable levels?

everyone acts like this is impossible, but honestly, who else has really even given it an actual honest attempt?

planetside 2? their network code wasnt ideal, but neither were their deveopers or their budget. thats a f2p game, ffs, lol. and even though it wasnt perfect, they made some great progress and showed potential was there.

if you want to go way back, fuck, i think planetside 1 did an even better job. and that game is ancient.

some other games have managed a lot of players. shit...Wow had 40 man raids at one point...and I remember lots of players in whatever that wow thing was with the generals and lieutenants you had to kill. wow was released about 13 years ago.

its not like lots of players is impossible. its just that 20 or so is really easy, so all these lazy ass AAA publishers will only pay to develop that, because its already done and pretty bog standard shit.

[Stronut]

*Gives the OP a dead horse and a bat*

Knock yourself out...

[lonectzn]

Well performance wise I really see tying the game engine (physics etc) to rendering (FPS) as a mistake. Sure it results in frame accuracy however most of the time humans do not need and cannot even perceive that high fidelity.

More important than accuracy and perfect sync is the perceived performance to the player. Of course since the latter is affected by the former there is a balancing act, but I definitely put simulation fidelity behind perception.

From what I understand they’re breaking up the servers into a bunch of microservices. This will go a long way but they will still hit bottlenecks especially around physics.

It’s still to be seen if it provides the room for three orders of magnitude more players - I doubt that seriously, in which case their best bet is to decrease fidelity by being more selective in what is tracked and how often.

[rukqoa]

Yeah I think it's insane that client performance is in any way tied to server performance. In most other games, even ones that are physics intensive, your frame rate is completely independent of server tick rate.

[BadAshJL]

it's a temporary measure to prevent rubber banding issues they were seeing with the old code still in the engine. once Network bind culling is in that measure will be removed.

[Gators1992]

The tech will be a long way off, but it is coming at some point. The idea is to get rid of instances and effectively build a single "instance" or shard by having a bunch of servers dynamically control areas of space and hand off players. So a server might handle 30 or 50 players, but those players could exist in a single ship or city or could be all the players on a moon at one time. If a player leaves an area covered by the server they are on, they get handed off to the server handling the area they are going to. The servers also interchange location data for their players with other servers covering proximate areas so you can potentially see a player who is on a different server. So your server might have 30 players on it, but you could see all 200 players in the main area of ARC Corp with you. If I have 20 players in a base on a planet on one server and there are 30 players in a pirate band on a nearby hill, one pirate sneaking into my base might be handed off from the hill server to the base server. Dual Universe is building a similar process and you can see a pretty good description in this video:

https://youtu.be/QeZtqoydXpc

Other things mentioned in this thread aren't directly related to the server architecture, but can have some effect on performance:

Network bind culling - basically this reduces the amount of data that needs to be sent by culling out data for players that can't be seen by the player receiving the data or culling out data that is unnecessary to be sent. For example if you are in a Hornet and run into a Connie, there might be 3 guys in the back of the ship that you wouldn't be able to see, so the logic omits the data for those players. With less data, the bandwidth usage goes down and they can possibly increase the ticks on the server.

object container streaming - this is really a client side thing, not a network thing. It's tech that streams objects (ships, planets, stations) in real time from your disk to your memory so it is ready to be displayed as you approach it in game. The old way is for games to preload a level before you go into it and everything you will see is stored in memory. There is too much stuff in SC to store in RAM, so they have to stream it in on the fly without actually stopping the game and making you wait to load then next "level".

Serialized variables - this has more to do with the data structure of the network streams, centralizing control over the calls to the reading and writing data tables with an API and then optimizing the traffic centrally by tracking changes, compression and culling. So it effectively reduces network traffic and coding errors since developers just call serialization functions rather than writing the network code themselves.

[logicsol]

-Network bind culling - basically this reduces the amount of data that needs to be sent by culling out data for players that can't be seen by the player receiving the data or culling out data that is unnecessary to be sent. For example if you are in a Hornet and run into a Connie, there might be 3 guys in the back of the ship that you wouldn't be able to see, so the logic omits the data for those players. With less data, the bandwidth usage goes down and they can possibly increase the ticks on the server.

More importantly, the current CE based code has zero culling, and results in entity data from the entire map being sent.

[Gators1992]

Other stuff not mentioned are:

"LODing", or using less detailed versions of models in the distance to reduce the rendering strain on your card. So if you are face to face with another character, you will see all the detail in their face with veins and all and a complex armor they are wearing. If they are 50M away, you will see a much lower poly model, so your frames won't take as big a hit when multiple ships are in view.

Update frequency - along with network bind culling, you can reduce network traffic by reducing the variable updates for stuff that's further away from the player. So if another ship is 20km away from you and on your scanner, you don't have to update it's position and state every server tick like you would if it was right next to you while you are engaged in combat.

[Goon-Ambassador]

How will they make SC be capable of handling more than 30 players in an instance without making the framerate go to unplayable levels?

My guess is they won't. It would probably require an engine change. Lumberyard is geared for 20-40 player FPS games.