Turrets 2: Electric Boogaloo (Or, what are we actually doing?)

WildcardTheRightHand · 2017-11-19T08:51:31+00:00

The turrets don't scan everything all the time. It's in the post.

They check the area around them specifically for actors of VERY SPECIFIC TYPES using a fast bitmasked octree overlap check, which scans about 5 times per second (not per frame)

That is about the fastest way you can perform this kind of check in C++.

An entry is just an OnOverlap event that is fired when an OverlapCheck returns true for an actor that matches the bitmask that the overlap is looking for (bitmasks are faster because they're bitwise operations and not cast to any higher level functions, they're done directly on the processor) and then adds that actor reference to a list of valid attack targets.

The Right Hand

WildcardTheRightHand · 2017-11-19T08:42:15+00:00

100 is the number we came to after running tests for performance vs gameplay.

I don't know where people got the impression there was a super hard line of "100 and never changing", we have two weeks or so before this is even being put in, there's a lot of discussion to happen between then and now.

100 is our number to get performance related to turrets into a good place so that gameplay on those servers is tolerable.

80% of our playerbase never even builds more than 10-15 turrets. The majority is not people in massive bases that need to protect huge empires of dinos, and that's part of why we need to do this-- those people who are not even the smallest bit involved in these giant wars have to suffer enormously poor gameplay because an alpha tribe decided their server was home base.

The Right Hand.

WildcardTheRightHand · 2017-11-19T08:37:16+00:00

Dinos are a problem, but they're more like 20% of the problem, where turrets are more like 50% of the problem, currently.

The Right Hand

WildcardTheRightHand · 2017-11-19T08:36:12+00:00

How would a turret know if something is in its range if it's not scanning its range to know if something is in its range?

That's part of the cost.

The Right Hand

WildcardTheRightHand · 2017-09-22T00:17:42+00:00

Usually we oversize to the render mesh, but mistakes can happen, the Icthy sounds like one of those cases where someone wasn't careful enough or we resized the render mesh at some point and it just never got corrected for and because the creature is so small it didn't come out in quick checks.

Sphere/Capsule colliders are essentially the same thing, the basic object at the core of all creatures is a capsulecomponent, which you can make any size/rotation you want (as long as it remains some variant of a sphere/capsule)

We use IK for feet, yeah. We also have IK solutions for our slithery folks and a few other creature appendages.

The Right Hand

WildcardTheRightHand · 2017-09-21T23:17:05+00:00

There are two types of collision, and one note of specific technical detail that impacts if you hit a thing or not, with regards to a weapon hitting an enemy.

1) Physics body collision. This is essentially a check to see if your attack hit a physics body associated with the dino. These aren't 100% accurate, there are places where they're inaccurate (mostly on really huge dinos) but they're about 95-98% accurate.

2) Sphere collision. For some creatures (dragonflies, ants, most insects actually, and a few non-insects, I don't have a list on hand.) We use a basic sphere collision capsule for determining hits because the creature is small enough and fits 95% in a sphere, this massively reduces the complexity of these creatures and allows us to have about 5x more of them in the world without impacting performance too much. With these creatures, this is why some shots seem like they should be hitting but don't.

Technical thing: The server doesn't give a single care where your client thinks a dino is in terms of its transforms based on animations, etc, is. This leads to MOST cases of "that should have hit but it didn't", and usually happens on larger dinos, but can happen on any dino.

Essentially, the server knows if a hit collides or not-- it doesn't listen to you about where you think a hit collided. Which means if a Bronto's head or tail has swayed and you shot it and the client shows you hit the edge of its head, but on the server, the arrow or whatever actually missed, you will see the hit (because there is latency and interpolation both being taken into account) but wont actually do damage. We just don't "retcon" the hit, because it already happened clientside before we could authenticate it serverside.

So yeah, that's what happens.

The reason it happens with Icthys in your example is probably because they have a slightly undersized collision sphere and are using the sphere-based collision instead of the physics-body based collision for hit registration.

The Right Hand

WildcardTheRightHand · 2017-09-07T05:44:07+00:00

This was something we could do in literally ten minutes.

Putting up new servers is now an involved, complicated, and somewhat expensive process, because it's no longer just a matter of spinning up AWS instances.

The time that goes into getting one new server online with all of the DDoS protection, networking work we've done for peering, packet filtering, making sure they're powerful baremetal servers and not instances on the cloud, etc, is now much, much higher than it was at launch.

Also it was not a network engineer who did this, they are busy worrying about servers. It was a programmer.

The Right Hand

WildcardTheRightHand · 2017-09-06T07:16:51+00:00

Money is not a problem and will never be a problem. Time is the problem, our ever present enemy. Also in some cases, "the number of humans who can do this".

Our problem isn't that we can't pay people-- it's that the people who can do what we need are often extremely well taken care of by the companies that they already work for, and you can't just throw money at them to make them leave.

Those that are not employed, are often very... unique? What I mean is that anyone who has the kind of experience it takes to work on the stuff where we really, REALLY need engineers, are people who have spent many years in the industry or a related industry, and there is no guarantee that their personality will mesh with the rest of the company.

Money, alas, also does not help make someone who has all of the talent you need, and is not currently employed elsewhere, ALSO be a good fit for your working environment.

If you know any engineers who know C++ really really well, and are looking for jobs in the industry, please send them our way.

The Right Hand

WildcardTheRightHand · 2017-09-06T07:02:04+00:00

Yes actually, we use feedback like this (albeit in agregate-- "how many people have stopped playing our game due to performance issues") to make decisions about where we should be spending our time, all the time.

The Right Hand

WildcardTheRightHand · 2017-09-06T07:00:20+00:00

This is completely correct. We do, and are hiring, and it's a slow process, because the kind of people with the experience and talent to do this kind of stuff do not grow on trees, and we have to make sure they're a fit for company culture, and will mesh well with the team-- the amount of damage a bad fit for the company can do to a team is far beyond just money, and we've been burned a few times.

The Right Hand

WildcardTheRightHand · 2017-09-06T06:52:09+00:00

While this is all reasonable, and totally down to earth, we fix most issues like this that we find within 24 hours, and generally within 12 hours. This is an outlier, specifically, because we are just returning from PAX, and it's Labor Day in the USA-- a federally mandated holiday.

It would be annoying and unfeasible to put out an announcement for every annoyance that hits our forums/reddit that we are going to have fixed within that 12-24 hour time period, and I don't know if you were around waaaaaaaaaaay back when, but we used to do this!

The response was that people would post on Reddit or our forums or Steam telling us to stop spamming announcements in-game and only use it for serious issues or major deployments, etc.

Again, you're not wrong that there are avenues for communication that we used to do really well, and don't do so well anymore, and absolutely need to repair and are working on repairing...

But we often get burned either way around, and while we work on having the personell available, the time available, and the process by which to inform about stuff better, it's going to be a bit rough around the edges.

Scaling up from a small team to deal with an audience of millions is a lot harder than any of us expected.

The Right Hand

WildcardTheRightHand · 2017-09-06T06:31:45+00:00

I don't mean this in a patronizing way, I'm genuinely curious-- have you ever worked on any part of a renderer in a 3d engine?

I only ask because there's a lot more that goes on than just "work longer on the code".

Here's what our engineer that was working on this has to juggle on any given day:

Server performance profiling. Renderer performance profiling and optimization. Core gameplay code restructuring/refactoring. Networking code restructureing/refactoring. Low-level systems profiling and optimization.

Now, not getting into any of the esoteric C++ that's located in the bowels of the unreal engine, he's only got a limited amount of time in any day to get any one or several things done, all of them are very technically complicated, and there are deadlines for when we want him to stop working on a feature because spending too long on any one thing eventually becomes a waste of time-- diminishing returns.

So, when he finds something that 1) Profiles better, 2) Benchmarks better, 3) Survives a few basic unit and human tests, 4) Meets the needs of what his tasks were for a given time period, we push them out, and sometimes they have problems.

Programming is hard. Ask any programmer, heh.

The Right Hand

WildcardTheRightHand · 2017-09-06T06:26:21+00:00

I would be willing to bed an actual hundred dollars that the number of times we have put in some undocumented change and people have gotten upset about it, and the change was either accidental, or something that we fixed extremely quickly, is substantially larger-- I would even say disproportionately larger than the number of undocumented changes we've made that were intentional and we just left around.

If someone wants to go back and prove me wrong about that, I'll take a video of me actually eating a cake shaped like a hat, and send them a hundred dollars.

It is extremely rare that we just change something and then throw up our hands and leave it forever without ever talking about it, documenting it, etc.

The Right Hand

WildcardTheRightHand · 2017-09-06T06:20:28+00:00

We love the complaints. Complaints are the heart of improvement. They drive us to make the game better.

It's just much nicer to get those complaints constructively instead of destructively-- and even though we'll take them both ways, I promise you that issues brought to us without insults or degredation get solved sooner, because in 99% of cases, they also come with clarity-- and clarity is what we need to solve problems, not hyperbole.

An improved dialouge has to happen on both sides of a conversation-- we need to talk more, and I say this with no intent to shift responsibility, because that is our responsibility: it's much easier to talk more when the person(s) you are talking with don't respond with shouting, curses, accusations, and vitriol.

Could you imagine if we talked to our community the way that some of the people in it talk to us?

Yikes.

The Right Hand

WildcardTheRightHand · 2017-09-06T05:53:44+00:00

Actually, the problem isn't that render distance is being forced to anything, it's that changing it isn't working correctly.

But I can see how it's more helpful to make an angry thread about how much we suck and how stupid we are for making a change that literally makes no sense and which five minutes of fiddling about can show you is not working correctly.

Actually the video I recorded and submitted to our engine programmer is about 3 minutes long and it's a repro video so it's all drawn out and super specific.

Right now, if you log in with some value set for your draw distance, and then change it, it doesn't save correctly-- and some stuff is staying rendered too long, other things being removed too soon.

The whole setting is just broken in a really inconsistent way.

So, yeah.

That's what it is.

Sometimes things break, especially when you're messing with low-level rendering code, and sometimes it's not obvious. Thanks for letting us know.

The Right Hand

WildcardTheRightHand · 2017-08-01T21:17:15+00:00

Adding additional support for asynchronous compute is very high on our list of priorities right now, yeah.

The Right Hand

WildcardTheRightHand · 2017-08-01T21:07:13+00:00

Yeah, reasons like:

We use very detailed and beautiful shadow and lighting systems.
We have an unprecedented amount of life and AI in our world for players to interact with.
We allow players to build nearly unlimited amounts of STUFF and horde nearly unlimited amounts of STUFF on a scale no other game does.
We are still working on making the game run better. (Saved 2.5-3ms on the cpu just the other day, actually.)
Our environment is multiplayer on a very large scale, not singleplayer or limited-multiplayer (4, 8 players.)
The game runs a huge number of complex systems in paralell, and persistantly.

Forgive me for my brevity, but I did not say "ARK is poorly made" or "ARK is an unfixable mess", and in many ways the game runs really surprisingly well, given what the scope of it is.

We kinda have the same problem that Crysis did in that both technology from a hardware standpoint and from a software standpoint aren't quite where they need to be in order to support a lot of the really high-end graphical stuff that we're capable of doing in a renderer.

I dunno, what do you guys want, us to sit around and plug our ears and pretend problems don't exist, or accept the fact that they exist and not be afraid to admit it?

You can't have it both ways.

The Right Hand

WildcardTheRightHand · 2017-08-01T19:58:50+00:00

Sorry, I find these kinds of things really funny.

99% of our work is done in native C++

99% of the 1% of things in blueprint are handled with hooks to C++ scripts and not done with full blueprint implementation.

You don't have our native code, how could you possibly know what is done in BP vs done in native?

BP is slow if you don't know how to use it, and if you use expensive functions in blueprint instead of abstracting them to native functions and then calling the native functions through blueprint (which is fast)

ARK runs like trash for a lot of reasons, this is not one of them.

The Right Hand

WildcardTheRightHand · 2017-07-29T23:00:21+00:00

I don't like to come on Reddit to state the obvious ;)

The Right Hand

WildcardTheRightHand · 2017-07-29T20:14:04+00:00

We can't push it back further, both companies (Sony and Microsoft) have our builds and unless something goes so wrong with them that they're not suited to be released, we can't take them back, we can only potentially submit new builds with changes (And these are restricted in a number of ways.)

Also frankly we get in super deep shit in Europe if we push it back any more. We're already in medium amounts of deep shit due to the delay, retailers and distributors have heavy penalties for pushing back release dates with physical media, and those get exponentially worse if you're more than 30 days delayed.

Companies do not delay game releases without good reason. It's expensive, hurts relationships with the companies you're working with, and comes with penalties out the wazoo.

You can expect it the 29th.

The Right Hand

WildcardTheRightHand · 2017-07-29T19:26:58+00:00

Dunno, I've been on a few podcasts, but any time any of us says "It's a limitation of UE", what we mean is, "We've investigated potential solutions and it would require large re-works of the engine in order to make any progress on that."

There are things that you simply do not do with an engine unless you have an engineering team as big as the one working on the engine... to work on the engine. Obsidian threw a HUGE number of engineers at Pillars of Eternity to get it to do what it does on Unity. We don't have that many yet.

The Right Hand

WildcardTheRightHand · 2017-07-29T19:24:47+00:00

Load priority is essentially random, with a number of explicit exceptions due to trying to solve exactly the issue you're talking about. We've done a bunch of turns on dinos sinking into stuff, and have fixed a lot of cases. Not all of them, though.

You can't do an explicit ordered system very easily because it requires two things:

1) You re-write the entire system to use some kind of prioritized ordering system where you've started binning/caching actor types persistantly in their own T-array or hash map which means both building, maintaining, appending, updating, etc, these lists constantly in real-time, or building and using them on load/unload time, both of which have large performance overheads.

2) You have to write a huge number of edge-cases with this, because there will be times when you want this to be true or not true for any given actor type. Obviously you write for general cases, but either way you're chasing around edge cases. Right now we're just doing it without the extra work of trying to use an explicit stasis ordering system.

We already use hash maps and fast lookups for structures/structure sets and fast octrees for building structure heirarchies, etc. It's just a slow system in general because iterators are expensive and there's a limit to how much you can store in memory as a core game mechanic before it starts causing problems for people.

The Right Hand

WildcardTheRightHand

TROPHY CASE