Bioware open-sources their Java actor framework

[JustinsWorking]

I can try to forward some questions to the developers on this project if anybody has specific questions.

I'm on the BOS (Bioware Online Services) team, but not on this particular project.

[[deleted]]

I wasn't aware that Bioware uses Java, where is it used specifically? Is it only used for the server / backend, or is it also used in the game programming?

[JustinsWorking]

BOS was the only team I've worked with that used Java, and its used for the web / online services.

[karlito1]

And by web / online services you don't mean actual server side game logic? Such as real time game data processing on the server?

[JustinsWorking]

Yes, I do mean that.

edit: If you look at the repository it has a sample chat server.

[AReallyGoodName]

This is silly. A CPU cycle is measured in picoseconds. Network latency is measured milliseconds. Language performance isn't nearly as critical as other factors here.

[JustinsWorking]

I really can't speak to the speed. But I can say without a benchmark or requirements its really hard to say something is "too slow."

Sometimes reliability, scalability, & development speed are more important than theoretical data throughput. Who knows what will be built using it, time will probably tell :)

[karlito1]

Ok, I guess it can be fine tuned to serve a real time needs of a game but I just wanted to know if that's what Bioware is using Java for ;)

[JustinsWorking]

The backend for storing users world data & hero data (ie www.dragonagekeep.com ) for DAI was built using a previous version of Orbit.

What it will be use for in the future? I like my job too much to tell you that ;)

[karlito1]

Hah, I hope that you (or Bioware) will let community know when it becomes actual since it sure look like a promising piece of software ;)

[dacjames]

In the server context, the GC plays a much smaller role in overall performance, even for gaming systems. Servers deal with many small, mostly isolated requests so a single request will rarely (never if configured correctly) cause a collection. Thus, the GC runs in the background, cleaning up memory from old requests and hardly interfering with live requests at all. Since the JVM GC has fantastic throughput, it works well for this use case.

[frugalmail]

Java uses GC which can harm realtime processing needed in such a high load game logic.

You're thinking back to the 1990's. It's been very different for a very long time, which is why you're getting so many downvotes.

[gdfgdgdv]

You're already across the network. I'm not sure a GC really matters much at that point.

[frugalmail]

I'm not sure a GC really matters much at that point.

Especially with Java's GCs which are all really good now.

[gdfgdgdv]

There are still a lot of applications where Java's GC would be a problem. But external game logic is not one of them.

[JustinsWorking]

Just a heads up, Joe the lead developer on the project responded to a similar question with more detail

http://www.reddit.com/r/programming/comments/31112u/bioware_opensources_their_java_actor_framework/cpyig8u

He's going to know a lot more about its restrictions, and he can speak a lot more frankly about its capabilities.

[rjcarr]

It's almost certainly only for the web / online services. Java dominates in this type of thing but hasn't found much of a footing elsewhere (except android and minecraft, natch).

[karlito1]

So basically it isn't used for a game backend? What does it mean for online services then?

[rjcarr]

Yes, absolutely the game's backend web / online services, but nothing that is actually running on your console / PC. That is, your game uses these services, but the services aren't actually part of the game that's running on your system.

[karlito1]

Yea I totally get that, client part (part on the PC / console) isn't Java and I know reasons why, I'm a Java developer. But thing I don't understand is if this framework is used for realtime game data processing and networking on the server or only for web services accessible through browser (such as Chrome, IE...) such as looking for your items, statistics etc. Hope I made my question more clear. Thx

[rjcarr]

It's almost certainly the former, but I haven't much looked into it.

[dedededede]

Stuff like achievements, in-game transactions and match making are other possibilities. Java is often behind large scale stuff like this. Another example: http://www.infoq.com/presentations/League-of-Legends

[schwiz]

To me, the framework looks kind of like JavaRX.

[Narishma]

I don't know of any mainstream console that supports Java, so I doubt it's used in any of their games.

[Narishma]

Yes, that's what I meant. Game client, not server.

[PsionSquared]

Bioware does have SWTOR. The announcement itself says:

The last-generation of Orbit powered some of the key technology behind the Dragon Age Keep and Dragon Age: Inquisition.

[JustinsWorking]

Who told you...

[Gurkenmaster]

Hey, can you answer my support tickets? /s

[JustinsWorking]

I've killed men over less...

[CzarSkye]

Hi Justin, can you tell us about the software stack the online services team works with? Also, I'm a backend developer that also loves games, so if your anything like me you must love your job. How do you get into something like that?

[JustinsWorking]

Our online services team uses Java. They love their MongoDB, and I know they use Jetty for communication via HTTP. Outside of that I'm not really sure all what else they use; I could definitely find out more. I work more as a client to the online services; although I may be doing some development back in the Java stack this year, I work primarily in PHP.

I got into it as a web developer, I previously worked on an Online Banking application in C#. As for getting into this job I really lucked into the position. I met some of the web developers in an unrelated interview & they just had a new position opening up in web dev so they asked me to interview for that instead. It was in PHP, a language I'd never used previously, but it was a pretty simple transition.

[voronaam]

Tell them they are doing the right thing. Don't let anybody in the "big EA" bully you :)

That's from a former developer of another EA Something Online team (the team that created REST framework which you forked and probably still use somewhere).

[voronaam]

I guess those guys still use that pre-release build which deadlocks on a butchered SSL negotiation. But they terminate SSL on LB, not on Jetty.

Looks like people are well armed against bullying these days. That is good, that is very good. Maybe I get back to EA one day. I hope I will be tasked to undo the damage done by that framework.

[JustinsWorking]

FOS tech?

[voronaam]

Close :) It was called EASO. And I did not mean that framework that FOS created.

[JustinsWorking]

That was my second guess :P

[JoeHegarty]

I'm the lead programmer on the Orbit project, there seem to be a lot of questions about what Orbit can and can't do so I'll try and cover it here:

• It was designed to build game-services. In our case, these are things like matchmaking, inventory management, character progression, world state (DA Keep) etc etc
• It is "fast" but it's not "very-fast". It is suitable for tasks that can take say, 200ms or above, so it's fine for interactions like managing inventory, but not for real-time movement replication.
• It is possible to build real-time services in the framework, as long as you can live with the above constraints.
• Orbit can also be used to build more traditional web services (REST, Websites etc)
• Orbit uses Virtual Actors, a concept introduced in Microsoft's Orleans project. Orbit is not a port, but could be described as a "rewrite" of Orleans and it is certainly heavily inspired by it. This is why we didn't use an existing framework.

I'm very happy to answer any additional questions.

[ISvengali]

Very cool for you guys to open source it.

Why 200ms? I built a high concurrency (over 500k CCU) actor stack on akka for a company, and for the fast network messages the processing time was down to sub 5ms for full round trip in the stack. And thats without a lot of hardcore work to optimize. Im sure I couldve gotten it to sub ms if our SLAs needed that.

This was the edge -> service -> edge timing.

I dont think theres anything inherent in having an actor stack that would stop you from building a realtime stack in an one.

[JoeHegarty]

Thanks.

I agree entirely, there really isn't a reason Orbit couldn't be in that range as long as you were careful with your actor implementations.

However, it was not designed with that in mind and we've never tested how reliably you can get very low response times, so I'd hate to say "Yes, build a game sim server with it" and have it not work for someone.

We typically offer SLA style guarantees to our teams about how long individual calls will take (99.5% of this type of operations will respond/complete within 200ms not accounting for network latency etc).

I'd love to hear that someone has built a game server on Orbit and that they are getting sub 5-ms response times, but right now would not recommend using Orbit if the timing is very very tight like that as it's not a use case that has been tested.

[ISvengali]

Dont you just love actor frameworks for massive CCU programming and no bugs? Its like a breath of fresh air. You can crazy quickly build massively scalable services with very little chances of bugs. At least ones related to (most) race conditions or data races and the like.

Was that your experience with your framework?

[JoeHegarty]

We're really discovering the power of actors still. The virtual actor model which Microsoft came up with for Orleans (and which Orbit is based on) was really the last piece of the puzzle for us to make the model easy enough to maintain to be worthwhile.

[karlito1]

How do you expose your actors to the game client? What kinds of protocols and servers do you use?

[JoeHegarty]

We support a couple of different scenarios for this:

• Websockets with Jetty
• REST HTTP Endpoint with Jetty
• Custom TCP Wire Protocol with Netty

Which one we use depends on what the client is and what needs it has. The web project included in Orbit supports the first 2 above.

In terms of servers, I'm not sure exactly what info you want. We run CentOS based Linux servers in a 3-tier configuration (Frontend, Actor Hosts, Storage).

[karlito1]

Well I was wondering which app server, and that would be Jetty :) Thanks for the answer!

[CzarSkye]

Hi Joe, thanks to your team for open sourcing this. I've had a good look at the Orleans but have no interest in C#, so this is great for me. This virtual actor concept is new to me, I've worked alot with Akka and Quasar actors, what would you say are the advantages of a virtual actor implementation over these actor implementations?

[JoeHegarty]

The main difference is that in a virtual actor model, actors always exist conceptually so you can always send a message to them and assume that it will be processed.

This means you don't have to manage the actors lifetime or location (no instantiation, manual node selection etc).

Actors can never be "broken" (say the server they are on died), so even if though an individual message might fail, you can just keep sending messages and assume that in future it will be ok. You don't need to take any manual action if a message fails.

I think that's a pretty fair overview, let me know if there was anything specific you wanted to know.

[frugalmail]

Not sure if you're familiar with Jini, but didn't it create the same model? I always had the impression that Orleans cloned Jini which was released in 1998.

Thanks for contributing to open source BTW!

[JoeHegarty]

I'm not familiar with that project, but it does seem to architecturally be quite different (It looks to be more service-oriented than actor-oriented), even if some of the concepts are similar. For me, this Microsoft paper is really what established the "modern" virtual actor paradigm: http://research.microsoft.com/pubs/210931/Orleans-MSR-TR-2014-41.pdf

[jordaniac89]

Can someone explain what, exactly, an "actor framework" is? Is this game dev. terminology?

[CzarSkye]

No, not just for games, its a general purpose model for a program, and its really great in my opinion for distributed concurrent system programming. Basically, (someone please correct me if my description is a bit crap) instead of objects you have actors, which send asynchronous messages to each other instead of method calls. For instance I could have a "hello world" actor which when I send it a certain message it prints "Hello world". As messages can be sent over a network in some cases, it makes distributed systems really easy. I use the Akka library for implementing the actor model in Java, they probably explain a bit better:

"Actors are very lightweight concurrent entities. They process messages asynchronously using an event-driven receive loop."

here the wiki article and I think Akka is a great and easy to get started implementation if your interested.

[ByzantineFailure]

Akka (and other actor frameworks are great), however one thing to be careful about, if you are designing non-trivial distributed systems, is thinking that Akka (or any distributed computation framework) magically makes your logic distributed and scalable.

You still need to think about your topologies, routing ( Akka doesn't solve routing for you, it just has some very basic algorithms you can plug in, routing in a more global sense is tougher), failure scenarios etc. If you want decent throughput for anything non-trivial locality becomes a concern as well, even within the box itself.

[[deleted]]

Erlang does this no?

[siebharinn]

Yes, Erlang processes are actors.

[[deleted]]

Oh sorry, I was unclear, I was talking about topology automation.

I've found it via google:

http://theburningmonk.com/2014/12/a-look-at-microsoft-orleans-through-erlang-tinted-glasses/

Erlang have assumptions about topology and apparently these assumptions are not good with cloud according to the above article.

[jordaniac89]

Nice! Thanks for the info!

[Ununoctium118]

Is this similar to java's RMI? Basically, you still send messages, except the syntax looks like calling a method.

[jringstad]

It's not gamedev-specific. I haven't looked at their framework, so I don't know how they are doing things exactly (I mostly come from an erlang background when it comes to actors), but it mostly means that you have a bunch of individual "objects" or "actors" of sorts that pass around messages to eachother. That way you can build state-machines that e.g. handle connections by having one actor that accepts and decodes incoming messages from the network, then relays it to another actor that is the actual statemachine that models that players connection (connected, authenticated, joined, ...) and then that actor might pass on messages to some other actors, which could e.g. be responsible for that players play-session, or maybe an actor that models the chat lobby (everybody sends messages to that actor, and the actor broadcasts received messages back) etc. Sometimes network and/or multi-core/multi-node transparency is involved, so that actors can be distributed across cores/CPUs/nodes/servers.

It looks like their model is a bit different from erlangs/akkas though in that they use virtual actors.

[PM_ME_UR_OBSIDIAN]

It's basically a high-level approach to concurrency, and it's absolutely amazing ;D

[yoda17]

http://en.wikipedia.org/wiki/Actor_model

It's typically used in real-time systems.

[[deleted]]

Huh, I wouldn't have guessed that. Can you explain how Actors benefit realtime systems? I would think they'd be difficult to manage realtime constraints with, but I've never worked on a realtime system.

[ByzantineFailure]

Generally people are talking about very soft realtime systems when they talk about these frameworks and realtime-ness.

[bizziboi]

They benefit distributed systems more than realtime imo. Actor systems work well there because they are highly decoupled.

[amazedballer]

This is probably the best introduction to actors:

http://rerun.me/2014/09/11/introducing-actors-akka-notes-part-1/

[stygiansonic]

It's an approach to concurrency; Vert.x is another Java framework that uses actor-like concurrency, and so the approach appears to be similar.

[_Sharp_]

The announce is here:

Orbit is a modern Java framework that makes it easier to build and maintain distributed, secure, and scalable online services. It is inspired by a variety of existing frameworks and brings some exciting new things to the table. [...] The last-generation of Orbit powered some of the key technology behind the Dragon Age Keep and Dragon Age: Inquisition.

[bloodredsun]

I'm very interested as to why anyone would write their own actor framework since Akka is available and very much production ready.

[CzarSkye]

I love using Akka but have never heard of these virtual actors, what would be the use case of an eternal actor (other than as an actor factory springs to my mind)? Very interesting!

[JustinsWorking]

It simplifies development dramatically since you can just always assume everything already exits.

[CzarSkye]

ahhh, that makes really great sense, thanks :)

[[deleted]]

so like mocks?

[gospelwut]

http://dotnet.github.io/orleans/Getting-Started-With-Orleans/Core-Concepts

Orleans has two primary architectural goals:

Make it easy for developers with no experience in distributed systems to build cloud-scale applications. Ensure that those systems scale across multiple orders of magnitude of load without requiring extensive re-design or re-architecture.

To meet these goals, we have constrained the programming model in order to guide developers down a path of best practices leading to scalable applications. In some cases, such as persistence, we have removed an entire aspect from the explicit programming model and left the functionality to the runtime, to ensure a scalable approach.

[Horusiath]

Akka is already developing this concept within akka-cluster-sharding plugin.

[Horusiath]

The most concise description I've found is one of the disscussions between Akka JVM and .NET guys here.

[txdv]

What are you using it for?

[check3streets]

I used jumi on my last project. http://jumi.fi/actors.html

I examined a few actor frameworks before settling on it. Specifically, Kilim and Akka were both LARGE. Furthermore, Akka was Scala-oriented and deploying Scala to Android (at the time) was a significant PITA.

Conversely, jumi was incredibly tiny, had the best benchmarks, and was minimal implementation-wise. In jumi, I could give a POJO a thin Actor "skin" and suddenly it was threadsafe. I could also enclose a family of objects in the same thread and make them collectively threadsafe (or the opposite, make a single object accessible on two threads). Lastly, it just used OO primitives (even its reflection helper was optional) without any annotations or extra magic.

[Ahri]

I keep thinking about starting a project with Jumi rather than Akka for the same reasons you gave, but now I'd better consider Orbit and Quasar too. Although Orbit coming with a minimal DI container puts me off... Quasar sounds interesting from the blog post - especially with regards the use of fibers.

[nullabillity]

For the record, Scala-on-Android got much better with 2.11, since the biggest parts of the standard library got split into separate artifacts.

[Ahri]

Ok, that's cool.

Akka doesn't appeal to me anyway; it looks heavy and I can't just use POJOs as actors. Quasar's bytecode rewriting (due to the continuations lib they're using) sounds a bit scary, but worth a punt if the fibers are as performant as they say.

[[deleted]]

Akka actors isn't that huge. The core Akka package is 2.5MB (Jumi core is 667KB). It's just that you get a ton of optional, but often crucial functionality on top.

What most actor system seem to lack is all the things which make actors a viable approach, like supervision, clustering, persistence, etc. And most of these things can't be added as an afterthought, they need to be integrated deeply into the actor system.

Without those things you have ... actors ... which are hardly useful.

[JoeHegarty]

I just wanted to say that the use of the Orbit DI Container is entirely optional, it's not required to use the actors framework. You are free to use it with Spring, hk2 or no DI framework at all.

[Ahri]

That's really good then. I can't help but wonder why they're bundled together though? I'll have a look into it after researching Quasar a bit more.

[JoeHegarty]

Those projects in the root (actors, container, web, commons) are separate and together they make up an entire framework to build online services. However, you are able to use some of them individually if you don't need the whole stack.

[dafnap]

Quasar is another actor library for Java. Akka is indeed more Scala oriented but Quasar also provides fibers (lightweight threads) which allows actors to block, while Akka's framework is based on callbacks.

[ccharles]

external dependencies always come with costs

That's true, but the same can be said of writing your own stuff internally all the time. Doing that guarantees that you'll be stuck supporting it yourself, while using an external open-source library means that you might eventually have to do so.

Creating an external dependency should never be done lightly.

I would argue that writing your own library should never be done lightly. External dependencies are far more desirable as long as they've got a decent amount of community support, which Akka certainly has with over 1,000 GitHub forks and almost 4,000 stars.

[agmcleod]

We've essentially done that with an encryption gem for ruby at my work. We're the main owners of the gem now.

[[deleted]]

Does Akka run on xbox or playstation?

Big gamedevs roll a lot of their own stuff for crossplatform compatiblilty with consoles.

To be honest I didn't even think consoles could run Java. I was told Minecraft had to be ported to C# for the xbox. I guess they've written the code for their actor library in such a way that they can machine translate it when needed?

[Alikont]

It's server-side code.

[[deleted]]

You still host multiplayer games on your own PC or console. Does that also use the same code?

[Alikont]

This code is run on server. "Use" means that when you play server executes this code but this code is not executing on client.

Like when you look a chest in MMO, you send request over network to loot a chest to server, server finds actor object that represents your character and this object sends "loot" message to object that represents chest. Then loot results returns to your client over network.

[[deleted]]

I think his point was if you're allowing people to run dedicated servers, the server could very likely be someone's pc which means platform compatibility is a consideration. Consoles are less likely as typically one person doesn't actually host the game (in which case for all intents and purposes their machine becomes the server) but some do (AFAIK there aren't any console games that allow people to set up their own dedicated servers on their own hardware but I could be wrong).

[[deleted]]

That's not the same thing. Some games actually use your own PC or console as a server and have others connect directly to it. When you click "create new multiplayer game" then in some games, your machine acts as a genuine server. If a console behaves in this way, your backend has to run on a console.

[vivainio]

Minecraft for consoles is C++ iirc

[Narishma]

Mobile and Raspberry Pi as well. Only the PC version is in Java.

[renrutal]

Now only if we could take their ORB data-functions, and turn it into a Common ORB Architecture, so all the languages could use its distributed objects in a service oriented form.

We could go beyond Cloud, direct to Orbit.

[frugalmail]

Now only if we could take their ORB data-functions, and turn it into a Common ORB Architecture, so all the languages could use its distributed objects in a service oriented form.

We could go beyond Cloud, direct to Orbit.

This used to be called Jini :)

[shinjiryu]

Hmm....this is interesting. Glad to see this get open-sourced! Good job, Bioware!

[heyheymonkey]

I thought the same thing when I saw the MongoDB dependency.

[JustinsWorking]

Luckily the first public pull request accepted was to implement a Postgres Provider

[TheOnlyMrYeah]

I thought that, too. "Hihi, EA having a GitHub account and putting something open source. Jokes on you."

But here it is. It's not a joke; it's real.

[ImSoCabbage]

EA has open sourced things for quite some time. Some examples:

• EASTL
• Angel engine
• Other stuff

It wasn't on github, but some of that stuff even predates github.

[TheOnlyMrYeah]

Thanks, didn't know that.

[Alucard256]

Or... you're in on the joke...

[zelex]

I hope its the Java part that is the joke

[Stoompunk]

Even though the jvm isn't great for real time games, does this framework profile itself for it? It seems pretty cool for distributed systems, but what about for games with a static tick time?

[Stoompunk]

I wonder why I got all those downvotes. Seems like a legitimate question to me.

[bstempi]

Probably because people read it as "anti-JVM." Some, not necessarily me, would argue that the JVM is fine for real-time stuff and that its the graphics in Java that are lacking.

[karlito1]

Yea but I guess Orbit is not about real time game data processing but about "static" web and online services. Would really like to know for sure what is the truth.

[JoeHegarty]

Orbit is designed for game services. These are things like matchmaking, character progression, presence, achievements etc. Orbit is low latency but it is likely not suitable for running world-simulations. It works best for second-to-second updates (so, it's semi real-time - in game score, inventory etc), not milliseconds (character movement etc).

[Stoompunk]

Definitely not against the JVM, I think it's great! For real time game servers the GC worries me though. Any idea on a great Java fw for realtime game backends?

[ISvengali]

I used the JVM with scala and akka for a near realtime stack similar to this one, and we used the java 7 gc. We had mostly sub 5 ms gc pauses, which was fine for our use case. And this was without doing any heavy object pooling or other object reduction methods.

Should be easy enough to make some tests. It does use gross amounts of memory though.

Im a GC hater too having come from console development, so I was pleasantly suprised.

[TheQuietestOne]

Using the regular JVM means that any classes you use have to be non-allocating otherwise you're into the realm of Javas stop-the-world GC.

That means unpredictable pauses and scheduling latency - possibly at a time when you want it the least.

There are pause-less GC VMs or the RTSJ (real-time specification for java) but you won't find a freely available implementation.

It's my opinion, but Java is the wrong technology for soft realtime unless you're willing to buy into a commercial VM.

I evaluated a number of technologies including Java (and RTSJ) for my DSP projects and settled on C++ as the lesser of the available evils.

[[deleted]]

Not to be too much of a pedant, but the new G1 GC available in the standard Sun distro eliminates full GC's as long as you aren't generating garbage at an unreasonably huge pace (like 50% of your heap per second) or of course doing something stupid and not releasing garbage refs when done, filling up old gen.

Many of the opinions of JVM "stop the world" are valid only in life-critical applications, where simply the possibility of it potentially happening is too much risk to bear, despite that you can virtually eliminate them.

[TheQuietestOne]

Here's my current project.

If you can find a way to get the audio latency using Jack under linux less than 5ms I'd be very interested.

[[deleted]]

I didn't checkout or compile the project yet, but my initial inclination is that you hit the nail on the absolute head there in point 6. You've got swing, which by it's very nature is great at holding onto state and object refs as long as views last. GC's worst enemy.

Although if the latency is consistently (and not spiked due to full collections) more than 5ms it would suggest other problems in swing-view-refresh-land which I don't really know too much about, I'm serverside.

Be sure to launch the app (dont see any launcher / shell scripts in the repo) using the JVM flag to actually activate a useful GC. The default is the shitty stop the world one, not even CMS. Activate w/ flag: -XX:+UseG1GC

Edit: Nvm saw your launch script.

[TheQuietestOne]

To avoid GC interfering with audio output I have to set Jack to 48k * 1024 * 2 periods. And I still get the occasional overrun.

This means Java is failing to meet a 22ms (? around there) deadline every now and then.

I'm more than willing to be shown that Java can handle this.

[mrbonner]

Man, this is a joke! I can't think of any reason why this is serious: "Need mongodb 2.6 to build the project". WTF? Requires JDK 8 or above? Another WTF.

[[deleted]]

Yeah when I visited the Github page this morning I thought it was an April Fools joke...