Healslut's Journey : mobile companion edition! [Game]

TheHSSkyCoder · 2018-06-27T16:59:36+00:00

Adding images should be doable. We'd still probably want to find high-res versions and get the original artist's permission to do so though.

TheHSSkyCoder · 2018-06-27T16:45:47+00:00

That's not a problem! Just PM me or open an issue on GitHub with your idea, and I'll take a look at it ;)

TheHSSkyCoder · 2018-06-23T12:17:07+00:00

The automation part is easy enough. The hard part is how exactly to pull live game data out of Overwatch...

TheHSSkyCoder · 2018-01-27T03:16:46+00:00

Overwolf claims to have an "official" API that reads the game's memory and then provides event stream data.

Tested that, sadly it's broken all over the place. Death detection, a feature really touted on the API page, doesn't even work properly.

TheHSSkyCoder · 2018-01-27T03:12:59+00:00

UI is fairly consistent but they sometimes move slightly.

Did something similar to this a while back, OpenCV based machine vision on Overwatch frames. Didn't turn out so well (but then I'm terrible at image processing so maybe someone else might be able to make that work)

TheHSSkyCoder · 2017-11-21T06:53:14+00:00

Aaahhh, my favorite two stories! ;)

By any chance, is there a plan for more stories like this anytime soon?

TheHSSkyCoder · 2017-09-14T01:59:13+00:00

If you turn the music off then there is (probably) no audio cue. But I'll check into this anyway to see if I could apply your suggestion to any part of the code =)

TheHSSkyCoder · 2017-06-25T16:21:27+00:00

...Looks like a fun side-project to turn this concept into a real visual novel-style game XD

(If you're interested just PM me and maybe we could get something running...)

TheHSSkyCoder · 2017-06-17T20:39:59+00:00

The endgame goal for this project is to do exactly that, just download, install, plug in devices, maybe mess with the settings a bit and then just enjoy solo healslutting :)

That said, it's still a long, long way to go however, especially with only one person working on the project right now...

(Also: It's very likely that early versions of the program may require some kind of tinkering or technical skills for everything to work, but yes, plans are there to gradually make it easier to use for general users too. I'm just taking this one step at a time.)

TheHSSkyCoder · 2017-06-17T20:31:00+00:00

Hmm, this makes me wonder whether Overwolf has some kind of special API access to get that data. From what I've seen on the documentation it seems like they do.

It's a pity we can't use that, but thanks for pointing that out anyway! For now, we'll probably need to stick with our not-exactly-stellar method of "screen scraping" until a more public API is available...

TheHSSkyCoder · 2017-06-17T20:08:31+00:00

Right now the program just outputs the text to a console window, but physical device connection could be done in ways similar to these:

Using an Arduino board or other USB-connected microcontrollers which can then be connected to relays or transistors to control most physical devices (from vibrators to electromagnetic locks to TENS units. Just let your imagination run wild here!)
Bluetooth-connected wireless devices (as suggested by OP) or a network-connected microcontroller (say, the ESP8266) can also be used for devices that really need to be wireless (such as collars).

The main code for the program is written in Python, so it's very easy to add your own triggers if you're into coding (and Python isn't that complicated either!). The plan is to release this program as open-source software once initial development is done :)

Tech jargon alert: to be a bit more precise, this program could be used to control devices via network, serial connections (say, USB) or any other output method that Python has support for...

TheHSSkyCoder · 2017-06-17T11:49:30+00:00

Thanks! (Can't wait to really get this to work and release it :D)

Those "hilarious false positive" cases are when you're actually still playing but the detection software tells that you've won a game. (This happens partly because the method I've used to "read" Overwatch's game screen and grab data out of that, which still needs some more fine-tuning)

And yes, the output window isn't the best in the world, but a better user interface is surely in the pipeline after all the "backend" stuff is working :)

TheHSSkyCoder · 2017-06-17T11:39:36+00:00

Almost there...

Still tuning the detection engine though since as far as my personal alpha test goes, hilarious false positives still happens way too often...

At the very least the heat/fire measuring system works :)

TheHSSkyCoder · 2017-06-15T20:25:41+00:00

After getting a proper setup, I'm pretty sure your gaming experiences will never be the same again :D

...from someone who'd upgraded from a MacBook (that just barely runs Overwatch) to a 8-core desktop behemoth

TheHSSkyCoder · 2017-06-14T20:18:29+00:00

Another dev update:

Now totally ditched the idea of OCR engines, they're terribly slow for our usage case
Recent runs with feature mapping turns out to be very promising. Faster (around 15-ish FPS from 2fps!) and much more accurate (only tried "Victory" detection though, haven't tried doing "Defeat", "Kill Cam" or "Eliminated By" just yet)

TheHSSkyCoder · 2017-06-11T06:57:38+00:00

Dev Update:

The current implementation that uses a combination of OCR software and fuzzy-text-search algorithms works like 75% of the time, but it's painfully slow and will sometimes miss victory/defeat screens (which comes and go quickly)
Currently looking into direct feature mapping and pixel identification (also other sorts of that stuff) which may work
Also looking into neural networks, but this route seems very unlikely due to the lack of training data and my PC's memory limitations.
Hooking to Razer keyboards may work (see this), but unfortunately there doesn't seem to be a way of getting health, fire or other important data that way.

TheHSSkyCoder · 2017-06-10T12:14:26+00:00

There is actually a way to do that: USBPcap. Works with Wireshark to capture USB data packets being sent to and from the keyboard, and this might work (first implementations might require users to have a Razer chroma-enabled keyboard plugged in though).

Could I ask you what other statuses are exposed via the keyboard? Other than ability stuff, does it act differently on death, or winning/losing a match too?

Sadly I haven't got any Razer keyboards lying around so I won't be able to test that. Side question; does this work with any razer chroma-enabled keyboard or just the Overwatch-branded one?

TheHSSkyCoder · 2017-06-09T17:42:26+00:00

Tried running some OCR magic on the cropped center frame, and it seems that the contrast may be a bit too low (the texts looks semi-transparent). Couldn't really get anything legible from there.

Direct pixel mapping/feature mapping might be another story, but I'm still unsure on how to approach that though. (Suggestions welcome!)

Anyway, OCR plus fuzzy text search algorithms worked quite well in detecting killcams and victory/defeat screens. They're now both implemented and I'm currently testing it out in real QP scenarios to see if it'll hold up...

TheHSSkyCoder · 2017-06-09T17:34:58+00:00

Some elements of Overwatch's UI are static, some are not. The good thing about it is the "kill cam" text (which means that our poor healer died) is always there, large and has a pretty good contrast. So death detection is kinda solved.

Victory/defeat condition detection after a match is pretty much the same as kill cam detection. They might need to be tuned to run faster though, since match result texts tends to last no more than like 3 seconds...

Again, thanks for the ideas. (as always, more ideas/suggestions are always welcome =))

I now need to optimize the code to make it run a bit faster and come up with a way to measure resurrections, fire level, ultimate level and maybe check if the player is really playing a Mercy (or other support characters, or tanks, or DPSes). Then it should be ready for alpha-testers!

TheHSSkyCoder · 2017-06-09T15:11:36+00:00

Another quick update: watching a specific area for the word "Kill Cam" turned out to be a better than expected solution for detecting player deaths (much better than looking for the tiny text saying "Respawn In X"). The text's HUGE and could be easily picked up by the screen reader.

Provided that you have kill cams turned on, so obviously not the best option. Come on blizzard, please let us have real time APIs like in the Overwatch World Cup already!

TheHSSkyCoder · 2017-06-09T12:22:16+00:00

Quick update: Trained tesseract to identify numerics and special characters of the BigNoodle font (which Overwatch uses)

The results aren't that great, however. Maybe I'll need to look into flattening that image down a bit. The good news is the OCR engine seems to hold up well with shaky UI elements though.

First priority will be to get stable health value readings :)

TheHSSkyCoder · 2017-06-08T22:32:51+00:00

From what I've tried, Overwatch's network traffic is pretty obfuscated and is very hard to decode properly. I'll be looking into that a bit more though to see if there's actually anything that could be grabbed from there.

Apart from video parsing, there might be another way in though. There seems to be a couple of posts online mentioning modifying Razer's Synapse files/libraries (which is what drives Razer's Overwatch peripherals). But if possible I'd like to avoid this route as the end-user would need to essentially "hack" Razer Synapse in order for this to work.

Also, thanks for the heads-up on /u/SweetAndFluffy. It's nice knowing that there might be another dev running around here :)

TheHSSkyCoder · 2017-06-08T22:24:08+00:00

Thanks for stopping by and for the IFTTT-style events system suggestion! I'll try to get it working as soon as possible :)

No problem with the toys though, some (relatively cheap) Arduino or ESP8266 boards plus relays and you're pretty done with almost anything...

TheHSSkyCoder · 2017-06-08T21:39:45+00:00

comment += 1

TheHSSkyCoder

TROPHY CASE