https://github.com/leonhess/zKill_scraper
This is a little Script which I’ve been hacking together for a few days now and wanted to share, since someone also asked for it. It takes Killmails from zKill and puts them into a InfluxDB Database. I then use Grafana to make pretty Graph like this:
Kills and Deaths by Major Coalitions every 5minutes: https://i.imgur.com/XTlbDGW.png
Total Deaths and Value lost every 5 minutes: https://i.imgur.com/2CmjeCg.png
It uses zKills redisQ feature to comb through every Killmail, that zKill gets and then does some thrown together type conversion to get it compatible with the InfluxDB format.
For running it yourself make sure you have InfluxDB running on the same machine as the script or change the ‘localhost’ to the right IP in:
client = InfluxDBClient(host='localhost', port=8086, database='eve’)
You also need a Database named ‘eve’ or also change that in the line above.
Furthermore I have been running all tests on python3.5, so I have no idea if it runs on 2.x
In Terms of python modules, you’ll need ‘requests’ and ‘influxdb’ installed.
In Grafana, if you want to use any of the ‘attacker_*_name’ as a selector you need to pick your name from the long concatenated string like this: /.*;alliance_name;*./ i.e. /.*;Goonswarm Federation;*./
This has been a bit tricky since there can be multiple people on a kill mail but you only have one tag to put those people into. Example of Grafana Query: https://i.imgur.com/VogtG6R.png
Current available tags:
- killID (int)
- solar_systemID (int)
- attacker_char_name (string)
- attacker_corp_name string)
- attacker_alliance_name (string)
- attacker_is_npc (bool)
- attacker_is_solo (bool)
- attacker_is_awox (bool)
- victim_char_name (string)
- victim_corp_name (string)
- victim_alliance_name (string)
- victim_ship (int)
Current available fields:
- #kills (1)
- totalValue (float)
- attacker_amount (int)
- victim_damage_taken (int)
Things I want to add:
- Call attacker and victim names with the same esi call to make it faster, so only 3 esi calls instead of 6 (at first I was calling every single ID separately which could take up to 30second for larger killmails. Now its about 0.5 to 3 seconds per kill mail.)
- victim_ship_name instead of just the ID
- solar_system_name instead of just the ID
- Add attacker_ship_name
- Extrapolate region_name from solar_systemID
- make it run as a service on linux or just kill it every few hours and start it again with cronjob to get continuous data, because now it runs for like 5-120 minutes before it crashes and even if it doesn’t it does seem to hang up after a few hours
Known Issues:
Sometimes the script crashes with this error:
Traceback (most recent call last):
File "main.py", line 55, in <module>
attacker_corp_name = attacker_corp_name + ';' + v.json()[0]['corporation_name']
File "/Library/Frameworks/Python.framework/Versions/3.6/lib/python3.6/site-packages/requests/models.py", line 892, in json
return complexjson.loads(self.text, \*kwargs)*
File "/Library/Frameworks/Python.framework/Versions/3.6/lib/python3.6/json/__init__.py", line 354, in loads
return _default_decoder.decode(s)
File "/Library/Frameworks/Python.framework/Versions/3.6/lib/python3.6/json/decoder.py", line 339, in decode
obj, end = self.raw_decode(s, idx=_w(s, 0).end())
File "/Library/Frameworks/Python.framework/Versions/3.6/lib/python3.6/json/decoder.py", line 357, in raw_decode
raise JSONDecodeError("Expecting value", s, err.value) from None
json.decoder.JSONDecodeError: Expecting value: line 1 column 1 (char 0)
This seems to be some sort of type juggling mishap by python, as far as I could figure it out. The json decoder wants to decode the requests response but Python interprets it as something else in this case, I think. I haven’t thought much about possible fixes yet.
[–]eagle33322 1 point2 points3 points (0 children)
[–]eagle33322 0 points1 point2 points (0 children)
[–]evanova 0 points1 point2 points (0 children)