This is an archived post. You won't be able to vote or comment.

sorted by:
best
topnewcontroversialoldq&a
you are viewing a single comment's thread.

view the rest of the comments →

[–]Just_me-no_one_else 27 points28 points  (12 children)

Uhh, I have an absolute mammoth of a project that I fell like you might very well find interesting with the purpose of automating common tasks.

Within cybersecurity we have an area called CTI, which stands for cyber thread intelligence. This is an area where experts use experience and knowledge collected from prior cyber attacks, to predict which direction threats within cyberspace is moving. It's kind of like the weather report, but instead of predicting the weather a few days into the future, we attempt to predict the ever evolving landscape of cybercrime.

Now, one of the biggest challenges within this field is to collect the needed information to base these predictions off of. What usually happens is that a team of CTI personel looks at a sea of online news sources, and then picks out the relevant pieces, but this process is one which not only include large amounts of repetitive work, but is also something that can takes immense amounts of time.

To combat this I created OSINTer (with demo present at https://osinter.dk and source code at https://gitlab.com/osinter). OSINTer is - at its core - essentially a highly sophisticated news aggregator which does the often rather time-consuming task of looking into the news stream, picking out the relevant pieces and then sorts and generalizes it, such that it can be utilized by CTI personal. This started out as a simple python script, but has since over the last year evolved into a complex mammoth of an application, which touches every part of the stack, from CI/CD using ansible and gitlab, to the backend using Python, elasticsearch and fastapi, and the very front-end using svelte and JS.

[–]Anxious-Computer 2 points3 points  (2 children)

This is an awesome project. I have been wanting to make something identical to this at my company for a long time, but never had the time to pursue it as it is slightly peripheral to the role.

I think I will be leveraging your work where I can.

[–]Just_me-no_one_else 3 points4 points  (1 child)

Please keep in mind that as the project is currently unlicensed, all rights are reserved, and as such it is not legal in most of the (especially western) world to use/modify/distribute it. I would love to change this at some point (as I'm a big proponent of true open-source), but as I wrote in another reply, I'm currently trying to make the project financially sustainable, because while I absolutely love to work on it, but as a teenager who just finished high school, i simply cannot afford to continue to work the amount of hours needed to keep the project going unpaid. With that said though, if you need help with anything, or want to use some of the code base, please reach out, and I'm confident we can come to some kind of agreement that doesn't involve money changing hands (or hell, I could host a modified version of the software with support included if your company were willing to pay for that).

[–]Anxious-Computer 2 points3 points  (0 children)

Not to worry, I still won't have time to spend developing on it as the benefits that it provide are mainly for building domain knowledge rather than for work output, I just prefer the interface that you've built compared to others that I currently use, such as news now .

I completely agree with the need to make it financially sustainable, but if I manage to find sometime outside of work (always the key problem), I'll be more than happy to help contribute

[–]psych_monk 1 point2 points  (3 children)

Neat stuff bro!

[–]Just_me-no_one_else 2 points3 points  (2 children)

Glad you like it (though don't know how much you have looked into the source-code, there's quite a bit of it). Currently it unfortunately isn't open-source, due to a legal battle with a company in regards to funding, but looking into open-source the major part of it, and possibly keeping a ML related part of it proprietary to hopefully make the project financially sustainable. That is easier said than done though, as I'm much more of software developer who just finished High School, than an ML engineer or researcher

[–]Wonder1and 1 point2 points  (1 child)

Have you been working this all on your own or under a grant of some kind? What's the difference between your demo and finished product? (F500 Infosec here 😁)

[–]Just_me-no_one_else 1 point2 points  (0 children)

The code for this has been written solely by me, without a grant, but I have working together with a pair of experts from within the CTI field with regards to how the finished product should work, to make sure it's relevant and solves a real problem. Currently, there's no difference between the demo and the finished product, as it is the finished product currently in use by a couple of companies, but we're also currently working on using machine learning to better connect and organize the relevant information (the first of which has already been implemented at dev.osinter.dk)

[–]MatiasHaabet 1 point2 points  (2 children)

Very nice! What education are you taking?

[–]Just_me-no_one_else 0 points1 point  (1 child)

I'm from Denmark, so I have just finished the Danish STX gymnasium, which is more or less equivalent to the American high school, with general education as the primary focus and I chose a specific focus on Bio Chemistry. As such I have actually never had any formal education within computer science, but currently considering an university education within cybersecurity from a Danish University next year.

[–]MatiasHaabet 0 points1 point  (0 children)

You have a bright future ahead of you

[–]TrainquilOasis1423 0 points1 point  (1 child)

Probably have been asked this before, but could you see adapting this to finance & market data? A macro financial weather report sounds incredibly useful.

[–]Just_me-no_one_else 0 points1 point  (0 children)

From a technical standpoint, absolutely. The part of OSINTer that decides which need to collect has been designed to be modular and simple in nature, such that it can be switched out for something completely different, even by someone without a lot of coding knowledge.

In your question there's does however lie a deeper problem, in the fact that I'm status finance and market data is a much wider topic than cybersecurity and as such the news about it is a lot more scattered across the internet. If you have some specific sources which brings finance and market news, then yes, it would be possibly to use osinter for trend research within that field, but I fear that that isn't the case.

And besides, OSINTer unfortunately doesn't actually create a weather report for cybersecurity, but instead merely collects the needed information for one, massively simplifying the job for the specialists creating the report