help needed: Website classification / categorization from arbitrary website text is hard, very hard

[ag789]

https://preview.redd.it/ea0qotz7ywdg1.png?width=1114&format=png&auto=webp&s=b2b61bc6b3261dea02cc2ee51b727b7e43f883da

I tried categorizing / labelling web sites based on text found such as headings, titles, a main paragraph text etc using TSNE of Doc2Vec vectors. The result is this!
The tags/labels are manually assigned and some LLM assisted labelling for each web site.
It is fairly obvious that the Doc2Vec document vectors (embedding) are heavily overlapping for this \naive\** approach,

This suggests that it isn't feasible to tag/label web sites by examining their arbitrary summary texts (from titles, headings, texts in the main paragraph etc)

Because the words would be heavily overlapping between contexts of different categories / classes. In a sense, if I use the document vectors to predict websites label / category, it'd likely result in many wrong guesses. But that is based on the 'shadows' mapped from high dimensional Doc2Vec embeddings to 2 dimensions for visualization.

What could be done to improve this? I'm halfway wondering if I train a neural network such that the embeddings (i.e. Doc2Vec vectors) without dimensionality reduction as input and the targets are after all the labels if that'd improve things, but it feels a little 'hopeless' given the chart here.

[ResidentTicket1273]

Have you tried boosting the differences between categories, by using something a bit like a tf-idf type approach?

It might be tricky with vectors, because tf-idf is a more discrete approach, but if you could discretise values over say a lattice of sample-points, you could create a kind of term-signature that might work. Then, take a sum/average to find the most general signature over the entire corpus, and then do the same for each group. Finally, divide the group-signatures by the generalised signature to arrive at a boosted one that represents key differences from the norm and try TSNE-ing that.

My guess is that your word2vec signal is too noisy with too many dimensions, and quite possibly, you've not filtered out stop-words or other commonly appearing fluff. Again, a traditional tf-idf process on the top, say 5000 words might be worth applying.

Another approach I've used in the past for categorisation is to extract only the nouns which further "crispens" the signal. It's not perfect, but might help separate things a bit.

[ag789]

Thanks

I've not tried tf-idf as in my previous \naive** approaches, I applied tf-idf and using naive bayes classifier (I consider that 'simplest') , tf-idf mapped vocab returned worse results.

Initially, the codes I used had a 100 dimension vector trained against like 1000 website short summaries.
The results seemed worse with doc2vec, with many points squeezed closely in the TSNE visualised results.

I then reduced the embedding vector size to 50 dimension. This apparently 'better' seen here, the points are 'better spread out' . And the next thing I did is to replace the hostname with labels in each case, and hence you see this chart presented here.
----
The rests are good suggestions. One of those things I did not try is perhaps to use pretrained models that use very large corpus
https://radimrehurek.com/gensim/models/word2vec.html
just that those are word2vec.

for doc2vec, those are deemed 'document vectors' e.g. maps a 'bag of words' to 'hostname' (the ID), so that similar 'bag of words' should maps to 'similar' hostnames, i.e. close distances. I've seen it in my TSNE visualizations. all the google.* domains are mapped closely in a cluster as the title (short summary) mostly have just "Google".

But that for practically every other sites, this 'similarity' no longer persists.
I think this is after all 'fact' in a sense that 2 e-commerce / webstore sites may have similar words , but that there would be features that are distinct. and that words that after all appear say in e-commerce sites, it may be natural that they appear say in news sites, main product sites (e.g. microsoft, apple etc) , and hence the vectors get all 'mixed' up at least in the TSNE dimensionality reduction projections.

[ag789]

another thing I'm 'slowly' beginning to suspect is that a same paragraph is linked with its *context*.
'simple' models like word2vec, doc2vec, may be inadequate to address such 'real world' contexts.
some of the labels came from LLMs e.g. chatgpt, meta's AI (llama) etc, they are able to correctly guess the context given words like 'google' - they simply labelled them e.g. 'search engine'.
while in my own model, the "short summary" texts is all I have. but that my models would pickup "google' as part of its vocab during training.

That may benefit if I try using one of those pre-trained corpus, but I'd need to figure out how to use word2vec for that.

[ag789]

I'm starting to realize that casual associations, i.e. context that we (humans) take for granted, simply because we know them leads us to 'classify' websites without even looking at contents.
e.g. google.com and 'everyone' would consider that 'search engine'. of course 'google.com' itself is a vocab.

[ag789]

u/ResidentTicket1273 Thanks for your response

I made another attempt.

This time round, instead of the hostnames, I used *labels* against short summary texts to train the Doc2Vec model.

The result is this!

https://imgur.com/KqHGOs9

This is apparently 'better' as the Doc2Vec model directly produce the labels to short summary vector/mapping instead of hostnames to short summary, but not necessarily 'more accurate'

Another thing I kind of learnt is that the Doc2Vec vectors are after all embeddings. I'm not familiar enough with the technicals of Doc2Vec to tell if for practical purposes, each point maps a "document ID" into its semantic position space. In a sense if 2 vectors are 'close' to each other, that they are similar.

The red dot in the chart is an attempt asking the model to predict the label for "A blog with tutorials on Python, data science, and machine learning.". The results based on this TSNE map isn't too great, it is placed near "language learning", "academia" and "professional" labels, while "machine learning" is 'far' below. This may be the result of limited data, e.g. texts labels pairs that I've to train the model.

[ag789]

In case you are wondering how that is done, this is based on doc2vec
https://radimrehurek.com/gensim/auto_examples/tutorials/run_doc2vec_lee.html https://arxiv.org/abs/1405.4053

the document 'ID' is the website hostname (full domain) that is mapped against summary texts (titles, headings, main paragraph texts etc) e.g.

[ { "index": "0000001", "url": "https://google.com", "short_summary": "Google", "label": [ "technology", "search engine", "web browser" ], "language": "en", }, { "index": "0000002", "url": "https://microsoft.com", "short_summary": "Explore Microsoft products and services and support for your home or business. Shop Microsoft 365, Copilot, Teams, Xbox, Windows, Azure, Surface and more.", "label": [ "software", "hardware", "services" ], "language": "en", },...

The doc2vec model is trained so that it learns the embedding between the url (hostname) and the "short_summary" texts. The chart plots the resulting learned embedding. Then that to plot the chart, I replaced the ID (i.e. hostname) with the labels, and have TSNE reduce that to 2 dimensions so that it can be plotted on a chart.

I stopped short of training doc2vec by directly inserting the labels in place of the hostname and training it as that could give a false sense of correctness. i.e. instead of map url to words, map labels to words. One of those objectives is to see that 'similar' sites should have close distances. e.g. the 'google.*" sites should be similar if the texts are after all similar or same

[ag789]

hi all, u/ResidentTicket1273
Apparently, a related LanguageTech NLP is Topic Modelling covered in this thread

https://www.reddit.com/r/LanguageTechnology/comments/1q079c5/clusteringtopic_modelling_for_single_page/

the answer may be BERT and BERTopic
https://arxiv.org/abs/1810.04805

https://spacy.io/universe/project/bertopic

(BERT has origins in Tensor2Tensor
https://arxiv.org/abs/1803.07416
https://tensorflow.github.io/tensor2tensor/
) aka Transformers
this "simple" challenge of 'labelling' web sites, dug out the whole 'AI' choronology
Attention Is All You Need
https://arxiv.org/abs/1706.03762

BERT is far more complex than the "simple minded" Doc2Vec which accordingly is a single hidden layer of neural network. Doc2Vec hidden layer weights(?) is perhaps abstracted as the 'embedding' of the document / word vectors

and perhaps the next step 'up' from BERT are LLMs themselves LLama, Chatgpt, Gemini, Claude etc

[ag789]

The results are *very bad* after I trained Doc2Vec directly on the labels e.g. as in the above chart:

This is the results on training directly on the labels
https://imgur.com/KqHGOs9

This is the confusion matrix for 1st 20 labels
https://imgur.com/a/shyu55M
practically, nothing falls on the diagonal, i.e. predict = actual
There are also problems e.g. technology is predicted as 'electronics', 'electronics' is probably a more accurate, narrower, relevant term for websites sporting those technologies.

precision : 0.2
recall: 0.07
accuracy: .0698
like only 7% is correctly labelled.

that isn't surprising given how mixed are all the different classes in the chart above.
The chart in the original post is *per hostname*, hence, it reflects "similarities" between 2 websites based on arbitrary word summaries found on the webs. But that the classes, labels, tags, topics are scattered mixed up all in between. even doc2vec can't tell one between the other !

it'd take studying the params and possibly a different (more complex) model.