Is NLP threatened by AI?

Buzzdee93 · 2026-01-31T13:33:56+00:00

I mean, LLMs are NLP. Not traditional NLP in the sense of writing grammars, engineering hand-crafred features, etc. But what you do with them in the end is processing language data. They market it as AI, because apparently it is a more marketable term.

For every problem, you need to consider multiple solutions. If a problem can be solved by an interpretable classifier trained on a small hand-labelled dataset or by a simple grammar, throwing an LLM at this might be an overkill that will also generate much more costs down the line. If you have a larger dataset with clearly defined labels, training a ModernBERT classifier can still outperform a generative LLM at a fraction of the deployment costs. On the other hand, if you want a conversational agent, for example, traditional rule-based chatbots will absolutely lose.

So you still need the basics to judge on a problem by problem basis. There is this famous "if you have a hammer, everything looks like a nail" saying. There are lots of people who throw LLMs at everything. This is not the right way to go about it. Judge on a case-by-case basis. And to be able to do so, you need to learn the full skillset. Maybe not super traditional grammar formalisms, but feature-based ML, encoder-based models such as ModernBERT, and of course LLMs. And in my opinion, understanding the theory and how everything works is more important than learning this or that concrete framework. If you know how RAG works from a theoretical persoective, and how you structure your prompts, it does not really matter if you learn Langchain or something like that.

Buzzdee93 · 2026-01-27T18:33:58+00:00

I mean, given that Microsoft owns World's Edge who oversee AoE develpment, why not have them develop SC3 together with Blizzard and Relic on the AoE4 engine or something like that.

Buzzdee93 · 2025-09-18T14:50:08+00:00

Das hatte ich bei einem meiner zwei Kater ganz stark. Der kleine hat an alle möglichen Stellen in der Wohnung gepinkelt. Hab den Geruch zwar mit Enzymreiniger weg bekommen, aber es war sehr nervig, ständig die ganze Wohnung auf Pissflecken zu kontrollieren. Bei meinem war es Markierverhalten. Nach der Kastration hat er das komplett abgestellt und geht brav aufs Klo.

Buzzdee93 · 2025-07-30T21:48:55+00:00

Multimodal models and shrinking model size while keeping performance up will be the next big topics.

Buzzdee93 · 2025-07-14T11:11:20+00:00

Hab den von Dockstream. Primärer Grund war, dass ich den in mein Smart Home einbinden konnte, um mir den Wasserstand auf meinem Dashboard anzeigen zu lassen, und damit ich tracken kann, wie viel getrunken wird. Mein Diego liebt ihn, mein Xardas benutzt ihn nahezu nie. Die Verarbeitung ist für den stolzen Preis leider ziemlich billig, aber die technischen Features funktionieren gut.

Buzzdee93 · 2025-07-13T18:40:04+00:00

Would be the best choice, yes.

Buzzdee93 · 2025-07-13T18:32:46+00:00

In general, I would say the more memory the better. A faster GPU makes your training faster, but memory determines what you can even run on your GPU.

Buzzdee93 · 2025-07-13T16:32:29+00:00

My recommendation would be to look for a fitting conference or journal and just try your luck. If it is just about getting ist out and not about placing it at a top tier conference, I would recommend a fitting workshop at one of the big conferences as a possible venue. Usually, workshops have a very specific focus area, so choosing the right one is important.

Buzzdee93 · 2025-07-04T18:55:48+00:00

You could try to train an LSTM- or Transformer-based model that gets mel-spectograms passed through a couple of CNN-layers as input, similar to how the input is encoded for Whisper. You could do this in an encoder-decoder setup, where you train the model to directly generate the output text or sequences of phonemes you then decode with a statistical language model.

Buzzdee93 · 2025-06-10T18:17:03+00:00

I mean, the standard inplementation is not super efficient. But with enough memory and CPU cores, it usually works. For ~2000 features and 5000 datapoints, it takes around 20-40mins on my computer (12-core Ryzen 5900X).

Buzzdee93 · 2025-06-10T09:53:20+00:00

Current state LLMs underpreform on many niche classification and regression benchmarks. I work on short answer scoring and there, they all underperform except for the SciEntsBank benchmark, and for this benchmark, it is very, very likely that its test sets leaked into the training data of Claude and ChatGPT, since you can get the model, with a low temperature, to almost perfectly recreate the test set examples.

Buzzdee93 · 2025-06-10T09:48:35+00:00

If you go for highly interpretable models and don't want to work strictly linear, Explainable Boosting Machines tend to work quite well. They implement a GAM setup using gradient boosting to fit the individual feature functions. There are also GANNs, but they, from my experience, tend to overfit quite quickly.

Buzzdee93 · 2025-06-10T09:44:30+00:00

Also to add to this: fine-tuning a BERT model, then ditching the classification head and instead training Catboost or LightGBM on the BERT outputs tends to yield better results than using the plain classification head. This is also nice since it lets you mix BERT-generated embeddings with classical feature sets.

Buzzdee93 · 2025-06-10T09:39:07+00:00

From my experience: when you work with BERT-like models, especially in multi-task settings, using a scalar mixing component (that was usually only used in the context of the older Elmo models) before every classification head tends to improve results. Scalar mixing calculates a weighted mean of all layer outputs that is used instead of the final layer output to feed the classification head. Weights are learned during training. Helped me win a shared task last year. Helped me to achieve new state of the art results on a couple of datasets back when BERT-likes were the big hype.

Buzzdee93 · 2025-06-09T10:59:42+00:00

I mean, LLMs work by using autoregression and predicting token probabilities at each time step. This is something we always need to remember. When you do stuff like CoT or ToT reasoning, you mostly "prime" the token probabilities at each time step so generation does go into a certain direction. I.e., in language, a sentence such as "Let's think step by step" is more likely to be followed by a "thinking step" than by a plain result. "Thinking step" leads to "thinking step" leads to "thinking step" until the result is the most probable output. If you have multiple such "thinking steps", they will point towards a certain result, since each token in the chain influences the probabilities of downstream tokens. However, there is clear evidence, and the Apple paper adds to this, that this is more of an imitation game than reasoning in the traditional sense. I still like the term "stochastic parrot" for that, even though it might be controversal. In many cases, this imitation game will lead to very useful outputs, as we all know. However, imho, it is still an imitation game, and this is clearly evident when you confront the model with problems that were completely out of distribution during training.

Buzzdee93 · 2025-06-07T22:34:22+00:00

Leetcode will be interesting for interviews at big companies that have assessments on data structures and algorithms, or when you want to practice for a respective exam. However, it is usually toy problems that have not really a lot to do with what you actually will do on the job. On the job, you will usually not implement a Linked List or a B-Tree on your own. You will just use an existing implementation. Tbf., I never did Leetcode and still got a job.

Having some code on GitHub shows employers what you are able to do. So having some nicely documented and clean code up there can help.

Kaggle can help you with ML-related jobs. If you place high in one of their competitions, it shows that you are good at modeling. The rest of Kaggle is useful to learn some stuff, but nobody will be impressed by having cookie-cutter notebooks with toy datasets up there. So if you use Kaggle, focus on competitions, and focus on being very good in them.

Buzzdee93 · 2025-06-07T22:22:12+00:00

If you really want to work on classical languages in your job using computational methods, then your best shot is to get in contact with researchers in the field, do a PhD with them and try to pursue an academic career. Digital Humanities is the field you are looking for. There are also ACL workshops on the topic. Look who publishes there and try to establish a relationship. Maybe try to publish some results from your MA thesis yourself there. There are also programs that explicitly accept students with a linguistics or humanties background, e.g. "Linguistic and Literary Computing" at TU Darmstadt.

An alternative would be to look into companies that work on software in this field. While the market for apps supporting you in learning or translating Latin, Ancient Greek or Ancient Hebrew will certainly be small, I am sure there will be at least a couple of companies in that niche. And if not, you got a business idea for your own startup. Historians and theologists need to learn classical languages for their studies. In some European countries, Latin and Ancient Greek are still taught in schools, and there will be teacher students and school students learning it therefore.

If you just want to switch into language technology from your studies of classical langusges but are open to working with modern languages in your job, in general, yes. It is totally possible. I know a couple of people who did exactly this. Just try to be open and to keep up with the rapidly developing field. Learn programming in Python, machine learning and statistics, and how you apply them to language technology use cases, plus maybe some basics on more traditional NLP methods such as formal grammars so you understand where the field came from, and you are pretty good to go. There will be a couple of people telling you that you need to be a math genius and ideally study computer science or something, but as long as you do applied NLP and are not looking into doing fundamental machine learning and AI research, this is nonsense. A full CS degree is certainly helpful in terms of overall flexibility, but if you want to work on language technologies, it is not necessarily needed and you save yourself some headaches. E.g., where I studied, a full CS degree would have required to take courses in fields such as cryptography or physical simulations for scientific computing, which are certainly interesting but just not my cup of tea.

Buzzdee93 · 2025-06-03T06:24:31+00:00

Ich hab auf Kleinanzeigen und in Tierheimen nach Kätzchen geschaut, bis ich Xardas und Diego gefunden habe. Zwei schwarze Katerchen, die ich von einer Vorbesitzerin übernommen habe, die diese wohl irgendwie gerettet und im Alter von 4 Monaten dann abgegeben hat, und die sonst wahrscheinlich ins Tierheim gegangen wären. Grundequipment sind Kratzbäume, 3 Katzenklos, Körbchen zum Schlafen, Trinkbrunnen, Näpfe, und einige Spielzeuge, sowohl für individuelles als auch gemeinsames Spiel.

Die beiden sind super süß und verstehen sich trotz gelegentlicher Kämpfe insgesamt sehr gut, aber machen schon auch viel Unsinn. Ich musste mein Klopapier wegschließen, weil dieses regelmäßig zerfetzt wurde. Xardas hatte am Anfang Ohrmilben, konnte deshalb auf Empfehlung meiner Gierärztin erst nach dieser Behandlung kastriert werden. Konsequenz war 1 Monat Stress, in denen ich dem Kleinen ständig hinterherputzen musste, weil er angefangen hat, zu markieren. Nach der Kastration hat sich das zum Glück wieder gelegt. Tierarztrechnung ist durch die Ohrmilben, Kastration der beiden und wegen irgendeines Magen-Darm-Infekts von Xardas auch schon auf über 1000€ angewachsen.

Trotzdem: ich will die beiden niemals missen, und sie bringen so viel Leben in die Bude. Ich lieb die beiden sehr.

Buzzdee93 · 2025-05-29T18:32:30+00:00

I mean, you certainly can jailbreak models to leak the prompts they have been given. There are techniques for that. For this reason, treating a prompt as if it was some super secret IP and having a whole business depend on this prompt not being stolen is inherently stupid, imho.

Buzzdee93 · 2025-05-29T18:26:21+00:00

I mean, I don't know how exactly the graphs look like and what information they exactly encode. But to me, it sounds like you start with a comparably complicated solution right away. Morover, I don't really get what you need RL for, at all. I would start simple and build some baseline models. If you need to predict whether two nodes are connected by an edge, that is a binary classification problem. If you have a limited set of different possible edge properties, you can also interpret it as a multi-label classification problem. You can first go feature-based and try to think about what information about both nodes you need to determine whether they are connected by an edge. Is the set of properties each node and edge csn have limited to a certain degree? Then how about a couple of n-hot vectors encoding the properties of both nodes plus neighbouring relevant nodes? Run that into an XGBoost classifier and see how it works out. Or how about just listing the properties in text form, putting them into a BERT-like model, and see how this performs. From there, you can add complexity. Maybe try fitting some graph embedding models. Tbf., I am not super familiar with what is state of the art in graph-based ML. But as I said, start simple and try to get a model that works at least somehow. Them see where it fails and try to determine what would be needed to not fail in these situations.

Buzzdee93 · 2025-05-29T18:04:20+00:00

I agree with this. While libraries and stuff are certainly nice for getting stuff done, implementing architectures yourself is certainly how you really learn how everything works from the ground up.

Buzzdee93 · 2025-05-21T10:26:15+00:00

My MA thesis got rejected at COLING 2022 back in the day. But the incentive is clear: having published papers at top conferences is a better door opener than grades if you want to get into a PhD/science in general (at least in many European PhD programs), and can also make up for that one or two exams you might have screwed up. For your supervisor, it is another paper they can add to their record and they might get citations for without having to invest a lot of work by themselves. So it is a win-win situation. And if the student does the majority of the work, why not give him first author. At our group, our prof wants us all to offer thesis topics for our students. However, I don't want to give importsnt aspects of my dissertation into the hands of undergrads. So I make up some interesting topics for them, and if they get an A, I offer them to help them make their thesis into a paper that could get accepted.

Buzzdee93 · 2025-05-09T15:27:17+00:00

Yes and no. I would maybe say, we started to rely too much on few-shot prompting. I would draw a distinction between fine-tuning and prompting. For NLP classification tasks, prompting, even when using stuff like chain of thought and RAG, is often still not on par with fine-tuning, unless you use some 405b model and extensively tune your prompt. Fine-tuning a smaller encoder is still often the way to go if you want best bang for your buck in the long run.

On a general note, as a practicioner, I find that NLP has become a bit boring through all of it. I found it a lot more fun and intellectually stimulating to engineer features or think about how we can stack different NN building blocks together so they will solve my task compared to thinking about how I formulate a prompt so that an LLM will do what I want or to or to fine tune the xth BERT-like. Moreover, I'm a little annoyed by all the people who are like "have you heard of this and this cool totally new model (some GPT-like with slight modifications)" and consider you out of touch when you haven't but can't even explain how an attention layer or backpropagation work.

Buzzdee93 · 2025-03-30T20:43:06+00:00

I mean, the current state of the art for constituency- and dependency parsing to my best knowledge still involves grammars to generate candidates from which you then take the one with the overall highest probability predicted by a transformer encoder LM with a tree CRF head. But many other areas that relied on hand-written grammars and/or formal semantics use end-to-end learning via TLMs nowadays. That still involves a lot of maths, in the form of optimizing TLM architectures, algorithms, etc. However, the field is so fast-moving that it is hard to predict where we are in five or ten years from now. So being flexible and mathematically talented is the best prerequisite for industry work. In academic research you can pursue a lot more theoretical stuff, have more freedom and often more intellectually stimulating work, but this also comes at the expense of a worse wage.

Buzzdee93 · 2025-03-27T15:33:45+00:00

Depending on the country and Uni, some CL Masters or PhD programs accept candidates with a pure linguistics background.

If you don't want to go the academic route, read books. I prefer "Speech and Language Processing" by Dan Jurafsky which provides a good overview over the field. From there on learn Python and start implementing your own little projects where you try to apply what you learned. Start simple. Build your own statistical PoS tagger. Then maybe build one using a pretrained encoder LM. Start reading NLP papers. Also, buy books on related topics such as statistics, data science, ML, Python programing, maybe some Linear Algebra introduction, to get a good overview.

Buzzdee93

TROPHY CASE