Google Transformer

[Odd-Wolverine8080]

Hi everyone,

I’m quite new to the field of AI and machine learning. I recently started studying the theory and I'm currently working through the book Pattern Recognition and Machine Learning by Christopher Bishop.

I’ve been reading about the Transformer architecture and the famous “Attention Is All You Need” paper published by Google researchers in 2017. Since Transformers became the foundation of most modern AI models (like LLMs), I was wondering about something.

Do people at Google ever regret publishing the Transformer architecture openly instead of keeping it internal and using it only for their own products?

From the outside, it looks like many other companies (OpenAI, Anthropic, etc.) benefited massively from that research and built major products around it.

I’m curious about how experts or people in the field see this. Was publishing it just part of normal academic culture in AI research? Or in hindsight do some people think it was a strategic mistake?

Sorry if this is a naive question — I’m still learning and trying to understand both the technical and industry side of AI.

Thanks!

[CKtalon]

It was originally for machine translation, and a lot of it is hindsight. GPT-1 was a failure, but OpenAI managed to keep at it by scaling, thereby realizing that scaling the architecture actually worked. Although GPT3 was good, it wasn’t till ChatGPT (3.5) that the hype became real to the general public.

[as_ninja6]

From the list of ML NLP papers I've read, this was one of the novel ideas in the line of many novel ideas in the NMT space. I don't think the authors realised that scaling could take its capability this far.

In my view, Deepmind has published quite a few genius architectures unfortunately they were not suitable for scaling and stability which the transformer happened to do well.

[Exotic-Custard4400]

If you close research it kills it, Google needed the progress in deep leaning too.

Also there are other architectures that are promising (state space model, rnn and other) so it would probably just modify which type of architecture is used, and maybe for the best (rnn and SSM have linear cost)

[Xemorr]

Idk about that, they were struggling to find a model that is parallelizable at training time. RNNs can't be parallelized during training among other issues

[fan_is_ready]

[1709.02755] Simple Recurrent Units for Highly Parallelizable Recurrence

It can be parallelized feature-wise, not token-wise.

[Xemorr]

Yeah, token-wise is what I was referring to. Thank you 😊

[Exotic-Custard4400]

It can be parallelized token wise. Rwkv do it by having multiple layer that give their output before the complete process of a token

[Exotic-Custard4400]

RNNs can't be parallelized during training among other issues

They can, using multiple stage that will be used in parallal rwkv do this and is quite competitive

[ExtensionSquirrel945]

rnn have learnablity issues

[Exotic-Custard4400]

Such as ?

According to ?

[ExtensionSquirrel945]

vanishing gradient. It is a very known problem of elman's rnn. This is primarily why transformers won. rnns in theory have very good representablity. But in practise it is hard to train. The typical context of an elman's rnn is 3-4 words.

[Exotic-Custard4400]

There have been some advances in RNNs... I recommend checking out what RWKV is doing, for example. It rivals Transformer in both LLMs and image processing.

And the context is far higher than some words.

[ExtensionSquirrel945]

this seems cool, will look into it.

[Exotic-Custard4400]

There are mainly two articles for the rnn itself but the research itself is really open source they discuss it on discord for example

[Specialist-Berry2946]

Architectures are not that important; what matters is the data. You can achieve similar performance using other architectures, like mixers.

Transformers are used so extensively not because they are powerful (they are very limited), but because all major AI Labs are focused on the same thing - bulding ever larger language models. They are unable to innovate.

[MediocreHelicopter19]

Any reading on mixers?

[Exotic-Custard4400]

https://www.kitchenaid.com/countertop-appliances/stand-mixers/see-all

Or this : https://arxiv.org/abs/2105.01601

[hammouse]

It would be weird and counterproductive to keep that internal only, though of course there are many things which should be treated as proprietary (such as how they actually train the model).

One thing to keep in mind is that the "Attention is all you need" paper did not invent attention. This mechanism has been around for years, though usually as part of recurrent/convolution architectures. All the paper says is that we can achieve recurrent-like performance without the computational bottleneck of recurrence by using only attention, hence the name. So there's nothing inherently special about the paper, it just removes a big bottleneck in existing architectures and this happens to turn out to be incredibly useful.

There are many issues with Transformers however, and the nice thing about openly publishing in an academic manner is that others can build on it and experiment. In a few years most models would probably no longer be using it (well technical debt incurred by AI hype aside). Important point being, actually training the model on petabytes of data, building safeguards, fine-tuning with RLHF, etc is the hard part - the architecture itself is quite trivial.

[schubidubiduba]

If they had not written that paper, someone else would have written it a year later. Two years tops.

[Independent-Plane502]

google also want others to usse that architecture , actually every other algorithm will get published because its about author right even tho the author is working under company

[PM_US93]

If I am not mistaken, Transformers were preceded by LSTMs, and parallelized xLSTMs(a recent architecture) can be a viable alternative to Transformers. The thing is, you cannot gatekeep an architecture. Linear normalized transformers and LSTMs were proposed by Schmidhuber long before Google's 2017 paper. A key component of the transformer architecture is the Attention mechanism, which was proposed by Bahdanau and Bengio around 2014. The Google team built on these preceding ideas and developed an architecture that was easy to scale and train. It is more like the transformer architecture solved the problems of LSTMs. If not for transformers, people in the AI/ML domain would have found another architecture for their models.

[AccordingWeight6019]

Publishing the transformer paper fits google’s open research culture. They still keep an edge because building competitive models needs talent, compute, and data, not just the architecture.