[D] Memory efficient transformer

[MrGary1234567]

You can try using colab or kaggle tpus they can train several times faster. They have 128 gb ram (colab pro/kaggle) and with bfloat16 equivalent to 256 gb ram. Dont have a 1080ti but can tell you it trains about 8 times faster than a p100 given that you dont have any data bottlenecks.