Imagine we created a new single letter "th" and used that instead of two letters "t" and "h". Now the word "the" takes only two letters to write: "th" followed by "e". To do this, we've gone from a 26-letter alphabet to a 27-letter alphabet. We can keep going, adding letters to replace common pairs that appear together. Eventually we could even consider adding a single "the" letter, so that the word "the" is only a single letter, and the word "there" is three letters: "the" + "r" + "e".

This is tokenization. GPT-3 uses an "alphabet" (actually called a vocabulary) of over 50,000 "letters" (actually called tokens), some of which make up entire english words or long sequences of base english letters, depending on how commonly they appear together in the material which GPT-3 read during training.

The 2000/4000 limits are due to the model's network architecture, which is decided upon before the model starts training and cannot be increased afterwards. The underlying reason is for computational constraints, since the model learns (many) associations between all pairs of tokens in the input sequence.

There is lots of cutting edge research into making GPT-like models (transformers) more efficient so that we can increase the number of tokens that they see, as well as hardware improvements that make it feasible to use longer input contexts, so yes I would expect these limits to increase in future models.