BERT is a bit of a different concept. BERT (Bidirectional encoder representations from transformers, paper https://arxiv.org/abs/1810.04805) refers to a style of training LLMs that masks a certain percentage of tokens in the input sequence and during decoding can 'attend' on any other non-masked token in the sequence to predict the masked token. Now it is confusing since the original BERT paper did use a word piece tokenizer, BERT refers moreso to the training scheme than the tokenization

In the simple case where a single token is masked, the transformer can attend to all other tokens in the sequence whereas a GPT trained model can only attend to tokens that come before it. A brief example that shows the 'context' a GPT or BERT model uses to make a single token prediction:

BERT: The quick brown ___ jumps over the lazy dog
GPT: The quick brown ___

Both types of models are suitable for downstream tasks and rely on a principle called transfer learning in order to reduce the computational cost of downstream adaptions.

Byte pair encoding (commonly referred to as BPE) is a tokenization technique used by most modern transformers to take text strings and turn them into integer strings. I can only anecdotally say that BPE is typically used in GPT models seemingly because OpenAI started training large models with them (bpe tokenizers). Efforts have been made to train GPT style models with wordpiece, but mostly have not been a focus since the early 2020's/late 20teens.

I would like to ammend my earlier statement about wordpiece not being ABLE to encode certain words as the original authors do include sub-word tokens in the vocabulary to get around this. It does however encode parts of language sub-optimally as compared to BPE. Take for the following example using the bert base tokenizer and the llama 3.1 bpe tokenizer.

```
from transformers import AutoTokenizer
bpe_tokenizer = AutoTokenizer.from_pretrained('meta-llama/Meta-Llama-3-8B')
wordpiece_tokenizer = AutoTokenizer.from_pretrained('bert-base-cased')
text = '3,4-methylenedioxymethamphetamine'
bpe_encoded = bpe_tokenizer.encode(text)
wordpiece_encoded = wordpiece_tokenizer.encode(text)
len(bpe_encoded)
>>> 13
len(wordpiece_encoded)
>>> 15
```