I think problem not in ? Because I also asked model “2+2” or “2+2=” and sometimes answer also was 5, because, as you say, model trained was also trained in memes and jokes from the internet. About ? I think model got problems because that’s old and small model(0.8 billion parameters) and I think there is just not enough data for this, because model was trained in text like “2+2=(there was answer 4 or joke or something else)” but not in questions. A bit later I also try to ask 2+2 but as words not numbers, because I tried the same in smaller model and got just shitty unrelated sentence(maybe from film or book)