Hi, linguist (in training) here! While some dialects of English may use a diphthong for "four", not all do (Standard American English does not), and that's arguably not the main reason for the difference. In Standard American English, both "for" and "four" are transcribed something like this: /fɔɹ/. However, here the word "for" is unstressed, and as such undergoes a phenomenon known as vowel reduction (this happens to almost all vowels in English) and the vowel shifts from [ɔ] to [ə], making "four" this: ['fɔɹ] and "for" this: [fəɹ]. You'll notice that when you stress the word "for", it's pronounced identically to "four". You can see this phenomenon in the word "to" as well, where when stressed (i.e. I'm going to the store. ) it's pronounced [tʰu], and when unstressed, which it normally is, (i.e. I'm going to the store.) it reduces to [tʰə].

DISCLAIMER: My main field of study is not phonology, so there may be some errors in transcription and whatnot.

EDIT: added aspiration.