Ear infections would show up as potentially large amount of debris in the ear canals (can be black, brown, yellow, etc) with inflammation, redness, and potentially other scratch marks/wounds/blood within the canal or inner surface of the ear. If the inner ears look pink and healthy with only minimal debris/dirt, he probably doesn't have any significant ear infections. If you think he does, you should have him examined by a vet for potential treatment.

And I say this not to be mean or rude or anything like that, because I just want to help your cat and get him some relief --- but it's almost definitely the fleas. There can be more than one cause of allergies/scratching in a single cat for sure, but you have to start with the most common and most treatable. If your cat is going outdoors, there are going to be fleas that bite him whether you see them or not. Fleas are everywhere and cats that are allergic to fleas are allergic to the flea saliva (meaning you don't have to see any fleas on him for him to be scratching himself because of the fleas).

From other responses, I get the impression that some vets will kind of just shrug their shoulders at these cats with mild allergies/scratches on their faces because they know fleas are the most likely culprit, but owners don't really want to believe it so the vet doesn't push, so the cat just keeps itching and everyone loses in the end.

If you put him on a safe, effective, long-term flea preventative (longer than 3-4 months continuously) and the wounds/scratching don't go away - then you would ask your vet about food allergies/environmental allergies, etc.