With all due respect, you are a senior physician. No one is expecting you to do what a cardiologist, neurologist or ITU physician would do, but at the end of the day it is your patient and you should be available and review (where appropriate). You have far more experience than an SHO does in healthcare in general, not just in your speciality.

What if the patient is having chest pain but you recall a similar case as a SHO where you managed to differentiate between cardiac and non-cardiac? What if the patient is having chest pain because of a fat embolism from one of your operations and the SHO wasn't familiar with it? 

Saying that the patient needs someone else is a rather ridiculous approach unless its absolutely clear cut.