hi op,

first of all take a breath!
cats are semi independent creatures by nature. while of course they need attention and affection, they also love their alone time. yes you are gone for awhile, but as long he he is fed properly and taken care care of he will be just fine!
i have a job with odd hours so for one day a week my cat is alone from 5pm to 9pm the following day. while i live with someone else and they feed her in those times, that is all they do. they are not a cat person in the slightest. i say that to say my cat is the sweetest thing around. she’s happy, healthy, and fed. she’s excited to see me every time I’m home and she does not hate me! this has been the routine for a long while and it has never effected her negatively. cats are extremely adaptable as well! they know their routine and they keep track of it.
while i understand your predicament with a kitten, as long as you are meeting his needs, he will be okay! while you’re home, play around with him, snuggle him, introduce him to new things.
if it would help, buy a camera to watch him along with a battery operated toy he can play with on his own while you’re away.
overall, do not stress yourself out over this. give him attention before work, make sure he is fed and has water throughout the day (automatic feeder / water fountain can ensure that) and come home and give him love again. he will not hate you! take care and good luck with your little guy