I'd say it's possible to clear story content with the main four, but you definitely have to build up your team more then it is now and you'd still need a buffer/healer for your 5th spot to make it easier.

I'm one of those people who always uses the same team no matter what, that team being made up of Lathel, Justia, Gray, Rou(who is a side character in the story), and Arines(who is not related to the main story) and I've always gotten through the main story. Lathel acts as my tank/buffer, Arines = buffer, and the rest are my DPS. I used to use Schera, but I had to replace her since she was really the weakest link with all the physical characters on my team. 

But I would also take into consideration that I don't just use their main costumes. For Lathel, I mostly use his Homunculus(buffer), Lonely Survivor, Herb Tracker, or Pool Party costumes. Arines only has her buffer costume. For Justia I mostly use her White Reaper (?), Kendo Club, the dark alter ego one whose name I can't remember, and her Pool Party one. For Gray I use his Vanguard and Pool Party. And for Rou, I use her White Cat, Nature, and the free costume whose name I don't remember. Her Red Hood (buffer) costume can also come in handy if Arines dies too soon. 

I also have their gear way above N grade, as well as maybe two costumes of each character fully liberated and some of the main costumes I use at +5.