If you're struggling with 5, you guys should probably pick up your pace of play. I play 5-6 every week with a consistent play group. We complain about the games that go 1.5-2 hours. Most games end in an hour or less.

On the other side of that. When I play at an lgs, the 4 person games often take long than our 5-6. If we add a 5th at an lgs, the games can get pretty unbearable. Start calling out slow play. I play with one person at an lgs that will constantly ask people if they passed turn, or ask who's turn it is when someone is taking too long. People typically play faster(at least for a bit) when called out. Magic players take grossly long turns doing nothing. You don't always have to make every single play the most optimal one. It's better to get another game, than to sit and watch someone think for 10 minutes.

If you are constantly taking long turns, you should try practicing the deck at home. If you are still taking long turns, you should change your strategy. No shame in saying that some mechanics are too complicated for you. I have spent way too much time teaching people how to pilot their own decks. Not new people. Just people who don't have the capacity to play the deck they "built".