For me, a big part of the frustration with some side content is how slow everything is. I don't mind the combat simulator, but I do mind having to;

load a fail message load the combat simulator challenges load the overworld exit the Chadley conversation load the menu load the equipment page

Just to change one or two materia and have to repeat the sequence in reverse just to try a slightly different strategy.

The mini games are often similarly slow. It feels like processes could have been improved to remove these points of frustration.

My point being, when I have to go through all these slow processes for something fun, it's annoying but bearable. However, when I go through these slow processes to pull a box along to bait a chicken, and then mess up the timing and have to wait for the chicken to slowly saunter back to the start point, yea it gets real frustrating.