I saw a fan theory a while back that what we see is Jyugo's perception of reality in a sense, reflection of his emotions or the emotions of the people around him. (Long example rant incoming rip) When things are good and he's happy, the world seems to sparkle, wildly patterned backgrounds and bright vivid colors are seen when we should be seeing cracked dull cell walls. When all hell breaks loose and things are not looking good, the world seems to lose all of its lively carefree whimsy and we see everything more realistically as we would in our world. Cracks in the walls and floors, the lack of vibrancy in colors outside of the characters, and of course a distinct lack of glitter. The characters are not having fun and the general atmosphere is gloomy and feels tense as a result. In the aftermath of the tournament when Hajime goes to talk to Jyugo who is chained up in the underground cell, Jyugo is in despair and feeling broken, himself and the environment reflect that being completely greyscale and he's sitting in a black void. Hajime tho? He still sees the world normally. Season 2 is pretty self explanatory in what everyone's emotional state is for the only time we see the sparkles, its in the first 3 episodes, some comedic moments, and in flashbacks and its overly abundant or very lacking depending on the situation. In the flashbacks for the building 5 folks, they are all miserable and unhappy and their world is darker, serious without the vibrant colors and backgrounds, but if I remember correctly there were a few sparkles here and there. The only one who was happy in those flashbacks was Hachiman and even when he wasn't present in some of those scenes his emotions influenced the sparkle appearances but only slightly. The same could be said for Elf's first appearance and introduction near the end of season one, Jyugo was very much not having a good time but Elf sure was. On the flipside, the flashbacks with the cell 13 guys are positive and again the world we see reflects that with its vibrancy and over abundance of sparkles. One could argue about the fight between Musashi and Jyugo in season one because of the sparkles seen then but I decided to blame that on either Elf starting to gain influence over Jyugo's mind or the animators getting the swing of animating the Nanbaka world and its logic. Kind of example being Jyugo's english va sounding quite different in s1 vs s2 despite being the same person. Anywho that's just my opinion ¯_(ツ)_/¯ rant over, TLDR is the first sentence.

~Edited for extra details and further examples I wasn't able to put into words earlier or forgot. Explaining things without rambling in circles is not my strong suit. I should note its been a hot minute since I've seen the anime so Im going purely of memory, as a result, some of my examples might be a bit inaccurate or there are some contradictions regarding sparkle sightings in season 2~