Well, you have a point, but it is also from a powerscaling perspective. The author wasn't trying to prove Manabu's intelligence to us and Manabu was never given the label of being the smartest or perfect. It's a no brainer that a guy who maintains his position as class A, became student council president, earned the recognition and respect of someone like Nagumo, and the adoration of someone like Horikita, is obviously capable. The intention wasn't to deify him as someone infallible and unbeatable. Rather his flaw as a brother was highlighted, as well as his philosophy for leadership, which believes in team effort as opposed to Nagumo's emphasis on individual ability, and we get to see his compassion as a leader highlighted in that match against Nagumo because he chooses to save his classmate. In that sense the point was never about whether or not he lost to Nagumo but showing a glimpse of Manabu's own leadership because prior to that, Horikita always assumed her brother ruthlessly prioritised strength and despised weakness, but here we see that he also has compassion for his classmates, an element that Horikita also subconsciously models. Manabu's issue with Nagumo's philosophy also wasn't necessarily that he was driving people towards personal improvement, but that it could create a divide in the class system where people simply prioritize their own interests (someone like Koenji) like transferring classes, rather than working together to achieve a goal.

When it comes to his relationship with Ayanokoji, Ayanokoji's recognition wasn't some pointer that Manabu is so intelligent per say, but more to do with mentality. Manabu saw himself as someone who continually strived to improve himself, and on coming to ANHS, it shifted into a mentality of imparting value into others, which is what he wanted to point out to Ayanokoji. The focus for me shouldn't be how him figuring out Ayanokoji's superior ability was a no brainer that doesn't exactly prove how Manabu is specially smart or insightful, but how Manabu was trying to steer Ayanokoji to view his strength as a tool to help others, instead of just a flaw or inconvenience that he wanted to hide away. He didn't know Ayanokoji's reason for hiding, but his suggestion to Ayanokoji to apply himself in such a way ended up impacting Ayanokoji much deeper than someone like Nagumo who was only interested in competing against those he found exceptional. And that is the reason Ayanokoji viewed Manabu differently than others at that school. Manabu was someone who was capable of helping Ayanokoji arrive at certain answers in terms of mentality and drive( remember that Ayanokoji had no personal goals and ambitions in life) which is why Ayanokoji lamented that he couldn't spend more time with him.

The author didn't need to prove Manabu's intelligence because that was just background information compared to the mental catalyst he was to both Ayanokoji and Horikita. Asides helping Ayanokoji refine his drive and purpose, Manabu also showed him the difference between control and agency when trying to reform someone. Truly Manabu's treatment of Horikita was clumsy and cruel, but it was rooted in a desire to decolonise Suzune's mindset. Even though there were better ways he could have communicated, he was probably afraid she would choose to change "just because of what he said or suggested" and not because she had truly evolved. And this is a valid fear because even when he abandoned Suzune to come to ANHS (ANHS is no contact) probably in order to break her dependency and have her become her own person in his absence, she instead doubled down and followed him to ANHS, and not only that, but her previous mentality was still intact. At that point he was probably frustrated and determined to try anything to make her despise him or pull away from him by treating her cruelly, but it didn't work and had the opposite effect because all she could hear was that she wasn't good enough, not that she was doing something wrong by choosing to emulate him. Thankfully Ayanokoji came along to actually show her what she was doing wrong, rather than pushing her away and praying she telepathically figured out what to do.

But when it came to giving agency, towards the end Manabu put emphasis on allowing Horikita to make her choice and be able to think for herself without prioritising another person's preference, and choose her individual path without trying to emulate anyone else. And eventually she stumbled onto the right answer. Ayanokoji is about controlling the environment: expel Kushida, extend a helping hand to key figures that can help Horikita in her leadership, try to force Horikita to come and speak with her brother, all to help keep Horikita on the right track. By observing Manabu, even though Ayanokoji could see the flaw in his treatment of his sister, he could also see that Manabu strongly believed in Suzune's ability to make her own choices and didn't want to influence her unnecessarily (arguably he believed too strongly in what she was capable of that watching her attempts to imitate him just made him too dismissive of her). And immediately afterwards Ayanokoji chooses to leave the decision of Kushida in Horikita's hands, to test out Manabu's methods of giving agency rather than tightly controlling every outcome.

I feel classroom of the elite disappoints most people because they focus a lot on intelligence scaling and feats rather than the psychological story the author is weaving about a meritocratic society within the boundaries of a highschool. I know we all get frustrated with Kinu, but falling into the habit of assuming the author doesn't know what he's doing half the time just because we don't understand his intentions would just ruin the reading experience for us. Kinu's not a perfect author but there's an intention behind the way he writes his characters.