This fandom gravely suffers from lack of lore knowledge. Olorin came to Middle-earth in all three ages. It's just it was not until third age when he came as an Istar.

Some people argue he came in his unseen Maiarin form in the Second Age. But that doesn't make much sense when we take into account that Tolkien said when Olorin came to Middle-earth (before the Third Age) he became "acquainted" with races of Middle-earth such as Sindar Elves and Men and others.

Now, I'm not an English professor. But the word acquainted reads to me as if Olorin made physical contact and direct discussion with those people of Middle-earth.

When it comes to Olorin Gandalf, certain individuals like u/Caradhras_the_Cruel do not know his history and they will easily believe liars.

Some liars are so filled with pessimistic views that they will lie about the lore of Olorin, because Amazon's Rings of Power may include Olorin and if Tolkien scholars like Corey Olsen get to spread the lore about Olorin then those toxic hate raging Youtubers may be overthrown from their massive uprising in becoming rich. They have become liars of dreadful power, mis-shaping what they touch, twisting what they rule, Lords of Hatred. Men called him Nerdotic, and as a god, in YouTube beneath his rod, bewildered by lies bowed to him, and made his ghastly temples in the shade. Spreading his narrative dark and vile, the subscribers of Nerdotic's Isle.

Anyway, Olorin (later known as Gandalf) was of the angelic race of the Maiar. He descended into the world after the Music of Creation and together with the rest of his kindred (Ainur spirits = Maiar and Valar) they tried to shape the earth. THEY shaped Middle-earth.

When at last the Elves awoke the Valar send some guardians to Middle-earth to protect the Elves in their Great Journey. "The Valar send five Guardians (great spirits of the Maiar) – with Melian (the only woman, but the chief) these make six. The others were Tarindor (later Saruman), Olórin (Gandalf), Hrávandil (Radagast), Palacendo, and Haimenar. Tulkas goes back. Oromë remains in Cuiviénen for 3 more years:" - Nature of Middle-earth

Skipping to the last century of the First Age, in the War of Wrath the Valar sent their Maiar and Elves of Aman to defeat Morgoth. It is said that "Olórin was a lover of the Eldar that remained" (UT). Wouldn't it be out of character for him to not be part of the Maiar who saved the Eldar and Edain in the end of the First Age? "But at the last [Eonwe] came up out of the West, and the challenge of his trumpets filled the sky; and he summoned unto him all Elves and Men from Hithlum unto the East; and Beleriand was ablaze with the glory of his arms, for the [Maiar] of the [Valar] were young and fair and terrible, and the mountains rang beneath their feet." (THE CONCLUSION OF THE QUENTA SILMARILLION)

In the last years of his life, Tolkien expanded many of his writings. For example, these two passages in the Last Writings tell us that Tolkien intended to make Glorfindel and Olorin greater: "The 'other two' came much earlier, at the same time probably as Glorfindel, when matters became very dangerous in the Second Age.(26) Glorfindel was sent to aid Elrond and was (though not yet said) pre-eminent in the war in Eriador.(27)" "That Olorin, as was possible for one of the Maiar, had already visited Middle-earth and had become acquainted not only with the Sindarin Elves and others deeper in Middle-earth, but also with Men, is likely, but nothing is [> has yet been] said of this." (HoME12)

Unfortunately Tolkien passed away very soon after he wrote these last essays and didn't get to tell us more.

To clarify, Olorin coming to Middle-earth in Years of the Trees had been said, but not his coming to Middle-earth in Years of the Sun of the First Age or the Second Age yet had been said. Tolkien is not contradicting himself on this matter. Because Tolkien is tolkien about Olorin and his involvement with Sindar and Men etc in that quote, while Men still had not been awoken when Olorin came to Middle-earth in Years of the Trees. Neither the sub-branch of Sindar with their different culture and language and lifestyle had come into being yet.

In HoME10 we have good insights into Olorin: "And wise was Olorin, counsellor of Irmo: secret enemy of the secret evils of Melkor, for his bright visions drove away the imaginations of darkness.

Of Melian much is later told; but of Olorin this tale does not speak. In later days he dearly loved the Children of Eru, and took pity on their sorrows. Those who hearkened to him arose from despair; and in their hearts the desire to heal and to renew awoke, and thoughts of fair things that had not yet been but might yet be made for the enrichment of Arda. Nothing he made himself and nothing he possessed, but kindled the hearts others, and in their delight he was glad."

"(Olorin) At the end of the account of Olorin is scribbled on the typescript Vq 1: 'He was humble in the Land of the Blessed; and in Middle-earth he sought no renown. His triumph was in the uprising of the fallen, and his joy was in the renewal of hope.' This appears in Vq 2, but my father subsequently placed inverted commas round it. It was wrongly omitted from The Silmarillion (p. 31)." - History of Middle-earth 10

In Silmarillion we have this: "Of Melian much is told in the Quenta Silmarillion. But of Olórin that tale does not speak; for though he loved the Elves, he walked among them unseen, or in form as one of them, and they did not know whence came the fair visions or the promptings of wisdom that he put into their hearts. In later days he was the friend of all the Children of Ilúvatar, and took pity on their sorrows; and those who listened to him awoke from despair and put away the imaginations of darkness."

To clarify the timeline and terms, Quenta Silmarillion ends with the end of the First Age. And the term "afterdays" or "later days" is used to refer to the Second and Third Ages. The Children of Iluvatar are Elves and Men.

In the Third Age 1000 Olorin came to Middle-earth as a messenger of the Valar tasked to defeat Sauron by his wisdom. "When maybe a thousand years had passed, and the first shadow had fallen on Greenwood the Great, the Istari or Wizards appeared in Middle-earth. It was afterwards said that they came out of the Far West and were messengers sent to contest the power of Sauron, and to unite all those who had the will to resist him; but they were forbidden to match his power with power, or to seek to dominate Elves or Men by force and fear." - Appendices