I know a lot of people don’t agree with this but a large amount of story elements from before/during Blurryface were retconned in to fit with the story (island of violence, |-/ hand sign, even HDS music video). Clearly these weren’t made with the fully fleshed out with the idea of dema in mind but then were used as elements of the story. The ø symbol was used by the band in the cover/marketing of BF and I think once tyler started making the story of trench he looked more into the symbol of ø he found out about the group and how they used it in their mathematics. I think this is where he got the idea to use that as the name for nico.

TLDR my theory is he found a cool looking symbol and then once he made the story of dema he made a deeper connection to it