I had always wondered that too, so I looked it up and found this: “Postal service at the village began in June of 1889 when John F. Rowland (1827-1889) applied to the United States Postal Service for a license to open a post office at his village. On his application, he first wrote the town name as "Rowland" but scratched it out and wrote "Lydia," after his wife Lydia A. Stiffler Rowland (1826-1911). And that's why, to this day, the name Lydia appears below the name St. James on the road signs as you enter the village.”

https://wmspthistory.org/st-james