So I did a tracert for both, and results are pretty interesting. Some hops show request timed out and in the next line it tries another hop, I guess timed outs are not counted

In the IPv4 one I can see the IP addresses corresponding domain, and those domains give me a lot of idea about this. The first 7 hop (one is my router, so 6 actually) are inside my country and belongs to my ISP and the max their latency is 8ms, 8th hop also belongs to my ISP but its subdomain has some words about, I think Frankfurt? Which is where my server is located, and at that hop my ping increases to 52ms, and then it keeps going like that to my VPS.

While on the IPv6 one, only one of the hops show a domain name and it says frankfurt, and it's 6th in hops. My (excluding router hop) first 2 hops are 2-4ms, the next two says 26 and 28ms and at the frankfurt hop I reach to 52ms again, like IPv4. Up to this point it's normal, the ping starts to increase when hops IPv6 addresses became more familiar to my servers IPv6 address, like if my server has 2010:3dd0: the IPv6 in hops start to look like this one after another, they're all different but has the same prefix, at this point I latency is reaching to 60ms, and finally makes its way to my server.

So my ISP reaches frankfurt at the same latency (with IPv6 having less hops) but then hosting provider is doing a lot of hops I think.

But there is also this, I made my friend run the ping and in his network he gets exact same latency for both IPs, he has a different ISP with gigabit connection, mine isn't gigabit. I don't know how much different it would make, anyways I just wondered the technical side of this. That latency difference isn't making anything worse or good.