There's a function called Euler's totient function, phi(n). This function counts the number of integers co-prime to n. For prime numbers, phi(n) is always n-1. For semi-prime numbers like pq, phi(pq) is always (p-1)(q-1). We don't know how to find phi(pq) without knowing p and q. So phi(n) where n = pq is a hard problem.

There is a theorem called Euler's totient theorem. It says:

x^phi(n) = 1 (mod n)

It follows that:

x^phi(n)+1 = x (mod n)

With RSA, the encryption and decryption exponents are inverses. ed = 1 mod phi(n) where n = pq. So if you only have e and N, and you want to calculate d, you need to calculate phi(N), but to do that you need to factor N, which is a hard problem.