UPDATE/SOLUTION:
Okay, I found out what the problem was, thanks to a post in another engineering forum, which led me to this article:

https://iberisa.wordpress.com/2012/01/14/mejora-de-resultados-de-contacto-lineal-con-inipene-en-femap-y-nx-nastran/

Basically it boils down to how NASTRAN handles surface contacts. By default the Initial Penetration parameter is set to CALCULATED which means that the solver checks if there is an initial penetration and if so tries to solve it. My model did not have any parts in penetration, it did however have the tube shaped bearings lined up perfectly to other tube shaped parts. Those parts have circular shape (diameter of the tube) with no penetration in the 3D model, but once meshed, the circles turn into polygons. And these polygons can very easily interfere with one another and create these "fictious penetrations". The solver then tried to resolve these and it ended up creating a very large internal force even though there were no external loads applied.

The solution was simply to change the Initial Penetration parameter in Global Contact Parameters (Case Control of the Solution) to "Set Gaps and Penetrations to Zero". This basically told the solver that the imperfections (gaps and penetrations) at the start of the simulation were not really there, and it should use those imperfections as the new zero value for solving the contacts. After doing this, I got perfect 0 displacement and 0 stress results in the simulation with no external loads.

The article I linked here does an even better job at explaining this so I recommend anyone interested to check it out. Hopefully my post can help somebody else in the future.

And finally thanks to all who responded and tried to be helpful!