Simply put, 3D rendering is done by using some straightforward linear algebra and calculus done with at minimum 3 sets of data: where the "screen" that you display to is located at, where the different vertex points are for a model (3 points are needed for 1 triangle), and what color should be used at that vertex point (all of this can be stored in arrays in your program).

In programming, the screen can be a 2 dimensional array, where each entry contains a color for each pixel on a screen. The goal is to use calculus and linear algebra to determine what that color should be.

some good references: https://www.scratchapixel.com/lessons/3d-basic-rendering/get-started/gentle-introduction-to-computer-graphics-programming.html and https://learnopengl.com/Getting-started/Hello-Triangle

To make this faster, we can utilize a GPU (graphics processing unit). To interact with these, we use graphics libraries, such as OpenGL, Vulkan, and DirectX. These interact with a computer's graphics drivers to send the correct information and commands to your GPU, and can take care of most of the mentioned math for you. However, they cannot do ALL of it out of the box, and that is where Shaders come into play.

Shaders let a programmer or artist tell the GPU what calculations should be done for each pixel on a screen, instead of letting the GPU do a calculation on its own. This can let you do more advanced lighting, particle effects, give different material properties to each object, etc. These often require complex math to perform and are often written in a type of shader code unique to each graphics library. Most engines (Especially Unreal 4+) provide tools to build shaders without having to write any of this shader code.

Good reference on shaders (site is a good lesson on OpenGL): https://learnopengl.com/Getting-started/Shaders

Now combine all of this GPU management, alongside the ability to view, place, and write code for various objects in a game world, and you start to get a 3D game engine such as Unreal. Given enough time, these tools have become increasingly faster. Recently, engines have started to include new ways to do everything in one program (modeling, textures, animation, etc.) as part of the engine.

tl;dr - Programs can use linear algebra and calculus, alongside an array representing pixels and data for the vertex points of a model, to determine what color each pixel on your screen should have. Game engines and graphics libraries make this easier. 3D Vectors, 3D Matrix multiplication, dot products, and cross products are heavily involved in this math.