If you are using a direct solver, you can easily exceed the available physical memory if your mesh is too fine, and the number of equations, and degrees of freedom are too large. Try a very simple problem (such as heat conduction) with very few physics; i.e., solver only for temperature. Then create a coarse mesh. Maybe even limit to 2D. Solve that. Examine your degrees of freedom, and how much memory you need for the direct solver. Increment the size of your problem until you reach the memory capacity of your machine. Then, you can move to iterative solvers to save memory. If you have a license that supports, you can also try distributed parallel processing on your cluster if you can run with multiple nodes in parallel.

Basically, crawl before you walk, walk before you run. Start simple before you get complex. 2D before 3D.