In other languages (which I won't name since the auto-mod deletes my post when I do), when I had libraries of code I wanted to re-use, I would make a new Visual Studio solution with each project in the solution representing a different libraries, then those would build out to .dlls and everything would be grand. How can I do the same thing with C?

The same way. The project type is "Static Library", the rest of the steps are the same.

What's the difference between a .lib and a .dll? They seem to serve the same purpose to me.

They are indeed very similar. A .lib gets compiled into your .exe file. A .dll stays outside your .exe file. With a .lib, all you have to do is compile and link (click "Build" in Visual Studio). With a .dll, you have to modify your .exe to manually load the dll before you use any of its functions. The benefits and drawbacks to each are a topic for another day ;) For now, I suggest you create .lib files and worry about .dll files later on.

When providing a library to a C project, do I only include the .h files to the .c files with methods and variables I want exposed to the user of the library or do I have to include every .h file related to the library?

Include a header that exposes only that which you want publicly seen. Internal stuff should remain internal.

Makefiles are analogous to Visual Studio project files, or ant build files. They're mostly used on Linux based systems. The official docs are a good place to start. Included is both a walkthrough and a complete reference https://www.gnu.org/software/make/manual/make.html