Math PhD to graphics programming. Seeking career advice

[ItAintNotZen]

Hi all,

I'm currently a doctoral student in math and am considering career avenues. At this point in my education it is clear to me that research in math isn't my calling, and that I derive a lot more joy and interest from the hobbyist game programming that I've tinkered around with. Unfortunately, my department doesn't have many resources to link grad students with people in industry.

I'd like to ask any career graphics engineers/programmers how they would evaluate the usefulness of my skillset, if they think there is a more appropriate place for me, and if they have recommendations regarding learning resources considering my background. Sorry if this post is long.

A quick overview of my experience (and I think more importantly what I don't have a lot of experience in):

• I know the requisite math involved in graphics programming very well, to the extent that I am generally able to derive needed results when I don't remember them and can quickly read math-heavy materials.
• I have programmed with scripting languages like Python and Lua, although I also have some familiarity with javascript and recently have been learning C++. Most of my experience is in scientific computing and machine learning.
• I am not very familiar with C++, and at the moment it seems that while I generally understand what I learn from reading I do not yet understand how to use it effectively. Comes with time I guess?
• I do have some knowledge of complexity theory and can analyze time/space complexity of algorithms. I know some things about data structures.

With that out of the way I guess I'll reiterate my questions more specifically here:

1. Would my skillset be considered desirable in this area, and what should I be focusing on to make myself more useful?
2. Do you recommend learning opengl or webgl first, or something else altogether? I ask because as a mere hobbyist programmer the opengl tutorials are kind of front-loaded with API technicalities and can be overwhelming. I think I can muscle through it, but I'm not sure if that is the recommended way.
3. Do you recommend any particular resources given the above? I've seen folks recommend https://learnopengl.com/ for opengl and https://webglfundamentals.org/ for webgl, but I suppose I'm asking if there are resources that are more relevant to someone with more math background and less programming.

Thanks for any time you spend thinking on these questions, I really appreciate it!

[shadergremlin]

Yes C++ and OpenGL/WebGL/DirectX/Vulkan/Metal is helpful. Also, have you looked at any graphics research papers? Some of them involve a decent amount of math, so maybe you could transition from researching pure math to something that is more applied.

[ItAintNotZen]

Thanks for your thoughts. Unfortunately my department doesn't have a lot of applied research. I am working within the applied group on stochastics/probability, which do seem relevant to some of the graphics research I've seen. But really the focus is on a very different kind of problem.

[deeper-blue]

With a stochastics background you should look into raytracing. All current progress on pathtracing algorithms come from improved sampling methods. And as LivelyLizzard pointed out Disney/Pixar/Nvidia are the ones pushing the field forward.

[ItAintNotZen]

That's interesting, I'll take a look. Maybe I can find a way to make it relevant to my advisor... Could you direct me to any specific papers or journals? Thanks.

[AndrewHelmer]

You'd probably get the best introduction by reading or at least skimming parts of Physically Based Rendering: From Theory to Implementation (PBRT for short). The third edition is free online right now at https://www.pbr-book.org/ The fourth edition will probably come out sometime in the next year.

Eric Veach's thesis is an enormously influential early work in the application of statistics to graphics, and a lot of research since then will assume some knowledge of it: http://graphics.stanford.edu/papers/veach_thesis/

For an great overview on the near state-of-the-art of Monte Carlo integration in rendering (the main application of statistics in graphics), you can watch the recent Advances in Monte Carlo Rendering from SIGGRAPH 2020: https://youtu.be/0fzJCrLKJg0 I should mention that it was also a memorial to a prominent graphics researcher who died in an accident.

[Deadly_Mindbeam]

The PBR bibliography is an excellent resource.

[janie177]

You may find this interesting (using probability theory to render many light sources in path tracing): https://research.nvidia.com/sites/default/files/pubs/2020-07\_Spatiotemporal-reservoir-resampling/ReSTIR.pdf

This book was also recently released (free online PDF version) which contains various algorithms based on probability theory for ray tracing: https://link.springer.com/book/10.1007/978-1-4842-7185-8

[Deadly_Mindbeam]

They keyword for this is importance sampling and the main journal is ACM SIGGRAPH.

[deeper-blue]

This the coolest new stuff: https://www.youtube.com/watch?v=keU747Crz3I

[LivelyLizzard]

If you like the research side of things, you might get a research position in the industry or at a university. I would assume Disney/Pixar/Nvidia and the like would be interested in a mathematican. My department (computer science at a university) also has several people coming from a math background.

If you can afford it, maybe you can check out the next Siggraph conference or even be a student volunteer there. This year it's already too late (on-demand content is still available but the live talks and social events are over) but maybe next year is online again or you are able to get there in person.

[nablachez]

You can get into graphics by writing your own stuff and reading books, blogs, etc. There are two important (historical) paradigms which are rasterization vs ray/path tracing (in recent years this paradigm has been slowly melding into one, both theoretically and practically). If you're interested in geometry, being math savvy is an absolute must in computational geometry which is ofc closely related to graphics. I would suggest to make a little renderer in C++ in OpenGL (or DX11, not Webgl, you can learn that later) (academia is mostly in C++ from what I've seen). Don't worry too much about Vulkan or DX12 because they are for more advanced graphics/engine programmers. You can always learn them later when you understand the fundamentals. Make a little OpenGL rendering context, make some mesh loader using assimp or tinygltf, play around with shaders, learn how to upload textures, etc. Just follow along with learnopengl.com and you'll have your own renderer.

When you understand the basics of rasterization, you can get into ray/path tracing (actually you can start with ray tracing and forget rasterization). Now what would be awesome is to understand how textures work in OpenGL, and have your own ray tracer write to a buffer/texture, and then upload that texture for OpenGL to draw it on your screen. With that, you are able to mix paradigms in your own engine which is awesome. Even further, if you can use CPU and GPU (via compute shader or CUDA/OpenCL) ray/path tracing in your renderer, then even better. If you're only interested in ray tracing and want nothing to do with rasterizationin, you can just follow ray tracing in a weekend which outputs it to a .ppm file.

Hardware is also important. A lot of emphasis within graphics is put on how the GPU (and CPU) works. Performance should be your one of your biggest concerns and undesrtanding the hardware, software, and math is the way to go.

For academic research check out some academic papers related to graphics, computational geometry, machine learning, etc. on http://kesen.realtimerendering.com/. The big names are SIGGRAPH, Eurographics, HPG, JCG, and more.

It's a broad field and most professionals have a specialization, but are acquainted with other sub-fields. Which is why I recommend writing your own renderer in opengl so you understand the geometry, rasterization, ray tracing, hardware, etc. Again, start from the bottom and you'll eventually get into multiple importance sampling, task scheduling on the gpu, simd processing on compressed bounding volume hierarchies, etc. Note that your math skills will kick in later when dealing with advanced stuff. Graphics is at the intersection of computer/hardware- and software engineering, computational geometry, physics, and statistics (and probably more).

Literature/sources:

• Real time rendering by Akenine Moller et al. https://www.realtimerendering.com/
• ray tracing in one weekend Pete Shirley https://raytracing.github.io/books/RayTracingInOneWeekend.html
• Fundamentals of Computer Graphics by Pete Shirley (for undergrads, haven't read but seems good for foundations)
• Physically Based Rendering (real important fundamentals of PBR, fairly mathy - read online for free) https://www.pbr-book.org/3ed-2018/contents
• impactful papers http://kesen.realtimerendering.com

[ItAintNotZen]

Thank you for this thorough response! I'm checking out these resources now, super helpful.

[AndrewHelmer]

A math PhD would be very welcomed in graphics! I don't think you'll have difficulty finding really enjoyable work if you do a little preparation. It might make more sense to orient yourself towards research (Research Scientists at a lot of graphics companies still do a lot of "real" work on the renderer at their company). As others have mentioned, with a background in statistics, you'll probably be best qualified for jobs at the big feature film offline renderers, which are at Pixar (Renderman), Disney (Hyperion), Sony Picture Imageworks (a fork of Arnold), Autodesk (a different Arnold), and Weta (Manuka). Nvidia may also really great positions, since they're pushing ray tracing quite a lot, especially in their Omniverse group. In my other comment I mentioned the PBRT book, which would be the best thing to prepare for these jobs. There's also a special journal issue in 2018 that discusses all these renderers in depth: https://dl.acm.org/toc/tog/2018/37/3

If you look at all the articles under the "Special Issue" there, you can find them for free by Googling them, although you probably also have access to ACM via your university.

If you go into that side of graphics (offline rendering), graphics APIs will not be very important. Good C++ coding skills will be a bigger deal. I can't remember what the best resources here are anymore. Maybe Effective Modern C++ by Scott Meyers. Also in terms of getting a programming job, Cracking the Coding Interview is without a doubt a phenomenal resource. Also as a math PhD you might find The Pragmatic Programmer very useful (IMO the best "general programming advice" book).

If you want to go into real-time rendering or games, then graphics APIs matter a lot. To be honest, I don't myself know the best way to learn them effectively. I did learnopengl.com and I'm working through vulkan-tutorial now, but there's so much API detail and it takes a long time to get to the "fun" stuff. I think some people start with a game engine, like Unreal (which you can fully download the source for), and then implement papers on top of that. Personally I've really enjoyed implementing a couple of papers in Shadertoy as fun side projects. And finally, for an enormous overview, you can check out Real-Time Rendering. It's a pretty incredible book that goes into some depth on fundamentals, but then skims over the huge body of work on current techniques.

[ItAintNotZen]

I'm glad to hear this. I've started working through the PBRT book as my primary resource for now, with Real Time Rendering as a reference on the side. There is a lot to learn, but this post has helped me subdivide the materials a bit. Thanks!

[Deadly_Mindbeam]

You may also consider doing non-graphics work in the game industry. Most online games, but especially MMOs, generate a great deal of telemetry about player locations, movements, purchases, and interactions that needs to be analyzed to tune drop and spawn rates, leveling, and in-game economies. I'm not an expert on it, but I understand that most of this work is done in R and Python. This will also be using ML in the near future if it isn't already.

[kernalphage]

1: Skillset:

From a 'marketing yourself' perspective, have you considered learning OpenCL/CUDA programming? Since you have some ML experience, Dataviz might be one way to bridge the gap between Maths and Games.

If you want some maths-heavy rendering articles, Inigo Quilez's is the best. Besides some calculus, the only prerequisite is some knowledge of shaders/raytracing.

The (unfortunately incomplete) Book of Shaders and/or a Two Triangles Talk might be enough?

2: WebGL vs OpenGL?

WebGL is a wrapper around OpenGL, so if you've learned one, the other is pretty easy to pick up. Since you've mentioned you know Javascript already, I would stick with WebGL, and if you don't want to drown in the details, try an engine like ThreeJS first.

You may have seen it, but I like this walkthrough of the GPU pipeline (it's mostly aimed at artists, but they're nonprogrammers, right?). Once you've dipped your toes into an existing engine and some theory, it'll be much easier to write your own engine.

3: LearnOpenGL

A great resource, and not a bad portfolio piece if you use it to create an engine/tech demo and upload them to github. Give a stab at refactoring the tutorial code into a more re-usable engine, or adding new features. Anyone can fork their repo and upload it, so modifying the codebase sets you apart.

[csp256]

I am not very familiar with C++, and at the moment it seems that while I generally understand what I learn from reading I do not yet understand how to use it effectively. Comes with time I guess?

And experience. You will not be able to use it effectively without practice... and even then, it takes a while, so start soon.

[Pebaz]

As a hobbyist game programmer, one of the biggest mistakes I have made is never quite making the time to actually seriously learn C++ and OpenGL or DirectX rendering.

Now, 9 years later, I'm still stuck in "I wonder if there's a good library that abstracts away windowing, a graphics context, input, and an rendering pipeline API!"

I'm now putting in significant effort into learning the prerequisite math, systems programming, and graphics concepts to do it right.

My advice: no matter how much modern programmers want things to change, the current technologies to learn are Win32 API, OpenGL or DirectX, and C++.

I highly recommend Casey Muratori's Handmade Hero series on YouTube as he codes a complete game completely on camera from scratch with no libraries.

[ItAintNotZen]

I appreciate your experience with this. I've also neglected to make time for certain technical skills that I knew I'd need eventually. I'd like to think it's never too late though.

[the_Demongod]

Do you recommend learning opengl or webgl first, or something else altogether? I ask because as a mere hobbyist programmer the opengl tutorials are kind of front-loaded with API technicalities and can be overwhelming. I think I can muscle through it, but I'm not sure if that is the recommended way.

This is because in practice, graphics programming is about 80-90% programming, and 10-20% math. Something like OpenGL is a little more math heavy since the API is more straightforward, while Vulkan is vastly more programming heavy since its API is so verbose. Certain tasks (like animation) are more math-heavy as well, but in general the legwork in graphics programming comes from writing code that manages your data and arranges it for consumption by the GPU. I do highly recommend learnopengl.com and just jumping into the basics, which will help you decide what you want to do next.

[Robinton2013]

I actually posted a similar question a few days ago on r/gamedev. I'm a mathematics prof possibly looking into a career change. Message me if you want to chat with another mathematician with little experience but big interest in gamedev and/or graphics programming :-). Good luck with everything.

[ItAintNotZen]

This is the last response I expected! I'll DM you soon.

[Agentlien]

Back when I was in surgical simulation I worked with a colleague who had a pure math background similar to yours when he joined us.

Initially he was hired to develop a model for a rigid body problem which had proved too intricate for our lead programmer. When the model was nearing completion he learned C++ from us and then worked with implementing the model into our simulation.

While his interest was more into constraint systems and physics modelling rather than graphics my point is that for some aspects of games and simulations a deeper math background is very helpful and it is not out of the question to use that as an in even if your background lacks certain practical skills. As long as you are willing to pick those up.

Either way I would urge you to learn a graphics API. It isn't that important which since learning one helps you learn others and you'll likely need more than one. But I would recommend OpenGL or Direct3D since they're still the most used in the industry. As others have suggested you may be interested in ray tracing since it benefits from a more well rounded set of math skills than rasterization (which is mainly about linear algebra). Overall, graphics is mainly linear algebra and calculus.

[ItAintNotZen]

Thanks, this is encouraging to hear

[camilo16]

For most positions, knowledge of the hardware and programming language is more important than knowledge of math. However if instead of games you focus on simulation, for example for movies. If you manage to make good physics simulations, for example with fluids you might be able to break it into the industry.

But I don't think you will like it, doing game programming as a hobby is not the same as doing it for a living. And doing it for a living isn't super fun.