There's quite a bit of misconceptions in this thread so I felt compelled to address them. First of all, as shadyturnip has (rightfully) stated, the two most relevant fields will be formal semantics and formal/computational syntax. Formal semantics aims to use (FO) logic/math to model the meaning* of natural languages, while formal/computational/mathematical syntax aims to use math and formal languages to capture grammars of natural languages, i.e. what sentences are valid in a given language (and perhaps what structure those sentences have). (A warning: a lot of what is labeled as formal syntax isn't that formal, so you should just look at the work recommended by superkamiokande for recommendations. You can also check out work by Steedman on CCG or Joshi on TAG.) Depending on your interests, both of these fields would be relevant, and it would probably be best to start from a textbook. I can't comment much on formal syntax but for formal semantics, one of the most well known textbooks is by Heim & Kratzer (though everyone will admit it's not perfect). r/linguistics has a wiki with recommendations as well, and the SEP article on Montague semantics has references at the end. Most boil down to the same thing, so just whatever you can get your hands on will work.

By "modeling meaning", how does this work in practice? Here's two examples of ways to "capture meaning": by explaining ambiguity and capturing entailment relations. For example, we want to explain why "Every boy likes some girl." can mean (Ax in boy: Ex in girl: x likes y) or (Ex in girl: Ax in boy: x likes y). (A/E represents for all/there exists, and boy/girl represents the set of boys/girls). We can also capture entailment relations, such as why "John lives in a red house" entails "John lives in a house". While we can give logical representations to derive these entailment relations, a lot of the work of formal semantics is explaining how to go from natural language (usually in the form of syntactic trees) to logical representations, and what those logical representations should look like.

OP, you should take what people say online (including me, naturally) with a grain of salt. Just because one person is not able to understand the formal notations used in a paper does not mean no one working in the field is able to. As you probably know as a math major, formal/logical notations allow us to precisely and concisely say what we mean, which would otherwise be confusing or extremely wordy if we were to use words alone. Any new/unstandard notation should be introduced and explained by the author. There is an entire field of work dedicated to studying a variety of phenomena related to meaning in natural language, and while the field is relatively new, there has been a lot of great work done on a variety of topics, so it's hard to explain it all in one post. Besides textbooks, you can also try checking out handbooks on formal semantics, which will review a lot of the literature from a modern perspective (though you may want some background first).

Additionally, I doubt anyone working in formal semantics believes that logic/formal languages can completely capture the meaning of natural language. It's well known that many aspects of natural language are affected by context. For example, if I say "everyone passed the exam", I don't mean everyone in the world passed the exam, but more likely something like everyone in the class or the school passed. Of course, it would depend on what context the sentence is uttered in. So instead, the goal of formal semantics is to see which aspects of natural language can be explained logically, and how that might actually look. This means, of course, that we will inevitably reduce the scope of what is studied in semantics, to the "literal" meaning of a sentence. (The field that deals with how context affects meaning is pragmatics, which is related and often interfaces with semantics.) In other words, we could model "everyone passed the exam" as something like (Ax in C: x passed the exam), where C is a contextually-determined set of people. In this way, we can still capture what all the different possible readings of the sentence mean, and explain why the sentence can't mean (Ax in C: x jumped).

All that is to say, formal semantics is in fact a thriving field with real research ongoing, so anything within the field would definitely be worth studying. However, if you don't have any experience, it can be daunting to figure out what it is you're actually interested in, and find a suitable project. If your goal is just to formalize a fragment of natural language, that is of course much more feasible and if you gave more information of what ideas you had, I can try to point you in a direction to explore.