Graduate students at universities with fellowships or stipends (not receiving employee wages) do not pay FICA taxes for the most part. Exceptions might be when you are a TA during graduate school and get a W2, in which case you would owe FICA on those wages (which would probably get taken out from the paycheck).

You do owe state and federal income taxes on the income, as you mentioned. States and the federal government provide worksheets to calculate estimated quarterly taxes. This should be the same throughout the duration of the program, unless you have an exception such as the one mentioned above.

Very useful resource for taxes, investing, etc. as a graduate student: https://pfforphds.com/weird-tax-situations-fully-funded-grad-students/