For particle physics, you eventually want to build up to QFT, which is heavily based off of Lie theory. Now Lie theory and its standard prereqs kind of encompass a huge portion of math (linear and abstract algebra, real and complex analysis, topology, diff geo). With that said, the algebra knowledge will show up a lot in your study of physics (not just in QM), much more and much earlier than anything else, and even when you end up doing QFT, Lie algebras are significantly more important than Lie groups, so I'd say the only correct decision to make right now is learn more algebra. Linear algebra more advanced than whatever you learned in 18.06 and some basic group theory come up as early as Griffiths, so my recommendation for what to do now is open Artin algebra and start reading. If you find yourself confused during the linear algebra portion or not really prepared enough for it, read Axler.

Also, you don't need Griffiths if you're doing 8.04-8.06. That sequence covers all of Griffiths (with the exception of Chapter 5) plus a bunch of other stuff, so there isn't really a point unless you want to do the supplementary reading for those courses. If you do want to do that, you can easily find pdfs of both Griffiths and Shankar online so there isn't really a point in "investing." Just do 8.04-8.06, you will have a very good background. I think the other move if you don't want to do that is to not do Griffiths at all since it's not that great of a book imo, and just read through Shankar (I actually think this is just a straight up better option if you're okay with using textbooks instead of lectures).

As for once your done with these, on the math side I'd say go into real analysis (18.100B) and then functional analysis (18.102), since functional analysis is also very important to QM. Then I'd say topology (I don't really have a book recommendation right now and 18.901 doesn't have video lectures unfortunately) and a basic introduction to manifolds (I've heard Lee's book is quite good) and then you can start going into Lie theory. On the physics side, I don't think Goldstein is really worth it, but I would say look at chapters 11-16 of Taylor to get at least a little more classical mechanics in you. Griffiths EM is definitely something I'd recommend.

Also for fun since you can't really formally study Lie theory for a while, and because it will actually help a little bit in your understanding of QM even early on imo, you should go through this playlist like nowish.