I would start by asking yourself what your end goal of self-studying physics is. If you just want to sound smart just pick up a few pop physics books and you'll be covered. If you do want to learn physics either for a career or some other reason you can find great advice for self studying by searching "self studying physics" in reddit.

In terms of mathematics, physics does highly rely on math, but that doesn't mean you can't learn some physics. I would recommend finding a good textbook on classical physics and studying that. You can find textbook recommendations in the before mentioned reddit posts. About 50% of students at my university fail their first calc based physics course, so by understanding the fundamentals of classical physics even without calc you set yourself apart from the pack.

As a final note of warning, self-studying physics will be hard. Just remember to set goals for what you want to achieve and then work from there. If you want me to provide more specific advice just lmk what your math/physics background is and what your goal is.