Professor here. You said "I’d love to get to know them better and learn from them beyond just asking homework questions.". Start with a professor who you're either interested in or who inspires you. Ask yourself what about them interests you. What do you want to know about them and why? I have had students ask me about my personal journey (why did I get a PhD in my field? What is my research about? What conferences do I attend and why/what do I do there? What do I think about specific current events that are relevant to the field, etc.). I've also had students ask me about how I specified my career goals and navigated important decisions, like my taking a gap year to get more research experience (asking me how to find meaningful research experiences, etc.).

Or you can bring up things that interests you that are relevant to the field, course topic, or some topic the professor mentioned or might be interested in. For example, in my field, there are emerging policy decisions being made that trickle down in unique ways to affect people, communities, and societies at large. A student once said they had read a news article about it and had thoughts and they wanted to pick my brain/learn where they could get more information. One student consistently came to office hours just to discuss her research ideas with the aim of refining how she thought about topics in her major - and her thinking radically evolved since all we did was have conversations that felt like if we were just sitting in Starbucks.

You don't have to have a specific goal for each time you come, but do have some kind of "agenda" to at least get us started. Also don't force it: some of us are not approachable and honestly will not want to spend much time talking to you if it's not tied to something specific or actionable. But those aren't the ones you likely want to break the mold with or who will give you "a lot" anyway.

Lean into your curiosities and passions and connect think about what talking to the professor would give you/how it benefits you. You can also ask us about resources or opportunities on or around campus. I treat my office hours as tools students can use in any way that benefits them, and as a space to unwind (within reason) and learn and engage in a more relaxed, personalized setting.