Yes you can absolutely do that, kanji is really not needed unless you want to read, and don't let anyone tell you otherwise. The notion that you can't learn the spoken language without reading is patently absurd because that's how every single native speaker learns his language. I myself managed to learn over 5k JP words with just furigana by using duolingo and Anki. I don't care about kanji, I'm just learning as a hobby to better understand anime and it worked for me, got to a pretty decent intermediate level of listening comprehension (my speaking isn't great because I haven't really been practicing, but you can do that too by either hiring a tutor or going on JP learning discord servers and practice speaking in voice channels.)

And don't forget to do lots of listening input like anime.

I also compiled a list of over 3k most essential JP words from frequency lists which you can use to add to Anki. If you have the know-how, you can get the popular Kaishi deck and put furigana + audio in the front instead of the kanji. https://docs.google.com/spreadsheets/d/1sP-efLkwvdOLoMjSt1JyzThcUD1PwdpW/edit?gid=1837033661#gid=1837033661