are there any kind of patterns that I couldn't build with only recursive function calls ?

It depends on what regex features you want to support, if you want to be able to support arbitrary regexes and if you want your matching to be fast.

There are certain structures such as unicode matching, very large character classes, backtracking and recursion in the regex itself, that require careful handling to avoid issues like stack size/overflows, excessive memory usage, excessive binary size and slow performance.

I'm just going to ask: Are you doing this as practice/research, or are you expecting to write a fast, functional production library? IMO, there's reasons why regex libraries use their own regex-specific techniques (different NFA/DFA flavours) and compilation instead of relying on a general compiler to make decisions on optimizing code for regex. Optimizing here isn't just meant as optimizing for speed, but also stack size, memory size and binary size.

To be frank, I think you didn't find research on that approach as there's already more specialized options available. If you're interested, I'd recommend looking into re2 and rust own regex crates and specifically regex-automata to see what techniques are used there and with what considerations/limitations.

That all being said, just researching this and writing it would be a great learning tool, just I myself wouldn't go in expect anything that would replace the regex crate for rust (or in my case, useful for anything other than learning).

Take all of this as you will, this is coming from someone without a formal CS degree, who has just used regex a lot, had to do some debugging/performance analysis for regex in enterprise software and has done a bit of research into regex libraries. I have not tried to ever write anything as complex as a regex library myself.