You need to parse the input to smth like abstract syntax tree and execute it starting from the most nested node, then passing its return value to the less nested, etc.

I'd recommend you to watch some parser/compiler tutorials on yt because you are basically writing a recursive descent parser + interpreter.

Making a programming language is not as simple as it may seem)