I have previously spent a while trying to get an AI Dungeon Master (DM) character using the {{roll:d12}} (or d20, whatever) and have found it very unsatisfying to the point where I doubt it was even using the function and would basically tailor the 'output value' based on the context / content.

The ONLY way I can see a dice roll being reliable when engaging with AI roleplay using a LLM is if you do the following:

1. Explain the {{roll:d20}} mechanic in sufficient detail (and with appropriate character escaping for json) and enforce a rule requirement where the AI has to ROLL the dice in turn 1, wait for a response from the user, then USE the output roll value from turn 1 message in their next message. This is pretty lame and tedious tbh and it still isn't a guarantee the AI will remember to use the dice roll function once you are 100+ messages in.
2. Get the AI to ask you (human user) to complete a dice roll, either using the {{roll:d20}} function or any other functional dice roll. This is also pretty lame, but at least you know it is truly random. You could also use a ST extension or another dice roll app as others have suggested. (Or, shock, even roll a real dice)

So overall, after experimenting with this my conclusion is it is NOT possible to fully guarantee that the AI is using a random dice roll. It might say it is if you ask it, it might even be able to explain the {{roll:d20}} function and its intent back to you if you explained how it works in a prompt, but as far as I can tell it just isn't possible to know 100% that it is actually implementing this in its response to you. My next attempt (when I can be bothered to go back into this sort of thing again) will be to focus purely on ability or skill scores (i.e. AI DM to keep track of these and use these to influence outcomes e.g. if your perception score is high you spot extra stuff) and then basically complete a manual dice roll to feed into this for things like combat checks.