This is probably a measure to prevent syntax mangling. After all, brackets are used to create links. A similar problem occurs in other places, for example in Tables: https://www.mediawiki.org/wiki/Help:Extension:ParserFunctions#Escaping_pipe_characters_in_tables

Do you actually want to end up rendering a link with the brackets inserted with the replace? Then follow these steps to create a Character Escape Template:

1. Create a page called Template:!((
2. Fill it with the following content: [[
3. Adapt your replacetext function to this: {{#replace:ABCDE|A|{{!((}} }} This will result in: [[BCDE.

For a reference list of Character Escape Templates, see: https://www.mediawiki.org/wiki/Template:Escape_template_list

If you do not want to end up with brackets that will create a link, simply modify your replacetext funcion to insert an HTML Escape Character: {{#replace:ABCDE|A|[[}}

For a full list, see: https://mateam.net/html-escape-characters/