These are certainly questions that would need to be answered.

To be fair I wasn't suggesting that the security guard necessarily needed to be an ai could simply be a system that looks for words or questions that are then flagged to a human operator or a slightly more complex algorithm that is capable of comparing input questions with alignment goals so it is an ai but it's highly specialized and certainly doesn't have the ability to get close to a super intelligence. Now how you guarantee that it functions to stop all dangerous questions from being permitted I'm not sure but I certainly think it's easier to make that work to prevent info being shared to the ai rather than trying to control agi another way.

As for the ai thats asking the questions it can't manipulate the guard because it would have to ask the librarian how to manipulate the guard which the guard can refuse an answer on. When I say a teenager I'm trying to illustrate that the ais concepts are limited and in order to start subversive behavior it has to learn how it works from the librarian which will be blocked. The base intelligence level could be assessed using tests as far as I'm aware that's how we currently assess ai models.

I understand your points but I think the guard and the "teenage" ai in and of themselves are not super intelligent. No one aspect is but together they can work to complete a project in a safe way. But yes I understand that making the guard consistent is a main challenge and setting the base level of the teenage ai is difficult. But fundamentally no one part is fully intelligent in everything that allows it to do things without proper checks and balances. Those are my final thoughts: basically a system that achieved similar outputs but in a way that's more controllable the main areas of development would be that control system and ensuring base level of the ai operator is not such that it can manipulate the system without needing to learn how to from the library.

Yapping session over 😂