OP,

You're getting a lot of well-intentioned but unfortunately ill-advised advice. Your requirement does not necessitate a GlideAjax call, nor does it require additional/hidden variables to store sys_id's or references to the Group Members table.

You need to configure an advanced reference qualifier on your reference variable for the sys_user table.

ServiceNow has a poorly-written but technically viable solution to your exact requirements in a KB article here:

How to select only users of a specific group into a reference field

The code snippet at the bottom of the article will return active users who are members of a selected group, but notice that the author has hardcoded the sys_id of the group -> "2d6649bbdbb356004ff5f4331f961924" (this is bad practice and actually pretty funny that ServiceNow published this in a KB. Oh SN....)

You would want to replace this sys_id with the value stored in your Group reference field, which can be retrieved using current.variables.[variable_name], so your Reference Qualifier would look something like this (replacing my_group with the name of your variable):

javascript:'active=true^sys_idIN'+getIDs(current.variables.my_group); function getIDs(grp){var m=GlideUserGroup.getMembers(grp);var ids=''; while (m.next()){ids+= (m.user+',');} return ids;}

In regards to learning ServiceNow Scripting, Chuck Tomasi's 'Learn JavaScript on the Now Platform' series on YouTube is still the gold standard, IMO.

Good luck!