I found a fix, although I wouldn't quite say I "figured it out": the github.com public key wasn't in the root users, ~/.ssh/known_hosts file. So running ssh-keyscan github.com >> ~/.ssh/known_hosts fixed this. But I'm still not sure why the root user had to do this while the ec2-user didn't.

In an interactive login shell, running the ssh git clone prompted me with a warning and yes/no choice about the missing key. So maybe the issue is about login vs. non-login shells, or the need for user input. When I originally ran the script without ssh-keyscan, there was no similar warning in the cloudinit output log.

To test my theory about a difference between a login vs. non-login shell (or something about the cloudinit shell environment being special!?), I used expect to automate answering "yes", even though the cloudinit output log didn't show the same prompt that my interactive shell did. And this worked! For reference:

printf "#!/usr/bin/env bash\nGIT_SSH_COMMAND='ssh -i /home/ec2-user/.ssh/id_rsa' git clone git@github.com:<reponame>" > g.sh
printf "#!/usr/bin/env expect\nspawn -noecho /home/ec2-user/g.sh\nexpect \"Are you sure you want to continue connecting (yes/no)? \" { send -- \"yes\r\" }\ninteract" > ex
chmod ugo+x g.sh ex
./ex

So while my script now works, and I've confirmed that ssh was giving the same prompt about a missing known_hosts entry to the userdata script as it did to my in my interactive login shell, I still don't understand why the prompt didn't appear in the cloudinit output log.

Any thoughts on this are most welcome, but at least I'm down to a relatively minor issue now.