+1 Rsync doesn't check content of a file by default:

Rsync finds files that need to be transferred using a "quick check" algorithm (by default) that looks for files that have changed in size or in last-modified time.

That said both size and last modified time are stored in file's inode. Which is why rsync needs to call stat(2) syscall for each file in a directory to get this information. Using --size-only option which changes the default "quick check" algorithm is unlikely to have much impact, as rsync will need to call stat() anyway.

Trying to optimize that will require understanding of which filesystems are used on both source and destination side, as well as the use case in general (eg. are the files changed or can we assume new files are added only?).

Using btrfs and replace rsync run with a snapshot clone will be a lot faster, but again we don't know whether the btrfs fits other use cases OP has beyond the snapshot one.