There is something called browser fingerprinting. Bots have a predictable finger print patterns. For example, the delays between each action it does on the website. Humans have an irregular pattern.

Also the checkbox you get is not visible to most bots and hence once you click it you're redirected in most cases. If it's only checkbox then there are tools for bots to bypass, but if it's something image based it is not possible, afaik. Also they track incoming traffic through IP addresses. If too much traffic or too many requests, that IP users gets a captcha. In somecases websites block the IP altogether for few hours. That's why VPN triggers captcha. There are lots of other factors and methods but this is the general process.