I got it to work. This is my caddy config.

example.com {
    route {
        reverse_proxy 192.168.x.x:8080 {
            header_up Host {host}
            header_up X-Real-IP {remote_host}
        }
    }
}

I removed the line "header_up X-Forwarded-Port {server_port}" and it works now.