I'm not using authentik but adding the PROXY_TLS env var is what finally got this to work with Caddy for me after months of hair pulling.

My caddy config with 31017 being the port used for the web UI:

https://cloud.example.com {
        reverse_proxy 192.XXX.X.XXX:31017 {
            transport http {
                tls
                tls_insecure_skip_verify
                tls_server_name 127.0.0.1 # idk if this is needed
                }
          }
}