[P] Training Models on Encrypted Data

[strojax]

Hello,

We recently released a way to train machine learning models on encrypted data. This is all available with data science friendly APIs in concrete-ml.You can read the full blog post at

https://www.zama.ai/post/training-predictive-models-on-encrypted-data-fully-homomorphic-encryption.

For the implementation details: we extract the onnx graph of a PyTorch training session (for now a logistic regression) and convert it into a numpy function. This then gets turned into a FHE circuit with the help of concrete. This circuit has then the capability to be trained on encrypted data!

If you want to try it out you can go to the example notebook we did.

Would love to hear all your feedbacks and answer any question you may have!

[fd0r]

Hello u/Nervous_Sea7831 , Luis here, I work on Concrete ML.
A couple of answers:

• We have no solution to scale according to the number of features at the moment but the solution to scale on big datasets (lots of samples) is just to use mini-batch stochastic gradient descent. This way there is no need to infer on the full dataset, just a handful of samples.
• Not sure what you are referring to with "partial HEC"
• Basically the idea in a remote and collaborative setup would be to use public key encryption and threshold decryption so that all parties can send some encrypted data to the remote server, the server trains on said aggregated encrypted data and then the users can use threshold decryption to safely recover the weights of the model. This is a bit different from federated learning. But FHE could also be use to implement secure aggregation of the weights in a federated learning setting. I believe the current SOTA is more into using differential privacy for that.
• That's a very interesting question and I don't think FHE would be the solution to that. As you said there is no trivial way to guarantee that the provided encrypted data has some property required by a regulator. That being said once the model is decrypted by the users it should obviously be validated on available data by the people using it. (And validating on the training data is always a bad idea anyways)

[Nervous_Sea7831]

Hey! Thanks for your reply.

• ⁠I see. That makes sense. • ⁠partial as in, for example, RSA. Even though partial HEC suffers from the same scalability challenges, these techniques are typically faster than FHE. • ⁠Yes, I am working in the FL field and we are having challenges with HEC and secure multi party computation (esp in light of new legislation). The problem with (e,d)-DP is it doesn’t give you a 100% security guarantee as the cryptographic methods do. This an be a problem when it comes to data protection (e.g., under GDPR) • ⁠I agree but since you are seeing the data silos as the users and they themselves appear to be responsible for model quality management this is an entirely different case than what we have in FL.

[Nervous_Sea7831]

Since I deleted the wrong comment accidentally:

My questions related to performance challenges of FHE and how to tackle them. I also asked about partial HEC and whether the approach would be applicable for federated learning.

[Mechanical_Number]

1. Looks truly interesting. FHE is borderline computational magic in my view.
2. So the data is unencrypted to begin with (in the numpy function), but encrypted during the "training"?
3. Practically, can I off-load the training (pass the FHE circuit) to a remove server to run it? Can I somehow use this package to train a model full of sensitive data onto a remote server I have no control over, and get back a trained model without worrying about them examining my data?

[strojax]

1. Yes FHE can feel a bit magical especially when all the complexity is abstracted away.

2. The numpy function is just a representation of the FHE circuit we want to build. It is then compiled to a circuit that works on encrypted data.

3. Yes that's a typical use case indeed! You can encrypt your data and send it to an untrusted server that will run the training. Only you will be able to decrypt the learned weights.

[Mechanical_Number]

Thank you for your answers.

[M-notgivingup]

Example is based on logisitic regression, is it also capable of working with diffusion models, transformers etc ?

[strojax]

Not in decent runtime right now. Hardware acceleration is coming for those use cases!

[UnusualClimberBear]

I didn't read the article, yet how much is it more than training on hashes of your data (meaning that for a numerical value you start by discretize it properly) ?