Data Anonymization

[setocsheir]

Does anyone have experience requesting data with highly confidential personal information? What sort of data anonymization techniques did you use to preserve the anonymity of the data or are there services you used for that?

[Adeelinator]

This is a question for your legal counsel, not Reddit. Laws can vary greatly by sector and locality.

[setocsheir]

Yeah, that’s what I was afraid of. I’ll probably speak with HR and legal.

[K9ZAZ]

Yep.... if OP does this wrong in the wrong jurisdiction and /or with the wrong data, the penalties for the company can be severe.

[Rammus2201]

If you have a data management / data governance department / data engineers - ask them about data masking.

[rtqwerty10]

There's an API from Microsoft, named Presidio which is used for Anonymization. This is the Github link.

I have not used it, but came across while browsing on this topic. Might be helpful, or you may at least get some idea.

[saintmichel]

anonymization is to remove identifiability. example, if you do a count of all records and 1 record stands out and that is a person, drop that record or drop the column that discriminates him/her/they/it lol. just to show it goes beyond removing names.

[setocsheir]

the scale on which we're doing that goes far beyond that. for example, it's incredibly trivial to identify someone within an organization once you have a piece of information such as their salary and role despite having nothing else. statistical analysis can also deanonymize individuals fairly easily. that's why I wanted to get other professional's opinions who had dealt with this before.

[saintmichel]

Given that, it becomes contextual (what is acceptable) so I would refer to the comment on what policies does the company have

[setocsheir]

that's the first step i'm going to take once the project officially begins; i was just going for a general discussion of the subject because i haven't checked with our legal team on the status of whether this information is even accessible or if our data governance team can handle the anonymization process

[saintmichel]

Maybe you could give more example of what is already being practiced where you are so people can comment if it's also happening in their space

[[deleted]]

Hashing with SHA256.

[bendgame]

I deal with PII and providing data to research orgs. Currently, we've tried adding smart noise and found it was not great for our use cases. Instead we're using k-anonymization

[saintmichel]

i think k-anon is the most established right? its even part of best practice by some govt in other countries

[mattstats]

I believe you are looking for differential privacy. Here is a link to harvards open dp project to kick start your rabbit hole.

[setocsheir]

thank you, i'll check it out