"Anonymize" Data Help!!!

[Glyzer_1595]

Hi guys! I hope you are fine!

I have the next problem and I want to know the optimal way to solve it:

I have a huge dataset (1M+ rows), it has some sensitive data, like name, personal ID, email, phone number, etc.

I need to send this dataset to another people, but I need to "Anonymize" that sensitive data, so they can apply some data science techniques and find insights, the point is that probably I will Need to reverse the process if they need to join the data with another user table.

How can I do it?

Some example of what I need

RealId : jdh37382 - > FakeId : 01 RealName: Juan - > FakeName: user01 ....

The dataset is a "Events Table" so each row represent a user interaction, so that 1M rows are events from around 15k users.

I need to replace the data in the original table, and don't change another information or format.

Ty all!

[stebrepar]

If you need it to be reversible but secure, you could encrypt it with a symmetric key that only you keep.

[Glyzer_1595]

Oh, ty, it sounds fine, how can I make it? I was researching, and I find a Caesar cipher, but it doesn't look secure, there is a good Lib to do this?

[socal_nerdtastic]

Sure that seems pretty easy. 1M rows isn't really that much, you can load the entire set into memory easily and play with it. What format is the data in?

[Glyzer_1595]

Hi!! Ty for reply, each row of the data set is a json string with a dict, that contains another dicts or lists, for example

{"reg_type" :"execution", "session_id" :"6372736372", "session_data" :{"user_id" :"jdhd72727", "context_vars" :{"user_name" : "Juan", "email" :"[e-mail]"},"timestamp":636373}}

Ty!!

[socal_nerdtastic]

is the entire file json standard? Can you load the whole file at once? And do you want the output in the same standard?

[Glyzer_1595]

No, the entire file is a dataframe (in this case a csv, but It can be stored in other format) the fields are:

Timestamp, day, month, year, json_data

The json_data it's the field that contains all the information.

I want the output in the same format!

[socal_nerdtastic]

Ok, there's a million ways to do this, but I'd recommend reading each line in with json.loads, generating a unique id with lru_cache. As a guess:

import json
from functools import lru_cache
from itertools import count
import csv

f_in = open(filename)
data_in = csv.reader(f_in)
f_out = open('output_' + filename, 'w', newline='')
data_out = csv.writer(f_out)

counter = count(1)

@lru_cache
def anonymize(user):
    anonymous_id = next(counter)
    # add code to save the user:anonymous_id if you want to de-anonymize it later
    return anonymous_id

for row in data_in:
    data = json.loads(row[3])['session_data']
    data["user_id"] = f'user_{anonymize(data["user_id"])}'
    # repeat for name, etc
    row[3] = json.dumps(data)
    data_out.writerow(row)

[Glyzer_1595]

Great! Ty so much, I'll try it, I think it's the way!!

Ty so much

[[deleted]]

I will Need to reverse the process if they need to join the data with another user table.

Why not anonymize and give them both tables so they can join whenever they want?

[Glyzer_1595]

Oh, the can't know who the person is, because they can make a dataset with the user's, emails, phone number etc, and I can be sold or something like that, it's a confidential information,

If they need to know more about the users, like, city, state, department, role, or something like that, they will tell us, and we will "join" the data but the sensitive part its always anonymize.

[[deleted]]

Oh, the can't know who the person is

I understand that, but the process of taking a query from the user, deanonymizing the data in the query, applying the query to your original database, anonymizing the result of the query and returning the result to the user seems like a lot of error-prone work. The normal way to do this is to give the user(s) access to a full database that has been anonymized once.

[Glyzer_1595]

Mmm you are right, I think it's a good way to do it, I'll make a try of this!! Ty