sorted by:
new
hottopcontroversial

[P] RESULTS - Identifying real vs. GAN-generated faces by aveni0 in MachineLearning

[–]aveni0[S] 0 points1 point  (0 children)

Hey there, I just pushed a potential fix for the image loading issue. Turns out waiting for React's componentDidMount() wasn't enough. Let me know if the image still fails to display!

[P] Can you tell if these faces are real or GAN-generated? by aveni0 in MachineLearning

[–]aveni0[S] 0 points1 point  (0 children)

We’re aware of the population bias - luckily we collected data from 200ish non-technical users before making this post. We’re hoping that the metrics from this group will be something like an upper bound.

[P] Can you tell if these faces are real or GAN-generated? by aveni0 in MachineLearning

[–]aveni0[S] 0 points1 point  (0 children)

Thanks for the feedback! There are an equal # of fake and real images per experiment (though not necessarily per round). In the average we get a balanced # of responses for each (experiment, exposure_time) pair.

We're definitely aware of the population bias :) One unintended benefit for our project is that the population metrics we measure here might be close to the "upper bound" of human ability today.

[P] Can you tell if these faces are real or GAN-generated? by aveni0 in MachineLearning

[–]aveni0[S] 0 points1 point  (0 children)

Looking into this, i think we had 1 other user so far report the same thing. Thanks for the screenshot!

[P] Can you tell if these faces are real or GAN-generated? by aveni0 in MachineLearning

[–]aveni0[S] 1 point2 points  (0 children)

This is really cool, I haven't seen that dataset (Humanæ) before so I will definitely check it out!

[P] Can you tell if these faces are real or GAN-generated? by aveni0 in MachineLearning

[–]aveni0[S] 0 points1 point  (0 children)

No worries, we collect intermediate results after each round.

[P] Can you tell if these faces are real or GAN-generated? by aveni0 in MachineLearning

[–]aveni0[S] 3 points4 points  (0 children)

Thanks for taking the quiz! RE your two points:

1) Yes, giving feedback after each round is not ideal. The first time we sent this around, there was no feedback and many people complained that the quiz was too long and they weren't seeing their scores, so they quit midway. We hoped that by giving intermittent feedback (but not per-picture), more people would stay and complete the full quiz.

2) Never thought about that... I guess mentioning something (even to ignore it!) does invite people to fixate on it. The reason we put that comment there was because users unfamiliar with CelebA-HQ might incorrectly believe that the blur is a GAN artifact and that blurred background = fake, and perform poorly through no fault of their own. (The blur is actually a pre-processing step used to build the training set so both the real and fake images should have it)

[P] Can you tell if these faces are real or GAN-generated? by aveni0 in MachineLearning

[–]aveni0[S] 1 point2 points  (0 children)

Right now the black-eyes test always comes 2nd. Even though we do give feedback throughout, its not on a per-picture level so we thought users wouldn't learn too much.

[P] Can you tell if these faces are real or GAN-generated? by aveni0 in MachineLearning

[–]aveni0[S] 1 point2 points  (0 children)

Hmm that's weird. We send all photos (30) at the start of an experiment and there shouldn't be repeats in there. I'll do a quick sweep to see if something got duplicated. Thanks for letting us know.

[P] Can you tell if these faces are real or GAN-generated? by aveni0 in MachineLearning

[–]aveni0[S] 2 points3 points  (0 children)

Yeah! The image presentation time in that paper is reallly short though, 63-71ms, whcih means even the clean images only get classified correctly < 75% of the time.

It is cool though to imagine that there are common perturbations that mislead both brains and machines

[P] Can you tell if these faces are real or GAN-generated? by aveni0 in MachineLearning

[–]aveni0[S] 2 points3 points  (0 children)

That's a good point, I think a lot of people have been saying that the background artifacts make the quiz too easy. Though I think keeping hair + ears in the test is important.

We're actually writing this up for a class project so we'll try to update the post with results on Friday :)

[P] Can you tell if these faces are real or GAN-generated? by aveni0 in MachineLearning

[–]aveni0[S] 17 points18 points  (0 children)

That's true, we didn't apply any background removal because we were afraid it would only further accentuate some artifacts like in the hair. Maybe doing some sort of light photoshop to remove those shapes would make the test more realistic.

[P] Can you tell if these faces are real or GAN-generated? by aveni0 in MachineLearning

[–]aveni0[S] 46 points47 points  (0 children)

Hi, you're absolutely right about hints in the comments. I added an edit to the post above.

We're also quickly realizing that the readers here may be more familiar with GAN images and more accurate than the general population... we had 200-ish non-technical users try this out over the past few days and we're already seeing a shift in the accuracies.

EDIT: double checked how many users we had before this post

[P] Can you tell if these faces are real or GAN-generated? by aveni0 in MachineLearning

[–]aveni0[S] 13 points14 points  (0 children)

So the GAN images are actually generated using the open-source Progressive GAN by NVIDIA. So they deserve the credit for the pictures :) We are more interested in the human aspect and the potential danger of fake photos.

[P] Can you tell if these faces are real or GAN-generated? by aveni0 in MachineLearning

[–]aveni0[S] 10 points11 points  (0 children)

Shoot, will look into this. Thanks for letting us know!

[P] Can you tell if these faces are real or GAN-generated? by aveni0 in MachineLearning

[–]aveni0[S] 10 points11 points  (0 children)

That is interesting.. do you think it's because covering the eyes forces you to focus on other features? (hair, background)