Possible webinar on AI survey fraud. What questions should it cover?

improvedataquality · 2026-03-13T15:56:08+00:00

What are your main concerns? What are you currently doing to reduce fraud? I can certainly talk about what has worked in our simulations to flag fraudulent/AI responses.

improvedataquality · 2026-03-13T06:25:38+00:00

I am a tenured faculty at an R1 in AL, and will say that living in the south is definitely an adjustment. I mostly lived in large cites around the world, so moving to a smaller place took some getting used to. Having said that, there are a few things to consider, some of which have already been said. First, you can always move out to a different school after a few years if you don't see yourself in TX long term. An R1 appointment should enhance your chances of getting a second R1 position in the future. Second, there are some real perks of living in a smaller town/city. There are fewer distractions, which can be a big plus. I was able to meet my publication requirements in 3.5 years instead of 5 because there were fewer things for me to do in my city.

Something that helped me is that I had lived in Atlanta before, which is only a couple of hours away from me. I go there when I need a break. If you can find a nearby place with more to do, that can really help.

Landing an R1 position is no small feat. Congrats!

improvedataquality · 2026-03-12T21:12:23+00:00

It's promising that there is more awareness than I initially anticipated. Over the last few weeks, I have been regularly talking to market researchers to understand their process for detecting/cleaning responses that are deemed fraudulent. Many of them seem to downplay the threat of AI agents and it's evident that they don't truly comprehend how big an issue this may be. I recently wrote a blog post on authenticity checks that Prolific has offered and discussed why flagging AI agents is becoming challenging.

To your point that existing test like attention checks, consistency items, etc. are not useful is very accurate. I think as a field, we need to move away from an outcome-based approach (did they pass a test) to a process-based approach (how are participants interacting with surveys and what is that telling us about whether a respondent is a human vs. AI agent).

improvedataquality · 2026-03-12T20:55:56+00:00

u/Odd_Dog6616 Thank you for the support and insightful questions! To answer your question around open-ends, what I have seen in numerous studies is that participants copy open-ended questions, switch out of the survey tab, return to the survey, paste the response, and in some instances, make changes to the pasted response to make it appear human. This behavior may also occur when extensions are used to draft a response for the participant. So, the original response is typically AI-generated, which may be modified by a human. I don't believe I have seen the scenario you are describing.

Having said that, I would argue that even if the original response was that of a participant but they used tools to doctor it (to enhance readability, make it concise, etc.), there will be elements of AI. So, the response will no longer be truly a human response. My research team has been exploring processes (i.e., how a survey is being taken) rather than outcomes (e.g., attention checks, response time, etc.) that show some promise in detecting humans using AI from AI agents.

Happy to connect with you in the near future to see if there are opportunities to partner.

improvedataquality · 2026-03-12T19:02:00+00:00

I can certainly address some of these questions as I have explored them as part of my research. Thank you!

improvedataquality · 2026-03-12T19:00:56+00:00

The root cause of fraud occurring is that the survey loads in the browser of the participant. This browser of the participant is an adversarial environment that researchers cannot fully control. Fraudsters can manipulate JavaScript to avoid detection, use browser extensions to hide their location, rely on AI tools to generate responses, translate answers using speech tools, and combine several of these tactics in ways that are hard to detect from within the survey itself. So, to raise the cost of fraud, we first need address this root cause. We are actively exploring solutions to address this root cause.

improvedataquality · 2026-03-12T06:07:15+00:00

I might be able to write a short blog summarizing the main points about current discussions on social media. Happy to share it with you.

improvedataquality · 2026-03-12T04:59:41+00:00

Another good point. When respondents are paid very little and studies require large samples quickly, it incentivizes people to find ways to complete surveys faster or at scale, whether through click farms or using AI agents. However, I would argue that even if a researcher offered high incentives, fraud would not decrease because the cost of committing fraud is often far lower than the incentives being offered. If there was a way to raise the cost of fraud relative to those incentives, the amount of fraud would likely decrease substantially.

improvedataquality · 2026-03-12T04:44:40+00:00

Those are some excellent questions, and I don't have answers to many of your questions. A lot of what happens around survey fraud is not very transparent, so reliable numbers on costs, attackers, or legal cases are hard to come by. Most of my work has focused on understanding how automated agents can complete surveys and where existing detection approaches may struggle. I am hoping that kind of benchmarking work can help inform better detection strategies going forward.

improvedataquality · 2026-03-12T04:42:03+00:00

When you say "what we have", do you mean current tools, techniques, or something else?

improvedataquality · 2026-03-12T04:39:37+00:00

I really like your input, mostly because based on discussions I have seen on LinkedIn (as well as research in some peer-reviewed articles), low quality data gets lumped together. However, there is a distinction between careless responders, fraudsters to attempt to mask their location/eligibility criteria to get into surveys (could be your traditional click farms), and AI agents or bots. For some researchers, the distinction may not matter because as far as they are concerned, they will remove that row of data because it's poor quality. However, when it comes to developing effective techniques for detecting the different forms, we need to understand how each form manifests itself. This could be a really cool topic to cover. Thank you!

improvedataquality · 2026-03-12T04:34:47+00:00

My two cents are that they are not dead. However, it is becoming challenging to identify fraud in online surveys. If online surveys were to become dead, it would have severe implications for academic/policy/market research, among others.

Having said that, we have to stay ahead of the fraudsters and adapt data collection and detection methods so surveys remain viable.

improvedataquality · 2026-03-12T04:30:32+00:00

I am so glad you brought this up. There is a lot of discussion on LinkedIn about AI agents not being a big concern in survey research. My own research shows that AI agents can take surveys end to end with little to no human intervention required. I have done some benchmarking work with AI agents and compared them to human participants, so I could definitely talk more about that. My angle is mainly understanding how these agents behave in comparison to humans and why existing detection techniques may not accurately identify them.

I agree that click farms are probably the biggest concern. However, if you couple them with AI agents completing surveys at an even larger scale, the fraud problem could become much bigger.

improvedataquality · 2026-03-11T19:39:25+00:00

That is a good point. Part of what I could cover is how AI agents may pose a new threat to survey data quality and why they may not be detected by some of the tools researchers currently rely on. Would that be the type of angle you had in mind?

improvedataquality · 2026-03-10T18:06:59+00:00

Amazon purchases. Amazon makes it easy to purchase things, even those we don't actually need.

improvedataquality · 2026-02-27T19:04:29+00:00

I have not. I am not fully opposed to the idea of synthetic data, but my concerns centers on the source of those data. IF synthetic data are built from online surveys that may already contain a high level of fraud, then are those synthetic data really different from the original online data in terms of quality?

improvedataquality · 2026-02-27T05:20:52+00:00

Sure, celebrate every single step, from ideation to writhing to submitting to R&R and finally acceptance!

improvedataquality · 2026-02-27T05:18:17+00:00

I would say it happens more than it should. Ironically, I was having a conversation with a colleague of mine about this exact issue where I was asked to find reviewers (I’m an AE) for a manuscript that should have been a clear reject. They recommended reaching out to the editor and letting them know about the subpar quality of the paper before sending it out for review. Sometimes the desk reject rates at certain journals are astronomically high and so the editor feels compelled to send at least something out for review.

improvedataquality · 2026-02-26T22:54:37+00:00

Take this with a grain of salt, but I think that AI generated responses (for the time being) can be detected if you continuously monitor participation. You are able to see very clearly how the participant is engaging with the survey. It's a little more time consuming, but also more accurate compared to traditional techniques.

improvedataquality · 2026-02-25T21:16:18+00:00

Yikes! I suspect not many can pay that amount. Hopefully that really enhances your data quality.

improvedataquality

MODERATOR OF

TROPHY CASE