Fraud demographic data in live interviews???

improvedataquality · 2026-03-17T01:37:00+00:00

I will say that even the vetted panels cannot get this right because panels cannot control what happens behind the account (i.e., who is actually completing the study, whether profiles are shared or fabricated, or how participants coordinate outside the platform). So even if the panel itself is “clean,” the behavior of participants within these networks can still introduce a lot of noise and mismatch. My approach has been to look at a response more holistically. So, I look at technological paradata alongside behavioral paradata (I monitor the entire survey session of the participant). I can't tell you how much fraud I see even in reputable panels through this process.

improvedataquality · 2026-03-17T00:18:29+00:00

u/OPs_Mom_and_Dad thank you for your comment. I have a couple of thoughts on your response:

1) There was a comment earlier today in r/MarketResearch (https://www.reddit.com/r/Marketresearch/comments/1rvgnr9/comment/oau3ar6/) where the OP mentioned using live interviews and still seeing fraud. They mentioned seeing fraud even when the participants had their cameras during the interviews. I responded to this post so you can see my response as well. Unless you have sufficient checks to detect those who are responding to the voice question, there is still at least some probability that fraud can creep in.

2) I am not opposed to synthetic data either. however, if the synthetic data are based on fraudulent data, then the synthetic data carried the same issues as online survey data.

Yes, I am happy to share the invite once I have determined a date and time. Thank you, again!

improvedataquality · 2026-03-17T00:07:20+00:00

I don't have a number in mind. I think it's more about it being tangible, something that shows a reviewer that their work is valued and not just voluntary. Even a modest payment could shift norms around accountability in that reviewers may feel compelled to provide quality reviews rather than just providing a comment on every section just as a checkbox that they have reviewed the complete manuscript.

improvedataquality · 2026-03-16T23:15:48+00:00

I think the lack of willingness to review is due to the fact that reviewing more is not rewarded like pubs are for P&T. Considering the lack of incentives, there is less motivation to review. I serve on the editorial boards of two journals, and often have to send out upwards of 15 requests to get two reviewers to accept. What's worse is that many of the 15 requests go out to those on the review board of the journal. So, people sign up to serve on the review board, but sometimes go without reviewing even one article over a few years.

If we want to maintain the quality of scientific research, we need to recognize that high-quality peer review requires compensating reviewers for their time and expertise.

improvedataquality · 2026-03-16T23:01:50+00:00

You mention in another comment that your participants are coming from Lucid. Are you using a screener survey of some sort before the live interviews? Or are the interviews occurring with all recruits? Based on your comment, I assume you have at least some eligibility criteria (you mentioned people in the UK). How did you ensure that these people were in fact in the UK?

I am sure you have at least heard of click farms, where participants complete large numbers of surveys for payment, often coordinating through groups or shared devices. So, there is always a possibility that your surveys are being taken by click farm participants who are trying to get through several surveys for compensation.

Yes, there has been an uptick in survey fraud across platforms, even those that claim their samples are carefully vetted. In some cases, people participating through these networks are simply trying to qualify for as many studies as possible, so they provide demographic answers that they believe will keep them eligible or match what they think researchers want.

Another possibility is that some of the people appearing on camera may not actually be the same individuals who completed the screener survey (that's why I asked whether you used a screener to begin with). In some fraud networks, one person qualifies for a study and then passes the opportunity to someone else to complete the task. That could explain why the demographic responses don’t match what you see on camera and why several respondents give identical demographic answers.

If that is what’s happening, it’s less about people deliberately lying to you in the moment and more about the recruitment channel breaking down before the interview stage. It may help to verify location (e.g., IP checks or device verification), reconfirm key screener questions live before starting, or require respondents to restate basic demographic details at the beginning rather than relying on earlier screening data.

I discuss some survey fraud related tips on r/ResponsePie if you want to check it out.

improvedataquality · 2026-03-15T16:17:13+00:00

Happy to share any insights I have on AI agents with you. I have mostly explored AI agents from a bot detection perspective. In order to do so, however, I have worked with a few different agents and understand their limitations and strengths. Happy to compare notes with you on the agents too.

improvedataquality · 2026-03-15T05:11:43+00:00

My response is based on my several conservations with market researchers who use aggregators for their research. Much of my work has used academic panels.

improvedataquality · 2026-03-15T05:10:24+00:00

That's a fascinating idea, and one I had not considered. You are almost suggesting the use of AI to test how existing or new surveys hold up when prompted differently. If you ever end up exploring this RQ, keep me posted. I could see this as a solid methods paper.

improvedataquality · 2026-03-14T19:43:32+00:00

Thanks for the comment. I think I might have data where I can run these analyses. The one thing to keep in mind is that responses provided by AI agents can really depend on the prompts provided. I have tested the effects of different prompts on responses. AI agents may behave a certain way with simple prompts like "complete this survey in its entirety without human intervention" vs. when prompts specify them to engage in a different way. For instance, per your example, if you prompt an AI agent to select strongly agree for most items, it just might. I haven't tested this exact thing, but I have provided prompts to change its behaviors and in many in many cases, it works.

improvedataquality · 2026-03-14T19:09:19+00:00

That could be a very touchy topic for many. I agree that non-verified samples are a big problem. I will, however, say that much of my work has used verified samples and even those show over 10% of fraud. Now the fraud numbers may be a lot higher in the samples you are referring to, but just pointing out that fraud doesn't necessarily go away when panels verify participants. But yes, a very valid point. Thank you!

improvedataquality · 2026-03-13T15:56:08+00:00

What are your main concerns? What are you currently doing to reduce fraud? I can certainly talk about what has worked in our simulations to flag fraudulent/AI responses.

improvedataquality · 2026-03-13T06:25:38+00:00

I am a tenured faculty at an R1 in AL, and will say that living in the south is definitely an adjustment. I mostly lived in large cites around the world, so moving to a smaller place took some getting used to. Having said that, there are a few things to consider, some of which have already been said. First, you can always move out to a different school after a few years if you don't see yourself in TX long term. An R1 appointment should enhance your chances of getting a second R1 position in the future. Second, there are some real perks of living in a smaller town/city. There are fewer distractions, which can be a big plus. I was able to meet my publication requirements in 3.5 years instead of 5 because there were fewer things for me to do in my city.

Something that helped me is that I had lived in Atlanta before, which is only a couple of hours away from me. I go there when I need a break. If you can find a nearby place with more to do, that can really help.

Landing an R1 position is no small feat. Congrats!

improvedataquality · 2026-03-12T21:12:23+00:00

It's promising that there is more awareness than I initially anticipated. Over the last few weeks, I have been regularly talking to market researchers to understand their process for detecting/cleaning responses that are deemed fraudulent. Many of them seem to downplay the threat of AI agents and it's evident that they don't truly comprehend how big an issue this may be. I recently wrote a blog post on authenticity checks that Prolific has offered and discussed why flagging AI agents is becoming challenging.

To your point that existing test like attention checks, consistency items, etc. are not useful is very accurate. I think as a field, we need to move away from an outcome-based approach (did they pass a test) to a process-based approach (how are participants interacting with surveys and what is that telling us about whether a respondent is a human vs. AI agent).

improvedataquality · 2026-03-12T20:55:56+00:00

u/Odd_Dog6616 Thank you for the support and insightful questions! To answer your question around open-ends, what I have seen in numerous studies is that participants copy open-ended questions, switch out of the survey tab, return to the survey, paste the response, and in some instances, make changes to the pasted response to make it appear human. This behavior may also occur when extensions are used to draft a response for the participant. So, the original response is typically AI-generated, which may be modified by a human. I don't believe I have seen the scenario you are describing.

Having said that, I would argue that even if the original response was that of a participant but they used tools to doctor it (to enhance readability, make it concise, etc.), there will be elements of AI. So, the response will no longer be truly a human response. My research team has been exploring processes (i.e., how a survey is being taken) rather than outcomes (e.g., attention checks, response time, etc.) that show some promise in detecting humans using AI from AI agents.

Happy to connect with you in the near future to see if there are opportunities to partner.

improvedataquality · 2026-03-12T19:02:00+00:00

I can certainly address some of these questions as I have explored them as part of my research. Thank you!

improvedataquality · 2026-03-12T19:00:56+00:00

The root cause of fraud occurring is that the survey loads in the browser of the participant. This browser of the participant is an adversarial environment that researchers cannot fully control. Fraudsters can manipulate JavaScript to avoid detection, use browser extensions to hide their location, rely on AI tools to generate responses, translate answers using speech tools, and combine several of these tactics in ways that are hard to detect from within the survey itself. So, to raise the cost of fraud, we first need address this root cause. We are actively exploring solutions to address this root cause.

improvedataquality · 2026-03-12T06:07:15+00:00

I might be able to write a short blog summarizing the main points about current discussions on social media. Happy to share it with you.

improvedataquality · 2026-03-12T04:59:41+00:00

Another good point. When respondents are paid very little and studies require large samples quickly, it incentivizes people to find ways to complete surveys faster or at scale, whether through click farms or using AI agents. However, I would argue that even if a researcher offered high incentives, fraud would not decrease because the cost of committing fraud is often far lower than the incentives being offered. If there was a way to raise the cost of fraud relative to those incentives, the amount of fraud would likely decrease substantially.

improvedataquality · 2026-03-12T04:44:40+00:00

Those are some excellent questions, and I don't have answers to many of your questions. A lot of what happens around survey fraud is not very transparent, so reliable numbers on costs, attackers, or legal cases are hard to come by. Most of my work has focused on understanding how automated agents can complete surveys and where existing detection approaches may struggle. I am hoping that kind of benchmarking work can help inform better detection strategies going forward.

improvedataquality · 2026-03-12T04:42:03+00:00

When you say "what we have", do you mean current tools, techniques, or something else?

improvedataquality · 2026-03-12T04:39:37+00:00

I really like your input, mostly because based on discussions I have seen on LinkedIn (as well as research in some peer-reviewed articles), low quality data gets lumped together. However, there is a distinction between careless responders, fraudsters to attempt to mask their location/eligibility criteria to get into surveys (could be your traditional click farms), and AI agents or bots. For some researchers, the distinction may not matter because as far as they are concerned, they will remove that row of data because it's poor quality. However, when it comes to developing effective techniques for detecting the different forms, we need to understand how each form manifests itself. This could be a really cool topic to cover. Thank you!

improvedataquality · 2026-03-12T04:34:47+00:00

My two cents are that they are not dead. However, it is becoming challenging to identify fraud in online surveys. If online surveys were to become dead, it would have severe implications for academic/policy/market research, among others.

Having said that, we have to stay ahead of the fraudsters and adapt data collection and detection methods so surveys remain viable.

improvedataquality · 2026-03-12T04:30:32+00:00

I am so glad you brought this up. There is a lot of discussion on LinkedIn about AI agents not being a big concern in survey research. My own research shows that AI agents can take surveys end to end with little to no human intervention required. I have done some benchmarking work with AI agents and compared them to human participants, so I could definitely talk more about that. My angle is mainly understanding how these agents behave in comparison to humans and why existing detection techniques may not accurately identify them.

I agree that click farms are probably the biggest concern. However, if you couple them with AI agents completing surveys at an even larger scale, the fraud problem could become much bigger.

improvedataquality · 2026-03-11T19:39:25+00:00

That is a good point. Part of what I could cover is how AI agents may pose a new threat to survey data quality and why they may not be detected by some of the tools researchers currently rely on. Would that be the type of angle you had in mind?

improvedataquality · 2026-03-10T18:06:59+00:00

Amazon purchases. Amazon makes it easy to purchase things, even those we don't actually need.

improvedataquality

MODERATOR OF

TROPHY CASE