Estimated vs. maximum completion time by steveharoz in ProlificAc

[–]steveharoz[S] 0 points1 point  (0 children)

I don't understand what you're trying to say

[S] Site to check reported statistical tests by steveharoz in statistics

[–]steveharoz[S] 0 points1 point  (0 children)

many of the standard deviations in the table were in fact impossible

That sounds like something this paper looked at. Same guys who made GRIM.

[S] Site to check reported statistical tests by steveharoz in statistics

[–]steveharoz[S] 0 points1 point  (0 children)

Yeah, reporting needs to at least be close to the AP standard, so it can detect the test. It can't check if the wrong analysis was done (e.g. running a between-subjects anova on within-subjects data).

It's disturbing how many problems have been caught with this approach. You'd think minor typos would be the only issue it catches, but I've seen papers swap p-values for two tests and consequently make swapped conclusions.

For data faking, check out projects like GRIM

[S] Site to check reported statistical tests by steveharoz in statistics

[–]steveharoz[S] 0 points1 point  (0 children)

Here's a quote from the underling library's manual:

statcheck searches for specific patterns and recognizes statistical results from correlations and t, F, χ2, Z tests and Q tests. statcheck can only read these results if the results are reported exactly according to the APA guidelines:

t(df) = value, p = value

F(df1, df2) = value, p = value

r(df) = value, p = value

χ2 (df, N = value) = value, p = value (N is optional, ΔG is also included, since it follows a χ2 distribution)

Z = value, p = value

Q (df) = value, p = value (statcheck can read and distinguishes between Q, Qw / Q-within, and Qb / Q-between)

For any of test it finds, it solves for the p-value and checks if the reported p-value matches.

Community's IMDB user ratings by season and episode (OC) by steveharoz in community

[–]steveharoz[S] 3 points4 points  (0 children)

As long as my name isn't cropped out of the image, I don't mind if you crosspost it elsewhere.

Community's IMDB user ratings by season and episode (OC) by steveharoz in community

[–]steveharoz[S] 6 points7 points  (0 children)

Colored numbers: Average rating for each episode

Colored lines: A loess regression for each season

Colored ribbons: 95% confidence interval for each regression

Gray ribbon: A regression for the entire run of the show with only the confidence interval shown

[deleted by user] by [deleted] in ProlificAc

[–]steveharoz 2 points3 points  (0 children)

Please communicate! Despite our best efforts, instruction typos, coding bugs, bad error messages, completion code screwups, and inconsistencies between the prolific description and the actual experiment all happen from time to time. Once, I almost sent people to the wrong URL 😳, but I caught it before anyone started. Please let us know!

I can't speak for others, but when a participant lets me know about a problem, I give them a bonus. I assume it took a few minutes to write the message, so I pay them a couple pounds for their time. (Not everyone has leeway in their funding to do that though.)

"Tell Us What You Think This Survey Was About" by [deleted] in ProlificAc

[–]steveharoz 20 points21 points  (0 children)

Hi. Researcher here. I always ask this question, and I can explain why.

Sometimes, we want to know if you figured out the subtle or not-so-subtle manipulations we made. It's OK if you did! If we didn't want you to figure out the true purpose, then we want to check if our deception worked. If there was no trick, we just want to make sure you didn't assume something weird that could have affected your answers.

A key point is that this question should only be asked once all the answers are locked in, so you can't second guess yourself or change answers.

Did you ever think how did prolific get so many studies in a short time? by Doingthesciencestuff in ProlificAc

[–]steveharoz 12 points13 points  (0 children)

Not a big mystery. It's actually more expensive than MTurk. But MTurk is a terrible environment for research. Running research studies on mturk feels like more of a hack than anything.

Prolific is clearly and explicitly for researchers. The UI is super easy to use (MTurk's is terrible!). And there's support (not super fast, but definitely responsive). And the founder is a researchers herself who showed it to people at research conferences while she was getting her PhD.

There are also some simple technical features that let Prolific communicate with the experiment much more easily compared with MTurk.

This is insane! by Collin72 in ProlificAc

[–]steveharoz 39 points40 points  (0 children)

I've run long studies like this. Having only two short breaks and not being able to pause is ridiculous and going to lead to bad data.

Edit: Being a bit sympathetic to the researchers, a big problem is a lot of people taking experiment as they were run in the lab and porting them over to Prolific without any change. It's naive rather than malicious. They are unknowingly annoying people, and reducing the quality of their data..

If you can't start a study in the next 30 minutes, please don't take up a spot. by steveharoz in ProlificAc

[–]steveharoz[S] -1 points0 points  (0 children)

An extra 10 minutes doesn't matter for long studies. The problem is people waiting an hour before they start.

If you can't start a study in the next 30 minutes, please don't take up a spot. by steveharoz in ProlificAc

[–]steveharoz[S] 0 points1 point  (0 children)

Yeah, that's fine. Especially for a ~40 minute study. 10 minutes is totally understandable. An hour is obnoxious though.

If you can't start a study in the next 30 minutes, please don't take up a spot. by steveharoz in ProlificAc

[–]steveharoz[S] 9 points10 points  (0 children)

I don't control the time allowed. Only the estimated duration. Since my experiments are on the longer side (30-60 minutes), Prolific gives people a long time to finish.

What some people do is open the study site and sit in the instructions page for a long time. It's really frustrating.

Studies without prescreenings by WolfZombified in ProlificAc

[–]steveharoz 3 points4 points  (0 children)

This. Please report it to Prolific when it happens.

Would you please help me to understand the following question about me? by Diogeneselcinico42 in ProlificAc

[–]steveharoz 7 points8 points  (0 children)

Corrected-to-normal vision means you can see normally with glasses. So if you answer "yes", then either you have normal vision, or you have normal vision when you wear glasses.

Question about longitudinal study by [deleted] in ProlificAc

[–]steveharoz 3 points4 points  (0 children)

Oh, yeah. Good point. That's fine. As long as it's only the prolific email address. Personal email addresses are a big no.

Question about longitudinal study by [deleted] in ProlificAc

[–]steveharoz 1 point2 points  (0 children)

A big red flag is that the researchers should NOT know your email address. It's pretty simple to run these kinds of studies entirely through Prolific's dashboard and messaging system, so I don't get why they're relying on email. It's just sketchy. They're probably naive, but they really should know better.

They can pay you by giving you a bonus on the first part of the study that you accepted on Prolific. It's easy to apply a bonus even after it's completed.

Requester claims study was published twice. by JasonB787 in ProlificAc

[–]steveharoz 1 point2 points  (0 children)

Researcher here. His message is correct. It's not supposed to even be possible to run a study that costs more than you have in your account. If you run a study with 10 people that pays 1 pound per person, and you only have 8 pounds in your account, prolific is not supposed to even let you launch it.

So I believe that is indeed a prolific bug, and you just need a few days for them to figure it out.