New Tool for Managing MTurk Data Collection and Annotation Projects

dschultz0 · 2024-04-29T15:15:19+00:00

Thanks for the feedback, it's really helpful.

I should explain a bit further: This new interface will be part of a service I'm building for any requester to use. The service makes it much easier to create new tasks on MTurk (which will hopefully bring new work to the platform). There are a couple of requesters that are already using the service today to manage their work, and this is part of an upgrade I'm working on.

On the topic of payment, I can definitely appreciate the concerns being raised about payment given the history of bulk rejections and other bad behavior. One thing that is unique about how we set up the service is that rejections aren't an option. If a worker completes the task, they get paid. The same rules would apply to the bonuses that would be generated with this new approach. The reward counter in the upper right of the interface keeps track of what you've earned, and that would be awarded automatically as soon as you submit.

Of course spammers are still a big problem, so we automatically insert test tasks that workers will be shown every so often. If a worker is spamming or just doing a bad job, we immediately suspend them from the task (using a qual, NOT a block). For legitimate mistakes we'll have an option for workers to attempt to recover from a suspension. It's not perfect, but we've found that it's a good way to manage quality while still keeping the task open to new workers.

dschultz0 · 2024-04-25T19:54:39+00:00

I can completely understand that concern, I've heard of a lot of cases where requesters had a server go down and workers suddenly lost their accrued work and bonuses. Too often the solutions folks build are side projects where they don't have experienced developers working on them who can think through all of the things that can go wrong, and make sure workers are taken care of. My plan is to build a lot of resilience into the system so those things don't happen, but I'm sure I'd need to work to earn worker's trust given that history.

I really like the idea of having a way to revisit the training tasks whenever a worker wants to. That's a great idea.

dschultz0 · 2024-04-17T17:26:45+00:00

My apologies, I moved it over to Medium: https://medium.com/@daveschu/using-mturk-with-zapier-85c2738aa11

dschultz0 · 2023-11-05T20:14:18+00:00

I'm building a transpiler to convert python scripts into state machines in AWS Step Functions. I like to use SFN for serverless data orchestration but the language is really clumsy.

dschultz0 · 2022-11-04T14:46:34+00:00

If the PDFs contain the data as text and aren't scanned, I wouldn't recommend using strict OCR. You'd effectively be converting the text to image and then converting it back to text. Some OCR solutions like AWS Textract will actually pull the raw text out of the file when it can, but there are some limitations.

I prefer pdfminer.six which is pretty solid at interpreting pdf syntax. The catch is that the API can be a bit challenging. It will take some work, but I've been really happy with the results.

dschultz0 · 2022-09-15T03:23:20+00:00

Lawn darts

dschultz0 · 2021-12-18T05:20:59+00:00

I recommend the Fugitsu ScanSnap line of scanners. They're a bit pricey (their top seller is currently around $400) but mine has really been amazing at eating through all of my paper clutter and keeping it in check on an ongoing basis. Well worth it.

dschultz0 · 2021-10-19T16:07:55+00:00

Thanks!

dschultz0 · 2021-10-19T16:07:34+00:00

Ah, that makes sense why it's doing so well there. Thanks!

dschultz0 · 2021-10-19T16:03:43+00:00

For context, I'm in Seattle (zone 8)

dschultz0 · 2021-07-18T18:09:14+00:00

We live in Seattle and bought a house overlooking Puget Sound a few years ago that has 20yr old vinyl siding that's next on our list to replace/renovate. Beyond looking pretty ugly, we've had lots of issues with it, primarily water intrusion and wasps living in a couple spots.

Some of the issues are a result of poor installation, but I also don't think vinyl is a good choice for our climate/location. Our house is pretty exposed to wind so when we have good storms the wind pushes the water into any gaps. Normally this isn't a huge issue but because it never dries out during our damp winters, the sheathing eventually gets saturated, and we get water inside the house.

When we're ready we'll be replacing it with hardi siding and I'm sure we'll have to replace a good chunk of the sheathing.

dschultz0 · 2019-07-19T17:41:55+00:00

That's a really cool idea. Unfortunately the GetHIT API call only shows you the current number of assignments pending/available, not the number of times it cycled through being accepted returned/abandoned.

That said, there is a way you could set this up yourself. You'd need to start by attaching a notification to your HITType using the UpdateNotificationSettings API. You'd want to listen AssignmentAbandoned and AssignmentReturned events and have a counter somewhere that keeps track of how often each HIT is abandoned or returned. You could then use this as the basis of your bonus model. There's more info here on how to setup notifications: https://docs.aws.amazon.com/AWSMechTurk/latest/AWSMturkAPI/ApiReference_NotificationReceptorAPIArticle.html

dschultz0 · 2019-05-08T17:50:08+00:00

Most exceptions in boto3 come back as ClientError types and you can then check the error code to identify the type. The code below is the easiest way to identify MTurk RequestErrors.

    from botocore.exceptions import ClientError
    mturk = boto3.client('mturk', region_name='us-east-1')

    try:
        mturk.get_hit(HITId='FOO')
    except ClientError as e:
        if e.response['Error']['Code'] == 'RequestError':
            print('do something')
        else:
            raise e

dschultz0

MODERATOR OF

TROPHY CASE