Rating AI responses

DataAnnotation says general projects such as chatbot response evaluation start at $25-$50/hr. That is a starting range for paid time, not what raters average once unpaid assessment and waiting time are counted. For comparison, BLS puts the median wage for data entry keyers at $18.17/hr, the closest official benchmark.

DataAnnotation asks for a bachelor's degree or equivalent experience and Outlier for at least an associate degree.

How the money works

Pay is hourly on some projects and per task on others. Outlier says it pays hourly or per task depending on the project, shows the rate before you start, and publishes no single dollar rate. Handshake AI sets hourly or per-task pay in each project's terms.

Rates rise with specialty. DataAnnotation's job board lists the Generalist role at up to $50/hr versus up to $125+/hr for many specialist roles.

What people actually earn

The AI Rankings, a review site, says it found no audited independent survey of AI annotation pay, only platform ads and unsourced self-reports.

Where the hours go

Paid minutes are only part of the time involved. DataAnnotation's assessment takes about an hour, or 1-2 hours for specialized work. Handshake AI says setup and technical issues are unpaid, and onboarding is paid only if the project terms say so. Mindrift doesn't pay expired or quality-rejected tasks, and only sometimes allows a second attempt.

What it costs to start

Most people already own the computer and internet connection. Mindrift's page says it deducts nothing itself, but Payoneer or Tipalti may charge withdrawal fees.

Where to find rating work

Mindrift offers all-access tasks that need no application, though they tend to pay less than specialized projects. Mindrift's restricted-country list names the U.S. Minor Outlying Islands but not the 50 states. Alignerr's job listings are location-specific, and some are open to U.S.-based workers. Most Alignerr projects use Labelbox to label, rate, create and align data, so rating is one task type among several.

Requirements and rules in the U.S.

Outlier states that workers are independent contractors, and Mindrift says contributors are freelancers with no tax withholding. The IRS says independent contractors count as self-employed, and self-employment tax usually applies once net earnings reach $400. Per IRS Topic 554, the rate is 12.4% for social security plus 2.9% for Medicare, applied to 92.35% of net earnings, and half of the tax is deductible.

The IRS says estimated payments apply if you expect to owe $1,000 or more when you file. Paying 90% of this year's tax or 100% of last year's avoids the underpayment penalty, and the payments cover self-employment tax too. State income tax rules vary, so check your state revenue agency. For general side-income tax, see the IRS gig economy tax center.

Risks and red flags

Platforms can end your access. DataAnnotation's terms let it terminate services for any or no reason.

Worker classification is being litigated in California. Scale AI, which operates Outlier, has a $12.5 million settlement with preliminary approval as of August 2026, and Scale denied the claims. Bloomberg Law reported that Surge AI was sued in California state court in May 2025 over classifying annotators as contractors. That report doesn't say how the suit ended.

Who this fits, and who should skip it

Good fit

• You meet the education minimums that DataAnnotation and Outlier set.
• You want to pick hours day to day. Outlier says there are no minimum hours and no obligation to accept projects.
• You can do careful judgment work, such as comparing AI responses and rating them on accuracy and helpfulness.
• You're fine with per-task pay that rises or falls with how fast you work.

Skip it if

• You need to know your pay before you start. Outlier says available projects vary and may not match your field.
• You can't risk a single attempt. DataAnnotation's Starter Assessment can be taken only once, and applicants may not hear back.
• You want a set price per visit in person, as with dog walking or house cleaning.

How to start

1. Gather a government photo ID, your resume and a mobile phone. Outlier's application asks for ID, phone, resume and LinkedIn.
2. Take DataAnnotation's Starter Assessment at dataannotation.tech when you have an uninterrupted hour. It can be taken only once.
3. Browse Alignerr's open roles and apply to the ones that match your skills. Each listing shows requirements and expected earnings.
4. Try Mindrift's all-access tasks if you want to start without an application.
5. Read each project's pay basis before you begin, then use the hourly earnings guide to count unpaid time.
6. Set aside money for tax. The IRS's estimated taxes page explains when quarterly payments apply.