Scoring

How Kimi evaluates conversation transcripts.

Pipeline

After the conversation closes, the full transcript goes to Kimi (Moonshot AI). Kimi returns a score between 0 and 1. Scores at or above KIMI_PASS_THRESHOLD trigger releasePayout. Scores below trigger a re-prompt.

The conversational LLM and Kimi are separate systems:

System	Input	Output
Conversational LLM	Advertiser brief + group context + conversation history	Next agent turn
Kimi	Full transcript + brief + group context	Quality score (0–1)

What Kimi evaluates

Response entropy
Did the user give varied, specific answers or pattern responses? Single-word replies, identical response structure across turns, and copy-paste answers score low. Responses with specific terminology, examples, or reasoning score higher.

Contextual coherence
Do the answers make sense given the questions asked? Does the user’s stated knowledge align with the group they’re joining? Responses that ignore the question or answer a different question score low.

Engagement trajectory
Did the user engage more on the follow-up, or give the same response quality across all turns? Humans typically elaborate when probed. Uniform response quality across turns regardless of question depth is a bot signal.

Timing
Response times are logged per turn. Responses under 1–2 seconds across all turns score lower. Human response times show natural variation.

Thresholds

Score	Outcome
≥ 0.7 (default)	Pass — releasePayout fires, user admitted
0.4–0.69	Re-prompt — user gets one attempt with a different question
< 0.4	Fail on re-prompt — user denied, 24h cooldown

KIMI_PASS_THRESHOLD is configurable per campaign. Higher thresholds produce fewer passing verifications with higher average quality. Lower thresholds pass more verifications with looser quality filtering.

Kimi outage handling

If Kimi is unavailable, the agent server retries up to 4 times with exponential backoff. If all 4 retries fail:

• Verification marked SCORING_UNAVAILABLE
• User admitted to group
• No payout released
• No cooldown applied
• Advertiser is not billed

What Kimi does not evaluate

Kimi does not assess whether answers are factually correct. For agent acquisition campaigns (no correct answer), Kimi scores engagement quality only — whether the response is genuine human engagement, not whether the user picked a specific option. For RLHF campaigns, Kimi scores whether the response reflects genuine reasoning, not whether the preference label matches a ground-truth answer.