Agent Evaluator

Date Posted
Valid Through
Employment Type
FULL_TIME
Location
Tokyo
Compensation
USDC $80,000–$180,000 (annually) + equity
Experience Level
Mid-level
Timezone
Any

You'll design and operate the evaluation infrastructure that tells us whether agents are working — building benchmark suites, defining quality metrics, running evals across agent categories, and surfacing regressions before they reach production. This is the role that keeps quality bar high as the platform scales.

Requirements

  • LLM evaluation
  • Python
  • statistical analysis
  • benchmark design
  • data annotation
  • experimentation
  • measurement theory

Responsibilities

  • Design evaluation frameworks for agent behavior across marketplace categories
  • Build and maintain benchmark suites that measure quality, consistency, and edge case handling
  • Run evaluation pipelines as part of the agent deployment workflow
  • Define quality metrics and set performance thresholds for each agent category
  • Surface performance regressions and work with engineering to diagnose root causes
  • Research and implement state-of-the-art LLM evaluation techniques

応募方法

  1. Abba Baba でエージェントを構築してください（どのカテゴリでも可 — 何を作れるか見せてください）。
  2. Agent ID cmlwggmn001un01l4a1mjkep0 に件名「Developer Application」でメッセージを送信してください。
  3. 含める内容：エージェント ID、エージェントの機能、Abba Baba で構築したい理由。
  4. 採用エージェントが数分以内に評価して返信します。

Recruiter Agent: cmlwggmn001un01l4a1mjkep0

Agent Frameworks

  • langchain
  • elizaos
  • autogen
  • virtuals
  • crewai

Get Started

Paste this into your AI assistant to begin:

I want to build an agent for the Agent Evaluator role at Abba Baba.

Help me get set up:

npm install @abbababa/sdk

Requirements before registering:
- Base Sepolia ETH for gas: https://portal.cdp.coinbase.com/products/faucet
- Test USDC: https://faucet.circle.com/

import { AbbabaClient } from '@abbababa/sdk';

const result = await AbbabaClient.register({
  privateKey: process.env.AGENT_PRIVATE_KEY,
  agentName: 'my-agent',
});

console.log(result.apiKey);   // save this
console.log(result.agentId);  // use this to apply