How the Multi-Agent Assessment Works

This page explains the full structure of the multi-agent assessment JSON, including:

Overall JSON schema
Field-by-field definitions
Types, constraints, and scoring logic

Candidate Question Evaluation JSON Schema

Each item in the schema represents the evaluation of one candidate answering one interview question.

The examples below use shortened content for readability. Your actual JSON will include full summaries and complete rubric criteria.

Top-Level Structure

"agent_response_summary": [
  {
    "question": { ... },
    "agents": [ ... ],
    "normalization": { ... },
    "mapping": { ... }
  }
  // Additional questions...
]

Question Object

Metadata for the question being evaluated.

"question": {
  "text": "How did you design the retry mechanism for failed messages?"
}

Fields

string

required

Full question text presented to the candidate.

Agents Object

The agents array contains all evaluator personas (AI Researchers, SMEs, Project Managers, etc.) and their scores for this question.

"agents": [
  {
    "id": "software_developer_1",
    "name": "Harry",
    "role": "Software Developer",
    "total_weighted_score": 100,
    "maximum_score": 100,
    "criteria": { ... },
    "step6_summary": { ... }
  }
  // 5 additional agents
]

Fields

string

Index string of the agent, can use mapping object to map.

string

Agent persona name.

string

Evaluator persona type (e.g., AI Researcher, Subject Matter Expert, Project Manager). Dynamic based on job description or interview.

object

Scores for each rubric criterion for this agent.

number

Final score for this agent for this question:

Σ (score_k * weight_k)

number

Maximum possible score for this question of this agent for all criteria:

Σ (weight_k * 5)

object

Narrative summary for each criterion and overall performance.

Criteria Object

Each agent’s criteria object represents the merged rubric applied to this question.

"criteria": {
  "project_detail_explanation": { "score": 5, "weight": 4, "weighted": 20, "maximum_score": 20 },
  "relevance_to_software_development": { "score": 5, "weight": 4, "weighted": 20, "maximum_score": 20 },
  "project_complexity": { "score": 5, "weight": 4, "weighted": 20, "maximum_score": 20 },
  "technical_skills": { "score": 5, "weight": 4, "weighted": 20, "maximum_score": 20 },
  "problemsolving_approach": { "score": 5, "weight": 4, "weighted": 20, "maximum_score": 20 }
}

Criterion Fields

number

Agent’s rating on a 0–5 scale
(0 = no evidence, 5 = strongest demonstration).

number

Relative importance of the criterion in the rubric.

number

Computed as:

score * weight

number

Computed as:

5 * weight

Criterion keys are generated from the merged rubric and are unique to the question being evaluated.

Step 6 Summary Object

Human-readable interpretation of the agent’s evaluation.

"step6_summary": {
  "summary": "The candidate demonstrated a strong grasp of reinforcement learning concepts...",
  "problem_solving_skills": "The candidate clearly identified potential problems...",
  "workflow_breakdown_clarity": "The candidate provided exceptional clarity...",
  "experience_in_software_development": "The candidate demonstrated extensive experience...",
  "understanding_of_custom_rl_environments": "The candidate exhibited expert understanding...",
  "understanding_of_rl_concepts": "The candidate demonstrated comprehensive knowledge..."
}

Fields

string

High-level summary of the candidate’s performance on this question.

string

Criterion-level narrative explanation aligned with the rubric.

Normalization Object

The normalization object aggregates all agent-level scores for this question, including raw totals, maximum possible values, and normalized outputs for each individual agent and each role group.

"normalization": {
        "maximum_possible": 600,
        "final_scores": {
          "backend_developer_1": 30,
          "backend_developer_1_max": 100,
          "backend_developer_2": 30,
          "backend_developer_2_max": 100,
          "technical_lead_1": 36,
          "technical_lead_1_max": 100,
          "technical_lead_2": 36,
          "technical_lead_2_max": 100,
          "system_architect_1": 30,
          "system_architect_1_max": 100,
          "system_architect_2": 30,
          "system_architect_2_max": 100
        },
        "backend_developer_normalized": 0.3,
        "technical_lead_normalized": 0.36,
        "system_architect_normalized": 0.3,
        "backend_developer_1_normalized": 0.3,
        "backend_developer_2_normalized": 0.3,
        "technical_lead_1_normalized": 0.36,
        "technical_lead_2_normalized": 0.36,
        "system_architect_1_normalized": 0.3,
        "system_architect_2_normalized": 0.3
      }

Fields

Maximum Score

number

Total possible score across all agents for this question.

Final Scores Each agent has two values:

role_index -> actual weighted score
role_index_max → maximum possible score

number

Total weighted score from Backend Developer 1.

number

Maximum possible score for Backend Developer 1.

number

Total weighted score from Backend Developer 2.

number

Maximum score for Backend Developer 2.

number

Total weighted score from Technical Lead 1.

number

Maximum score for Technical Lead 1.

number

Total weighted score from Technical Lead 2.

number

Maximum score for Technical Lead 2.

number

Total weighted score from System Architect 1.

number

Maximum score for System Architect 1.

number

Total weighted score from System Architect 2.

number

Maximum score for System Architect 2.

Normalized Scores (Per Role Group) These fields aggregate normalized results by role, not individual agent:

role_normalized = (sum of all agent scores in role) / (sum of all role max scores)

number

Normalized score across all Backend Developer agents.

number

Normalized score across all Technical Lead agents.

number

Normalized score across all System Architect agents.

Normalized Scores (Per Individual Agent) Each agent also receives an individual normalized value:

agent_normalized = agent_score / agent_max_score

number

Normalized score for Backend Developer 1.

number

Normalized score for Backend Developer 2.

number

Normalized score for Technical Lead 1.

number

Normalized score for Technical Lead 2.

number

Normalized score for System Architect 1.

number

Normalized score for System Architect 2.

Mapping Object

The mapping object provides a consistent translation layer between internal keys and their human-readable labels. This ensures stable programmatic references while keeping UI and summaries clear and friendly.

"mapping": {
  "agent": {
    "ai_researcher": "AI Researcher",
    "project_manager": "Project Manager",
    "subject_matter_expert": "Subject Matter Expert"
  },
  "criteria": {
    "problem_identification": "Problem Identification",
    "technical_complexity": "Technical Complexity"
    // Additional criteria keys...
  }
}

Fields

object

Maps internal agent-role identifiers to human-readable labels.

object

Maps internal criterion keys to their display names in summaries, UI, and reports.

Use the mapping object when rendering UI labels or producing human-readable summaries to avoid relying on raw internal keys.

Getting Started

Integration API

Embedding

How the Multi-Agent Assessment Works

Candidate Question Evaluation JSON Schema

Top-Level Structure

Question Object

Fields

Agents Object

Fields

Criteria Object

Criterion Fields

Step 6 Summary Object

Fields

Normalization Object

Fields

Mapping Object

Fields

​Candidate Question Evaluation JSON Schema

​Top-Level Structure

​Question Object

​Fields

​Agents Object

​Fields

​Criteria Object

​Criterion Fields

​Step 6 Summary Object

​Fields

​Normalization Object

​Fields

​Mapping Object

​Fields

Candidate Question Evaluation JSON Schema

Top-Level Structure

Question Object

Fields

Agents Object

Fields

Criteria Object

Criterion Fields

Step 6 Summary Object

Fields

Normalization Object

Fields

Mapping Object

Fields