> ## Documentation Index
> Fetch the complete documentation index at: https://docs.qode.world/llms.txt
> Use this file to discover all available pages before exploring further.

# How the Multi-Agent Assessment Works

> A detailed explanation of the JSON structure, field definitions, and scoring logic used in the multi-agent candidate evaluation.

This page explains the full structure of the **multi-agent assessment JSON**, including:

* Overall JSON schema
* Field-by-field definitions
* Types, constraints, and scoring logic

## Candidate Question Evaluation JSON Schema

Each item in the schema represents the evaluation of **one candidate** answering **one interview question**.

<Info>
  The examples below use shortened content for readability. Your actual JSON will include full summaries and complete rubric criteria.
</Info>

## Top-Level Structure

```json theme={null}
"agent_response_summary": [
  {
    "question": { ... },
    "agents": [ ... ],
    "normalization": { ... },
    "mapping": { ... }
  }
  // Additional questions...
]
```

***

## Question Object

Metadata for the question being evaluated.

```json theme={null}
"question": {
  "text": "How did you design the retry mechanism for failed messages?"
}
```

### Fields

<ParamField path="text" type="string" required>
  Full question text presented to the candidate.
</ParamField>

***

## Agents Object

The `agents` array contains all evaluator personas (AI Researchers, SMEs, Project Managers, etc.) and their scores for this question.

```json theme={null}
"agents": [
  {
    "id": "software_developer_1",
    "name": "Harry",
    "role": "Software Developer",
    "total_weighted_score": 100,
    "maximum_score": 100,
    "criteria": { ... },
    "step6_summary": { ... }
  }
  // 5 additional agents
]
```

### Fields

<ParamField path="id" type="string">
  Index string of the agent, can use `mapping` object to map.
</ParamField>

<ParamField path="name" type="string">
  Agent persona name.
</ParamField>

<ParamField path="role" type="string">
  Evaluator persona type (e.g., AI Researcher, Subject Matter Expert, Project Manager). Dynamic based on job description or interview.
</ParamField>

<ParamField path="criteria" type="object">
  Scores for each rubric criterion for this agent.
</ParamField>

<ParamField path="total_weighted_score" type="number">
  Final score for this agent for this question:

  ```
  Σ (score_k * weight_k)
  ```
</ParamField>

<ParamField path="maximum_score" type="number">
  Maximum possible score for this question of this agent for all criteria:

  ```
  Σ (weight_k * 5)
  ```
</ParamField>

<ParamField path="step6_summary" type="object">
  Narrative summary for each criterion and overall performance.
</ParamField>

***

## Criteria Object

Each agent's `criteria` object represents the merged rubric applied to this question.

```json theme={null}
"criteria": {
  "project_detail_explanation": { "score": 5, "weight": 4, "weighted": 20, "maximum_score": 20 },
  "relevance_to_software_development": { "score": 5, "weight": 4, "weighted": 20, "maximum_score": 20 },
  "project_complexity": { "score": 5, "weight": 4, "weighted": 20, "maximum_score": 20 },
  "technical_skills": { "score": 5, "weight": 4, "weighted": 20, "maximum_score": 20 },
  "problemsolving_approach": { "score": 5, "weight": 4, "weighted": 20, "maximum_score": 20 }
}
```

### Criterion Fields

<ParamField path="<criterion_key>.score" type="number">
  Agent's rating on a **0–5** scale\
  (0 = no evidence, 5 = strongest demonstration).
</ParamField>

<ParamField path="<criterion_key>.weight" type="number">
  Relative importance of the criterion in the rubric.
</ParamField>

<ParamField path="<criterion_key>.weighted" type="number">
  Computed as:

  ```
  score * weight
  ```
</ParamField>

<ParamField path="<criterion_key>.maximum_score" type="number">
  Computed as:

  ```
  5 * weight
  ```
</ParamField>

<Note>
  Criterion keys are generated from the merged rubric and are unique to the question being evaluated.
</Note>

***

## Step 6 Summary Object

Human-readable interpretation of the agent's evaluation.

```json theme={null}
"step6_summary": {
  "summary": "The candidate demonstrated a strong grasp of reinforcement learning concepts...",
  "problem_solving_skills": "The candidate clearly identified potential problems...",
  "workflow_breakdown_clarity": "The candidate provided exceptional clarity...",
  "experience_in_software_development": "The candidate demonstrated extensive experience...",
  "understanding_of_custom_rl_environments": "The candidate exhibited expert understanding...",
  "understanding_of_rl_concepts": "The candidate demonstrated comprehensive knowledge..."
}
```

### Fields

<ParamField path="summary" type="string">
  High-level summary of the candidate's performance on this question.
</ParamField>

<ParamField path="<criterion_key>" type="string">
  Criterion-level narrative explanation aligned with the rubric.
</ParamField>

***

## Normalization Object

The `normalization` object aggregates **all agent-level scores for this question**, including raw totals, maximum possible values, and normalized outputs for each individual agent and each role group.

```json theme={null}
"normalization": {
        "maximum_possible": 600,
        "final_scores": {
          "backend_developer_1": 30,
          "backend_developer_1_max": 100,
          "backend_developer_2": 30,
          "backend_developer_2_max": 100,
          "technical_lead_1": 36,
          "technical_lead_1_max": 100,
          "technical_lead_2": 36,
          "technical_lead_2_max": 100,
          "system_architect_1": 30,
          "system_architect_1_max": 100,
          "system_architect_2": 30,
          "system_architect_2_max": 100
        },
        "backend_developer_normalized": 0.3,
        "technical_lead_normalized": 0.36,
        "system_architect_normalized": 0.3,
        "backend_developer_1_normalized": 0.3,
        "backend_developer_2_normalized": 0.3,
        "technical_lead_1_normalized": 0.36,
        "technical_lead_2_normalized": 0.36,
        "system_architect_1_normalized": 0.3,
        "system_architect_2_normalized": 0.3
      }
```

### Fields

**Maximum Score**

<ParamField path="maximum_possible" type="number">
  Total possible score across all agents for this question.
</ParamField>

**Final Scores**

Each agent has two values:

* role\_index -> actual weighted score
* role\_index\_max → maximum possible score

<ParamField path="final_scores.backend_developer_1" type="number">
  Total weighted score from Backend Developer 1.
</ParamField>

<ParamField path="final_scores.backend_developer_1_max" type="number">
  Maximum possible score for Backend Developer 1.
</ParamField>

<ParamField path="final_scores.backend_developer_2" type="number">
  Total weighted score from Backend Developer 2.
</ParamField>

<ParamField path="final_scores.backend_developer_2_max" type="number">
  Maximum score for Backend Developer 2.
</ParamField>

<ParamField path="final_scores.technical_lead_1" type="number">
  Total weighted score from Technical Lead 1.
</ParamField>

<ParamField path="final_scores.technical_lead_1_max" type="number">
  Maximum score for Technical Lead 1.
</ParamField>

<ParamField path="final_scores.technical_lead_2" type="number">
  Total weighted score from Technical Lead 2.
</ParamField>

<ParamField path="final_scores.technical_lead_2_max" type="number">
  Maximum score for Technical Lead 2.
</ParamField>

<ParamField path="final_scores.system_architect_1" type="number">
  Total weighted score from System Architect 1.
</ParamField>

<ParamField path="final_scores.system_architect_1_max" type="number">
  Maximum score for System Architect 1.
</ParamField>

<ParamField path="final_scores.system_architect_2" type="number">
  Total weighted score from System Architect 2.
</ParamField>

<ParamField path="final_scores.system_architect_2_max" type="number">
  Maximum score for System Architect 2.
</ParamField>

**Normalized Scores (Per Role Group)**

These fields aggregate normalized results by **role**, not individual agent:

```
role_normalized = (sum of all agent scores in role) / (sum of all role max scores)
```

<ParamField path="backend_developer_normalized" type="number">
  Normalized score across all Backend Developer agents.
</ParamField>

<ParamField path="technical_lead_normalized" type="number">
  Normalized score across all Technical Lead agents.
</ParamField>

<ParamField path="system_architect_normalized" type="number">
  Normalized score across all System Architect agents.
</ParamField>

**Normalized Scores (Per Individual Agent)**

Each agent also receives an individual normalized value:

```
agent_normalized = agent_score / agent_max_score
```

<ParamField path="backend_developer_1_normalized" type="number">
  Normalized score for Backend Developer 1.
</ParamField>

<ParamField path="backend_developer_2_normalized" type="number">
  Normalized score for Backend Developer 2.
</ParamField>

<ParamField path="technical_lead_1_normalized" type="number">
  Normalized score for Technical Lead 1.
</ParamField>

<ParamField path="technical_lead_2_normalized" type="number">
  Normalized score for Technical Lead 2.
</ParamField>

<ParamField path="system_architect_1_normalized" type="number">
  Normalized score for System Architect 1.
</ParamField>

<ParamField path="system_architect_2_normalized" type="number">
  Normalized score for System Architect 2.
</ParamField>

## Mapping Object

The `mapping` object provides a consistent translation layer between **internal keys** and their **human-readable labels**. This ensures stable programmatic references while keeping UI and summaries clear and friendly.

```json theme={null}
"mapping": {
  "agent": {
    "ai_researcher": "AI Researcher",
    "project_manager": "Project Manager",
    "subject_matter_expert": "Subject Matter Expert"
  },
  "criteria": {
    "problem_identification": "Problem Identification",
    "technical_complexity": "Technical Complexity"
    // Additional criteria keys...
  }
}
```

### Fields

<ParamField path="agent" type="object">
  Maps internal agent-role identifiers to human-readable labels.
</ParamField>

<ParamField path="criteria" type="object">
  Maps internal criterion keys to their display names in summaries, UI, and reports.
</ParamField>

<Tip>
  Use the **mapping object** when rendering UI labels or producing human-readable summaries to avoid relying on raw internal keys.
</Tip>