Skip to main content

Documentation Index

Fetch the complete documentation index at: https://docs.qode.world/llms.txt

Use this file to discover all available pages before exploring further.

This page explains the full structure of the multi-agent assessment JSON, including:
  • Overall JSON schema
  • Field-by-field definitions
  • Types, constraints, and scoring logic

​
Candidate Question Evaluation JSON Schema

Each item in the schema represents the evaluation of one candidate answering one interview question.
The examples below use shortened content for readability. Your actual JSON will include full summaries and complete rubric criteria.

​
Top-Level Structure

"agent_response_summary": [
  {
    "question": { ... },
    "agents": [ ... ],
    "normalization": { ... },
    "mapping": { ... }
  }
  // Additional questions...
]

​
Question Object

Metadata for the question being evaluated. 
"question": {
  "text": "How did you design the retry mechanism for failed messages?"
}

​
Fields

​
text
string
required
Full question text presented to the candidate.

​
Agents Object

The agents array contains all evaluator personas (AI Researchers, SMEs, Project Managers, etc.) and their scores for this question. 
"agents": [
  {
    "id": "software_developer_1",
    "name": "Harry",
    "role": "Software Developer",
    "total_weighted_score": 100,
    "maximum_score": 100,
    "criteria": { ... },
    "step6_summary": { ... }
  }
  // 5 additional agents
]

​
Fields

​
id
string
Index string of the agent, can use mapping object to map.
​
name
string
Agent persona name.
​
role
string
Evaluator persona type (e.g., AI Researcher, Subject Matter Expert, Project Manager). Dynamic based on job description or interview.
​
criteria
object
Scores for each rubric criterion for this agent.
​
total_weighted_score
number
Final score for this agent for this question:
Σ (score_k * weight_k)
​
maximum_score
number
Maximum possible score for this question of this agent for all criteria:
Σ (weight_k * 5)
​
step6_summary
object
Narrative summary for each criterion and overall performance.

​
Criteria Object

Each agent’s criteria object represents the merged rubric applied to this question. 
"criteria": {
  "project_detail_explanation": { "score": 5, "weight": 4, "weighted": 20, "maximum_score": 20 },
  "relevance_to_software_development": { "score": 5, "weight": 4, "weighted": 20, "maximum_score": 20 },
  "project_complexity": { "score": 5, "weight": 4, "weighted": 20, "maximum_score": 20 },
  "technical_skills": { "score": 5, "weight": 4, "weighted": 20, "maximum_score": 20 },
  "problemsolving_approach": { "score": 5, "weight": 4, "weighted": 20, "maximum_score": 20 }
}

​
Criterion Fields

​
<criterion_key>.score
number
Agent’s rating on a 0–5 scale
(0 = no evidence, 5 = strongest demonstration).
​
<criterion_key>.weight
number
Relative importance of the criterion in the rubric.
​
<criterion_key>.weighted
number
Computed as:
score * weight
​
<criterion_key>.maximum_score
number
Computed as:
5 * weight
Criterion keys are generated from the merged rubric and are unique to the question being evaluated.

​
Step 6 Summary Object

Human-readable interpretation of the agent’s evaluation. 
"step6_summary": {
  "summary": "The candidate demonstrated a strong grasp of reinforcement learning concepts...",
  "problem_solving_skills": "The candidate clearly identified potential problems...",
  "workflow_breakdown_clarity": "The candidate provided exceptional clarity...",
  "experience_in_software_development": "The candidate demonstrated extensive experience...",
  "understanding_of_custom_rl_environments": "The candidate exhibited expert understanding...",
  "understanding_of_rl_concepts": "The candidate demonstrated comprehensive knowledge..."
}

​
Fields

​
summary
string
High-level summary of the candidate’s performance on this question.
​
<criterion_key>
string
Criterion-level narrative explanation aligned with the rubric.

​
Normalization Object

The normalization object aggregates all agent-level scores for this question, including raw totals, maximum possible values, and normalized outputs for each individual agent and each role group. 
"normalization": {
        "maximum_possible": 600,
        "final_scores": {
          "backend_developer_1": 30,
          "backend_developer_1_max": 100,
          "backend_developer_2": 30,
          "backend_developer_2_max": 100,
          "technical_lead_1": 36,
          "technical_lead_1_max": 100,
          "technical_lead_2": 36,
          "technical_lead_2_max": 100,
          "system_architect_1": 30,
          "system_architect_1_max": 100,
          "system_architect_2": 30,
          "system_architect_2_max": 100
        },
        "backend_developer_normalized": 0.3,
        "technical_lead_normalized": 0.36,
        "system_architect_normalized": 0.3,
        "backend_developer_1_normalized": 0.3,
        "backend_developer_2_normalized": 0.3,
        "technical_lead_1_normalized": 0.36,
        "technical_lead_2_normalized": 0.36,
        "system_architect_1_normalized": 0.3,
        "system_architect_2_normalized": 0.3
      }

​
Fields

Maximum Score
​
maximum_possible
number
Total possible score across all agents for this question.
Final Scores Each agent has two values:
  • role_index -> actual weighted score
  • role_index_max → maximum possible score
​
final_scores.backend_developer_1
number
Total weighted score from Backend Developer 1.
​
final_scores.backend_developer_1_max
number
Maximum possible score for Backend Developer 1.
​
final_scores.backend_developer_2
number
Total weighted score from Backend Developer 2.
​
final_scores.backend_developer_2_max
number
Maximum score for Backend Developer 2.
​
final_scores.technical_lead_1
number
Total weighted score from Technical Lead 1.
​
final_scores.technical_lead_1_max
number
Maximum score for Technical Lead 1.
​
final_scores.technical_lead_2
number
Total weighted score from Technical Lead 2.
​
final_scores.technical_lead_2_max
number
Maximum score for Technical Lead 2.
​
final_scores.system_architect_1
number
Total weighted score from System Architect 1.
​
final_scores.system_architect_1_max
number
Maximum score for System Architect 1.
​
final_scores.system_architect_2
number
Total weighted score from System Architect 2.
​
final_scores.system_architect_2_max
number
Maximum score for System Architect 2.
Normalized Scores (Per Role Group) These fields aggregate normalized results by role, not individual agent: 
role_normalized = (sum of all agent scores in role) / (sum of all role max scores)
​
backend_developer_normalized
number
Normalized score across all Backend Developer agents.
​
technical_lead_normalized
number
Normalized score across all Technical Lead agents.
​
system_architect_normalized
number
Normalized score across all System Architect agents.
Normalized Scores (Per Individual Agent) Each agent also receives an individual normalized value: 
agent_normalized = agent_score / agent_max_score
​
backend_developer_1_normalized
number
Normalized score for Backend Developer 1.
​
backend_developer_2_normalized
number
Normalized score for Backend Developer 2.
​
technical_lead_1_normalized
number
Normalized score for Technical Lead 1.
​
technical_lead_2_normalized
number
Normalized score for Technical Lead 2.
​
system_architect_1_normalized
number
Normalized score for System Architect 1.
​
system_architect_2_normalized
number
Normalized score for System Architect 2.

​
Mapping Object

The mapping object provides a consistent translation layer between internal keys and their human-readable labels. This ensures stable programmatic references while keeping UI and summaries clear and friendly. 
"mapping": {
  "agent": {
    "ai_researcher": "AI Researcher",
    "project_manager": "Project Manager",
    "subject_matter_expert": "Subject Matter Expert"
  },
  "criteria": {
    "problem_identification": "Problem Identification",
    "technical_complexity": "Technical Complexity"
    // Additional criteria keys...
  }
}

​
Fields

​
agent
object
Maps internal agent-role identifiers to human-readable labels.
​
criteria
object
Maps internal criterion keys to their display names in summaries, UI, and reports.
Use the mapping object when rendering UI labels or producing human-readable summaries to avoid relying on raw internal keys.