This page explains the full structure of the multi-agent assessment JSON, including:
- Overall JSON schema
- Field-by-field definitions
- Types, constraints, and scoring logic
Candidate Question Evaluation JSON Schema
Each item in the schema represents the evaluation of one candidate answering one interview question.
The examples below use shortened content for readability. Your actual JSON will include full summaries and complete rubric criteria.
Top-Level Structure
"agent_response_summary": [
{
"question": { ... },
"agents": [ ... ],
"normalization": { ... },
"mapping": { ... }
}
// Additional questions...
]
Question Object
Metadata for the question being evaluated.
"question": {
"text": "How did you design the retry mechanism for failed messages?"
}
Fields
Full question text presented to the candidate.
Agents Object
The agents array contains all evaluator personas (AI Researchers, SMEs, Project Managers, etc.) and their scores for this question.
"agents": [
{
"id": "software_developer_1",
"name": "Harry",
"role": "Software Developer",
"total_weighted_score": 100,
"maximum_score": 100,
"criteria": { ... },
"step6_summary": { ... }
}
// 5 additional agents
]
Fields
Index string of the agent, can use mapping object to map.
Evaluator persona type (e.g., AI Researcher, Subject Matter Expert, Project Manager). Dynamic based on job description or interview.
Scores for each rubric criterion for this agent.
Final score for this agent for this question: Maximum possible score for this question of this agent for all criteria: Narrative summary for each criterion and overall performance.
Criteria Object
Each agent’s criteria object represents the merged rubric applied to this question.
"criteria": {
"project_detail_explanation": { "score": 5, "weight": 4, "weighted": 20, "maximum_score": 20 },
"relevance_to_software_development": { "score": 5, "weight": 4, "weighted": 20, "maximum_score": 20 },
"project_complexity": { "score": 5, "weight": 4, "weighted": 20, "maximum_score": 20 },
"technical_skills": { "score": 5, "weight": 4, "weighted": 20, "maximum_score": 20 },
"problemsolving_approach": { "score": 5, "weight": 4, "weighted": 20, "maximum_score": 20 }
}
Criterion Fields
Agent’s rating on a 0–5 scale
(0 = no evidence, 5 = strongest demonstration).
Relative importance of the criterion in the rubric.
<criterion_key>.maximum_score
Criterion keys are generated from the merged rubric and are unique to the question being evaluated.
Step 6 Summary Object
Human-readable interpretation of the agent’s evaluation.
"step6_summary": {
"summary": "The candidate demonstrated a strong grasp of reinforcement learning concepts...",
"problem_solving_skills": "The candidate clearly identified potential problems...",
"workflow_breakdown_clarity": "The candidate provided exceptional clarity...",
"experience_in_software_development": "The candidate demonstrated extensive experience...",
"understanding_of_custom_rl_environments": "The candidate exhibited expert understanding...",
"understanding_of_rl_concepts": "The candidate demonstrated comprehensive knowledge..."
}
Fields
High-level summary of the candidate’s performance on this question.
Criterion-level narrative explanation aligned with the rubric.
Normalization Object
The normalization object aggregates all agent-level scores for this question, including raw totals, maximum possible values, and normalized outputs for each individual agent and each role group.
"normalization": {
"maximum_possible": 600,
"final_scores": {
"backend_developer_1": 30,
"backend_developer_1_max": 100,
"backend_developer_2": 30,
"backend_developer_2_max": 100,
"technical_lead_1": 36,
"technical_lead_1_max": 100,
"technical_lead_2": 36,
"technical_lead_2_max": 100,
"system_architect_1": 30,
"system_architect_1_max": 100,
"system_architect_2": 30,
"system_architect_2_max": 100
},
"backend_developer_normalized": 0.3,
"technical_lead_normalized": 0.36,
"system_architect_normalized": 0.3,
"backend_developer_1_normalized": 0.3,
"backend_developer_2_normalized": 0.3,
"technical_lead_1_normalized": 0.36,
"technical_lead_2_normalized": 0.36,
"system_architect_1_normalized": 0.3,
"system_architect_2_normalized": 0.3
}
Fields
Maximum Score
Total possible score across all agents for this question.
- role_index -> actual weighted score
- role_index_max → maximum possible score
final_scores.backend_developer_1
Total weighted score from Backend Developer 1.
final_scores.backend_developer_1_max
Maximum possible score for Backend Developer 1.
final_scores.backend_developer_2
Total weighted score from Backend Developer 2.
final_scores.backend_developer_2_max
Maximum score for Backend Developer 2.
final_scores.technical_lead_1
Total weighted score from Technical Lead 1.
final_scores.technical_lead_1_max
Maximum score for Technical Lead 1.
final_scores.technical_lead_2
Total weighted score from Technical Lead 2.
final_scores.technical_lead_2_max
Maximum score for Technical Lead 2.
final_scores.system_architect_1
Total weighted score from System Architect 1.
final_scores.system_architect_1_max
Maximum score for System Architect 1.
final_scores.system_architect_2
Total weighted score from System Architect 2.
final_scores.system_architect_2_max
Maximum score for System Architect 2.
role_normalized = (sum of all agent scores in role) / (sum of all role max scores)
backend_developer_normalized
Normalized score across all Backend Developer agents.
technical_lead_normalized
Normalized score across all Technical Lead agents.
system_architect_normalized
Normalized score across all System Architect agents.
agent_normalized = agent_score / agent_max_score
backend_developer_1_normalized
Normalized score for Backend Developer 1.
backend_developer_2_normalized
Normalized score for Backend Developer 2.
technical_lead_1_normalized
Normalized score for Technical Lead 1.
technical_lead_2_normalized
Normalized score for Technical Lead 2.
system_architect_1_normalized
Normalized score for System Architect 1.
system_architect_2_normalized
Normalized score for System Architect 2.
Mapping Object
The mapping object provides a consistent translation layer between internal keys and their human-readable labels. This ensures stable programmatic references while keeping UI and summaries clear and friendly.
"mapping": {
"agent": {
"ai_researcher": "AI Researcher",
"project_manager": "Project Manager",
"subject_matter_expert": "Subject Matter Expert"
},
"criteria": {
"problem_identification": "Problem Identification",
"technical_complexity": "Technical Complexity"
// Additional criteria keys...
}
}
Fields
Maps internal agent-role identifiers to human-readable labels.
Maps internal criterion keys to their display names in summaries, UI, and reports.
Use the mapping object when rendering UI labels or producing human-readable summaries to avoid relying on raw internal keys.
