Design CodeSignal Online Assessment Tool: System Design & Architecture

Scope

Back-of-the-Envelope Estimates

• Assume ~2M assessments/year; peak hiring seasons reach ~5K assessments running at the same time
• Per assessment: ~3 problems over 90 minutes, with ~15 graded submissions plus ~30 quick "run" executions → ~45 executions per assessment
• Peak execution rate: 5K concurrent assessments x 45 executions ÷ 90 minutes ≈ 2,500/min ≈ 42 executions/sec on average - but submissions cluster near deadlines, so bursts run 5-10x → design for ~400/sec
• Each execution costs real compute: ~1 vCPU and 512MB for up to ~10 seconds → at a 400/sec burst with ~5s average runtime, that is ~2,000 sandboxes alive at once ≈ ~2,000 vCPUs of burst capacity, a few hundred at steady state - the fleet size is derived, not guessed
• Storage is a footnote: code is ~10KB per submission → 2M x 45 x 10KB ≈ ~1TB/year of code; logs and artifacts add a few TB more. Nothing here challenges a database
• The punchline of the numbers: 42/sec is laughably small next to Design a Newsfeed's 46K feed writes/sec - but each of these requests costs ~5 CPU-seconds and a fresh sandbox. The load is measured in CPU-seconds and isolation overhead, not requests per second

Data Model & Storage

• Assessments (PostgreSQL): title, problem list, time limit, availability window, settings. Small, relational, created by the thousands - not the millions.
• Problems (PostgreSQL): statement, starter code per language, resource limits, and test_cases as JSONB where each case carries a hidden flag. The hidden ones are the crown jewels - see the note below.
• Sessions (PostgreSQL): one per candidate per assessment - started_at, deadline (computed server-side at start), current code per problem (the durable autosave target), and an event log (joins, saves, submissions) that doubles as the audit trail proctoring would consume.
• Submissions (PostgreSQL): code, language, status, verdict, per-test summary, timing and memory numbers, and a pointer to full logs in object storage. Append-only in spirit: this table is the evidence hiring decisions cite.
• Job Queue (Redis Streams): one entry per execution with a consumer group across the runner fleet. The queue is delivery, not truth - if it were lost, jobs re-enqueue from Submissions rows still marked queued.
• Autosave buffer (Redis): the last few keystrokes' worth of code per session, debounced in, flushed to the Sessions row every few seconds - so a browser crash costs seconds of typing, not minutes.
• Artifacts (Object Storage): stdout/stderr per test, compile output, execution traces - access-controlled, since candidate code is both their intellectual property and personally identifying.

API Design

Low-level Design

Database Schemas (PostgreSQL throughout - the estimates say so)

- Assessments
  assessment_id UUID PRIMARY KEY,
  recruiter_id UUID NOT NULL,
  title VARCHAR(255),
  problem_ids UUID[] NOT NULL,
  time_limit_minutes INT DEFAULT 90,
  window_start TIMESTAMP, window_end TIMESTAMP,
  settings JSONB,                    -- allowed languages, attempts per problem
  created_at TIMESTAMP DEFAULT NOW()

- Problems
  problem_id UUID PRIMARY KEY,
  title VARCHAR(255), statement TEXT,
  starter_code JSONB,                -- { "python": "def solve():...", ... }
  limits JSONB,                      -- { "cpu_seconds": 2, "memory_mb": 512 }
  test_cases JSONB
    -- [{ "input": "...", "expected": "...", "hidden": true }, ...]
    -- hidden cases are NEVER serialized into any client-facing response;
    -- the trust boundary is this API shape, not the UI

- Sessions (one per candidate per assessment)
  session_id UUID PRIMARY KEY,
  assessment_id UUID NOT NULL,
  candidate_email VARCHAR(255),
  started_at TIMESTAMP NOT NULL,
  deadline TIMESTAMP NOT NULL,       -- server clock; the only timer that counts
  code JSONB,                        -- durable autosave target per problem
  event_log JSONB,                   -- joins, saves, submits: the audit trail
  UNIQUE (assessment_id, candidate_email)

- Submissions (append-only in spirit: the evidence table)
  submission_id UUID PRIMARY KEY,
  session_id UUID NOT NULL,
  problem_id UUID NOT NULL,
  mode run_mode_enum,                -- 'run' (visible tests) | 'submit' (all)
  language VARCHAR(20), code TEXT,
  status status_enum DEFAULT 'queued',  -- queued|running|done
  verdict verdict_enum,              -- accepted|wrong_answer|tle|mle|rte|ce
  tests_passed INT, tests_total INT,
  cpu_time_ms INT, memory_mb INT,
  logs_ref VARCHAR(512),             -- object-storage pointer
  submitted_at TIMESTAMP DEFAULT NOW(),
  INDEX (session_id, problem_id, submitted_at DESC)

Redis

- Job queue (Redis Streams)
  XADD exec-jobs * submission_id ... language ... limits ...
  Consumer group "runners": each job delivered to exactly one runner;
  leases + XAUTOCLAIM reclaim jobs from dead runners.
  The queue is DELIVERY, not truth - rows with status=queued re-enqueue
  if the stream is ever lost.

- Autosave buffer
  "code:{session_id}:{problem_id}" -> latest draft
  flushed to the Sessions row every few seconds

Object storage
  /artifacts/{submission_id}/compile.txt, test_{n}.stdout, test_{n}.stderr
  -- access-controlled: candidate code is their IP and their PII

{
  "session_id": "sess_8f3e2d1c",
  "problem_id": "prob_two_sum",
  "mode": "submit",
  "language": "python",
  "code": "def solve(nums, target):\n    seen = {}\n    ..."
}

{
  "submission_id": "sub_7a2b8f4e",
  "status": "queued",
  "queue_position": 12
}

// Returns immediately - execution is asynchronous behind the queue.
// queue_position is honest feedback during deadline spikes: candidates
// see movement instead of a spinner. A submission after the session's
// server-side deadline is rejected here with 410, whatever the client's
// timer claimed.

{ "type": "status", "submission_id": "sub_7a2b8f4e",
  "status": "running", "current_test": 3, "total_tests": 8 }

// ...then, a few seconds later:

{ "type": "verdict", "submission_id": "sub_7a2b8f4e",
  "verdict": "wrong_answer",
  "tests_passed": 6, "tests_total": 8,
  "cpu_time_ms": 340, "memory_mb": 41 }

{
  "submission_id": "sub_7a2b8f4e",
  "verdict": "wrong_answer",
  "tests": [
    { "test": 1, "hidden": false, "passed": true,
      "input": "[2,7,11,15], 9", "expected": "[0,1]", "output": "[0,1]",
      "cpu_time_ms": 12 },
    { "test": 7, "hidden": true, "passed": false,
      "cpu_time_ms": 45 }
  ],
  "logs_url": "https://artifacts.example.com/sub_7a2b8f4e/...(signed)"
}

// The trust boundary in action: visible tests return full detail
// (that is where feedback teaches); hidden tests return pass/fail and
// timing ONLY - never the input or expected output, even on failure.

{
  "assessment_id": "a_9d4f1a2b",
  "candidates": [
    {
      "candidate_email": "dev@example.com",
      "completed_in_minutes": 74,
      "total_score": 2.5,
      "problems": [
        { "problem_id": "prob_two_sum", "best_verdict": "accepted",
          "attempts": 3, "best_cpu_time_ms": 120 },
        { "problem_id": "prob_intervals", "best_verdict": "wrong_answer",
          "attempts": 7, "tests_passed": "6/8" }
      ]
    }
  ]
}

// Computed on read - at this data size, aggregation needs an index,
// not a pipeline. Attempt counts and timing come straight from the
// Submissions evidence table.

// 410 Gone - the server-side deadline has passed
{
  "error": "session_expired",
  "message": "The assessment time limit has been reached"
}

// 429 - submission rate limit (per session)
{
  "error": "rate_limited",
  "message": "Please wait before submitting again",
  "retry_after_seconds": 5
}
// Rate limiting per session blunts both abuse and accidental
// submit-spamming during deadline panic.

Concluding Discussion

Critical Knowledge