The best predictor of coding is coding.

Software occupies a position in hiring that almost no other occupation shares: the job itself fits inside the assessment. A candidate for a sales or leadership role can only be asked about the work; a candidate for an engineering role can be handed the work itself, real code to read, change, test, and defend, and be observed doing it. That makes the coding assessment the most direct measurement opportunity in selection, and the strange fact of technical hiring is how routinely the opportunity is spent on something else: trivia about language internals, competitive puzzles the job will never pose, or take-home projects produced under conditions nobody saw.

The underlying idea is the oldest one in the field's playbook: if you want to know whether someone can do the work, watch them do a sample of it, a logic running through the century of selection research that Schmidt and Hunter (1998) synthesized. That logic is not in dispute. What the tradition never guaranteed is that any exercise involving code inherits its force.

A coding task predicts only when it is built like an instrument. That means an execution environment every candidate meets identically; a scope constrained enough that performance is attributable to the person; scoring on stated dimensions against anchored criteria; and integrity designed into the task rather than bolted on afterward. Most coding rounds fail at least one of those conditions, and each failure has a signature: the trivia quiz samples the wrong construct, the unbounded take-home measures unmeasurable help, and the pass/fail autograder discards the signal in how the candidate worked, keeping only whether the tests went green. Each failure is a design choice, and each has a design remedy.

Why the sample predicts, and what the modern estimate says

Work samples predict for a reason that is almost embarrassingly direct: they reproduce the behavior they are trying to forecast. Callinan and Robertson (2000), reviewing the work-sample literature in the International Journal of Selection and Assessment, put the case in terms of fidelity, the degree to which an assessment reproduces actual job behavior. A personality inventory asks what a person tends to do; an ability test asks what they can figure out; a work sample watches what they do when handed the job in miniature. Every layer of inference removed is a place where prediction error cannot enter.

The modern estimate of what that directness earns comes from Sackett, Zhang, Berry, and Lievens (2022), the re-analysis that re-estimated the field's meta-analytic validity evidence after applying less aggressive range-restriction corrections than earlier syntheses had. Work samples stand at .33 on the revised table, in the upper tier of selection methods, with the structured interview, at .42, at the top. The full reordering, and what it did and did not change, is traced in the series' flagship review of what predicts job performance.

A generation of practitioners learned larger numbers, and the correction is part of the record. Roth, Bobko, and McFarland (2005), updating the classic work-sample meta-analyses in Personnel Psychology, found those older estimates optimistic, and found work samples still substantial predictors once the evidence is treated more conservatively. The practical conclusion survives the correction intact: sampling the work is among the better-supported moves in selection. It is simply not automatic, and nothing in the coefficient transfers to an exercise that abandons the conditions under which the coefficient was earned.

One more boundary keeps the claim inside the evidence. A work sample reads the skill a person has now, and says much less about trajectory: it under-reads the strong learner who has not yet met your stack, which is why sampling alone serves experienced pools better than early-career ones. The sensible response is composition rather than abandonment. Pair the sample with measures of ability and role-relevant disposition, the combination logic that the science behind modern assessment batteries rests on; in that arrangement the sample carries the criterion, and the other instruments carry the range the sample cannot see.

That gap, between work samples as studied and coding rounds as run, is the subject of everything below. The studied version is standardized, scoped, and scored deliberately. The common version is assembled from puzzle banks and folklore, administered under whatever conditions the calendar allowed, and graded on instinct or on a green checkmark. Same label, different instruments, and only one of them has an evidence base.

A coding assessment measures whatever it samples

A software job is made, hour by hour, of reading code other people wrote, modifying it without breaking what already works, writing tests that pin behavior down, debugging from symptom to cause, and making small design decisions under constraints someone else set. Inventing novel algorithms from a blank page is a real but small fraction of most engineering work. A task built from the job's actual verbs samples the construct the hire is about; a task built from a different set of verbs measures that other thing, however impressive the other thing looks in a leaderboard sense.

Hold the common formats against that standard. The trivia quiz samples recall of language facts, which is cheap to test and cheaply faked by recency of study. The algorithm puzzle samples competitive-programming fluency, a trainable specialty with its own preparation industry, which means over time it measures preparation as much as ability. The multi-day take-home has the opposite defect: high apparent fidelity, but with time, effort, and assistance left as free parameters, the artifact arrives carrying no trail of what produced it. In measurement terms, an unconstrained task has surrendered attribution, and a score you cannot attribute to the candidate is not a score of the candidate.

Figure 1 lays the four formats on the fidelity axis, and the shape of the trade is the point. Fidelity rises to the right, but so does cost, and past a certain point standardization falls away, which is Callinan and Robertson's (2000) other lesson: fidelity and standardization pull against each other, and a work sample needs both. The operating point for most hiring is the bounded work sample, a 30–60 minute task inside a controlled environment: long enough to observe the job's real moves, short enough that the performance is attributable, comparable, and respectful of the candidate's time. The project simulation to its right buys realism at the price of everything the next two sections argue for.

Deriving the task from the job is not a slogan; it is a procedure. List what the role actually requires people to do in a normal week, in the codebase they will actually inherit. Then build the smallest exercise that requires those same moves: a service with a bug and a failing test, a working module that needs one capability added without regressions, a pull request to review and improve. The work sample coding test earns its validity from that derivation, and an organization that cannot say which job behaviors its task samples has not chosen an assessment yet, only a ritual.

Score the how, not only the whether

The default scoring model in technical screening is binary at heart: the tests pass or they do not. Autograding against a hidden suite is fast, cheap, and consistent, and for exactly one dimension of performance it is the right tool. The trouble is what the checkmark cannot see. Two candidates can both reach green, one by forming a hypothesis about the failure, confirming it with a targeted test, and making a minimal, well-named change, the other by mutating code at random until the suite relented. The first candidate is the hire. A pass/fail gate scores them identically, which means the format has paid for a work sample and kept only the fraction of it a multiple-choice item could have delivered.

The remedy is the same structure that rescued the interview: decide the dimensions before anyone is scored, and anchor each level of each dimension to described behavior, so that a 2 means the same thing on Tuesday as it did on Monday and the same thing to both reviewers. A workable set for most engineering roles, offered as illustration rather than canon, is four: correctness (does the change do what was asked, including at the edges), design (is the solution shaped so the next change is easy), readability (does the code explain itself to the colleague who inherits it), and process (how the candidate moved through the problem). Anchored criteria are what separate a rubric from a mood: "readable with effort" and "reads like an explanation" are decisions a reviewer can defend, where a 7-out-of-10 is a feeling with a numeral attached.

Figure 2 sets out the anatomy, and the row shading carries the operational rule: let machines grade what machines grade well, and put human judgment where judgment is the instrument. Correctness against a versioned test suite is the machine's home ground, deterministic and instantly repeatable. Design, readability, and process are judgment calls, and the discipline for judgment is evidence: every rating attached to the thing that justifies it, the diff, the run history, the sequence in which the work took shape. Review of that evidence can be assisted by software; what it cannot be is unattached to anything a second reviewer could inspect and dispute.

The process row deserves the most attention, because it is the one only a work sample can score. Recorded execution environments preserve the making, not just the made: what the candidate ran, when the tests entered, how the solution converged. That log is where the difference between the two green checkmarks above becomes visible, and it is the part of the sample no take-home artifact and no puzzle leaderboard contains.

Anchored scoring also gives the program its quality loop. Because levels are tied to described behavior, two reviewers can score the same submission independently and their disagreement becomes information: a drifting reviewer shows up in the comparison, and an anchor that reviewers read differently gets rewritten. The same property makes the score answerable after the fact. When a declined candidate asks what happened, "level 2 on design, and here is the anchor beside the diff" is an answer; a composite impression is not. Defensibility, seen from outside the organization, is what reliability looks like from inside it.

Same task, same conditions, or the scores do not compare

Every score in hiring is an implicit comparison, and a comparison is only as good as what is held constant across it. The moment two candidates meet different conditions, the conditions enter the score: a different runtime, a more helpful editor, an extra twenty minutes, a stricter interpretation of what "no external help" means. None of that variation is the candidate, and all of it is indistinguishable from the candidate once the number is written next to a name. This is the register in which the environment is the instrument: the task states the question, but the environment decides what a given score can mean.

Figure 3 lists the five pins; two repay a closer look because they feed the objection that follows. The time box is a construct decision disguised as logistics: pace carries signal only when every candidate faces the same clock, and an elastic one grades circumstance instead. Permitted resources are the same decision applied to help: the threat to measurement is not assistance but unstated assistance, so whatever the task admits must be admitted for the whole pool and named before anyone sits. The remaining pins carry their reasons in the figure. Together they amount to the commitments the field's consensus standards ask of any test, standardization and documentation above all (American Educational Research Association, American Psychological Association, & National Council on Measurement in Education, 2014), a shelf the series' capstone on building a standards-based assessment program takes up in full; delivering the environment over the internet has its own published guidelines (International Test Commission, 2006).

The standing objection is realism: working engineers search documentation, lean on autocomplete, and increasingly delegate to AI assistants, so an environment that forbids all of that samples a job nobody does anymore. The objection is right about the job and wrong about the remedy. Fidelity argues for including a resource, standardization argues for pinning it, and both are satisfied by pinning it open: allow the resource for every candidate, visibly, inside the environment, so that its use becomes part of the observed performance rather than an unmeasured advantage. What a coding assessment cannot survive is not AI in the room but variance in the room, the candidate-to-candidate difference in what was available and unrecorded.

One further pin is calibration to the role. A task tuned for a senior platform engineer measures little but frustration when given to a graduate, and a graduate-level task ceilings out on the senior, compressing real differences into identical top scores. Difficulty tiers mapped to role levels keep the instrument on scale for the population actually being hired, the same way a well-run testing program matches form difficulty to the examinee pool rather than administering one form to everyone. Tiering is a standardization decision, not a content decision: within a tier, everyone still meets the identical task under identical pins.

The constraints that make it fair also make it hard to fake

Everything the previous two sections pinned for measurement reasons does a second job. A sealed, standardized execution environment narrows the surface through which unmeasured help can reach a score: unknown tooling is ruled out by the tooling pin, unbounded outside work is ruled out by the time box, and the process record means the making is observed rather than inferred from a finished artifact. Constrained environments as an adversarial defense, the analysis of which formats survive once questions can leave the room and answers can come back in, is the territory of the companion article on AI-resistant assessment design; the point here comes before it. The constraints were already required just to make the scores comparable. A coding assessment acquires most of its integrity as a by-product of taking its measurement seriously.

Construction cannot close every channel. Identity and unobserved assistance remain, and each is handled by a move a companion article owns: a short supervised verification stage for finalists, the two-stage design set out in the study of unproctored testing and verification, and observation proportionate to the stakes, disclosed before the sitting begins, per the deterrence case in the proctoring article. Figure 4 maps the division of labor: what the design closes by construction, and the residue it leaves for those two moves.

The working rules: derive, publish, score, validate, retire

For an organization rebuilding its technical funnel, the evidence and the design logic above reduce to five working rules. Each is cheap to state, workable without a research department, and together they are the difference between owning a coding assessment and owning a queue of opinions about code.

• Derive the task from the job, and record the derivation. List the role's actual weekly verbs, build the smallest exercise that requires them, and keep the mapping from task to job behavior in writing. A technical hiring test derived from a puzzle bank is a valid measure of a job nobody is hiring for.
• Publish the rules of the room. Time box, runtime, tooling, and permitted resources, including whether AI assistance is allowed, stated in the invitation and identical for everyone. An unstated rule is enforced only against the candidates who guessed conservatively.
• Score stated dimensions against anchored levels. Fix the dimensions before the first sitting, anchor every level to described behavior, and split the labor: machines on correctness, reviewers on the rest, every judgment tied to evidence a second reviewer could inspect. Platforms differ in how much of this arrives built; the delivery features worth checking are the standardized environment, role-tiered tasks, and the review workflow.
• Validate against outcomes once cohorts allow. The coefficient in the literature licenses the format, not your instance of it. When enough hires accumulate, compare assessment scores with on-the-job results, because a constrained task that no one ever validates measures nothing anyone can defend.
• Retire tasks on a schedule. Treat exposure as certain and rotation as maintenance rather than scandal response. The time-box layer of Figure 4 only holds if tasks actually age: leaked answers expire when the task does.

Organizations searching for a coding interview alternative are mostly searching for relief from one of the failures catalogued above: the puzzle that filters on the wrong construct, the take-home no one can attribute, the debrief that cannot say what its 7 meant. The relief is not a newer format; it is the oldest one, finally built with care: a sample of the job, delivered under pinned conditions, scored as evidence. Software hiring holds the rare privilege of a criterion that fits inside an assessment. The rest is the discipline of not wasting it.