Directed Tests
Exhaustive testing proves an 8-bit function and is out of reach past a dozen or so input bits. Beyond that you choose. Directed testing is the discipline of choosing: the inputs most likely to fail, most likely to be misspecified, most likely to matter in use. It does not replace a sweep where a sweep is possible; it runs first, is small, and finds the obvious breakage in seconds.
The categories
Any experienced verification engineer can name half a dozen inputs worth trying before running anything, because the same cases break designs across architectures.
- Zero. Identity paths are often implemented separately from the general case. An adder that gives 1 for 0 + 0 has a specific, findable bug.
- Maximum, all ones. Overflow, saturation and carry-chain bugs live at the top of the range: 255 + 1, 0xFFFF_FFFF as an operand, a counter at its limit.
- Powers of two. One bit set, probing one position of a carry chain, a shifter, a decoder. A bug in bit 7 of a counter shows at 0x80.
- Off by one. N-1 and N+1 around every modulo point, threshold and limit. A counter that resets at 100 is tested at 98, 99, 100 and 101.
- The sign bit. 0x7F and 0x80 for 8 bits: sign extension, overflow detection and signed comparison all behave specially there.
- Repeated inputs. Stateful designs, FIFOs, shift registers, accumulators, can fail on the same value twice in a row, which a test that varies every input never tries.
Named cases
A failure reported as test 7 has to be decoded. One reported as OVERFLOW_MAX_PLUS_ONE says what broke. Parallel arrays of names, inputs and expected values give that:
`timescale 1ns / 1ps
module tb_adder_8bit_directed();
reg [7:0] a, b;
wire [8:0] sum;
integer error_count, test_count, k;
adder_8bit dut (.a(a), .b(b), .sum(sum));
localparam NUM_DIRECTED = 14;
reg [8*24-1:0] test_names [0:NUM_DIRECTED-1]; // 24 characters
reg [7:0] test_a [0:NUM_DIRECTED-1];
reg [7:0] test_b [0:NUM_DIRECTED-1];
reg [8:0] test_expected [0:NUM_DIRECTED-1];
task run_directed_tests;
integer idx;
begin
for (idx = 0; idx < NUM_DIRECTED; idx = idx + 1) begin
a = test_a[idx];
b = test_b[idx];
#10;
test_count = test_count + 1;
if (sum !== test_expected[idx]) begin
$error("[%0s] a=%0d b=%0d expected=%0d actual=%0d",
test_names[idx], test_a[idx], test_b[idx], test_expected[idx], sum);
error_count = error_count + 1;
end
end
end
endtask
integer i, j;
reg [8:0] expected_sweep;
initial begin
error_count = 0;
test_count = 0;
// zero
test_names[0] = "ZERO_PLUS_ZERO"; test_a[0] = 8'd0; test_b[0] = 8'd0; test_expected[0] = 9'd0;
test_names[1] = "ZERO_PLUS_MAX"; test_a[1] = 8'd0; test_b[1] = 8'd255; test_expected[1] = 9'd255;
test_names[2] = "MAX_PLUS_ZERO"; test_a[2] = 8'd255; test_b[2] = 8'd0; test_expected[2] = 9'd255;
// maximum and overflow
test_names[3] = "OVERFLOW_MAX_PLUS_ONE"; test_a[3] = 8'd255; test_b[3] = 8'd1; test_expected[3] = 9'd256;
test_names[4] = "OVERFLOW_MAX_PLUS_MAX"; test_a[4] = 8'd255; test_b[4] = 8'd255; test_expected[4] = 9'd510;
// powers of two
test_names[5] = "POW2_128_PLUS_128"; test_a[5] = 8'd128; test_b[5] = 8'd128; test_expected[5] = 9'd256;
test_names[6] = "POW2_64_PLUS_64"; test_a[6] = 8'd64; test_b[6] = 8'd64; test_expected[6] = 9'd128;
test_names[7] = "POW2_1_PLUS_1"; test_a[7] = 8'd1; test_b[7] = 8'd1; test_expected[7] = 9'd2;
// off by one around 128
test_names[8] = "OFFBYONE_127_PLUS_1"; test_a[8] = 8'd127; test_b[8] = 8'd1; test_expected[8] = 9'd128;
test_names[9] = "OFFBYONE_128_PLUS_255"; test_a[9] = 8'd128; test_b[9] = 8'd255; test_expected[9] = 9'd383;
// the sign bit
test_names[10] = "SIGNED_MIN_PLUS_ZERO"; test_a[10] = 8'h80; test_b[10] = 8'h00; test_expected[10] = 9'h080;
test_names[11] = "SIGNED_MAX_PLUS_ONE"; test_a[11] = 8'h7F; test_b[11] = 8'h01; test_expected[11] = 9'h080;
// alternating
test_names[12] = "ALTERNATING_AA_55"; test_a[12] = 8'hAA; test_b[12] = 8'h55; test_expected[12] = 9'hFF;
test_names[13] = "ALTERNATING_55_AA"; test_a[13] = 8'h55; test_b[13] = 8'hAA; test_expected[13] = 9'hFF;
$info("running %0d directed tests", NUM_DIRECTED);
run_directed_tests();
$info("directed tests done: %0d failed", error_count);
$info("running the sweep");
for (i = 0; i < 256; i = i + 1) begin
for (j = 0; j < 256; j = j + 1) begin
a = i[7:0];
b = j[7:0];
#10;
expected_sweep = i + j;
test_count = test_count + 1;
if (sum !== expected_sweep) begin
$error("[SWEEP a=%0d b=%0d] expected=%0d actual=%0d", i, j, expected_sweep, sum);
error_count = error_count + 1;
end
end
end
if (error_count == 0)
$display("PASSED: all %0d tests.", test_count);
else
$display("FAILED: %0d of %0d tests.", error_count, test_count);
$finish;
end
endmodule
Directed tests first, sweep second. If 0 + 0 is wrong you want to know in the first millisecond, not after 65 536 iterations. When a named test fails the message names the behaviour, and an array of failed flags printed in the summary classifies the failures at a glance: if the add and and cases fail and the xor cases pass, the fault is in specific paths, not random corruption.
From a file
For hundreds of vectors the arrays move out to a file and $readmemh loads them, as in section 6.2. A script generates the file against a software model; the specification changes, the script is edited, the file regenerated, and no Verilog moves.
Tiers
A test suite settles into three tiers. Directed tests, a few dozen, hand-written for the danger zones, run in seconds, and every bug ever found in the field gets one so that it cannot come back. A sweep or structured random run, thousands to millions of vectors, minutes. System-level tests, the design under a realistic workload, hours, for the behaviour that only appears in use. This section has been tier one, the fastest and the most valuable during active development.
The testbench can now catch every error, name the input, and name the behaviour that broke. What it cannot do is show why, and that is the next section: reading waveforms with the same discipline.