$error, $fatal and Assertions

Not every failure is the same. A wrong output on one vector should be recorded and the run should continue, because the next fifty vectors may show a pattern. A design in an illegal state that can never recover should stop the run, because everything after is meaningless. The severity tasks give you that choice.

Four grades

$info prints a progress message and implies nothing. $warning flags something odd that is not necessarily wrong and continues. $error records a failure and continues; it is the workhorse of a self-checking testbench. $fatal(level, message) ends the run with a non-zero exit code; the level, 0 to 2, controls how much context is printed.

$info("starting passthrough tests");

if (data_out !== 8'h00)
    $warning("output non-zero on an invalid transaction: %0h", data_out);

if (data_out !== data_in)
    $error("passthrough mismatch: in=%0h out=%0h", data_in, data_out);

if (data_out === 8'hxx)
    $fatal(1, "output is x; the state is corrupt, stopping");

All four are SystemVerilog tasks that Icarus accepts, with the same format strings as $display.

The choice between $error and $fatal comes down to one question: is there any point continuing? A failing vector, yes; the design can still answer the next one, and twenty failures tell you more than one. An x on an output, a violated protocol invariant, or a watchdog timeout, no. Using $fatal for every mismatch feels decisive and throws away the pattern; a broken carry chain fails thousands of vectors, and stopping at the first tells you nothing about that.

Immediate assertions

The assert statement tests an expression when it is reached and runs its else branch if the expression is false or x:

// as an if
if (dut_sum !== expected_sum)
    $error("sum mismatch: expected=%0h got=%0h", expected_sum, dut_sum);

// as an assertion
assert (dut_sum === expected_sum)
else $error("sum mismatch: expected=%0h got=%0h", expected_sum, dut_sum);

Same behaviour. The assertion reads as a statement of what must be true, and it is the form you will see in professional code. Always write the else, so that the message and the severity are yours and not the simulator’s default.

The message

A message has to carry three things: which test, what was expected, what came out.

ERROR: output mismatch                                             // useless
ERROR: [test 47] a=0xFE b=0x03 expected=0x101 actual=0x001         // enough to reproduce

`timescale 1ns / 1ps

module tb_multiplier_8bit();
    reg  [7:0]  a, b;
    wire [15:0] product;
    integer     error_count, test_id;
    integer     i, j;
    reg  [15:0] expected;

    multiplier_8bit dut (.a(a), .b(b), .product(product));

    initial begin
        error_count = 0;
        test_id     = 0;

        for (i = 0; i < 256; i = i + 1) begin
            for (j = 0; j < 256; j = j + 1) begin
                a = i[7:0];
                b = j[7:0];
                #10;
                expected = i * j;
                test_id  = test_id + 1;

                assert (product === expected)
                else begin
                    $error("[test %0d] a=%0d b=%0d expected=%0d actual=%0d", test_id, i, j, expected, product);
                    error_count = error_count + 1;
                end
            end
        end

        if (error_count == 0)
            $display("PASSED: all %0d tests.", test_id);
        else
            $display("FAILED: %0d of %0d tests.", error_count, test_id);
        $finish;
    end
endmodule

$error does not hand you a count at the end in any portable way, so keep your own error_count and increment it on every call.

All four in one task

`timescale 1ns / 1ps

module tb_alu_8bit();
    reg  [7:0] a, b;
    reg  [2:0] op;
    wire [7:0] result;
    wire       overflow;

    integer error_count, test_count;

    alu_8bit dut (.a(a), .b(b), .op(op), .result(result), .overflow(overflow));

    task run_check;
        input [7:0]  t_a, t_b;
        input [2:0]  t_op;
        input [7:0]  t_expected;
        input        t_exp_ovf;
        input [31:0] t_id;
        begin
            a  = t_a;
            b  = t_b;
            op = t_op;
            #10;
            test_count = test_count + 1;

            if (result === 8'hxx || overflow === 1'bx)
                $fatal(1, "[test %0d] output is x: undriven net or uninitialised state", t_id);

            if (overflow !== t_exp_ovf)
                $warning("[test %0d] overflow flag got=%b expected=%b", t_id, overflow, t_exp_ovf);

            assert (result === t_expected)
            else begin
                $error("[test %0d] op=%0b a=%0h b=%0h expected=%0h got=%0h", t_id, t_op, t_a, t_b, t_expected, result);
                error_count = error_count + 1;
            end
        end
    endtask

    initial begin
        error_count = 0;
        test_count  = 0;

        $info("ALU tests starting");

        run_check(8'd10,  8'd20,  3'b000, 8'd30,  1'b0, 1);     // add
        run_check(8'd200, 8'd100, 3'b000, 8'd44,  1'b1, 2);     // add, wraps
        run_check(8'd0,   8'd0,   3'b000, 8'd0,   1'b0, 3);
        run_check(8'd255, 8'd255, 3'b000, 8'd254, 1'b1, 4);

        run_check(8'hAA, 8'h55, 3'b001, 8'h00, 1'b0, 5);        // and
        run_check(8'hFF, 8'hF0, 3'b001, 8'hF0, 1'b0, 6);

        run_check(8'hAA, 8'hAA, 3'b010, 8'h00, 1'b0, 7);        // xor
        run_check(8'h00, 8'hFF, 3'b010, 8'hFF, 1'b0, 8);

        $info("%0d tests run", test_count);

        if (error_count == 0)
            $display("PASSED: all %0d tests.", test_count);
        else
            $display("FAILED: %0d of %0d tests.", error_count, test_count);
        $finish;
    end
endmodule

The $fatal guards against an x that would corrupt every comparison after it; the $warning notes an odd flag without counting it as a failure; the $error records a wrong result; the $info marks progress. The next article removes the hand-written expected values: a reference model computes them.