fault

KO | EN

gitlite — search

#python #android #java #machine-learning #typescript #linux #game #docker #golang #cpp #csharp #rust

★ 48 Open GitHub ↗

A Python package for testing hardware (part of the magma ecosystem)

// repository documentation

Was this content helpful?

★ 0 (0 ratings)

# Fault
![Linux Test](https://github.com/leonardt/fault/workflows/Linux%20Test/badge.svg)
![MacOS Test](https://github.com/leonardt/fault/workflows/MacOS%20Test/badge.svg)
[![BuildKite Status](https://badge.buildkite.com/c724929c65201c6ed5aebc027ffac02b5092d0bd8fad4341b6.svg?branch=master)](https://buildkite.com/stanford-aha/fault)
[![Code Coverage](https://codecov.io/gh/leonardt/fault/branch/master/graph/badge.svg)](https://codecov.io/gh/leonardt/fault)
[![License](https://img.shields.io/badge/License-BSD%203--Clause-blue.svg)](https://opensource.org/licenses/BSD-3-Clause)

A Python package for testing hardware (part of the magma ecosystem).

[API Documentation](http://truong.io/fault/)

[CHANGELOG](./CHANGELOG.md)

## Installation
```
pip install fault
```

## Documentation
Check out the [fault tutorial](https://github.com/leonardt/fault/tree/master/tutorial)

* [Actions](https://github.com/leonardt/fault/blob/master/doc/actions.md)
* [Tester](https://github.com/leonardt/fault/blob/master/doc/tester.md)
* [Integrating External Verilog](https://github.com/leonardt/fault/blob/master/doc/verilog_integration.ipynb)
* [Properties](https://github.com/leonardt/fault/blob/master/doc/property.md)

## Supported Simulators

* Digital simulation
  * [Verilator](https://www.veripool.org/wiki/verilator)
  * [Icarus Verilog](http://iverilog.icarus.com)
  * [Cadence Incisive/Xcelium](https://www.cadence.com/en_US/home/tools/system-design-and-verification/simulation-and-testbench-verification/xcelium-parallel-simulator.html)
  * [Synopsys VCS](https://www.synopsys.com/verification/simulation/vcs.html)
  * [Xilinx Vivado](https://www.xilinx.com/products/design-tools/vivado.html)
* Formal verification
  * Engines supported by [pySMT](https://github.com/pysmt/pysmt)
* Analog simulation
  * [ngspice](http://ngspice.sourceforge.net)
  * [Cadence Spectre](https://www.cadence.com/en_US/home/tools/custom-ic-analog-rf-design/circuit-simulation/spectre-simulation-platform.html)
  * [Synopsys HSPICE](https://www.synopsys.com/verification/ams-verification/hspice.html)
* Mixed-signal simulation
  * [Verilog-AMS](https://www.verilogams.com) via Cadence Incisive/Xcelium

## Example
Here is a simple ALU defined in magma.
```python
import magma as m

class ConfigReg(m.Circuit):
    io = m.IO(D=m.In(m.Bits[2]), Q=m.Out(m.Bits[2])) + \
        m.ClockIO(has_ce=True)

reg = m.Register(m.Bits[2], has_enable=True)(name="conf_reg")
    io.Q @= reg(io.D, CE=io.CE)

class SimpleALU(m.Circuit):
    io = m.IO(
        a=m.In(m.UInt[16]),
        b=m.In(m.UInt[16]),
        c=m.Out(m.UInt[16]),
        config_data=m.In(m.Bits[2]),
        config_en=m.In(m.Enable)
    ) + m.ClockIO()

opcode = ConfigReg(name="config_reg")(io.config_data, CE=io.config_en)
    io.c @= m.mux(
        [io.a + io.b, io.a - io.b, io.a * io.b, io.a ^ io.b], opcode)
```

Here's an example test in fault that uses the configuration interface, expects
a value on the internal register, and checks the result of performing the
expected operation.

```python
import operator
import fault

ops = [operator.add, operator.sub, operator.mul, operator.floordiv]
tester = fault.Tester(SimpleALU, SimpleALU.CLK)
tester.circuit.CLK = 0
tester.circuit.config_en = 1
for i in range(0, 4):
    tester.circuit.config_data = i
    tester.step(2)
    tester.circuit.a = 3
    tester.circuit.b = 2
    tester.eval()
    tester.circuit.c.expect(ops[i](3, 2))
```

We can run this with three different simulators

```python
tester.compile_and_run("verilator", flags=["-Wno-fatal"], directory="build")
tester.compile_and_run("system-verilog", simulator="ncsim", directory="build")
tester.compile_and_run("system-verilog", simulator="vcs", directory="build")
```

### Working with internal signals
Fault supports peeking, expecting, and printing internal signals. For the
`verilator` target, you should use the keyword argument `magma_opts` with
`"verilator_debug"` set to true.  This will cause coreir to compile the verilog
with the required debug comments.  Example:
```python
tester.compile_and_run("verilator", flags=["-Wno-fatal"], 
                       magma_opts={"verilator_debug": True}, directory="build")
```

If you're using `mantle.Register` from the `coreir` implementation, you can
also poke the internal register value directly using the `value` field.  Notice
that `conf_reg` is defined in `ConfigReg` to be an instance of
`mantle.Register` and the test bench pokes it by setting `confg_reg.value`
equal to `1`.

```python
tester = fault.Tester(SimpleALU, SimpleALU.CLK)
tester.circuit.CLK = 0
# Set config_en to 0 so stepping the clock doesn't clobber the poked value
tester.circuit.config_en = 0
# Initialize
tester.step(2)
for i in reversed(range(4)):
    tester.circuit.config_reg.conf_reg.value = i
    tester.step(2)
    tester.circuit.config_reg.conf_reg.O.expect(i)
    # You can also print these internal signals using the getattr interface
    tester.print("O=%d\n", tester.circuit.config_reg.conf_reg.O)
```

## FAQ

### How can I write test bench logic that depends on the runtime state of the circuit?
A common pattern in testing is to only perform certain actions depending on the
state of the circuit.  For example, one may only want to expect an output value
when a valid signal is high, ignoring it otherwise.  Another pattern is to change
the expected value over time by using a looping structure.  Finally, one may
want to expect a value that is a function of other runtime values.  To support
these pattern, `fault` provides support "peeking" values, performing expressions
on "peeked" values, if statements, and while loops.

#### Peek Expressions
Suppose we had a circuit as follows:
```python
class BinaryOpCircuit(m.Circuit):
    io = m.IO(I0=m.In(m.UInt[5]), I1=m.In(m.UInt[5]), O=m.Out(m.UInt[5]))

io.O @= io.I0 + io.I1 & (io.I1 - io.I0)
```
We can write a generic test that expects the output `O` in terms
of the inputs `I0` and `I1` (rather than computing the expected value in
Python).
```python
tester = fault.Tester(BinaryOpCircuit)
for _ in range(5):
    tester.poke(tester._circuit.I0, hwtypes.BitVector.random(5))
    tester.poke(tester._circuit.I1, hwtypes.BitVector.random(5))
    tester.eval()
    expected = tester.circuit.I0 + tester.circuit.I1
    expected &= tester.circuit.I1 - tester.circuit.I0
    tester.circuit.O.expect(expected)
```
This is a useful pattern for writing reuseable test components (e.g.  composign
the output checking logic with various input stimuli generators).

#### Control Structures
The `tester._while(<test>)` action accepts a Peek value or expression as the test condition for a loop and returns a child tester that allows the user to add actions to the body of the loop.  Here's a simple example that loops until a done signal is asserted, printing some debug information in the loop body:
```python
# Wait for loop to complete
loop = tester._while(dut.n_done)
debug_print(loop, dut)
loop.step()
loop.step()

# check final state
tester.circuit.count.expect(expected_num_cycles - 1)
```
Notice that you can also add actions after the loop to check expected behavior
after the loop has completed.

The `tester._if(<test>)` action behaves similarly by accepting a test peek value or expression and conditionally executes actions depending on the
result of the expression.  Here is a simple example:
```python
if_tester = tester._if(tester.circuit.O == 0)
if_tester.circuit.I = 1
else_tester = if_tester._else()
else_tester.circuit.I = 0
tester.eval()
```

The `tester._for(<num_iter>)` action provides a simple way to write a loop over
a fixed number of iterations.  Use the attribute `index` to get access to the
current iteration, for example:
```python
loop = tester._for(8)
loop.poke(circ.I, loop.index)
loop.eval()
tester.expect(circ.O, loop.index)
```

### What Python values can I use to poke/expect ports?
Here are the supported Python values for poking the following port types:
* `m.Bit` - `bool` (`True`/`False`) or `int` (`0`/`1`) or `hwtypes.Bit`
* `m.Bits[N]` - `hwtypes.BitVector[N]`, `int` (where the number of bits used to
  express it is equal to `N`)
* `m.SInt[N]` - `hwtypes.SIntVector[N]`, `int` (where the number of bits used to
  express it is equal to `N`)
* `m.UInt[N]` - `hwtypes.UIntVector[N]`, `int` (where the number of bits used to
  express it is equal to `N`)
* `m.Array[N, T]` - `list` (where the length of the list is equal to `N` and
  the elements recursively conform to the supported types of values for `T`).
  For example, suppose I have a port `I` of type `m.Array[3, m.Bits[3]]`. 
  I can poke it as follows:
  ```python
  val = [random.randint(0, (1 << 4) - 1) for _ in range(3)]
  tester.poke(circ.I, val)
  ```
  You can also poke it by element as follows:
  ```python
  for i in range(3):
      val = random.randint(0, (1 << 4) - 1)
      tester.poke(circ.I[i], val)
      tester.eval()
      tester.expect(circ.O[i], val)
  ```
* `m.Tuple(a=m.Bits[4], b=m.Bits[4])` - `tuple` (where the length of the tuple is equal to the number of fields), `dict` (where there is a one-to-one mapping between key/value pairs to the tuple fields).  Example:
  ```python
  tester.circuit.I = (4, 2)
  tester.eval()
  tester.circuit.O.expect((4, 2))
  tester.circuit.I = {"a": 4, "b": 2}
  tester.eval()
  tester.circuit.O.expect({"a": 4, "b": 2})
  ```

### How do I generate waveforms with fault?

Fault supports generating `.vcd` dumps when using the `verilator` and
`system-verilog/ncsim` target.

For the `verilator` target, use the `flags` keyword argument to pass the
`--trace` flag.  For example,

tester.compile_and_run("verilator", flags=["-Wno-fatal", "--trace"])

The `--trace` flag must be passed through to verilator so it generates code
that supports waveform dumping. The test harness generated by fault will
include the required logic for invoking `tracer->dump(main_time)` for every
call to `eval` and `step`.  `main_time` is incremented for every call to step.
The output `.vcd` file will be saved in the file `logs/{circuit_name}` where
`circuit_name` is the name of the ciruit passed to `Tester`.  The `logs`
directory will be placed in the same directory as the generated harness, which
is controlled by the `directory` keyword argument (by default this is
`"build/"`).

For the `system-verilog` target, enable this feature using the
`compile_and_run` parameter `dump_waveform=True`.  By default, the waveform
file will be named `waveforms.vcd` for `ncsim` and `waveforms.vpd` for `vcs`.
The name of the file can be changed using the parameter
`waveform_file="<file_name>"`.

The `vcs` simulator also supports dumping `fsdb` by using the argument
`waveform_type="fsdb"`.  For this to work, you'll need to also use the `flags`
argument using the path defined in your verdi manual.  For example,
`$VERDI_HOME/doc/linking_dumping.pdf`.

Here is an example using an older version of verdi (using the VERDIHOME
environment variable):
```python
verdi_home = os.environ["VERDIHOME"]
# You may need to change the 'vcs_latest' and 'LINUX64' parts of the path
# depending on your verdi version, please consult
# $VERDI_HOME/doc/linking_dumping.pdf
flags = ['-P', 
         f' {verdi_home}/share/PLI/vcs_latest/LINUX64/novas.tab',
         f' {verdi_home}/share/PLI/vcs_latest/LINUX64/pli.a']
tester.compile_and_run(target="system-verilog", simulator="vcs",
                       waveform_type="fsdb", dump_waveforms=True, flags=flags)
```

Here's an example for a newer version of verdi
```python
verdi_home = os.environ["VERDI_HOME"]
flags = ['-P',
         f' {verdi_home}/share/PLI/VCS/linux64/novas.tab',
         f' {verdi_home}/share/PLI/VCS/linux64/pli.a']
tester.compile_and_run(target="system-verilog", simulator="vcs",
                       waveform_type="fsdb", dump_waveforms=True, flags=flags)
```

To configure fsdb dumping, use the `fsdb_dumpvars_args` parameter of the
compile_and_run command to pass a string to the `$fsdbDumpvars()` function.

For example:
```python
tester.compile_and_run(target="system-verilog", simulator="vcs",
                       waveform_type="fsdb", dump_waveforms=True,
                       fsdb_dumpvars_args='0, "dut"')
```

will produce:
```verilog
  $fsdbDumpvars(0, "dut");
```

inside the generated test bench.

### How do I pass through flags to the simulator?
The `verilator` and `system-verilog` target support the parameter `flags` which
accepts a list of flags (strings) that will be passed through to the simulator
command (`verilator` for verilator, `irun` for ncsim, `vcs` for vcs, and
`iverilog` for iverilog).

### Can I include a message to print when an expect fails?
Use the `msg` argument to the expect action. You can either pass a standalone
string, e.g.
```python
tester.circuit.O.expect(0, msg="my error message")
```

or you can pass a printf/$display style message using a tuple.  The first argument
should be the format string, the subsequent arguments are the format values,
e.g.
```python
tester.circuit.O.expect(0, msg=("MY_MESSAGE: got %x, expected 0!",
                                tester.circuit.O))
```

### Can I display or print values from my testbench?
Yes, you can use the `tester.print` API which accepts a format string and a
variable number of arguments.  Here's an example:
```python
tester = fault.Tester(circ, circ.CLK)
tester.poke(circ.I, 0)
tester.eval()
tester.expect(circ.O, 0)
tester.poke(circ.CLK, 0)
tester.step()
tester.print("%08x\n", circ.O)
```

### Can I just generate a test bench without running it?
Yes, here's an example:
```python
# compile the tester
tester.compile("verilator")
# generate the test bench file (returns the name of the file)
tb_file = tester.generate_test_bench("verilator")
```

or for system verilog
```python
tester.compile("system-verilog", simulator="ncsim")
tb_file = tester.generate_test_bench("system-verilog")
```

### Using the ReadyValid Tester
Fault provides a `ReadyValidTester` that provides a few convenient features for
unit testing `ReadyValid` interfaces with sequences.

Consider the following circuit:
```python
class Main2(m.Circuit):
    io = m.IO(I=m.Consumer(m.ReadyValid[m.UInt[8]]),
              O=m.Producer(m.ReadyValid[m.UInt[8]]),
              inc=m.In(m.UInt[8]),
              ) + m.ClockIO()
    count = m.Register(m.UInt[2])()
    count.I @= count.O + 1
    enable = io.I.valid & (count.O == 3) & io.O.ready
    io.I.ready @= enable
    io.O.data @= m.Register(m.UInt[8], has_enable=True)()(io.I.data + io.inc,
                                                          CE=enable)
    io.O.valid @= enable
```

The output stream `O` is the input stream `I` incremented by the value of `inc`
and delayed by 4 cycles.

Here's a simple test that provides the input sequence `I` and expected output
sequence `O`
```python
def test_lifted_ready_valid_sequence_simple():
    I = [BitVector.random(8) for _ in range(8)] + [0]
    O = [0] + [i + 2 for i in I[:-1]]
    tester = f.ReadyValidTester(Main2, {"I": I, "O": O})
    tester.circuit.inc = 2
    tester.finish_sequences()
    tester.compile_and_run("verilator", disp_type="realtime")

```

Notice that we provide the sequences as a dictionary mapping port name to
sequence in the constructor.  Afterwards, we are free to `poke` values like in a normal tester, in this case provide `2` for `inc` which will satisfy the provided stream.

**NOTE:** The user must explicitly use the `tester.circuit` peek/poke
interface, or call `tester.poke(tester._circuit, value)` since the user circuit
is wrapped internally (cannot call `tester.poke(Main2, value)`).

The test finishes by calling `tester.finish_sequences()` which is a convenience
API that waits for the provided sequences to finish.

Here's a different version of the above test that should fail (changing the `inc` value mid test).
```python

def test_lifted_ready_valid_sequence_simple_fail():
    I = [BitVector.random(8) for _ in range(8)] + [0]
    O = [0] + [i + 2 for i in I[:-1]]
    tester = f.ReadyValidTester(Main2, {"I": I, "O": O})
    tester.circuit.inc = 2
    # Should work for a few cycles
    for i in range(9):
        tester.advance_cycle()
    # Bad inc should fail
    tester.circuit.inc = 3
    tester.finish_sequences()
    with pytest.raises(AssertionError):
        tester.compile_and_run("verilator", disp_type="realtime")
```

Finally, here's a variant that changes the `inc` value over time to match the expected sequence.
```python
def test_lifted_ready_valid_sequence_changing_inc():
    I = [BitVector.random(8) for _ in range(8)] + [0]
    O = [0] + [I[i] + ((i + 1) % 2) for i in range(8)]
    tester = f.ReadyValidTester(Main2, {"I": I, "O": O})
    # Sequence expects inc to change over time
    for i in range(8):
        tester.circuit.inc = i % 2
        tester.advance_cycle()
        tester.wait_until_high(tester.circuit.O.ready & tester.circuit.O.valid)
    # Advance one cycle to finish last handshake
    tester.advance_cycle()
    tester.expect_sequences_finished()
    tester.compile_and_run("verilator", disp_type="realtime")
```

**NOTE:** At the end of the test, we call `expect_sequences_finished()` to
assert that the sequences have all been processed, otherwise it's possible tha
the test could pass without completing the sequences.

fault

Explore Similar Repositories

Was this content helpful?

Recent Feedback

Download README