Reading tables of results

• A table that lists test results
• Comparison tables

Our experiment reports mostly consist of tables of results. In such tables, rows correspond to test executions, while columns correspond to model or hardware output. Model output is given as Allow or Forbid, expressing that the executions are either allowed or forbidden by the model; while hardware output is given as counts of observed executions over the total number of runs, supplemented by some comments.

Tables come in two varieties, which we now detail.

A table that lists test results

The following table simply shows model and hardware output for a few ARM tests on our current model (column “Model”) and on some ARM machines columns from “Tegra2” to “Exynos4412”). The additional “ARM” column corresponds to the sum of all runs on all ARM machines.

Executions of a given test derive from its final condition, which is the one from our test base. Executions are somehow hidden as one as to click on the test link to discover them.

From the above table we can conclude:

1. One execution of CoRR1 is forbidden by our model, yet observed on hardware (this is a hardware bug acknowledged by ARM).
2. One execution of WRC+ctrl+addr is allowed by our model, yet not observed on hardware. Notice that the execution is clearly allowed by ARM documentation [1, Sec. 6.4] “Causal consistency issues with multiple observers”.

So far so good, the above table suffices to illustrate two important facts about the ARM architecture. However, the table does not tell the whole story on comparing model and hardware.

Comparison tables

We thus often use another kind of table whose rows result from comparing Model and hardware. We usually display two tables: the first one lists all the executions that are forbidden by the model, yet observed on hardware — which we entitle Invalid behaviours; while the second one lists all the executions that are allowed by the model, yet unobserved on hardware — which we entitle Unseen behaviours.

Invalid behaviours

We now see that there are more invalid executions than the one of CoRR1: the test RDW has three invalid executions, which are similar to the one of CoRR1.

Unseen behaviours