Previous Up Next

Reading tables of results

Our experiment reports mostly consist of tables of results. In such tables, rows correspond to test executions, while columns correspond to model or hardware output. Model output is given as Allow or Forbid, expressing that the executions are either allowed or forbidden by the model; while hardware output is given as counts of observed executions over the total number of runs, supplemented by some comments.

Tables come in two varieties, which we now detail.

A table that lists test results

The following table simply shows model and hardware output for a few ARM tests on our current model (column “Model”) and on some ARM machines columns from “Tegra2” to “Exynos4412”). The additional “ARM” column corresponds to the sum of all runs on all ARM machines.

Executions of a given test derive from its final condition, which is the one from our test base. Executions are somehow hidden as one as to click on the test link to discover them.

There are 8 such tests
 ModelARMTegra2APQ8060A5XA6XExynos5250Tegra3Exynos4412Exynos5410APQ8064
CoRR1ForbidNo, 10M/101GNo, 928k/12GNo, 87/24GNo, 249k/3.9GOk, 0/13GOk, 0/7.9GNo, 7.7M/18GNo, 1.0M/11GOk, 0/5.7GNo, 612/5.4G
  Forbid invalidatedForbid invalidatedForbid invalidatedForbid invalidated  Forbid invalidatedForbid invalidated Forbid invalidated
CoRWForbidOk, 0/191GOk, 0/34GOk, 0/33GOk, 0/9.1GOk, 0/13GOk, 0/7.9GOk, 0/61GOk, 0/22GOk, 0/5.7GOk, 0/4.2G
CoRW1ForbidOk, 0/358GOk, 0/60GOk, 0/63GOk, 0/14GOk, 0/26GOk, 0/16GOk, 0/116GOk, 0/44GOk, 0/11GOk, 0/8.4G
CoWRForbidOk, 0/191GOk, 0/34GOk, 0/33GOk, 0/9.1GOk, 0/13GOk, 0/7.9GOk, 0/61GOk, 0/22GOk, 0/5.7GOk, 0/4.2G
CoWWForbidOk, 0/381GOk, 0/68GOk, 0/66GOk, 0/18GOk, 0/26GOk, 0/16GOk, 0/123GOk, 0/44GOk, 0/11GOk, 0/8.4G
MPAllowOk, 269M/30GOk, 41M/5.0GOk, 6.6M/2.3GOk, 55M/6.3GOk, 14M/6.6GOk, 2.6M/1.2GOk, 12M/1.2GOk, 134M/7.0GOk, 504k/40MOk, 2.2M/400M
RDWForbidOk, 0/39GOk, 0/23GOk, 0/11GOk, 0/2.8GOk, 0/2.1G
WRC+ctrl+addrAllowNo, 0/23GNo, 0/6.7GNo, 0/11GNo, 0/2.8GNo, 0/2.1G
  Allow unseen     Allow unseenAllow unseenAllow unseenAllow unseen

From the above table we can conclude:

  1. One execution of CoRR1 is forbidden by our model, yet observed on hardware (this is a hardware bug acknowledged by ARM).
  2. One execution of WRC+ctrl+addr is allowed by our model, yet not observed on hardware. Notice that the execution is clearly allowed by ARM documentation [1, Sec. 6.4] “Causal consistency issues with multiple observers”.

So far so good, the above table suffices to illustrate two important facts about the ARM architecture. However, the table does not tell the whole story on comparing model and hardware.

Comparison tables

We thus often use another kind of table whose rows result from comparing Model and hardware. We usually display two tables: the first one lists all the executions that are forbidden by the model, yet observed on hardware — which we entitle Invalid behaviours; while the second one lists all the executions that are allowed by the model, yet unobserved on hardware — which we entitle Unseen behaviours.

Invalid behaviours

There are 2 such tests
 ModelARMTegra2APQ8060A5XA6XExynos5250Tegra3Exynos4412Exynos5410APQ8064
CoRR1ForbidOk, 10M/102GOk, 928k/12GOk, 87/24GOk, 249k/3.9GNo, 0/14GNo, 0/8.3GOk, 7.7M/18GOk, 1.0M/11GNo, 0/6.3GOk, 612/5.4G
  Forbid invalidatedForbid invalidatedForbid invalidatedForbid invalidated  Forbid invalidatedForbid invalidated Forbid invalidated
RDWForbidOk, 450k/40GOk, 295k/23GOk, 155k/11GNo, 0/3.1GNo, 0/2.3G
  Forbid invalidated     Forbid invalidatedForbid invalidated  

We now see that there are more invalid executions than the one of CoRR1: the test RDW has three invalid executions, which are similar to the one of CoRR1.

Unseen behaviours

There is one such test
 ModelARMTegra3Exynos4412Exynos5410APQ8064
WRC+ctrl+addrAllowNo, 0/24GNo, 0/7.3GNo, 0/11GNo, 0/3.1GNo, 0/2.3G
  Allow unseenAllow unseenAllow unseenAllow unseenAllow unseen

Previous Up Next