Cost evaluation of coverage directed test generation for the IBM mainframe.

Full text

Page 1

- -
1
Cost Evaluation of Coverage Directed Test Generation for the IBM Mainframe
Gilly Nativ, Steven Mittermaier, Shmuel Ur, Avi Ziv
IBM Corporation
Abstract
Test generation and simulation tools have input stimuli that
can direct them to cover specific events. However, the cost
of completely covering a verification plan is still very high.
While coverage analysis tools can find events that have not
been covered, they do not provide an automated covering
method. This paper presents the first implementation of a
generation framework that uses feedback from coverage
analysis to direct microarchitecture simulation. This
framework uses a coverage analysis tool to find events that
have not been simulated and then utilizes information about
the design to determine which directives should be given to
the simulation environment. This paper describes, in detail,
the system and its operation process, an experiment that
uses the system, and the results of the experiment. This
system was shown to reduce the machine time and person
time required to cover the test plan. Implications of this
work suggest the types of verification plans appropriate for
the utilization of this system and the further experiments and
developments required.


1. Introduction
Nowadays, microarchitecture simulation is the major
technique used for processor verification. Microarchitecture
simulation requires a lot of expert time and computer
resources. Formal Verification techniques [1][2] cannot
handle the size of modern processors and can only be used
for specific portions of the design.
In current industrial practice, test generation and simulation
tools perform large portions of the verification plan. In this
technique, huge amounts of tests are generated and executed
over a simulation model and the results are checked to
match the expected results. The test space is enormous and
even a large number of tests can cover only a small portion
of the test space. In practice, actual coverage of the global
test space is unknown and the verification process does not
provide feedback on the quality of the tests that are
simulated.
The main technique used to measure the thoroughness of the
verification is called coverage analysis [3][4]. The idea of
coverage analysis is to compile, in some systematic fashion,
a large and comprehensive list of tasks called a coverage
model. Each item in this list is called a coverage task and
represents some event that should be covered in verification.
Coverage analysis tools can provide data about the
verification status and progress over the subject coverage
model. The coverage analysis tools can find the coverage
tasks that have not been covered, since the tasks require
changes and biases to be performed in the verification
environment.
Coverage analysis provides a measurement of the quality of
verification1 and helps find tasks that have not been
covered. However, it does not provide a method to cover
these tasks or achieve better coverage in the verification
process. Covering each task by manually redirecting the
verification process would take a large amount of expert
time. A verification methodology called Coverage Directed
Generation (CDG) aims to solve this problem.
There are two major techniques for CDG: by Construction,
and by Feedback. CDG by Construction [5][6][7][8] is
based on a provided translation engine that can translate a
verification task into a simulation test. In general, it
involves describing the coverage tasks in some fashion. It
then uses the translation engine to translate each one of
these tasks into a simulation test. This method's underlying
assumption is that the translation is exact, and therefore, the
derived test covers the verification tasks with complete
certainty. Task coverage is assured as early as the time of
construction of the test. CDG by Feedback also requires a
translation engine, but it does not assume that it is faultless.
This technique uses coverage analysis to obtain feedback on
the success of the translation and reiterates the translation
process for the tasks that have not been covered. Here, the
underlying assumption is that if the translation is inaccurate
one or more times, it can still succeed on an additional
attempt to produce a covering test. Given the huge size and
complexity of the simulation model, building a faultless
translation engine would require an enormous investment of
expert time, and would only be suitable for a specific
coverage model. Feedback CDG, which demands less from


1 In the environment used for this work, test generation is
done cycle-by-cycle, alongside simulation. For each cycle,
there is a generation phase, immediately followed by a
simulation phase. Therefore, the term simulation, in this
paper, also includes generation.

Page 2

- -
2
the translation mechanism, appears to have potential for this
type of simulation environment. It provides the benefits of
CDG—better quality tests and reduced machine time—
without heavy investments in expert time.
This paper describes an implementation of a Feedback CDG
framework for a coverage model on a microarchitecture
simulation environment. The implementation used IBM’s
simulation tools and the functional coverage analysis tool,
Meteor [9]. It also involved building a Feedback CDG
mechanism, referred to hereafter as the CDG System, which
will be used for other coverage models in the future.
Results of this work show that the coverage model was
covered in a significantly shorter amount of machine time
than it would take without a CDG system, (i.e., only by
random simulation). Also, the person time invested in the
CDG system, specifically for this coverage model, was
shorter than the person time required to perform a manual
feedback process.
Feedback CDG framework advantages, with the addition of
proposed improvements, offer the potential for saving
additional work time for certain types of coverage models.
The rest of the paper is organized as follows: Section 2
includes an overview of the simulation environment
involved in this work. Section 3 presents a description of the
CDG framework developed. Section 4 explains the
experiment and its results and Section 5 concludes the
paper.
2. The Simulation Environment
The CDG framework described here was built and used
with the Z-series (formerly S390) simulation environment
and coverage tools. However, the CDG methodology used
for this framework is not specific to this environment, and is
applicable to other environments as well.
2.1 Random Simulation Environment
2.1.1 Background
In the IBM Z-series server (mainframe) chip development,
the verification team tries to find design bugs as early in the
process as possible. This involves multiple levels of
simulation, including macro, unit (several macros), chip,
and system (several chips). The Random Environment [10]
serves as our primary environment for verification at the
unit level and for switch/cache chip simulation.
2.1.2 Description and Methodology
The Random Environment is a methodology and a set of
C++ classes. In each cycle, pseudo random programs
(drivers) choose stimuli for the next cycle. They drive the
inputs to the logic or model under test, as randomly as
possible, bound only by the limits and assumptions of the
logic itself. Any limits implied by the chip architecture or
by the limits of the "real" logic that drive the logic under
test once fabricated in silicon, are ignored. Separate
"intelligent" monitor programs are created to watch and
understand the stimulus applied to the chip and to verify its
correct behavior.
The C++ classes provide tools that simplify the creation of
the drivers and make them configurable without the need to
modify source code or recompile. One of these tools is a
configuration file that is used as input to the simulation
environment at run-time. This configuration file contains
parameters that are used as directives to the drivers. These
parameters control key values inside the environment, or the
distribution of values that are randomly drawn during the
test case.
2.2 Switch/Cache Unit
The unit used in this work combines the functions of a
coherent switch with a shared cache (referred to hereafter as
a switch/cache) [11]. It stands at the intersection of multiple,
high-performance processors and a very large high-
bandwidth memory subsystem.
Internally, two pipelines handle transactions on an address
basis, as opposed to an availability basis. The unit is often
referred to as the system controller because it connects the
memory subsystem, a two-part store-in L2 cache, each
processor’s store-through L1 cache, and high-throughput
I/O controllers. It consists of five chips, one controller chip,
and four data chips; all of which are included but not shown
inside the "Coherent Switch with a Shared Cache" symbol
in Figure 1.
Figure 1 - System Structure
Memory subsystem
I/O
Bridge
. . .
Coherent Switch with a Shared Cache
Data
Command
Processor
Chip n
Processor
Chip 0
Processor
Chip 1

Page 3

- -
3
2.3 Structure of the Switch/Cache Simulation
Environment
The structure of the simulation environment, shown in
Figure 2, substitutes programs for devices outside the scope
of testing. The environment consists of the switch/cache
microarchitecture model, and drivers that simulate the
processors, the memory subsystem, and the I/O Bridge. At
every interface, a monitor collects data on the current
transactions and compares the results with the actual events.
2.4 The Coverage Model
• Motivation
Although there are a manageable number of commands the
processors can send to the switch/cache, and a limited
number of responses to each, the interactions of these
command-response sequences can create very complex and
perhaps unexpected conditions and contention within the
unit when many processors are active. We must be sure
these conditions are tested to be confident that the
verification is complete.
• Definition
The switch/cache coverage model comprises all possible
types of command-response events that can take place
between the CPs and the switch/cache. The model is the list
of all possible events of a command that come from a
specific source and a corresponding response.
We used Cross Product Generation [9] methodology to
create the coverage model. In this methodology, the model
is created by a definition of a few coverage attributes, each
of which has a set of possible values. The list of coverage
tasks is generated by the cross product of the sets of values
of the attributes. Additional specifications, called
restrictions, determine which of these combinations are
legal.
The switch/cache coverage model was defined by five
coverage attributes relevant to the command-response
events. The attributes were selected by the Functional
Coverage Methodology [4]. That is, these are functional
attributes of the design and not necessarily physical entities
or signals. The attributes are:
1. Commands (from the CPs to the switch/cache) – 31
possible values.
2. Responses (from the switch/cache to the CPs) – 16
values.
3. CPs in the system – eight values.
4. Command generators per CP chip – two values.
5. Pipes in each switch/cache – two values.

The cross product is all of the possible combinations of
these values: 31 x 16 x 8 x 2 x 2 = 15,872 combinations.
The following are the model's restrictions:
1. The possible responses for each command.
2. The commands/responses that are still not implemented
in the simulation model.
3. The commands executed only on pipe #1.
Altogether, there are 1968 legal combination, or tasks, that
should be covered.
An example of a task would be: (Command = 20, Response
= 01, CP = 5, CG = 0, Pipe = 1). This task will be covered
if, in one of the simulation runs, Command Generator 0 in
CP 5 generates command 20, and the switch/cache
processes the command in pipe 1 and sends response 01.
2.5 Reasons for Selecting Simulation Environment
A great deal of the verification engineer's time and effort is
expended in the Random Environment. While this is a topic
for debate, we strongly suspect that a Random Environment
quickly covers a given space and then continues to dwell in
that same space unless human redirection takes place. A
method that automates this redirection would save a great
deal of development resources in the form of expert time
and computer run-time.
The complexity of a coverage model for CDG
implementation is derived from: 1) the functional distance
between the outputs of the simulation environment, which
are the coverage attributes, and 2) the inputs of the
simulation environment. In other words, the complexity lies
in figuring out which input values will generate a specific
combination of output values.
Using this measurement, the selected coverage model is
relatively simple, but not trivial. A coverage model with this
level of complexity is simple enough for a first of a kind
experiment, and is still appropriate since it is hard to cover
otherwise.
Figure 2 - Simulation Environment
CP Driver
Object
CP Driver
Object
Object
CP Monitor
Processor 1
Processor 2
Processor n
Object
CP Monitor
Memory subsystem driver & model
CP Driver
Object
Object
CP Monitor
Object
Switch/Cache
Model
I/O Bridge Driver
Object
Bridge Monitor
I/O

Page 4

- -
4
3. The CDG Framework
3.1 Motivation for Using CDG
When coverage is measured over a space of simulation
tasks, we can get information about the distribution of the
events. Common coverage analysis results show that many
of the tasks are not covered. Few tasks are covered few
times, and very few are covered a large number of times.
The greatest benefit from the coverage information is
attained when the generation tools are directed towards the
less covered tasks. This grants an even distribution of the
tests over the space, which provides us with a good chance
to find bugs in areas seldom reached by the simulation
environment.
The process of directing simulation in order to cover a set of
tasks consists of the following phases (Figure 3):
1. Analyze the coverage state and decide which tasks
should be attempted to cover. If none, finish. This step is
done using a coverage analysis tool.
2. Find out which directives should be given to simulation
in order to cover these tasks and send them to the
simulation tools. This is the most complex step, since it
requires specific knowledge of the design. Therefore, it
forces the involvement of verification experts in the
process.
3. Run simulation. This step is done using a simulation
tool.
4. Verify that the specified tasks were covered. This step is
also performed using a coverage analysis tool.
The process of directing simulation is extremely time-
consuming for verification experts. The goal of CDG is to
automate this process and reduce the required time.
3.2 Components of the Framework
The CDG framework used in this work is presented in
Figure 4. Its components are:
•
Random – switch/cache unit Random Simulation
Environment.
•
UPX – file transfer application, used for transferring
the simulation trace files to the coverage server.
•
Meteor Server – the server side of the coverage
analysis tool, that measures and stores coverage data.
•
CRE (Coverage Report Engine) – generates coverage
reports. The CRE is a component of the client side of
the coverage analysis tool, Meteor.
•
CDG System – reads the coverage reports and
redirects the simulation environment in order to cover
new tasks.
3.2.1 Coverage Analysis Tool
Coverage of the model is measured by Meteor, IBM’s
functional coverage analysis tool. Simulation engineers
define a coverage model on the Meteor server—in this work
it was the switch/cache coverage model (described above).
The server then collects trace data from the simulation
environment and uses it to measure coverage over the given
model. The simulation environment produces the trace data,
which consists of the list of tasks that were covered during a
simulation run. Accumulated measurement results are stored
in a database.
The Meteor analysis client (CRE) provides a set of reports
about the coverage status in either a graphical or a textual
presentation. CRE can find the coverage tasks that have not
been covered, that is, did not appear in the trace data of any
of the simulation runs. It then groups them into sets of
related tasks with common attributes called coverage holes.
A coverage hole is defined as a set of coverage attributes
and their values.
For instance, coverage hole ‘A’ can be defined as follows:
(Command = 15, Response = 4, CP = 3, Generator = 1, Pipe
= 0). This means that response 4 was never sent to
































Figure 3 – CDG Process
coverage data
set of tasks
to be covered
directives to
simulation
trace data
4. check
coverage of
tasks
1. analyze
coverage
status
2. select simulation
directives
3. run
simulation
Finish
Start

Page 5

- -
5
command 15, which came from generator 1 in CP 3, if the
command was loaded on pipe 0. This coverage hole
comprises the following single coverage task:
Task # Command Response CP
1
15
Generator Pipe
1
4 3 0
A coverage hole ‘B’ can be defined as (Command = 15,
Response = 4, CP = 3, Pipe = 0). This means that response 4
was never sent to command 15, which came from any
generator in CP 3, if the command was loaded on pipe
0.This coverage hole comprises the following two tasks:
Task # Command Response CP
1
15
2
15
Generator Pipe
0
1
4
4
3
3
0
0
Finally, a coverage hole ‘C’ can be defined as (Command =
15, Response = 4, CP = 3). This means that none of these
tasks were covered; in other words, response 4 was never
sent to command 15, which came from CP 3, regardless of
the command generator or pipe on which it was loaded.
This hole comprises the following four tasks:
Task # Command Response CP
1
15
2
15
3
15
4
15
Generator Pipe
0
1
0
1
4
4
4
4
3
3
3
3
0
1
0
1
In the cross product methodology, coverage hole size is
measured by the number of coverage attributes by which the
coverage hole is defined. Hole X is considered larger than
hole Y if the number of attributes that define hole X is
smaller than the number of attributes that define hole Y. The
rationale is that hole X would represent a larger fault in the
verification environment and would usually include more
coverage tasks.
In the example above, coverage hole A is defined by all of
the (five) attributes of the coverage model, and therefore it
comprises a single coverage task. Coverage hole B is
defined by four attributes and coverage hole C is defined by
only three attributes. Therefore we would sort them:
A < B < C.
CRE generates the Holes Report, which specifies the current
coverage holes in the model. This report is used by the CDG
system to select directives for the simulation environment,
and prioritize them according to their size—largest first.
3.2.2 CDG System Components
•
Dispatcher – manages the CDG process. It controls all
framework components and calls them iteratively in
the required order.
•
Holes Report Image – parses the Holes Report created
by the CRE. It reads the report and creates C++
objects that represent its contents.
•
Director – directs simulation to cover a specific hole.
Each time the Dispatcher calls the Director, it
specifies a hole to be covered. The Director uses the
Rules Module to select simulation directives and uses
the Config Director to redirect the simulation
environment.
•
Rules Module – consists of a set of direction rules,
which specify which simulation directives should be
given to the Random Environment in order to cover a
specific hole. The verification expert specifies the
direction rules, since they require deep knowledge of
the microarchitecture and the simulation environment.
Therefore, this is the only component of the CDG
System that is specific to the subject coverage Model.
•
Config Director – implements the simulation
directives. In this work, the directives were
implemented by adding configuration file statements.




















































Figure 4 - CDG Framework
Holes Report
Image
Dispatcher
Director
Rules
Module
Config Director
Config file
trace
trace
UPX
Holes Report
Meteor
Server
CRE
Control
Data transfer
CDG system
External Framework
Random