12th Int`l Symposium on Quality Electronic Design

Tracking Hardware Evolution
José Augusto Nacif1,2, Thiago S. F. Silva1, Luiz Filipe M. Vieira1,
Alex Borges Vieira3, Antônio O. Fernandes1, Claudionor Coelho Jr.1
1
Universidade Federal de Minas Gerais, Belo Horizonte, Brazil
2
Universidade Federal de Viçosa, Florestal, Brazil
3
Universidade Federal de Juiz de Fora, Juiz de Fora, Brazil
E-mail: {jnacif,thiagosf,lfvieira,borges,otavio,coelho}@dcc.ufmg.br
(SVN) [4] and Concurrent Version System (CVS)
[5];
• Bug Tracking Systems (BTS): Store information
about all reported software bugs. Popular opensource systems are Bugzilla [6] and Mantis [7];
• E-mail lists or software documentation.
In the last years, software evolution research focused
mainly on [8]: estimation of developer effort, impact of
changes in source code and defect prediction. In order to
facilitate software evolution research, several frameworks
were proposed [9]. These frameworks are suitable to track
historical and software complexity metrics, but
unfortunately do not offer support to extract and visualize
Hardware Description Language (HDL) specific metrics.
Software evolution techniques can be used to
complement and deeply improve hardware verification
methodologies. For example, large industrial System-on-achips (SoCs) often reuse IP (Intellectual Property) blocks.
Many high-value IP blocks are developed over long periods
of time, being expected to live on for a decade [10]. Such
systems are designed by large teams, generating repositories
that contain millions of RTL lines of code and thousands of
bug reports. Open source hardware also plays an important
role on making hardware evolution research possible.
Designs like SUN’s Opensparc [11] and OpenRisc [12] offer
repositories that have plenty of information about how these
designs where implemented and verified.
We coin the term hardware evolution to describe a new
paradigm that offers many advantages. As happens in
software construction, hardware development can also
benefit from hardware evolution. It allows hardware bug
prediction, mitigating hardware development complexity
and verification issues. Furthermore, it may provide
hardware
module complexity profile, facilitating
development management.
The contributions of this work are threefold. First, we
propose the use of information available in HDL repositories
to track hardware evolution. Second, we provide EyesOn, an
open source, extensible framework designed to extract,
store, visualize, and export HDL complexity and historical
metrics. Third, we present scenarios where hardware
evolution tracking is inserted in traditional integrated design
flows, providing accurate and visual information.
This paper is outlined as follows. Section II reviews the
main software evolution tracking tools. In Section III, we
present the methodology to track hardware evolution, while
Abstract
Software evolution is the term used to describe the
process of developing and updating software systems.
Software repositories such as versioning systems and bug
tracking systems are used to manage the evolution of
software projects. The mining of this information is used to
support predictions and improve design and reuse.
Integrated circuit development can also benefit from these
techniques. Nowadays, both software and hardware
development use repositories and bug tracking systems.
There are many hardware open source projects such as
SUN's OpenSparc and designs at Opencores.org. We
propose a methodology to track specific HDL metrics in
order to improve design quality. Our results present a case
study that correlates HDL metrics and bug proneness of
Verilog HDL modules. We also present EyesOn, an open
source framework designed to automate historical and
complexity metrics tracking of HDL projects.
Keywords
HDL, design flow, quality, metrics, validation
1. Introduction
Verifying industrial scale designs is still a challenging
task that consumes most part of total development efforts.
International Technology Roadmap for Semiconductors
reported that verification stage is a bottleneck that has
actually reached crisis proportions [1]. Considering that
there is no formal or simulation tool capable to stimulate all
possible states in an industrial scale design, verification
team must work smarter, wisely spending their resources. In
order to handle the still increasing design complexity,
improvements in current verification methodologies are
needed. A step in this direction is to benefit from software
engineering experience. Currently, hardware and software
development methodologies have many aspects in common:
requirements specification; use of description languages;
identification of functional design errors; use of versioning
systems and bug tracking tools.
Software evolution is a mature and well established
research area in software engineering [2,3]. Software
evolution research relies on extracting and processing
information from:
• Version Control Systems (VCS) repositories: These
systems store all revisions of the software source
code. Freely available platforms are Subversion
978-1-61284-914-0/11/$26.00 ©2011 IEEE
442
12th Int'l Symposium on Quality Electronic Design
in Section IV we provide details of the framework. In
Section V a few hardware evolution key visualization
possibilities are presented and a case study is discussed.
Finally, in Section VI, we conclude our remarks and present
future work directions.
2. Related work
While software evolution has been studied for at least the
last three decades [13, 14], research on hardware evolution
is still incipient. Therefore, there are no tools or frameworks
designed to track HDL evolution. A preliminary work was
proposed by [15]. In this paper, some historical metrics and
bug reports of HDL repositories are retrieved and stored on
a database. The main contribution of this work is to define a
language to describe bug reports. This language is used to
attach bug reports on the commit messages, making possible
to automatically retrieve and display this information. In
[16], Nacif et al. use design history information to identify
error-prone modules and allocate verification resources.
The most complete tools and frameworks designed
process repositories are legacy from software evolution
research community. The most importants are CVSAnalY,
Release History Database (RHDB), Kenyon, and Alitheia.
Unfortunately, none of them are capable of handling HDL
specific complexity metrics.
CVSAnalY [17] is a repository data extractor. It was
initially built to collect revision data from CVS repositories.
They analyzed the GNOME project and presented statistics
about inactivity, commiters, commits, lines changed, lines
modified, lines removed, first and last commit. Based on
these statistics, they identified relationships and
characteristics among modules and commiters.
RHDB [18] is used to join historical information from
CVS and Bugzilla, correlating their relationships. RHDB
was used on Mozilla web browser, a well-known open
source project. Internal function calls were tracked and
mapped to call graphs, registering their evolution with
collected modification and bug reports.
Kenyon [19] is a software evolution tracker. It was
designed to facilitate software evolution research providing
an environment with repositories connectors, metric
extractors, and API points that could be extended. The
collected data is stored on a relational database through
Object-Relational Mapping.
Alitheia [20] tool aggregates similar functionalities, but
is designed to be a service oriented architecture and focused
in having good performance on large scale data extraction.
They are the most complete tools designed to facilitate
software evolution tracking. A detailed work comparing
different software evolution trackers is presented in [9].
Considering the great number of metrics and the huge
amount of objects being analyzed the visualization of this
data is a challenging problem. RelVis [21] is a tool that
provides multiple metrics visualization in time using Kiviat
diagrams, also known as radar charts. RelVis also supports
the inclusion and visualization of software entities and their
relationships over releases. GEVOL [22] is a system that
uses graphs and colors to represent Java classes evolution.
Eick et al. [23] presented different visualization options that
vary from simple pie charts to complex diagrams like
network, matrix and 3D views.
3. Methodology
In this Section, we describe the methodology to track
hardware evolution. Basically, two different application
scenarios are described. These application scenarios are
illustrated on Figure 1.
In the first scenario, called hardware evolution tracking,
the metrics are retrieved from the Repository and displayed
on different formats that include:
• Bar and line charts: present metrics evolution over
commits or periods of time;
• Radar charts: useful to compare the evolution of
metrics on different axis and periods of time;
• Heatmap diagrams: good for visualizing module’s
specific metrics, such as modification rate and bug
proneness.
Figure 1: Hardware evolution application scenarios.
It’s important to note that hardware evolution tracking
can include also metrics extracted by commercial Electronic
Design Automation (EDA) tools.
The second application scenario facilitates the use of
advanced data mining and statistics techniques. This is
where hardware evolution research starts, experimenting the
effectiveness of different software evolution techniques on
HDL hardware projects.
12th Int’l Symposium on Quality Electronic Design
4. Framework architecture
Finding integrated environments to automate hardware
evolution research is a key problem. Having that in mind,
we designed and implemented EyesOn, an open source,
extensible framework. Our framework not only automates
the extraction and storage of HDL complexity metrics, but
also provides visualization and exporting capabilities. The
organization of the proposed framework is illustrated on
Figure 2.
information of HDL designs. Although VCS tools store
revision information in different formats, key data like tags,
commits, comments, and authors are somehow standardized.
Repository Extractor is responsible for preprocessing the
Repository, retrieving revision history information that will
be stored in the Database. The revision information will
guide metrics extraction.
4.2. Metrics extractor
Metrics Extractor is the module responsible for
extracting and storing different design metrics: HDL,
External, Bug, and Repository.
Revision
history
information
gathered by Repository Extractor is
useful to observe project specific
state points over a timeline.
4.2.1. HDL metrics.
In order to extract static metrics
from source code, Repository
versions are retrieved on demand by
Source Extractor. Some metrics
such as LoC (Lines of Code) or
comment lines are very simple to
implement, but there are other
metrics like gate count that depend
on HDL parsers and synthesis tools.
The framework HDL parser is
capable to extract metrics like
statements, I/O signals, operators,
and operands.
Figure 2: Framework architecture.
There are two storage components, namely Repository
and Database. The first is populated by the designers during
the hardware development process and holds important
information about the modules development history. This
information include: which module was modified; when the
module was modified; how many lines of code were
included, removed or modified; who was the designer
responsible for a modification. The Database stores the same
information, but in a relational model facilitating queries
and flexibilizing the relationship among different metrics. In
addition, we have modules for data manipulation (extractors,
parsers, metrics, visualization, and exporter).
The framework architecture is designed to allow
integration among repositories and metric extractors. The
data is extracted by the Repository Extractor, preprocessed
and inserted into the Database. EyesOn provides support for
researchers to retrieve project files and extract metrics,
which can be visualized using the Data Visualization
module. It is also possible to build queries and to exchange
data. In the next Sections, we explain the main modules
characteristics and the features presented in the framework
architecture.
4.1. Repository extractor
Repository Extractor is a module that connects to VCS
tools like CVS, Subversion, and Git retrieving temporal
4.2.2. External metrics.
In order to increase the framework flexibility, it is
possible to include metrics extracted by external tools. As an
example, we used a commercial EDA tool that provides the
following metrics: latches, flip-flops, equivalent gate count,
nets, and instances.
4.2.3. Bug metrics.
EyesOn is also capable of extracting bug data from BTS
and VCS repositories. Simple VCS transaction detection
heuristics are provided based on similar commit messages
and intersections of VCS with BTS modification intervals.
4.2.4. Repository metrics.
Repository metrics are useful to contextualize the other
metrics in time. This information provides support to study
the relation among different classes of events like commit or
bug reports. Examples of Repository Metrics are: commits
comments, contributions by author, revision differences,
transactions, and coupling among modifications.
4.3. Extension points
Our framework extensibility relies on its capability to
adapt to new tools, metrics and to accept parameterized data
from external applications. In order to support these new
features we provided extension points for metrics,
repositories, and external extractors.
12th Int’l Symposium on Quality Electronic Design
4.4. Flexible database
Despite
making
metric
extraction more agile, other
motivation to use a relational
database is to facilitate queries.
Due to the capabilities of
Hibernate, an Object-Relational
Mapping (ORM) framework,
different Database Management
System (DBMS)
can
be
configured to store data.
4.5. Data Visualization
Efficient data visualization to
track
files
and
software
evolution has become a specific
research area [24]. The data
visualization module allows
users to integrate and generate
parameterized charts. We also
provide
a
set
of
data
visualization formats like bar, radar and liness.
5. Results
As a case study, we used revision histtory information
from a MIPS32 processor core. This core w
was divided into
17 modules that were developed during fourr months. All the
modifications on these modules were reportted and stored at
the repository as commit messages.
In order to observe when bugs were discovered during
the development process, we built a bbi-weekly chart
depicted in Figure 3. The bars represent tthe accumulated
bugs found by the time interval.
Figure 3: Number of errors per time intervall.
Figure 4 depicts extracted metrics changging as hardware
designers commit new versions of MIPS32 Decode pipeline
stage module. The commit motivation is aalso depicted by
colored bars behind the chart. Each colorr represents one
motivation.
The resume modification means that a designer just
commited modifications to the design, withoout reporting any
error. Errors could be found by inspection, simulation, or
Figure 4: Metrics changing betweeen commits.
synthesis. There are also code op
ptimization, simplification
and documentation motivations.
The first five commits are cllassified as resume. The
increase of metrics values between
n commits #1 and #5 can
be interpreted as designers inserrting RTL code on this
module. After commit #5, the main
n reason for modifications
is the existence of errors that coulld be found by inspection
and simulation. The number of reg
gisters (latches and flops)
and gates decreases its value. On commit #10, the
ptimization but only the
modification is classified as op
number of nets has changed. During
D
a series of bug
correction between commit #12 and #15, the number of flipflops has an expressive variation
n. During a resume after
commit #16 all flip-flops are synthetized as latches, but after
the commit #18 the code is correcteed and the number of flipflops returns to its original value. After the commit #23 the
RTL code becomes more sttable requiring smaller
modifications.
A simplified view of metricss evolution is shown in
Figure 5. This chart presents metrics
m
from the MIPS32
Memory pipeline stage module. The
T presented metrics are
Registers, Gates, Nets, Lines of Code, and Accumulated
w
normalized. This
Fixed Errors. The values were
visualization is fragmented on tiime, showing only three
evolution steps. They are composed
d by commits #6, #18 and
#23, respectively colored by red (g
gray in gray scale copies),
blue (dark gray) and green (light gray). We can observe that
the Accumulated Fixed Errors meetric increase its value as
new revisions are done. Both Lines
L
of Code and Gate
metrics increase their values on co
ommit #18, but shrink on
commit #23, possibly due to a deesign simplification. Nets
metric value is also decreased in commit #23. Finally, the
Registers metrics keep its value over the three presented
revisions.
12th Int’l Symposium on Quality Electronic Design
Figure 5: Radar visualization.
5.1. Case study: module error proneneess
In this Section we present a case sstudy that uses
hardware evolution metrics to calculate a suubset of modules
that are most prone to contain undetectted errors. The
algorithm, presented on Figure 6, uses the cooncept of a small
cache (in our case, of size 3) that stores the M
Most Frequently
Fixed Modules (MFF). This algorithm presented good
results for prediction of software modules error proneness.
More details can be found in [25].
01:
02:
03:
04:
05:
06:
07:
08:
09:
10:
11:
12:
13:
14:
15:
16:
17:
ComputeMFF ( FixedModuleLis
st )
initialize cache
for each module on FixedMod
duleList
if cache is empty
store module with freq=1
else
if module is already on the cache
freq = freq + 1 (max=3)
store module order nu
umber
else
if cache size == MAX_CACHE_SIZE
choose the oldest m
module with ↵
the least freq valu
ue
drop this module fr
rom cache
store module with freq
eq=1
store module order nu
umber
return cache
Table I: Most Frequently Fixed algorithm execution on
MIPS32 project.
Cache Slots
Next Fixed
Time
Freq.
F
Order
Module
1
2
3
1
Alu
2
1
1
Alu
Alu
3
2
2
Alu
Alu
4
Alu
3
3
Alu
5
Alu
3
4
Ram
6
Alu Ram
1
5
Ram
7
2
6
Registers
Alu Ram
8
7
Alu
Alu Ram Registers 1
9
8
Shifter
Alu Ram Registers 3
10
Alu Ram Shifter
1
9
Shifter
11
Alu Ram Shifter
2
10
Comp
12
Alu Comp Shifter
1
11
Gdm
13
1
12
Alu Gdm Shifter
...
Figure 7 shows a different vissualization for most error
prone modules, the heat map chaart. This chart illustrates
modules that are colored according to the module presence
in cache. Green (light gray in gray
yscale copies) means low
error proneness, while read (dark gray) means high chance
of error. Each module was represen
nted by a rectangle where
its area is proportional to its LoC size.
s
This is a much more
intuitive way of visualizing the mo
odules error proneness. In
this chart the modules names are represented by numbers.
c
respectively to
Modules number #2, #6, and #3 correspond
modules Alu, Gdm, and Shifter.
Figure 6: Most Frequently Fixed algorithm.
The cache frequencies are classified bby numbers, (3)
very frequently, (2) frequently, (1) loaded inn cache. As new
modules are fixed, it populates a cache slott. If a module is
more frequently fixed it have more chancees to remain on
cache. Always when a new module is fixed the least
frequently module is dropped. When the frequencies are
equal, the module order number is used tto eliminate the
oldest module. Table I depicts the usage off MFF algorithm
presented on Figure 6. There are three cachee slots that store
the most error prone modules.
Figure 7: Heatmap visualization
n.
12th Int’l Symposium on Quality Electronic Design
6. Conclusion and future work
Tracking hardware evolution offers many benefits. It can
improve the quality of electronic design. We proposed the
use of information available in HDL repositories to track
hardware evolution. We presented EyesOn, an extensible,
open-source framework designed to automate hardware
evolution tracking. We illustrated the advantages of
hardware evolution tracking presenting and discussing a
case study where hardware evolution tracking were used to
correlate HDL metrics and bug proneness of Verilog HDL
modules.
As future work, we intent to extract HDL metrics in
order to build HDL quality predictor models. We are also
improving the set of EyesOn supported metrics as well as
visualization options.
7. Acknowledgements
This work is supported by CNPq and FAPEMIG under
grants #PNM-141201/2005-3, #CEX-1485-06, and #APQ02217-10.
8. References
[1]
“International technology roadmap for semiconductors
design,”.
2009.
[Online].
Available:
http://www.itrs.net
[2] H. Kagdi, M. L. Collard, and J. I. Maletic, “A survey
and taxonomy of approaches for mining software
repositories in the context of software evolution,” J.
Softw. Maint. Evol., vol. 19, no. 2, pp. 77–131, 2007.
[3] K. H. Bennett and V. T. Rajlich, “Software
maintenance and evolution: a roadmap,” in ICSE ’00:
Proceedings of the Conference on The Future of
Software Engineering. New York, NY, USA: ACM,
2000, pp. 73– 87.
[4] B. Collins-Sussman, B. W. Fitzpatrick, and C. M.
Pilato, Version Control with Subversion. O’Reilly
Media, 2004.
[5] P. Cederqvist, Version Management with CVS. Free
Software Foundation, Inc., 2004.
[6] The Bugzilla Guide, 2010. [Online]. Available:
http://www.bugzilla.org
[7] Mantis Bug Tracker, 2010. [Online]. Available:
http://www.mantisbt.org
[8] M. DAmbros, H. C. Gall, M. Lanza, and M. Pinzger,
Software Evolution. Springer, 2008, ch. Analyzing
software repositories to understand software evolution,
pp. 37–67.
[9] G. Gousios, “Tools and methods for large scale
software engineering research,” Ph.D. dissertation,
Athens University of Economics and Business, 2009.
[10] W. Stapleton and P. Tobin, “Verification problems in
reusing internal design components,” in DAC ’09:
Proceedings of the 46th Annual Design Automation
Conference. New York, NY, USA, 2009, pp. 209–211.
[11] D. L. Weaver, OpenSPARCTMInternals, 2008.
[12] OpenRISC 1000 Architecture Manual, 2004. [Online].
Available: http://www.opencores.org
[13] I J. P. Cavano and J. A. McCall, “A framework for the
measurement of software quality,” in Proceedings of
the software quality assurance workshop on Functional
and performance issues, 1978, pp. 133–139.
[14] V. R. Basili, L. C. Briand, and W. L. Melo, “A
validation of objectoriented design metrics as quality
indicators,” IEEE Trans. Softw. Eng., vol. 22, no. 10,
pp. 751–761, 1996.
[15] T. N. C. Cardoso, J.A. Nacif, A. O. Fernandes, C. N.
Coelho. “BugTracer: A system for integrated circuit
development tracking and statistics retrieval” in
Proceedings of Latin American Test Workshop
(LATW), Búzios, Rio de Janeiro, Brazil, 2009.
[16] J.A. Nacif, T. Silva, A. I. Tavares, A. O. Fernandes,
C.N. Coelho. “Efficient Allocation of Verification
Resources using Revision History Information” in
Proceedings of the Workshop on Design and
Diagnostics of Electronic Circuits and Systems
(DDECS), Bratislava, Slovakia, 2008.
[17] G. Robles, S. Koch, and J. M. Gonzalez-Barahona,
“Remote analysis and measurement of libre software
systems by means of the CVSAnalY tool,” in
Proceedings of Remote Analysis and Measurement of
Software Systems (RAMSS), Edinburg, UK, 2004.
[18] M. Fischer, M. Pinzger, and H. Gall, “Analyzing and
relating bug report data for feature tracking,” in
WCRE ’03: Proceedings of the 10th Working
Conference on Reverse Engineering. Washington, DC,
USA, 2003.
[19] J. Bevan, E. J. Whitehead, Jr., S. Kim, and M. Godfrey,
“Facilitating software evolution research with
kenyon,” in Proceedings of the European software
engineering conference, New York, USA, 20056.
[20] G. Gousios and D. Spinellis, “Alitheia core: An
extensible software quality monitoring platform,”
Software Engineering, International Conference on,
vol. 0, pp. 579–582, 2009.
[21] M. Pinzger, H. Gall, M. Fischer, and M. Lanza,
“Visualizing multiple evolution metrics,” in
Proceedings of the 2005 ACM symposium on
Software visualization. ACM, 2005, pp. 67–75.
[22] C. Collberg, S. Kobourov, J. Nagra, J. Pitts, and K.
Wampler, “A system for graph-based visualization of
the evolution of software,” in Proceedings of the 2003
ACM symposium on Software visualization. 2003.
[23] S. Eick, T. Graves, A. Karr, A. Mockus, and P.
Schuster, “Visualizing software changes,” IEEE
Transactions on Software Engineering, vol. 28, no. 4,
p. 412, 2002.
[24] L. Voinea, A. Telea, and M. Chaudron, “Version
centric visualization of code evolution,” in
Proceedings of Eurographics/IEEE-VGTC Symposium
on Visualization. Citeseer, 2005.
[25] A. Hassan and R. Holt, “The top ten list: Dynamic fault
prediction,” in Software Maintenance, 2005. ICSM’05.
Proceedings of the 21st IEEE International Conference
on, 2005, pp. 263–272.
12th Int’l Symposium on Quality Electronic Design