Guide < NetFPGA/OneGig < Fo

Transcrição

Hide Contents...
Contents
file:///C:/Users/Cesar/Desktop/Guide.htm
Introduction
Usage Models
Major Components
How to read this Guide
to set up a laboratory
to use the NetFPGA packages
Connecting with the Community
Track Bugs with Bugzilla
NetFPGA Forums
NetFPGA-Beta Email list
NO GUARANTEES
Obtain Hardware and Software
Obtaining NetFPGA Hardware
Ordering From the Web
Ordering with a Purchase Order by Email or Phone
Obtaining a Host PC for the NetFPGA
Assemble your PC from Components
List of PC Components
Motherboard
CPU
Host Memory
DVD Reader/Writer (for boot disk)
MicroATX Chassis with clear covers
Intel Pro/1000 Dual-port Gigabit PCI-Express PCI-express x4 NIC
Hard Disk
Cat5E or Cat6 Ethernet Cables
Other Misc. Parts
Total estimated cost to build a cube
Purchase a Dell 2950
Purchase a Pre-built Machine
Obtaining Gateware/Software Package
Register to download the NetFPGA Package (NFP)
Download the NetFPGA Package (NFP)
Install Software
Quick Start
Minimal Install
Quick Development Install
Quick Xilinx Tool Install
Installing an Operating System on the Host PC
8/8/2010 10:20 PM
The NetFPGA is a low-cost platform, primarily
designed as a tool for teaching networking
hardware and router design. It has also proved to
be a useful tool for networking researchers. Through partnerships and donations from sponsor of the
project, the NetFPGA is widely available to students, teachers, researchers, andNF anyone else interested
in experimenting with new ideas in high-speed networking hardware.
Introduction
You are here: Foswiki>NetFPGA/OneGig Web>Guide (26 Jun 2010, Main.JonathanEllithorpe)EditAttach
Guide < NetFPGA/OneGig < Foswiki
1 de 56
CentOS Installation Instructions
Other tested but unsupported operating systems
Software Installation
Log in as root
Install RPMforge Yum repository
Install NetFPGA Base Package
Create netfpga directory in your user account
Reboot your machine
Install CAD Tools
Install Xilinx ISE
Install ModelSim
Debug with ChipScope
Install Memory Modules for Simulation
Micron DDR2 SDRAM
Cypress SRAM
Verify the software and hardware
Compile and Load Driver
Compile driver and tools
Load driver and tools
Verify NetFPGA interfaces
Reprogram the CPCI
Run Selftest
Connect loopback cables
Bring nf2cX interfaces up
Load self-test bitfile
Run Selftest
Run Regression Tests
Connect Ethernet test cables
Log in as root through X session
Load reference_router bitfile
Run regression test suite
Run regression scripts on new bitfile
Synthesize reference_router bitfile, from source
Load new bitfile
Run regression-test suite on new bitfile
Walkthrough the Reference Designs
Reference NIC Walkthrough
Using counterdump
Using send_pkts
Understanding the Hardware/Software Interface
Reference Pipeline
What to do From Here
SCONE Walkthrough
What is SCONE?
How to use SCONE?
How does SCONE work with NetFPGA?
Writing MAC Addresses to the NetFPGA
Writing interface IP Addresses to the NetFPGA
Writing routing entries to the NetFPGA
Writing ARP entries to the NetFPGA
Router Kit Walkthrough
Overview
Running Router Kit
Using Router Kit
2 de 56
8/8/2010 10:20 PM
The
NetFPGA
offloads
8/8/2010 10:20 PM
At a high level, the board contains four 1 Gigabit/second Ethernet (GigE) interfaces, a user programmable
Field Programmable Gate Array (FPGA), and four banks of locally-attached Static and Dynamic Random
Access Memory (SRAM and DRAM). It has a standard PCI interface allowing it to be connected to a
desktop PC or server. A reference design can be downloaded from the http://NetFPGA.org website that
contains a hardware-accelerated Network Interface Card (NIC) or an Internet Protocol Version 4 (IPv4)
router that can be readily configured into the NetFPGA hardware. The router kit allows the NetFPGA to
interoperate with other IPv4 routers.
Usage Models
How it Works
Reference Router Walkthrough
Java GUI
Command Line Interpreter
Modifying the Reference Router
Reference Pipeline Details
Register Pipeline
Outside the Reference Pipeline
Using a Library Module
Adding New Sources
Simulating the design
Implementing the Design
Testing the New Router
Buffer Monitoring System
Introduction
Using the system
monitor_ctrl
rcv_evts
Design Details
evt_rcrdr
evt_pkt_wrtr
Schematic
Packet Format
Links
Schematic and board layout
Contributed Packages
Tutorial Setup
Other Pages
License
3 de 56
8/8/2010 10:20 PM
1. Use the hardware as is as an accelerator and modify the software to implement new protocols. In
this scenario, the NetFPGA board is programmed with IPv4 hardware and the Linux host uses the
Router Kit Software distributed in the NFP. The Router Kit daemon mirrors the routing table and
ARP cache from software to the tables in the hardware allowing for IPv4 routing at line rate. The
user can modify Linux to implement new protocols and test them using the full system.
2. Start with the provided hardware from the official NFP (or from a third-party NFP), modify it by
using modules from the NFP's library or by writing your own Verilog code, then compile the source
code using industry standard design tools. The implemented bitfile can then be downloaded to the
FPGA. The new functionality can be complemented by additional software or modifications to the
existing software. For the IPv4 router, an example of this would be implementing a Trie longest
prefix match (LPM) lookup instead of the currently implemented CAM LPM lookup for the
hardware routing table. Another example would be to modify the router to implement NAT or a
firewall.
3. Implement a new design from scratch: The design can use modules from the official NFP's library or
third party modules to implement the needed functionality or can use completely new source code.
NetFPGA is to start with the reference router and extend the design with a custom user module. Finally, it
is also possible to implement a completely new design where the user can place their own logic and data
processing functions directly in the FPGA.
Another
way to
modify
the
NetFPGA packages (NFPs) are available that contain source code (both for hardware and software) that
implement networking functions. Using the reference router as an example, there are three main ways that
a developer can use the NFP. In the first usage model, the default router hardware can be configured into
the FPGA and the software can be modified to implement a custom protocol.
processing from a host processor. The host's CPU has access to main memory and can DMA to read and
write registers and memories on the NetFPGA. Unlike other open-source projects, the NetFPGA provides
a hardware-accelerated hardware datapath. The NetFPGA provides a direct hardware interface connected
to four GigE ports and multiple banks of local memory installed on the card.
4 de 56
8/8/2010 10:20 PM
If your task is to set up machines, start by reading the steps to obtain hardware and software , follow the
steps to install software , then verify the software and hardware .
to set up a laboratory
Depending on your goals, you may find certain chapters of this guide more relevant than others.
How to read this Guide
http://www.broadcom.com/products/Enterprise-Networking/Gigabit-Ethernet-Transceivers/BCM5464SR
More information about BCM5464SR chip:
The Broadcom BCM5464SR Gigabit/second external physical-layer transceiver (PHY) sends packets
over standard category 5, 5e, or 6 twisted-pair cables. The quad PHY interfaces with four Gigabit
Ethernet Media Access Controllers (MACs) instantiated as a soft core on the FPGA. The NetFPGA also
includes two interfaces with Serial ATA (SATA) connectors that enable multiple NetFPGA boards in a
system to exchange traffic directly without use of the PCI bus. In order to get
Two 18 MBit external Cypress SRAMs are arranged in a configuration of 512k words by 36 bits (4.5
Mbytes total) and operate synchronously with the FPGA logic at 125 MHz. One bank of external Micron
DDR2 SDRAM is arranged in a configuration of 16M words by 32 bits (64 MBytes total). Using both
edges of a separate 200 MHz clock, the memory has a bandwidth of 400 MWords/second (1,600
MBytes/s = 12,800 Mbits/s).
Virtex2-Pro 50 FPGA which is programmed with user-defined logic and has a core clock that runs at
125MHz. The NetFPGA platform also contains one small Xilinx Spartan II FPGA holding the logic that
implements the control logic for the PCI interface to the host processor.
The
NetFPGA
platform
contains
one large
Xilinx
A block diagram that shows the major components of NetFPGA platform is shown below.
Major Components
5 de 56
Obtaining NetFPGA Hardware : How you can acquire NetFPGA hardware
8/8/2010 10:20 PM
The first thing to be done is putting the board in the the box and making sure it runs. To get started, you'll
need to perform the following steps:
Obtain Hardware and Software
We do not guarantee that any or all of the NetFPGA components will work for you. FPGAs allow for an
enormous range of freedom in the implementation of circuits. We do not guarantee that anything you get
from us will not damage the hardware on the NetFPGA, the software on the PC, or anything else. And
finally we do not guarantee that you will get support. However, we do guarantee that we did/will do our
best. Hence, the license.
NO GUARANTEES
This is here for historic purposes. The email list has been superseded by the NetFPGA Forums (see
above). Please DO NOT use the beta e-mail list!
NetFPGA-Beta Email list
Feel free to use the forum for:
Announcements on progress with the NetFPGA
General questions about the scripts and code
Answers to questions (feel free to contribute)
Submit patches! We always welcome these
We've created a forum where users can post their questions and have them answered. The forum is easily
searchable and anyone can register. It is available here.
NetFPGA Forums
Create an account for yourself
Search for a bug
View summaries
Report a bug
We track and maintain bugs using BugZilla
Track Bugs with Bugzilla
We encourage feedback and discussion about the progress and problems with the NetFPGA. An
bug-tracking system called Bugzilla is available to read about and post bugs. A forum is available to
communicate with other members of the community and submit patches.
Connecting with the Community
If you already have NetFPGA systems up and running in your laboratory and want to understand how it
works, read the walkthroughs of the Reference Designs to understand the operation of the reference
NIC, the software component of the router (SCONE), the router kit, the reference router hardware, and
the buffer monitoring system.
to use the NetFPGA packages
6 de 56
(BOM) below to do the same.
Obtaining a Host PC for the NetFPGA : How you can buy or build your Host PC.
CPU
8/8/2010 10:20 PM
Use Micro ATX (uATX) for small case
Option 1: Gigabyte MA78GM-US2H mATX MB
AMD 780G Chipset / SB 700 / Rev 1.0
Includes one port of GigE on the motherboard
Includes ATI Radeon HE3200 Video (leaves PCI-E slot open)
DDR2 1200 DRAM support (supports RAM faster than DDR2 800)
2PCI+PCIe x1+PCIe x16
AM2+ Phenom II Support (allows for use of quad-core CPU)
Be sure to upgrade BIOS to latest available to make use of all features
Availability
$84.99 @ Frys
$79.99 at NewEgg as Item=N82E16813128379
Option 2: ASUS M2N-VM DVI - Micro ATX Motherboard
Item=N82E16813131214 from NewEgg.com : $59.99
Set the BIOS to use the on-board Video. Use the PCI-express bus for the NIC
We built a dozen nf-test cube machines at Stanford using this motherboard in 2007-2008
combined with the dual-core CPU. If you can't still locate this (now) older board, use option
(1)
Motherboard
At Stanford, we built 11 nf-test PCs that we use in the lab and used at the North American tutorials. This
is the least expensive way to build a high-end development system, but does take some time to assemble
the parts.
Components
List
of PC
Obtain Designs : Projects:Packet_generator
Contributed Designs
Obtaining NetFPGA Hardware
The NetFPGA boards can be obtained from a third-party company, Digilent Inc.
The cards are sold for a discounted price when used for Educational purposes.
They are also available for commercial use, but pricing is higher. Stanford
University provided the open reference design to Digilent Inc., but is not involved
in the sale of the hardware.
Complete NetFPGA systems can also be ordered on-line that include the NetFPGA hardware pre-installed
in a host PC.
Ordering From the Web
The easiest way to purchase hardware is to order on-line from Digilent
Ordering with a Purchase Order by Email or Phone
Academic institutions can order the hardware with a discount by placing a purchase order.
To start the process, send an email to request a quote from: [email protected]
Have your university execute a purchase order and have that sent to Digilent Inc.
For help with ordering, call: (509) 334-6306
Obtaining a Host PC for the NetFPGA
NetFPGAs host systems can be built from commodity, off-the-shelf (COTS) parts. The NetFPGA card fits
into a standard PCI slot in a desktop or server-class PC. We have only tested the NetFPGA in a few of a
few widely-available systems. The NetFPGA may work with other PCs as well, but we do not support
such configurations. In fact, according to this thread, current versions of board will work on most PC's.
There are currently multiple ways to obtain a NetFPGA host system:
1. Assemble your on PC using from components
2. Purchase a Dell 2950 from Dell.com then add the NetFPGA.
3. Purchase a complete pre-built system
To install a NetFPGA, you will need to open the case to your computer. To minimize the chance that you
damage your computer or the NetFPGA module, we suggest that you wear an anti-static wrist strap when
handing the hardware.
Assemble your PC from Components
The most cost-effective way to build a high-performance NetFPGA host system is to purchase the
components from on-line vendors and assemble your own machine. This effort is not for the faint of heart,
however, as you will need to place multiple orders for components and have the time to assemble and test
the PC. We assembled all of the nf-test machines at Stanford University. You can use the Bill of Materials
8/8/2010 10:20 PM
8 de 56
Obtaining Gateware/Software Package: How you can get an account and download the Beta package
from NetFPGA.org
7 de 56
Other Misc. Parts
Power Strip for PC and monitor with localized plug:
Use existing or purchase for $5
SATA Cable
Typically included in a new motherboard box
8/8/2010 10:20 PM
c:74|&Sort=0&Recs=10 Category 5e or c:74|&Sort=0&Recs=10 Category 6 Ethernet Cables
Short-length: 1 foot ~= 30 cm, Blue (for host)
Short-length: 1 foot ~= 30 cm, Orange (for host)
Medium-length: 6 foot ~= 2m, White (for neighbor machine)
Medium-length: 6 foot ~= 2m, Red (for neighbor machine)
Long-length: 12 foot ~= 4m, Blue (for Internet)
Cat5E or Cat6 Ethernet Cables
250 GByte minimum
500GB for $64
Or larger
Hard Disk
Mfg Part#: EXPI9402PT from Buy.com : $155.99
Intel Pro/1000 Dual-port Gigabit PCI-Express PCI-express x4 NIC
APEVIA X-QPACK2-BK/500 Black Aluminum Body
Item=N82E16811144139 from NewEgg.com: $99
MicroATX Chassis with clear covers
DVD Reader/Writer (for boot disk)
DDR2 DRAM: 2GB to 4GB
Minimum of 2GB is needed to efficiently run Xilinx CAD tools
Host Memory
We use fast dual or quad-core systems to maximize efficiency of developers
Option 1: AMD X4 940 Quad-Core (3 GHz) AM2+ CPU
For use in AM2+ Motherboard, (1) above
Item=N82E16819103471 from NewEgg.com: $189, or
Item CP1-AM2-940 from TigerDirect.com, Model HDZ940XCGIBOX- $199
Option 2: AMD X2 Dual-Core (3.2 GHz) AM2 6400+ CPU
For use in AM2 Motherboard, (2) above
Item=N82E16819103228 from NewEgg.com : $169
Option 3: AMD X2 Dual-Core (3.0 GHz) AM2 6000+ CPU
Item=N82E16819103773 from NewEgg.com : $99
Use the fastest clock frequency to minimize time to place and route the FPGA
9 de 56
8/8/2010 10:20 PM
During shipment, the printed circuit board can vibrate or shake within the chasis causing mechanical
damage. Systems should not be shipped with the NetFPGA card pre-installed.
To use the card in the Dell 2950, a laser-cut extender was built to enable the card to extend to the end of
the card slot. This may help if you ship the NetFPGA in the 2950, or if the 2950 is in a high-vibration
environment. Support bracket design files and ordering info.
The actual bracket required depends on the size of the chassis. http://www.bracket.com/retainerslist.asp
For PCs that have standard full-length PCI slots, Retainers to secure the back of the NetFPGA to a chassis
are available from Gompf as: http://www.bracket.com/downloads/brackets/pdf/91060000AFG.pdf
In addition to
locking in the
faceplate at the
front of the
system, the
card should
also be locked
in at the rear of
the card using a
mounting
bracket. As
shown below,
there is a gap
between the
end of the card
and the slot that
holds a
full-length PCI card, shown shown below:
Note: When
installing the
NetFPGA in a
system, it is
important that
the card is
securely
fastened to the
chassis.
A pre-configured Dell 2950 2U Rackmount PC can be purchased from Dell. We have verified that the
NetFPGA works in the PCI-X slot of the Dell 2950 2U Rack-mount server. The cost for a pre-built Dell
server typically ranges from $3,000 to $5,000 depending on the configuration you select. Running the
selftest requires purchasing a SATA cable and two Ethernet cables.
Purchase a Dell 2950
About $700 USD
Note that prices will vary
(but generally become less expensive over time)
Total estimated cost to build a cube
10 de 56
http://www.accenttechnologyinc.com/product_details.php?category_id=0&item_id=1
During shipment, the
printed circuit board can
vibrate or shake within
the chasis causing
mechanical damage.
Systems should not be
shipped with the
NetFPGA card
pre-installed. Cards
should be shipped
separately from the
chassis and installed on
site to avoid damage.
Obtaining
8/8/2010 10:20 PM
A third-party vendor has just started building complete system with the NetFPGA hardware and software
pre-installed. The complete turn-key system, including the NetFPGA card, are available from Accent
Technolgy Inc.
Purchase a Pre-built Machine
11 de 56
8/8/2010 10:20 PM
Install CAD Tools: Install Computer Aided Design tools to enable synthesis and simulation of hardware
circuits (Optional)
Software Installation: Install the NetFPGA device driver and self-test program & bitfile
Installing an Operating System on the Host PC: Describes how to install CentOS on the host computer
Quick Start
Install Software
Download the tarballs from Releases. Later, you will unpack them in the same directory.
netfpga_full_x_x.tar.gz, which includes regression scripts and binary versions of the reference
projects. Replace 'x' with the latest version.
netfpga_lib.tar.gz, which includes all external java code needed by the router gui
The NFP currently comprises two tarballs:
Download the NetFPGA Package (NFP)
1. Sign up for a FosWiki account as:
http://netfpga.org/foswiki/bin/view/NetFPGA/UserRegistration
2. when your new account is created:
1. you will be automatically added the NetFPGA Announce mailing list.
1. This email list will be used to post announcements about the NetFPGA
2. Let your SPAM filter pass email for: [email protected]
2. you will be sent an email for a message from NetFPGAwiki.
1. within that message will be URL that needs to be opened
2. click on the URL to verify that the email address you provided is valid.
3. if you have an account but have forgotten your password, click the e-mail password button on:
http://www.netfpga.org/foswiki/bin/view/NetFPGA/ResetPassword
To download the NFP:
Register to download the NetFPGA Package (NFP)
The instructions in this section have been superseded by the instructions in the Install Software 1.2
section below.
The Beta release of the NetFPGA Package (NFP) contains the source code for gateware, system software,
and regression tests. The NFP includes an IPv4 Router, a four-port NIC, an IPv4 Router with Output
Queues Monitoring System, the PW-OSPF software that interacts with the IPv4 Router (SCONE), and the
Router Kit which is a daemon that reflects the routing table and ARP cache from the Linux host to the
IPv4 router on NetFPGA.
Gateware/Software Package
12 de 56
8/8/2010 10:20 PM
If you want a full development install (that builds the reference bitfiles and runs their tests), first do the
Minimal Install, then:
Quick Development Install
Install NetFPGA software (because you at least need to install and build nf_download and the
kernel driver for this machine)
Either: tgz package:
untar the official tgz into your home directory from release: http://netfpga.org/foswiki
/NetFPGA/OneGig/Releases
Or: git repository:
install git from the latest bzip (./configure; make; sudo make install)
in your home directory: git clone git://netfpga.org/netfpga.git # also see gitweb
you'll need at least the cpci*.bit bitfiles from the tgz release above; just copy all bitfiles from
their bitfile directory into your ~/netfpga/bitfiles.
plug in NetFPGA card boot up. If boot crashes first time, reboot and it should work.
add uppermem and vmalloc to /boot/grub/grub.conf and reboot (check forum for exact syntax)
I'm afraid that to start up the device with the cpci_reprogram.pl tool, you're going to have to put the
necessary NF environment variables from netfpga/bashrc_addon into your ~/.bashrc then re-login
link /usr/local to your own user copy so that you don't have to keep two copies:
sudo ln -s ~/netfpga /usr/local/netfpga
sudo ln -s ~/netfpga/lib/scripts/cpci_config_reg_access/loadregs.sh /usr/local/sbin/loadregs.sh
sudo ln -s ~/netfpga/lib/scripts/cpci_config_reg_access/dumpregs.sh /usr/local
/sbin/dumpregs.sh
# WARNING: don't forget last step or cpci_reprogram.pl won't work but will tell it has.
If you only care about installing the driver and .bit files but don't care about building absolutely
everything:
cd ~/netfpga/lib/C
make
sudo make install # this also does kernel dir "make install" to install kernel module nf2.c
/sbin/modprobe nf2
see if init things work:
~/netfpga/lib/scripts/cpci_config_reg_access/dumpregs.sh -f ~/defaultregs # for problem
recovery later
~/netfpga/lib/scripts/cpci_config_reg_access/loadregs.sh -f ~/defaultregs # do if
cpci_reprogram fails half way (usually permissions or path problems)
sudo ~/netfpga/lib/scripts/cpci_reprogram/cpci_reprogram.pl
sudo /usr/local/bin/nf_download /usr/local/netfpga/bitfiles/reference_nic.bit
add the same stuff and more to /etc/rc.local as follows:
/usr/local/netfpga/lib/scripts/cpci_reprogram/cpci_reprogram.pl
/usr/local/bin/nf_download /usr/local/netfpga/bitfiles/reference_nic.bit
ifconfig nf2c[0-3] 10.0.1[0-3].1/24 for the 4 ports
NOTE: if you forget to run cpci_reprogram with root permissions, it will hang your card and you'll
need to reboot or run the loaadregs.sh command mentioned above.
so far this should get you to the point of downloading and running other people's hardware (.bit)
files.
How to perform a minimal install (without yum) of the netfpga card on a new machine:
Minimal Install
This quick start is tested for the 2.1.1 release on CentOS 5 and RHEL4 (Add your OS if it works for you).
Quick Start
13 de 56
You can create your own bootable CentOS DVD by downloading an ISO [2]
Burn the image onto a DVD
Install CentOS http://netfpga.org/CentOS_Install.pdf (note this pdf was created for CentOS 4)
We use GNOME as the default window manager
Do not install Virtualization
Set SELinux to: Not enforcing
Otherwise you will need to manually adjust security settings
Attach your Eth0 (motherboard's primary interface) to the external network
It is easiest to use DHCP to set the IP address, gateway, DNS
Apply CentOS Updates
After installation, Package updater should prompt for an update
Select to install all updates
8/8/2010 10:20 PM
The current version of the NetFPGA package is only supported for the 32-bit version of CentOS 5 (i.e.
CentOS 5.1 -- CentOS 5.4). Packages before the 2.0.0 release were tested with the 32-bit version of
CentOS 4.4, CentOS 4.5, CentOS 5.1, and CentOS 5.2 operating systems.
CentOS Installation Instructions
We support use of the popular Linux distribution CentOS as the operating system for the Host PC.
CentOS is a free variation of the popular RedHat distribution.
Installing an Operating System on the Host PC
Install Xilinx ISE Foundation 10.1. (Academic users should be able to obtain this for free;
commercial users will need to purchase this from Xilinx or a distributor.) These days 10.1 is old, so
your distributor will make you buy 11 or 12 for the same price and then give you 10.1. I have
downloaded them from the web, but I'm sure a CD copy would work as well. The free webpack
version of Xilinx ISE will not work with the NetFPGA's virtex2p50, and it will NOT give you
easy-to-find error messages.
This will also install isim to let you simulate using: nf_run_test.pl --isim
Update the Xilinx tools to service pack SP3. If you use Xilinx's web update tool beware that it has
to download a gigabyte so takes ages. I've had more success downloading the update files myself
from the Xilinx downloads centre (also big).
install modelsim if your budget is nearly $10k
to synthesize or simulate the first time, coregen will run, and it needs X windows. Even though it
doesn't actually use X windows, it will fail and may not give you a sensible error message.
Quick Xilinx Tool Install
check that everything builds:
cd ~/netfpga run:
make
make install
# If you get perl error "Can't locate XML/Simple.pm" then you have a version of perl before
5.8.8, so get a later version of perl or copy these files from a new version of perl: /usr/lib
/perl5/vendor_perl/5.8.*/XML/Simple, /usr/lib/perl5/5.8.*/Digest/file.pm
# RHEL4 needs libnet which is best got from http://dag.wieers.com/rpm/packages/libnet/
To simulate, you'll need to install the memory models
14 de 56
Reboot your machine
PERL5LIB
Use of other operating systems is possible, but we do not support them.
We provide the Verilog source code the modules so that users can compile, simulate, and synthesize
gateware for the NetFPGA. We have tested simulation and synthesis using a specific version of the Xilinx
tools (as described below). Use of other versions of the tools (older or newer) is not supported. If you do
not plan to rebuild the hardware circuits, you can skip installation of CAD tools.
Install CAD Tools
NetFPGA Package 1.0 Install_Software_1.0
NetFPGA Package 1.2 Install_Software_1.2
Install Xilinx ISE
Log in as root or 'su -' to root
Install RPMforge Yum repository
Install the RPMforge repository.
CentOS 5 wiki documentation on installing RPMforge
Install NetFPGA Base Package
Install NetFPGA yum repository and GPG Key
rpm -Uhv http://netfpga.org/yum/el5/RPMS/noarch/netfpga-repo-1-1_CentOS5.noarch.rpm
Next run the following command to install the NetFPGA base package
yum install netfpga-base
Note that there may be some dependencies. Select 'y' to install these dependent packages.
From release 2.0.0 and beyond the Java GUI has been split into its own package. To install this
package follow the Java GUI Install
Create netfpga directory in your user account
Run the following script to copy the entire netfpga directory into your account (typically: /root/netfpga).
WARNING: Running this command WILL overwrite any existing netfpga directory or files in your user
account! If you have files that you want to preserve, 'mv' your netfpga directory to another location, such
as netfpga_backup.
To copy the NetFPGA directory and set the environment variables run the following command
/usr/local/netfpga/lib/scripts/user_account_setup/user_account_setup.pl
It also adds the following environment variables to your .bashrc file.
NF_ROOT
NF_DESIGN_DIR
NF_WORK_DIR
PYTHONPATH
8/8/2010 10:20 PM
Install Memory Modules for Simulation
Micron DDR2 SDRAM
Download the model from Micron
http://download.micron.com/downloads/models/verilog/sdram/ddr2/256Mb_ddr2.zip
Extract and copy ddr2_parameters.vh, and ddr2.v to netfpga/lib/verilog/core/common/src
8/8/2010 10:20 PM
To debug signals on the FPGA using an on-chip logic analyzer, install:
Xilinx: ChipScope Pro
Version 9.1.02i or 10.1 SP3 (ChipScope 10.1 license comes with Xilinx 10.1 these days)
Allows visibility of verilog signals in synthesized verilog on the NetFPGA
Requires use of a Xilinx USB Platform Cable or this compatible cable
Debug with ChipScope
To simulate Verilog, install:
Mentor Graphics: ModelSim
Version SE 6.2G (Also tested with ModelSim DE v6.6). But ModelSim PE and the
student version are windows-only so not so good for the NetFPGA Unix environment.
Allows simulation of circuits and viewing of simulated waveforms.
Testbench software assumes use of this version of ModelSim.
Install ModelSim
Xilinx: ISE Foundation, Version: 10.1 SP3
Install Service Pack 3
Install IP Update 3
Use of other versions of the tools (older or newer) is not supported.
Obtain a license for the V2Pro TEMAC core from Xilinx.
Part Number: DO-DI-TEMAC, Ver 3.3
For a free evaluation copy
Request "Full System hardware Evaluation"
Allows use of the TEMAC for 30 days, 8 hour run-time
Academic users can request a donation of the core and CAD tools
Mention use of the NetFPGA when you submit the request
Commercial users can purchase the core through their local sales representative.
Log in as root
For archival purposes the install instructions for older packages are linked below. Use the instructions on
this page to install newest version NetFPGA package.
Reboot your machine in order to finalize the installation.
16 de 56
Software Installation
Other tested but unsupported operating systems
15 de 56
8/8/2010 10:20 PM
Sample correct output:
make install
Install the driver and reboot. The driver will be stored in /lib/modules/ùname -r`/kernel/drivers
/nf2.ko
Load driver and tools
If you get an error message such as "make: * /lib/modules/2.6.9-42.ELsmp/build: No such file or
directory. Stop.", then kernel sources are need to build the driver.
make[1]: Entering directory `/home/gac1/temp/NF2/lib/scripts'
make -C cpci_reprogram
make[2]: Entering directory `/home/gac1/temp/NF2/lib/scripts/cpci_reprogram'
make[2]: Nothing to be done for àll'.
make[2]: Leaving directory `/home/gac1/temp/NF2/lib/scripts/cpci_reprogram'
make -C cpci_config_reg_access
make[2]: Entering directory `/home/gac1/temp/NF2/lib/scripts/cpci_config_reg_access'
make[2]: Leaving directory `/home/gac1/temp/NF2/lib/scripts/cpci_config_reg_access'
make[1]: Leaving directory `/home/gac1/temp/NF2/lib/scripts'
18 de 56
8/8/2010 10:20 PM
for dir in lib bitfiles projects/scone/base projects/selftest/sw ; do \
make -C $dir install; \
Compile driver and tools
done
make[1]: Entering directory `/home/gac1/temp/NF2/lib'
for dir in C scripts java/gui ; do \
Compile
done
cd ~/netfpga/ make
make[2]: Entering directory `/home/gac1/temp/NF2/lib/C'
for dir in kernel download reg_access router ; do \
done
make -C C
make[3]: Entering directory `/home/gac1/temp/NF2/lib/C/kernel'
make[1]: Entering directory `/home/gac1/temp/NF2/lib/C'
make -C /lib/modules/2.6.9-55.0.9.ELsmp/build M=/home/gac1/temp/NF2/lib/C/kernel LDDINC=/home
make -C kernel
make[4]: Entering directory `/usr/src/kernels/2.6.9-55.0.9.EL-smp-i686'
make[2]: Entering directory `/home/gac1/temp/NF2/lib/C/kernel'
Building modules, stage 2.
make -C /lib/modules/2.6.9-55.0.9.ELsmp/build M=/home/gac1/temp/NF2/lib/C/kernel
LDDINC=/home/gac1/te MODPOST
make[3]: Entering directory `/usr/src/kernels/2.6.9-55.0.9.EL-smp-i686'
make[4]: Leaving directory `/usr/src/kernels/2.6.9-55.0.9.EL-smp-i686'
Building modules, stage 2.
install -m 644 nf2.ko /lib/modules/ùname -r`/kernel/drivers/nf2.ko /sbin/depmod -a
MODPOST
make[3]: Leaving directory `/home/gac1/temp/NF2/lib/C/kernel'
make[3]: Leaving directory `/usr/src/kernels/2.6.9-55.0.9.EL-smp-i686'
make[3]: Entering directory `/home/gac1/temp/NF2/lib/C/download'
make[2]: Leaving directory `/home/gac1/temp/NF2/lib/C/kernel'
install nf2_download /usr/local/bin
make -C download
make[3]: Leaving directory `/home/gac1/temp/NF2/lib/C/download'
make[2]: Entering directory `/home/gac1/temp/NF2/lib/C/download'
make[3]: Entering directory `/home/gac1/temp/NF2/lib/C/reg_access'
make -C ../common
install regread /usr/local/bin
make[3]: Entering directory `/home/gac1/temp/NF2/lib/C/common'
install regwrite /usr/local/bin
make[3]: Leaving directory `/home/gac1/temp/NF2/lib/C/reg_access'
make[3]: Leaving directory `/home/gac1/temp/NF2/lib/C/common'
make[3]: Entering directory `/home/gac1/temp/NF2/lib/C/router'
make[2]: Leaving directory `/home/gac1/temp/NF2/lib/C/download'
make[3]: Nothing to be done for ìnstall'.
make -C reg_access
make[3]: Leaving directory `/home/gac1/temp/NF2/lib/C/router'
make[2]: Entering directory `/home/gac1/temp/NF2/lib/C/reg_access'
make[2]: Leaving directory `/home/gac1/temp/NF2/lib/C'
make -C ../common
make[2]: Entering directory `/home/gac1/temp/NF2/lib/scripts'
make[3]: Entering directory `/home/gac1/temp/NF2/lib/C/common'
for dir in cpci_reprogram cpci_config_reg_access ; do \
make[3]: Leaving directory `/home/gac1/temp/NF2/lib/C/common'
done
make[2]: Leaving directory `/home/gac1/temp/NF2/lib/C/reg_access'
make[3]: Entering directory `/home/gac1/temp/NF2/lib/scripts/cpci_reprogram'
make -C router
install cpci_reprogram.pl /usr/local/sbin
make[2]: Entering directory `/home/gac1/temp/NF2/lib/C/router'
make[3]: Leaving directory `/home/gac1/temp/NF2/lib/scripts/cpci_reprogram'
-o cli
gcc -lncurses cli.o ../common/nf2util.o ../common/util.o ../common/reg_defines.h
make[3]: Entering directory `/home/gac1/temp/NF2/lib/scripts/cpci_config_reg_access'
gcc -lncurses regdump.o ../common/nf2util.o ../common/reg_defines.h
-o regdump
install dumpregs.sh /usr/local/sbin
-o show_statinstall loadregs.sh /usr/local/sbin
gcc -lncurses show_stats.o ../common/nf2util.o ../common/util.o ../common/reg_defines.h
make[2]: Leaving directory `/home/gac1/temp/NF2/lib/C/router'
make[3]: Leaving directory `/home/gac1/temp/NF2/lib/scripts/cpci_config_reg_access'
make[1]: Leaving directory `/home/gac1/temp/NF2/lib/C'
make[2]: Leaving directory `/home/gac1/temp/NF2/lib/scripts'
make -C scripts
make[2]: Entering directory `/home/gac1/temp/NF2/lib/java/gui'
Compile and Load Driver
Run Regression Tests: Each project has a set of regression tests that verify the functionality of the
distributed code.
These should be run before starting to use any of the projects from the NFP.
Run Selftest: Verify the functionality of your NetFPGA system
Compile and Load Driver: Run the makefile to build the executables
Verify the software and hardware
Download the model from Cypress
http://download.cypress.com.edgesuite.net/design_resources/models/contents
/cy7c1370d_verilog_10.zip
Extract and copy cy7c1370d.v to netfpga/lib/verilog/core/common/src
Rename cy7c1370d.v to cy7c1370.v
Cypress SRAM
17 de 56
reboot
Leaving directory `/home/gac1/temp/NF2/bitfiles'
Entering directory `/home/gac1/temp/NF2/projects/scone/base'
Nothing to be done for ìnstall'.
Leaving directory `/home/gac1/temp/NF2/projects/scone/base'
Entering directory `/home/gac1/temp/NF2/projects/selftest/sw'
Nothing to be done for ìnstall'.
Leaving directory `/home/gac1/temp/NF2/projects/selftest/sw'
After reboot log in as root.
28428
Verify that the driver loaded:
| grep nf2
lsmod
nf2
Verify NetFPGA interfaces
encap:Ethernet
encap:Ethernet
encap:Ethernet
encap:Ethernet
0
HWaddr
HWaddr
HWaddr
HWaddr
00:4E:46:32:43:00
00:4E:46:32:43:01
00:4E:46:32:43:02
00:4E:46:32:43:03
Verify that four nf2cX interfaces have successfully loaded:
Link
Link
Link
Link
ifconfig -a | grep nf2
nf2c0
nf2c1
nf2c2
nf2c3
Reprogram the CPCI
Run the cpci reprogramming script
/usr/local/sbin/cpci_reprogram.pl --all
(to reprogram all NetFPGAs in a system)
Expected output:
Loading the CPCI Reprogrammer on NetFPGA 0
Loading the CPCI on NetFPGA 0
CPCI on NetFPGA 0 has been successfully reprogrammed
Every time you restart the computer, you need to reload the CPCI!
8/8/2010 10:20 PM
Reboot the machine. The driver currently crashes upon rmmod, so a reboot is required to load the
newly compiled driver. You may want to check if other users are on the machine with the 'who'
command first. If you don't like the other current machine users or you're the only one on the
machine, run the following:
done
make[1]:
make[1]:
make[1]:
make[1]:
make[1]:
make[1]:
make[1]:
make[2]: Nothing to be done for ìnstall'.
make[2]: Leaving directory `/home/gac1/temp/NF2/lib/java/gui'
make[1]: Leaving directory `/home/gac1/temp/NF2/lib'
make[1]: Entering directory `/home/gac1/temp/NF2/bitfiles'
for bitfile in CPCI_2.1.bit cpci_reprogrammer.bit ; do \
install -D -m 0644 $bitfile /usr/local/netfpga/bitfiles/$bitfile ; \
19 de 56
Connect loopback cables
Install a SATA cable to loopback the board-to-board high-speed serial I/O.
Note: To minimize the chance that you damage your computer or the NetFPGA module, wear an
anti-static wrist strap when handing the hardware.
8/8/2010 10:20 PM
The following instructions assume that you have successfully installed a NetFPGA card with CentOS. The
selftest is an enhanced version of the test run on every NetFPGA at the factory by Digilent to verify
proper hardware operation.
We provide the self-test bitfile and the software to end-users so that the self-test can be run when the
hardware is delivered. When you receive a NetFPGA card, we suggest that you run the self-test to ensure
that the card is still fully functional and that the card works properly in your environment. Before running
the self-test, be sure that you have connected the loopback cables as shown in the directions on how to set
up a NetFPGA in your system. If the loopback cables are not connected, the self-test will correctly report
that an interface appears non-functional.
The self-test bitfile performs rigorous testing of the SRAM and DDR2 DRAM to ensure that all memory
lines can be properly written to and read back with the same data. Multiple data patterns are used to
ensure that no address or data lines have faults. The network test sends bursts of packets on the Ethernet
interfaces and the loopback cables are put in place so that packets can be read and compared to the data
that was transmitted. The SATA loopback test transmits data using the Multi-Gigabit I/O lines (MGIOs) to
ensure that data can be reliably transmitted on the high-speed I/O interfaces. The DMA test exercises the
PCI Controller (CPCI), the VirtexII, and the PCI bus to ensure that large blocks of data can be sent
between the NetFPGA the host computer's memory. The selftest bitfile runs all of the tests above in
parallel and continously runs until it is terminated. The self-test software displays the results of testing on
a console.
The NetFPGA self-test is an FPGA bitfile and software that ensures that all of the components on your
platform are fully functional. The self-test consists of both an FPGA bitfile that contains logic and
interfaces to external components as well the software that displays the results. The self-test excercises all
of the hardware in parallel. The test continues to run repeatedly until terminated by the user. The self-test
was run at the factory just after the cards were manufactured. Cards are not distributed unless they
completely pass all functions of the self-test process.
Run Selftest
If the NetFPGA refuses to send packets, and the regression or selftest is failing, make sure you've
reprogrammed the cpci.
/usr/local/netfpga/lib/scripts/cpci_reprogram/cpci_reprogram.pl --all
To have the CPCI reprogrammed when the computer boots add the following line to /etc/rc.local
20 de 56
Install two Ethernet cables as shown:
21 de 56
8/8/2010 10:20 PM
8/8/2010 10:20 PM
The features exercised by regression test suite are the only features we will try to provide support for.
The regression test suite is a set of tests that exercise the functionality of the released gateware and
software. On a fast machine, this test should take approximately 10 minutes.
Run Regression Tests
Found net device: nf2c0
NetFPGA selftest 1.00 alpha
Running..... PASSED
Expected Output:
~/netfpga/projects/selftest/sw/selftest -n
Otherwise, type the following command.
~/netfpga/projects/selftest/sw/selftest
If you have connected a SATA cable to the NetFPGA, type the following command.
Run Selftest
nf_download ~/netfpga/bitfiles/selftest.bit
Type:
Load self-test bitfile
for i in `seq 0 3`; do ifconfig nf2c$i up; done
Type:
Bring nf2cX interfaces up
22 de 56
Connect Ethernet test cables
Connect 'eth1' to 'nf2c0' (c0 is the port closest to the mainboard)
Connect 'eth2' to 'nf2c1' (c1 is the port one away from the mainboard)
The location of your eth1 and eth2 ports may vary depending on your NIC
The photo below shows the configuration of a nf-test machine
Log in as root through X session
Log in as root or 'su -' to root using an X session, because we will be testing the GUI Scone
Load reference_router bitfile
Download the reference bitfile to the NetFPGA board:
8/8/2010 10:20 PM
Please make sure that you have successfully completed the selftest before proceeding to the regression
tests.
The NIC supports a set of features
The details of how each feature is tested is described in the NIC Regression test document
available both on the Wiki and Web.
The Reference Router (RR) supports a set of features
The details of how each feature is tested is described in the RR Regression test, a large
document available both on the Wiki and Web.
For more information on the features we support, as defined by tests, see the following:
Additional features might be available and functional in the released gateware, but they are not
supported.
23 de 56
Running tests on project 'scone'...
Running global setup... PASS
Running test 'test_build'... PASS
Running test 'test_mac_set'... PASS
Running tests on project 'reference_router'...
Running test 'test_router_cpusend/run.pl'... PASS
Running test 'test_wrong_dest_mac'... PASS
Running test 'test_nonip_packet'... PASS
Running test 'test_nonipv4_packet'... PASS
Running test 'test_invalidttl_packet'... PASS
Running test 'test_lpm_misses'... PASS
Running test 'test_arp_misses'... PASS
Running test 'test_badipchecksum_packet'... PASS
Running test 'test_ipdest_filter_hit'... PASS
Running test 'test_packet_forwarding'... PASS
Running test 'test_lpm'... PASS
Running test 'test_lpm_next_hop'... PASS
Running test 'test_queue_overflow'... PASS
Running test 'test_oq_limit'... PASS
Running test 'test_ipdest_filter'... PASS
Running test 'test_oq_sram_sz_cpu'... PASS
Running test 'test_oq_sram_sz_mac'... PASS
Running test 'test_router_table/run.pl'... PASS
Running test 'test_send_rec/run.pl'... PASS
Running test 'test_lut_forward'... PASS
Running global teardown... PASS
Running tests on project 'reference_nic'...
Running test 'download_nic'... PASS
Running test 'test_loopback_random'... PASS
Running test 'test_loopback_minsize'... PASS
Running test 'test_loopback_maxsize'... PASS
Running test 'test_loopback_drop'... PASS
Running test 'test_ip_interface'... PASS
Running tests on project 'driver'...
Running test 'driver_compile'... PASS
Running test 'driver_install'... PASS
Running test 'verify_mtu'... PASS
~/netfpga/bin/nf_regress_test.pl
8/8/2010 10:20 PM
Run the regression test suite. The tests should take about 10 minutes total.
Note: Prior to release 2.0.0 the command was nf21_regress_test.pl
Run regression test suite
Bit file built from: nf2_top_par.ncd
Part: 2vp50ff1152
Date: 2007/10/ 9
Time: 22: 3: 4
Error Registers: 1000000
Good, after resetting programming interface the FIFO is empty
Download completed - 2377668 bytes. (expected 2377668).
DONE went high - chip has been successfully programmed.
nf_download ~/netfpga/bitfiles/reference_router.bit
24 de 56
test 'test_ip_set'... PASS
test 'test_rtable_set'... PASS
test 'test_disabled_interfaces/run.pl'... PASS
test 'test_noniparp_ethtype'... PASS
test 'test_arp_rpl/run.pl'... PASS
test 'test_arp_norpl/run.pl'... PASS
test 'test_arp_quepkt/run.pl'... PASS
test 'test_ip_error/run.pl'... PASS
test 'test_ip_rtblmiss/run.pl'... PASS
test 'test_ip_intfc/run.pl'... PASS
test 'test_ip_checksum/run.pl'... PASS
test 'test_ttl_expired/run.pl'... PASS
test 'test_send_receive/run.pl'... PASS
test 'test_arp_req/run.pl'... PASS
test 'test_tcp_port/run.pl'... PASS
test 'test_udp_packet/run.pl'... PASS
test 'test_icmp_echo/run.pl'... PASS
test 'test_icmp_notecho/run.pl'... PASS
global teardown... PASS
8/8/2010 10:20 PM
Go to the synthesis directory for the reference_nic and run make. This step should take under
Set up the Xilinx ISE tools (see Xilinx's website for instructions). Make sure the Xilnx tools
are in your path and that the XILINX environment variable is set.
Login, either direct in an X session or via ssh -X. This step causes ~/nf2_profile, plus
environment variables, to be sourced. You may say "But I'm not running anything graphical!"
and you'd be right. Unfortunately, even when called with no gui, the Xilinx tools require X to
be running. A bugreport on this issue has been filed to Xilinx.
If you are a hardware developer and would like to synthesize your own NetFPGA Router
hardware using the Verilog source code, follow the steps below. To synthesize FPGA
hardware, you will need to have all of the FPGA Development tools installed.
Note: This step will take about 45-60 mins. This can be used to verify the setup of the machine
for synthesis. You will need to have the NetFPGA Beta Plus package. The Beta (not Plus)
package does not include the sources for this step.
Synthesize reference_router bitfile, from source
If you installed the CAD tools, you should run this test to verify that you can build a new circuit.
Skip this step if you do not plan to modify hardware.
Run regression scripts on new bitfile
If there are no errors (all tests say PASS), you can play with your router, or go on to creating
a bitfile from source.
Running tests on project 'router_buffer_sizing'...
Running test 'test_time_stamp/run'... PASS
Running test 'test_store_event/run'... PASS
Running tests on project 'router_kit'...
Running test 'test_00_make/run.sh'... PASS
Running test 'test_01_ip_dst_filter/run.pl'... PASS
Running test 'test_02_route_table/run.pl'... PASS
Running test 'test_03_arp_table/run.pl'... PASS
Running test 'test_04_ip_packets/run.pl'... PASS
Running
Running
Running
Running
Running
Running
Running
Running
Running
Running
Running
Running
Running
Running
Running
Running
Running
Running
Running
25 de 56
Running tests on project 'reference_router'...
Running test 'test_router_cpusend/run.pl'... PASS
Running test 'test_wrong_dest_mac'... PASS
Running test 'test_nonip_packet'... PASS
Running test 'test_nonipv4_packet'... PASS
Running test 'test_invalidttl_packet'... PASS
Running test 'test_lpm_misses'... PASS
Running test 'test_arp_misses'... PASS
Running test 'test_badipchecksum_packet'... PASS
Running test 'test_ipdest_filter_hit'... PASS
Running test 'test_packet_forwarding'... PASS
Running test 'test_lpm'... PASS
Running test 'test_lpm_next_hop'... PASS
Running test 'test_queue_overflow'... PASS
Running test 'test_oq_limit'... PASS
Running test 'test_ipdest_filter'... PASS
Running test 'test_oq_sram_sz_cpu'... PASS
Running test 'test_oq_sram_sz_mac'... PASS
Running test 'test_router_table/run.pl'... PASS
Running test 'test_send_rec/run.pl'... PASS
Running tests on project 'reference_nic'...
Running test 'download_nic'... PASS
Running test 'test_loopback_random'... PASS
Running test 'test_loopback_minsize'... PASS
Running test 'test_loopback_maxsize'... PASS
Running test 'test_loopback_drop'... PASS
Running test 'test_ip_interface'... PASS
Running tests on project 'driver'...
Running test 'driver_compile'... PASS
Running test 'driver_install'... PASS
Running test 'verify_mtu'... PASS
~/netfpga/bin/nf2_regress_test.pl
Re-run the regression test suite.
Note: Prior to release 2.0.0 the command was nf21_regress_test.pl
Run regression-test suite on new bitfile
nf_download nf2_top_par.bit
Download the fresh bitfile to the NetFPGA board:
Load new bitfile
nf2_top_par.bit
ls | grep nf2_top_par.bit
8/8/2010 10:20 PM
Verify the reference_router bitfile (nf2_top_par.bit) has been created.
cd ~/netfpga/projects/reference_router/synth
time make
an hour on a well-endowed machine.
26 de 56
Reference NIC
Walkthrough: Mainly an
introduction to the
software/hardware
interface. How to make
software talk to the
hardware.
SCONE Walkthrough: More
complex example of user
process communicating to
hardware. How to talk to
the router and set entries.
8/8/2010 10:20 PM
This section describes some walkthroughs to help in using the distributed source code. Each
walkthrough will deal with a different aspect of using the reference designs. Some sections will be
marked as optional reading. These sections explain what is going on behind the scenes to put things
together and are not within the main focus of the walkthrough. The walkthroughs are the following:
Walkthrough the Reference Designs
Running tests on project 'router_buffer_sizing'...
Running test 'test_time_stamp/run'... PASS
Running test 'test_store_event/run'... PASS
Running tests on project 'router_kit'...
Running test 'test_00_make/run.sh'... PASS
Running test 'test_01_ip_dst_filter/run.pl'... PASS
Running test 'test_02_route_table/run.pl'... PASS
Running test 'test_03_arp_table/run.pl'... PASS
Running test 'test_04_ip_packets/run.pl'... PASS
Running tests on project 'scone'...
Running test 'test_build'... PASS
Running test 'test_mac_set'... PASS
Running test 'test_ip_set'... PASS
Running test 'test_rtable_set'... PASS
Running test 'test_disabled_interfaces/run.pl'... PASS
Running test 'test_noniparp_ethtype'... PASS
Running test 'test_arp_rpl/run.pl'... PASS
Running test 'test_arp_norpl/run.pl'... PASS
Running test 'test_arp_quepkt/run.pl'... PASS
Running test 'test_ip_error/run.pl'... PASS
Running test 'test_ip_rtblmiss/run.pl'... PASS
Running test 'test_ip_intfc/run.pl'... PASS
Running test 'test_ip_checksum/run.pl'... PASS
Running test 'test_ttl_expired/run.pl'... PASS
Running test 'test_send_receive/run.pl'... PASS
Running test 'test_arp_req/run.pl'... PASS
Running test 'test_tcp_port/run.pl'... PASS
Running test 'test_udp_packet/run.pl'... PASS
Running test 'test_icmp_echo/run.pl'... PASS
Running test 'test_icmp_notecho/run.pl'... PASS
Running test 'test_lut_forward'... PASS
27 de 56
./counterdump
You should see an output similar to this:
8/8/2010 10:20 PM
One of the built tools is called counterdump. This simple tool reads several hardware counters and
dumps the counts to the terminal. To use it, type:
cd ~/netfpga/lib/C/nic
make
To compile the utilities type:
Bit file built from: nf2_top_par.ncd
Part: 2vp50ff1152
Date: 2007/11/21
Time: 11: 0: 3
Error Registers: 1000000
Good, after resetting programming interface the FIFO is empty
Download completed - 2377668 bytes. (expected 2377668).
DONE went high - chip has been successfully programmed.
You should see output similar to the following:
~/netfpga/lib/C/download/nf_download ~/netfpga/bitfiles/reference_nic.bit
Next, we need to download the NIC bitfile onto the NetFPGA. As root, run the following:
/sbin/ifconfig nf2cX x.x.x.x
Assuming the regression tests have been completed successfully and the NFP is installed correctly,
we can now proceed to use one of the distributed projects: the NIC. In the rest of the exercises, we
assume that NFP is installed in the user's home directory. Replace the '~' with the full path to the
installation location if not. To run the tools, IP addresses must be assigned to the nf2cX interfaces.
To do that, run the following command as root after replacing all the x's:
Using counterdump
The reference NIC walkthrough will go through an example of using some of the tools that are
distributed with the release, and an example of how to write a simple C program to interface to the
hardware.
Reference NIC Walkthrough
Buffer Monitoring System:
Describes how a circuits
was added to measure the
lengths of buffers.
Reference Router
Walkthrough: Introduces
more details about the
hardware, and how to
modify it.
Router Kit Walkthrough:
How to install and use the
router kit.
28 de 56
pkts received on port 1:
pkts dropped (rx queue 1 full):
pkts dropped (bad fcs q 1):
bytes received on port 1:
pkts sent from port 1:
bytes sent from port 1:
Num pkts received on port 0:
Num pkts dropped (rx queue 0 full):
Num pkts dropped (bad fcs q 0):
Num bytes received on port 0:
Num pkts sent from port 0:
Num bytes sent from port 0:
0
0
0
0
0
0
0
0
0
0
0
0
0
0
0
0
0
0
Num
Num
Num
Num
Num
Num
Num
Num
Num
Num
Num
Num
Num
Num
Num
Num
Num
Num
0
0
0
0
0
0
0
0
0
0
0
0
0
0
0
0
0
0
Num
Num
Num
Num
Num
Num
Num
Num
Num
Num
Num
Num
0
0
0
0
0
0
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
#include <stdio.h>
#include <stdlib.h>
#include <unistd.h>
#include <net/if.h>
80
"nf2c0"
#include "../common/reg_defines.h"
#include "../common/nf2.h"
#include "../common/nf2util.h"
#define PATHLEN
#define DEFAULT_IFACE
/* Global vars */
static struct nf2device nf2;
/* Function declarations */
void dumpCounts();
8/8/2010 10:20 PM
/***************************************************************************
* vim:set shiftwidth=2 softtabstop=2 expandtab:
* $Id: Guide.txt,v 1.27 2010/06/26 02:12:38 JonathanEllithorpe Exp $
*
* Module: counterdump.c
* Project: NetFPGA NIC
* Description: dumps the MAC Rx/Tx counters to stdout
* Author: Jad Naous
*
* Change history:
*
*/
The counters that have been dumped using counterdump are actually memory-mapped I/O registers.
These are counters that exist in the NetFPGA FPGA hardware and are exported via the PCI
interface. The software uses ioctl calls to do reads and writes into these registers. The ioctl calls are
wrapped in two simple functions readReg and writeReg. In this section, we will open up
counterdump.c and understand the software/hardware interface. counterdump.c is shown below
with line numbers for reference.
Understanding the Hardware/Software Interface
If your NetFPGA ports are not connected to a quiet network, then you'll probably see different
results.
Using send_pkts
Now let's try to send and receive some packets and check the counter outputs again. One of the
tools that are included in the NFP is a tool called send_pkts. This tool can send arbitrary Ethernet
packets from any given port, but you will need root access to use it. To use this tool:
cd ~/netfpga/lib/C/tools/send_pkts
make
If everything goes correctly, you should see an output similar to this:
gcc `libnet-config --defines --cflags` -O2 -o send_pkts send_pkts.c `libnet-config
sudo ./send_pkts -i nf2c0 -s 10 -l 100
For the next part, we will test sending a few packets from one of the ports. To send packets, issue
the following commands:
cd ~/netfpga/lib/C/tools/send_pkts
0
0
0
0
10
1000
The last command sends 10 100-byte packets out of port 0 on the NetFPGA (port 0 is the port
closest to the PCI connector on the NetFPGA). If you have a machine connected to the same LAN
section as that port, you should be able to capture the packets using Wireshark.
Now check the counters again:
~/netfpga/lib/C/nic/counterdump
You should see an output similar to this:
Num pkts received on port 0:
Num pkts dropped (rx queue 0 full):
Num pkts dropped (bad fcs q 0):
Num bytes received on port 0:
Num pkts sent from port 0:
Num bytes sent from port 0:
8/8/2010 10:20 PM
30 de 56
Num
Num
Num
Num
Num
Num
29 de 56
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
91
92
93
94
95
96
97
98
99
100
101
processArgs(argc, argv);
exit(1);
// Open the interface if possible
if (check_iface(&nf2))
{
exit(1);
}
if (openDescriptor(&nf2))
{
}
dumpCounts();
closeDescriptor(&nf2);
return 0;
8/8/2010 10:20 PM
readReg(&nf2, RX_QUEUE_2_NUM_PKTS_STORED_REG, &val);
printf("Num pkts received on port 2:
%u\n", val);
readReg(&nf2, RX_QUEUE_2_NUM_PKTS_DROPPED_FULL_REG, &val);
printf("Num pkts dropped (rx queue 2 full):
%u\n", val);
readReg(&nf2, RX_QUEUE_2_NUM_PKTS_DROPPED_BAD_REG, &val);
printf("Num pkts dropped (bad fcs q 2):
%u\n", val);
readReg(&nf2, RX_QUEUE_2_NUM_BYTES_PUSHED_REG, &val);
printf("Num bytes received on port 2:
%u\n", val);
readReg(&nf2, TX_QUEUE_2_NUM_PKTS_SENT_REG, &val);
printf("Num pkts sent from port 2:
%u\n", val);
readReg(&nf2, TX_QUEUE_2_NUM_BYTES_PUSHED_REG, &val);
printf("Num bytes sent from port 2:
%u\n\n", val);
%u\n", val);
%u\n", val);
%u\n", val);
%u\n", val);
%u\n", val);
%u\n\n", val);
%u\n", val);
%u\n", val);
%u\n", val);
%u\n", val);
%u\n", val);
%u\n\n", val);
void dumpCounts()
{
unsigned val;
}
int main(int argc, char *argv[])
{
nf2.device_name = DEFAULT_IFACE;
void processArgs (int , char **);
void usage (void);
31 de 56
}
%u\n", val);
readReg(&nf2, R<script type="text/javascript" src="http://netfpga.org/fosw
%u\n", val);
%u\n", val);
%u\n", val);
%u\n", val);
%u\n\n", val);
}
}
while ((c = getopt (argc, argv, "i:h")) != -1)
{
switch (c)
{
case 'i':
/* interface name */
nf2.device_name = optarg;
break;
case '?':
if (isprint (optopt))
fprintf (stderr, "Unknown option `-%c'.\n", optopt);
else
fprintf (stderr,
"Unknown option character `\\x%x'.\n",
optopt);
case 'h':
default:
usage();
exit(1);
/* don't want getopt to moan - I can do that just fine thanks! */
opterr = 0;
/*
* Process the arguments.
*/
void processArgs (int argc, char **argv )
{
char c;
}
printf("Usage: ./counterdump <options> \n\n");
printf("Options: -i <iface> : interface name (default nf2c0)\n");
printf("
-h : Print this message and exit.\n");
/*
* Describe usage of this program.
*/
void usage (void)
{
}
#include "../common/reg_defines.h"
#include "../common/nf2.h"
#include "../common/nf2util.h"
8/8/2010 10:20 PM
Line 20 includes the header file that contains all the register addresses on the NetFPGA. This is
needed to refer to register addresses as constant names rather than numeric addresses. Lines 21 and
22 include macros to access registers. These functions are used later in the code.
20
21
22
Let's go through the code.
102
103
104
105
106
107
108
109
110
111
112
113
114
115
116
117
118
119
120
121
122
123
124
125
126
127
128
129
130
131
132
133
134
135
136
137
138
139
140
141
142
143
144
145
146
147
148
149
150
151
152
153
154
155
156
157
32 de 56
static struct nf2device nf2;
nf2.device_name = DEFAULT_IFACE;
processArgs(argc, argv);
if (check_iface(&nf2))
if (openDescriptor(&nf2))
dumpCounts();
closeDescriptor(&nf2);
%u\n", val);
8/8/2010 10:20 PM
Next, we will modify this file to dump the device ID which is assigned at implementation time. To
The registers available for access are documented in the Register Map. Unfortunately, registers
keep getting added and removed as the design evolves, so the most current list of registers is in the
reg_defines.h file that defines the register addresses. This file is generated automatically from the
Verilog source code when it is simulated so it always has the most recent list of registers and their
addresses. The registers in the Register Map are divided into groups corresponding to the modules
described in the Verilog. The next section will go into more details on these modules.
Line 63 reads the number of packets received into Rx Queue 0 into val, and line 64 prints out the
result.
63
64
As an example we will look at two lines in the dumpCounts() function. The rest of the lines are
similar:
int writeReg(nf2device *nf2, unsigned int addr, unsigned int val): Writes val into the register at
address addr from device nf2 . Returns 1 on fail, 0 on success.
int readReg(nf2device *nf2, unsigned int addr, unsigned int *val): Reads the register at address addr
from device nf2 and writes the value in *val. Returns 1 on fail, 0 on success.
Reading and writing registers uses two functions:
Closes the interface after we are done using it to be polite.
54
calls the function to dump all the counts.
52
Tries to open the interface for reading/writing using ioctl calls. The interface has to be up and
assigned an IP address for a non-root user to be able to access it using ioctl calls.
47
Checks that the interface exists and can be reached.
43
Parses the command line options. In this simple program, the only command line option is to change
the name of the interface we are trying to access.
40
Set a default name for the device we are trying to access. This is useful so that the user of this
program doesn't have to specify the name if she is using the default device (which is true in most
cases).
38
Now let's go into our main function.
The nf2 struct will hold information about the device we are trying to access.
29
33 de 56
readReg(&nf2, XXXX, &val);
printf("Device ID:
%u\n\n", val);
Then after saving the file, type make in the directory containing counterdump.c and run the
program again. You should now see an output similar to the following:
Num
Num
Num
Num
Num
Num
Num
Num
Num
Num
Num
Num
0
0
0
0
0
0
0
0
0
0
0
0
0
0
0
0
0
0
0
0
0
0
10
1000
1
Num
Num
Num
Num
Num
Num
Device ID:
Num
Num
Num
Num
Num
Num
Reference Pipeline
8/8/2010 10:20 PM
do that, open file netfpga/lib/C/common/reg_defines.h file and copy the macro that defines the
device ID to replace the XXXX in the following lines, and copy the lines into the start of
dumpCounts() function after the declaration of val.
34 de 56
SCONE Walkthrough
8/8/2010 10:20 PM
You can go through the SCONE Software Walkthrough if you are interested in writing
software for the NetFPGA to run in the host
You can go through the Linux Router Kit Walkthrough if you are interested in using the
NetFPGA as a high performance router
You can go through the Reference Router Walkthrough if you are interested in modifying the
gateware on the FPGA
What to do From Here
For almost each of these modules, there is a set of registers that access status information and set
control signals for the module. These registers are described in the Register Map.
The Tx queues are analogous to the Rx queues and they send packets out of the IO ports instead of
receiving. Tx queues are also interleaved so that packets sent out of User Data Path port 0 are sent
to Ethernet Tx queue 0, and packets sent out of User Data Path port 1 are sent to CPU DMA Tx
queue 0, and so on. Packets that are handed to DMA Tx queue X pop out of interface nf2cX. For
more information on these modules, you can go here.
In the User Data Path, the first module a packet passes through is the Input Arbiter. The input
arbiter decides which Rx queue to service next, and pulls the packet from that Rx queue and hands
it to the next module in the pipeline: The output port lookup module. The output port lookup
module is responsible for deciding which port a packet goes out of. After that decision is made, the
packet is then handed to the output queues module which stores the packet in the output queues
corresponding to the output port until the Tx queue is ready to accept the packet for transmission.
The first stage in the pipeline consists of
several queues which we call the Rx
queues. These queues receive packets
from IO ports such as the Ethernet ports
and the PCI over DMA and provide a
unified interface to the rest of the
system. These ports are connected into a
wrapper called the User Data Path
which contains the processing stages.
The current design (version 1.0 Beta)
has 4 Ethernet Rx queues and 4 CPU
Diagram of the reference pipeline
DMA queues. The Ethernet and DMA
queues are interleaved so that to the
User Data Path, Rx Queue 0 is Ethernet Port 0, Rx Queue 1 is DMA port 0, Rx Queue 2 is Ethernet
Port 1, and so on. Packets that arrive into CPU DMA Rx Queue X are packets that have been sent
by the software out of interface nf2cX.
The division of the hardware into
modules was hinted at in the previous
section. Understanding these modules is
essential in making the most of the
available designs. The distributed
projects in the NFP, including the NIC,
all follow the same modular structure.
This design is a pipeline where each
stage is a separate module. A diagram of
the pipeline is shown on the right.
35 de 56
1.
2.
3.
4.
5.
8/8/2010 10:20 PM
Ensure your NETFPGA is programmed with the Reference Router bitfile
Check that your nf2c0, nf2c1, nf2c2, nfc3 interfaces are up and do not have an assigned IPv4
address
To remove an IPv4 address from an interface run 'ifconfig nf2cX 0.0.0.0' replacing X
with the interface name
Setup the cpuhw file
A text file named cpuhw must exist in the sw directory that you execute the scone
binary. The format is as follows:
<interface name> <ip address> <ip mask> <mac address>
An example:
eth0 192.168.0.2 255.255.255.0 00:00:00:00:00:01
Inside SCONE the interfaces are typically named and referred to as eth0-3, which
correspond to the nf2c0-3 ports.
(Optional) Specify a text file containing static routes
A text file containing static routes to be added on startup can be specified from the
command line using the '-r' parameter, ie '-r rtable.netfpga'. The format is as follows:
<net> <next hop> <mask> <outgoing interface name>
An example:
192.168.130.14 192.168.130.9 255.255.255.254 eth1
(Optional) Specify a file to log packets in pcap format
Specify a file using the '-l' command line parameter, ie '-l scone.log', all packets
received by SCONE will be logged using pcap format to this file. This file can be
opened and examined using Wireshark. Note packets that take the hardware
forwarding path will not appear in this log file.
How to use SCONE?
The router SCONE ( S oftware C omponent O f Ne tFPGA) is a user level router that performs
IPv4 forwarding, handles ARPs and various ICMP messages, has telnet (port 23) and web (port
8080) interfaces to handle router control, and also implements a subset of OSPF named PW-OSPF.
SCONE mirrors a copy of its MAC addresses, IP addresses, routing table, and ARP table to the
NetFPGA card which hardware accelerates the forwarding path.
What is SCONE?
36 de 56
Writing MAC Addresses to the NetFPGA
uint8_t mac_addr[6];
unsigned int mac_hi = 0;
8/8/2010 10:20 PM
For an incoming packet to take the hardware forwarding path the packet must get a hit on the
destination MAC address table, the routing table, and the ARP table for the next hop. SCONE
builds and maintains the routing table containing static and dynamic routes, and the ARP table in
software. When changes are detected in the software tables, SCONE copies them down into the
NetFPGA's hardware tables. If you want packets to be passed up to software that match the IP
addresses assigned to the interfaces you must also push these IP addresses down into a hardware
table. Following are some code snippets for writing to these various tables. In general to write you
first write the registers representing the data for the row, then you write the row number into the
corresponding WR register. To read a row, first write the row number you want into the RD register,
then read the data from the corresponding registers.
How does SCONE work with NetFPGA?
To modify the way SCONE operates after launch, use the telnet command line interface, or the web
interface. The web interface supports all the commands available from the telnet interface, and
provides AJAX style capabilities to keep the data current as the router runs. To use either of these
interfaces, connect to one of the IP addresses specified in the cpuhw file on port 23 for telnet, or
port 8080 for http. To get a list of commands using telnet issue the ? command. A graphical version
of this printout is available from the Command List link of the web interface. For help with a
command enter the command and put ? for its arguments. IE 'show ip ?'. Many of the commands are
documented in the Required Commands section here.
SCONE's web interface
37 de 56
Writing ARP entries to the NetFPGA
uint8_t mac_addr[6];
unsigned int mac_hi = 0;
unsigned int mac_lo = 0;
struct in_addr ip;
8/8/2010 10:20 PM
struct in_addr ip, mask, gw;
int i;
char* iface;
/* write the ip */
writeReg(&netfpga, ROUTER_OP_LUT_RT_IP_REG, ntohl(ip.s_addr));
/* write the mask */
writeReg(&netfpga, ROUTER_OP_LUT_RT_MASK_REG, ntohl(mask.s_addr));
/* write the next hop */
writeReg(&netfpga, ROUTER_OP_LUT_RT_NEXT_HOP_IP_REG, ntohl(gw.s_addr));
/* write the port */
writeReg(&netfpga, ROUTER_OP_LUT_RT_OUTPUT_PORT_REG, getOneHotPortNumber(iface));
/* write the row number */
writeReg(&netfpga, ROUTER_OP_LUT_RT_LUT_WR_ADDR_REG, i);
Writing to the Routing table is similar to writing to the Destination IP filter table. Note that the
output port(s) is specified in one-hot-encoded format corresponding to the output port of the User
Data Path to the Tx Queues. The function getOneHotPortNumber returns the values (in decimal): 1
, 4, 16, 64 which correspond to physical output ports 0-3 on the NetFPGA card. You can also tell a
route to specifically send to software by specifying 2, 8, 32, 128 corresponding to the nf2c0,1,2,3
interfaces. To send out of MAC ports 0 and 1, write 1+4=5 as the output port entry.
Writing routing entries to the NetFPGA
struct in_addr ip;
writeReg(&rs->netfpga, ROUTER_OP_LUT_DST_IP_FILTER_IP_REG, ntohl(ip.s_addr));
// i is the row to write the IP address to
writeReg(&rs->netfpga, ROUTER_OP_LUT_DST_IP_FILTER_WR_ADDR_REG, i);
The HCORR has a table that stores IP addresses called the destination IP filter table. Packets whose
destination IP address matches an entry in this table will be sent to the hardware. The number of
entries in this table can be found in the Register Map. We use this table to filter out the IP addresses
of the router and the IP addresses of PW-OSPF multicast packets. We write into the table as
follows:
Writing interface IP Addresses to the NetFPGA
NOTE : One confusing aspect of using a user level process to control the router is that the MAC
addresses of the router will not match the kernel MAC addresses. That is, the MAC addresses of the
nf2cX interfaces do not necessarily reflect those that are actually in the hardware. The nf2cX
interfaces are software independent entities that do not necessarily reflect the state of the hardware.
In the next section we will describe the Router Kit which reflects the Linux's view of the network to
HCORR (Hardware Component of Reference Router). This includes the ports' MAC addresses, IP
addresses, the routing table, and the ARP cache.
unsigned int mac_lo = 0;
mac_hi |= ((unsigned int)mac_addr[0]) << 8;
mac_hi |= ((unsigned int)mac_addr[1]);
mac_lo |= ((unsigned int)mac_addr[2]) << 24;
mac_lo |= ((unsigned int)mac_addr[5]);
writeReg(&netfpga, ROUTER_OP_LUT_MAC_0_HI_REG, mac_hi);
writeReg(&netfpga, ROUTER_OP_LUT_MAC_0_LO_REG, mac_lo);
// 1,2,3 can be substituted in for 0 to set the MAC for the other ports
38 de 56
8/8/2010 10:20 PM
rkd continuously polls the routing table and ARP cache state from /proc/net/route and /proc/net/arp
respectively. When a change in state is detected, ./rkd writes the updated state to the NetFPGA
How it Works
Router kit is not limited to manual manipulation from the command line. All state (including
dynamic state) is mirrored. To wit, running rkd alongside a standard routing daemon, such as XoRP,
or Zebra/Quagga, should provide hardware acceleration of the forwarding table without any further
configuration (provided the routing software is using the NetFPGA interfaces for forwarding).
route add default gw 10.0.0.1 dev nfc0
To add an entry into the routing table, use route(8) (or ip(8)). For example, adding a default entry
with a next hop of 10.0.0.1 out of the first port would look something like:
arp -s 1.2.3.4 ca:fe:de:ad:d0:d0 -i nfc0
rkd will attempt to mirror all ARP cache and routing table entries associate with a NetFPGA
interface into hardware. This provides a very simple (and familiar) method of of adding entries to
the hardware. For example, to add a static ARP entry, simply use the arp(8) command. The
following command will add a static ARP entry.
Router Kit is only useful on a Linux host with NetFPGA2 installed, and the ipv4 reference router
bitfile loaded. Given this setup each port on the NetFPGA2 card is available to Linux via a nfc*
interface (i.e. nfc0, nfc1, nfc2, and nfc3 assuming a single card is installed).
Using Router Kit
rkd should work from the command line without any external configuration options. Simply run
(./rkd). To run the process in the background use -d. You may specify the polling time in
milliseconds using the -i option.
rkd [-h|--help} [-d|--daemon} [-i|--interval]
Running Router Kit
Router Kit is a simple approach to providing hardware acceleration to an unmodified Linux system.
It is comprised of a single program, rkd (Router Kit Daemon), which monitors the Linux routing
table and ARP cache and mirrors it down to the NetFPGA IPv4 reference router implementation.
Overview
Router Kit Walkthrough
int i;
mac_hi |= ((unsigned int)mac_addr[0]) << 8;
mac_hi |= ((unsigned int)mac_addr[1]);
mac_lo |= ((unsigned int)mac_addr[5]);
writeReg(&netfpga, ROUTER_OP_LUT_ARP_MAC_HI_REG, mac_hi);
writeReg(&netfpga, ROUTER_OP_LUT_ARP_MAC_LO_REG, mac_lo);
/* write the next hop ip data */
writeReg(&netfpga, ROUTER_OP_LUT_ARP_NEXT_HOP_IP_REG, ntohl(ip.s_addr));
/* set the row */
writeReg(&netfpga, ROUTER_OP_LUT_ARP_LUT_WR_ADDR_REG, i);
39 de 56
gcc
gcc
gcc
gcc
gcc
8/8/2010 10:20 PM
-g
-c -o cli.o cli.c
-g
-c -o ../common/util.o ../common/util.c
-lncurses cli.o ../common/nf2util.o ../common/util.o ../common/reg_defines.h
-g
-c -o regdump.o regdump.c
-lncurses regdump.o ../common/nf2util.o ../common/reg_defines.h
-o regdump
A standalone small CLI that allows you to change routing table entries, ARP cache entries, and
other settings is also provided in case the user doesn't want to run SCONE. The CLI is under
netfpga/lib/C/router. To build it, type make in that directory. You should see output similar to the
following:
Command Line Interpreter
The Quickstart Panel provides a summary of things that can be done with the hardware. There is
also a tab for viewing statistics and a tab for details. The details page will show the data path
pipeline we saw earlier under the Reference NIC Walkthrough with clickable buttons. Clicking on
those buttons will open up new buttons with more details and configuration settings.
To run the GUI for the router, cd into netfpga/lib/java/gui and type ./router.sh. The GUI should
pop-up. The GUI constantly polls the data it reads from the hardware. To make updates faster, you
can change the update rate under the File menu at the top left.
make[1]: Entering directory `/home/jnaous/NF2/lib/C/common'
gcc -fpic -c nf2util.c
gcc -shared nf2util.o -o libnf2.so
make[1]: Leaving directory `/home/jnaous/NF2/lib/C/common'
Building java...
Done
Writing router gui manifest...
Building router jar...
Writing script to start router gui...
Writing event capturing router gui manifest...
Building event capturing router jar...
Writing script to start event capturing router gui...
To build the GUI, first make sure that you have Sun's Java Development Kit (version >=1.6.0)
installed and make sure the java, javac, and jar binaries are in your path (otherwise, you can edit the
Makefile under lib/java/gui to reflect the actual locations). Then cd into NF2/lib/java/gui and type
make clean. Then type make. You should get output similar to below:
The Java GUI allows the user to change entries in the Routing Table and ARP cache as well as the
router's MAC and IP addresses. It also provides updates on counter values and graphs of throughput
and much more. The GUI has a part that is written in C that provides the interface between the Java
binaries and the driver. This native library is compiled from the nf2util.c file that contains the
readReg and writeReg utilities used in the previous section. The library connects to the GUI using
the Java Native Access (jna) library.
Java GUI
In this section we will go through the available tools to communicate with the Hardware Component
of the Reference Router (HardCORR) and go through the process of modifying the design,
simulating it, and finally implementing it. The tools that we will go over quickly are a Java GUI and
a Standalone Command Line Interface (CLI).
Reference Router Walkthrough
through the register interface. All traffic not handled by the hardware is DMA'd to software where it
is processed by the Linux kernel.
40 de 56
8/8/2010 10:20 PM
The CTRL bus is used for two purposes. The first is to distinguish module headers from each other
As a packet is processed by the modules in the data path, each module has the choice of either
adding a 64-bit word to the beginning of the packet that contains the result of the processing or
modifying a word that already exists. We call this word a module header. Modules that come later
in the pipeline can use the information in the module header(s) to do further processing on the
packet. In the reference designs we use only one module header that is modified by almost all
modules in the pipeline.
Packet hand-off between two consecutive modules.
ModuleInterface.svg
The user data path is 64-bits wide running at
125MHz. This would give the data path a peak
bandwidth of 8Gbps. Packets are passed between
modules using a fifo-like simple push interface
with four signals: WR, RDY, DATA, and CTRL.
The WR and RDY signals are each one bit wide
while the CTRL is 8 bits wide and the DATA is 64
bits wide. Say module i wants to push a packet to
module i+1 . When i+1 is ready to accept some
data, it will assert the RDY signal. Module i then
puts the packet data on the DATA bus, sets the
CTRL bus according to the data it is transmitting
Diagram of the reference pipeline
(the values of the CTRL bus are discussed later),
and raises the WR signal whenever it is ready to
send. Module i+1 should be ready to latch the data transmitted the next clock cycle after asserting
the RDY signal. If module i+1 cannot accept anymore data, it will deassert the RDY signal at least
one clock cycle before the data should stop arriving. The figure below demonstrates a valid
transaction.
Hopefully you still remember the reference
pipeline and what each major module in the
pipeline does. We show it again here for
reference. Please go over it again if you need to
know what each module does.
Reference Pipeline Details
This section will guide you through the process of creating your own project based on the reference
router and adding a library module in the data path that would limit the rate at which the NetFPGA
output packets on a particular port. We will first go through an overview of the router design and
the interface between modules. Then we will explain how to add a library module into the design
and put it in the data path. Following that, we will go through verifying the design by simulation and
finally implementing it as a downloadable bitfile.
Modifying the Reference Router
We invite you to extend this CLI and any of our software tools and contribute them back so we can
expand our library and make it easier for anybody to use NetFPGA.
For help on using the CLI, start it by typing ./cli and then type help in the CLI.
gcc -g
-c -o show_stats.o show_stats.c
gcc -lncurses show_stats.o ../common/nf2util.o ../common/util.o ../common/reg_defin
41 de 56
8/8/2010 10:20 PM
A module identifies whether it is the target of a request by inspecting the REG_ADDR_IN signal. If
the address matches the address range assigned to the module then the module should process the
Register requests/replies are signaled by a high on REG_REQ_*. REG_REQ_* should only be high
for a single clock cycle otherwise it indicates multiple register acccess. Note that a module is
permitted to take more than one clock cycle to produce a reply but it should ensure that requests
following the initial request are not dropped.) The REG_RD_WR_L_* signal indicates whether the
transaction is a read (high) or a write (low). REG_ACK_* should be low when the request is
generated and should only be brought high by the module responding to the request.
The register pipeline is 32-bits wide and runs at 125 MHz. Each module should have two pairs of
ports: one for incoming requests and one for outgoing replies. The following set of signals are the
input signals for a single module: REG_REQ_IN, REG_ACK_IN, REG_RD_WR_L_IN,
REG_ADDR_IN (23-bits), REG_DATA_IN (32-bits), REG_SRC_IN (2-bits). Equivalent signals
ending in _OUT exist for the output port.
Module authors may wish to incorporate registers that are accessible from the host within their
modules. To simplify the process of adding modules the register interfaces of the modules are
connected together in a pipeline manner instead of being connected in a star topology to a central
arbiter. This greatly simplifies the process of adding a module to a design as it does not require the
central arbiter to be modified.
Register Pipeline
The Output Queues module looks at the IOQ module header to decide which output queue to store
the packet in and uses the lengths from the IOQ module header to store the packet efficiently. After
the packet is removed from its output queue and pushed into its destination Tx Queue, the IOQ
module is finally removed before sending the packet out of the appropriate port. The diagram on the
right shows the format of a packet as it passes through the reference pipeline.
The Input Arbiter selects an Rx
PacketFormat.svg
queue to service and pushes a packet
Format of a packet as it passes through the hardware pipeline.
into the Output Port Lookup module
without modifying the module
header. The Output Port Lookup module decides which output port(s) a packet goes out of and
writes the output ports selection as a one-hot-encoded number into bits 48–63 of the IOQ module
header. This number has a one for every port the packet should go out on where port 0 is MAC0,
port 1 is CPU0, port 2 is MAC1, ...
The Rx Queues create a module header when they receive a packet and prepend it to the beginning
of the packet. The Rx queues store the length of the packet in bytes at the lowest 16 bits of the
module header, the source port as a binary-encoded number (port 0 is MAC port 0 and port 1 is
CPU port 0, ...) in bits 16–31, and the packet length in 64-bit words in bits 32–47. We call this
module header the IOQ module header.
(see next paragraph), and to determine the end of the packet. The CTRL bus is non-zero for module
headers and distinguishes module headers from each other when there are multiple module headers.
When the actual packet received starts after the module headers, the CTRL bus is reset to 0, and
then at the last word of the packet, the CTRL lines will indicate which byte is the last byte in the
last word. This is done by setting a 1 in the position of the last byte. Note that the first byte of the
packet is stored in the most significant byte position (byte 7, i.e. bits 63-56) of the first word of the
packet, and so on. For example, if the last word has only 1 byte, then the last word will have the last
byte in byte 7, and the CTRL word associated with that DATA word is 0b10000000. On the other
hand, if the last word has six valid bytes (i.e. packet length in bytes mod 8 = 6) then the CTRL word
that signifies the end-of-packet will be 0b00000100.
42 de 56
234a235,251
>
//------- Rate limiter wires/regs ----->
wire [CTRL_WIDTH-1:0]
rate_limiter_in_ctrl[0:3];
>
wire [DATA_WIDTH-1:0]
rate_limiter_in_data[0:3];
>
wire
rate_limiter_in_wr[0:3];
>
wire
rate_limiter_in_rdy[0:3];
>
>
wire [CTRL_WIDTH-1:0]
rate_limiter_out_ctrl[0:3];
>
wire [DATA_WIDTH-1:0]
rate_limiter_out_data[0:3];
>
wire
rate_limiter_out_wr[0:3];
>
wire
rate_limiter_out_rdy[0:3];
>
>
wire
rate_limiter_in_reg_req[0:4];
>
wire
rate_limiter_in_reg_ack[0:4];
>
wire
rate_limiter_in_reg_rd_wr_L[0:4];
>
wire [ÙDP_REG_ADDR_WIDTH-1:0]
rate_limiter_in_reg_addr[0:4];
>
wire [`CPCI_NF2_DATA_WIDTH-1:0] rate_limiter_in_reg_data[0:4];
>
wire [UDP_REG_SRC_WIDTH-1:0]
rate_limiter_in_reg_src[0:4];
In the above we have added wires to connect the new modules.
8/8/2010 10:20 PM
We have now created a copy of the reference HCORR and made a local copy of the
user_data_path.v file for overriding the reference one. The following assumes that you know
Verilog. We will now connect four rate_limiter modules in the pipeline between the output_queues
module and the MAC output ports of the user_data_path.v. The rate_limiter module is under
netfpga/lib/verilog/rate_limiter. You can take a look at it for reference. After modifying
netfpga/projects/rate_limited_router/src/udp/user_data_path.v to add the rate_limiter modules, the
diff of the original user_data_path.v with the new one should look similar to below. You can find
the modified user_data_path.v here:
~$ cd ~/NF2/projects/
~/NF2/projects$ cp -r reference_router/ rate_limited_router
~/NF2/projects$ mkdir rate_limited_router/src/udp
~/NF2/projects$ cp ../lib/verilog/user_data_path/reference_user_data_path/src/user_d
Type the following:
The NFP contains several modules that can be used to add more features to your hardware design.
The modules all exist under netfpga/lib/verilog. The module we will add in this walkthrough is a rate
limiter which allows you to control the rate at which packets aresent out of a port. The module will
be added into the user_data_path.v file.
Using a Library Module
There is a number of modules that are not described by the Reference Pipeline. These are the
nf2_top and nf2_core modules which contain the Reference Pipeline as well as modules that are
needed to generate the various clocks on the Virtex chip and interface with the PCI controller,
SRAM, DRAM, ... These modules are outside the scope of this document. We invite you to help us
extend this section of the documentation so that others may benefit from your experience.
Outside the Reference Pipeline
The REG_SRC_* signals are used by register request initiators to identify the responses that are
destined to the requestor. Each requestor should use a unique value as their source address.
request and generate a response. Once the module has completed any necessary processing it
should raise REG_ACK_OUT, set REG_DATA_OUT to the correct value in the case of a read, and
forward all other inputs to the outputs, all for a single cycle. If a module determines that it is not the
target of a request then it should forward all inputs unmodified to the outputs on the next clock
cycle.
43 de 56
(out_data_0),
(out_ctrl_0),
(out_wr_0),
(out_rdy_0),
(rate_limiter_in_data[0]),
(rate_limiter_in_ctrl[0]),
(rate_limiter_in_wr[0]),
(rate_limiter_in_rdy[0]),
(out_data_2),
(out_ctrl_2),
(out_wr_2),
(out_rdy_2),
(out_data_4),
(out_ctrl_4),
(out_wr_4),
(out_rdy_4),
(out_data_6),
(out_ctrl_6),
(out_wr_6),
(out_rdy_6),
(udp_reg_req_in),
(udp_reg_ack_in),
(udp_reg_rd_wr_L_in),
(udp_reg_addr_in),
(udp_reg_data_in),
(udp_reg_src_in),
(rate_limiter_in_reg_req[0]),
(rate_limiter_in_reg_ack[0]),
(rate_limiter_in_reg_rd_wr_L[0]),
(rate_limiter_in_reg_addr[0]),
(rate_limiter_in_reg_data[0]),
(rate_limiter_in_reg_src[0]),
437c454,525
<
-->
generate
>
genvar i;
>
for (i=0; i<4; i=i+1) begin: gen_rate_limiters
>
rate_limiter #(
>
.DATA_WIDTH
(DATA_WIDTH),
>
.UDP_REG_SRC_WIDTH
(UDP_REG_SRC_WIDTH)
8/8/2010 10:20 PM
Above: Instead of connecting the output ports of the output_queues module to the user_data_path
output ports, connect them to the rate limiter modules. The same goes for the register ring
connections.
360,363c377,380
<
.out_data_0
<
.out_ctrl_0
<
.out_wr_0
<
.out_rdy_0
-->
.out_data_0
>
.out_ctrl_0
>
.out_wr_0
>
.out_rdy_0
370,373c387,390
<
.out_data_2
<
.out_ctrl_2
<
.out_wr_2
<
.out_rdy_2
-->
.out_data_2
>
.out_ctrl_2
>
.out_wr_2
>
.out_rdy_2
380,383c397,400
<
.out_data_4
<
.out_ctrl_4
<
.out_wr_4
<
.out_rdy_4
-->
.out_data_4
>
.out_ctrl_4
>
.out_wr_4
>
.out_rdy_4
390,393c407,410
<
.out_data_6
<
.out_ctrl_6
<
.out_wr_6
<
.out_rdy_6
-->
.out_data_6
>
.out_ctrl_6
>
.out_wr_6
>
.out_rdy_6
414,419c431,436
<
.reg_req_out
<
.reg_ack_out
<
.reg_rd_wr_L_out
<
.reg_addr_out
<
.reg_data_out
<
.reg_src_out
-->
.reg_req_out
>
.reg_ack_out
>
.reg_rd_wr_L_out
>
.reg_addr_out
>
.reg_data_out
>
.reg_src_out
44 de 56
.out_data
.out_ctrl
.out_wr
.out_rdy
(rate_limiter_in_data[i]),
(rate_limiter_in_ctrl[i]),
(rate_limiter_in_wr[i]),
(rate_limiter_in_rdy[i]),
(rate_limiter_out_data[i]),
(rate_limiter_out_ctrl[i]),
(rate_limiter_out_wr[i]),
(rate_limiter_out_rdy[i]),
.in_data
.in_ctrl
.in_wr
.in_rdy
(rate_limiter_in_reg_req[i]),
(rate_limiter_in_reg_ack[i]),
(rate_limiter_in_reg_rd_wr_L[i]),
(rate_limiter_in_reg_addr[i]),
(rate_limiter_in_reg_data[i]),
(rate_limiter_in_reg_src[i]),
) rate_limiter
(
// --- Register interface
.reg_req_in
.reg_ack_in
.reg_rd_wr_L_in
.reg_addr_in
.reg_data_in
.reg_src_in
out_data_2
out_ctrl_2
out_wr_2
rate_limiter_out_rdy[1]
=
=
=
=
=
=
=
=
rate_limiter_out_data[3];
rate_limiter_out_ctrl[3];
rate_limiter_out_wr[3];
out_rdy_6;
out_rdy_4;
out_rdy_2;
`RATE_LIMIT_0
`RATE_LIMIT_1
`RATE_LIMIT_2
`RATE_LIMIT_3
(rate_limiter_in_reg_req[i+1]),
(rate_limiter_in_reg_ack[i+1]),
(rate_limiter_in_reg_rd_wr_L[i+1])
(rate_limiter_in_reg_addr[i+1]),
(rate_limiter_in_reg_data[i+1]),
(rate_limiter_in_reg_src[i+1]),
assign
assign
assign
assign
out_data_4
out_ctrl_4
out_wr_4
=
=
=
=
rate_limiter_in_reg_req[4];
rate_limiter_in_reg_ack[4];
rate_limiter_in_reg_rd_wr_L[4];
rate_limiter_in_reg_addr[4];
rate_limiter_in_reg_data[4];
rate_limiter_in_reg_src[4];
=
=
=
=
.reg_req_out
.reg_ack_out
.reg_rd_wr_L_out
.reg_addr_out
.reg_data_out
.reg_src_out
(clk),
(reset));
gen_rate_limiters[0].rate_limiter.RATE_LIMIT_BLOCK_TAG
// --- Misc
.clk
.reset
end // block: gen_rate_limiters
endgenerate
defparam
defparam
defparam
defparam
assign
assign
assign
assign
out_data_6
out_ctrl_6
out_wr_6
=
=
=
=
=
=
rate limiters
= rate_limiter_out_data[0];
= rate_limiter_out_ctrl[0];
= rate_limiter_out_wr[0];
= out_rdy_0;
assign
assign
assign
assign
udp_reg_req_in
udp_reg_ack_in
udp_reg_rd_wr_L_in
udp_reg_addr_in
udp_reg_data_in
udp_reg_src_in
//--- Connect the wires from the
assign out_data_0
assign out_ctrl_0
assign out_wr_0
assign rate_limiter_out_rdy[0]
assign
assign
assign
assign
assign
assign
8/8/2010 10:20 PM
Above: Add the rate limiter modules on each output port going to an Ethernet Tx queue. The
register ring goes through each of the rate limiter modules. Note the defparams used to assign a
register block for each rate limiter module. More on that is coming later.
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
45 de 56
io_queues/cpu_dma_queue
io_queues/ethernet_mac
input_arbiter/rr_input_arbiter
nf2/generic_top
nf2/reference_core
user_data_path/generic_cntr_reg
output_port_lookup/cam_router
output_queues/sram_rr_output_queues
sram_arbiter/sram_weighted_rr
user_data_path/reference_user_data_path
io/mdio
cpci_bus
dma
user_data_path/udp_reg_master
io_queues/add_rm_hdr
rate_limiter
8/8/2010 10:20 PM
If you wish to add registers, you will need to create a module XML file for each module added to
Copy the .ngc and .v files: Copy the generated .v files to your /src directory and the .ngc/.edn to
your /synth directory. This will avoid reimplementing the core.
To add IP cores generated with the Xilinx's tool, you can do one of two things: Copy .xco file
(recommended): Simply copy the .xco file from the directory where you generated the IP core to
the /synth directory. The build environment will automatically regenerate the .v wrapper for
simulation/synthesis and the .ngc or .edn needed for synthesis.
If you wish to add new source code (i.e. not from the library) to your project, you can simply put
the new verilog files under the src dir in your project. You could also put them in a directory one
level below the src directory. They will automatically be included for synthesis and simulation. Note
that in case some files are only used for simulation, you will have to encapsulate the
unsynthesizable code with synthesis translate_off and synthesis translate_on directives.
Adding New Sources
All these library modules can be found under netfpga/projects/lib/verilog. Some library modules are
only used for simulation, such as the testbench module. Some libraries offer alternatives such as the
user_data_path module. If you look under netfpga/lib/verilog/user_data_path, you will find two
directories, one has the user data path used for the buffer monitoring router, and one used in the
reference router. If you have the full source package (the teacher/researcher package) you will also
find an alternative for the output _port_lookup module: The cam_router which is a router using a
CAM to perform LPM lookups and the learning_cam_switch which makes the NetFPGA act as a
four-port learning switch. Note that the library modules that are not used in the reference NIC
design and the reference IPv4 Router design have not received the same thorough testing as those
that are. This includes the learning_cam_switch module and the rate_limiter module.
Note that the reference_user_data_path module is still mentioned in the lib_modules.txt even
though we are overriding one of the files. This is because we would still like to use the other files
that are in that module. The build environment will automatically handle the override and make sure
that it uses the file in the project src dir instead of the library one. Also note that the build system
only handles source files that are directlys under the src dir in the project directory and one level
below it. So if you put a file under rate_limited_router/src/udp/some_dir/some_file.v it will not be
included in the build.
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
Now all that is left is telling the build system that we want to include the rate limiter in compilation.
All the library modules that a project uses are found in projects//include/lib_modules.txt. For the
rate limited router, we will have the rate limiter module as well as all the modules that are normally
used by the reference router. The modified lib_modules.txt looks as follows, where line 16 was
added:
46 de 56
run
13
#!/usr/local/bin/perl -w
# make_pkts.pl
#
#
use NF2::PacketGen;
use NF2::PacketLib;
use SimLib;
use RouterLib;
require "reg_defines.ph";
$delay = 2000;
# use strict AFTER the $delay, $batch and %reg are declared
use strict;
use vars qw($delay $batch %reg);
$batch = 0;
nf_set_environment( { PORT_MODE => 'PHYSICAL', MAX_PORTS => 4 } );
Required lines to initiate the libraries.
14
15
16
17
18
19
8/8/2010 10:20 PM
Set the delay to send a packet. More information can be found in the comments and documentation
in the libraries.
11
Include the file that defines all register addresses.
7
8
9
10
Select the libraries to be used for simulation. Theses libraries provide functions to create/send
/expect packets and to generate register requests. In particular, the SimLib encapsulates the
nf_PCI_regread and the nf_PCI_regwrite functions which access registers. Writing of router
specific tables the RouterLib can be used. These libraries and some others that are used for real
hardware (as opposed to simulation) testing can be found under NF2/lib/Perl5/.
1
2
3
4
File header. Nothing fancy.
test_router_short/make_pkts.pl is shown below ready for dissection:
make_pkts.pl
Perl script that generates all the packets and register requests.
generates packets and runs the simulation. Usually this should not be modified except for
unusual circumstances such as simulation time parameter definitions.
config.txt
specifies when a simulation should end.
The next step after writing the code is to simulate the design. We will use the same testbenches used
for the router. These testbenches are defined under netfpga/projects/rate_limited_router/verif. Each
directory contains a description of the packets to send, when to send them and which packets we
expect to come out of the NetFPGA whether via DMA or Ethernet. The simulation environment
also allows you to specify register reads and writes. Each testbench consists of a three files:
Simulating the design
the design and modify the project XML file to include the additional modules. The high level
description of the register system can be found in the Developers Guide. A more detailed
description of the register system can be found on the Register System page.
47 de 56
70
71
72
73
74
75
my
my
my
my
my
my
my
my
$ROUTER_PORT_1_IP
$ROUTER_PORT_2_IP
$ROUTER_PORT_3_IP
$ROUTER_PORT_4_IP
$ROUTER_PORT_1_MAC
$ROUTER_PORT_2_MAC
$ROUTER_PORT_3_MAC
$ROUTER_PORT_4_MAC
=
=
=
=
=
=
=
=
'00:ca:fe:00:00:01';
'00:ca:fe:00:00:02';
'00:ca:fe:00:00:03';
'00:ca:fe:00:00:04';
'192.168.1.1';
'192.168.2.1';
'192.168.3.1';
'192.168.4.1';
my $next_hop_1_DA = '00:fe:ed:01:d0:65';
my $next_hop_2_DA = '00:fe:ed:02:d0:65';
# Prepare the DMA and enable interrupts
prepare_DMA('@3.9us');
enable_interrupts(0);
# Write the ip addresses and mac addresses, routing table, filter, ARP entri
$delay = '@4us';
set_router_MAC(1, $ROUTER_PORT_1_MAC);
$delay = 0;
add_dst_ip_filter_entry(0,$ROUTER_PORT_1_IP);
add_LPM_table_entry(0,'171.64.2.0', '255.255.255.0', '171.64.2.1', 0x04);
add_LPM_table_entry(15, '0.0.0.0', '0.0.0.0', '171.64.1.1', 0x01);
$length = 100;
$TTL = 30;
$DA = 0;
$SA = 0;
$dst_ip = 0;
$src_ip = 0;
$pkt;
# Add the ARP table entries
add_ARP_table_entry(0, '171.64.1.1', $next_hop_1_DA);
add_ARP_table_entry(1, '171.64.2.1', $next_hop_2_DA);
my
my
my
my
my
my
my
$delay = '@80us';
$length = 64;
$DA = $ROUTER_PORT_1_MAC;
$SA = '01:55:55:55:55:55';
$dst_ip = '171.64.2.7';
$src_ip = '171.64.8.1';
$pkt = make_IP_pkt($length, $DA, $SA, $TTL, $dst_ip, $src_ip);
Create the packet:
76
8/8/2010 10:20 PM
Send the first packet into port 1 (MAC port 0). Note that the ports in the simulation libraries are all
defined starting from 1 as opposed to 0.
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
Write the registers to setup routes through the router:
34
35
36
Initiate the DMA:
21
22
23
24
25
26
27
28
29
30
31
32
33
Define some variables for the tests:
48 de 56
Send it in:
77
$pkt);
$DA = $next_hop_2_DA;
$SA = $ROUTER_PORT_2_MAC;
$pkt = make_IP_pkt($length, $DA, $SA, $TTL-1, $dst_ip, $src_ip);
nf_expected_packet(2, $length, $pkt);
$length = 60;
$DA = $ROUTER_PORT_2_MAC;
$SA = '02:55:55:55:55:55';
$dst_ip = $ROUTER_PORT_1_IP;
$src_ip = '171.64.9.1';
$pkt = make_IP_pkt($length, $DA, $SA, $TTL, $dst_ip, $src_ip);
nf_packet_in(2, $length, '@82us', $batch, $pkt);
nf_expected_dma_data(2, $length, $pkt);
$delay = '@100us';
PCI_create_and_send_pkt(2, $length);
# *********** Finishing Up - need this in all scripts ! ********************
my $t = nf_write_sim_files();
print "--- make_pkts.pl: Generated all configuration packets.\n";
printf "--- make_pkts.pl: Last packet enters system at approx %0d microsecon
if (nf_write_expected_files()) {
die "Unable to write expected files\n";
}
nf_create_hardware_file('LITTLE_ENDIAN');
nf_write_hardware_file('LITTLE_ENDIAN');
nf_run_test.pl --help
8/8/2010 10:20 PM
The test should then generate the packets and register requests and run the simulation in console
mode. When the test is done, it will search for the word ERROR in the log to find out if an error
occurred and let you know. If you would like to run the test in a GUI to view the waveforms and
have complete control over the simulation, you can add the --gui switch to the command. For more
information on other options type:
nf_run_test.pl --major router --minor short
Now to run the simulation, we need to make sure that our environment is set. Make sure that your
NF_ROOT environment variable points to the path where the netfpga directory is (e.g. ~/netfpga)
and that your NF_DESIGN_DIR points to ${NF_ROOT}/projects/rate_limited_router. Also make
sure that you are sourcing the settings from ${NF_ROOT}/bin/nf2_profile or ${NF_ROOT}/bin
/nf2_cshrc depending on your shell. Of course we will need to have Modelsim installed. To run the
simulations:
163
164
165
166
167
168
169
170
171
172
The rest of the lines till 163 test different size packets. After that there's some code that generates
all this info and puts it in files. The rest should not be changed in general:
98
99
Now send a packet out from nf2c1. This also says that we should expect the same packet to come
out of MAC port 1:
96
Specify that we expect the packet to come on DMA port 2 (a.k.a nf2c1):
88
89
90
91
92
93
94
Create a new packet from a different port that is destined for the router itself. The packet should be
sent to the CPU via DMA.
79
80
81
82
Create the packet that we expect to see coming out of port 2 (MAC port 1):
nf_packet_in(1, $length, $delay, $batch,
49 de 56
TIMING := 0
FAIL: 0
You can set these switches to 1 or just comment them out to re-enable Smartguide or timing.
Testing the New Router
8/8/2010 10:20 PM
Also, by default, map uses the timing switch to improve timing. This can in some cases lead to weird
errors during the map process. To disable the use of the timing switch, add the following line to
/synth/Makefile:
USE_SMARTGUIDE := 0
The synthesis process uses Smartguide by default when rebuilding a project. In cases where the
netlist is changed dramatically between synthesis runs, or where the place and route process does
not manage to route the nets to meet timing, Smartguide will produce results that fail to meet timing
or will take a very very long time to finish (and still fail to meet timing). In these cases, you can
disable Smartguide by adding the following line before the include in the synth/Makefile:
nf_download ./nf2_top_par.bit
When make is done, you should have a bitfile called nf2_top_par.bit that should be downloaded to
the NetFPGA to run. To download it, use:
cd rate_limited_router/synth
make
Implementing the design is a very simple process:
Implementing the Design
FAILING TESTS:
TOTAL: 1 PASS: 1
...
# Timecheck: 493645.00ns
#
500100 Simulation has reached finish time - ending.
# ** Note: $finish
: /home/jnaous/mckeown/NF2/new_tree/lib/verilog/testbench/targ
#
Time: 500100 ns Iteration: 0 Instance: /testbench/target32
--- Simulation is complete. Validating the output.
Comparing simulation output for port 1 ...
Port 1 matches [0 packets]
Comparing simulation output for DMA queue 1 ...
DMA queue 1 matches [0 packets]
--- Test PASSED
Test test_router_short passed!
------------SUMMARY--------------PASSING TESTS:
test_router_short
After you run the simulation you should see output similar to what is seen below (this is only the
last part):
50 de 56
8/8/2010 10:20 PM
On every clock cycle, there are 4 types of possible events: A packet stored, a packet removed, a
evt_rcrdr
The system is composed of two main modules: evt_rcrdr and evt_pkt_wrtr. The evt_rcrdr module
records individual events as they happen and serializes them to be sent out to the evt_pkt_wrtr
module. The evt_pkt_wrtr then reads each event and stores it, and then when the send_egress_pkt
is ready, it sends out a complete event packet.
Design Details
where nf2c3 is the interface receiving the event packets.
rcv_evts -i nf2c3 -v
The rcv_evts program is supposed to record incoming evts, parse them, and display the output to
stdout. It can be used as a basis on which to do more interesting things with the event packet
information (such as build the queue occupancy in time). To use the program, simply run:
rcv_evts
For more information on buffer monitoring registers please see the Register Map.
The monitor_ctrl program is found in netfpga/projects/router_buffer_sizing/sw. It implements all the
functions to control and view the status of the buffer monitoring system. The program works by
reading and changing the control register of the buffer monitoring system. In addition, several
read-only registers were added to observe the behavior of the subsystem. Executing monitor_ctrl -h
will show available options. monitor_ctrl -p will print the status.
monitor_ctrl
The system has two software components: monitor_ctrl and rcv_evts. These two components
control the buffer monitoring system and record the events at the receiver respectively.
Using the system
In order to record the development of buffer sizes in switches, a system is needed to record arrival
and departure of packets into the queue. The buffer monitoring subsystem is designed to meet this
need. The subsystem works by monitoring events such as arrival/departure/drop of a packet on the
queues, then recording a Timestamp as well as the length of the packet. These events are then
output in a packet with a certain format called the event packet.
Introduction
Buffer Monitoring System
A full Perl library is available to test the actual hardware. The library netfpga/lib/Perl5/TestLib.pm
contains many functions that can be used to send/receive packets and read/write registers. For more
information on using these libraries, you can look at the various regression tests under
netfpga/projects/reference_router/regress as well as looking through the library code to see which
functions are available and read the comments to know what they do. We invite you to help us by
expanding this section or submitting patches that provide additional documentation on using the
Perl library.
51 de 56
8/8/2010 10:20 PM
Following the Ethernet, IP, and UDP headers, there are 4 reserved bits, then 4 bits indicating the
version of the buffer monitoring system, then 8 bits indicating the number of event types excluding
the Timestamp event type. This is followed by a 32 bit sequence number (starting at 0) and a list of
the queue sizes before the first short event. A series of short events then follows. Periodic Time
Stamps make it possible to keep track of the exact time of each event.
Packet Format
The system follows the reference user data path.
Schematic
This module reads events from the evt_rcrdr module and delineates them into packets. The module
also monitors the datapath for activity and only injects event packets when the datapath is idle.
evt_pkt_wrtr
packet dropped, and a Timestamp event. The first three event types we call short events because
they only need 32 bits. Whereas the Timestamp is a long event since it uses 64 bits. The short
events carry the 19 least significant bits of the clock. Periodically, a Timestamp event is signaled
and recorded. The evt_rcrdr rearranges the events as they come in and stores them in a single clock
cycle into the event fifo. The event fifo is a shallow fifo (depth=8) with a variable input size. The
input is composed of five 32-bit words of which we can store a variable number of words (the first x
words are stored). This is the number of events at the current clock cycle (the Timestamp event
takes 2 words). Note that the evt_rcrdr assumes that the events are all independent. The maximum
sustained event recording capability is 62.5 million events per second, whereas the peak event
recording capability is 8 events in any 32 ns interval (after which events should not go over the
average or events would be lost). It is possible to adapt the evt_rcrdr to record any signal that meets
these requirements with an additional field of 9-bits (usually the length field) to record any
additional data. In addition, the evt_rcrdr make no assumption on the sizes of the fields. These
could be modified by changing the sizes in unet_defines.v. However, note that the evt_pkt_wrtr
does make the assumption that the word sizes are 32 bits since that would simplify writing to the
send_egress_pkt. However, this could be easily adapted as well.
52 de 56
8/8/2010 10:20 PM
1- Arrival/Departure/Drop (short) Events. Note that the packet length here is in words of 64-bits.
The default time resolution is 8ns.
There are four event types:
53 de 56
Contributed Packages
Packages contributed by Alpha testers and Beta Users
Wiki: editable by Dev and Alpha Users
Web page viewable by all
Tutorial Setup
Contains Tutorial setup instructions
Tutorial Setup
Other Pages
8/8/2010 10:20 PM
Revision 2:
PDF
Orcad Capture file
Revision 3:
PDF
Orcad Capture file
An Allegro layout of the board is also available:
Revision 2:
http://netfpga.org/net_fpga-2p1_051807_v15.zip
Revision 3:
Allegro layout will be released when we have a confirmed working revision 3 board
The schematic of the NetFPGA board is available as both a PDF and an Orcad Capture file:
Schematic and board layout
Links
2- Timestamp Event: Periodically recorded to keep the time updated.
54 de 56
► Attachments 22 22
Edit | Attach | Print version | History: r27 < r26 < r25 < r24 < r23 | Backlinks | Raw View | Raw
edit | More topic actions
Topic revision: r27 - 26 Jun 2010 - 02:12:37 - Main.JonathanEllithorpe
Log In or Register
Learn
Learn More
Get Started
Users Guide
License
Develop
Develop
Developers Guide
Report Bugs
Contribute
Community
Projects
8/8/2010 10:20 PM
'''The name and trademarks of copyright holder(s) may NOT be used in advertising or publicity
pertaining to the Software or any derivatives without specific, written prior permission.
The above copyright notice and this permission notice shall be included in all copies or
substantial portions of the Software. THE SOFTWARE IS PROVIDED "AS IS", WITHOUT
WARRANTY OF ANY KIND, EXPRESS OR IMPLIED, INCLUDING BUT NOT LIMITED TO
THE WARRANTIES OF MERCHANTABILITY, FITNESS FOR A PARTICULAR PURPOSE
AND NONINFRINGEMENT. IN NO EVENT SHALL THE AUTHORS OR COPYRIGHT
HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER LIABILITY, WHETHER
IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM, OUT OF OR IN
CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE
SOFTWARE.
Copyright (c) 2006 The Board of Trustees of The Leland Stanford Junior University We are
making the NetFPGA tools and associated documentation (Software) available for public use and
benefit with the expectation that others will use, modify and enhance the Software and contribute
those enhancements back to the community. However, since we would like to make the Software
available for broadest use, with as few restrictions as possible permission is hereby granted, free of
charge, to any person obtaining a copy of this Software) to deal in the Software under the
copyrights without restriction, including without limitation the rights to use, copy, modify, merge,
publish, distribute, sublicense, and/or sell copies of the Software, and to permit persons to whom the
Software is furnished to do so, subject to the following conditions:
The NetFPGA code is distributed under a BSD-style license shown below. Please make sure you
read and understand it. The design of the board itself is also available freely.
License
Register Map - Information on all available registers and how to use them.
Verilog Library Modules - Place holder
55 de 56
Home
Powered by Foswiki.
© NetFPGA. NetFPGA Menu by LRD Group.
Webs
NetFPGA
OneGig
TenGig
System
Toolbox
Create New Topic
Index
Search
Changes
Notifications
RSS Feed
Statistics
Preferences
Forums
Events
Publications
56 de 56
Applications Events News Wiki Forums About
8/8/2010 10:20 PM

Guide < NetFPGA/OneGig < Fo

Transcrição

Documentos relacionados

Port Governance in Santos, Brazil: A Qualitative Approach

SbA760 - Loja DJ

r egistr atio n

Israel`s Battles in the Wilderness.pmd

Keys router zon

day of AD 1955. THAT the organisations named in the

CANADIAN CANAL SOCIETY PRESENTATION

The history of port - in brief

KHSTVQCDONB Agora Smart+ HDMI TV Dongle User

LapBand long term results