ProtoFlex   Carnegie Mellon

IISWC 2010 Tutorial

Welcome to the IISWC 2010 Protoflex tutorial! Download the tutorial slides here.


During this tutorial we will first briefly cover the basic ProtoFlex simulation architecture concepts and we will then move onto a hands-on session, where we will test-drive the recently released ProtoFlex Simulator. In particular we will go over the hardware and software installation procedures, and the process for staging and running your first simulation on the FPGA. This tutorial assumes you are familiar with basic computer architecture concepts and general simulation tools. No prior knowledge of FPGAs is required.

The ProtoFlex Simulator

The ProtoFlex Simulator is an open-sourced simulator developed at Carnegie Mellon University to facilitate scalable, shared-memory multiprocessor research using FPGAs. In its basic form, the ProtoFlex Simulator simulates a functional model of an N-way UltraSPARC III server system and is able to run unmodified, multithreaded applications on a Solaris operating system. The ProtoFlex Simulator is a parameterizable simulator and has been shown to simulate up to 16 processors on a BEE2 FPGA platform. The version of the ProtoFlex Simulator that you will be using has been ported over to the XUPV5-LX110T platform, which is a widely-available commodity FPGA platform.


Throughout this tutorial, we will assume the following terminology. A target system refers to the simulated machine that we are interested in modeling (in the case of ProtoFlex, this is the Serengeti-based UltraSPARC III server). A host system refers to the underlying collection of hardware and software used to support the simulation of the target system. This includes the FPGA platform as well as software components that run on an x86-based workstation.

The target machine that we will be simulating on the FPGA is a functional model of a 4-CPU UltraSPARC III shared-memory server. The target application that runs on this model will be the Solaris 10 operating system. We will also stage and run a simple multithreaded microbenchmark within the operating system.

The Primary PC is a PC with a PCI Express x1 slot that hosts the XUPV5 FPGA board. The Secondary PC is another PC that is used for configuring the FPGA and monitoring the RS232 output. The figure below shows a high-level view of our remote setup, which consists of 4 PC pairs.

Since we only have four FPGA boards available in our remote infrastructure, please divide up into four separate groups (protoflex1, protoflex2, protoflex4). Each group should have a Windows laptop with the Remote Desktop client available.

Remote Desktop Access to CMU's FPGA infrastructure

For this tutorial you will only need to connect to the Secondary PC that is assigned to your group; all steps below will be performed through this Secondary PC, running Windows XP. To connect, first open up the Remote Desktop Connection tool and connect to the following address: where X is your designated group number (so for example, if you are in the protoflex2 group, connect to The username/password is: pf_user/protoflex. This username/password pair is common for all remaining steps of this tutorial, unless otherwise noted.

Now please go ahead and connect to your designated Secondary PC! Once connected to the Secondary PC you will need to establish an SSH connection to the Primary PC that runs Linux and hosts the FPGA board. From the desktop open SSH Secure Shell Client and click on “Quick Connect”. In the “Host Name” field type protoflexN.scotch.ece.cmu.local, where N is your group number (e.g. protoflex1.scotch.ece.cmu) and in the “User Name” field type pf_user. Hit connect and enter the password “protoflex”. Now open “SSH Secure File Transfer” and repeat the same steps one more time. At this point you should see something like this

Hands-on Session

During the hands-on session of this tutorial you will become familiar and try out most of the steps involved in setting up and using the ProtoFlex Simulator. Some time-consuming steps will be omitted. Such steps are marked with SKIP in their title. For instance instead of going through the lengthy and automated process of generating the FPGA bitstream, we will directly provide you with pre-generated files. More details on the ProtoFlex Simulator can be found in the User Guide. Below is a time line highlighting the basic steps we will be going through. Note, any steps that are marked with CLICK should be followed through.

Installing OS and Software

OpenSuse 11.1 and all necessary software packages (Simics, Bluespec, Xilinx Tools) have already been installed for you.

SKIP - Expand/Collapse

1. Downloading and compiling the ProtoFlex source code to the Primary PC

 All of these steps should be performed on the Primary PC (Linux). Note: For your convenience we have already placed all of the files that you would normally download in the tutorial_files folder in your home directory.

CLICK - Expand/Collapse

Installing Xilinx Software and IP

To save time Xilinx Software (ISE, EDK) has already been installed and the steps needed to prepare the PCI Express Verilog code has already been done for you.

SKIP - Expand/Collapse

SKIP - 2. Preparing a Simics checkpoint

 Since we will be using the checkpoint you created during the Flexus part of the tutorial (~/checkpoints/final), you can skip this step.

SKIP- Expand/Collapse

SKIP - Installing Solaris in a simulated machine

We will omit this step, because installing an OS on the simulated target machine is an (uninteresting) process that can take many hours. We have provided you with a fresh disk image of the target system (prior to boot).

SKIP - Expand/Collapse

SKIP - Boot Solaris and Save Checkpoint

CLICK - Expand/Collapse

SKIP - 3. Preparing a test workload

 Since we will be using the checkpoint you created during the Flexus part of the tutorial (~/checkpoints/final), you can skip this step.

SKIP - Expand/Collapse

Validating a Workload for ProtoFlex

CLICK - Expand/Collapse

4. Generating the Bitstream

CLICK - Expand/Collapse

Generating and synthesizing RTL on the Primary PC

CLICK - Expand/Collapse

5. Preparing the PCI express driver on the Primary PC

SKIP - Updating Linux kernel

SKIP - Expand/Collapse

Build PCI express driver

CLICK - Expand/Collapse

6. Downloading the bitstream to FPGA

CLICK - Expand/Collapse

Using HyperTerminal on the Secondary PC to connect to the FPGA serial port

CLICK - Expand/Collapse

Programming the FPGA from the Secondary PC

CLICK - Expand/Collapse

Configuring the PCI express driver

CLICK - Expand/Collapse

7. Running the PFMON Tool

CLICK - Expand/Collapse


CLICK - Expand/Collapse

Congratulations! You made it!


ProtoFlex: Towards Scalable, Full-System Multiprocessor Simulations Using FPGAs
Eric S. Chung, Michael K. Papamichael, Eriko Nurvitadhi, James C. Hoe, Babak Falsafi, and Ken Mai.
ACM Transactions on Reconfigurable Technology and Systems, 2009.

Implementing a High-performance Multithreaded Microprocessor: A Case Study in High-level Design and Validation
Eric S. Chung and James C. Hoe.
Formal Methods and Models for Codesign (MEMOCODE), July 2009.

A Complexity-Effective Architecture for Accelerating Full-System Multiprocessor Simulations Using FPGAs
Eric S. Chung, Eriko Nurvitadhi, James C. Hoe, Babak Falsafi, and Ken Mai.
International Symposium on Field Programmable Gate Arrays, February 2008, Monterey, CA.