Design and Implementation of Non-Volatile Memory Express

Abstract

Flash-memory-based solid-state disks (SSDs) provide faster random access and data transfer rates than electromechanical drives and today it can often serve as rotating-disk replacements, but the host interface to SSDs remains a performance bottleneck and also I/O subsystem causes unnecessary latencies, translations in the Read/Write commands. In order to completely utilize the performance of SSDs a Non Volatile Memory Subsystem was designed based on the NVM Express Specification. The communication to this I/O subsystem is through PCI Express interface and the command set is based on NVMe 1.0c Specification. The designed subsystem typically consists of PCIe Core, PCIe controller, NVMe controller, NAND Flash Controller and several NAND Chips. The present paper deals with the design and implementation of PCIe controller and the NVMe controller. The PCIe controller was designed as a generic bridge between any PCIe device and the PCIe Core. The NVMe controller was designed as a PCIe device which implements the NVMe Specification. Keywords—HDD, SSD, PCI Express, NVM Express and NAND Flash Controller.

Full text

2014 International Conference on Recent Trends in Information Technology
978-1-4799-4989-2/14/$31.00 © 2014 IEEE
Abstract— Flash-memory-based solid-state disks (SSDs) provide
faster random access and data transfer rates than
electromechanical drives and today it can often serve as rotating-
disk replacements, but the host interface to SSDs remains a
performance bottleneck and also I/O subsystem causes
unnecessary latencies, translations in the Read/Write commands.
In order to completely utilize the performance of SSDs a Non
Volatile Memory Subsystem was designed based on the NVM
Express Specification. The communication to this I/O subsystem
is through PCI Express interface and the command set is based
on NVMe 1.0c Specification. The designed sub-system typically
consists of PCIe Core, PCIe controller, NVMe controller, NAND
Flash Controller and several NAND Chips. The present paper
deals with the design and implementation of PCIe controller and
the NVMe controller. The PCIe controller was designed as a
generic bridge between any PCIe device and the PCIe Core. The
NVMe controller was designed as a PCIe device which
implements the NVMe Specification.
Keywords—HDD, SSD, PCI Express, NVM Express and NAND
Flash Controller.
I. INTRODUCTION
HE
design of a high-end in-house processor for defense
and security related applications as let to the development
of the subsequent. The Figure 1 shows an overall view of the
targeted processor. The processor system has quad-core
architecture, with a support for two levels of cache hierarchy
and a single on-chip DRAM.
It implements cache coherency at L1 cache level, with
coherency bus. The CPU core is based on 64-bit PowerPC
Instruction Set Architecture. It supports “dual-issue” and “out
of order execution”. A Memory Management Unit was
designed for an efficient data transfer between the Processor
Core and the Main Memory (DRAM). It supports NVM
Express based I/O Subsystem with a PCI Express interface.
This I/O subsystem typically implements the “File System”
for the processor.
The solid-state drives (SSDs) introduction in conventional
computing schemes is restructuring the patron storage site.
Sivashankar P is a P.G research Scholar doing Masters in VLSI Design at
R.M.K Engineering College/Anna University, Chennai, Tamil Nadu, India (E-
mail: sivashankaryadav@outlook.com).
Dr. S. Ramasamy, is Professor in department of Electronics &
Communication Engineering, R.M.K Engineering College/Anna University,
Chennai, Tamil Nadu, India. (E-mail: srs.ece@rmkec.ac.in).
Two major transformations are ongoing: a new storage device
interface architected for nonvolatile memory (NVM), and
novel caching solutions that extend the benefits of NVM to a
wider segment of users at a more affordable price. At the
hardware interface level, Serial ATA (SATA) is giving way to
Peripheral Component Interconnect Express (PCIe) to deliver
higher performance. To realize the full benefits of NVM over
the next decade, the engineering has defined NVM Express
(NVMe), the software interface optimized for PCIe SSDs.
NVMe was architected from the ground up for NVM to ensure
a high-throughput, low latency, robust solution.
Fig1. Overview of Processor Architecture
In order to store large amounts of data, an NVM Express
based I/O subsystem was designed, which functions as a back
end for On-chip DRAM. PCI Express has been used as the
interface between the processor and the NVM Subsystem. The
highlighted portion in the Figure 1 shows my contribution to
the NVM Subsystem.
Sivashankar, P.G Scholar, Dr. S. Ramasamy, Professor
Design and Implementation of Non-Volatile
Memory Express
T

2014 International Conference on Recent Trends in Information Technology
978-1-4799-4989-2/14/$31.00 © 2014 IEEE
The theme is to design IP of NVM Express controller with
reference to the specification of NVM Express spec 1.0c and
test it for its performance and take the same to the VLSI
backend tools for back end GDS extraction and verify for its
area, speed and power.
II. B
ACKGROUND
SSDs based on future NVM technologies promise to deliver
speeds on the order of millions of IOP Seven for client
devices. However, a key question for the PCIe transition is
whether the existing software landscape will continue to
support SSD needs through the next decade.
A. PCIe as an SSD Interface
In the Enterprise and Client Systems, PCIe based SSDs are
emerging as a back-end for DRAMS. The main factors for the
wide-spread adoption of PCIe as an interface for SSDs are
given below:
PCIe is High Performance
1) It is a full-duplex system, which can support
multiple outstanding requests, and out of order
processing.
2) It has a scalable port width ranging from single
lane (x1) to sixteen lanes (x16).
3) It also has a scalable link speed. Links speeds
include 2.5GTps, 5GTps and 8GTps.
4) It is a direct attach to CPU subsystem hence, it has
no HBA latency.
PCIe is Low Cost
1) It is a high volume commodity interconnects.
2) It has less number of pins. Hence lower area and
hence lower cost.
3) It is a direct attach to CPU subsystem hence, it
eliminates HBA cost.
PCIe also provides effective Power Management
1) Direct attachment to CPU subsystem eliminates
HBA power.
2) It implements power budgeting and Dynamic
Power Allocation [1].
B. Need for NVM Express
Standardized Interface
A standardized interface is need for the easy adoption of
PCIe based SSDs. NVM express specification provides this
standard interface for PCIe SSDs.
Scalable
NVM Express is scalable host controller interface standard,
which is designed for Enterprise and Client systems that use
PCI express SSDs.
Efficient Command Set
It provides a simple, streamlined and efficient command set
which eliminates the legacy HDD to SSD command
conversion overhead.
Optimized Register Interface
It provides an optimized register interface that allows the
host software to communicate with the non-volatile memory
subsystem.
Industry Support
It was developed by industry consortium of 80+ members
and hence enjoys a very wide industry support.
III. K
EY ATTRIBUTES OF NVM EXPRESS CONTROLLER
1. It does not require uncatchable/MIMO register reads
in the command issue or command completion path.
Each such access would otherwise take 2000 clock
cycles
2. A maximum of one MMIO register write is necessary
in the command issue path
3. Support for up to 64K I/O queues, with each queue
supporting up to 64K commands
4. Simplified command decoding and processing with
fixed size (64B) command format
5. All information to complete a 4KB read request is
included in the 64B command itself, ensuring
efficient small I/O operation
6. Support for 2k MSI-X interrupts or 32 multiple
message MSI
7. Support for simple and efficient Interrupt
Aggregation.
IV. N
ON VOLATILE MEMORY SUB-SYSTEM
Fig2. NVM Sub-system
The present developed Non Volatile Memory Sub-system
consists of a PCIe Core, PCIe controller, NVMe Controller
and NAND Flash Controller to access the NAND Flash Chips.
The Sub-system is shown in Figure 2.

2014 International Conference on Recent Trends in Information Technology
978-1-4799-4989-2/14/$31.00 © 2014 IEEE
The Xilinx 1-lane Integrated Endpoint Block was chosen as
the PCI Express Core Architecture. It provides a user interface
of width 32 bits. Link speeds of up to 2.5 GB/s is supported.
The core provides PCIe Express Base Specification compliant
with v1.1. The detail about the core can be found in the
Endpoint Block User Guide [2]. A PCIe controller was
developed as a generic interface between any PCIe device and
the Core. It also provides a device interface of width 32 bits.
The NVMe controller is implemented as a PCIe device. It
provides a 32 bit interface on both PCIe side and NAND Flash
Controller side. The controller is compliant to NVM
Express1.0c Specification. An NVMe compliant NAND Flash
Controller was developed in order to process the NAND Flash
Commands. It provides a 32 bit interface on NVMe side and
supports 16 bit interface to NAND Flash chips. Specifically
Micron’s MT29F2G16A NAND Flash chips are targeted.
V. B
LUESPEC SYSTEM VERILOG LANGUAGE BACKGROUND
BlueSpec System Verilog is a Hardware centric language
based on industry standard System Verilog and Verilog [3]. It
uses Rules and interface methods for behavioral description,
which adds a powerful way to express complex concurrency
and control:
a. Across multiple shared resources.
b. Across module boundaries.
Parallelization
a. Concurrent behavior is expressed implicitly.
b. It has a traditional hardware semantic model of
cooperating FSMs.
c. Rules express concurrent operations by simply
describing the conditions under whichthe state
element(s) are updated.
d. Rules are implemented as un-sequenced atomic
transactions.
e. Compiler introduces the scheduling and the mixing
for the shared resources.
Level of Abstraction
a. BSV provides significantly higher level of abstraction
than Verilog, System Verilog, VHDL and SystemC.
b. It provides Behavioral description through: Rules and
Interface Methods.
c. Various attributes for Structural description are:
1. High level abstraction types.
2. Powerful Static checking.
3. Powerful parameterization.
4. Powerful Static elaboration.
Standalone Function Libraries
BSV has a large set of function libraries, some of them are
provided below.
a. It has Data Containers such as FIFO, registers, BRAMs
etc...
b. Circuits such LFSR and completion buffer [4].
c. Interface types, GET, PUT transactors.
d. Multiple Clock domain circuits such as synchronizers.
e. Bus interfaces such as common data bus, Z-bus etc...
VI. NVM
EXPRESS CONTROLLER
All the steps and structures mentioned here are as per the
NVM Express 1.0c Specification [6].
A. NVMe Structure.
The basic structure of the NVM is given in the Figure 3. This
is structured under the reference of specification
Fig3. NVMe Structure
B. Queuing Interface
The basic steps involved in queuing interface are
Command Submission, Command Processing and Command
Completion. Where,
Command Submission involves:
1. Host writes command to submission queue.
2. Host write updated submission queue tail pointer to
doorbell.
Command Processing
3. Controller fetches command.
4. Controller processes commend.
Command Completion
5. Controller writes completion to completion queue
6. Controller generates MSI-X interrupt
7. Host processer completion.
8. Host write updated completion queue head pointer to
doorbell.
C. Module Description
The entire design of the controller is divided into seven
logical modules as shown in Figure 5. Six state machines are
defined to assist the modules in deriving their respective
functionality. State machines are highlighted in italics.
VII. D
ESIGN CHALLENGES
Some of the challenges faced design decisions taken
during the design are explained below. This section will be
very helpful while improving the features of the present
controller for future use.

2014 International Conference on Recent Tre
n
978-1-4799-4989-2/14/$31.00 © 2014 IEEE
Fig4. Queuing interface
Fig5. NVMe Modules and State Mach
i
Multiple Requests to PCIe controller
NVMe controller: Because of the parallel
Fetch, Command Execution and Register Fil
p
ossibility for simultaneous requests for rea
d
command, write completion and send co
m
order to resolve this issue a PCIe Request S
t
developed which responds to one of the req
u
of assigned priority. The decision to assi
g
priority was taken with a view to provide
complete existing commands than to acquir
e
for execution. The order is as follows:
w
>completion TLP> write data > read data > r
e
Sequence of operations for Com
m
Command Arbitration: Commands can be
f
arbitrated or first arbitrated and then fetched.
are fetched into a buffer and then arbitrated
additional buffers to store the fetched comm
a
they are arbitrated and fetched, then the nu
m
b
uffer spaces for storing the commands is
m
this option was chosen. Abort Command Li
m
Command has to be checked if it is to be ab
o
comparison adds a lot of hardware and
h
number of outstanding commands to be abor
t
support five outstanding commands is taken.
n
ds in Information Technology
i
nes
from within the
execution of the
e units, there is a
d
-write data, read
m
pletion TLP. In
t
ate Machine was
u
ests on the basis
g
n the following
more priority to
e
new commands
w
rite completion
e
ad command.
m
and Fetch and
fetched and then
If the commands
, then we require
a
nds. However, if
m
ber of additional
m
inimized. Hence
m
its: Every NVM
o
rted or not. This
hence limits the
t
ed. A decision to
VIII. V
ERIFIC
A
The entire verification of the co
n
the “BlueSim” simulator. The N
V
integrated with NAND Flash
C
developed in BlueSpec, and the
w
The test setup used and the verified
the following sub-sections.
Fig6. Verificati
o
A. Verification Setup
The verification setup for verifyin
g
shown in the Figure 6.
NVM EXPRESS CONTROLL
E
controller, which is the Design Und
e
NAND FLASH CONTROLLER:
T
integrated with the NAND Flash C
o
N
VMe Controller’s functionality. T
h
is a fully developed system with th
e
System Verilog.
NAND FLASH TARGET: This is
in BSV to simulate the function
a
CHIPS”, with ONFI Interface.
PCIe MODEL: This is a function
a
abstract the functionality of the PC
I
Core in the form of Interface level
data buffers which are used to stor
e
N
VMe. This is used to verify
operation. It also has a read buffe
r
some random data at the reset of th
e
to verify the correctness of the Rea
d
TEST CASE COMMANDS: Thi
s
BSV, which stores the required co
m
This is accessed just as the Host M
a
commands from the submission que
B. Verification Test Cases
The controller has been veri
fi
cases:
1) Test cases for the I/O c
o
Write and Read command
s
2) Test cases for all the
c
Command set.
3) Test case to verify th
e
responses.
4) Test case to verify t
h
responses. This includes
and “Normal Shutdown”.
A
TION
n
troller was carried out in
V
M express controller was
C
ontroller, which is also
w
hole system was verified.
test cases are described in
o
n Setup
g
the NVMe controller is
E
R: This is the present
e
r Verification (DUV).
T
he NVMe controller was
o
ntroller, in order to verify
h
e NAND Flash Controller
e
source code in BlueSpec
a functional model written
a
lity of “NAND FLASH
a
l model written in BSV to
I
e Controller and the PCIe
Transactions. It has write
e
the data coming from the
the correctness of write
r
, which is initialized with
e
system. This can be used
d
command.
s
is the memory model in
m
mands for NVM Express.
a
in Memory is accessed for
ues.
fi
ed for the following test
o
mmand set, that includes
.
c
ommands in the Admin
Controller Initialization
h
e Controller Shutdown
both “Abrupt Shutdown”

2014 International Conference on Recent Trends in Information Technology
978-1-4799-4989-2/14/$31.00 © 2014 IEEE
5) Test case to verify the Read-Write operation to the
Register File.
6) Test case to verify the “Arbitration Mechanism”.
7) Test cases to verify interrupt mechanism. This
includes verification for Interrupt Aggregation for
Successful I/O Commands.
8) Test cases to verify that the Aggregation is not
applied to Admin Commands and to I/O commands
that complete in error.
9) Test case for abort command with “Abort Command
Limit Exceeded” error.
10) Creation of I/O CQ with:
I. Invalid Q ID.
II. Queue size greater than Max Queue size
exceeded.
III. Invalid Interrupt vector.
11) Creation of I/O CQ with:
I. Invalid Q ID.
II. Non-existent associated CQ.
III. Max Queue size exceeds.
12) Delete a CQ that does not exist.
13) Delete a SQ that does not exist.
C. Synthesis Report
The design was synthesized for the device Virtex-6
XC6VLX720T-FF1156. The slice utilization and timing
summary is provided below:
Number of Slice Registers used: 4773
Number of Slice LUTs used: 6803
Number of LUT Flip Flop Pairs used: 2851
Number of Block RAMs used: 13
Max Frequency of operation: 204 MHz
D. Net-list and SDF
The net-list and the SDF for the design are taken out in
Synopsys Design Vision with 180nm technology library from
Faraday Technology Corporation. These files are taken to the
backend development tools for the ASIC implementation
E. Backend Design
The backend design started with importing the net list,
libraries namely
,“fsd0a_generic_core, header6m15_V55,
FSD0A_GENERIC_CORE_ANT_V55.6m15”
and the constraints file
defined for clock and the IO Ports, where clock is defined as,
“create_clock -name CLK -period 10 -waveform {0 5}
[get_ports "CLK"]
set_clock_transition -rise 0.1 [get_clocks "CLK"]
set_clock_transition -fall 0.1 [get_clocks "CLK"]
set_clock_uncertainty 1.0 [get_ports "CLK"]”.
With all the above contributions the design reports the area as,
Number of ports: 977
Combinational area: 79661.456480
Non-combinational area: 47834.976524
Total cell area: 127496.433004
And the power reports Total power as 51.8846mW. As
compared to the privies design
[13] [
TABLE I
]
which is of SSD+HDD
hybrid models the current design power and area will be
fewer, as it is of pure NVMe based SSD. NVMe prototype
delivers lower clocks per I/O while at the same time delivering
higher performance on the workloads. A lower clock per I/O is
a proxy for efficiency and lower power – the CPU & system
can go to a sleep state more quickly
[14]
.
TABLE
I
P
OWER AND LATENCY COMPARISON
Design Max. Ideal
Power
Max. Power Max. Latency
SSD+HDD(SATA)
<=8mW 63mW <855mS
NVMe
<=5mW 52mW <500mS
IX. CONCLUSION
The NVM Express 1.0c Specification compliant controller
was designed with all the basic functionality and mandatory
command sets. The PCIe controller was designed as a bridge
between the Xilinx PCIe Endpoint Block and the NVMe
controller. Both the controllers were individually verified for
their functionality. PCIe controller was verified as a
standalone controller between any PCIe device and Core.
NVMe controller was verified with the implemented NAND
Flash controller and the Transactional model of PCIe core and
controller. Finally all the three controllers were integrated.
The core and the three controllers form a basic working Non-
Volatile Memory system.
Fig7. Proof Point: NVMe Efficiency & Power [14].
The ASIC implementation for the same has carried out with
the obtained Net-list and SDF.
The NVM subsystem can be improved in the following
ways:
1. NVMe command set can be enriched with some optional
commands, if required.
2. An FTL(Flash Translation Layer) processor can be added
to the subsystem to carryout
Flash Management activities at software level [7]. A brief
detail about FTL and its necessity is provided in the appendix.

2014 International Conference on Recent Trends in Information Technology
978-1-4799-4989-2/14/$31.00 © 2014 IEEE
A
CKNOWLEDGMENT
I would like to express my sincere gratitude and thanks to
Dr. G.S. Madhusudan (RISE Lab IIT Madras) who has
provided me with lots of ideas and suggestions. I am very
much thankful for the valuable inputs that he has provided
through the Industry experts.
R
EFERENCES
[1] J. S. Adam Wilen and R. Thornburg, “Introduction to PCI Express:
Hardware and Software Developer’s Guide”. Intel Press, 2003.
[2] Xilinx, “Spartan-6 FPGA Integrated Endpoint Block for PCI Express”,
ug672 ed., January 18, 2012.
[3] Bluespec, Inc, “Bluespec System Verilog Reference Guide”, 2012.
[4] R. S. Nikhil and K. Czeck, “BSV by Example”. Bluespec, Inc, 2010.
[5] “PCI express base specification” April 29, 2002.
[6] A. Huffman, “NVM express 1.0c” February 16, 2012.
[7] LC. RinoMicheloni, A.Marelli, “Inside NAND Flash Memories”,
Springer, 2010.
[8] Walker, Don H. “A Comparison of NVMe and AHCI”, 31 July 2012.
The Serial ATA International Organization. Retrieved 3 July 2013.
[9] “New Promoter Group Formed to Advance NVM Express”, Press
release. June 1, 2011. Retrieved September 18, 2013.
[10] Amber Huffman editor (October 11, 2012). “NVM Express Revision
1.1”, Specification. Retrieved September 18, 2013.
[11] Amber Huffman editor (January 23, 2013). “NVM Express Revision
1.0e”. Specification. Retrieved September 18, 2013.
[12] Dave Landsman. “AHCI and NVMe as Interfaces for SATA Express™
Devices – Overview” (PDF). SanDisk. Retrieved 2013-10-02.
[13] Amber Huffman and Dale Juenemann, “The Nonvolatile Memory
Transformation of Client Storage”, Published by the IEEE Computer
Society 0018-9162/13 2013 IEEE.
[14] Danny Cobb, CTO Flash Memory Business Unit, EMC. Amber
Huffman, Sr. Principal Engineer, Intel “NVM Express and the PCI
Express* SSD Revolution” SSDS003, IDF2012.