docs: nvdimm: convert to ReST

=============================

BTT - Block Translation Table

=============================

1. Introduction

---------------

===============

Persistent memory based storage is able to perform IO at byte (or more

accurately, cache line) granularity. However, we often want to expose such

2. Static Layout

----------------

================

The underlying storage on which a BTT can be laid out is not limited in any way.

The BTT, however, splits the available space into chunks of up to 512 GiB,

Each arena follows the same layout for its metadata, and all references in an

arena are internal to it (with the exception of one field that points to the

next arena). The following depicts the "On-disk" metadata layout:

next arena). The following depicts the "On-disk" metadata layout::

    Backing Store     +------->  Arena

3. Theory of Operation

----------------------

======================

a. The BTT Map

block. Each map entry is 32 bits. The two most significant bits are special

flags, and the remaining form the internal block number.

======== =============================================================

Bit      Description

31 - 30	: Error and Zero flags - Used in the following way:

	 Bit		      Description

	31 30

	-----------------------------------------------------------------------

======== =============================================================

31 - 30	 Error and Zero flags - Used in the following way:

	   == ==  ====================================================

	   31 30  Description

	   == ==  ====================================================

	   0  0	  Initial state. Reads return zeroes; Premap = Postmap

	   0  1	  Zero state: Reads return zeroes

	   1  0	  Error state: Reads fail; Writes clear 'E' bit

	   1  1	  Normal Block – has valid postmap

	   == ==  ====================================================

29 - 0	: Mappings to internal 'postmap' blocks

29 - 0	 Mappings to internal 'postmap' blocks

======== =============================================================

Some of the terminology that will be subsequently used:

External LBA  : LBA as made visible to upper layers.

ABA           : Arena Block Address - Block offset/number within an arena

Premap ABA    : The block offset into an arena, which was decided upon by range

============	================================================================

External LBA	LBA as made visible to upper layers.

ABA		Arena Block Address - Block offset/number within an arena

Premap ABA	The block offset into an arena, which was decided upon by range

		checking the External LBA

Postmap ABA   : The block number in the "Data Blocks" area obtained after

Postmap ABA	The block number in the "Data Blocks" area obtained after

		indirection from the map

nfree	      : The number of free blocks that are maintained at any given time.

nfree		The number of free blocks that are maintained at any given time.

		This is the number of concurrent writes that can happen to the

		arena.

============	================================================================

For example, after adding a BTT, we surface a disk of 1024G. We get a read for

maintained in the form of the BTT flog. 'Flog' is a combination of the words

"free list" and "log". The flog contains 'nfree' entries, and an entry contains:

lba     : The premap ABA that is being written to

old_map : The old postmap ABA - after 'this' write completes, this will be a

========  =====================================================================

lba       The premap ABA that is being written to

old_map   The old postmap ABA - after 'this' write completes, this will be a

	  free block.

new_map : The new postmap ABA. The map will up updated to reflect this

new_map   The new postmap ABA. The map will up updated to reflect this

	  lba->postmap_aba mapping, but we log it here in case we have to

	  recover.

seq	: Sequence number to mark which of the 2 sections of this flog entry is

seq	  Sequence number to mark which of the 2 sections of this flog entry is

	  valid/newest. It cycles between 01->10->11->01 (binary) under normal

	  operation, with 00 indicating an uninitialized state.

lba'	: alternate lba entry

old_map': alternate old postmap entry

new_map': alternate new postmap entry

seq'	: alternate sequence number.

lba'	  alternate lba entry

old_map'  alternate old postmap entry

new_map'  alternate new postmap entry

seq'	  alternate sequence number.

========  =====================================================================

Each of the above fields is 32-bit, making one entry 32 bytes. Entries are also

padded to 64 bytes to avoid cache line sharing or aliasing. Flog updates are

While 'nfree' describes the number of concurrent IOs an arena can process

concurrently, 'nlanes' is the number of IOs the BTT device as a whole can

process.

process::

	nlanes = min(nfree, num_cpus)

A lane number is obtained at the start of any IO, and is used for indexing into

all the on-disk and in-memory data structures for the duration of the IO. If

there are more CPUs than the max number of available lanes, than lanes are

--------------------------------------

Consider a case where two writer threads are writing to the same LBA. There can

be a race in the following sequence of steps:

be a race in the following sequence of steps::

	free[lane] = map[premap_aba]

	map[premap_aba] = postmap_aba

through all the entries, and for each lane, of the set of two possible

'sections', we always look at the most recent one only (based on the sequence

number). The reconstruction rules/steps are simple:

- Read map[log_entry.lba].

- If log_entry.new matches the map entry, then log_entry.old is free.

- If log_entry.new does not match the map entry, then log_entry.new is free.

An arena would be in an error state if any of the metadata is corrupted

irrecoverably, either due to a bug or a media error. The following conditions

indicate an error:

- Info block checksum does not match (and recovering from the copy also fails)

- All internal available blocks are not uniquely and entirely addressed by the

  sum of mapped blocks and free blocks (from the BTT flog).

(pmem, or blk mode). The easiest way to set up such a namespace is using the

'ndctl' utility [1]:

For example, the ndctl command line to setup a btt with a 4k sector size is:

For example, the ndctl command line to setup a btt with a 4k sector size is::

    ndctl create-namespace -f -e namespace0.0 -m sector -l 4k

See ndctl create-namespace --help for more options.

[1]: https://github.com/pmem/ndctl

:orphan:

===================================

Non-Volatile Memory Device (NVDIMM)

===================================

.. toctree::

   :maxdepth: 1

   nvdimm

   btt

   security

===============================

LIBNVDIMM: Non-Volatile Devices

===============================

libnvdimm - kernel / libndctl - userspace helper library

linux-nvdimm@lists.01.org

				      v13

Version 13

.. contents:

	Glossary

	Overview

Glossary

--------

========

PMEM: A system-physical-address range where writes are persistent.  A

PMEM:

  A system-physical-address range where writes are persistent.  A

  block device composed of PMEM is capable of DAX.  A PMEM address range

  may span an interleave of several DIMMs.

BLK: A set of one or more programmable memory mapped apertures provided

BLK:

  A set of one or more programmable memory mapped apertures provided

  by a DIMM to access its media.  This indirection precludes the

  performance benefit of interleaving, but enables DIMM-bounded failure

  modes.

DPA: DIMM Physical Address, is a DIMM-relative offset.  With one DIMM in

DPA:

  DIMM Physical Address, is a DIMM-relative offset.  With one DIMM in

  the system there would be a 1:1 system-physical-address:DPA association.

  Once more DIMMs are added a memory controller interleave must be

  decoded to determine the DPA associated with a given

  system-physical-address.  BLK capacity always has a 1:1 relationship

  with a single-DIMM's DPA range.

DAX: File system extensions to bypass the page cache and block layer to

DAX:

  File system extensions to bypass the page cache and block layer to

  mmap persistent memory, from a PMEM block device, directly into a

  process address space.

DSM: Device Specific Method: ACPI method to to control specific

DSM:

  Device Specific Method: ACPI method to to control specific

  device - in this case the firmware.

DCR: NVDIMM Control Region Structure defined in ACPI 6 Section 5.2.25.5.

DCR:

  NVDIMM Control Region Structure defined in ACPI 6 Section 5.2.25.5.

  It defines a vendor-id, device-id, and interface format for a given DIMM.

BTT: Block Translation Table: Persistent memory is byte addressable.

BTT:

  Block Translation Table: Persistent memory is byte addressable.

  Existing software may have an expectation that the power-fail-atomicity

  of writes is at least one sector, 512 bytes.  The BTT is an indirection

  table with atomic update semantics to front a PMEM/BLK block device

  driver and present arbitrary atomic sector sizes.

LABEL: Metadata stored on a DIMM device that partitions and identifies

LABEL:

  Metadata stored on a DIMM device that partitions and identifies

  (persistently names) storage between PMEM and BLK.  It also partitions

  BLK storage to host BTTs with different parameters per BLK-partition.

  Note that traditional partition tables, GPT/MBR, are layered on top of a

Overview

--------

========

The LIBNVDIMM subsystem provides support for three types of NVDIMMs, namely,

PMEM, BLK, and NVDIMM devices that can simultaneously support both PMEM

for exclusive access via one mode a time.

Supporting Documents

ACPI 6: http://www.uefi.org/sites/default/files/resources/ACPI_6.0.pdf

NVDIMM Namespace: http://pmem.io/documents/NVDIMM_Namespace_Spec.pdf

DSM Interface Example: http://pmem.io/documents/NVDIMM_DSM_Interface_Example.pdf

Driver Writer's Guide: http://pmem.io/documents/NVDIMM_Driver_Writers_Guide.pdf

--------------------

ACPI 6:

	http://www.uefi.org/sites/default/files/resources/ACPI_6.0.pdf

NVDIMM Namespace:

	http://pmem.io/documents/NVDIMM_Namespace_Spec.pdf

DSM Interface Example:

	http://pmem.io/documents/NVDIMM_DSM_Interface_Example.pdf

Driver Writer's Guide:

	http://pmem.io/documents/NVDIMM_Driver_Writers_Guide.pdf

Git Trees

LIBNVDIMM: https://git.kernel.org/cgit/linux/kernel/git/djbw/nvdimm.git

LIBNDCTL: https://github.com/pmem/ndctl.git

PMEM: https://github.com/01org/prd

---------

LIBNVDIMM:

	https://git.kernel.org/cgit/linux/kernel/git/djbw/nvdimm.git

LIBNDCTL:

	https://github.com/pmem/ndctl.git

PMEM:

	https://github.com/01org/prd

LIBNVDIMM PMEM and BLK

------------------

======================

Prior to the arrival of the NFIT, non-volatile memory was described to a

system in various ad-hoc ways.  Usually only the bare minimum was

       the spec also allows for vendor specific layouts, and non-NFIT BLK

       implementations may have other designs for BLK I/O.  For this reason

       "nd_blk" calls back into platform-specific code to perform the I/O.

       One such implementation is defined in the "Driver Writer's Guide" and "DSM

       Interface Example".

Why BLK?

--------

========

While PMEM provides direct byte-addressable CPU-load/store access to

NVDIMM storage, it does not provide the best system RAS (recovery,

to a corrupted address through an BLK-aperture causes that block window

to raise an error status in a register.  The latter is more aligned with

the standard error model that host-bus-adapter attached disks present.

Also, if an administrator ever wants to replace a memory it is easier to

service a system at DIMM module boundaries.  Compare this to PMEM where

data could be interleaved in an opaque hardware specific manner across

several DIMMs.

PMEM vs BLK

-----------

BLK-apertures solve these RAS problems, but their presence is also the

major contributing factor to the complexity of the ND subsystem.  They

complicate the implementation because PMEM and BLK alias in DPA space.

extents.

BLK-REGIONs, PMEM-REGIONs, Atomic Sectors, and DAX

--------------------------------------------------

^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^

One of the few

reasons to allow multiple BLK namespaces per REGION is so that each

BLK-namespace can be configured with a BTT with unique atomic sector

sizes.  While a PMEM device can host a BTT the LABEL specification does

not provide for a sector size to be specified for a PMEM namespace.

This is due to the expectation that the primary usage model for PMEM is

via DAX, and the BTT is incompatible with DAX.  However, for the cases

where an application or filesystem still needs atomic sector update

Example NVDIMM Platform

-----------------------

=======================

For the remainder of this document the following diagram will be

referenced for any example sysfs layouts.

referenced for any example sysfs layouts::

                               (a)               (b)           DIMM   BLK-REGION

LIBNVDIMM Kernel Device Model and LIBNDCTL Userspace API

----------------------------------------------------

========================================================

What follows is a description of the LIBNVDIMM sysfs layout and a

corresponding object hierarchy diagram as viewed through the LIBNDCTL

test.

LIBNDCTL: Context

-----------------

Every API call in the LIBNDCTL library requires a context that holds the

logging parameters and other library instance state.  The library is

based on the libabc template:

	https://git.kernel.org/cgit/linux/kernel/git/kay/libabc.git

LIBNDCTL: instantiate a new library context example

^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^

::

	struct ndctl_ctx *ctx;

		return NULL;

LIBNVDIMM/LIBNDCTL: Bus

-------------------

-----------------------

A bus has a 1:1 relationship with an NFIT.  The current expectation for

ACPI based systems is that there is only ever one platform-global NFIT.

test.

LIBNVDIMM: control class device in /sys/class

---------------------------------------------

This character device accepts DSM messages to be passed to DIMM

identified by its NFIT handle.

identified by its NFIT handle::

	/sys/class/nd/ndctl0

	|-- dev

LIBNVDIMM: bus

--------------

::

	struct nvdimm_bus *nvdimm_bus_register(struct device *parent,

	       struct nvdimm_bus_descriptor *nfit_desc);

::

	/sys/devices/platform/nfit_test.0/ndbus0

	|-- commands

	|-- nd

	`-- wait_probe

LIBNDCTL: bus enumeration example

Find the bus handle that describes the bus from Example NVDIMM Platform

^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^

Find the bus handle that describes the bus from Example NVDIMM Platform::

	static struct ndctl_bus *get_bus_by_provider(struct ndctl_ctx *ctx,

			const char *provider)

LIBNVDIMM/LIBNDCTL: DIMM (NMEM)

---------------------------

-------------------------------

The DIMM device provides a character device for sending commands to

hardware, and it is a container for LABELs.  If the DIMM is defined by

be physical DIMMs, so we use a more generic name.

LIBNVDIMM: DIMM (NMEM)

^^^^^^^^^^^^^^^^^^^^^^

::

	struct nvdimm *nvdimm_create(struct nvdimm_bus *nvdimm_bus, void *provider_data,

			const struct attribute_group **groups, unsigned long flags,

			unsigned long *dsm_mask);

::

	/sys/devices/platform/nfit_test.0/ndbus0

	|-- nmem0

	|   |-- available_slots

LIBNDCTL: DIMM enumeration example

^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^

Note, in this example we are assuming NFIT-defined DIMMs which are

identified by an "nfit_handle" a 32-bit value where:

Bit 3:0 DIMM number within the memory channel

Bit 7:4 memory channel number

Bit 11:8 memory controller ID

Bit 15:12 socket ID (within scope of a Node controller if node controller is present)

Bit 27:16 Node Controller ID

Bit 31:28 Reserved

   - Bit 3:0 DIMM number within the memory channel

   - Bit 7:4 memory channel number

   - Bit 11:8 memory controller ID

   - Bit 15:12 socket ID (within scope of a Node controller if node

     controller is present)

   - Bit 27:16 Node Controller ID

   - Bit 31:28 Reserved

::

	static struct ndctl_dimm *get_dimm_by_handle(struct ndctl_bus *bus,

	       unsigned int handle)

	dimm = get_dimm_by_handle(bus, DIMM_HANDLE(0, 0, 0, 0, 0));

LIBNVDIMM/LIBNDCTL: Region

----------------------

--------------------------

A generic REGION device is registered for each PMEM range or BLK-aperture

set.  Per the example there are 6 regions: 2 PMEM and 4 BLK-aperture

at the 'add' event, and finally, the optional "spa_index" is provided in

the case where the region is defined by a SPA.

LIBNVDIMM: region

LIBNVDIMM: region::

	struct nd_region *nvdimm_pmem_region_create(struct nvdimm_bus *nvdimm_bus,

			struct nd_region_desc *ndr_desc);

	struct nd_region *nvdimm_blk_region_create(struct nvdimm_bus *nvdimm_bus,

			struct nd_region_desc *ndr_desc);

::

	/sys/devices/platform/nfit_test.0/ndbus0

	|-- region0

	|   |-- available_size

	[..]

LIBNDCTL: region enumeration example

^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^

Sample region retrieval routines based on NFIT-unique data like

"spa_index" (interleave set id) for PMEM and "nfit_handle" (dimm id) for

BLK.

BLK::

	static struct ndctl_region *get_pmem_region_by_spa_index(struct ndctl_bus *bus,

			unsigned int spa_index)

looking at the kernel header (/usr/include/linux/ndctl.h) to decode the

"nstype" integer attribute, here are some other options.

    1. module alias lookup:

1. module alias lookup

^^^^^^^^^^^^^^^^^^^^^^

    The whole point of region/namespace device type differentiation is to

    decide which block-device driver will attach to a given LIBNVDIMM namespace.

    the resulting namespaces.  The output from module resolution is more

    accurate than a region-name or region-devtype.

    2. udev:

2. udev

^^^^^^^

    The kernel "devtype" is registered in the udev database::

    The kernel "devtype" is registered in the udev database

	# udevadm info --path=/devices/platform/nfit_test.0/ndbus0/region0

	P: /devices/platform/nfit_test.0/ndbus0/region0

	E: DEVPATH=/devices/platform/nfit_test.0/ndbus0/region0

    "devtype" does not indicate sub-type variations and scripts should

    really be understanding the other attributes.

    3. type specific attributes:

3. type specific attributes

^^^^^^^^^^^^^^^^^^^^^^^^^^^

    As it currently stands a BLK-aperture region will never have a

    "nfit/spa_index" attribute, but neither will a non-NFIT PMEM region.  A

LIBNVDIMM/LIBNDCTL: Namespace

-------------------------

-----------------------------

A REGION, after resolving DPA aliasing and LABEL specified boundaries,

surfaces one or more "namespace" devices.  The arrival of a "namespace"

and register a disk/block device.

LIBNVDIMM: namespace

^^^^^^^^^^^^^^^^^^^^

Here is a sample layout from the three major types of NAMESPACE where

namespace0.0 represents DIMM-info-backed PMEM (note that it has a 'uuid'

attribute), namespace2.0 represents a BLK namespace (note it has a

'sector_size' attribute) that, and namespace6.0 represents an anonymous

PMEM namespace (note that has no 'uuid' attribute due to not support a

LABEL).

LABEL)::

	/sys/devices/platform/nfit_test.0/ndbus0/region0/namespace0.0

	|-- alt_name

	`-- uevent

LIBNDCTL: namespace enumeration example

^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^

Namespaces are indexed relative to their parent region, example below.

These indexes are mostly static from boot to boot, but subsystem makes

no guarantees in this regard.  For a static namespace identifier use its

'uuid' attribute.

static struct ndctl_namespace *get_namespace_by_id(struct ndctl_region *region,

                unsigned int id)

::

  static struct ndctl_namespace

  *get_namespace_by_id(struct ndctl_region *region, unsigned int id)

  {

          struct ndctl_namespace *ndns;

  }

LIBNDCTL: namespace creation example

^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^

Idle namespaces are automatically created by the kernel if a given

region has enough available capacity to create a new namespace.

Namespace instantiation involves finding an idle namespace and

configuring it.  For the most part the setting of namespace attributes

can occur in any order, the only constraint is that 'uuid' must be set

before 'size'.  This enables the kernel to track DPA allocations

internally with a static identifier.

internally with a static identifier::

  static int configure_namespace(struct ndctl_region *region,

                  struct ndctl_namespace *ndns,

Why the Term "namespace"?

^^^^^^^^^^^^^^^^^^^^^^^^^

    1. Why not "volume" for instance?  "volume" ran the risk of confusing

       ND (libnvdimm subsystem) to a volume manager like device-mapper.

LIBNVDIMM/LIBNDCTL: Block Translation Table "btt"

---------------------------------------------

-------------------------------------------------

A BTT (design document: http://pmem.io/2014/09/23/btt.html) is a stacked

block device driver that fronts either the whole block device or a

partition of a block device emitted by either a PMEM or BLK NAMESPACE.

LIBNVDIMM: btt layout

^^^^^^^^^^^^^^^^^^^^^

Every region will start out with at least one BTT device which is the

seed device.  To activate it set the "namespace", "uuid", and

"sector_size" attributes and then bind the device to the nd_pmem or

nd_blk driver depending on the region type.

nd_blk driver depending on the region type::

	/sys/devices/platform/nfit_test.1/ndbus0/region0/btt0/

	|-- namespace

	`-- uuid

LIBNDCTL: btt creation example

^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^

Similar to namespaces an idle BTT device is automatically created per

region.  Each time this "seed" btt device is configured and enabled a new

seed is created.  Creating a BTT configuration involves two steps of

finding and idle BTT and assigning it to consume a PMEM or BLK namespace.

finding and idle BTT and assigning it to consume a PMEM or BLK namespace::

	static struct ndctl_btt *get_idle_btt(struct ndctl_region *region)

	{

------------------------

For the given example above, here is the view of the objects as seen by the

LIBNDCTL API:

LIBNDCTL API::

              +---+

              |CTX|    +---------+   +--------------+  +---------------+

              +-+-+  +-> REGION0 +---> NAMESPACE0.0 +--> PMEM8 "pm0.0" |

                     | +---------+   +--------------+  +----------------------+

                     +-> REGION5 +---> NAMESPACE5.0 +--> ND1  "blk5.0" | BTT0 |

                       +---------+   +--------------+  +---------------+------+

NVDIMM SECURITY

===============

NVDIMM Security

===============

1. Introduction

by the extended security status.

[1]: http://pmem.io/documents/NVDIMM_DSM_Interface-V1.8.pdf

[2]: http://www.t13.org/documents/UploadedDocuments/docs2006/e05179r4-ACS-SecurityClarifications.pdf

	  Documentation/admin-guide/kernel-parameters.rst).  This driver converts

	  these persistent memory ranges into block devices that are

	  capable of DAX (direct-access) file system mappings.  See

	  Documentation/nvdimm/nvdimm.txt for more details.

	  Documentation/nvdimm/nvdimm.rst for more details.

	  Say Y if you want to use an NVDIMM