docs: Update pvrdma device documentation

modifications.

While it complies with the VMware device, it can also communicate with bare

metal RDMA-enabled machines and does not require an RDMA HCA in the host, it

can work with Soft-RoCE (rxe).

metal RDMA-enabled machines as peers.

It does not require an RDMA HCA in the host, it can work with Soft-RoCE (rxe).

It does not require the whole guest RAM to be pinned allowing memory

over-commit and, even if not implemented yet, migration support will be

3. Usage

========

3.1 VM Memory settings

======================

Currently the device is working only with memory backed RAM

and it must be mark as "shared":

   -m 1G \

   -object memory-backend-ram,id=mb1,size=1G,share \

   -numa node,memdev=mb1 \

The pvrdma device is composed of two functions:

 - Function 0 is a vmxnet Ethernet Device which is redundant in Guest

   but is required to pass the ibdevice GID using its MAC.

   Examples:

     For an rxe backend using eth0 interface it will use its mac:

       -device vmxnet3,addr=<slot>.0,multifunction=on,mac=<eth0 MAC>

     For an SRIOV VF, we take the Ethernet Interface exposed by it:

       -device vmxnet3,multifunction=on,mac=<RoCE eth MAC>

 - Function 1 is the actual device:

       -device pvrdma,addr=<slot>.1,backend-dev=<ibdevice>,backend-gid-idx=<gid>,backend-port=<port>

   where the ibdevice can be rxe or RDMA VF (e.g. mlx5_4)

 Note: Pay special attention that the GID at backend-gid-idx matches vmxnet's MAC.

 The rules of conversion are part of the RoCE spec, but since manual conversion

 is not required, spotting problems is not hard:

    Example: GID: fe80:0000:0000:0000:7efe:90ff:fecb:743a

             MAC: 7c:fe:90:cb:74:3a

    Note the difference between the first byte of the MAC and the GID.

3.2 MAD Multiplexer

===================

MAD Multiplexer is a service that exposes MAD-like interface for VMs in

order to overcome the limitation where only single entity can register with

MAD layer to send and receive RDMA-CM MAD packets.

To build rdmacm-mux run

# make rdmacm-mux

The application accepts 3 command line arguments and exposes a UNIX socket

to pass control and data to it.

-d rdma-device-name  Name of RDMA device to register with

-s unix-socket-path  Path to unix socket to listen (default /var/run/rdmacm-mux)

-p rdma-device-port  Port number of RDMA device to register with (default 1)

The final UNIX socket file name is a concatenation of the 3 arguments so

for example for device mlx5_0 on port 2 this /var/run/rdmacm-mux-mlx5_0-2

will be created.

pvrdma requires this service.

Please refer to contrib/rdmacm-mux for more details.

3.3 Service exposed by libvirt daemon

=====================================

The control over the RDMA device's GID table is done by updating the

device's Ethernet function addresses.

Usually the first GID entry is determined by the MAC address, the second by

the first IPv6 address and the third by the IPv4 address. Other entries can

be added by adding more IP addresses. The opposite is the same, i.e.

whenever an address is removed, the corresponding GID entry is removed.

The process is done by the network and RDMA stacks. Whenever an address is

added the ib_core driver is notified and calls the device driver add_gid

function which in turn update the device.

To support this in pvrdma device the device hooks into the create_bind and

destroy_bind HW commands triggered by pvrdma driver in guest.

Whenever changed is made to the pvrdma port's GID table a special QMP

messages is sent to be processed by libvirt to update the address of the

backend Ethernet device.

pvrdma requires that libvirt service will be up.

3.4 PCI devices settings

========================

RoCE device exposes two functions - an Ethernet and RDMA.

To support it, pvrdma device is composed of two PCI functions, an Ethernet

device of type vmxnet3 on PCI slot 0 and a PVRDMA device on PCI slot 1. The

Ethernet function can be used for other Ethernet purposes such as IP.

3.5 Device parameters

=====================

- netdev: Specifies the Ethernet device function name on the host for

  example enp175s0f0. For Soft-RoCE device (rxe) this would be the Ethernet

  device used to create it.

- ibdev: The IB device name on host for example rxe0, mlx5_0 etc.

- mad-chardev: The name of the MAD multiplexer char device.

- ibport: In case of multi-port device (such as Mellanox's HCA) this

  specify the port to use. If not set 1 will be used.

- dev-caps-max-mr-size: The maximum size of MR.

- dev-caps-max-qp:      Maximum number of QPs.

- dev-caps-max-sge:     Maximum number of SGE elements in WR.

- dev-caps-max-cq:      Maximum number of CQs.

- dev-caps-max-mr:      Maximum number of MRs.

- dev-caps-max-pd:      Maximum number of PDs.

- dev-caps-max-ah:      Maximum number of AHs.

Notes:

- The first 3 parameters are mandatory settings, the rest have their

  defaults.

- The last 8 parameters (the ones that prefixed by dev-caps) defines the top

  limits but the final values is adjusted by the backend device limitations.

- netdev can be extracted from ibdev's sysfs

  (/sys/class/infiniband/<ibdev>/device/net/)

3.6 Example

===========

Define bridge device with vmxnet3 network backend:

<interface type='bridge'>

  <mac address='56:b4:44:e9:62:dc'/>

  <source bridge='bridge1'/>

  <model type='vmxnet3'/>

  <address type='pci' domain='0x0000' bus='0x00' slot='0x10' function='0x0' multifunction='on'/>

</interface>

Define pvrdma device:

<qemu:commandline>

  <qemu:arg value='-object'/>

  <qemu:arg value='memory-backend-ram,id=mb1,size=1G,share'/>

  <qemu:arg value='-numa'/>

  <qemu:arg value='node,memdev=mb1'/>

  <qemu:arg value='-chardev'/>

  <qemu:arg value='socket,path=/var/run/rdmacm-mux-rxe0-1,id=mads'/>

  <qemu:arg value='-device'/>

  <qemu:arg value='pvrdma,addr=10.1,ibdev=rxe0,netdev=bridge0,mad-chardev=mads'/>

</qemu:commandline>