Merge branch 'xdp: hints via kfuncs'

   xfrm_proc

   xfrm_sync

   xfrm_sysctl

   xdp-rx-metadata

.. only::  subproject and html

===============

XDP RX Metadata

===============

This document describes how an eXpress Data Path (XDP) program can access

hardware metadata related to a packet using a set of helper functions,

and how it can pass that metadata on to other consumers.

General Design

==============

XDP has access to a set of kfuncs to manipulate the metadata in an XDP frame.

Every device driver that wishes to expose additional packet metadata can

implement these kfuncs. The set of kfuncs is declared in ``include/net/xdp.h``

via ``XDP_METADATA_KFUNC_xxx``.

Currently, the following kfuncs are supported. In the future, as more

metadata is supported, this set will grow:

.. kernel-doc:: net/core/xdp.c

   :identifiers: bpf_xdp_metadata_rx_timestamp bpf_xdp_metadata_rx_hash

An XDP program can use these kfuncs to read the metadata into stack

variables for its own consumption. Or, to pass the metadata on to other

consumers, an XDP program can store it into the metadata area carried

ahead of the packet.

Not all kfuncs have to be implemented by the device driver; when not

implemented, the default ones that return ``-EOPNOTSUPP`` will be used.

Within an XDP frame, the metadata layout (accessed via ``xdp_buff``) is

as follows::

  +----------+-----------------+------+

  | headroom | custom metadata | data |

  +----------+-----------------+------+

             ^                 ^

             |                 |

   xdp_buff->data_meta   xdp_buff->data

An XDP program can store individual metadata items into this ``data_meta``

area in whichever format it chooses. Later consumers of the metadata

will have to agree on the format by some out of band contract (like for

the AF_XDP use case, see below).

AF_XDP

======

:doc:`af_xdp` use-case implies that there is a contract between the BPF

program that redirects XDP frames into the ``AF_XDP`` socket (``XSK``) and

the final consumer. Thus the BPF program manually allocates a fixed number of

bytes out of metadata via ``bpf_xdp_adjust_meta`` and calls a subset

of kfuncs to populate it. The userspace ``XSK`` consumer computes

``xsk_umem__get_data() - METADATA_SIZE`` to locate that metadata.

Note, ``xsk_umem__get_data`` is defined in ``libxdp`` and

``METADATA_SIZE`` is an application-specific constant (``AF_XDP`` receive

descriptor does _not_ explicitly carry the size of the metadata).

Here is the ``AF_XDP`` consumer layout (note missing ``data_meta`` pointer)::

  +----------+-----------------+------+

  | headroom | custom metadata | data |

  +----------+-----------------+------+

                               ^

                               |

                        rx_desc->address

XDP_PASS

========

This is the path where the packets processed by the XDP program are passed

into the kernel. The kernel creates the ``skb`` out of the ``xdp_buff``

contents. Currently, every driver has custom kernel code to parse

the descriptors and populate ``skb`` metadata when doing this ``xdp_buff->skb``

conversion, and the XDP metadata is not used by the kernel when building

``skbs``. However, TC-BPF programs can access the XDP metadata area using

the ``data_meta`` pointer.

In the future, we'd like to support a case where an XDP program

can override some of the metadata used for building ``skbs``.

bpf_redirect_map

================

``bpf_redirect_map`` can redirect the frame to a different device.

Some devices (like virtual ethernet links) support running a second XDP

program after the redirect. However, the final consumer doesn't have

access to the original hardware descriptor and can't access any of

the original metadata. The same applies to XDP programs installed

into devmaps and cpumaps.

This means that for redirected packets only custom metadata is

currently supported, which has to be prepared by the initial XDP program

before redirect. If the frame is eventually passed to the kernel, the

``skb`` created from such a frame won't have any hardware metadata populated

in its ``skb``. If such a packet is later redirected into an ``XSK``,

that will also only have access to the custom metadata.

bpf_tail_call

=============

Adding programs that access metadata kfuncs to the ``BPF_MAP_TYPE_PROG_ARRAY``

is currently not supported.

Example

=======

See ``tools/testing/selftests/bpf/progs/xdp_metadata.c`` and

``tools/testing/selftests/bpf/prog_tests/xdp_metadata.c`` for an example of

BPF program that handles XDP metadata.

	return hi | lo;

}

void mlx4_en_fill_hwtstamps(struct mlx4_en_dev *mdev,

			    struct skb_shared_hwtstamps *hwts,

			    u64 timestamp)

u64 mlx4_en_get_hwtstamp(struct mlx4_en_dev *mdev, u64 timestamp)

{

	unsigned int seq;

	u64 nsec;

		nsec = timecounter_cyc2time(&mdev->clock, timestamp);

	} while (read_seqretry(&mdev->clock_lock, seq));

	return ns_to_ktime(nsec);

}

void mlx4_en_fill_hwtstamps(struct mlx4_en_dev *mdev,

			    struct skb_shared_hwtstamps *hwts,

			    u64 timestamp)

{

	memset(hwts, 0, sizeof(struct skb_shared_hwtstamps));

	hwts->hwtstamp = ns_to_ktime(nsec);

	hwts->hwtstamp = mlx4_en_get_hwtstamp(mdev, timestamp);

}

/**

	.ndo_bpf		= mlx4_xdp,

};

static const struct xdp_metadata_ops mlx4_xdp_metadata_ops = {

	.xmo_rx_timestamp		= mlx4_en_xdp_rx_timestamp,

	.xmo_rx_hash			= mlx4_en_xdp_rx_hash,

};

struct mlx4_en_bond {

	struct work_struct work;

	struct mlx4_en_priv *priv;

		dev->netdev_ops = &mlx4_netdev_ops_master;

	else

		dev->netdev_ops = &mlx4_netdev_ops;

	dev->xdp_metadata_ops = &mlx4_xdp_metadata_ops;

	dev->watchdog_timeo = MLX4_EN_WATCHDOG_TIMEOUT;

	netif_set_real_num_tx_queues(dev, priv->tx_ring_num[TX]);

	netif_set_real_num_rx_queues(dev, priv->rx_ring_num);

#define MLX4_CQE_STATUS_IP_ANY (MLX4_CQE_STATUS_IPV4)

#endif

struct mlx4_en_xdp_buff {

	struct xdp_buff xdp;

	struct mlx4_cqe *cqe;

	struct mlx4_en_dev *mdev;

	struct mlx4_en_rx_ring *ring;

	struct net_device *dev;

};

int mlx4_en_xdp_rx_timestamp(const struct xdp_md *ctx, u64 *timestamp)

{

	struct mlx4_en_xdp_buff *_ctx = (void *)ctx;

	if (unlikely(_ctx->ring->hwtstamp_rx_filter != HWTSTAMP_FILTER_ALL))

		return -EOPNOTSUPP;

	*timestamp = mlx4_en_get_hwtstamp(_ctx->mdev,

					  mlx4_en_get_cqe_ts(_ctx->cqe));

	return 0;

}

int mlx4_en_xdp_rx_hash(const struct xdp_md *ctx, u32 *hash)

{

	struct mlx4_en_xdp_buff *_ctx = (void *)ctx;

	if (unlikely(!(_ctx->dev->features & NETIF_F_RXHASH)))

		return -EOPNOTSUPP;

	*hash = be32_to_cpu(_ctx->cqe->immed_rss_invalid);

	return 0;

}

int mlx4_en_process_rx_cq(struct net_device *dev, struct mlx4_en_cq *cq, int budget)

{

	struct mlx4_en_priv *priv = netdev_priv(dev);

	struct mlx4_en_xdp_buff mxbuf = {};

	int factor = priv->cqe_factor;

	struct mlx4_en_rx_ring *ring;

	struct bpf_prog *xdp_prog;

	bool doorbell_pending;

	bool xdp_redir_flush;

	struct mlx4_cqe *cqe;

	struct xdp_buff xdp;

	int polled = 0;

	int index;

	ring = priv->rx_ring[cq_ring];

	xdp_prog = rcu_dereference_bh(ring->xdp_prog);

	xdp_init_buff(&xdp, priv->frag_info[0].frag_stride, &ring->xdp_rxq);

	xdp_init_buff(&mxbuf.xdp, priv->frag_info[0].frag_stride, &ring->xdp_rxq);

	doorbell_pending = false;

	xdp_redir_flush = false;

						priv->frag_info[0].frag_size,

						DMA_FROM_DEVICE);

			xdp_prepare_buff(&xdp, va - frags[0].page_offset,

					 frags[0].page_offset, length, false);

			orig_data = xdp.data;

			xdp_prepare_buff(&mxbuf.xdp, va - frags[0].page_offset,

					 frags[0].page_offset, length, true);

			orig_data = mxbuf.xdp.data;

			mxbuf.cqe = cqe;

			mxbuf.mdev = priv->mdev;

			mxbuf.ring = ring;

			mxbuf.dev = dev;

			act = bpf_prog_run_xdp(xdp_prog, &xdp);

			act = bpf_prog_run_xdp(xdp_prog, &mxbuf.xdp);

			length = xdp.data_end - xdp.data;

			if (xdp.data != orig_data) {

				frags[0].page_offset = xdp.data -

					xdp.data_hard_start;

				va = xdp.data;

			length = mxbuf.xdp.data_end - mxbuf.xdp.data;

			if (mxbuf.xdp.data != orig_data) {

				frags[0].page_offset = mxbuf.xdp.data -

					mxbuf.xdp.data_hard_start;

				va = mxbuf.xdp.data;

			}

			switch (act) {

			case XDP_PASS:

				break;

			case XDP_REDIRECT:

				if (likely(!xdp_do_redirect(dev, &xdp, xdp_prog))) {

				if (likely(!xdp_do_redirect(dev, &mxbuf.xdp, xdp_prog))) {

					ring->xdp_redirect++;

					xdp_redir_flush = true;

					frags[0].page = NULL;