Commit d75858ef authored by Jakub Kicinski's avatar Jakub Kicinski
Browse files
Daniel Borkmann says:

====================
bpf-next 2023-01-04

We've added 45 non-merge commits during the last 21 day(s) which contain
a total of 50 files changed, 1454 insertions(+), 375 deletions(-).

The main changes are:

1) Fixes, improvements and refactoring of parts of BPF verifier's
   state equivalence checks, from Andrii Nakryiko.

2) Fix a few corner cases in libbpf's BTF-to-C converter in particular
   around padding handling and enums, also from Andrii Nakryiko.

3) Add BPF_F_NO_TUNNEL_KEY extension to bpf_skb_set_tunnel_key to better
  support decap on GRE tunnel devices not operating in collect metadata,
  from Christian Ehrig.

4) Improve x86 JIT's codegen for PROBE_MEM runtime error checks,
   from Dave Marchevsky.

5) Remove the need for trace_printk_lock for bpf_trace_printk
   and bpf_trace_vprintk helpers, from Jiri Olsa.

6) Add proper documentation for BPF_MAP_TYPE_SOCK{MAP,HASH} maps,
   from Maryam Tahhan.

7) Improvements in libbpf's btf_parse_elf error handling, from Changbin Du.

8) Bigger batch of improvements to BPF tracing code samples,
   from Daniel T. Lee.

9) Add LoongArch support to libbpf's bpf_tracing helper header,
   from Hengqi Chen.

10) Fix a libbpf compiler warning in perf_event_open_probe on arm32,
    from Khem Raj.

11) Optimize bpf_local_storage_elem by removing 56 bytes of padding,
    from Martin KaFai Lau.

12) Use pkg-config to locate libelf for resolve_btfids build,
    from Shen Jiamin.

13) Various libbpf improvements around API documentation and errno
    handling, from Xin Liu.

* tag 'for-netdev' of https://git.kernel.org/pub/scm/linux/kernel/git/bpf/bpf-next: (45 commits)
  libbpf: Return -ENODATA for missing btf section
  libbpf: Add LoongArch support to bpf_tracing.h
  libbpf: Restore errno after pr_warn.
  libbpf: Added the description of some API functions
  libbpf: Fix invalid return address register in s390
  samples/bpf: Use BPF_KSYSCALL macro in syscall tracing programs
  samples/bpf: Fix tracex2 by using BPF_KSYSCALL macro
  samples/bpf: Change _kern suffix to .bpf with syscall tracing program
  samples/bpf: Use vmlinux.h instead of implicit headers in syscall tracing program
  samples/bpf: Use kyscall instead of kprobe in syscall tracing program
  bpf: rename list_head -> graph_root in field info types
  libbpf: fix errno is overwritten after being closed.
  bpf: fix regs_exact() logic in regsafe() to remap IDs correctly
  bpf: perform byte-by-byte comparison only when necessary in regsafe()
  bpf: reject non-exact register type matches in regsafe()
  bpf: generalize MAYBE_NULL vs non-MAYBE_NULL rule
  bpf: reorganize struct bpf_reg_state fields
  bpf: teach refsafe() to take into account ID remapping
  bpf: Remove unused field initialization in bpf's ctl_table
  selftests/bpf: Add jit probe_mem corner case tests to s390x denylist
  ...
====================

Link: https://lore.kernel.org/r/20230105000926.31350-1-daniel@iogearbox.net


Signed-off-by: default avatarJakub Kicinski <kuba@kernel.org>
parents 1f47510e acd3b776
Loading
Loading
Loading
Loading
+498 −0
Original line number Diff line number Diff line
.. SPDX-License-Identifier: GPL-2.0-only
.. Copyright Red Hat

==============================================
BPF_MAP_TYPE_SOCKMAP and BPF_MAP_TYPE_SOCKHASH
==============================================

.. note::
   - ``BPF_MAP_TYPE_SOCKMAP`` was introduced in kernel version 4.14
   - ``BPF_MAP_TYPE_SOCKHASH`` was introduced in kernel version 4.18

``BPF_MAP_TYPE_SOCKMAP`` and ``BPF_MAP_TYPE_SOCKHASH`` maps can be used to
redirect skbs between sockets or to apply policy at the socket level based on
the result of a BPF (verdict) program with the help of the BPF helpers
``bpf_sk_redirect_map()``, ``bpf_sk_redirect_hash()``,
``bpf_msg_redirect_map()`` and ``bpf_msg_redirect_hash()``.

``BPF_MAP_TYPE_SOCKMAP`` is backed by an array that uses an integer key as the
index to look up a reference to a ``struct sock``. The map values are socket
descriptors. Similarly, ``BPF_MAP_TYPE_SOCKHASH`` is a hash backed BPF map that
holds references to sockets via their socket descriptors.

.. note::
    The value type is either __u32 or __u64; the latter (__u64) is to support
    returning socket cookies to userspace. Returning the ``struct sock *`` that
    the map holds to user-space is neither safe nor useful.

These maps may have BPF programs attached to them, specifically a parser program
and a verdict program. The parser program determines how much data has been
parsed and therefore how much data needs to be queued to come to a verdict. The
verdict program is essentially the redirect program and can return a verdict
of ``__SK_DROP``, ``__SK_PASS``, or ``__SK_REDIRECT``.

When a socket is inserted into one of these maps, its socket callbacks are
replaced and a ``struct sk_psock`` is attached to it. Additionally, this
``sk_psock`` inherits the programs that are attached to the map.

A sock object may be in multiple maps, but can only inherit a single
parse or verdict program. If adding a sock object to a map would result
in having multiple parser programs the update will return an EBUSY error.

The supported programs to attach to these maps are:

.. code-block:: c

	struct sk_psock_progs {
		struct bpf_prog *msg_parser;
		struct bpf_prog *stream_parser;
		struct bpf_prog *stream_verdict;
		struct bpf_prog	*skb_verdict;
	};

.. note::
    Users are not allowed to attach ``stream_verdict`` and ``skb_verdict``
    programs to the same map.

The attach types for the map programs are:

- ``msg_parser`` program - ``BPF_SK_MSG_VERDICT``.
- ``stream_parser`` program - ``BPF_SK_SKB_STREAM_PARSER``.
- ``stream_verdict`` program - ``BPF_SK_SKB_STREAM_VERDICT``.
- ``skb_verdict`` program - ``BPF_SK_SKB_VERDICT``.

There are additional helpers available to use with the parser and verdict
programs: ``bpf_msg_apply_bytes()`` and ``bpf_msg_cork_bytes()``. With
``bpf_msg_apply_bytes()`` BPF programs can tell the infrastructure how many
bytes the given verdict should apply to. The helper ``bpf_msg_cork_bytes()``
handles a different case where a BPF program cannot reach a verdict on a msg
until it receives more bytes AND the program doesn't want to forward the packet
until it is known to be good.

Finally, the helpers ``bpf_msg_pull_data()`` and ``bpf_msg_push_data()`` are
available to ``BPF_PROG_TYPE_SK_MSG`` BPF programs to pull in data and set the
start and end pointers to given values or to add metadata to the ``struct
sk_msg_buff *msg``.

All these helpers will be described in more detail below.

Usage
=====
Kernel BPF
----------
bpf_msg_redirect_map()
^^^^^^^^^^^^^^^^^^^^^^
.. code-block:: c

	long bpf_msg_redirect_map(struct sk_msg_buff *msg, struct bpf_map *map, u32 key, u64 flags)

This helper is used in programs implementing policies at the socket level. If
the message ``msg`` is allowed to pass (i.e., if the verdict BPF program
returns ``SK_PASS``), redirect it to the socket referenced by ``map`` (of type
``BPF_MAP_TYPE_SOCKMAP``) at index ``key``. Both ingress and egress interfaces
can be used for redirection. The ``BPF_F_INGRESS`` value in ``flags`` is used
to select the ingress path otherwise the egress path is selected. This is the
only flag supported for now.

Returns ``SK_PASS`` on success, or ``SK_DROP`` on error.

bpf_sk_redirect_map()
^^^^^^^^^^^^^^^^^^^^^
.. code-block:: c

    long bpf_sk_redirect_map(struct sk_buff *skb, struct bpf_map *map, u32 key u64 flags)

Redirect the packet to the socket referenced by ``map`` (of type
``BPF_MAP_TYPE_SOCKMAP``) at index ``key``. Both ingress and egress interfaces
can be used for redirection. The ``BPF_F_INGRESS`` value in ``flags`` is used
to select the ingress path otherwise the egress path is selected. This is the
only flag supported for now.

Returns ``SK_PASS`` on success, or ``SK_DROP`` on error.

bpf_map_lookup_elem()
^^^^^^^^^^^^^^^^^^^^^
.. code-block:: c

    void *bpf_map_lookup_elem(struct bpf_map *map, const void *key)

socket entries of type ``struct sock *`` can be retrieved using the
``bpf_map_lookup_elem()`` helper.

bpf_sock_map_update()
^^^^^^^^^^^^^^^^^^^^^
.. code-block:: c

    long bpf_sock_map_update(struct bpf_sock_ops *skops, struct bpf_map *map, void *key, u64 flags)

Add an entry to, or update a ``map`` referencing sockets. The ``skops`` is used
as a new value for the entry associated to ``key``. The ``flags`` argument can
be one of the following:

- ``BPF_ANY``: Create a new element or update an existing element.
- ``BPF_NOEXIST``: Create a new element only if it did not exist.
- ``BPF_EXIST``: Update an existing element.

If the ``map`` has BPF programs (parser and verdict), those will be inherited
by the socket being added. If the socket is already attached to BPF programs,
this results in an error.

Returns 0 on success, or a negative error in case of failure.

bpf_sock_hash_update()
^^^^^^^^^^^^^^^^^^^^^^
.. code-block:: c

    long bpf_sock_hash_update(struct bpf_sock_ops *skops, struct bpf_map *map, void *key, u64 flags)

Add an entry to, or update a sockhash ``map`` referencing sockets. The ``skops``
is used as a new value for the entry associated to ``key``.

The ``flags`` argument can be one of the following:

- ``BPF_ANY``: Create a new element or update an existing element.
- ``BPF_NOEXIST``: Create a new element only if it did not exist.
- ``BPF_EXIST``: Update an existing element.

If the ``map`` has BPF programs (parser and verdict), those will be inherited
by the socket being added. If the socket is already attached to BPF programs,
this results in an error.

Returns 0 on success, or a negative error in case of failure.

bpf_msg_redirect_hash()
^^^^^^^^^^^^^^^^^^^^^^^
.. code-block:: c

    long bpf_msg_redirect_hash(struct sk_msg_buff *msg, struct bpf_map *map, void *key, u64 flags)

This helper is used in programs implementing policies at the socket level. If
the message ``msg`` is allowed to pass (i.e., if the verdict BPF program returns
``SK_PASS``), redirect it to the socket referenced by ``map`` (of type
``BPF_MAP_TYPE_SOCKHASH``) using hash ``key``. Both ingress and egress
interfaces can be used for redirection. The ``BPF_F_INGRESS`` value in
``flags`` is used to select the ingress path otherwise the egress path is
selected. This is the only flag supported for now.

Returns ``SK_PASS`` on success, or ``SK_DROP`` on error.

bpf_sk_redirect_hash()
^^^^^^^^^^^^^^^^^^^^^^
.. code-block:: c

    long bpf_sk_redirect_hash(struct sk_buff *skb, struct bpf_map *map, void *key, u64 flags)

This helper is used in programs implementing policies at the skb socket level.
If the sk_buff ``skb`` is allowed to pass (i.e., if the verdict BPF program
returns ``SK_PASS``), redirect it to the socket referenced by ``map`` (of type
``BPF_MAP_TYPE_SOCKHASH``) using hash ``key``. Both ingress and egress
interfaces can be used for redirection. The ``BPF_F_INGRESS`` value in
``flags`` is used to select the ingress path otherwise the egress path is
selected. This is the only flag supported for now.

Returns ``SK_PASS`` on success, or ``SK_DROP`` on error.

bpf_msg_apply_bytes()
^^^^^^^^^^^^^^^^^^^^^^
.. code-block:: c

    long bpf_msg_apply_bytes(struct sk_msg_buff *msg, u32 bytes)

For socket policies, apply the verdict of the BPF program to the next (number
of ``bytes``) of message ``msg``. For example, this helper can be used in the
following cases:

- A single ``sendmsg()`` or ``sendfile()`` system call contains multiple
  logical messages that the BPF program is supposed to read and for which it
  should apply a verdict.
- A BPF program only cares to read the first ``bytes`` of a ``msg``. If the
  message has a large payload, then setting up and calling the BPF program
  repeatedly for all bytes, even though the verdict is already known, would
  create unnecessary overhead.

Returns 0

bpf_msg_cork_bytes()
^^^^^^^^^^^^^^^^^^^^^^
.. code-block:: c

    long bpf_msg_cork_bytes(struct sk_msg_buff *msg, u32 bytes)

For socket policies, prevent the execution of the verdict BPF program for
message ``msg`` until the number of ``bytes`` have been accumulated.

This can be used when one needs a specific number of bytes before a verdict can
be assigned, even if the data spans multiple ``sendmsg()`` or ``sendfile()``
calls.

Returns 0

bpf_msg_pull_data()
^^^^^^^^^^^^^^^^^^^^^^
.. code-block:: c

    long bpf_msg_pull_data(struct sk_msg_buff *msg, u32 start, u32 end, u64 flags)

For socket policies, pull in non-linear data from user space for ``msg`` and set
pointers ``msg->data`` and ``msg->data_end`` to ``start`` and ``end`` bytes
offsets into ``msg``, respectively.

If a program of type ``BPF_PROG_TYPE_SK_MSG`` is run on a ``msg`` it can only
parse data that the (``data``, ``data_end``) pointers have already consumed.
For ``sendmsg()`` hooks this is likely the first scatterlist element. But for
calls relying on the ``sendpage`` handler (e.g., ``sendfile()``) this will be
the range (**0**, **0**) because the data is shared with user space and by
default the objective is to avoid allowing user space to modify data while (or
after) BPF verdict is being decided. This helper can be used to pull in data
and to set the start and end pointers to given values. Data will be copied if
necessary (i.e., if data was not linear and if start and end pointers do not
point to the same chunk).

A call to this helper is susceptible to change the underlying packet buffer.
Therefore, at load time, all checks on pointers previously done by the verifier
are invalidated and must be performed again, if the helper is used in
combination with direct packet access.

All values for ``flags`` are reserved for future usage, and must be left at
zero.

Returns 0 on success, or a negative error in case of failure.

bpf_map_lookup_elem()
^^^^^^^^^^^^^^^^^^^^^

.. code-block:: c

	void *bpf_map_lookup_elem(struct bpf_map *map, const void *key)

Look up a socket entry in the sockmap or sockhash map.

Returns the socket entry associated to ``key``, or NULL if no entry was found.

bpf_map_update_elem()
^^^^^^^^^^^^^^^^^^^^^
.. code-block:: c

	long bpf_map_update_elem(struct bpf_map *map, const void *key, const void *value, u64 flags)

Add or update a socket entry in a sockmap or sockhash.

The flags argument can be one of the following:

- BPF_ANY: Create a new element or update an existing element.
- BPF_NOEXIST: Create a new element only if it did not exist.
- BPF_EXIST: Update an existing element.

Returns 0 on success, or a negative error in case of failure.

bpf_map_delete_elem()
^^^^^^^^^^^^^^^^^^^^^^
.. code-block:: c

    long bpf_map_delete_elem(struct bpf_map *map, const void *key)

Delete a socket entry from a sockmap or a sockhash.

Returns	0 on success, or a negative error in case of failure.

User space
----------
bpf_map_update_elem()
^^^^^^^^^^^^^^^^^^^^^
.. code-block:: c

	int bpf_map_update_elem(int fd, const void *key, const void *value, __u64 flags)

Sockmap entries can be added or updated using the ``bpf_map_update_elem()``
function. The ``key`` parameter is the index value of the sockmap array. And the
``value`` parameter is the FD value of that socket.

Under the hood, the sockmap update function uses the socket FD value to
retrieve the associated socket and its attached psock.

The flags argument can be one of the following:

- BPF_ANY: Create a new element or update an existing element.
- BPF_NOEXIST: Create a new element only if it did not exist.
- BPF_EXIST: Update an existing element.

bpf_map_lookup_elem()
^^^^^^^^^^^^^^^^^^^^^
.. code-block:: c

    int bpf_map_lookup_elem(int fd, const void *key, void *value)

Sockmap entries can be retrieved using the ``bpf_map_lookup_elem()`` function.

.. note::
	The entry returned is a socket cookie rather than a socket itself.

bpf_map_delete_elem()
^^^^^^^^^^^^^^^^^^^^^
.. code-block:: c

    int bpf_map_delete_elem(int fd, const void *key)

Sockmap entries can be deleted using the ``bpf_map_delete_elem()``
function.

Returns 0 on success, or negative error in case of failure.

Examples
========

Kernel BPF
----------
Several examples of the use of sockmap APIs can be found in:

- `tools/testing/selftests/bpf/progs/test_sockmap_kern.h`_
- `tools/testing/selftests/bpf/progs/sockmap_parse_prog.c`_
- `tools/testing/selftests/bpf/progs/sockmap_verdict_prog.c`_
- `tools/testing/selftests/bpf/progs/test_sockmap_listen.c`_
- `tools/testing/selftests/bpf/progs/test_sockmap_update.c`_

The following code snippet shows how to declare a sockmap.

.. code-block:: c

	struct {
		__uint(type, BPF_MAP_TYPE_SOCKMAP);
		__uint(max_entries, 1);
		__type(key, __u32);
		__type(value, __u64);
	} sock_map_rx SEC(".maps");

The following code snippet shows a sample parser program.

.. code-block:: c

	SEC("sk_skb/stream_parser")
	int bpf_prog_parser(struct __sk_buff *skb)
	{
		return skb->len;
	}

The following code snippet shows a simple verdict program that interacts with a
sockmap to redirect traffic to another socket based on the local port.

.. code-block:: c

	SEC("sk_skb/stream_verdict")
	int bpf_prog_verdict(struct __sk_buff *skb)
	{
		__u32 lport = skb->local_port;
		__u32 idx = 0;

		if (lport == 10000)
			return bpf_sk_redirect_map(skb, &sock_map_rx, idx, 0);

		return SK_PASS;
	}

The following code snippet shows how to declare a sockhash map.

.. code-block:: c

	struct socket_key {
		__u32 src_ip;
		__u32 dst_ip;
		__u32 src_port;
		__u32 dst_port;
	};

	struct {
		__uint(type, BPF_MAP_TYPE_SOCKHASH);
		__uint(max_entries, 1);
		__type(key, struct socket_key);
		__type(value, __u64);
	} sock_hash_rx SEC(".maps");

The following code snippet shows a simple verdict program that interacts with a
sockhash to redirect traffic to another socket based on a hash of some of the
skb parameters.

.. code-block:: c

	static inline
	void extract_socket_key(struct __sk_buff *skb, struct socket_key *key)
	{
		key->src_ip = skb->remote_ip4;
		key->dst_ip = skb->local_ip4;
		key->src_port = skb->remote_port >> 16;
		key->dst_port = (bpf_htonl(skb->local_port)) >> 16;
	}

	SEC("sk_skb/stream_verdict")
	int bpf_prog_verdict(struct __sk_buff *skb)
	{
		struct socket_key key;

		extract_socket_key(skb, &key);

		return bpf_sk_redirect_hash(skb, &sock_hash_rx, &key, 0);
	}

User space
----------
Several examples of the use of sockmap APIs can be found in:

- `tools/testing/selftests/bpf/prog_tests/sockmap_basic.c`_
- `tools/testing/selftests/bpf/test_sockmap.c`_
- `tools/testing/selftests/bpf/test_maps.c`_

The following code sample shows how to create a sockmap, attach a parser and
verdict program, as well as add a socket entry.

.. code-block:: c

	int create_sample_sockmap(int sock, int parse_prog_fd, int verdict_prog_fd)
	{
		int index = 0;
		int map, err;

		map = bpf_map_create(BPF_MAP_TYPE_SOCKMAP, NULL, sizeof(int), sizeof(int), 1, NULL);
		if (map < 0) {
			fprintf(stderr, "Failed to create sockmap: %s\n", strerror(errno));
			return -1;
		}

		err = bpf_prog_attach(parse_prog_fd, map, BPF_SK_SKB_STREAM_PARSER, 0);
		if (err){
			fprintf(stderr, "Failed to attach_parser_prog_to_map: %s\n", strerror(errno));
			goto out;
		}

		err = bpf_prog_attach(verdict_prog_fd, map, BPF_SK_SKB_STREAM_VERDICT, 0);
		if (err){
			fprintf(stderr, "Failed to attach_verdict_prog_to_map: %s\n", strerror(errno));
			goto out;
		}

		err = bpf_map_update_elem(map, &index, &sock, BPF_NOEXIST);
		if (err) {
			fprintf(stderr, "Failed to update sockmap: %s\n", strerror(errno));
			goto out;
		}

	out:
		close(map);
		return err;
	}

References
===========

- https://github.com/jrfastab/linux-kernel-xdp/commit/c89fd73cb9d2d7f3c716c3e00836f07b1aeb261f
- https://lwn.net/Articles/731133/
- http://vger.kernel.org/lpc_net2018_talks/ktls_bpf_paper.pdf
- https://lwn.net/Articles/748628/
- https://lore.kernel.org/bpf/20200218171023.844439-7-jakub@cloudflare.com/

.. _`tools/testing/selftests/bpf/progs/test_sockmap_kern.h`: https://git.kernel.org/pub/scm/linux/kernel/git/torvalds/linux.git/tree/tools/testing/selftests/bpf/progs/test_sockmap_kern.h
.. _`tools/testing/selftests/bpf/progs/sockmap_parse_prog.c`: https://git.kernel.org/pub/scm/linux/kernel/git/torvalds/linux.git/tree/tools/testing/selftests/bpf/progs/sockmap_parse_prog.c
.. _`tools/testing/selftests/bpf/progs/sockmap_verdict_prog.c`: https://git.kernel.org/pub/scm/linux/kernel/git/torvalds/linux.git/tree/tools/testing/selftests/bpf/progs/sockmap_verdict_prog.c
.. _`tools/testing/selftests/bpf/prog_tests/sockmap_basic.c`: https://git.kernel.org/pub/scm/linux/kernel/git/torvalds/linux.git/tree/tools/testing/selftests/bpf/prog_tests/sockmap_basic.c
.. _`tools/testing/selftests/bpf/test_sockmap.c`: https://git.kernel.org/pub/scm/linux/kernel/git/torvalds/linux.git/tree/tools/testing/selftests/bpf/test_sockmap.c
.. _`tools/testing/selftests/bpf/test_maps.c`: https://git.kernel.org/pub/scm/linux/kernel/git/torvalds/linux.git/tree/tools/testing/selftests/bpf/test_maps.c
.. _`tools/testing/selftests/bpf/progs/test_sockmap_listen.c`: https://git.kernel.org/pub/scm/linux/kernel/git/torvalds/linux.git/tree/tools/testing/selftests/bpf/progs/test_sockmap_listen.c
.. _`tools/testing/selftests/bpf/progs/test_sockmap_update.c`: https://git.kernel.org/pub/scm/linux/kernel/git/torvalds/linux.git/tree/tools/testing/selftests/bpf/progs/test_sockmap_update.c
+39 −31
Original line number Diff line number Diff line
@@ -1003,6 +1003,7 @@ static int do_jit(struct bpf_prog *bpf_prog, int *addrs, u8 *image, u8 *rw_image
		u8 b2 = 0, b3 = 0;
		u8 *start_of_ldx;
		s64 jmp_offset;
		s16 insn_off;
		u8 jmp_cond;
		u8 *func;
		int nops;
@@ -1369,57 +1370,52 @@ st: if (is_imm8(insn->off))
		case BPF_LDX | BPF_PROBE_MEM | BPF_W:
		case BPF_LDX | BPF_MEM | BPF_DW:
		case BPF_LDX | BPF_PROBE_MEM | BPF_DW:
			insn_off = insn->off;

			if (BPF_MODE(insn->code) == BPF_PROBE_MEM) {
				/* Though the verifier prevents negative insn->off in BPF_PROBE_MEM
				 * add abs(insn->off) to the limit to make sure that negative
				 * offset won't be an issue.
				 * insn->off is s16, so it won't affect valid pointers.
				/* Conservatively check that src_reg + insn->off is a kernel address:
				 *   src_reg + insn->off >= TASK_SIZE_MAX + PAGE_SIZE
				 * src_reg is used as scratch for src_reg += insn->off and restored
				 * after emit_ldx if necessary
				 */
				u64 limit = TASK_SIZE_MAX + PAGE_SIZE + abs(insn->off);
				u8 *end_of_jmp1, *end_of_jmp2;

				/* Conservatively check that src_reg + insn->off is a kernel address:
				 * 1. src_reg + insn->off >= limit
				 * 2. src_reg + insn->off doesn't become small positive.
				 * Cannot do src_reg + insn->off >= limit in one branch,
				 * since it needs two spare registers, but JIT has only one.
				u64 limit = TASK_SIZE_MAX + PAGE_SIZE;
				u8 *end_of_jmp;

				/* At end of these emitted checks, insn->off will have been added
				 * to src_reg, so no need to do relative load with insn->off offset
				 */
				insn_off = 0;

				/* movabsq r11, limit */
				EMIT2(add_1mod(0x48, AUX_REG), add_1reg(0xB8, AUX_REG));
				EMIT((u32)limit, 4);
				EMIT(limit >> 32, 4);

				if (insn->off) {
					/* add src_reg, insn->off */
					maybe_emit_1mod(&prog, src_reg, true);
					EMIT2_off32(0x81, add_1reg(0xC0, src_reg), insn->off);
				}

				/* cmp src_reg, r11 */
				maybe_emit_mod(&prog, src_reg, AUX_REG, true);
				EMIT2(0x39, add_2reg(0xC0, src_reg, AUX_REG));
				/* if unsigned '<' goto end_of_jmp2 */
				EMIT2(X86_JB, 0);
				end_of_jmp1 = prog;

				/* mov r11, src_reg */
				emit_mov_reg(&prog, true, AUX_REG, src_reg);
				/* add r11, insn->off */
				maybe_emit_1mod(&prog, AUX_REG, true);
				EMIT2_off32(0x81, add_1reg(0xC0, AUX_REG), insn->off);
				/* jmp if not carry to start_of_ldx
				 * Otherwise ERR_PTR(-EINVAL) + 128 will be the user addr
				 * that has to be rejected.
				 */
				EMIT2(0x73 /* JNC */, 0);
				end_of_jmp2 = prog;
				/* if unsigned '>=', goto load */
				EMIT2(X86_JAE, 0);
				end_of_jmp = prog;

				/* xor dst_reg, dst_reg */
				emit_mov_imm32(&prog, false, dst_reg, 0);
				/* jmp byte_after_ldx */
				EMIT2(0xEB, 0);

				/* populate jmp_offset for JB above to jump to xor dst_reg */
				end_of_jmp1[-1] = end_of_jmp2 - end_of_jmp1;
				/* populate jmp_offset for JNC above to jump to start_of_ldx */
				/* populate jmp_offset for JAE above to jump to start_of_ldx */
				start_of_ldx = prog;
				end_of_jmp2[-1] = start_of_ldx - end_of_jmp2;
				end_of_jmp[-1] = start_of_ldx - end_of_jmp;
			}
			emit_ldx(&prog, BPF_SIZE(insn->code), dst_reg, src_reg, insn->off);
			emit_ldx(&prog, BPF_SIZE(insn->code), dst_reg, src_reg, insn_off);
			if (BPF_MODE(insn->code) == BPF_PROBE_MEM) {
				struct exception_table_entry *ex;
				u8 *_insn = image + proglen + (start_of_ldx - temp);
@@ -1428,6 +1424,18 @@ st: if (is_imm8(insn->off))
				/* populate jmp_offset for JMP above */
				start_of_ldx[-1] = prog - start_of_ldx;

				if (insn->off && src_reg != dst_reg) {
					/* sub src_reg, insn->off
					 * Restore src_reg after "add src_reg, insn->off" in prev
					 * if statement. But if src_reg == dst_reg, emit_ldx
					 * above already clobbered src_reg, so no need to restore.
					 * If add src_reg, insn->off was unnecessary, no need to
					 * restore either.
					 */
					maybe_emit_1mod(&prog, src_reg, true);
					EMIT2_off32(0x81, add_1reg(0xE8, src_reg), insn->off);
				}

				if (!bpf_prog->aux->extable)
					break;

+12 −4
Original line number Diff line number Diff line
@@ -189,7 +189,7 @@ struct btf_field_kptr {
	u32 btf_id;
};

struct btf_field_list_head {
struct btf_field_graph_root {
	struct btf *btf;
	u32 value_btf_id;
	u32 node_offset;
@@ -201,7 +201,7 @@ struct btf_field {
	enum btf_field_type type;
	union {
		struct btf_field_kptr kptr;
		struct btf_field_list_head list_head;
		struct btf_field_graph_root graph_root;
	};
};

@@ -2795,10 +2795,18 @@ struct btf_id_set;
bool btf_id_set_contains(const struct btf_id_set *set, u32 id);

#define MAX_BPRINTF_VARARGS		12
#define MAX_BPRINTF_BUF			1024

struct bpf_bprintf_data {
	u32 *bin_args;
	char *buf;
	bool get_bin_args;
	bool get_buf;
};

int bpf_bprintf_prepare(char *fmt, u32 fmt_size, const u64 *raw_args,
			u32 **bin_buf, u32 num_args);
void bpf_bprintf_cleanup(void);
			u32 num_args, struct bpf_bprintf_data *data);
void bpf_bprintf_cleanup(struct bpf_bprintf_data *data);

/* the implementation of the opaque uapi struct bpf_dynptr */
struct bpf_dynptr_kern {
+20 −20

File changed.

Preview size limit exceeded, changes collapsed.

+4 −0
Original line number Diff line number Diff line
@@ -2001,6 +2001,9 @@ union bpf_attr {
 * 			sending the packet. This flag was added for GRE
 * 			encapsulation, but might be used with other protocols
 * 			as well in the future.
 * 		**BPF_F_NO_TUNNEL_KEY**
 * 			Add a flag to tunnel metadata indicating that no tunnel
 * 			key should be set in the resulting tunnel header.
 *
 * 		Here is a typical usage on the transmit path:
 *
@@ -5764,6 +5767,7 @@ enum {
	BPF_F_ZERO_CSUM_TX		= (1ULL << 1),
	BPF_F_DONT_FRAGMENT		= (1ULL << 2),
	BPF_F_SEQ_NUMBER		= (1ULL << 3),
	BPF_F_NO_TUNNEL_KEY		= (1ULL << 4),
};

/* BPF_FUNC_skb_get_tunnel_key flags. */
Loading