Commit a52171ae authored by David S. Miller's avatar David S. Miller
Browse files

Merge git://git.kernel.org/pub/scm/linux/kernel/git/bpf/bpf-next 

Daniel Borkmann says:

====================
pull-request: bpf-next 2021-06-17

The following pull-request contains BPF updates for your *net-next* tree.

We've added 50 non-merge commits during the last 25 day(s) which contain
a total of 148 files changed, 4779 insertions(+), 1248 deletions(-).

The main changes are:

1) BPF infrastructure to migrate TCP child sockets from a listener to another
   in the same reuseport group/map, from Kuniyuki Iwashima.

2) Add a provably sound, faster and more precise algorithm for tnum_mul() as
   noted in https://arxiv.org/abs/2105.05398

, from Harishankar Vishwanathan.

3) Streamline error reporting changes in libbpf as planned out in the
   'libbpf: the road to v1.0' effort, from Andrii Nakryiko.

4) Add broadcast support to xdp_redirect_map(), from Hangbin Liu.

5) Extends bpf_map_lookup_and_delete_elem() functionality to 4 more map
   types, that is, {LRU_,PERCPU_,LRU_PERCPU_,}HASH, from Denis Salopek.

6) Support new LLVM relocations in libbpf to make them more linker friendly,
   also add a doc to describe the BPF backend relocations, from Yonghong Song.

7) Silence long standing KUBSAN complaints on register-based shifts in
   interpreter, from Daniel Borkmann and Eric Biggers.

8) Add dummy PT_REGS macros in libbpf to fail BPF program compilation when
   target arch cannot be determined, from Lorenz Bauer.

9) Extend AF_XDP to support large umems with 1M+ pages, from Magnus Karlsson.

10) Fix two minor libbpf tc BPF API issues, from Kumar Kartikeya Dwivedi.

11) Move libbpf BPF_SEQ_PRINTF/BPF_SNPRINTF macros that can be used by BPF
    programs to bpf_helpers.h header, from Florent Revest.
====================

Signed-off-by: default avatarDavid S. Miller <davem@davemloft.net>
parents 4de77251 f20792d4

Documentation/bpf/index.rst

+1 −0
Original line number Diff line number Diff line
@@ -84,6 +84,7 @@ Other 
   :maxdepth: 1



   ringbuf

   llvm_reloc



.. Links:

.. _networking-filter: ../networking/filter.rst

Documentation/bpf/llvm_reloc.rst

0 → 100644
+240 −0
Original line number Diff line number Diff line 
.. SPDX-License-Identifier: (LGPL-2.1 OR BSD-2-Clause)



====================

BPF LLVM Relocations

====================



This document describes LLVM BPF backend relocation types.



Relocation Record

=================



LLVM BPF backend records each relocation with the following 16-byte

ELF structure::



  typedef struct

  {

    Elf64_Addr    r_offset;  // Offset from the beginning of section.

    Elf64_Xword   r_info;    // Relocation type and symbol index.

  } Elf64_Rel;



For example, for the following code::



  int g1 __attribute__((section("sec")));

  int g2 __attribute__((section("sec")));

  static volatile int l1 __attribute__((section("sec")));

  static volatile int l2 __attribute__((section("sec")));

  int test() {

    return g1 + g2 + l1 + l2;

  }



Compiled with ``clang -target bpf -O2 -c test.c``, the following is

the code with ``llvm-objdump -dr test.o``::



       0:       18 01 00 00 00 00 00 00 00 00 00 00 00 00 00 00 r1 = 0 ll

                0000000000000000:  R_BPF_64_64  g1

       2:       61 11 00 00 00 00 00 00 r1 = *(u32 *)(r1 + 0)

       3:       18 02 00 00 00 00 00 00 00 00 00 00 00 00 00 00 r2 = 0 ll

                0000000000000018:  R_BPF_64_64  g2

       5:       61 20 00 00 00 00 00 00 r0 = *(u32 *)(r2 + 0)

       6:       0f 10 00 00 00 00 00 00 r0 += r1

       7:       18 01 00 00 08 00 00 00 00 00 00 00 00 00 00 00 r1 = 8 ll

                0000000000000038:  R_BPF_64_64  sec

       9:       61 11 00 00 00 00 00 00 r1 = *(u32 *)(r1 + 0)

      10:       0f 10 00 00 00 00 00 00 r0 += r1

      11:       18 01 00 00 0c 00 00 00 00 00 00 00 00 00 00 00 r1 = 12 ll

                0000000000000058:  R_BPF_64_64  sec

      13:       61 11 00 00 00 00 00 00 r1 = *(u32 *)(r1 + 0)

      14:       0f 10 00 00 00 00 00 00 r0 += r1

      15:       95 00 00 00 00 00 00 00 exit



There are four relations in the above for four ``LD_imm64`` instructions.

The following ``llvm-readelf -r test.o`` shows the binary values of the four

relocations::



  Relocation section '.rel.text' at offset 0x190 contains 4 entries:

      Offset             Info             Type               Symbol's Value  Symbol's Name

  0000000000000000  0000000600000001 R_BPF_64_64            0000000000000000 g1

  0000000000000018  0000000700000001 R_BPF_64_64            0000000000000004 g2

  0000000000000038  0000000400000001 R_BPF_64_64            0000000000000000 sec

  0000000000000058  0000000400000001 R_BPF_64_64            0000000000000000 sec



Each relocation is represented by ``Offset`` (8 bytes) and ``Info`` (8 bytes).

For example, the first relocation corresponds to the first instruction

(Offset 0x0) and the corresponding ``Info`` indicates the relocation type

of ``R_BPF_64_64`` (type 1) and the entry in the symbol table (entry 6).

The following is the symbol table with ``llvm-readelf -s test.o``::



  Symbol table '.symtab' contains 8 entries:

     Num:    Value          Size Type    Bind   Vis       Ndx Name

       0: 0000000000000000     0 NOTYPE  LOCAL  DEFAULT   UND

       1: 0000000000000000     0 FILE    LOCAL  DEFAULT   ABS test.c

       2: 0000000000000008     4 OBJECT  LOCAL  DEFAULT     4 l1

       3: 000000000000000c     4 OBJECT  LOCAL  DEFAULT     4 l2

       4: 0000000000000000     0 SECTION LOCAL  DEFAULT     4 sec

       5: 0000000000000000   128 FUNC    GLOBAL DEFAULT     2 test

       6: 0000000000000000     4 OBJECT  GLOBAL DEFAULT     4 g1

       7: 0000000000000004     4 OBJECT  GLOBAL DEFAULT     4 g2



The 6th entry is global variable ``g1`` with value 0.



Similarly, the second relocation is at ``.text`` offset ``0x18``, instruction 3,

for global variable ``g2`` which has a symbol value 4, the offset

from the start of ``.data`` section.



The third and fourth relocations refers to static variables ``l1``

and ``l2``. From ``.rel.text`` section above, it is not clear

which symbols they really refers to as they both refers to

symbol table entry 4, symbol ``sec``, which has ``STT_SECTION`` type

and represents a section. So for static variable or function,

the section offset is written to the original insn

buffer, which is called ``A`` (addend). Looking at

above insn ``7`` and ``11``, they have section offset ``8`` and ``12``.

From symbol table, we can find that they correspond to entries ``2``

and ``3`` for ``l1`` and ``l2``.



In general, the ``A`` is 0 for global variables and functions,

and is the section offset or some computation result based on

section offset for static variables/functions. The non-section-offset

case refers to function calls. See below for more details.



Different Relocation Types

==========================



Six relocation types are supported. The following is an overview and

``S`` represents the value of the symbol in the symbol table::



  Enum  ELF Reloc Type     Description      BitSize  Offset        Calculation

  0     R_BPF_NONE         None

  1     R_BPF_64_64        ld_imm64 insn    32       r_offset + 4  S + A

  2     R_BPF_64_ABS64     normal data      64       r_offset      S + A

  3     R_BPF_64_ABS32     normal data      32       r_offset      S + A

  4     R_BPF_64_NODYLD32  .BTF[.ext] data  32       r_offset      S + A

  10    R_BPF_64_32        call insn        32       r_offset + 4  (S + A) / 8 - 1



For example, ``R_BPF_64_64`` relocation type is used for ``ld_imm64`` instruction.

The actual to-be-relocated data (0 or section offset)

is stored at ``r_offset + 4`` and the read/write

data bitsize is 32 (4 bytes). The relocation can be resolved with

the symbol value plus implicit addend. Note that the ``BitSize`` is 32 which

means the section offset must be less than or equal to ``UINT32_MAX`` and this

is enforced by LLVM BPF backend.



In another case, ``R_BPF_64_ABS64`` relocation type is used for normal 64-bit data.

The actual to-be-relocated data is stored at ``r_offset`` and the read/write data

bitsize is 64 (8 bytes). The relocation can be resolved with

the symbol value plus implicit addend.



Both ``R_BPF_64_ABS32`` and ``R_BPF_64_NODYLD32`` types are for 32-bit data.

But ``R_BPF_64_NODYLD32`` specifically refers to relocations in ``.BTF`` and

``.BTF.ext`` sections. For cases like bcc where llvm ``ExecutionEngine RuntimeDyld``

is involved, ``R_BPF_64_NODYLD32`` types of relocations should not be resolved

to actual function/variable address. Otherwise, ``.BTF`` and ``.BTF.ext``

become unusable by bcc and kernel.



Type ``R_BPF_64_32`` is used for call instruction. The call target section

offset is stored at ``r_offset + 4`` (32bit) and calculated as

``(S + A) / 8 - 1``.



Examples

========



Types ``R_BPF_64_64`` and ``R_BPF_64_32`` are used to resolve ``ld_imm64``

and ``call`` instructions. For example::



  __attribute__((noinline)) __attribute__((section("sec1")))

  int gfunc(int a, int b) {

    return a * b;

  }

  static __attribute__((noinline)) __attribute__((section("sec1")))

  int lfunc(int a, int b) {

    return a + b;

  }

  int global __attribute__((section("sec2")));

  int test(int a, int b) {

    return gfunc(a, b) +  lfunc(a, b) + global;

  }



Compiled with ``clang -target bpf -O2 -c test.c``, we will have

following code with `llvm-objdump -dr test.o``::



  Disassembly of section .text:



  0000000000000000 <test>:

         0:       bf 26 00 00 00 00 00 00 r6 = r2

         1:       bf 17 00 00 00 00 00 00 r7 = r1

         2:       85 10 00 00 ff ff ff ff call -1

                  0000000000000010:  R_BPF_64_32  gfunc

         3:       bf 08 00 00 00 00 00 00 r8 = r0

         4:       bf 71 00 00 00 00 00 00 r1 = r7

         5:       bf 62 00 00 00 00 00 00 r2 = r6

         6:       85 10 00 00 02 00 00 00 call 2

                  0000000000000030:  R_BPF_64_32  sec1

         7:       0f 80 00 00 00 00 00 00 r0 += r8

         8:       18 01 00 00 00 00 00 00 00 00 00 00 00 00 00 00 r1 = 0 ll

                  0000000000000040:  R_BPF_64_64  global

        10:       61 11 00 00 00 00 00 00 r1 = *(u32 *)(r1 + 0)

        11:       0f 10 00 00 00 00 00 00 r0 += r1

        12:       95 00 00 00 00 00 00 00 exit



  Disassembly of section sec1:



  0000000000000000 <gfunc>:

         0:       bf 20 00 00 00 00 00 00 r0 = r2

         1:       2f 10 00 00 00 00 00 00 r0 *= r1

         2:       95 00 00 00 00 00 00 00 exit



  0000000000000018 <lfunc>:

         3:       bf 20 00 00 00 00 00 00 r0 = r2

         4:       0f 10 00 00 00 00 00 00 r0 += r1

         5:       95 00 00 00 00 00 00 00 exit



The first relocation corresponds to ``gfunc(a, b)`` where ``gfunc`` has a value of 0,

so the ``call`` instruction offset is ``(0 + 0)/8 - 1 = -1``.

The second relocation corresponds to ``lfunc(a, b)`` where ``lfunc`` has a section

offset ``0x18``, so the ``call`` instruction offset is ``(0 + 0x18)/8 - 1 = 2``.

The third relocation corresponds to ld_imm64 of ``global``, which has a section

offset ``0``.



The following is an example to show how R_BPF_64_ABS64 could be generated::



  int global() { return 0; }

  struct t { void *g; } gbl = { global };



Compiled with ``clang -target bpf -O2 -g -c test.c``, we will see a

relocation below in ``.data`` section with command

``llvm-readelf -r test.o``::



  Relocation section '.rel.data' at offset 0x458 contains 1 entries:

      Offset             Info             Type               Symbol's Value  Symbol's Name

  0000000000000000  0000000700000002 R_BPF_64_ABS64         0000000000000000 global



The relocation says the first 8-byte of ``.data`` section should be

filled with address of ``global`` variable.



With ``llvm-readelf`` output, we can see that dwarf sections have a bunch of

``R_BPF_64_ABS32`` and ``R_BPF_64_ABS64`` relocations::



  Relocation section '.rel.debug_info' at offset 0x468 contains 13 entries:

      Offset             Info             Type               Symbol's Value  Symbol's Name

  0000000000000006  0000000300000003 R_BPF_64_ABS32         0000000000000000 .debug_abbrev

  000000000000000c  0000000400000003 R_BPF_64_ABS32         0000000000000000 .debug_str

  0000000000000012  0000000400000003 R_BPF_64_ABS32         0000000000000000 .debug_str

  0000000000000016  0000000600000003 R_BPF_64_ABS32         0000000000000000 .debug_line

  000000000000001a  0000000400000003 R_BPF_64_ABS32         0000000000000000 .debug_str

  000000000000001e  0000000200000002 R_BPF_64_ABS64         0000000000000000 .text

  000000000000002b  0000000400000003 R_BPF_64_ABS32         0000000000000000 .debug_str

  0000000000000037  0000000800000002 R_BPF_64_ABS64         0000000000000000 gbl

  0000000000000040  0000000400000003 R_BPF_64_ABS32         0000000000000000 .debug_str

  ......



The .BTF/.BTF.ext sections has R_BPF_64_NODYLD32 relocations::



  Relocation section '.rel.BTF' at offset 0x538 contains 1 entries:

      Offset             Info             Type               Symbol's Value  Symbol's Name

  0000000000000084  0000000800000004 R_BPF_64_NODYLD32      0000000000000000 gbl



  Relocation section '.rel.BTF.ext' at offset 0x548 contains 2 entries:

      Offset             Info             Type               Symbol's Value  Symbol's Name

  000000000000002c  0000000200000004 R_BPF_64_NODYLD32      0000000000000000 .text

  0000000000000040  0000000200000004 R_BPF_64_NODYLD32      0000000000000000 .text

Documentation/networking/ip-sysctl.rst

+25 −0
Original line number Diff line number Diff line
@@ -761,6 +761,31 @@ tcp_syncookies - INTEGER 
	network connections you can set this knob to 2 to enable

	unconditionally generation of syncookies.



tcp_migrate_req - BOOLEAN

	The incoming connection is tied to a specific listening socket when

	the initial SYN packet is received during the three-way handshake.

	When a listener is closed, in-flight request sockets during the

	handshake and established sockets in the accept queue are aborted.



	If the listener has SO_REUSEPORT enabled, other listeners on the

	same port should have been able to accept such connections. This

	option makes it possible to migrate such child sockets to another

	listener after close() or shutdown().



	The BPF_SK_REUSEPORT_SELECT_OR_MIGRATE type of eBPF program should

	usually be used to define the policy to pick an alive listener.

	Otherwise, the kernel will randomly pick an alive listener only if

	this option is enabled.



	Note that migration between listeners with different settings may

	crash applications. Let's say migration happens from listener A to

	B, and only B has TCP_SAVE_SYN enabled. B cannot read SYN data from

	the requests migrated from A. To avoid such a situation, cancel

	migration by returning SK_DROP in the type of eBPF program, or

	disable this option.



	Default: 0



tcp_fastopen - INTEGER

	Enable TCP Fast Open (RFC7413) to send and accept data in the opening

	SYN packet.

include/linux/bpf.h

+23 −0
Original line number Diff line number Diff line
@@ -70,6 +70,8 @@ struct bpf_map_ops { 
	void *(*map_lookup_elem_sys_only)(struct bpf_map *map, void *key);

	int (*map_lookup_batch)(struct bpf_map *map, const union bpf_attr *attr,

				union bpf_attr __user *uattr);

	int (*map_lookup_and_delete_elem)(struct bpf_map *map, void *key,

					  void *value, u64 flags);

	int (*map_lookup_and_delete_batch)(struct bpf_map *map,

					   const union bpf_attr *attr,

					   union bpf_attr __user *uattr);
@@ -1499,8 +1501,13 @@ int dev_xdp_enqueue(struct net_device *dev, struct xdp_buff *xdp, 
		    struct net_device *dev_rx);

int dev_map_enqueue(struct bpf_dtab_netdev *dst, struct xdp_buff *xdp,

		    struct net_device *dev_rx);

int dev_map_enqueue_multi(struct xdp_buff *xdp, struct net_device *dev_rx,

			  struct bpf_map *map, bool exclude_ingress);

int dev_map_generic_redirect(struct bpf_dtab_netdev *dst, struct sk_buff *skb,

			     struct bpf_prog *xdp_prog);

int dev_map_redirect_multi(struct net_device *dev, struct sk_buff *skb,

			   struct bpf_prog *xdp_prog, struct bpf_map *map,

			   bool exclude_ingress);

bool dev_map_can_have_prog(struct bpf_map *map);



void __cpu_map_flush(void);
@@ -1668,6 +1675,13 @@ int dev_map_enqueue(struct bpf_dtab_netdev *dst, struct xdp_buff *xdp, 
	return 0;

}



static inline

int dev_map_enqueue_multi(struct xdp_buff *xdp, struct net_device *dev_rx,

			  struct bpf_map *map, bool exclude_ingress)

{

	return 0;

}



struct sk_buff;



static inline int dev_map_generic_redirect(struct bpf_dtab_netdev *dst,
@@ -1677,6 +1691,14 @@ static inline int dev_map_generic_redirect(struct bpf_dtab_netdev *dst, 
	return 0;

}



static inline

int dev_map_redirect_multi(struct net_device *dev, struct sk_buff *skb,

			   struct bpf_prog *xdp_prog, struct bpf_map *map,

			   bool exclude_ingress)

{

	return 0;

}



static inline void __cpu_map_flush(void)

{

}
@@ -2026,6 +2048,7 @@ struct sk_reuseport_kern { 
	struct sk_buff *skb;

	struct sock *sk;

	struct sock *selected_sk;

	struct sock *migrating_sk;

	void *data_end;

	u32 hash;

	u32 reuseport_id;

include/linux/bpf_local_storage.h

+2 −2
Original line number Diff line number Diff line
@@ -58,7 +58,7 @@ struct bpf_local_storage_data { 
	 * from the object's bpf_local_storage.

	 *

	 * Put it in the same cacheline as the data to minimize

	 * the number of cachelines access during the cache hit case.

	 * the number of cachelines accessed during the cache hit case.

	 */

	struct bpf_local_storage_map __rcu *smap;

	u8 data[] __aligned(8);
@@ -71,7 +71,7 @@ struct bpf_local_storage_elem { 
	struct bpf_local_storage __rcu *local_storage;

	struct rcu_head rcu;

	/* 8 bytes hole */

	/* The data is stored in aother cacheline to minimize

	/* The data is stored in another cacheline to minimize

	 * the number of cachelines access during a cache hit.

	 */

	struct bpf_local_storage_data sdata ____cacheline_aligned;