Merge branch 'New nf_conntrack kfuncs for insertion, changing timeout, status'

   faq

   syscall_api

   helpers

   kfuncs

   programs

   maps

   bpf_prog_run

=============================

BPF Kernel Functions (kfuncs)

=============================

1. Introduction

===============

BPF Kernel Functions or more commonly known as kfuncs are functions in the Linux

kernel which are exposed for use by BPF programs. Unlike normal BPF helpers,

kfuncs do not have a stable interface and can change from one kernel release to

another. Hence, BPF programs need to be updated in response to changes in the

kernel.

2. Defining a kfunc

===================

There are two ways to expose a kernel function to BPF programs, either make an

existing function in the kernel visible, or add a new wrapper for BPF. In both

cases, care must be taken that BPF program can only call such function in a

valid context. To enforce this, visibility of a kfunc can be per program type.

If you are not creating a BPF wrapper for existing kernel function, skip ahead

to :ref:`BPF_kfunc_nodef`.

2.1 Creating a wrapper kfunc

----------------------------

When defining a wrapper kfunc, the wrapper function should have extern linkage.

This prevents the compiler from optimizing away dead code, as this wrapper kfunc

is not invoked anywhere in the kernel itself. It is not necessary to provide a

prototype in a header for the wrapper kfunc.

An example is given below::

        /* Disables missing prototype warnings */

        __diag_push();

        __diag_ignore_all("-Wmissing-prototypes",

                          "Global kfuncs as their definitions will be in BTF");

        struct task_struct *bpf_find_get_task_by_vpid(pid_t nr)

        {

                return find_get_task_by_vpid(nr);

        }

        __diag_pop();

A wrapper kfunc is often needed when we need to annotate parameters of the

kfunc. Otherwise one may directly make the kfunc visible to the BPF program by

registering it with the BPF subsystem. See :ref:`BPF_kfunc_nodef`.

2.2 Annotating kfunc parameters

-------------------------------

Similar to BPF helpers, there is sometime need for additional context required

by the verifier to make the usage of kernel functions safer and more useful.

Hence, we can annotate a parameter by suffixing the name of the argument of the

kfunc with a __tag, where tag may be one of the supported annotations.

2.2.1 __sz Annotation

---------------------

This annotation is used to indicate a memory and size pair in the argument list.

An example is given below::

        void bpf_memzero(void *mem, int mem__sz)

        {

        ...

        }

Here, the verifier will treat first argument as a PTR_TO_MEM, and second

argument as its size. By default, without __sz annotation, the size of the type

of the pointer is used. Without __sz annotation, a kfunc cannot accept a void

pointer.

.. _BPF_kfunc_nodef:

2.3 Using an existing kernel function

-------------------------------------

When an existing function in the kernel is fit for consumption by BPF programs,

it can be directly registered with the BPF subsystem. However, care must still

be taken to review the context in which it will be invoked by the BPF program

and whether it is safe to do so.

2.4 Annotating kfuncs

---------------------

In addition to kfuncs' arguments, verifier may need more information about the

type of kfunc(s) being registered with the BPF subsystem. To do so, we define

flags on a set of kfuncs as follows::

        BTF_SET8_START(bpf_task_set)

        BTF_ID_FLAGS(func, bpf_get_task_pid, KF_ACQUIRE | KF_RET_NULL)

        BTF_ID_FLAGS(func, bpf_put_pid, KF_RELEASE)

        BTF_SET8_END(bpf_task_set)

This set encodes the BTF ID of each kfunc listed above, and encodes the flags

along with it. Ofcourse, it is also allowed to specify no flags.

2.4.1 KF_ACQUIRE flag

---------------------

The KF_ACQUIRE flag is used to indicate that the kfunc returns a pointer to a

refcounted object. The verifier will then ensure that the pointer to the object

is eventually released using a release kfunc, or transferred to a map using a

referenced kptr (by invoking bpf_kptr_xchg). If not, the verifier fails the

loading of the BPF program until no lingering references remain in all possible

explored states of the program.

2.4.2 KF_RET_NULL flag

----------------------

The KF_RET_NULL flag is used to indicate that the pointer returned by the kfunc

may be NULL. Hence, it forces the user to do a NULL check on the pointer

returned from the kfunc before making use of it (dereferencing or passing to

another helper). This flag is often used in pairing with KF_ACQUIRE flag, but

both are orthogonal to each other.

2.4.3 KF_RELEASE flag

---------------------

The KF_RELEASE flag is used to indicate that the kfunc releases the pointer

passed in to it. There can be only one referenced pointer that can be passed in.

All copies of the pointer being released are invalidated as a result of invoking

kfunc with this flag.

2.4.4 KF_KPTR_GET flag

----------------------

The KF_KPTR_GET flag is used to indicate that the kfunc takes the first argument

as a pointer to kptr, safely increments the refcount of the object it points to,

and returns a reference to the user. The rest of the arguments may be normal

arguments of a kfunc. The KF_KPTR_GET flag should be used in conjunction with

KF_ACQUIRE and KF_RET_NULL flags.

2.4.5 KF_TRUSTED_ARGS flag

--------------------------

The KF_TRUSTED_ARGS flag is used for kfuncs taking pointer arguments. It

indicates that the all pointer arguments will always be refcounted, and have

their offset set to 0. It can be used to enforce that a pointer to a refcounted

object acquired from a kfunc or BPF helper is passed as an argument to this

kfunc without any modifications (e.g. pointer arithmetic) such that it is

trusted and points to the original object. This flag is often used for kfuncs

that operate (change some property, perform some operation) on an object that

was obtained using an acquire kfunc. Such kfuncs need an unchanged pointer to

ensure the integrity of the operation being performed on the expected object.

2.5 Registering the kfuncs

--------------------------

Once the kfunc is prepared for use, the final step to making it visible is

registering it with the BPF subsystem. Registration is done per BPF program

type. An example is shown below::

        BTF_SET8_START(bpf_task_set)

        BTF_ID_FLAGS(func, bpf_get_task_pid, KF_ACQUIRE | KF_RET_NULL)

        BTF_ID_FLAGS(func, bpf_put_pid, KF_RELEASE)

        BTF_SET8_END(bpf_task_set)

        static const struct btf_kfunc_id_set bpf_task_kfunc_set = {

                .owner = THIS_MODULE,

                .set   = &bpf_task_set,

        };

        static int init_subsystem(void)

        {

                return register_btf_kfunc_id_set(BPF_PROG_TYPE_TRACING, &bpf_task_kfunc_set);

        }

        late_initcall(init_subsystem);

				struct bpf_reg_state *regs);

int btf_check_kfunc_arg_match(struct bpf_verifier_env *env,

			      const struct btf *btf, u32 func_id,

			      struct bpf_reg_state *regs);

			      struct bpf_reg_state *regs,

			      u32 kfunc_flags);

int btf_prepare_func_args(struct bpf_verifier_env *env, int subprog,

			  struct bpf_reg_state *reg);

int btf_check_type_match(struct bpf_verifier_log *log, const struct bpf_prog *prog,

#define BTF_TYPE_EMIT(type) ((void)(type *)0)

#define BTF_TYPE_EMIT_ENUM(enum_val) ((void)enum_val)

enum btf_kfunc_type {

	BTF_KFUNC_TYPE_CHECK,

	BTF_KFUNC_TYPE_ACQUIRE,

	BTF_KFUNC_TYPE_RELEASE,

	BTF_KFUNC_TYPE_RET_NULL,

	BTF_KFUNC_TYPE_KPTR_ACQUIRE,

	BTF_KFUNC_TYPE_MAX,

};

/* These need to be macros, as the expressions are used in assembler input */

#define KF_ACQUIRE	(1 << 0) /* kfunc is an acquire function */

#define KF_RELEASE	(1 << 1) /* kfunc is a release function */

#define KF_RET_NULL	(1 << 2) /* kfunc returns a pointer that may be NULL */

#define KF_KPTR_GET	(1 << 3) /* kfunc returns reference to a kptr */

/* Trusted arguments are those which are meant to be referenced arguments with

 * unchanged offset. It is used to enforce that pointers obtained from acquire

 * kfuncs remain unmodified when being passed to helpers taking trusted args.

 *

 * Consider

 *	struct foo {

 *		int data;

 *		struct foo *next;

 *	};

 *

 *	struct bar {

 *		int data;

 *		struct foo f;

 *	};

 *

 *	struct foo *f = alloc_foo(); // Acquire kfunc

 *	struct bar *b = alloc_bar(); // Acquire kfunc

 *

 * If a kfunc set_foo_data() wants to operate only on the allocated object, it

 * will set the KF_TRUSTED_ARGS flag, which will prevent unsafe usage like:

 *

 *	set_foo_data(f, 42);	   // Allowed

 *	set_foo_data(f->next, 42); // Rejected, non-referenced pointer

 *	set_foo_data(&f->next, 42);// Rejected, referenced, but wrong type

 *	set_foo_data(&b->f, 42);   // Rejected, referenced, but bad offset

 *

 * In the final case, usually for the purposes of type matching, it is deduced

 * by looking at the type of the member at the offset, but due to the

 * requirement of trusted argument, this deduction will be strict and not done

 * for this case.

 */

#define KF_TRUSTED_ARGS (1 << 4) /* kfunc only takes trusted pointer arguments */

struct btf;

struct btf_member;

struct btf_kfunc_id_set {

	struct module *owner;

	union {

		struct {

			struct btf_id_set *check_set;

			struct btf_id_set *acquire_set;

			struct btf_id_set *release_set;

			struct btf_id_set *ret_null_set;

			struct btf_id_set *kptr_acquire_set;

		};

		struct btf_id_set *sets[BTF_KFUNC_TYPE_MAX];

	};

	struct btf_id_set8 *set;

};

struct btf_id_dtor_kfunc {

const char *btf_name_by_offset(const struct btf *btf, u32 offset);

struct btf *btf_parse_vmlinux(void);

struct btf *bpf_prog_get_target_btf(const struct bpf_prog *prog);

bool btf_kfunc_id_set_contains(const struct btf *btf,

u32 *btf_kfunc_id_set_contains(const struct btf *btf,

			       enum bpf_prog_type prog_type,

			       enum btf_kfunc_type type, u32 kfunc_btf_id);

			       u32 kfunc_btf_id);

int register_btf_kfunc_id_set(enum bpf_prog_type prog_type,

			      const struct btf_kfunc_id_set *s);

s32 btf_find_dtor_kfunc(struct btf *btf, u32 btf_id);

{

	return NULL;

}

static inline bool btf_kfunc_id_set_contains(const struct btf *btf,

static inline u32 *btf_kfunc_id_set_contains(const struct btf *btf,

					     enum bpf_prog_type prog_type,

					     enum btf_kfunc_type type,

					     u32 kfunc_btf_id)

{

	return false;

	return NULL;

}

static inline int register_btf_kfunc_id_set(enum bpf_prog_type prog_type,

					    const struct btf_kfunc_id_set *s)

	u32 ids[];

};

struct btf_id_set8 {

	u32 cnt;

	u32 flags;

	struct {

		u32 id;

		u32 flags;

	} pairs[];

};

#ifdef CONFIG_DEBUG_INFO_BTF

#include <linux/compiler.h> /* for __PASTE */

#define BTF_IDS_SECTION ".BTF_ids"

#define ____BTF_ID(symbol)				\

#define ____BTF_ID(symbol, word)			\

asm(							\

".pushsection " BTF_IDS_SECTION ",\"a\";       \n"	\

".local " #symbol " ;                          \n"	\

".size  " #symbol ", 4;                        \n"	\

#symbol ":                                     \n"	\

".zero 4                                       \n"	\

word							\

".popsection;                                  \n");

#define __BTF_ID(symbol) \

	____BTF_ID(symbol)

#define __BTF_ID(symbol, word) \

	____BTF_ID(symbol, word)

#define __ID(prefix) \

	__PASTE(prefix, __COUNTER__)

 * to 4 zero bytes.

 */

#define BTF_ID(prefix, name) \

	__BTF_ID(__ID(__BTF_ID__##prefix##__##name##__))

	__BTF_ID(__ID(__BTF_ID__##prefix##__##name##__), "")

#define ____BTF_ID_FLAGS(prefix, name, flags) \

	__BTF_ID(__ID(__BTF_ID__##prefix##__##name##__), ".long " #flags "\n")

#define __BTF_ID_FLAGS(prefix, name, flags, ...) \

	____BTF_ID_FLAGS(prefix, name, flags)

#define BTF_ID_FLAGS(prefix, name, ...) \

	__BTF_ID_FLAGS(prefix, name, ##__VA_ARGS__, 0)

/*

 * The BTF_ID_LIST macro defines pure (unsorted) list

".popsection;                                 \n");	\

extern struct btf_id_set name;

/*

 * The BTF_SET8_START/END macros pair defines sorted list of

 * BTF IDs and their flags plus its members count, with the

 * following layout:

 *

 * BTF_SET8_START(list)

 * BTF_ID_FLAGS(type1, name1, flags)

 * BTF_ID_FLAGS(type2, name2, flags)

 * BTF_SET8_END(list)

 *

 * __BTF_ID__set8__list:

 * .zero 8

 * list:

 * __BTF_ID__type1__name1__3:

 * .zero 4

 * .word (1 << 0) | (1 << 2)

 * __BTF_ID__type2__name2__5:

 * .zero 4

 * .word (1 << 3) | (1 << 1) | (1 << 2)

 *

 */

#define __BTF_SET8_START(name, scope)			\

asm(							\

".pushsection " BTF_IDS_SECTION ",\"a\";       \n"	\

"." #scope " __BTF_ID__set8__" #name ";        \n"	\

"__BTF_ID__set8__" #name ":;                   \n"	\

".zero 8                                       \n"	\

".popsection;                                  \n");

#define BTF_SET8_START(name)				\

__BTF_ID_LIST(name, local)				\

__BTF_SET8_START(name, local)

#define BTF_SET8_END(name)				\

asm(							\

".pushsection " BTF_IDS_SECTION ",\"a\";      \n"	\

".size __BTF_ID__set8__" #name ", .-" #name "  \n"	\

".popsection;                                 \n");	\

extern struct btf_id_set8 name;

#else

#define BTF_ID_LIST(name) static u32 __maybe_unused name[5];

#define BTF_ID(prefix, name)

#define BTF_ID_FLAGS(prefix, name, flags)

#define BTF_ID_UNUSED

#define BTF_ID_LIST_GLOBAL(name, n) u32 __maybe_unused name[n];

#define BTF_ID_LIST_SINGLE(name, prefix, typename) static u32 __maybe_unused name[1];

#define BTF_SET_START(name) static struct btf_id_set __maybe_unused name = { 0 };

#define BTF_SET_START_GLOBAL(name) static struct btf_id_set __maybe_unused name = { 0 };

#define BTF_SET_END(name)

#define BTF_SET8_START(name) static struct btf_id_set8 __maybe_unused name = { 0 };

#define BTF_SET8_END(name) static struct btf_id_set8 __maybe_unused name = { 0 };

#endif /* CONFIG_DEBUG_INFO_BTF */