Commit a48b4bf9 authored by Alexei Starovoitov's avatar Alexei Starovoitov
Browse files

Merge branch 'bpf: Implement cgroup local storage available to non-cgroup-attached bpf progs' 



Yonghong Song says:

====================

There already exists a local storage implementation for cgroup-attached
bpf programs. See map type BPF_MAP_TYPE_CGROUP_STORAGE and helper
bpf_get_local_storage(). But there are use cases such that non-cgroup
attached bpf progs wants to access cgroup local storage data. For example,
tc egress prog has access to sk and cgroup. It is possible to use
sk local storage to emulate cgroup local storage by storing data in socket.
But this is a waste as it could be lots of sockets belonging to a particular
cgroup. Alternatively, a separate map can be created with cgroup id as the key.
But this will introduce additional overhead to manipulate the new map.
A cgroup local storage, similar to existing sk/inode/task storage,
should help for this use case.

This patch implemented new cgroup local storage available to
non-cgroup-attached bpf programs. In the patch series, Patches 1 and 2
are preparation patches. Patch 3 implemented new cgroup local storage
kernel support. Patches 4 and 5 implemented libbpf and bpftool support.
Patches 6-8 fixed one existing test and added four new tests to validate
kernel/libbpf implementations. Patch 9 added documentation for new
BPF_MAP_TYPE_CGRP_STORAGE map type and comparison of the old and new
cgroup local storage maps.

Changelogs:
  v5 -> v6:
    . fix selftest test_libbpf_str/bpf_map_type_str due to marking
      BPF_MAP_TYPE_CGROUP_STORAGE as deprecated.
    . add cgrp_local_storage test in s390x denylist since the test
      has some fentry/fexit programs.
  v4 -> v5:
    . additional refactoring in patch 2
    . fix the call site for bpf_cgrp_storage_free() in kernel/cgroup/cgroup.c.
    . add a test for progs attaching to cgroups
    . add a negative test (the helper key is a task instead of expected cgroup)
    . some spelling fixes
  v3 -> v4:
    . fix a config guarding problem in kernel/cgroup/cgroup.c when
      cgrp_storage is deleted (CONFIG_CGROUP_BPF => CONFIG_BPF_SYSCALL).
    . rename selftest from cgroup_local_storage.c to cgrp_local_storage.c
      so the name can better align with map name.
    . fix a few misspellings.
  v2 -> v3:
    . fix a config caused kernel test complaint.
    . better description/comments in uapi bpf.h and bpf_cgrp_storage.c.
    . factor code for better resue for map_alloc/map_free.
    . improved explanation in map documentation.
  v1 -> v2:
    . change map name from BPF_MAP_TYPE_CGROUP_LOCAL_STORAGE to
      BPF_MAP_TYPE_CGRP_STORAGE.
    . removed support of sleepable programs.
    . changed the place of freeing cgrp local storage from put_css_set_locked()
      to css_free_rwork_fn().
    . added map documentation.
====================

Signed-off-by: default avatarAlexei Starovoitov <ast@kernel.org>
parents fdf45787 d4319801

Documentation/bpf/map_cgrp_storage.rst

0 → 100644
+109 −0
Original line number Diff line number Diff line 
.. SPDX-License-Identifier: GPL-2.0-only

.. Copyright (C) 2022 Meta Platforms, Inc. and affiliates.



=========================

BPF_MAP_TYPE_CGRP_STORAGE

=========================



The ``BPF_MAP_TYPE_CGRP_STORAGE`` map type represents a local fix-sized

storage for cgroups. It is only available with ``CONFIG_CGROUPS``.

The programs are made available by the same Kconfig. The

data for a particular cgroup can be retrieved by looking up the map

with that cgroup.



This document describes the usage and semantics of the

``BPF_MAP_TYPE_CGRP_STORAGE`` map type.



Usage

=====



The map key must be ``sizeof(int)`` representing a cgroup fd.

To access the storage in a program, use ``bpf_cgrp_storage_get``::



    void *bpf_cgrp_storage_get(struct bpf_map *map, struct cgroup *cgroup, void *value, u64 flags)



``flags`` could be 0 or ``BPF_LOCAL_STORAGE_GET_F_CREATE`` which indicates that

a new local storage will be created if one does not exist.



The local storage can be removed with ``bpf_cgrp_storage_delete``::



    long bpf_cgrp_storage_delete(struct bpf_map *map, struct cgroup *cgroup)



The map is available to all program types.



Examples

========



A BPF program example with BPF_MAP_TYPE_CGRP_STORAGE::



    #include <vmlinux.h>

    #include <bpf/bpf_helpers.h>

    #include <bpf/bpf_tracing.h>



    struct {

            __uint(type, BPF_MAP_TYPE_CGRP_STORAGE);

            __uint(map_flags, BPF_F_NO_PREALLOC);

            __type(key, int);

            __type(value, long);

    } cgrp_storage SEC(".maps");



    SEC("tp_btf/sys_enter")

    int BPF_PROG(on_enter, struct pt_regs *regs, long id)

    {

            struct task_struct *task = bpf_get_current_task_btf();

            long *ptr;



            ptr = bpf_cgrp_storage_get(&cgrp_storage, task->cgroups->dfl_cgrp, 0,

                                       BPF_LOCAL_STORAGE_GET_F_CREATE);

            if (ptr)

                __sync_fetch_and_add(ptr, 1);



            return 0;

    }



Userspace accessing map declared above::



    #include <linux/bpf.h>

    #include <linux/libbpf.h>



    __u32 map_lookup(struct bpf_map *map, int cgrp_fd)

    {

            __u32 *value;

            value = bpf_map_lookup_elem(bpf_map__fd(map), &cgrp_fd);

            if (value)

                return *value;

            return 0;

    }



Difference Between BPF_MAP_TYPE_CGRP_STORAGE and BPF_MAP_TYPE_CGROUP_STORAGE

============================================================================



The old cgroup storage map ``BPF_MAP_TYPE_CGROUP_STORAGE`` has been marked as

deprecated (renamed to ``BPF_MAP_TYPE_CGROUP_STORAGE_DEPRECATED``). The new

``BPF_MAP_TYPE_CGRP_STORAGE`` map should be used instead. The following

illusates the main difference between ``BPF_MAP_TYPE_CGRP_STORAGE`` and

``BPF_MAP_TYPE_CGROUP_STORAGE_DEPRECATED``.



(1). ``BPF_MAP_TYPE_CGRP_STORAGE`` can be used by all program types while

     ``BPF_MAP_TYPE_CGROUP_STORAGE_DEPRECATED`` is available only to cgroup program types

     like BPF_CGROUP_INET_INGRESS or BPF_CGROUP_SOCK_OPS, etc.



(2). ``BPF_MAP_TYPE_CGRP_STORAGE`` supports local storage for more than one

     cgroup while ``BPF_MAP_TYPE_CGROUP_STORAGE_DEPRECATED`` only supports one cgroup

     which is attached by a BPF program.



(3). ``BPF_MAP_TYPE_CGROUP_STORAGE_DEPRECATED`` allocates local storage at attach time so

     ``bpf_get_local_storage()`` always returns non-NULL local storage.

     ``BPF_MAP_TYPE_CGRP_STORAGE`` allocates local storage at runtime so

     it is possible that ``bpf_cgrp_storage_get()`` may return null local storage.

     To avoid such null local storage issue, user space can do

     ``bpf_map_update_elem()`` to pre-allocate local storage before a BPF program

     is attached.



(4). ``BPF_MAP_TYPE_CGRP_STORAGE`` supports deleting local storage by a BPF program

     while ``BPF_MAP_TYPE_CGROUP_STORAGE_DEPRECATED`` only deletes storage during

     prog detach time.



So overall, ``BPF_MAP_TYPE_CGRP_STORAGE`` supports all ``BPF_MAP_TYPE_CGROUP_STORAGE_DEPRECATED``

functionality and beyond. It is recommended to use ``BPF_MAP_TYPE_CGRP_STORAGE``

instead of ``BPF_MAP_TYPE_CGROUP_STORAGE_DEPRECATED``.

include/linux/bpf.h

+7 −0
Original line number Diff line number Diff line
@@ -2041,6 +2041,7 @@ struct bpf_link *bpf_link_by_id(u32 id); 


const struct bpf_func_proto *bpf_base_func_proto(enum bpf_func_id func_id);

void bpf_task_storage_free(struct task_struct *task);

void bpf_cgrp_storage_free(struct cgroup *cgroup);

bool bpf_prog_has_kfunc_call(const struct bpf_prog *prog);

const struct btf_func_model *

bpf_jit_find_kfunc_model(const struct bpf_prog *prog,
@@ -2295,6 +2296,10 @@ static inline bool has_current_bpf_ctx(void) 
static inline void bpf_prog_inc_misses_counter(struct bpf_prog *prog)

{

}



static inline void bpf_cgrp_storage_free(struct cgroup *cgroup)

{

}

#endif /* CONFIG_BPF_SYSCALL */



void __bpf_free_used_btfs(struct bpf_prog_aux *aux,
@@ -2535,6 +2540,8 @@ extern const struct bpf_func_proto bpf_copy_from_user_task_proto; 
extern const struct bpf_func_proto bpf_set_retval_proto;

extern const struct bpf_func_proto bpf_get_retval_proto;

extern const struct bpf_func_proto bpf_user_ringbuf_drain_proto;

extern const struct bpf_func_proto bpf_cgrp_storage_get_proto;

extern const struct bpf_func_proto bpf_cgrp_storage_delete_proto;



const struct bpf_func_proto *tracing_prog_func_proto(

  enum bpf_func_id func_id, const struct bpf_prog *prog);

include/linux/bpf_local_storage.h

+7 −10
Original line number Diff line number Diff line
@@ -116,21 +116,22 @@ static struct bpf_local_storage_cache name = { \ 
	.idx_lock = __SPIN_LOCK_UNLOCKED(name.idx_lock),	\

}



u16 bpf_local_storage_cache_idx_get(struct bpf_local_storage_cache *cache);

void bpf_local_storage_cache_idx_free(struct bpf_local_storage_cache *cache,

				      u16 idx);



/* Helper functions for bpf_local_storage */

int bpf_local_storage_map_alloc_check(union bpf_attr *attr);



struct bpf_local_storage_map *bpf_local_storage_map_alloc(union bpf_attr *attr);

struct bpf_map *

bpf_local_storage_map_alloc(union bpf_attr *attr,

			    struct bpf_local_storage_cache *cache);



struct bpf_local_storage_data *

bpf_local_storage_lookup(struct bpf_local_storage *local_storage,

			 struct bpf_local_storage_map *smap,

			 bool cacheit_lockit);



void bpf_local_storage_map_free(struct bpf_local_storage_map *smap,

bool bpf_local_storage_unlink_nolock(struct bpf_local_storage *local_storage);



void bpf_local_storage_map_free(struct bpf_map *map,

				struct bpf_local_storage_cache *cache,

				int __percpu *busy_counter);



int bpf_local_storage_map_check_btf(const struct bpf_map *map,
@@ -141,10 +142,6 @@ int bpf_local_storage_map_check_btf(const struct bpf_map *map, 
void bpf_selem_link_storage_nolock(struct bpf_local_storage *local_storage,

				   struct bpf_local_storage_elem *selem);



bool bpf_selem_unlink_storage_nolock(struct bpf_local_storage *local_storage,

				     struct bpf_local_storage_elem *selem,

				     bool uncharge_omem, bool use_trace_rcu);



void bpf_selem_unlink(struct bpf_local_storage_elem *selem, bool use_trace_rcu);



void bpf_selem_link_map(struct bpf_local_storage_map *smap,

include/linux/bpf_types.h

+1 −0
Original line number Diff line number Diff line
@@ -86,6 +86,7 @@ BPF_MAP_TYPE(BPF_MAP_TYPE_PROG_ARRAY, prog_array_map_ops) 
BPF_MAP_TYPE(BPF_MAP_TYPE_PERF_EVENT_ARRAY, perf_event_array_map_ops)

#ifdef CONFIG_CGROUPS

BPF_MAP_TYPE(BPF_MAP_TYPE_CGROUP_ARRAY, cgroup_array_map_ops)

BPF_MAP_TYPE(BPF_MAP_TYPE_CGRP_STORAGE, cgrp_storage_map_ops)

#endif

#ifdef CONFIG_CGROUP_BPF

BPF_MAP_TYPE(BPF_MAP_TYPE_CGROUP_STORAGE, cgroup_storage_map_ops)

include/linux/btf_ids.h

+1 −0
Original line number Diff line number Diff line
@@ -265,5 +265,6 @@ MAX_BTF_TRACING_TYPE, 
};



extern u32 btf_tracing_ids[];

extern u32 bpf_cgroup_btf_id[];



#endif