Skip to content
  1. Mar 06, 2009
    • Tejun Heo's avatar
      x86, percpu: setup reserved percpu area for x86_64 · 6b19b0c2
      Tejun Heo authored 
      

Impact: fix relocation overflow during module load

x86_64 uses 32bit relocations for symbol access and static percpu
symbols whether in core or modules must be inside 2GB of the percpu
segement base which the dynamic percpu allocator doesn't guarantee.
This patch makes x86_64 reserve PERCPU_MODULE_RESERVE bytes in the
first chunk so that module percpu areas are always allocated from the
first chunk which is always inside the relocatable range.

This problem exists for any percpu allocator but is easily triggered
when using the embedding allocator because the second chunk is located
beyond 2GB on it.

This patch also changes the meaning of PERCPU_DYNAMIC_RESERVE such
that it only indicates the size of the area to reserve for dynamic
allocation as static and dynamic areas can be separate.  New
PERCPU_DYNAMIC_RESERVED is increased by 4k for both 32 and 64bits as
the reserved area separation eats away some allocatable space and
having slightly more headroom (currently between 4 and 8k after
minimal boot sans module area) makes sense for common case
performance.

x86_32 can address anywhere from anywhere and doesn't need reserving.

Mike Galbraith first reported the problem first and bisected it to the
embedding percpu allocator commit.

Signed-off-by: default avatarTejun Heo <tj@kernel.org>
Reported-by: default avatarMike Galbraith <efault@gmx.de>
Reported-by: default avatarJaswinder Singh Rajput <jaswinder@kernel.org>
      6b19b0c2
    • Tejun Heo's avatar
      percpu, module: implement reserved allocation and use it for module percpu variables · edcb4639
      Tejun Heo authored 
      

Impact: add reserved allocation functionality and use it for module
	percpu variables

This patch implements reserved allocation from the first chunk.  When
setting up the first chunk, arch can ask to set aside certain number
of bytes right after the core static area which is available only
through a separate reserved allocator.  This will be used primarily
for module static percpu variables on architectures with limited
relocation range to ensure that the module perpcu symbols are inside
the relocatable range.

If reserved area is requested, the first chunk becomes reserved and
isn't available for regular allocation.  If the first chunk also
includes piggy-back dynamic allocation area, a separate chunk mapping
the same region is created to serve dynamic allocation.  The first one
is called static first chunk and the second dynamic first chunk.
Although they share the page map, their different area map
initializations guarantee they serve disjoint areas according to their
purposes.

If arch doesn't setup reserved area, reserved allocation is handled
like any other allocation.

Signed-off-by: default avatarTejun Heo <tj@kernel.org>
      edcb4639
    • Tejun Heo's avatar
      percpu: add an indirection ptr for chunk page map access · 3e24aa58
      Tejun Heo authored 
      

Impact: allow sharing page map, no functional difference yet

Make chunk->page access indirect by adding a pointer and renaming the
actual array to page_ar.  This will be used by future changes.

Signed-off-by: default avatarTejun Heo <tj@kernel.org>
      3e24aa58
    • Tejun Heo's avatar
      x86: make embedding percpu allocator return excessive free space · 9a4f8a87
      Tejun Heo authored 
      

Impact: reduce unnecessary memory usage on certain configurations

Embedding percpu allocator allocates unit_size *
smp_num_possible_cpus() bytes consecutively and use it for the first
chunk.  However, if the static area is small, this can result in
excessive prellocated free space in the first chunk due to
PCPU_MIN_UNIT_SIZE restriction.

This patch makes embedding percpu allocator preallocate only what's
necessary as described by PERPCU_DYNAMIC_RESERVE and return the
leftover to the bootmem allocator.

Signed-off-by: default avatarTejun Heo <tj@kernel.org>
      9a4f8a87
    • Tejun Heo's avatar
      percpu: use negative for auto for pcpu_setup_first_chunk() arguments · cafe8816
      Tejun Heo authored 
      

Impact: argument semantic cleanup

In pcpu_setup_first_chunk(), zero @unit_size and @dyn_size meant
auto-sizing.  It's okay for @unit_size as 0 doesn't make sense but 0
dynamic reserve size is valid.  Alos, if arch @dyn_size is calculated
from other parameters, it might end up passing in 0 @dyn_size and
malfunction when the size is automatically adjusted.

This patch makes both @unit_size and @dyn_size ssize_t and use -1 for
auto sizing.

Signed-off-by: default avatarTejun Heo <tj@kernel.org>
      cafe8816
    • Tejun Heo's avatar
      percpu: improve first chunk initial area map handling · 61ace7fa
      Tejun Heo authored 
      

Impact: no functional change

When the first chunk is created, its initial area map is not allocated
because kmalloc isn't online yet.  The map is allocated and
initialized on the first allocation request on the chunk.  This works
fine but the scattering of initialization logic between the init
function and allocation path is a bit confusing.

This patch makes the first chunk initialize and use minimal statically
allocated map from pcpu_setpu_first_chunk().  The map resizing path
still needs to handle this specially but it's more straight-forward
and gives more latitude to the init path.  This will ease future
changes.

Signed-off-by: default avatarTejun Heo <tj@kernel.org>
      61ace7fa
    • Tejun Heo's avatar
      percpu: cosmetic renames in pcpu_setup_first_chunk() · 2441d15c
      Tejun Heo authored 
      

Impact: cosmetic, preparation for future changes

Make the following renames in pcpur_setup_first_chunk() in preparation
for future changes.

* s/free_size/dyn_size/
* s/static_vm/first_vm/
* s/static_chunk/schunk/

Signed-off-by: default avatarTejun Heo <tj@kernel.org>
      2441d15c
    • Tejun Heo's avatar
      percpu: clean up percpu constants · 6a242909
      Tejun Heo authored 
      

Impact: cleaup

Make the following cleanups.

* There isn't much arch-specific about PERCPU_MODULE_RESERVE.  Always
  define it whether arch overrides PERCPU_ENOUGH_ROOM or not.

* blackfin overrides PERCPU_ENOUGH_ROOM to align static area size.  Do
  it by default.

* percpu allocation sizes doesn't have much to do with the page size.
  Don't use PAGE_SHIFT in their definition.

Signed-off-by: default avatarTejun Heo <tj@kernel.org>
Cc: Bryan Wu <cooloney@kernel.org>
      6a242909
  3. Mar 04, 2009
  5. Mar 03, 2009