Merge branches 'bitmaprange.2021.05.10c', 'doc.2021.05.10c',...

to see the effects of all accesses prior to the beginning of that grace

period that are within RCU read-side critical sections.

Similarly, any code that happens before the beginning of a given RCU grace

period is guaranteed to see the effects of all accesses following the end

period is guaranteed to not see the effects of all accesses following the end

of that grace period that are within RCU read-side critical sections.

Note well that RCU-sched read-side critical sections include any region

leftmost ``rcu_node`` structure offlines its last CPU and if the next

``rcu_node`` structure has no online CPUs).

.. kernel-figure:: TreeRCU-gp-init-1.svg

.. kernel-figure:: TreeRCU-gp-init-2.svg

The final ``rcu_gp_init()`` pass through the ``rcu_node`` tree traverses

breadth-first, setting each ``rcu_node`` structure's ``->gp_seq`` field

to the newly advanced value from the ``rcu_state`` structure, as shown

in the following diagram.

.. kernel-figure:: TreeRCU-gp-init-1.svg

.. kernel-figure:: TreeRCU-gp-init-3.svg

This change will also cause each CPU's next call to

``__note_gp_changes()`` to notice that a new grace period has started,

			whole algorithm to behave better in low memory

			condition.

	rcutree.rcu_delay_page_cache_fill_msec= [KNL]

			Set the page-cache refill delay (in milliseconds)

			in response to low-memory conditions.  The range

			of permitted values is in the range 0:100000.

	rcutree.jiffies_till_first_fqs= [KNL]

			Set delay from grace-period initialization to

			first attempt to force quiescent states.

#define RCU_LOCKDEP_WARN(c, s)						\

	do {								\

		static bool __section(".data.unlikely") __warned;	\

		if (debug_lockdep_rcu_enabled() && !__warned && (c)) {	\

		if ((c) && debug_lockdep_rcu_enabled() && !__warned) {	\

			__warned = true;				\

			lockdep_rcu_suspicious(__FILE__, __LINE__, s);	\

		}							\

#define rcu_check_sparse(p, space)

#endif /* #else #ifdef __CHECKER__ */

/**

 * unrcu_pointer - mark a pointer as not being RCU protected

 * @p: pointer needing to lose its __rcu property

 *

 * Converts @p from an __rcu pointer to a __kernel pointer.

 * This allows an __rcu pointer to be used with xchg() and friends.

 */

#define unrcu_pointer(p)						\

({									\

	typeof(*p) *_________p1 = (typeof(*p) *__force)(p);		\

	rcu_check_sparse(p, __rcu);					\

	((typeof(*p) __force __kernel *)(_________p1)); 		\

})

#define __rcu_access_pointer(p, space) \

({ \

	typeof(*p) *_________p1 = (typeof(*p) *__force)READ_ONCE(p); \

 * @p: The pointer to read, prior to dereferencing

 * @c: The conditions under which the dereference will take place

 *

 * This is the RCU-bh counterpart to rcu_dereference_check().

 * This is the RCU-bh counterpart to rcu_dereference_check().  However,

 * please note that starting in v5.0 kernels, vanilla RCU grace periods

 * wait for local_bh_disable() regions of code in addition to regions of

 * code demarked by rcu_read_lock() and rcu_read_unlock().  This means

 * that synchronize_rcu(), call_rcu, and friends all take not only

 * rcu_read_lock() but also rcu_read_lock_bh() into account.

 */

#define rcu_dereference_bh_check(p, c) \

	__rcu_dereference_check((p), (c) || rcu_read_lock_bh_held(), __rcu)

 * @c: The conditions under which the dereference will take place

 *

 * This is the RCU-sched counterpart to rcu_dereference_check().

 * However, please note that starting in v5.0 kernels, vanilla RCU grace

 * periods wait for preempt_disable() regions of code in addition to

 * regions of code demarked by rcu_read_lock() and rcu_read_unlock().

 * This means that synchronize_rcu(), call_rcu, and friends all take not

 * only rcu_read_lock() but also rcu_read_lock_sched() into account.

 */

#define rcu_dereference_sched_check(p, c) \

	__rcu_dereference_check((p), (c) || rcu_read_lock_sched_held(), \

 * sections, invocation of the corresponding RCU callback is deferred

 * until after the all the other CPUs exit their critical sections.

 *

 * In v5.0 and later kernels, synchronize_rcu() and call_rcu() also

 * wait for regions of code with preemption disabled, including regions of

 * code with interrupts or softirqs disabled.  In pre-v5.0 kernels, which

 * define synchronize_sched(), only code enclosed within rcu_read_lock()

 * and rcu_read_unlock() are guaranteed to be waited for.

 *

 * Note, however, that RCU callbacks are permitted to run concurrently

 * with new RCU read-side critical sections.  One way that this can happen

 * is via the following sequence of events: (1) CPU 0 enters an RCU

/**

 * rcu_read_unlock() - marks the end of an RCU read-side critical section.

 *

 * In most situations, rcu_read_unlock() is immune from deadlock.

 * However, in kernels built with CONFIG_RCU_BOOST, rcu_read_unlock()

 * is responsible for deboosting, which it does via rt_mutex_unlock().

 * Unfortunately, this function acquires the scheduler's runqueue and

 * priority-inheritance spinlocks.  This means that deadlock could result

 * if the caller of rcu_read_unlock() already holds one of these locks or

 * any lock that is ever acquired while holding them.

 *

 * That said, RCU readers are never priority boosted unless they were

 * preempted.  Therefore, one way to avoid deadlock is to make sure

 * that preemption never happens within any RCU read-side critical

 * section whose outermost rcu_read_unlock() is called with one of

 * rt_mutex_unlock()'s locks held.  Such preemption can be avoided in

 * a number of ways, for example, by invoking preempt_disable() before

 * critical section's outermost rcu_read_lock().

 *

 * Given that the set of locks acquired by rt_mutex_unlock() might change

 * at any time, a somewhat more future-proofed approach is to make sure

 * that that preemption never happens within any RCU read-side critical

 * section whose outermost rcu_read_unlock() is called with irqs disabled.

 * This approach relies on the fact that rt_mutex_unlock() currently only

 * acquires irq-disabled locks.

 *

 * The second of these two approaches is best in most situations,

 * however, the first approach can also be useful, at least to those

 * developers willing to keep abreast of the set of locks acquired by

 * rt_mutex_unlock().

 * In almost all situations, rcu_read_unlock() is immune from deadlock.

 * In recent kernels that have consolidated synchronize_sched() and

 * synchronize_rcu_bh() into synchronize_rcu(), this deadlock immunity

 * also extends to the scheduler's runqueue and priority-inheritance

 * spinlocks, courtesy of the quiescent-state deferral that is carried

 * out when rcu_read_unlock() is invoked with interrupts disabled.

 *

 * See rcu_read_lock() for more information.

 */

/**

 * rcu_read_lock_bh() - mark the beginning of an RCU-bh critical section

 *

 * This is equivalent of rcu_read_lock(), but also disables softirqs.

 * Note that anything else that disables softirqs can also serve as

 * an RCU read-side critical section.

 * This is equivalent to rcu_read_lock(), but also disables softirqs.

 * Note that anything else that disables softirqs can also serve as an RCU

 * read-side critical section.  However, please note that this equivalence

 * applies only to v5.0 and later.  Before v5.0, rcu_read_lock() and

 * rcu_read_lock_bh() were unrelated.

 *

 * Note that rcu_read_lock_bh() and the matching rcu_read_unlock_bh()

 * must occur in the same context, for example, it is illegal to invoke

/**

 * rcu_read_lock_sched() - mark the beginning of a RCU-sched critical section

 *

 * This is equivalent of rcu_read_lock(), but disables preemption.

 * Read-side critical sections can also be introduced by anything else

 * that disables preemption, including local_irq_disable() and friends.

 * This is equivalent to rcu_read_lock(), but also disables preemption.

 * Read-side critical sections can also be introduced by anything else that

 * disables preemption, including local_irq_disable() and friends.  However,

 * please note that the equivalence to rcu_read_lock() applies only to

 * v5.0 and later.  Before v5.0, rcu_read_lock() and rcu_read_lock_sched()

 * were unrelated.

 *

 * Note that rcu_read_lock_sched() and the matching rcu_read_unlock_sched()

 * must occur in the same context, for example, it is illegal to invoke

static inline void rcu_irq_exit_irqson(void) { }

static inline void rcu_irq_enter_irqson(void) { }

static inline void rcu_irq_exit(void) { }

static inline void rcu_irq_exit_preempt(void) { }

static inline void rcu_irq_exit_check_preempt(void) { }

#define rcu_is_idle_cpu(cpu) \

	(is_idle_task(current) && !in_nmi() && !in_irq() && !in_serving_softirq())

void rcu_idle_exit(void);

void rcu_irq_enter(void);

void rcu_irq_exit(void);

void rcu_irq_exit_preempt(void);

void rcu_irq_enter_irqson(void);

void rcu_irq_exit_irqson(void);

bool rcu_is_idle_cpu(int cpu);