Merge branch 'lkmm.2020.11.06a' into HEAD

     These are for use with atomic RMW functions that do not imply memory

     barriers, but where the code needs a memory barrier. Examples for atomic

     RMW functions that do not imply are memory barrier are e.g. add,

     RMW functions that do not imply a memory barrier are e.g. add,

     subtract, (failed) conditional operations, _relaxed functions,

     but not atomic_read or atomic_set. A common example where a memory

     barrier may be required is when atomic ops are used for reference

It has been said that successful communication requires first identifying

what your audience knows and then building a bridge from their current

knowledge to what they need to know.  Unfortunately, the expected

Linux-kernel memory model (LKMM) audience might be anywhere from novice

to expert both in kernel hacking and in understanding LKMM.

This document therefore points out a number of places to start reading,

depending on what you know and what you would like to learn.  Please note

that the documents later in this list assume that the reader understands

the material provided by documents earlier in this list.

o	You are new to Linux-kernel concurrency: simple.txt

o	You have some background in Linux-kernel concurrency, and would

	like an overview of the types of low-level concurrency primitives

	that the Linux kernel provides:  ordering.txt

	Here, "low level" means atomic operations to single variables.

o	You are familiar with the Linux-kernel concurrency primitives

	that you need, and just want to get started with LKMM litmus

	tests:  litmus-tests.txt

o	You are familiar with Linux-kernel concurrency, and would

	like a detailed intuitive understanding of LKMM, including

	situations involving more than two threads:  recipes.txt

o	You would like a detailed understanding of what your compiler can

	and cannot do to control dependencies:  control-dependencies.txt

o	You are familiar with Linux-kernel concurrency and the use of

	LKMM, and would like a quick reference:  cheatsheet.txt

o	You are familiar with Linux-kernel concurrency and the use

	of LKMM, and would like to learn about LKMM's requirements,

	rationale, and implementation:	explanation.txt

o	You are interested in the publications related to LKMM, including

	hardware manuals, academic literature, standards-committee

	working papers, and LWN articles:  references.txt

====================

DESCRIPTION OF FILES

====================

README

	This file.

cheatsheet.txt

	Quick-reference guide to the Linux-kernel memory model.

control-dependencies.txt

	Guide to preventing compiler optimizations from destroying

	your control dependencies.

explanation.txt

	Detailed description of the memory model.

litmus-tests.txt

	The format, features, capabilities, and limitations of the litmus

	tests that LKMM can evaluate.

ordering.txt

	Overview of the Linux kernel's low-level memory-ordering

	primitives by category.

recipes.txt

	Common memory-ordering patterns.

references.txt

	Background information.

simple.txt

	Starting point for someone new to Linux-kernel concurrency.

	And also a reminder of the simpler approaches to concurrency!

CONTROL DEPENDENCIES

====================

A major difficulty with control dependencies is that current compilers

do not support them.  One purpose of this document is therefore to

help you prevent your compiler from breaking your code.  However,

control dependencies also pose other challenges, which leads to the

second purpose of this document, namely to help you to avoid breaking

your own code, even in the absence of help from your compiler.

One such challenge is that control dependencies order only later stores.

Therefore, a load-load control dependency will not preserve ordering

unless a read memory barrier is provided.  Consider the following code:

	q = READ_ONCE(a);

	if (q)

		p = READ_ONCE(b);

This is not guaranteed to provide any ordering because some types of CPUs

are permitted to predict the result of the load from "b".  This prediction

can cause other CPUs to see this load as having happened before the load

from "a".  This means that an explicit read barrier is required, for example

as follows:

	q = READ_ONCE(a);

	if (q) {

		smp_rmb();

		p = READ_ONCE(b);

	}

However, stores are not speculated.  This means that ordering is

(usually) guaranteed for load-store control dependencies, as in the

following example:

	q = READ_ONCE(a);

	if (q)

		WRITE_ONCE(b, 1);

Control dependencies can pair with each other and with other types

of ordering.  But please note that neither the READ_ONCE() nor the

WRITE_ONCE() are optional.  Without the READ_ONCE(), the compiler might

fuse the load from "a" with other loads.  Without the WRITE_ONCE(),

the compiler might fuse the store to "b" with other stores.  Worse yet,

the compiler might convert the store into a load and a check followed

by a store, and this compiler-generated load would not be ordered by

the control dependency.

Furthermore, if the compiler is able to prove that the value of variable

"a" is always non-zero, it would be well within its rights to optimize

the original example by eliminating the "if" statement as follows:

	q = a;

	b = 1;  /* BUG: Compiler and CPU can both reorder!!! */

So don't leave out either the READ_ONCE() or the WRITE_ONCE().

In particular, although READ_ONCE() does force the compiler to emit a

load, it does *not* force the compiler to actually use the loaded value.

It is tempting to try use control dependencies to enforce ordering on

identical stores on both branches of the "if" statement as follows:

	q = READ_ONCE(a);

	if (q) {

		barrier();

		WRITE_ONCE(b, 1);

		do_something();

	} else {

		barrier();

		WRITE_ONCE(b, 1);

		do_something_else();

	}

Unfortunately, current compilers will transform this as follows at high

optimization levels:

	q = READ_ONCE(a);

	barrier();

	WRITE_ONCE(b, 1);  /* BUG: No ordering vs. load from a!!! */

	if (q) {

		/* WRITE_ONCE(b, 1); -- moved up, BUG!!! */

		do_something();

	} else {

		/* WRITE_ONCE(b, 1); -- moved up, BUG!!! */

		do_something_else();

	}

Now there is no conditional between the load from "a" and the store to

"b", which means that the CPU is within its rights to reorder them:  The

conditional is absolutely required, and must be present in the final

assembly code, after all of the compiler and link-time optimizations

have been applied.  Therefore, if you need ordering in this example,

you must use explicit memory ordering, for example, smp_store_release():

	q = READ_ONCE(a);

	if (q) {

		smp_store_release(&b, 1);

		do_something();

	} else {

		smp_store_release(&b, 1);

		do_something_else();

	}

Without explicit memory ordering, control-dependency-based ordering is

guaranteed only when the stores differ, for example:

	q = READ_ONCE(a);

	if (q) {

		WRITE_ONCE(b, 1);

		do_something();

	} else {

		WRITE_ONCE(b, 2);

		do_something_else();

	}

The initial READ_ONCE() is still required to prevent the compiler from

knowing too much about the value of "a".

But please note that you need to be careful what you do with the local

variable "q", otherwise the compiler might be able to guess the value

and again remove the conditional branch that is absolutely required to

preserve ordering.  For example:

	q = READ_ONCE(a);

	if (q % MAX) {

		WRITE_ONCE(b, 1);

		do_something();

	} else {

		WRITE_ONCE(b, 2);

		do_something_else();

	}

If MAX is compile-time defined to be 1, then the compiler knows that

(q % MAX) must be equal to zero, regardless of the value of "q".

The compiler is therefore within its rights to transform the above code

into the following:

	q = READ_ONCE(a);

	WRITE_ONCE(b, 2);

	do_something_else();

Given this transformation, the CPU is not required to respect the ordering

between the load from variable "a" and the store to variable "b".  It is

tempting to add a barrier(), but this does not help.  The conditional

is gone, and the barrier won't bring it back.  Therefore, if you need

to relying on control dependencies to produce this ordering, you should

make sure that MAX is greater than one, perhaps as follows:

	q = READ_ONCE(a);

	BUILD_BUG_ON(MAX <= 1); /* Order load from a with store to b. */

	if (q % MAX) {

		WRITE_ONCE(b, 1);

		do_something();

	} else {

		WRITE_ONCE(b, 2);

		do_something_else();

	}

Please note once again that each leg of the "if" statement absolutely

must store different values to "b".  As in previous examples, if the two

values were identical, the compiler could pull this store outside of the

"if" statement, destroying the control dependency's ordering properties.

You must also be careful avoid relying too much on boolean short-circuit

evaluation.  Consider this example:

	q = READ_ONCE(a);

	if (q || 1 > 0)

		WRITE_ONCE(b, 1);

Because the first condition cannot fault and the second condition is

always true, the compiler can transform this example as follows, again

destroying the control dependency's ordering:

	q = READ_ONCE(a);

	WRITE_ONCE(b, 1);

This is yet another example showing the importance of preventing the

compiler from out-guessing your code.  Again, although READ_ONCE() really

does force the compiler to emit code for a given load, the compiler is

within its rights to discard the loaded value.

In addition, control dependencies apply only to the then-clause and

else-clause of the "if" statement in question.  In particular, they do

not necessarily order the code following the entire "if" statement:

	q = READ_ONCE(a);

	if (q) {

		WRITE_ONCE(b, 1);

	} else {

		WRITE_ONCE(b, 2);

	}

	WRITE_ONCE(c, 1);  /* BUG: No ordering against the read from "a". */

It is tempting to argue that there in fact is ordering because the

compiler cannot reorder volatile accesses and also cannot reorder

the writes to "b" with the condition.  Unfortunately for this line

of reasoning, the compiler might compile the two writes to "b" as

conditional-move instructions, as in this fanciful pseudo-assembly

language:

	ld r1,a

	cmp r1,$0

	cmov,ne r4,$1

	cmov,eq r4,$2

	st r4,b

	st $1,c

The control dependencies would then extend only to the pair of cmov

instructions and the store depending on them.  This means that a weakly

ordered CPU would have no dependency of any sort between the load from

"a" and the store to "c".  In short, control dependencies provide ordering

only to the stores in the then-clause and else-clause of the "if" statement

in question (including functions invoked by those two clauses), and not

to code following that "if" statement.

In summary:

  (*) Control dependencies can order prior loads against later stores.

      However, they do *not* guarantee any other sort of ordering:

      Not prior loads against later loads, nor prior stores against

      later anything.  If you need these other forms of ordering, use

      smp_load_acquire(), smp_store_release(), or, in the case of prior

      stores and later loads, smp_mb().

  (*) If both legs of the "if" statement contain identical stores to

      the same variable, then you must explicitly order those stores,

      either by preceding both of them with smp_mb() or by using

      smp_store_release().  Please note that it is *not* sufficient to use

      barrier() at beginning and end of each leg of the "if" statement

      because, as shown by the example above, optimizing compilers can

      destroy the control dependency while respecting the letter of the

      barrier() law.

  (*) Control dependencies require at least one run-time conditional

      between the prior load and the subsequent store, and this

      conditional must involve the prior load.  If the compiler is able

      to optimize the conditional away, it will have also optimized

      away the ordering.  Careful use of READ_ONCE() and WRITE_ONCE()

      can help to preserve the needed conditional.

  (*) Control dependencies require that the compiler avoid reordering the

      dependency into nonexistence.  Careful use of READ_ONCE() or

      atomic{,64}_read() can help to preserve your control dependency.

  (*) Control dependencies apply only to the then-clause and else-clause

      of the "if" statement containing the control dependency, including

      any functions that these two clauses call.  Control dependencies

      do *not* apply to code beyond the end of that "if" statement.

  (*) Control dependencies pair normally with other types of barriers.

  (*) Control dependencies do *not* provide multicopy atomicity.  If you

      need all the CPUs to agree on the ordering of a given store against

      all other accesses, use smp_mb().

  (*) Compilers do not understand control dependencies.  It is therefore

      your job to ensure that they do not break your code.

This document contains brief definitions of LKMM-related terms.  Like most

glossaries, it is not intended to be read front to back (except perhaps

as a way of confirming a diagnosis of OCD), but rather to be searched

for specific terms.

Address Dependency:  When the address of a later memory access is computed

	based on the value returned by an earlier load, an "address

	dependency" extends from that load extending to the later access.

	Address dependencies are quite common in RCU read-side critical

	sections:

	 1 rcu_read_lock();

	 2 p = rcu_dereference(gp);

	 3 do_something(p->a);

	 4 rcu_read_unlock();

	 In this case, because the address of "p->a" on line 3 is computed

	 from the value returned by the rcu_dereference() on line 2, the

	 address dependency extends from that rcu_dereference() to that

	 "p->a".  In rare cases, optimizing compilers can destroy address

	 dependencies.	Please see Documentation/RCU/rcu_dereference.txt

	 for more information.

	 See also "Control Dependency" and "Data Dependency".

Acquire:  With respect to a lock, acquiring that lock, for example,

	using spin_lock().  With respect to a non-lock shared variable,

	a special operation that includes a load and which orders that

	load before later memory references running on that same CPU.

	An example special acquire operation is smp_load_acquire(),

	but atomic_read_acquire() and atomic_xchg_acquire() also include

	acquire loads.

	When an acquire load returns the value stored by a release store

	to that same variable, then all operations preceding that store

	happen before any operations following that load acquire.

	See also "Relaxed" and "Release".

Coherence (co):  When one CPU's store to a given variable overwrites

	either the value from another CPU's store or some later value,

	there is said to be a coherence link from the second CPU to

	the first.

	It is also possible to have a coherence link within a CPU, which

	is a "coherence internal" (coi) link.  The term "coherence

	external" (coe) link is used when it is necessary to exclude

	the coi case.

	See also "From-reads" and "Reads-from".

Control Dependency:  When a later store's execution depends on a test

	of a value computed from a value returned by an earlier load,

	a "control dependency" extends from that load to that store.

	For example:

	 1 if (READ_ONCE(x))

	 2   WRITE_ONCE(y, 1);

	 Here, the control dependency extends from the READ_ONCE() on

	 line 1 to the WRITE_ONCE() on line 2.	Control dependencies are

	 fragile, and can be easily destroyed by optimizing compilers.

	 Please see control-dependencies.txt for more information.

	 See also "Address Dependency" and "Data Dependency".

Cycle:	Memory-barrier pairing is restricted to a pair of CPUs, as the

	name suggests.	And in a great many cases, a pair of CPUs is all

	that is required.  In other cases, the notion of pairing must be

	extended to additional CPUs, and the result is called a "cycle".

	In a cycle, each CPU's ordering interacts with that of the next:

	CPU 0                CPU 1                CPU 2

	WRITE_ONCE(x, 1);    WRITE_ONCE(y, 1);    WRITE_ONCE(z, 1);

	smp_mb();            smp_mb();            smp_mb();

	r0 = READ_ONCE(y);   r1 = READ_ONCE(z);   r2 = READ_ONCE(x);

	CPU 0's smp_mb() interacts with that of CPU 1, which interacts

	with that of CPU 2, which in turn interacts with that of CPU 0

	to complete the cycle.	Because of the smp_mb() calls between

	each pair of memory accesses, the outcome where r0, r1, and r2

	are all equal to zero is forbidden by LKMM.

	See also "Pairing".

Data Dependency:  When the data written by a later store is computed based

	on the value returned by an earlier load, a "data dependency"

	extends from that load to that later store.  For example:

	 1 r1 = READ_ONCE(x);

	 2 WRITE_ONCE(y, r1 + 1);

	In this case, the data dependency extends from the READ_ONCE()

	on line 1 to the WRITE_ONCE() on line 2.  Data dependencies are

	fragile and can be easily destroyed by optimizing compilers.

	Because optimizing compilers put a great deal of effort into

	working out what values integer variables might have, this is

	especially true in cases where the dependency is carried through

	an integer.

	See also "Address Dependency" and "Control Dependency".

From-Reads (fr):  When one CPU's store to a given variable happened

	too late to affect the value returned by another CPU's

	load from that same variable, there is said to be a from-reads

	link from the load to the store.

	It is also possible to have a from-reads link within a CPU, which

	is a "from-reads internal" (fri) link.  The term "from-reads

	external" (fre) link is used when it is necessary to exclude

	the fri case.

	See also "Coherence" and "Reads-from".

Fully Ordered:  An operation such as smp_mb() that orders all of

	its CPU's prior accesses with all of that CPU's subsequent

	accesses, or a marked access such as atomic_add_return()

	that orders all of its CPU's prior accesses, itself, and

	all of its CPU's subsequent accesses.

Marked Access:  An access to a variable that uses an special function or

	macro such as "r1 = READ_ONCE(x)" or "smp_store_release(&a, 1)".

	See also "Unmarked Access".

Pairing: "Memory-barrier pairing" reflects the fact that synchronizing

	data between two CPUs requires that both CPUs their accesses.

	Memory barriers thus tend to come in pairs, one executed by

	one of the CPUs and the other by the other CPU.  Of course,

	pairing also occurs with other types of operations, so that a

	smp_store_release() pairs with an smp_load_acquire() that reads

	the value stored.

	See also "Cycle".

Reads-From (rf):  When one CPU's load returns the value stored by some other

	CPU, there is said to be a reads-from link from the second

	CPU's store to the first CPU's load.  Reads-from links have the

	nice property that time must advance from the store to the load,

	which means that algorithms using reads-from links can use lighter

	weight ordering and synchronization compared to algorithms using

	coherence and from-reads links.

	It is also possible to have a reads-from link within a CPU, which

	is a "reads-from internal" (rfi) link.	The term "reads-from

	external" (rfe) link is used when it is necessary to exclude

	the rfi case.

	See also Coherence" and "From-reads".

Relaxed:  A marked access that does not imply ordering, for example, a

	READ_ONCE(), WRITE_ONCE(), a non-value-returning read-modify-write

	operation, or a value-returning read-modify-write operation whose

	name ends in "_relaxed".

	See also "Acquire" and "Release".

Release:  With respect to a lock, releasing that lock, for example,

	using spin_unlock().  With respect to a non-lock shared variable,

	a special operation that includes a store and which orders that

	store after earlier memory references that ran on that same CPU.

	An example special release store is smp_store_release(), but

	atomic_set_release() and atomic_cmpxchg_release() also include

	release stores.

	See also "Acquire" and "Relaxed".

Unmarked Access:  An access to a variable that uses normal C-language

	syntax, for example, "a = b[2]";

	See also "Marked Access".

	carrying a dependency, then the compiler can break that dependency

	by substituting a constant of that value.

	Conversely, LKMM sometimes doesn't recognize that a particular

	optimization is not allowed, and as a result, thinks that a

	dependency is not present (because the optimization would break it).

	The memory model misses some pretty obvious control dependencies

	because of this limitation.  A simple example is:

		r1 = READ_ONCE(x);

		if (r1 == 0)

			smp_mb();

		WRITE_ONCE(y, 1);

	There is a control dependency from the READ_ONCE to the WRITE_ONCE,

	even when r1 is nonzero, but LKMM doesn't realize this and thinks

	that the write may execute before the read if r1 != 0.  (Yes, that

	doesn't make sense if you think about it, but the memory model's

	intelligence is limited.)

2.	Multiple access sizes for a single variable are not supported,

	and neither are misaligned or partially overlapping accesses.