Unverified Commit 2e63e7d0 authored by openeuler-ci-bot's avatar openeuler-ci-bot Committed by Gitee
Browse files

!1517 [OLK-5.10] Rework CPU capacity asymmetry detection 

Merge Pull Request from: @liujie-248683921 
 
As of now, the asym_cpu_capacity_level will try to locate the lowest
topology level where the highest available CPU capacity is being
visible to all CPUs. This works perfectly fine for most of existing
asymmetric designs out there, though for some possible and completely
valid setups, combining different cpu microarchitectures within
clusters, this might not be the best approach, resulting in pointing
at a level, at which some of the domains might not see any asymmetry
at all. This could be problematic for misfit migration and/or energy
aware placement. And as such, for affected platforms it might result
in custom changes to wake-up and CPU selection paths.

As mentioned in the previous version, based on the available sources out there,
one of the potentially affected (by original approach) platforms might be
Exynos 9820/990 with it's 'sliced' LLC(SLC) divided between the two custom (big)
cores and the remaining A75/A55 cores, which seems to be reflected in the
made available dt entries for those platforms.

The following patches rework how the asymmetric detection is being
carried out, allowing pinning the asymmetric topology level to the lowest one,
where full range of CPU capacities is visible to all CPUs within given
sched domain. The asym_cpu_capacity_level will also keep track of those
levels where any scope of asymmetry is being observed, to denote
corresponding sched domains with the SD_ASYM_CPUCAPACITY flag
and to enable misfit migration for those.

In order to distinguish the sched domains with partial vs full range
of CPU capacity asymmetry, new sched domain flag has been introduced:
SD_ASYM_CPUCAPACITY_FULL.

The overall idea of changing the asymmetry detection has been suggested
by Valentin Schneider <valentin.schneider@arm.com>

Beata Michalska (3):
  sched/core: Introduce SD_ASYM_CPUCAPACITY_FULL sched_domain flag
  sched/topology: Rework CPU capacity asymmetry detection
  sched/doc: Update the CPU capacity asymmetry bits

 Documentation/scheduler/sched-capacity.rst |   6 +-
 Documentation/scheduler/sched-energy.rst   |   2 +-
 include/linux/sched/sd_flags.h             |  10 +
 kernel/sched/topology.c                    | 206 +++++++++++++--------
 4 files changed, 143 insertions(+), 81 deletions(-)

-- 
2.17.1 
 
Link:https://gitee.com/openeuler/kernel/pulls/1517

 

Reviewed-by: default avatarZucheng Zheng <zhengzucheng@huawei.com>
Signed-off-by: default avatarJialin Zhang <zhangjialin11@huawei.com>
parents d0424233 cd499e19

Documentation/scheduler/sched-capacity.rst

+4 −2
Original line number Diff line number Diff line
@@ -284,8 +284,10 @@ whether the system exhibits asymmetric CPU capacities. Should that be the 
case:



- The sched_asym_cpucapacity static key will be enabled.

- The SD_ASYM_CPUCAPACITY flag will be set at the lowest sched_domain level that

  spans all unique CPU capacity values.

- The SD_ASYM_CPUCAPACITY_FULL flag will be set at the lowest sched_domain

  level that spans all unique CPU capacity values.

- The SD_ASYM_CPUCAPACITY flag will be set for any sched_domain that spans

  CPUs with any range of asymmetry.



The sched_asym_cpucapacity static key is intended to guard sections of code that

cater to asymmetric CPU capacity systems. Do note however that said key is

Documentation/scheduler/sched-energy.rst

+1 −1
Original line number Diff line number Diff line
@@ -328,7 +328,7 @@ section lists these dependencies and provides hints as to how they can be met. 


As mentioned in the introduction, EAS is only supported on platforms with

asymmetric CPU topologies for now. This requirement is checked at run-time by

looking for the presence of the SD_ASYM_CPUCAPACITY flag when the scheduling

looking for the presence of the SD_ASYM_CPUCAPACITY_FULL flag when the scheduling

domains are built.



See Documentation/scheduler/sched-capacity.rst for requirements to be met for this

include/linux/sched/sd_flags.h

+10 −0
Original line number Diff line number Diff line
@@ -90,6 +90,16 @@ SD_FLAG(SD_WAKE_AFFINE, SDF_SHARED_CHILD) 
 */

SD_FLAG(SD_ASYM_CPUCAPACITY, SDF_SHARED_PARENT | SDF_NEEDS_GROUPS)



/*

 * Domain members have different CPU capacities spanning all unique CPU

 * capacity values.

 *

 * SHARED_PARENT: Set from the topmost domain down to the first domain where

 *		  all available CPU capacities are visible

 * NEEDS_GROUPS: Per-CPU capacity is asymmetric between groups.

 */

SD_FLAG(SD_ASYM_CPUCAPACITY_FULL, SDF_SHARED_PARENT | SDF_NEEDS_GROUPS)



/*

 * Domain members share CPU capacity (i.e. SMT)

 *

kernel/sched/topology.c

+130 −78
Original line number Diff line number Diff line
@@ -704,7 +704,7 @@ static void update_top_cache_domain(int cpu) 
	sd = highest_flag_domain(cpu, SD_ASYM_PACKING);

	rcu_assign_pointer(per_cpu(sd_asym_packing, cpu), sd);



	sd = lowest_flag_domain(cpu, SD_ASYM_CPUCAPACITY);

	sd = lowest_flag_domain(cpu, SD_ASYM_CPUCAPACITY_FULL);

	rcu_assign_pointer(per_cpu(sd_asym_cpucapacity, cpu), sd);

}
@@ -1295,6 +1295,116 @@ static void init_sched_groups_capacity(int cpu, struct sched_domain *sd) 
	update_group_capacity(sd, cpu);

}



/*

 * Asymmetric CPU capacity bits

 */

struct asym_cap_data {

	struct list_head link;

	unsigned long capacity;

	unsigned long cpus[];

};



/*

 * Set of available CPUs grouped by their corresponding capacities

 * Each list entry contains a CPU mask reflecting CPUs that share the same

 * capacity.

 * The lifespan of data is unlimited.

 */

static LIST_HEAD(asym_cap_list);



#define cpu_capacity_span(asym_data) to_cpumask((asym_data)->cpus)



/*

 * Verify whether there is any CPU capacity asymmetry in a given sched domain.

 * Provides sd_flags reflecting the asymmetry scope.

 */

static inline int

asym_cpu_capacity_classify(const struct cpumask *sd_span,

			   const struct cpumask *cpu_map)

{

	struct asym_cap_data *entry;

	int count = 0, miss = 0;



	/*

	 * Count how many unique CPU capacities this domain spans across

	 * (compare sched_domain CPUs mask with ones representing  available

	 * CPUs capacities). Take into account CPUs that might be offline:

	 * skip those.

	 */

	list_for_each_entry(entry, &asym_cap_list, link) {

		if (cpumask_intersects(sd_span, cpu_capacity_span(entry)))

			++count;

		else if (cpumask_intersects(cpu_map, cpu_capacity_span(entry)))

			++miss;

	}



	WARN_ON_ONCE(!count && !list_empty(&asym_cap_list));



	/* No asymmetry detected */

	if (count < 2)

		return 0;

	/* Some of the available CPU capacity values have not been detected */

	if (miss)

		return SD_ASYM_CPUCAPACITY;



	/* Full asymmetry */

	return SD_ASYM_CPUCAPACITY | SD_ASYM_CPUCAPACITY_FULL;



}



static inline void asym_cpu_capacity_update_data(int cpu)

{

	unsigned long capacity = arch_scale_cpu_capacity(cpu);

	struct asym_cap_data *entry = NULL;



	list_for_each_entry(entry, &asym_cap_list, link) {

		if (capacity == entry->capacity)

			goto done;

	}



	entry = kzalloc(sizeof(*entry) + cpumask_size(), GFP_KERNEL);

	if (WARN_ONCE(!entry, "Failed to allocate memory for asymmetry data\n"))

		return;

	entry->capacity = capacity;

	list_add(&entry->link, &asym_cap_list);

done:

	__cpumask_set_cpu(cpu, cpu_capacity_span(entry));

}



/*

 * Build-up/update list of CPUs grouped by their capacities

 * An update requires explicit request to rebuild sched domains

 * with state indicating CPU topology changes.

 */

static void asym_cpu_capacity_scan(void)

{

	struct asym_cap_data *entry, *next;

	int cpu;



	list_for_each_entry(entry, &asym_cap_list, link)

		cpumask_clear(cpu_capacity_span(entry));



	for_each_cpu_and(cpu, cpu_possible_mask, housekeeping_cpumask(HK_FLAG_DOMAIN))

		asym_cpu_capacity_update_data(cpu);



	list_for_each_entry_safe(entry, next, &asym_cap_list, link) {

		if (cpumask_empty(cpu_capacity_span(entry))) {

			list_del(&entry->link);

			kfree(entry);

		}

	}



	/*

	 * Only one capacity value has been detected i.e. this system is symmetric.

	 * No need to keep this data around.

	 */

	if (list_is_singular(&asym_cap_list)) {

		entry = list_first_entry(&asym_cap_list, typeof(*entry), link);

		list_del(&entry->link);

		kfree(entry);

	}

}



/*

 * Initializers for schedule domains

 * Non-inlined to reduce accumulated stack pressure in build_sched_domains()
@@ -1430,11 +1540,12 @@ int __read_mostly node_reclaim_distance = RECLAIM_DISTANCE; 
static struct sched_domain *

sd_init(struct sched_domain_topology_level *tl,

	const struct cpumask *cpu_map,

	struct sched_domain *child, int dflags, int cpu)

	struct sched_domain *child, int cpu)

{

	struct sd_data *sdd = &tl->data;

	struct sched_domain *sd = *per_cpu_ptr(sdd->sd, cpu);

	int sd_id, sd_weight, sd_flags = 0;

	struct cpumask *sd_span;



#ifdef CONFIG_NUMA

	/*
@@ -1451,9 +1562,6 @@ sd_init(struct sched_domain_topology_level *tl, 
			"wrong sd_flags in topology description\n"))

		sd_flags &= TOPOLOGY_SD_FLAGS;



	/* Apply detected topology flags */

	sd_flags |= dflags;



	*sd = (struct sched_domain){

		.min_interval		= sd_weight,

		.max_interval		= 2*sd_weight,
@@ -1485,13 +1593,19 @@ sd_init(struct sched_domain_topology_level *tl, 
#endif

	};



	cpumask_and(sched_domain_span(sd), cpu_map, tl->mask(cpu));

	sd_id = cpumask_first(sched_domain_span(sd));

	sd_span = sched_domain_span(sd);

	cpumask_and(sd_span, cpu_map, tl->mask(cpu));

	sd_id = cpumask_first(sd_span);



	sd->flags |= asym_cpu_capacity_classify(sd_span, cpu_map);



	WARN_ONCE((sd->flags & (SD_SHARE_CPUCAPACITY | SD_ASYM_CPUCAPACITY)) ==

		  (SD_SHARE_CPUCAPACITY | SD_ASYM_CPUCAPACITY),

		  "CPU capacity asymmetry not supported on SMT\n");



	/*

	 * Convert topological properties into behaviour.

	 */



	/* Don't attempt to spread across CPUs of different capacities. */

	if ((sd->flags & SD_ASYM_CPUCAPACITY) && sd->child)

		sd->child->flags &= ~SD_PREFER_SIBLING;
@@ -2153,9 +2267,9 @@ static void sd_llc_free_all(const struct cpumask *cpu_map) 


static struct sched_domain *build_sched_domain(struct sched_domain_topology_level *tl,

		const struct cpumask *cpu_map, struct sched_domain_attr *attr,

		struct sched_domain *child, int dflags, int cpu)

		struct sched_domain *child, int cpu)

{

	struct sched_domain *sd = sd_init(tl, cpu_map, child, dflags, cpu);

	struct sched_domain *sd = sd_init(tl, cpu_map, child, cpu);



	if (child) {

		sd->level = child->level + 1;
@@ -2217,65 +2331,6 @@ static bool topology_span_sane(struct sched_domain_topology_level *tl, 
	return true;

}



/*

 * Find the sched_domain_topology_level where all CPU capacities are visible

 * for all CPUs.

 */

static struct sched_domain_topology_level

*asym_cpu_capacity_level(const struct cpumask *cpu_map)

{

	int i, j, asym_level = 0;

	bool asym = false;

	struct sched_domain_topology_level *tl, *asym_tl = NULL;

	unsigned long cap;



	/* Is there any asymmetry? */

	cap = arch_scale_cpu_capacity(cpumask_first(cpu_map));



	for_each_cpu(i, cpu_map) {

		if (arch_scale_cpu_capacity(i) != cap) {

			asym = true;

			break;

		}

	}



	if (!asym)

		return NULL;



	/*

	 * Examine topology from all CPU's point of views to detect the lowest

	 * sched_domain_topology_level where a highest capacity CPU is visible

	 * to everyone.

	 */

	for_each_cpu(i, cpu_map) {

		unsigned long max_capacity = arch_scale_cpu_capacity(i);

		int tl_id = 0;



		for_each_sd_topology(tl) {

			if (tl_id < asym_level)

				goto next_level;



			for_each_cpu_and(j, tl->mask(i), cpu_map) {

				unsigned long capacity;



				capacity = arch_scale_cpu_capacity(j);



				if (capacity <= max_capacity)

					continue;



				max_capacity = capacity;

				asym_level = tl_id;

				asym_tl = tl;

			}

next_level:

			tl_id++;

		}

	}



	return asym_tl;

}





/*

 * Build sched domains for a given set of CPUs and attach the sched domains

 * to the individual CPUs
@@ -2288,7 +2343,6 @@ build_sched_domains(const struct cpumask *cpu_map, struct sched_domain_attr *att 
	struct s_data d;

	struct rq *rq = NULL;

	int i, ret = -ENOMEM;

	struct sched_domain_topology_level *tl_asym;

	bool has_asym = false;

	bool has_cluster = false;
@@ -2299,25 +2353,19 @@ build_sched_domains(const struct cpumask *cpu_map, struct sched_domain_attr *att 
	if (alloc_state != sa_rootdomain)

		goto error;



	tl_asym = asym_cpu_capacity_level(cpu_map);



	/* Set up domains for CPUs specified by the cpu_map: */

	for_each_cpu(i, cpu_map) {

		struct sched_domain_topology_level *tl;

		int dflags = 0;



		sd = NULL;

		for_each_sd_topology(tl) {

			if (tl == tl_asym) {

				dflags |= SD_ASYM_CPUCAPACITY;

				has_asym = true;

			}



			if (WARN_ON(!topology_span_sane(tl, cpu_map, i)))

				goto error;



			sd = build_sched_domain(tl, cpu_map, attr, sd, dflags, i);

			sd = build_sched_domain(tl, cpu_map, attr, sd, i);



			has_asym |= sd->flags & SD_ASYM_CPUCAPACITY;

			has_cluster |= sd->flags & SD_CLUSTER;



			if (tl == sched_domain_topology)
@@ -2511,6 +2559,7 @@ int sched_init_domains(const struct cpumask *cpu_map) 
	zalloc_cpumask_var(&fallback_doms, GFP_KERNEL);



	arch_update_cpu_topology();

	asym_cpu_capacity_scan();

	ndoms_cur = 1;

	doms_cur = alloc_sched_domains(ndoms_cur);

	if (!doms_cur)
@@ -2600,6 +2649,9 @@ void partition_sched_domains_locked(int ndoms_new, cpumask_var_t doms_new[], 


	/* Let the architecture update CPU core mappings: */

	new_topology = arch_update_cpu_topology();

	/* Trigger rebuilding CPU capacity asymmetry data */

	if (new_topology)

		asym_cpu_capacity_scan();



	if (!doms_new) {

		WARN_ON_ONCE(dattr_new);