Unverified Commit c19681d5 authored by openeuler-ci-bot's avatar openeuler-ci-bot Committed by Gitee
Browse files

!3689 Intel: Backport Sierra Forest(SRF) core PMU support to OLK-5.10 

Merge Pull Request from: @yunyingsun 
 
Title:  Add core PMU support for Intel Sierra Forest(SRF)

Content:
This PR is backporting Intel Sierra Forest (SRF) core PMU support to OpenEuler kernel devel-5.10, so that end users can access the core PMU counters via perf on next Intel E-core Xeon server SRF.

Upstream commit(v6.6-rc1):
a430021f perf/x86/intel: Add Crestmont PMU

Since SRF is an E-core only server platform and it shares same Crestmont core with Meteor Lake, its core PMU enabling is based on hybrid PMU driver which was firstly introduced since Alder Lake and later evolved with Raptor Lake. There are lots of hybrid PMU related commits needed as dependencies.

Besides the upstream commit, totally there're 55 dependent patches identified and backported.
The complete commit list of dependent patches:

v6.6-rc1(3): //mem_attr = grt_mem_attrs;
53544562 x86/cpu: Update Hybrids
0cfd8fba x86/cpu: Fix Crestmont uarch
882cdb06 x86/cpu: Fix Gracemont uarch

v6.3-rc1(7): // x86_pmu.pebs_latency_data = mtl_latency_data_small;
b0bd3336 perf/x86/msr: Add Meteor Lake support
eaef048c perf/x86/cstate: Add Meteor Lake support
eb467aaa perf/x86/intel: Support Architectural PerfMon Extension leaf
a018d2e3 x86/cpufeatures: Add Architectural PerfMon Extension bit
c87a3109 perf/x86: Support Retire Latency
38aaf921 perf/x86: Add Meteor Lake support

v6.0 (3):
b6c00fb9 perf: Add PMU_FORMAT_ATTR_SHOW
24919fde perf/x86/intel: Fix unchecked MSR access error for Alder Lake N
5515d21c x86/cpu: Add CPU model numbers for Meteor Lake

v6.0-rc1(2):
ccf170e9 perf/x86/intel: Fix PEBS data source encoding for ADL
39a41278 perf/x86/intel: Fix PEBS memory access info encoding for ADL

v5.19-rc1(5) // case INTEL_FAM6_ALDERLAKE_N:
f758bc5a perf/x86/uncore: Add new Alder Lake and Raptor Lake support
e5ae168e perf/x86/uncore: Clean up uncore_pci_ids[]
cd971104 perf/x86/cstate: Add new Alder Lake and Raptor Lake support
d773a733 perf/x86/msr: Add new Alder Lake and Raptor Lake support
c2a960f7 perf/x86: Add new Alder Lake and Raptor Lake support

v5.18-rc5(1): // case INTEL_FAM6_ALDERLAKE_N:
3ccce934 x86/cpu: Add new Alderlake and Raptorlake CPU model numbers

v5.18-rc2(4): // case INTEL_FAM6_RAPTORLAKE:
ad4878d4 perf/x86/uncore: Add Raptor Lake uncore support
82cd8304 perf/x86/msr: Add Raptor Lake CPU support
2da202aa perf/x86/cstate: Add Raptor Lake support
c61759e5 perf/x86: Add Intel Raptor Lake support

5.17-rc2(2):
5a4487f9 perf/x86/intel/uncore: Add IMC uncore support for ADL
7fa981ca perf/x86/intel: Add a quirk for the calculation of the number of counters on Alder Lake

v5.16-rc4(1):
7d697f0d x86/cpu: Drop spurious underscore from RAPTOR_LAKE #define

v5.16-rc1(1):
fbdb5e8f x86/cpu: Add Raptor Lake to Intel family

v5.14-rc5(1):
acade637 perf/x86/intel: Apply mid ACK for small core

v5.14-rc1(1)
ee72a94e perf/x86/intel: Fix fixed counter check warning for some Alder Lake

5.13-rc1(24):
6a5f4386 perf/x86/rapl: Add support for Intel Alder Lake
d0ca946b perf/x86/cstate: Add Alder Lake CPU support
19d3a81f perf/x86/msr: Add Alder Lake CPU support
772ed05f perf/x86/intel/uncore: Add Alder Lake support
55bcf6ef perf: Extend PERF_TYPE_HARDWARE and PERF_TYPE_HW_CACHE
f83d2f91 perf/x86/intel: Add Alder Lake Hybrid support
3e9a8b21 perf/x86: Support filter_match callback
58ae30c2 perf/x86/intel: Add attr_update for Hybrid PMUs
a9c81ccd perf/x86: Add structures for the attributes of Hybrid PMUs
d9977c43 perf/x86: Register hybrid PMUs
e11c1a7e perf/x86: Factor out x86_pmu_show_pmu_cap
b9856729 perf/x86: Remove temporary pmu assignment in event_init
34d5b61f perf/x86/intel: Factor out intel_pmu_check_extra_regs
bc14fe1b perf/x86/intel: Factor out intel_pmu_check_event_constraints
b8c4d1a8 perf/x86/intel: Factor out intel_pmu_check_num_counters
183af736 perf/x86: Hybrid PMU support for extra_regs
24ee38ff perf/x86: Hybrid PMU support for event constraints
0d18f2df perf/x86: Hybrid PMU support for hardware cache event
eaacf07d perf/x86: Hybrid PMU support for unconstrained
d4b294bf perf/x86: Hybrid PMU support for counters
fc4b8fca perf/x86: Hybrid PMU support for intel_ctrl
d0946a88 perf/x86/intel: Hybrid PMU support for perf capabilities
250b3c0d x86/cpu: Add helper function to get the type of the current hybrid CPU
a161545a x86/cpufeatures: Enumerate Intel Hybrid Technology feature bit

v5.11-rc1(1):
c2208046 perf/x86/intel: Add Tremont Topdown support

Some of the backported patches have slightly code deviations from its upstream version, mainly because upstream commit contains code or functions that OLK-5.10 not yet supported.

The backported patch set have been verified on Intel internal Sierra Forest SP system.

Intel-kernel issue:
https://gitee.com/openeuler/intel-kernel/issues/I8RWG5

Test:
Platform dependent core PMU events do not work on SRF without this PR, like "L1-dcache-loads":
`# perf stat -a -e L1-dcache-loads -- sleep 1

 Performance counter stats for 'system wide':

   <not supported>      L1-dcache-loads

       1.029285911 seconds time elapsed`

With this PR applied to kernel OLK-5.10, it works:
`# perf stat -a -e L1-dcache-loads -- sleep 1

 Performance counter stats for 'system wide':

         7,354,296      L1-dcache-loads

       1.030988869 seconds time elapsed`

Known issue:
N/A

Default config change:
N/A 
 
Link:https://gitee.com/openeuler/kernel/pulls/3689

 

Reviewed-by: default avatarXu Kuohai <xukuohai@huawei.com>
Reviewed-by: default avatarJason Zeng <jason.zeng@intel.com>
Reviewed-by: default avatarAichun Shi <aichun.shi@intel.com>
Signed-off-by: default avatarJialin Zhang <zhangjialin11@huawei.com>
parents 37fee35b ffc6aabf

arch/x86/events/core.c

+243 −69
Original line number Diff line number Diff line
@@ -54,6 +54,7 @@ DEFINE_PER_CPU(struct cpu_hw_events, cpu_hw_events) = { 


DEFINE_STATIC_KEY_FALSE(rdpmc_never_available_key);

DEFINE_STATIC_KEY_FALSE(rdpmc_always_available_key);

DEFINE_STATIC_KEY_FALSE(perf_is_hybrid);



/*

 * This here uses DEFINE_STATIC_CALL_NULL() to get a static_call defined
@@ -147,15 +148,16 @@ u64 x86_perf_event_update(struct perf_event *event) 
 */

static int x86_pmu_extra_regs(u64 config, struct perf_event *event)

{

	struct extra_reg *extra_regs = hybrid(event->pmu, extra_regs);

	struct hw_perf_event_extra *reg;

	struct extra_reg *er;



	reg = &event->hw.extra_reg;



	if (!x86_pmu.extra_regs)

	if (!extra_regs)

		return 0;



	for (er = x86_pmu.extra_regs; er->msr; er++) {

	for (er = extra_regs; er->msr; er++) {

		if (er->event != (config & er->config_mask))

			continue;

		if (event->attr.config1 & ~er->valid_mask)
@@ -178,16 +180,29 @@ static DEFINE_MUTEX(pmc_reserve_mutex); 


#ifdef CONFIG_X86_LOCAL_APIC



static inline int get_possible_num_counters(void)

{

	int i, num_counters = x86_pmu.num_counters;



	if (!is_hybrid())

		return num_counters;



	for (i = 0; i < x86_pmu.num_hybrid_pmus; i++)

		num_counters = max_t(int, num_counters, x86_pmu.hybrid_pmu[i].num_counters);



	return num_counters;

}



static bool reserve_pmc_hardware(void)

{

	int i;

	int i, num_counters = get_possible_num_counters();



	for (i = 0; i < x86_pmu.num_counters; i++) {

	for (i = 0; i < num_counters; i++) {

		if (!reserve_perfctr_nmi(x86_pmu_event_addr(i)))

			goto perfctr_fail;

	}



	for (i = 0; i < x86_pmu.num_counters; i++) {

	for (i = 0; i < num_counters; i++) {

		if (!reserve_evntsel_nmi(x86_pmu_config_addr(i)))

			goto eventsel_fail;

	}
@@ -198,7 +213,7 @@ static bool reserve_pmc_hardware(void) 
	for (i--; i >= 0; i--)

		release_evntsel_nmi(x86_pmu_config_addr(i));



	i = x86_pmu.num_counters;

	i = num_counters;



perfctr_fail:

	for (i--; i >= 0; i--)
@@ -209,9 +224,9 @@ static bool reserve_pmc_hardware(void) 


static void release_pmc_hardware(void)

{

	int i;

	int i, num_counters = get_possible_num_counters();



	for (i = 0; i < x86_pmu.num_counters; i++) {

	for (i = 0; i < num_counters; i++) {

		release_perfctr_nmi(x86_pmu_event_addr(i));

		release_evntsel_nmi(x86_pmu_config_addr(i));

	}
@@ -224,7 +239,7 @@ static void release_pmc_hardware(void) {} 


#endif



static bool check_hw_exists(void)

bool check_hw_exists(struct pmu *pmu, int num_counters, int num_counters_fixed)

{

	u64 val, val_fail = -1, val_new= ~0;

	int i, reg, reg_fail = -1, ret = 0;
@@ -235,7 +250,7 @@ static bool check_hw_exists(void) 
	 * Check to see if the BIOS enabled any of the counters, if so

	 * complain and bail.

	 */

	for (i = 0; i < x86_pmu.num_counters; i++) {

	for (i = 0; i < num_counters; i++) {

		reg = x86_pmu_config_addr(i);

		ret = rdmsrl_safe(reg, &val);

		if (ret)
@@ -249,13 +264,13 @@ static bool check_hw_exists(void) 
		}

	}



	if (x86_pmu.num_counters_fixed) {

	if (num_counters_fixed) {

		reg = MSR_ARCH_PERFMON_FIXED_CTR_CTRL;

		ret = rdmsrl_safe(reg, &val);

		if (ret)

			goto msr_fail;

		for (i = 0; i < x86_pmu.num_counters_fixed; i++) {

			if (fixed_counter_disabled(i))

		for (i = 0; i < num_counters_fixed; i++) {

			if (fixed_counter_disabled(i, pmu))

				continue;

			if (val & (0x03 << i*4)) {

				bios_fail = 1;
@@ -356,8 +371,7 @@ set_ext_hw_attr(struct hw_perf_event *hwc, struct perf_event *event) 
		return -EINVAL;

	cache_result = array_index_nospec(cache_result, PERF_COUNT_HW_CACHE_RESULT_MAX);



	val = hw_cache_event_ids[cache_type][cache_op][cache_result];



	val = hybrid_var(event->pmu, hw_cache_event_ids)[cache_type][cache_op][cache_result];

	if (val == 0)

		return -ENOENT;
@@ -365,7 +379,7 @@ set_ext_hw_attr(struct hw_perf_event *hwc, struct perf_event *event) 
		return -EINVAL;



	hwc->config |= val;

	attr->config1 = hw_cache_extra_regs[cache_type][cache_op][cache_result];

	attr->config1 = hybrid_var(event->pmu, hw_cache_extra_regs)[cache_type][cache_op][cache_result];

	return x86_pmu_extra_regs(val, event);

}
@@ -460,7 +474,7 @@ int x86_setup_perfctr(struct perf_event *event) 
		local64_set(&hwc->period_left, hwc->sample_period);

	}



	if (attr->type == PERF_TYPE_RAW)

	if (attr->type == event->pmu->type)

		return x86_pmu_extra_regs(event->attr.config, event);



	if (attr->type == PERF_TYPE_HW_CACHE)
@@ -595,7 +609,7 @@ int x86_pmu_hw_config(struct perf_event *event) 
	if (!event->attr.exclude_kernel)

		event->hw.config |= ARCH_PERFMON_EVENTSEL_OS;



	if (event->attr.type == PERF_TYPE_RAW)

	if (event->attr.type == event->pmu->type)

		event->hw.config |= event->attr.config & X86_RAW_EVENT_MASK;



	if (event->attr.sample_period && x86_pmu.limit_period) {
@@ -718,7 +732,17 @@ void x86_pmu_enable_all(int added) 


static inline int is_x86_event(struct perf_event *event)

{

	int i;



	if (!is_hybrid())

		return event->pmu == &pmu;



	for (i = 0; i < x86_pmu.num_hybrid_pmus; i++) {

		if (event->pmu == &x86_pmu.hybrid_pmu[i].pmu)

			return true;

	}



	return false;

}



struct pmu *x86_get_pmu(unsigned int cpu)
@@ -935,6 +959,7 @@ EXPORT_SYMBOL_GPL(perf_assign_events); 


int x86_schedule_events(struct cpu_hw_events *cpuc, int n, int *assign)

{

	int num_counters = hybrid(cpuc->pmu, num_counters);

	struct event_constraint *c;

	struct perf_event *e;

	int n0, i, wmin, wmax, unsched = 0;
@@ -1010,7 +1035,7 @@ int x86_schedule_events(struct cpu_hw_events *cpuc, int n, int *assign) 


	/* slow path */

	if (i != n) {

		int gpmax = x86_pmu.num_counters;

		int gpmax = num_counters;



		/*

		 * Do not allow scheduling of more than half the available
@@ -1031,7 +1056,7 @@ int x86_schedule_events(struct cpu_hw_events *cpuc, int n, int *assign) 
		 * the extra Merge events needed by large increment events.

		 */

		if (x86_pmu.flags & PMU_FL_PAIR) {

			gpmax = x86_pmu.num_counters - cpuc->n_pair;

			gpmax = num_counters - cpuc->n_pair;

			WARN_ON(gpmax <= 0);

		}
@@ -1095,8 +1120,9 @@ static void del_nr_metric_event(struct cpu_hw_events *cpuc, 
static int collect_event(struct cpu_hw_events *cpuc, struct perf_event *event,

			 int max_count, int n)

{

	union perf_capabilities intel_cap = hybrid(cpuc->pmu, intel_cap);



	if (x86_pmu.intel_cap.perf_metrics && add_nr_metric_event(cpuc, event))

	if (intel_cap.perf_metrics && add_nr_metric_event(cpuc, event))

		return -EINVAL;



	if (n >= max_count + cpuc->n_metric)
@@ -1117,10 +1143,12 @@ static int collect_event(struct cpu_hw_events *cpuc, struct perf_event *event, 
 */

static int collect_events(struct cpu_hw_events *cpuc, struct perf_event *leader, bool dogrp)

{

	int num_counters = hybrid(cpuc->pmu, num_counters);

	int num_counters_fixed = hybrid(cpuc->pmu, num_counters_fixed);

	struct perf_event *event;

	int n, max_count;



	max_count = x86_pmu.num_counters + x86_pmu.num_counters_fixed;

	max_count = num_counters + num_counters_fixed;



	/* current number of events already accepted */

	n = cpuc->n_events;
@@ -1499,18 +1527,19 @@ void perf_event_print_debug(void) 
{

	u64 ctrl, status, overflow, pmc_ctrl, pmc_count, prev_left, fixed;

	u64 pebs, debugctl;

	struct cpu_hw_events *cpuc;

	int cpu = smp_processor_id();

	struct cpu_hw_events *cpuc = &per_cpu(cpu_hw_events, cpu);

	int num_counters = hybrid(cpuc->pmu, num_counters);

	int num_counters_fixed = hybrid(cpuc->pmu, num_counters_fixed);

	struct event_constraint *pebs_constraints = hybrid(cpuc->pmu, pebs_constraints);

	unsigned long flags;

	int cpu, idx;

	int idx;



	if (!x86_pmu.num_counters)

	if (!num_counters)

		return;



	local_irq_save(flags);



	cpu = smp_processor_id();

	cpuc = &per_cpu(cpu_hw_events, cpu);



	if (x86_pmu.version >= 2) {

		rdmsrl(MSR_CORE_PERF_GLOBAL_CTRL, ctrl);

		rdmsrl(MSR_CORE_PERF_GLOBAL_STATUS, status);
@@ -1522,7 +1551,7 @@ void perf_event_print_debug(void) 
		pr_info("CPU#%d: status:     %016llx\n", cpu, status);

		pr_info("CPU#%d: overflow:   %016llx\n", cpu, overflow);

		pr_info("CPU#%d: fixed:      %016llx\n", cpu, fixed);

		if (x86_pmu.pebs_constraints) {

		if (pebs_constraints) {

			rdmsrl(MSR_IA32_PEBS_ENABLE, pebs);

			pr_info("CPU#%d: pebs:       %016llx\n", cpu, pebs);

		}
@@ -1533,7 +1562,7 @@ void perf_event_print_debug(void) 
	}

	pr_info("CPU#%d: active:     %016llx\n", cpu, *(u64 *)cpuc->active_mask);



	for (idx = 0; idx < x86_pmu.num_counters; idx++) {

	for (idx = 0; idx < num_counters; idx++) {

		rdmsrl(x86_pmu_config_addr(idx), pmc_ctrl);

		rdmsrl(x86_pmu_event_addr(idx), pmc_count);
@@ -1546,8 +1575,8 @@ void perf_event_print_debug(void) 
		pr_info("CPU#%d:   gen-PMC%d left:  %016llx\n",

			cpu, idx, prev_left);

	}

	for (idx = 0; idx < x86_pmu.num_counters_fixed; idx++) {

		if (fixed_counter_disabled(idx))

	for (idx = 0; idx < num_counters_fixed; idx++) {

		if (fixed_counter_disabled(idx, cpuc->pmu))

			continue;

		rdmsrl(MSR_ARCH_PERFMON_FIXED_CTR0 + idx, pmc_count);
@@ -1583,6 +1612,7 @@ void x86_pmu_stop(struct perf_event *event, int flags) 
static void x86_pmu_del(struct perf_event *event, int flags)

{

	struct cpu_hw_events *cpuc = this_cpu_ptr(&cpu_hw_events);

	union perf_capabilities intel_cap = hybrid(cpuc->pmu, intel_cap);

	int i;



	/*
@@ -1622,7 +1652,7 @@ static void x86_pmu_del(struct perf_event *event, int flags) 
	}

	cpuc->event_constraint[i-1] = NULL;

	--cpuc->n_events;

	if (x86_pmu.intel_cap.perf_metrics)

	if (intel_cap.perf_metrics)

		del_nr_metric_event(cpuc, event);



	perf_event_update_userpage(event);
@@ -1836,6 +1866,49 @@ ssize_t events_ht_sysfs_show(struct device *dev, struct device_attribute *attr, 
			pmu_attr->event_str_noht);

}



ssize_t events_hybrid_sysfs_show(struct device *dev,

				 struct device_attribute *attr,

				 char *page)

{

	struct perf_pmu_events_hybrid_attr *pmu_attr =

		container_of(attr, struct perf_pmu_events_hybrid_attr, attr);

	struct x86_hybrid_pmu *pmu;

	const char *str, *next_str;

	int i;



	if (hweight64(pmu_attr->pmu_type) == 1)

		return sprintf(page, "%s", pmu_attr->event_str);



	/*

	 * Hybrid PMUs may support the same event name, but with different

	 * event encoding, e.g., the mem-loads event on an Atom PMU has

	 * different event encoding from a Core PMU.

	 *

	 * The event_str includes all event encodings. Each event encoding

	 * is divided by ";". The order of the event encodings must follow

	 * the order of the hybrid PMU index.

	 */

	pmu = container_of(dev_get_drvdata(dev), struct x86_hybrid_pmu, pmu);



	str = pmu_attr->event_str;

	for (i = 0; i < x86_pmu.num_hybrid_pmus; i++) {

		if (!(x86_pmu.hybrid_pmu[i].cpu_type & pmu_attr->pmu_type))

			continue;

		if (x86_pmu.hybrid_pmu[i].cpu_type & pmu->cpu_type) {

			next_str = strchr(str, ';');

			if (next_str)

				return snprintf(page, next_str - str + 1, "%s", str);

			else

				return sprintf(page, "%s", str);

		}

		str = strchr(str, ';');

		str++;

	}



	return 0;

}

EXPORT_SYMBOL_GPL(events_hybrid_sysfs_show);



EVENT_ATTR(cpu-cycles,			CPU_CYCLES		);

EVENT_ATTR(instructions,		INSTRUCTIONS		);

EVENT_ATTR(cache-references,		CACHE_REFERENCES	);
@@ -1960,6 +2033,37 @@ static void _x86_pmu_read(struct perf_event *event) 
	x86_perf_event_update(event);

}



void x86_pmu_show_pmu_cap(int num_counters, int num_counters_fixed,

			  u64 intel_ctrl)

{

	pr_info("... version:                %d\n",     x86_pmu.version);

	pr_info("... bit width:              %d\n",     x86_pmu.cntval_bits);

	pr_info("... generic registers:      %d\n",     num_counters);

	pr_info("... value mask:             %016Lx\n", x86_pmu.cntval_mask);

	pr_info("... max period:             %016Lx\n", x86_pmu.max_period);

	pr_info("... fixed-purpose events:   %lu\n",

			hweight64((((1ULL << num_counters_fixed) - 1)

					<< INTEL_PMC_IDX_FIXED) & intel_ctrl));

	pr_info("... event mask:             %016Lx\n", intel_ctrl);

}



/*

 * The generic code is not hybrid friendly. The hybrid_pmu->pmu

 * of the first registered PMU is unconditionally assigned to

 * each possible cpuctx->ctx.pmu.

 * Update the correct hybrid PMU to the cpuctx->ctx.pmu.

 */

void x86_pmu_update_cpu_context(struct pmu *pmu, int cpu)

{

	struct perf_cpu_context *cpuctx;



	if (!pmu->pmu_cpu_context)

		return;



	cpuctx = per_cpu_ptr(pmu->pmu_cpu_context, cpu);

	cpuctx->ctx.pmu = pmu;

}



static int __init init_hw_perf_events(void)

{

	struct x86_pmu_quirk *quirk;
@@ -1993,7 +2097,7 @@ static int __init init_hw_perf_events(void) 
	pmu_check_apic();



	/* sanity check that the hardware exists or is emulated */

	if (!check_hw_exists())

	if (!check_hw_exists(&pmu, x86_pmu.num_counters, x86_pmu.num_counters_fixed))

		return 0;



	pr_cont("%s PMU driver.\n", x86_pmu.name);
@@ -2020,15 +2124,11 @@ static int __init init_hw_perf_events(void) 


	pmu.attr_update = x86_pmu.attr_update;



	pr_info("... version:                %d\n",     x86_pmu.version);

	pr_info("... bit width:              %d\n",     x86_pmu.cntval_bits);

	pr_info("... generic registers:      %d\n",     x86_pmu.num_counters);

	pr_info("... value mask:             %016Lx\n", x86_pmu.cntval_mask);

	pr_info("... max period:             %016Lx\n", x86_pmu.max_period);

	pr_info("... fixed-purpose events:   %lu\n",

			hweight64((((1ULL << x86_pmu.num_counters_fixed) - 1)

					<< INTEL_PMC_IDX_FIXED) & x86_pmu.intel_ctrl));

	pr_info("... event mask:             %016Lx\n", x86_pmu.intel_ctrl);

	if (!is_hybrid()) {

		x86_pmu_show_pmu_cap(x86_pmu.num_counters,

				     x86_pmu.num_counters_fixed,

				     x86_pmu.intel_ctrl);

	}



	if (!x86_pmu.read)

		x86_pmu.read = _x86_pmu_read;
@@ -2055,9 +2155,46 @@ static int __init init_hw_perf_events(void) 
	if (err)

		goto out1;



	if (!is_hybrid()) {

		err = perf_pmu_register(&pmu, "cpu", PERF_TYPE_RAW);

		if (err)

			goto out2;

	} else {

		u8 cpu_type = get_this_hybrid_cpu_type();

		struct x86_hybrid_pmu *hybrid_pmu;

		int i, j;



		if (!cpu_type && x86_pmu.get_hybrid_cpu_type)

			cpu_type = x86_pmu.get_hybrid_cpu_type();



		for (i = 0; i < x86_pmu.num_hybrid_pmus; i++) {

			hybrid_pmu = &x86_pmu.hybrid_pmu[i];



			hybrid_pmu->pmu = pmu;

			hybrid_pmu->pmu.type = -1;

			hybrid_pmu->pmu.attr_update = x86_pmu.attr_update;

			hybrid_pmu->pmu.capabilities |= PERF_PMU_CAP_HETEROGENEOUS_CPUS;

			hybrid_pmu->pmu.capabilities |= PERF_PMU_CAP_EXTENDED_HW_TYPE;



			err = perf_pmu_register(&hybrid_pmu->pmu, hybrid_pmu->name,

						(hybrid_pmu->cpu_type == hybrid_big) ? PERF_TYPE_RAW : -1);

			if (err)

				break;



			if (cpu_type == hybrid_pmu->cpu_type)

				x86_pmu_update_cpu_context(&hybrid_pmu->pmu, raw_smp_processor_id());

		}



		if (i < x86_pmu.num_hybrid_pmus) {

			for (j = 0; j < i; j++)

				perf_pmu_unregister(&x86_pmu.hybrid_pmu[j].pmu);

			pr_warn("Failed to register hybrid PMUs\n");

			kfree(x86_pmu.hybrid_pmu);

			x86_pmu.hybrid_pmu = NULL;

			x86_pmu.num_hybrid_pmus = 0;

			goto out2;

		}

	}



	return 0;
@@ -2182,16 +2319,27 @@ static void free_fake_cpuc(struct cpu_hw_events *cpuc) 
	kfree(cpuc);

}



static struct cpu_hw_events *allocate_fake_cpuc(void)

static struct cpu_hw_events *allocate_fake_cpuc(struct pmu *event_pmu)

{

	struct cpu_hw_events *cpuc;

	int cpu = raw_smp_processor_id();

	int cpu;



	cpuc = kzalloc(sizeof(*cpuc), GFP_KERNEL);

	if (!cpuc)

		return ERR_PTR(-ENOMEM);

	cpuc->is_fake = 1;



	if (is_hybrid()) {

		struct x86_hybrid_pmu *h_pmu;



		h_pmu = hybrid_pmu(event_pmu);

		if (cpumask_empty(&h_pmu->supported_cpus))

			goto error;

		cpu = cpumask_first(&h_pmu->supported_cpus);

	} else

		cpu = raw_smp_processor_id();

	cpuc->pmu = event_pmu;



	if (intel_cpuc_prepare(cpuc, cpu))

		goto error;
@@ -2210,7 +2358,7 @@ static int validate_event(struct perf_event *event) 
	struct event_constraint *c;

	int ret = 0;



	fake_cpuc = allocate_fake_cpuc();

	fake_cpuc = allocate_fake_cpuc(event->pmu);

	if (IS_ERR(fake_cpuc))

		return PTR_ERR(fake_cpuc);
@@ -2244,7 +2392,27 @@ static int validate_group(struct perf_event *event) 
	struct cpu_hw_events *fake_cpuc;

	int ret = -EINVAL, n;



	fake_cpuc = allocate_fake_cpuc();

	/*

	 * Reject events from different hybrid PMUs.

	 */

	if (is_hybrid()) {

		struct perf_event *sibling;

		struct pmu *pmu = NULL;



		if (is_x86_event(leader))

			pmu = leader->pmu;



		for_each_sibling_event(sibling, leader) {

			if (!is_x86_event(sibling))

				continue;

			if (!pmu)

				pmu = sibling->pmu;

			else if (pmu != sibling->pmu)

				return ret;

		}

	}



	fake_cpuc = allocate_fake_cpuc(event->pmu);

	if (IS_ERR(fake_cpuc))

		return PTR_ERR(fake_cpuc);

	/*
@@ -2272,35 +2440,26 @@ static int validate_group(struct perf_event *event) 


static int x86_pmu_event_init(struct perf_event *event)

{

	struct pmu *tmp;

	struct x86_hybrid_pmu *pmu = NULL;

	int err;



	switch (event->attr.type) {

	case PERF_TYPE_RAW:

	case PERF_TYPE_HARDWARE:

	case PERF_TYPE_HW_CACHE:

		break;

	if ((event->attr.type != event->pmu->type) &&

	    (event->attr.type != PERF_TYPE_HARDWARE) &&

	    (event->attr.type != PERF_TYPE_HW_CACHE))

		return -ENOENT;



	default:

	if (is_hybrid() && (event->cpu != -1)) {

		pmu = hybrid_pmu(event->pmu);

		if (!cpumask_test_cpu(event->cpu, &pmu->supported_cpus))

			return -ENOENT;

	}



	err = __x86_pmu_event_init(event);

	if (!err) {

		/*

		 * we temporarily connect event to its pmu

		 * such that validate_group() can classify

		 * it as an x86 event using is_x86_event()

		 */

		tmp = event->pmu;

		event->pmu = &pmu;



		if (event->group_leader != event)

			err = validate_group(event);

		else

			err = validate_event(event);



		event->pmu = tmp;

	}

	if (err) {

		if (event->destroy)
@@ -2485,6 +2644,14 @@ static int x86_pmu_aux_output_match(struct perf_event *event) 
	return 0;

}



static int x86_pmu_filter_match(struct perf_event *event)

{

	if (x86_pmu.filter_match)

		return x86_pmu.filter_match(event);



	return 1;

}



static struct pmu pmu = {

	.pmu_enable		= x86_pmu_enable,

	.pmu_disable		= x86_pmu_disable,
@@ -2512,6 +2679,8 @@ static struct pmu pmu = { 
	.check_period		= x86_pmu_check_period,



	.aux_output_match	= x86_pmu_aux_output_match,



	.filter_match		= x86_pmu_filter_match,

};



void arch_perf_update_userpage(struct perf_event *event,
@@ -2785,6 +2954,11 @@ unsigned long perf_misc_flags(struct pt_regs *regs) 
void perf_get_x86_pmu_capability(struct x86_pmu_capability *cap)

{

	cap->version		= x86_pmu.version;

	/*

	 * KVM doesn't support the hybrid PMU yet.

	 * Return the common value in global x86_pmu,

	 * which available for all cores.

	 */

	cap->num_counters_gp	= x86_pmu.num_counters;

	cap->num_counters_fixed	= x86_pmu.num_counters_fixed;

	cap->bit_width_gp	= x86_pmu.cntval_bits;

arch/x86/events/intel/core.c

+900 −101

File changed.

Preview size limit exceeded, changes collapsed.

arch/x86/events/intel/cstate.c

+36 −10
Original line number Diff line number Diff line
@@ -40,7 +40,8 @@ 
 * Model specific counters:

 *	MSR_CORE_C1_RES: CORE C1 Residency Counter

 *			 perf code: 0x00

 *			 Available model: SLM,AMT,GLM,CNL,ICX,TNT

 *			 Available model: SLM,AMT,GLM,CNL,ICX,TNT,ADL,RPL

 *					  MTL

 *			 Scope: Core (each processor core has a MSR)

 *	MSR_CORE_C3_RESIDENCY: CORE C3 Residency Counter

 *			       perf code: 0x01
@@ -51,46 +52,50 @@ 
 *			       perf code: 0x02

 *			       Available model: SLM,AMT,NHM,WSM,SNB,IVB,HSW,BDW,

 *						SKL,KNL,GLM,CNL,KBL,CML,ICL,ICX,

 *						TGL,TNT,SPR

 *						TGL,TNT,SPR,RPL,MTL

 *			       Scope: Core

 *	MSR_CORE_C7_RESIDENCY: CORE C7 Residency Counter

 *			       perf code: 0x03

 *			       Available model: SNB,IVB,HSW,BDW,SKL,CNL,KBL,CML,

 *						ICL,TGL

 *						ICL,TGL,ADL,RPL,MTL

 *			       Scope: Core

 *	MSR_PKG_C2_RESIDENCY:  Package C2 Residency Counter.

 *			       perf code: 0x00

 *			       Available model: SNB,IVB,HSW,BDW,SKL,KNL,GLM,CNL,

 *						KBL,CML,ICL,ICX,TGL,TNT,SPR

 *						KBL,CML,ICL,ICX,TGL,TNT,SPR,ADL,

 *						RPL,MTL

 *			       Scope: Package (physical package)

 *	MSR_PKG_C3_RESIDENCY:  Package C3 Residency Counter.

 *			       perf code: 0x01

 *			       Available model: NHM,WSM,SNB,IVB,HSW,BDW,SKL,KNL,

 *						GLM,CNL,KBL,CML,ICL,TGL,TNT

 *						GLM,CNL,KBL,CML,ICL,TGL,TNT,

 *						ADL,RPL,MTL

 *			       Scope: Package (physical package)

 *	MSR_PKG_C6_RESIDENCY:  Package C6 Residency Counter.

 *			       perf code: 0x02

 *			       Available model: SLM,AMT,NHM,WSM,SNB,IVB,HSW,BDW,

 *						SKL,KNL,GLM,CNL,KBL,CML,ICL,ICX,

 *						TGL,TNT,SPR

 *						TGL,TNT,SPR,ADL,RPL,MTL

 *			       Scope: Package (physical package)

 *	MSR_PKG_C7_RESIDENCY:  Package C7 Residency Counter.

 *			       perf code: 0x03

 *			       Available model: NHM,WSM,SNB,IVB,HSW,BDW,SKL,CNL,

 *						KBL,CML,ICL,TGL

 *						KBL,CML,ICL,TGL,ADL,RPL,MTL

 *			       Scope: Package (physical package)

 *	MSR_PKG_C8_RESIDENCY:  Package C8 Residency Counter.

 *			       perf code: 0x04

 *			       Available model: HSW ULT,KBL,CNL,CML,ICL,TGL

 *			       Available model: HSW ULT,KBL,CNL,CML,ICL,TGL,ADL,

 *			       RPL,MTL

 *			       Scope: Package (physical package)

 *	MSR_PKG_C9_RESIDENCY:  Package C9 Residency Counter.

 *			       perf code: 0x05

 *			       Available model: HSW ULT,KBL,CNL,CML,ICL,TGL

 *			       Available model: HSW ULT,KBL,CNL,CML,ICL,TGL,ADL,

 *			       RPL,MTL

 *			       Scope: Package (physical package)

 *	MSR_PKG_C10_RESIDENCY: Package C10 Residency Counter.

 *			       perf code: 0x06

 *			       Available model: HSW ULT,KBL,GLM,CNL,CML,ICL,TGL,

 *						TNT

 *						TNT,ADL,RPL,MTL

 *			       Scope: Package (physical package)

 *

 */
@@ -571,6 +576,20 @@ static const struct cstate_model icx_cstates __initconst = { 
				  BIT(PERF_CSTATE_PKG_C6_RES),

};



static const struct cstate_model adl_cstates __initconst = {

	.core_events		= BIT(PERF_CSTATE_CORE_C1_RES) |

				  BIT(PERF_CSTATE_CORE_C6_RES) |

				  BIT(PERF_CSTATE_CORE_C7_RES),



	.pkg_events		= BIT(PERF_CSTATE_PKG_C2_RES) |

				  BIT(PERF_CSTATE_PKG_C3_RES) |

				  BIT(PERF_CSTATE_PKG_C6_RES) |

				  BIT(PERF_CSTATE_PKG_C7_RES) |

				  BIT(PERF_CSTATE_PKG_C8_RES) |

				  BIT(PERF_CSTATE_PKG_C9_RES) |

				  BIT(PERF_CSTATE_PKG_C10_RES),

};



static const struct cstate_model slm_cstates __initconst = {

	.core_events		= BIT(PERF_CSTATE_CORE_C1_RES) |

				  BIT(PERF_CSTATE_CORE_C6_RES),
@@ -652,6 +671,7 @@ static const struct x86_cpu_id intel_cstates_match[] __initconst = { 
	X86_MATCH_INTEL_FAM6_MODEL(ATOM_TREMONT_D,	&glm_cstates),

	X86_MATCH_INTEL_FAM6_MODEL(ATOM_TREMONT,	&glm_cstates),

	X86_MATCH_INTEL_FAM6_MODEL(ATOM_TREMONT_L,	&glm_cstates),

	X86_MATCH_INTEL_FAM6_MODEL(ATOM_GRACEMONT,	&adl_cstates),



	X86_MATCH_INTEL_FAM6_MODEL(ICELAKE_L,		&icl_cstates),

	X86_MATCH_INTEL_FAM6_MODEL(ICELAKE,		&icl_cstates),
@@ -662,6 +682,12 @@ static const struct x86_cpu_id intel_cstates_match[] __initconst = { 


	X86_MATCH_INTEL_FAM6_MODEL(TIGERLAKE_L,		&icl_cstates),

	X86_MATCH_INTEL_FAM6_MODEL(TIGERLAKE,		&icl_cstates),

	X86_MATCH_INTEL_FAM6_MODEL(ALDERLAKE,		&adl_cstates),

	X86_MATCH_INTEL_FAM6_MODEL(ALDERLAKE_L,		&adl_cstates),

	X86_MATCH_INTEL_FAM6_MODEL(RAPTORLAKE,		&adl_cstates),

	X86_MATCH_INTEL_FAM6_MODEL(RAPTORLAKE_P,	&adl_cstates),

	X86_MATCH_INTEL_FAM6_MODEL(METEORLAKE,		&adl_cstates),

	X86_MATCH_INTEL_FAM6_MODEL(METEORLAKE_L,	&adl_cstates),

	{ },

};

MODULE_DEVICE_TABLE(x86cpu, intel_cstates_match);

arch/x86/events/intel/ds.c

+166 −49

File changed.

Preview size limit exceeded, changes collapsed.

arch/x86/events/intel/uncore.c

+10 −0
Original line number Diff line number Diff line
@@ -1771,6 +1771,11 @@ static const struct intel_uncore_init_fun tgl_l_uncore_init __initconst = { 
	.mmio_init = tgl_l_uncore_mmio_init,

};



static const struct intel_uncore_init_fun adl_uncore_init __initconst = {

	.cpu_init = adl_uncore_cpu_init,

	.mmio_init = adl_uncore_mmio_init,

};



static const struct intel_uncore_init_fun icx_uncore_init __initconst = {

	.cpu_init = icx_uncore_cpu_init,

	.pci_init = icx_uncore_pci_init,
@@ -1832,9 +1837,14 @@ static const struct x86_cpu_id intel_uncore_match[] __initconst = { 
	X86_MATCH_INTEL_FAM6_MODEL(ICELAKE_X,		&icx_uncore_init),

	X86_MATCH_INTEL_FAM6_MODEL(TIGERLAKE_L,		&tgl_l_uncore_init),

	X86_MATCH_INTEL_FAM6_MODEL(TIGERLAKE,		&tgl_uncore_init),

	X86_MATCH_INTEL_FAM6_MODEL(ALDERLAKE,		&adl_uncore_init),

	X86_MATCH_INTEL_FAM6_MODEL(ALDERLAKE_L,		&adl_uncore_init),

	X86_MATCH_INTEL_FAM6_MODEL(RAPTORLAKE,		&adl_uncore_init),

	X86_MATCH_INTEL_FAM6_MODEL(RAPTORLAKE_P,	&adl_uncore_init),

	X86_MATCH_INTEL_FAM6_MODEL(SAPPHIRERAPIDS_X,	&spr_uncore_init),

	X86_MATCH_INTEL_FAM6_MODEL(EMERALDRAPIDS_X,	&spr_uncore_init),

	X86_MATCH_INTEL_FAM6_MODEL(ATOM_TREMONT_D,	&snr_uncore_init),

	X86_MATCH_INTEL_FAM6_MODEL(ATOM_GRACEMONT,	&adl_uncore_init),

	{},

};

MODULE_DEVICE_TABLE(x86cpu, intel_uncore_match);