Commit 67561efe authored by Kefeng Wang's avatar Kefeng Wang
Browse files

arm64: optimize flush tlb kernel range 

mainline inclusion
from mainline-v6.13-rc1
commit a923705c69f7f4ebe6a5488c1f556bed12d28031
category: performance
bugzilla: https://gitee.com/openeuler/kernel/issues/IB82FR


CVE: NA

-------------------------------------------------

Currently the kernel TLBs is flushed page by page if the target
VA range is less than MAX_DVM_OPS * PAGE_SIZE, otherwise we'll
brutally issue a TLBI ALL.

But we could optimize it when CPU supports TLB range operations,
convert to use __flush_tlb_range_op() like other tlb range flush
to improve performance.

Co-developed-by: default avatarYicong Yang <yangyicong@hisilicon.com>
Signed-off-by: default avatarYicong Yang <yangyicong@hisilicon.com>
Signed-off-by: default avatarKefeng Wang <wangkefeng.wang@huawei.com>
Reviewed-by: default avatarAnshuman Khandual <anshuman.khandual@arm.com>
Link: https://lore.kernel.org/r/20240923131351.713304-3-wangkefeng.wang@huawei.com


Signed-off-by: default avatarCatalin Marinas <catalin.marinas@arm.com>
(cherry picked from commit a923705c69f7f4ebe6a5488c1f556bed12d28031)
[KF: no lpa2 support]
Signed-off-by: default avatarKefeng Wang <wangkefeng.wang@huawei.com>
parent 7d6fd04f

arch/arm64/include/asm/tlbflush.h

+8 −7
Original line number Diff line number Diff line
@@ -469,19 +469,20 @@ static inline void flush_tlb_range(struct vm_area_struct *vma, 


static inline void flush_tlb_kernel_range(unsigned long start, unsigned long end)

{

	unsigned long addr;

	const unsigned long stride = PAGE_SIZE;

	unsigned long pages;



	if ((end - start) > (MAX_DVM_OPS * PAGE_SIZE)) {

	start = round_down(start, stride);

	end = round_up(end, stride);

	pages = (end - start) >> PAGE_SHIFT;



	if (__flush_tlb_range_limit_excess(start, end, pages, stride)) {

		flush_tlb_all();

		return;

	}



	start = __TLBI_VADDR(start, 0);

	end = __TLBI_VADDR(end, 0);



	dsb(ishst);

	for (addr = start; addr < end; addr += 1 << (PAGE_SHIFT - 12))

		__tlbi(vaale1is, addr);

	__flush_tlb_range_op(vaale1is, start, pages, stride, 0, 0, false);

	dsb(ish);

	isb();

}