Commit d772b086 authored by Guangguan Wang's avatar Guangguan Wang Committed by Yingyu Zeng
Browse files

net/smc: send cdc msg inline if qp has sufficient inline space 

mainline inclusion
from mainline-v5.19-rc1
commit b632eb06
category: bugfix
bugzilla: https://gitee.com/openeuler/kernel/issues/I77V5Z
CVE: NA

Reference: https://git.kernel.org/pub/scm/linux/kernel/git/torvalds/linux.git/commit/net/smc?id=b632eb06973209dfac1eba3a9fbd13f0041f3e45



--------------------------------

As cdc msg's length is 44B, cdc msgs can be sent inline in
most rdma devices, which can help reducing sending latency.

In my test environment, which are 2 VMs running on the same
physical host and whose NICs(ConnectX-4Lx) are working on
SR-IOV mode, qperf shows 0.4us-0.7us improvement in latency.

Test command:
server: smc_run taskset -c 1 qperf
client: smc_run taskset -c 1 qperf <server ip> -oo \
		msg_size:1:2K:*2 -t 30 -vu tcp_lat

The results shown below:
msgsize     before       after
1B          11.9 us      11.2 us (-0.7 us)
2B          11.7 us      11.2 us (-0.5 us)
4B          11.7 us      11.3 us (-0.4 us)
8B          11.6 us      11.2 us (-0.4 us)
16B         11.7 us      11.3 us (-0.4 us)
32B         11.7 us      11.3 us (-0.4 us)
64B         11.7 us      11.2 us (-0.5 us)
128B        11.6 us      11.2 us (-0.4 us)
256B        11.8 us      11.2 us (-0.6 us)
512B        11.8 us      11.4 us (-0.4 us)
1KB         11.9 us      11.4 us (-0.5 us)
2KB         12.1 us      11.5 us (-0.6 us)

Signed-off-by: default avatarGuangguan Wang <guangguan.wang@linux.alibaba.com>
Reviewed-by: default avatarTony Lu <tonylu@linux.alibaba.com>
Tested-by: default avatarkernel test robot <lkp@intel.com>
Acked-by: default avatarKarsten Graul <kgraul@linux.ibm.com>
Signed-off-by: default avatarJakub Kicinski <kuba@kernel.org>
Signed-off-by: default avatarYingyu Zeng <zengyingyu@sangfor.com.cn>
parent 92b03342

net/smc/smc_ib.c

+1 −0
Original line number Diff line number Diff line
@@ -371,6 +371,7 @@ int smc_ib_create_queue_pair(struct smc_link *lnk) 
			.max_recv_wr = SMC_WR_BUF_CNT * 3,

			.max_send_sge = SMC_IB_MAX_SEND_SGE,

			.max_recv_sge = 1,

			.max_inline_data = 0,

		},

		.sq_sig_type = IB_SIGNAL_REQ_WR,

		.qp_type = IB_QPT_RC,

net/smc/smc_wr.c

+4 −1
Original line number Diff line number Diff line
@@ -488,9 +488,10 @@ void smc_wr_remember_qp_attr(struct smc_link *lnk) 
static void smc_wr_init_sge(struct smc_link *lnk)

{

	u32 i;

	bool send_inline = (lnk->qp_attr.cap.max_inline_data > SMC_WR_TX_SIZE);



	for (i = 0; i < lnk->wr_tx_cnt; i++) {

		lnk->wr_tx_sges[i].addr =

		lnk->wr_tx_sges[i].addr = send_inline ? (uintptr_t)(&lnk->wr_tx_bufs[i]) :

			lnk->wr_tx_dma_addr + i * SMC_WR_BUF_SIZE;

		lnk->wr_tx_sges[i].length = SMC_WR_TX_SIZE;

		lnk->wr_tx_sges[i].lkey = lnk->roce_pd->local_dma_lkey;
@@ -508,6 +509,8 @@ static void smc_wr_init_sge(struct smc_link *lnk) 
		lnk->wr_tx_ibs[i].opcode = IB_WR_SEND;

		lnk->wr_tx_ibs[i].send_flags =

			IB_SEND_SIGNALED | IB_SEND_SOLICITED;

		if (send_inline)

			lnk->wr_tx_ibs[i].send_flags |= IB_SEND_INLINE;

		lnk->wr_tx_rdmas[i].wr_tx_rdma[0].wr.opcode = IB_WR_RDMA_WRITE;

		lnk->wr_tx_rdmas[i].wr_tx_rdma[1].wr.opcode = IB_WR_RDMA_WRITE;

		lnk->wr_tx_rdmas[i].wr_tx_rdma[0].wr.sg_list =