Skip to content
  1. Jan 08, 2020
    • Wenpeng Liang's avatar
      RDMA/hns: Avoid printing address of mtt page · eca44507
      Wenpeng Liang authored
      Address of a page shouldn't be printed in case of security issues.
      
      Link: https://lore.kernel.org/r/1578313276-29080-2-git-send-email-liweihang@huawei.com
      
      
      Signed-off-by: default avatarWenpeng Liang <liangwenpeng@huawei.com>
      Signed-off-by: default avatarWeihang Li <liweihang@huawei.com>
      Signed-off-by: default avatarJason Gunthorpe <jgg@mellanox.com>
      eca44507
    • Chuck Lever's avatar
      RDMA/core: Add trace points to follow MR allocation · 622db5b6
      Chuck Lever authored
      Track the lifetime of ib_mr objects. Here's sample output from a test run
      with NFS/RDMA:
      
                 <...>-361   [009] 79238.772782: mr_alloc:             pd.id=3 mr.id=11 type=MEM_REG max_num_sg=30 rc=0
                 <...>-361   [009] 79238.772812: mr_alloc:             pd.id=3 mr.id=12 type=MEM_REG max_num_sg=30 rc=0
                 <...>-361   [009] 79238.772839: mr_alloc:             pd.id=3 mr.id=13 type=MEM_REG max_num_sg=30 rc=0
                 <...>-361   [009] 79238.772866: mr_alloc:             pd.id=3 mr.id=14 type=MEM_REG max_num_sg=30 rc=0
                 <...>-361   [009] 79238.772893: mr_alloc:             pd.id=3 mr.id=15 type=MEM_REG max_num_sg=30 rc=0
                 <...>-361   [009] 79238.772921: mr_alloc:             pd.id=3 mr.id=16 type=MEM_REG max_num_sg=30 rc=0
                 <...>-361   [009] 79238.772947: mr_alloc:             pd.id=3 mr.id=17 type=MEM_REG max_num_sg=30 rc=0
                 <...>-361   [009] 79238.772974: mr_alloc:             pd.id=3 mr.id=18 type=MEM_REG max_num_sg=30 rc=0
                 <...>-361   [009] 79238.773001: mr_alloc:             pd.id=3 mr.id=19 type=MEM_REG max_num_sg=30 rc=0
                 <...>-361   [009] 79238.773028: mr_alloc:             pd.id=3 mr.id=20 type=MEM_REG max_num_sg=30 rc=0
                 <...>-361   [009] 79238.773055: mr_alloc:             pd.id=3 mr.id=21 type=MEM_REG max_num_sg=30 rc=0
                 <...>-361   [009] 79240.270942: mr_alloc:             pd.id=3 mr.id=22 type=MEM_REG max_num_sg=30 rc=0
                 <...>-361   [009] 79240.270975: mr_alloc:             pd.id=3 mr.id=23 type=MEM_REG max_num_sg=30 rc=0
                 <...>-361   [009] 79240.271007: mr_alloc:             pd.id=3 mr.id=24 type=MEM_REG max_num_sg=30 rc=0
                 <...>-361   [009] 79240.271036: mr_alloc:             pd.id=3 mr.id=25 type=MEM_REG max_num_sg=30 rc=0
                 <...>-361   [009] 79240.271067: mr_alloc:             pd.id=3 mr.id=26 type=MEM_REG max_num_sg=30 rc=0
                 <...>-361   [009] 79240.271095: mr_alloc:             pd.id=3 mr.id=27 type=MEM_REG max_num_sg=30 rc=0
                 <...>-361   [009] 79240.271121: mr_alloc:             pd.id=3 mr.id=28 type=MEM_REG max_num_sg=30 rc=0
                 <...>-361   [009] 79240.271153: mr_alloc:             pd.id=3 mr.id=29 type=MEM_REG max_num_sg=30 rc=0
                 <...>-361   [009] 79240.271181: mr_alloc:             pd.id=3 mr.id=30 type=MEM_REG max_num_sg=30 rc=0
                 <...>-361   [009] 79240.271208: mr_alloc:             pd.id=3 mr.id=31 type=MEM_REG max_num_sg=30 rc=0
                 <...>-361   [009] 79240.271236: mr_alloc:             pd.id=3 mr.id=32 type=MEM_REG max_num_sg=30 rc=0
                 <...>-4351  [001] 79242.299400: mr_dereg:             mr.id=32
                 <...>-4351  [001] 79242.299467: mr_dereg:             mr.id=31
                 <...>-4351  [001] 79242.299554: mr_dereg:             mr.id=30
                 <...>-4351  [001] 79242.299615: mr_dereg:             mr.id=29
                 <...>-4351  [001] 79242.299684: mr_dereg:             mr.id=28
                 <...>-4351  [001] 79242.299748: mr_dereg:             mr.id=27
                 <...>-4351  [001] 79242.299812: mr_dereg:             mr.id=26
                 <...>-4351  [001] 79242.299874: mr_dereg:             mr.id=25
                 <...>-4351  [001] 79242.299944: mr_dereg:             mr.id=24
                 <...>-4351  [001] 79242.300009: mr_dereg:             mr.id=23
                 <...>-4351  [001] 79242.300190: mr_dereg:             mr.id=22
                 <...>-4351  [001] 79242.300263: mr_dereg:             mr.id=21
                 <...>-4351  [001] 79242.300326: mr_dereg:             mr.id=20
                 <...>-4351  [001] 79242.300388: mr_dereg:             mr.id=19
                 <...>-4351  [001] 79242.300450: mr_dereg:             mr.id=18
                 <...>-4351  [001] 79242.300516: mr_dereg:             mr.id=17
                 <...>-4351  [001] 79242.300629: mr_dereg:             mr.id=16
                 <...>-4351  [001] 79242.300718: mr_dereg:             mr.id=15
                 <...>-4351  [001] 79242.300784: mr_dereg:             mr.id=14
                 <...>-4351  [001] 79242.300879: mr_dereg:             mr.id=13
                 <...>-4351  [001] 79242.300945: mr_dereg:             mr.id=12
                 <...>-4351  [001] 79242.301012: mr_dereg:             mr.id=11
      
      Some features of the output:
      - The lifetime and owner PD of each MR is clearly visible.
      - The type of MR is captured, as is the SGE array size.
      - Failing MR allocation can be recorded.
      
      Link: https://lore.kernel.org/r/20191218201820.30584.34636.stgit@manet.1015granger.net
      
      
      Signed-off-by: default avatarChuck Lever <chuck.lever@oracle.com>
      Signed-off-by: default avatarJason Gunthorpe <jgg@mellanox.com>
      622db5b6
    • Chuck Lever's avatar
      RDMA/core: Trace points for diagnosing completion queue issues · 3e5901cb
      Chuck Lever authored
      Sample trace events:
      
         kworker/u29:0-300   [007]   120.042217: cq_alloc:             cq.id=4 nr_cqe=161 comp_vector=2 poll_ctx=WORKQUEUE
                <idle>-0     [002]   120.056292: cq_schedule:          cq.id=4
          kworker/2:1H-482   [002]   120.056402: cq_process:           cq.id=4 wake-up took 109 [us] from interrupt
          kworker/2:1H-482   [002]   120.056407: cq_poll:              cq.id=4 requested 16, returned 1
                <idle>-0     [002]   120.067503: cq_schedule:          cq.id=4
          kworker/2:1H-482   [002]   120.067537: cq_process:           cq.id=4 wake-up took 34 [us] from interrupt
          kworker/2:1H-482   [002]   120.067541: cq_poll:              cq.id=4 requested 16, returned 1
                <idle>-0     [002]   120.067657: cq_schedule:          cq.id=4
          kworker/2:1H-482   [002]   120.067672: cq_process:           cq.id=4 wake-up took 15 [us] from interrupt
          kworker/2:1H-482   [002]   120.067674: cq_poll:              cq.id=4 requested 16, returned 1
      
       ...
      
               systemd-1     [002]   122.392653: cq_schedule:          cq.id=4
          kworker/2:1H-482   [002]   122.392688: cq_process:           cq.id=4 wake-up took 35 [us] from interrupt
          kworker/2:1H-482   [002]   122.392693: cq_poll:              cq.id=4 requested 16, returned 16
          kworker/2:1H-482   [002]   122.392836: cq_poll:              cq.id=4 requested 16, returned 16
          kworker/2:1H-482   [002]   122.392970: cq_poll:              cq.id=4 requested 16, returned 16
          kworker/2:1H-482   [002]   122.393083: cq_poll:              cq.id=4 requested 16, returned 16
          kworker/2:1H-482   [002]   122.393195: cq_poll:              cq.id=4 requested 16, returned 3
      
      Several features to note in this output:
       - The WCE count and context type are reported at allocation time
       - The CPU and kworker for each CQ is evident
       - The CQ's restracker ID is tagged on each trace event
       - CQ poll scheduling latency is measured
       - Details about how often single completions occur versus multiple
         completions are evident
       - The cost of the ULP's completion handler is recorded
      
      Link: https://lore.kernel.org/r/20191218201815.30584.3481.stgit@manet.1015granger.net
      
      
      Signed-off-by: default avatarChuck Lever <chuck.lever@oracle.com>
      Reviewed-by: default avatarParav Pandit <parav@mellanox.com>
      Signed-off-by: default avatarJason Gunthorpe <jgg@mellanox.com>
      3e5901cb
    • Chuck Lever's avatar
      RDMA/cma: Add trace points in RDMA Connection Manager · ed999f82
      Chuck Lever authored
      Record state transitions as each connection is established. The IP address
      of both peers and the Type of Service is reported. These trace points are
      not in performance hot paths.
      
      Also, record each cm_event_handler call to ULPs. This eliminates the need
      for each ULP to add its own similar trace point in its CM event handler
      function.
      
      These new trace points appear in a new trace subsystem called "rdma_cma".
      
      Sample events:
      
                 <...>-220   [004]   121.430733: cm_id_create:         cm.id=0
                 <...>-472   [003]   121.430991: cm_event_handler:     cm.id=0 src=192.168.2.51:35090 dst=192.168.2.55:20049 tos=0 ADDR_RESOLVED (0/0)
                 <...>-472   [003]   121.430995: cm_event_done:        cm.id=0 src=192.168.2.51:35090 dst=192.168.2.55:20049 tos=0 result=0
                 <...>-472   [003]   121.431172: cm_event_handler:     cm.id=0 src=192.168.2.51:35090 dst=192.168.2.55:20049 tos=0 ROUTE_RESOLVED (2/0)
                 <...>-472   [003]   121.431174: cm_event_done:        cm.id=0 src=192.168.2.51:35090 dst=192.168.2.55:20049 tos=0 result=0
                 <...>-220   [004]   121.433480: cm_qp_create:         cm.id=0 src=192.168.2.51:35090 dst=192.168.2.55:20049 tos=0 pd.id=2 qp_type=RC send_wr=4091 recv_wr=256 qp_num=521 rc=0
                 <...>-220   [004]   121.433577: cm_send_req:          cm.id=0 src=192.168.2.51:35090 dst=192.168.2.55:20049 tos=0 qp_num=521
           kworker/1:2-973   [001]   121.436190: cm_send_mra:          cm.id=0 src=192.168.2.51:35090 dst=192.168.2.55:20049 tos=0
           kworker/1:2-973   [001]   121.436340: cm_send_rtu:          cm.id=0 src=192.168.2.51:35090 dst=192.168.2.55:20049 tos=0
           kworker/1:2-973   [001]   121.436359: cm_event_handler:     cm.id=0 src=192.168.2.51:35090 dst=192.168.2.55:20049 tos=0 ESTABLISHED (9/0)
           kworker/1:2-973   [001]   121.436365: cm_event_done:        cm.id=0 src=192.168.2.51:35090 dst=192.168.2.55:20049 tos=0 result=0
                 <...>-1975  [005]   123.161954: cm_disconnect:        cm.id=0 src=192.168.2.51:35090 dst=192.168.2.55:20049 tos=0
                 <...>-1975  [005]   123.161974: cm_sent_dreq:         cm.id=0 src=192.168.2.51:35090 dst=192.168.2.55:20049 tos=0
                 <...>-220   [004]   123.162102: cm_disconnect:        cm.id=0 src=192.168.2.51:35090 dst=192.168.2.55:20049 tos=0
           kworker/0:1-13    [000]   123.162391: cm_event_handler:     cm.id=0 src=192.168.2.51:35090 dst=192.168.2.55:20049 tos=0 DISCONNECTED (10/0)
           kworker/0:1-13    [000]   123.162393: cm_event_done:        cm.id=0 src=192.168.2.51:35090 dst=192.168.2.55:20049 tos=0 result=0
                 <...>-220   [004]   123.164456: cm_qp_destroy:        cm.id=0 src=192.168.2.51:35090 dst=192.168.2.55:20049 tos=0 qp_num=521
                 <...>-220   [004]   123.165290: cm_id_destroy:        cm.id=0 src=192.168.2.51:35090 dst=192.168.2.55:20049 tos=0
      
      Some features to note:
      - restracker ID of the rdma_cm_id is tagged on each trace event
      - The source and destination IP addresses and TOS are reported
      - CM event upcalls are shown with decoded event and status
      - CM state transitions are reported
      - rdma_cm_id lifetime events are captured
      - The latency of ULP CM event handlers is reported
      - Lifetime events of associated QPs are reported
      - Device removal and insertion is reported
      
      This patch is based on previous work by:
      
      Saeed Mahameed <saeedm@mellanox.com>
      Mukesh Kacker <mukesh.kacker@oracle.com>
      Ajaykumar Hotchandani <ajaykumar.hotchandani@oracle.com>
      Aron Silverton <aron.silverton@oracle.com>
      Avinash Repaka <avinash.repaka@oracle.com>
      Somasundaram Krishnasamy <somasundaram.krishnasamy@oracle.com>
      
      Link: https://lore.kernel.org/r/20191218201810.30584.3052.stgit@manet.1015granger.net
      
      
      Signed-off-by: default avatarChuck Lever <chuck.lever@oracle.com>
      Signed-off-by: default avatarJason Gunthorpe <jgg@mellanox.com>
      ed999f82
  2. Jan 04, 2020
  3. Dec 30, 2019
  4. Dec 29, 2019
  5. Dec 28, 2019