Skip to content
  1. Mar 30, 2014
    • Alex Elder's avatar
      rbd: drop an unsafe assertion · 638c323c
      Alex Elder authored
      
      
      Olivier Bonvalet reported having repeated crashes due to a failed
      assertion he was hitting in rbd_img_obj_callback():
      
          Assertion failure in rbd_img_obj_callback() at line 2165:
      	rbd_assert(which >= img_request->next_completion);
      
      With a lot of help from Olivier with reproducing the problem
      we were able to determine the object and image requests had
      already been completed (and often freed) at the point the
      assertion failed.
      
      There was a great deal of discussion on the ceph-devel mailing list
      about this.  The problem only arose when there were two (or more)
      object requests in an image request, and the problem was always
      seen when the second request was being completed.
      
      The problem is due to a race in the window between setting the
      "done" flag on an object request and checking the image request's
      next completion value.  When the first object request completes, it
      checks to see if its successor request is marked "done", and if
      so, that request is also completed.  In the process, the image
      request's next_completion value is updated to reflect that both
      the first and second requests are completed.  By the time the
      second request is able to check the next_completion value, it
      has been set to a value *greater* than its own "which" value,
      which caused an assertion to fail.
      
      Fix this problem by skipping over any completion processing
      unless the completing object request is the next one expected.
      Test only for inequality (not >=), and eliminate the bad
      assertion.
      
      Tested-by: default avatarOlivier Bonvalet <ob@daevel.fr>
      Signed-off-by: default avatarAlex Elder <elder@linaro.org>
      Reviewed-by: default avatarSage Weil <sage@inktank.com>
      Reviewed-by: default avatarIlya Dryomov <ilya.dryomov@inktank.com>
      638c323c
  2. Mar 03, 2014
  3. Mar 02, 2014
  4. Mar 01, 2014
  5. Feb 28, 2014