io_uring: only flush workqueues on fileset removal

commit 8a997340 upstream.

We should not remove the workqueue, we just need to ensure that the
workqueues are synced. The workqueues are torn down on ctx removal.

Cc: stable@vger.kernel.org
Fixes: 6b06314c ("io_uring: add file set registration")
Reported-by: Stefan Hajnoczi <stefanha@redhat.com>
Signed-off-by: Jens Axboe <axboe@kernel.dk>
Signed-off-by: Greg Kroah-Hartman <gregkh@linuxfoundation.org>
[PG: use 5.3.x-stable version for this v5.2.x codebase.]
Signed-off-by: Paul Gortmaker <paul.gortmaker@windriver.com>