Revert "cpufreq: Fix governor module removal race"

This reverts commit 0049457b which was
commit a8b149d3 upstream.

The backport was not correct, so just drop it entirely.

Reported-by: Ben Hutchings <ben.hutchings@codethink.co.uk>
Cc: Rafael J. Wysocki <rafael.j.wysocki@intel.com>
Cc: Viresh Kumar <viresh.kumar@linaro.org>
Cc: Sasha Levin <alexander.levin@microsoft.com>
Signed-off-by: Greg Kroah-Hartman <gregkh@linuxfoundation.org>