mlxsw: spectrum: Move port used check outside port remove function

Be symmentrical with create and do the check outside the remove function.

Signed-off-by: Jiri Pirko <jiri@mellanox.com>
Signed-off-by: Elad Raz <eladr@mellanox.com>
Signed-off-by: David S. Miller <davem@davemloft.net>