net/mlx4_core: Adjust port number in qp_attach wrapper when detaching

When using single ported VFs and the VF is using port 2, we need
to adjust the port accordingly (change it from 1 to 2).

Fixes: 449fc488

 ('net/mlx4: Adapt code for N-Port VF')
Signed-off-by: Matan Barak <matanb@mellanox.com>
Signed-off-by: Jack Morgenstein <jackm@mellanox.com>
Signed-off-by: Or Gerlitz <ogerlitz@mellanox.com>
Signed-off-by: David S. Miller <davem@davemloft.net>