iommu/vt-d: Only call domain_remove_one_dev_info to detach old domain

There is no need to make a difference here between VM and
non-VM domains, so simplify this code here.

Signed-off-by: Joerg Roedel <jroedel@suse.de>