xfs: also remove cached ACLs when removing the underlying attr

We should not just invalidate the ACL when setting the underlying
attribute, but also when removing it.  The ioctl interface gets that
right, but the normal xattr inteface skipped the xfs_forget_acl due
to an early return.

Signed-off-by: Christoph Hellwig <hch@lst.de>
Reviewed-by: Darrick J. Wong <darrick.wong@oracle.com>
Signed-off-by: Darrick J. Wong <darrick.wong@oracle.com>