Yes you can remove some metl from the back of the adapters. Increase the ID of the back side just enough to clear the OD of the raised part on the drums. It will be ok. IF YOU ARE NOT USING ANTI SIEZE ON THE NUT CHAMFERS DO IT!
Why are the holes chamfered on the back side??? This would concentrate the torque applied by the drive axles through the studs into a sharp edge in each hole. By design it would seem to act like a "Knife" and attempt to shear the studs ???
Questionable design
Rick