One simple thing to check before you dig too deeply - check to see if you can detect a pull to one side or the other when the car is coasting. It might be very subtle but if you do detect a pull, jack up the rear and see if the rear wheel on the side it pulls towards is stiff or difficult to spin. Anything dragging on that wheel - brake caliper or even a bad bearing - will be overcome by torque under power but will slow the wheel enough when you let off to cause a change of direction. The faster you are going and the more suddenly you lift off, the more dramatic the effect will be. Letting up on the throttle at speed produces the same effect that would occur if you only had brakes on that one wheel and applied them at say, 60 or 70 mph. Could be a bit unsettling, to say the least.
