Quite likely it was some other reason - not the flywheel. Maybe he had to brake more because his car was running up on your tail as it accelerated faster coming out of the turns
.
The heavier flywheel is generally more pleasant to drive on the street as there is less rpm drop during shifts making them smoother and less peaky. A lighter flywheel pays dividends on a high-revving car on a road course but could be more of an annoyance than an advantage behind a high torque, higher compression engine; especially in a heavy car. Was this in the lil' Cobra or the bigger, heavier one?