I agree with previous posts.
The only drawback is that you have to relearn your startup a little bit. This is minor -- you just have to add a few revs more than normal. The other things that you mention (slower launch, lower gas mileage, rough idle) are not something that I've experienced (VE with 8.5# flywheel).
Although a heavy flywheel may make a difference for a drag launch in a DE, with my VE, I'm spinning wheels all the way through first with a feathered clutch take off so I don't see any advantage to having more flywheel momemtum for a VE.
On a road course, the light flywheel makes matching revs for downshifting smoother -- covering less precise throttle work with a smooth downshift.
Overall, there is no significant downside, IMHO. I'd do it again. There are lots of mods I don't say that about.