Johnny, I thought you were talking real big turbos so I just avoided this thread. From the sounds of it you're talking about stuff I've dealt with directly. Having driven Dan's swapped S13 with a GT3076R at around 450 WHP (and other RWD cars with turbos larger than typical for their displacement) on countless occasions I'll say you're dead on with your assumptions in your first post. There's a lot to be said about smaller turbos and quick boost onset for a daily driver, but when you want serious power and are more concerned with the goals you outlined in your first post, the larger turbos are the only thing that can deliver.
As for "living with it" on a daily basis, it is not bad at all, and like you said, as long as you have about 3k rpm of power band, you're all set. There are trade offs with everything, but the gain in traction (from a roll) you get from a bigger turbo is great to have, especially for FWD cars. The delay in boost onset can also change your driving habits. Many find they drive much slower and avoid attracting the wrong kind of attention from the police with a bigger turbo since spooling it turns into an event instead of just a typical thing because of the time it takes, the need to prepare yourself, make sure you're going straight, etc. etc.
Sorry I'm rambling. If you have the mindset I think you have, you'll fully enjoy a T3 flanged turbo.