There shouldn't be any "loss of torque", although it may feel like it as you are moving the power band up a bit higher with a step-up in exhaust size. What you lose on the bottom you gain on the top. However if your exhaust system flows nicely altogether you probably won't feel much of a difference.
If you are keeping you primaries/secondaries all the same size, and the scavenging effect of the exhaust gases has already taken place than going bigger after that is a good thing! Anybody want to add to that??
Edit: See below what cory said.
I was beating around the bush but I kinda said it here (see bold above) ^ Bigger is good and sounds better/deeper too! First part of statement was in general.