If you go too big, you'll start to shift the power band higher. It might feel like turbo lag on the low end. Other than that, it wouldn't hurt anything except your ears, maybe your wallet. Drone would be terrible. You might have new rattles in your interior. Anything over 3 inches for 99% of the cars out on the road is moot.
This 100% contradicts your last statement. Assuming the flow into the pipe is reasonably smooth you would see no reduction at all by switching to a larger diameter exhaust. It simply is not true. The turbine operates on a pressure delta, and because the pressure before the turbo is the same, if you reduce the pressure after the turbine you will have a greater delta, and thus a more efficient turbo.
Yes, it will be LOUDer. And you really do not need (want) the flow advantage of a 4in pipe until you'er over the 500-600hp region.
Even N/A engines don't "need" the back pressure. This is an unfortunate "old hot rodder's tale" back from when they did not understand the differences between gas velocity and back pressure. But you're right, your turbo certainly does not need any back pressure behind it.
