Is there a difference? Yes.
Is it a big deal? Depends on how anal you are about efficiency (and from the looks of the end tanks on that IC, I can assume you're not terribly anal and more interested in making it actually fit in your car nicely).
Are you doing blow through or draw through?
If draw through just make your turbo intake pipes 4" and then do 3" (reducing to 2.5" for the IC).
If blow through, do 3" pipes as well with the appropriate reducers.
If you have a 3" throttle body though, and are willing to run 3" charge pipes, I'd seriously consider an intercooler with 3" in/outs.