I have a friend with a 78 Malibu, street car, with a 406 stroker, fully built, with a nice shot of nitrous and all the supporting mods. He has dual 2.5 inch pipes with a set of Flowmasters that are quieter than mine. I'm not sure about horsepower numbers, but he ran 6.20s in the 1/8 a couple weeks ago.
Restrictive? Maybe.
Necessary to have big exhaust to run good times: definitely not.