BlueThunder- Yes, potentially, there is downside. Working properly, the egr is all upside. It descreases emissions, increases fuel mileage, and is non-functional at w.o.t. so there's no performance downside. It works by allowing small amounts of exhaust gas (egr - exhaust gas recirculation) to be injected into the intake air under part throttle, low load conditions. The exhaust is inert, and displaces air/oxygen that would otherwise be present. That bit of inert gas in the mixture actually reduces (cools) combustion temperatures allowing the computer to add more timing advance, and lean the fuel mixture out without risking detonation. If you remove the egr gas from the equation, the computer doesn't know that, and it continues to add timing and pull fuel under low load. Sometimes this results in detonation that wasn't present before. Since detonation is also impacted by ambient temps, it's cold enough now in most of the country, that you might not have a problem until next spring/summer when temps rise. Unfortunately there's no way to predict what's gonna happen with your car. Variables include mods (compression ratio), engine condition and mileage, geographic location (temps, elevation, etc.). Many others will likely post and tell you they had a problem or didn't have a problem. Unfortunately, that's not really helpful to you. The only way to know what's gonna happen to yours is to remove it and see. But since it won't help wot performance, and it will likely hurt emissions and fuel mileage, why remove it?
Having said all that - I removed all of mine because I didn't like the clutter, and I used a chip to turn off the egr function in the computer - that's the proper way to solve any potential detonation issues.
Good luck with it.