It still draws fenderwell air with the silencer removed. Where my airbox is, is about 2 inches from the fenderwell. The air people are concerned about drawing in to the filter (commonly referred to as "hot air") is the air coming off the radiator and fan, and the air down by and radiating from the exhaust manifolds. Forward movement of the vehicle draws most of that air back down the bottom of the engine compartment, and the intake still is sucking whatever air comes from the passenger side near the fenderwell (which is the air pouring in from the fenderwell itself.)
Either way, removing the silencer isn't worth enough to feel SOTP wise, but probably would be worth something if you had the power adders to require that much air volume to be sucked in at a time.