If it was flying at Mach 3, I would get worried and do some smoothing. Since it isn't moving anywhere near that fast, it is whatever makes you feel good.
An air powered die grinder (less than $30 at Harbor Freight) and another $30 spent on a 1/2" diameter carbide rotary file will do the job. If you don't have a good 40 gallon tank or larger air compressor, or the $60 for the proper tools, forget about it. A Dremel tool or electric drill just isn't up to the job.