If you dont want to sand it smooth, I would take some 220 and sand down the whole thing with light pressure. This should help to remove any loose dirt or debris. In essence, it is more like if you just cleaned it good, but the sanding would ensure all the loose particles and dirt are removed.
RC