Well, don't use oven cleaner, just in case you were considering it. I generally use lacquer thinner to clean bare metal. Aluminum is rather porous, so you need something that will get in and remove residues. You can buy 2 gals. of thinner and pour it into a 5 gal bucket and soak one half at a time. If the level doesn't quite come up half-way on the manifold, put something else in the bucket, too, to displace liquid and raise the level. I'd also use a self-etching primer.
Be aware that silver paint has some insulating properties and will hinder disipation of heat. Hot manifolds make less power, but it will be very minor. I believe that if you used a black paint, heat will disipate better, but that may not go with your paint scheme.
You might also try an aluminum polish, like Mother's, and see if it will remove the stains, then leave it natural. Mothers acts a little like silver polish and doesn't require a lot of rubbing.
