The friction modifier serves just as it's name implies. It modifies the friction through reduction.
Have you ever smelled that stuff? If I were friction, I wouldn't want to be around that either.
Excessive backlash (excessive = greater than .013") leads to banging and slamming in the rear housing. Backlash too tight (tight = less than .008") leads to busted teeth off the ring gear.
A toed in or out gear pattern produces a whining noise.
Now, it can appear that your gears were set-up "just right," and they may have been, but the actual casting in the pinion or ring gear could be flawed, which is producing the sound.
With that said, if my lawnmower had a ring and pinion gear, I wouldn't put a Precision gear set in it.
I'd be willing to bet that if these people are as honest and good as you say they are, your problem is the actual gear set. FMS is the only way to go.
Joe