So, first I replaced the starter on my 85, but that didn't help. What fixed it was replacing my starter power cable with a good quality 2 gauge cable - starts better than new now, hot or cold with 11:1 compression and the stock starter (not a mini).
As other have stated it sounds like a cable, ground or heat soak issue. So go through and check the grounds, make sure they are tight, the cables aren't frayed and there isn't any corrosion on the connections. If the problem persists I would replace the starter cable. It is probably the original and could stand to be replaced, unless it was recently done. The last problem would be a heat soaked starter. Basically once the starter gets hot the resistance in the windings increases and causes the starter to turn slow. I had a problem similar to your with an old Challenger and it turned out to be a heat soaked starter. Of course the combo of LTs and a stock syle starter was the cause but I was to cheap to buy one of the mini high torque starter so I ended up swapping starter about every 3 months.