Easiest solution is to put a relay between key barrel and starter. This ensures the starter solenoid get a good, solid 12V signal. After this I'd be wrapping the manifold/headers and/or installing a heat shield between them.
Depending on how old the starter motor itself is, it could be the solenoid contacts that are worn. Pulling apart and filing them has helped me before (on a different car).