That's great news Guy, so glad you were able to fix it yourself. I am having a bit of a tough time figuring out exactly what you did, but it sounds like the ground wire to the relay/solenoid for the contactor is what you replaced. That doesn't surprise me at all that that fixed it. It is almost always something small like that which causes problems. A little corrosion inside the ring terminal or whatever was there will cause enough resistance to keep the solenoid from operating. Great job man! If it was only a ground wire you didn't bypass the fuse box.
Oh, and the reason it works with the key is that it probably is a three wire solenoid, in which the grounds are shared between the actuator circuit and the switched power circuit. So, the one ground being bad left both the solenoid's switching function and the power to the contactor circuit without a ground. Is this car still grounded through the chassis like a regular car?