It sounds to me as if there were two separate problems. The original problem was when you went to start it and nothing happen, is that correct?
If so, that could very well have been a loose connection.
As I understand it, you then replaced the battery and solenoid, thats when the new problem started, now it wont shut off? Is that correct?
If the above is the correct sequence of events, then I am inclined to agree with Shayne (he's correct 99.999% of the time), it could very well be a bad solenoid.
There is a brown wire coming off of one of the small posts on your solenoid, that wire supplies a full +12 volts to your ignition coil when the starter is engaged. When the engine starts and you let go of the key, an auxiliary contact in the start solenoid is supposed to open, and the brown wire should no longer supply the ignition coil with 12 volts. The ignition coil should now be getting a reduced voltage through the pink resistor wire fed from the ignition switch.
So, in theory, there are 2 sources of power to your ignition coil, the brown wire from the start solenoid and the pink wire from your ignition switch. So if either of those components were malfunctioning, it could cause the problem you describe.
Do you have a test light or a meter? The first thing I would check would be, with the engine off, remove the brown wire from the starter solenoid and see if you have +12 volts on that post. You shouldn't, there should only be 12 volts there when you are cranking the starter. If you don't have 12 volts, then have someone crank the starter and see if you have get voltage then. If you do and it goes away when they stop cranking the starter, then I would say the solenoid is good.
Im going to have to let Thunderfoot get you through this one though, I'm going on vacation for 5 days and wont be near a computer.
I know it's a little overwhelming, but don't worry, we'll figure this one out. Just take it a step at a time.
"I plan to start procastinating immediately"