The solenoid should only engage for a few seconds while you are cranking the engine, so should not overheat (unless it is a 6V solenoid in a 12V car). Remove the spark plugs and try turning the engine over by turning the nut on the alternator clockwise. The engine should turn easily just using a < 1' long wrench. If it does, try cranking the engine with the starter. It should spin pretty easily.
You can check the starter motor by powering the starter from the bolt at the bottom of the solenoid instead of the bolt at the top of the solenoid. Normally the power is at the top bolt and the bottom bolt gets its power from the contacts in the solenoid. This bypasses the solenoid altogether, which will probably not engage the bendix gear. The starter should spin freely and fast. If it does, there may be a bad solenoid and you could open up the solenoid and look for a contact problem.
If the solenoid windings are shorted together or shorted to ground internally, that would make the solenoid get hot real fast. Since the winding resistance is so low per turn, it is nearly impossible to measure a resistance fault. Testing the current draw of the solenoid windings (current to the small starting terminal) might reveal a short as an unusually high current, like 20A. I cannot remember the value range and would have to grab a starter to make a test, if needed.
If the engine is hard to turn over with the plugs removed, and possibly also hard to turn over with the starter with the plugs removed, you have an engine problem. I would suspect the shims at the flywheel end of the crank are too tight (common rebuilding problem), or the cylinders are rusty.