I'm not sure exactly what you did with the alligator clips but the red wire on the loom side of the '77 connector is connected directly to the battery via the 30amp fuse. The original '77 connector was rated at 45amps and the regulator is able to supply 32amps I think so to have the wires smoking would require a pretty hefty current draw (ie dead short and that may well have taken out the 30 amp fuse) - I'm guessing the wires with the alligator clips were a lot thinner and they smoked? You would have to think you had the positive shorted to ground.
The negative output of the regulator is bolted directly to the case of the regulator (see picture below - the yellow connector is the ground lug), so I would unplug the regulator and measure the resistance of each of the output wires to the case of the regulator to be REALLY certain that the positive is connected to positive and negative to negative. The worst that would have happened if the wires were swapped should have been the 32 amp fuse popping (assuming the engine wasn't running).