Could they have pulled it off with special effects at the time? Certainly. The makeup effects for the apes in the opening scenes of
2001: A Space Odyssey and for the evolved apes throughout the entirety of
The Planet of the Apes were entirely convincing for their time, and both films were produced over twenty years before SMB. Don't forget that SMB also had its share of brilliant makeup effects, including Scapelli's de-evolution.
Would Donkey Kong have worked for a villain in SMB2? I'm not quite sure he would have. The entire first movie ran off the theme of dinosaurs, reptiles, and cold-bloodedness. A mammal villain just seems... off. It would completely skew away from what the first movie had established. I'm just not quite sure it would feel right.
Having said that, I have speculated that re-evolving Scapelli could have had technicalities and resulted in an "ape man" with a strong desire for revenge against the Mario Bros.
