I think the term "de-evolution" is a misnomer, as it doesn't truly de-evolve a being-- it mutates them. I mean, the people in the Mushroom Kingdom probably didn't evolve from Goombas, but rather something that sort of looked like Goombas. The "de-evolution" machine is probably a mix of complex science that Koopa and his team of scientists concocted over the years that infuses the ability to mutate cells, simplifying them and creates a new organism that has far less complex thinking processes than a normal being. The Goombas are not just a simpler lifeform, but they're in a sense super-soldiers, with unusual strength and (usually) a sense of loyalty. However, due to their simplistic nature, they often tend to wander from their missions and in the end, wind up dancing in the streets and if the movie had gone on longer, may have even rebelled against Koopa and his army of cops. We may have even seen a "Goomba civil war" where they fought each other, the ones who sided with Koopa and the ones who rebelled against him...
As for the "evolve" setting on the machine, that's probably similar to the "learning machine" in the L. Ron Hubbard novel Battlefield Earth (Redstar will know what I'm talking about, here.) where the Psychlos leave Johnny Goodboy out in that mud pen for weeks with the learning machine which teaches him the Psychlo language and the workings of math, science, geography and other things that would take a normal being years to come to understand in a matter of moments. So, in a way, Koopa is sort of similar to Terl, in that he gives someone he wants working for him the ability to learn quickly, boosting their intelligence to a level far beyond that of a normal person and inadvertently leads to his own downfall. I'm sure that as Koopa was getting "de-evolved" in the finale, the thought of evolving Iggy and Spike went through his head and he probably thought something along the lines of "That seems kind of stupid in retrospect..."
CURRENT TROLL ALERT LEVEL STATUS
KOOPA THE DESPOT (Level 4)
High risk of Troll Attacks.