I believe the reason is that if you remove the cable, the port is still active, so to speak. It is not like you're issuing a "shutdown" on the port. The way to clear the mismatch usually is to correct it, so I guess when you change the VLAN, because the port is not connected to anything, it will accept the change and clear the error. When you change the VLAN back to original VLAN, again, there's nothing to mismatch with on the other side until you connect the cable again and the error appears again.
This is what I think is going on, let me know what you think?