Diversion into politics
It is clear to me that Bush's only responsible choice would be to fire Rove. Bush isn't stupid-- he is a brilliant propagandist and someone who has been remarkably successful at pushing through his agenda items (both foreign and domestic). If Bush has a failing here, it is in his aquiescence to scandal within his administration. It would have certainly been better politically for him to fire Rumsfeld after Abu Ghraib and try to put that matter behind the administration.
In international law (IANAL) there is a concept called "Command Responsibility." Under this concept, a leader is responsible for the conduct of those who report to him. These responsiblities include the requirement that the leader appropriately discipline subordinates who act in violation of such laws (this has been a large part of the case against Milosevic). The idea is that the failure to provide such discipline when the leader knew or should have known about the offense actually means that the leader was aquiescent to the offence (sort of the "Don't violate the law, now" wink, wink, nudge, nudge approach).
Bush owes it to his administration, his party, and ultimately his country to appropriately discipline those who are responsible for wrongdoing within his administration. He has publically stated that whoever was involved in the Plame leak would be fired, and backing off from that position now would be ultimately damaging to his administration and his party. It would also help the Democrats cement their opposition to the Bush agenda.
In politics, nothing works better than an enemy to mobilize forces. I wonder how well right-wing politicians would like it if Roe v. Wade was overturned as this would significantly dampen their propaganda capabilities. I say right-wing because there are many Republicans who support abortion rights (along with the majority of Americans). Similarly, I wonder if the Democrats (this is largely a partisan issue) would really like it if the Bush administration did fire Rove, as this would reduce their propaganda capabilities. Bush has nothing to lose from firing Rove. His party has nothing to lose. And Bush has everything to gain by stating that he will not tolerate such behavior even from those in the most privileged positions of his administration.