Who should Recuse?

In the Proposition 8 case working its way through the courts the question arises : should the former federal district court judge who first heard the case and found Proposition 8 unconstitutional have recused himself because he is gay?

According to the SF Gate story: "Proposition 8 supporters argued that Walker should have been disqualified from presiding over the case because his 10-year same-sex relationship gave him an interest in the outcome of the trial."

But what about a heterosexual judge married to someone of the opposite sex? Why would that judge not have an interest in the outcome of the trial? If we take the core argument by the Prop 8 supporters at face value, that the extension of marriage rights to gays constitutes some sort of harm to the institution as practiced by straight people, how then does a straight married judge not have an interest in the outcome of the case?

Any ideas?