The ruling by the US Court of Appeals for the Armed Forces yesterday also made invalid an order from the judge, Colonel Gregory Gross, for Major Nidal Hasan to appear clean-shaven or for him to be forcibly shaved before his court-martial.
Hasan, charged with the November 2009 shooting at Fort Hood, had argued that he had a right to keep his beard as an expression of his Muslim faith.
The court said that in light of a variety of factors, a reasonable person "would harbour doubts about the military judge's impartiality."
Although it did not say that Gross was biased, the court ordered him removed due to an appearance of bias.
Gross had found Hasan in contempt of court at six pre-trial hearings because he had appeared with his beard, but the appeals court vacated those convictions.
However, the court said it was not issuing a ruling on whether Gross' order had violated Hasan's religious rights.
A new trial judge has not yet been named.
The argument over Hasan's beard has held up his court-martial, which was originally due to get under way in August.
Hasan, a US Army psychiatrist, is accused of opening fire at the Fort Hood base on November 5, 2009, in an attack that killed 12 soldiers and a civilian, and wounded 32 others.
His court-martial is due to take place in Texas and, if convicted, Hasan could face a death sentence.