It happened. Legal History Blog plucks out a gem from John Frank’s article on judicial disqualification.
With a little ingenuity, a situation of “necessity” can occasionally be avoided. Thus in a case in which all the members of the Teas Supreme Court were Woodmen of the World and hence the entire court vas disqualified in a case involving that group, the Governor appointed a special court of three women. For the resultant three opinions see Johnson v. Darr, 144 Tex. 516, 272 S. W. 1098 (1925).