As a preface, recognize that this has never happened, so we can't know for sure. We can only interpret the relevant statutes, constitutional provisions, and vaguely analogous precedents, and consider the issue in the context of the structure of the U.S. court system.
Implications of a lack of quorum in appeals to SCOTUS
In the case in question, the judgment of the U.S. Court of Appeals, or of the state supreme court in question, would be affirmed without creating a binding precedent. The outcome would be the same as the outcome in the 99% of cases appealed to the U.S. Supreme Court where certiorari is denied.
Indeed, if at least six justices recused, there would not be a sufficient number of non-recusing justices to grant certiorari in the first place (which takes four justices to vote in favor of the grant). So, the seemly fraught and anomalous cases of conflicts of interest by the justices would be pretty much invisible to the general public.
In those cases, the parties would already have received an adjudication at the trial court level, and a direct appeal of that decision, and sometimes one discretionary appeal beyond that (either by a state supreme court in a state with an intermediate appellate court, or in the form of en banc review of a U.S. Court of Appeals panel), so no great harm is done to the individual parties. This would be the normal result anyway, and certiorari can be, and routinely is, denied in cases where the claim is believed by a majority of the justices to be meritorious, due to a limited supply of court resources to review certiorari petitions.
Indeed, it wouldn't even be necessary for all of the justices to recuse. All that would be necessary to force that result would be for the recusals to deny the U.S. Supreme Court a quorum. So four recusals would be enough.
The Supreme Court of the United States shall consist of a Chief
Justice of the United States and eight associate justices, any six of
whom shall constitute a quorum.
28 U.S.C. § 1.
One side effect of the quorum rule is that if the justices really want to play hardball, a four judge minority can recuse and prevent the U.S. Supreme Court from ruling on a case (in what would possibly be an unenforceable breach of the "duty to sit" arising under judicial ethics guidelines). I'm not aware of any such minority boycott cases ever actually occurring, however.
In the tiny sliver of cases, under 28 U.S.C. § 1253, which are appeals of cases decided by three judge panels in the U.S. District Court, mostly or entirely in Voting Rights Act cases, where there is a direct appeal of right to the U.S. Supreme Court, the case can be referred to the Court of Appeals if there is not a U.S. Supreme Court quorum, pursuant to 28 U.S.C. § 2109, as noted in another answer, which also confirms the rule that in discretionary appeals, if there is a lack of a quorum, the lower court ruling is affirmed, "if a majority of the qualified justices shall be of opinion that the case cannot be heard and determined at the next ensuing term."
Implications in SCOTUS original jurisdiction cases
28 U.S.C. § 2109 does not expressly address the question of what the U.S. Supreme Court should do if there is a lack of a quorum in a case in its original jurisdiction.
The only situation where the application of these rules would really be problematic would be in a case where the U.S. Supreme Court has original and exclusive jurisdiction.
In all cases affecting ambassadors, other public ministers and
consuls, and those in which a state shall be party, the Supreme Court
shall have original jurisdiction.
U.S. Constitution, Article III, § 2.
The U.S. Supreme Court's original jurisdiction is also explained and established at 28 U.S.C. § 1251, which states:
(a) The Supreme Court shall have original and exclusive jurisdiction
of all controversies between two or more States.
(b) The Supreme Court shall have original but not exclusive
jurisdiction of:
(1) All actions or proceedings to which ambassadors, other public
ministers, consuls, or vice consuls of foreign states are parties;
(2) All controversies between the United States and a State;
(3) All actions or proceedings by a State against the citizens of
another State or against aliens.
Thus, the U.S. Supreme Court has original jurisdiction over cases involve diplomats, but this is not exclusive of U.S. District Court jurisdiction over these cases. See 28 U.S.C. §§ 1351 and 1364.
Thus, the only cases in the exclusive original jurisdiction of the U.S. Supreme Court are cases between two or more U.S. states as parties (with or without the United States government as a party), in which it would be hard to see a fact pattern where an ethical obligation to recuse that would leave the court with less than six qualified justices would arise.
Usually, there are one to three cases filed in the U.S. Supreme Court's original jurisdiction each year (out of more than 5,000 cases filed in the court each year). Probably a majority of those cases involve water rights disputes between U.S. states arising under interstate compacts.
Also, notwithstanding the plain language of the relevant language of Article III of the U.S. Constitution and the relevant jurisdictional statutes, and over the objections of a couple of sitting U.S. Supreme Court justices, the U.S. Supreme Court has consistently treated original jurisdiction cases as if they were discretionary, rather than as cases that the justices are required to consider on the merits. So, the U.S. Supreme Court might just decline to rule on an exclusive, original jurisdiction case in that situation.
Would the justices recuse and would they have to do so?
If the case was one that inherently implicated the U.S. Supreme Court, e.g., a challenge to the constitutionality of the statute creating the U.S. Supreme Court, no justice would recuse based upon the rule of necessity (this rule could also be applied to cases in the original and exclusive jurisdiction of the U.S. Supreme Court). The rule of necessity is:
a rule permitting or requiring a judge or other official to adjudicate
a case despite bias or personal interest when disqualification would
result in the lack of any competent tribunal.
Furthermore, there are no circumstance where recusal is mandatory in an enforceable way at the U.S. Supreme Court level. There are ethical mandates that say that judge's should recuse in certain circumstances, but the U.S. Supreme Court has no binding and legally enforceable ethics code.