The "actual" reason that the U.S. Supreme Court ruled that part of the Judiciary Act was unconstitutional, in Marbury v. Madison, 5 U.S. 137 (1803), is, by definition, the reason stated in the linked opinion of the U.S. Supreme Court.
Marybury filed his case directly in the U.S. Supreme Court requesting that Secretary of State Madison recognize his appointment as a Justice of the Peace in the District of Columbia.
The U.S. Supreme Court broke the case up into three issues:
Has the applicant a right to the commission he demands?
If he has a right, and that right has been violated, do the laws of his country afford him a remedy?
If they do afford him a remedy, is it a mandamus issuing from this court?
The Court concluded that Marbury had a right to the commission he demanded, and that this was a judicially enforceable right.
The resolution of the issue that the case is remembered for is whether a writ of mandamus from the U.S. Supreme Court was the proper remedy when Marbury was denied his commission.
The U.S. Supreme Court noted that the Judiciary Act gave it the power to issue writs without limitation. But it also noted that the U.S. Constitution did not give the U.S. Supreme Court original jurisdiction over the subject-matter of the case in question.
Article III, Section 2 of the U.S. Constitution stated in the pertinent part:
In all Cases affecting Ambassadors, other public Ministers and
Consuls, and those in which a State shall be Party, the supreme Court
shall have original Jurisdiction. In all the other Cases before
mentioned, the supreme Court shall have appellate Jurisdiction, both
as to Law and Fact, with such Exceptions, and under such Regulations
as the Congress shall make.
The U.S. Supreme Court summarized the pertinent part of the Judiciary Act as follows:
The act to establish the judicial courts of the United States
authorizes the Supreme Court
"to issue writs of mandamus, in cases warranted by the principles and
usages of law, to any courts appointed, or persons holding office,
under the authority of the United States."
So, it held that the Judiciary Act language allowing it to issue writs, even in a case that was not an appeal from another court, and was not within constitutionally prescribed original jurisdiction of the U.S. Supreme Court, was unconstitutional.
In other words, it told Marbury that he should have sought relief in the U.S. District Court for the District of Columbia, and then appeal the case to the U.S. Supreme Court if he was unsatisfied, rather than filing his case directly with the U.S. Supreme Court.
This may seem like a petty procedural question, unlike landmark cases resolving questions of the individual rights of large numbers of people under the U.S. Constitution. But it had great importance to the U.S. Supreme Court as an institution.
If the U.S. Supreme Court had ruled the other way, every single federal petition for a writ of mandamus (or any other common law writ) could have been brought directly in the U.S. Supreme Court. This would have deluged the U.S. Supreme Court with far more cases than it would otherwise have to deal with, if those cases were resolved in the first instance by federal trial courts.
To provide some context, as of 2023, more than 10% of cases filed in U.S. District Courts (in excess of 33,000 cases a year) are petitions for common law writs. Also, keep in mind that until 1891, the U.S. Supreme Court was the sole appellate court in the federal system, because no intermediate appellate courts had been established yet, so dealing with appeals alone already provided plenty of work to keep the U.S. Supreme Court busy. Then, as now, cases in the U.S. Supreme Court's constitutionally prescribed original jurisdiction only account for 1-2% of the U.S. Supreme Court's case load.
Given the seeming clarity of the portion of the Judiciary Act known as the All Writs Act at the time, though, from a statutory perspective, Marbury was correct. He was statutorily entitled to file his case seeking a writ of mandamus against the U.S. Secretary of State in the U.S. Supreme Court. So, holding that the statute was unconstitutional was the clearest and most legally supportable way for the U.S. Supreme Court to align the law regarding its own jurisdiction with the U.S. Constitution.
Notably, this was not just meddling with a decision of Congress based upon a policy disagreement. It was fundamentally a separation of powers issue.
Could Congress order the U.S. Supreme Court to deal with cases in its original jurisdiction in a manner directly contrary to the allocation of cases between the U.S. Supreme Courts and lower courts set forth in the Article III of the U.S. Constitution?
The U.S. Supreme Court held that Congress could not order it to take on this additional case load.
Also, the principle of U.S. Constitutional supremacy over federal statutes, while a landmark determination to establish in case law and a stark difference from the structure of the British government, was really not that big of a deal. It was a bedrock assumption of the U.S. Constitution that was widely shared by the Founders and the legal community of the time, that just happened to finally come up in this case. It was not a "shocking" in the eyes of the legal community at the time.
In the wake of this decision, the All Writs Act, now codified at 28 U.S.C. § 1651, was rewritten. It now says:
(a)The Supreme Court and all courts established by Act of Congress may
issue all writs necessary or appropriate in aid of their respective
jurisdictions and agreeable to the usages and principles of law.
(b)An alternative writ or rule nisi may be issued by a justice or
judge of a court which has jurisdiction.
