Is this right?
No. I think you've arrived at this because you're not considering the gravitational degrees of freedom. Sheldon Goldstein puts it this way:
[T]he attractive nature of the gravitational interaction is such that
gravitating matter tends to clump, clumped states having larger
entropy. . . . For an ordinary gas, increasing entropy tends to make
the distribution more uniform. For a system of gravitating bodies the
reverse is true. High entropy is achieved by gravitational clumping —
and the highest of all, by collapse to a black hole.
So, gravitationally speaking, the young universe had very low entropy because the distribution of mass-energy was nearly uniform rather than clumped.
Now, the question of why this was the case is much more difficult to answer.