(CW to expand on other answer.)
We will try to apply this approach to two labeled graphs $G$ and $H$ on three vertices as below. We wish to quantumly test whether these graphs are isomorphic, by preparing two states $|G\rangle$ and $|H\rangle$ corresponding to the uniform superposition of all permutations of $G$, respectively $H$. We'll see what happens when we do a SWAP test therebetween.

Let's go in order; we'll begin with graph $G$, and then later work on graph $H$. At the end we will see what happens when we SWAP them.
- Let's prepare a first register $|g\rangle$, where $g$ is the adjacency matrix of $G$. Let's let $\mathbf{G}$ be the operator that prepares $|g\rangle$ from the all-zero's ket, e.g., $\mathbf{G}|00\cdots0\rangle=|g\rangle$.
$$g=\begin{pmatrix}
0 & 1 & 0 \\
1 & 0 & 0 \\
0 & 0 & 0 \\
\end{pmatrix}.$$
- In another register, let's also apply the operator $\mathbf{A}$ to prepare the uniform superposition over all $n!$ elements of $S_n$ in another register, starting with a register initialized to the all-zero's ket $|00\cdots0\rangle$; we will use cycle notation. It's (mostly) mechanical to do this with something like Grover-Rudolph and/or Stirling's Approximation. But with only $n=3$ vertices, there are only six elements to $S_3$:
$$\mathbf{A}|00\cdots0\rangle=\frac{1}{\sqrt 6}\big(|(1)\rangle+|(12)\rangle+|(13)\rangle+|(23)\rangle+|(123)\rangle+|(132)\rangle\big).$$
Now, let's apply, in superposition, operator $\mathbf B$ on the first and second registers. This does the appropriate permutation to $g$ over all such permutations stored in the second register. You suggest applying a conditional SWAP between columns $i$ and $j$ - I would do something like that too, but I think you'd also need to SWAP the rows as well. But it doesn't matter too much how this operator $\mathbf B$ is implemented; it's reversible and scales gracefully with $n$, and its implementation is also mechanical.
In particular, after applying $\mathbf B$ and $\mathbf{A}$ to the registers, we will have $\mathbf{B}\mathbf{A}\mathbf{G}|00\cdots0\rangle|00\cdots0\rangle=|G\rangle$, where, assuming I did my permutations correctly:
$$|G\rangle=\frac{1}{\sqrt 6}\big(
|\begin{pmatrix}
0 & 1 & 0 \\
1 & 0 & 0 \\
0 & 0 & 0 \\
\end{pmatrix}\rangle|(1)\rangle+
|\begin{pmatrix}
0 & 1 & 0 \\
1 & 0 & 0 \\
0 & 0 & 0 \\
\end{pmatrix}\rangle|(12)\rangle+
|\begin{pmatrix}
0 & 0 & 0 \\
0 & 0 & 1 \\
0 & 1 & 0 \\
\end{pmatrix}\rangle|(13)\rangle+
|\begin{pmatrix}
0 & 0 & 1 \\
0 & 0 & 0 \\
1 & 0 & 0 \\
\end{pmatrix}\rangle|(23)\rangle+
|\begin{pmatrix}
0 & 0 & 0 \\
0 & 0 & 1 \\
0 & 1 & 0 \\
\end{pmatrix}\rangle|(123)\rangle+
|\begin{pmatrix}
0 & 0 & 1 \\
0 & 0 & 0 \\
1 & 0 & 0 \\
\end{pmatrix}\rangle|(132)\rangle\big).
$$
Let's now do exactly steps 1-4 above, but for $h$ instead of $g$. That is, we will prepare $|H\rangle$ by acting on $\mathbf{B}\mathbf{A}\mathbf{H}|00\cdots0\rangle|00\cdots0\rangle$, which, if I permuted correctly, gives:
$$|H\rangle=\frac{1}{\sqrt 6}\big(
|\begin{pmatrix}
0 & 0 & 1 \\
0 & 0 & 0 \\
1 & 0 & 0 \\
\end{pmatrix}\rangle|(1)\rangle+
|\begin{pmatrix}
0 & 0 & 0 \\
0 & 0 & 1 \\
0 & 1 & 0 \\
\end{pmatrix}\rangle|(12)\rangle+
|\begin{pmatrix}
0 & 0 & 1 \\
0 & 0 & 0 \\
1 & 0 & 0 \\
\end{pmatrix}\rangle|(13)\rangle+
|\begin{pmatrix}
0 & 1 & 0 \\
1 & 0 & 0 \\
0 & 0 & 0 \\
\end{pmatrix}\rangle|(23)\rangle+
|\begin{pmatrix}
0 & 1 & 0 \\
1 & 0 & 0 \\
0 & 0 & 0 \\
\end{pmatrix}\rangle|(123)\rangle+
|\begin{pmatrix}
0 & 0 & 0 \\
0 & 0 & 1 \\
0 & 1 & 0 \\
\end{pmatrix}\rangle|(132)\rangle\big).
$$
Lining up the matrices or alternatively lining up the permutations and taking the dot product lets us visually see that $\langle H|G\rangle=0$. Thus, the SWAP test between $|G\rangle$ and $|H\rangle$ won't tell you anything!
In the comments it was also suggested to apply $\mathbf{H}^\dagger\mathbf{A}^\dagger\mathbf{B}^\dagger\mathbf{B}\mathbf{A}\mathbf{G}|0\rangle|00\cdots0\rangle$. But this is equal to $\mathbf{H}^\dagger\mathbf{G}|00\cdots0\rangle$, which is also going to be junk.
Really what you need is another operator, call it, say, $\mathbf{T}$ for tigerjack, that uncomputes the second register back down to the all-zeroes ket. That is, ideally we'd like $\mathbf{T}\mathbf{B}\mathbf{A}\mathbf{G}|00\cdots0\rangle|00\cdots0\rangle$ to give:
$$|G'\rangle=\frac{1}{\sqrt 6}\big(
|\begin{pmatrix}
0 & 1 & 0 \\
1 & 0 & 0 \\
0 & 0 & 0 \\
\end{pmatrix}\rangle|00\cdots0\rangle+
|\begin{pmatrix}
0 & 1 & 0 \\
1 & 0 & 0 \\
0 & 0 & 0 \\
\end{pmatrix}\rangle|00\cdots0\rangle+
|\begin{pmatrix}
0 & 0 & 0 \\
0 & 0 & 1 \\
0 & 1 & 0 \\
\end{pmatrix}\rangle|00\cdots0\rangle+
|\begin{pmatrix}
0 & 0 & 1 \\
0 & 0 & 0 \\
1 & 0 & 0 \\
\end{pmatrix}\rangle|00\cdots0\rangle+
|\begin{pmatrix}
0 & 0 & 0 \\
0 & 0 & 1 \\
0 & 1 & 0 \\
\end{pmatrix}\rangle|00\cdots0\rangle+
|\begin{pmatrix}
0 & 0 & 1 \\
0 & 0 & 0 \\
1 & 0 & 0 \\
\end{pmatrix}\rangle|00\cdots0\rangle\big).
$$
- If we had such an operator, we could apply $\mathbf{T}\mathbf{B}\mathbf{A}\mathbf{H}|00\cdots0\rangle|00\cdots0\rangle$ to get:
$$|H'\rangle=\frac{1}{\sqrt 6}\big(
|\begin{pmatrix}
0 & 0 & 1 \\
0 & 0 & 0 \\
1 & 0 & 0 \\
\end{pmatrix}\rangle|00\cdots0\rangle+
|\begin{pmatrix}
0 & 0 & 0 \\
0 & 0 & 1 \\
0 & 1 & 0 \\
\end{pmatrix}\rangle|00\cdots0\rangle+
|\begin{pmatrix}
0 & 0 & 1 \\
0 & 0 & 0 \\
1 & 0 & 0 \\
\end{pmatrix}\rangle|00\cdots0\rangle+
|\begin{pmatrix}
0 & 1 & 0 \\
1 & 0 & 0 \\
0 & 0 & 0 \\
\end{pmatrix}\rangle|00\cdots0\rangle+
|\begin{pmatrix}
0 & 1 & 0 \\
1 & 0 & 0 \\
0 & 0 & 0 \\
\end{pmatrix}\rangle|00\cdots0\rangle+
|\begin{pmatrix}
0 & 0 & 0 \\
0 & 0 & 1 \\
0 & 1 & 0 \\
\end{pmatrix}\rangle|00\cdots0\rangle\big).
$$
- If I did it right (there's a good chance I made a mistake) we see that $\langle H'|G'\rangle=1$! But no one knows how to make any such $\mathbf{T}$ for general $n$ that runs in polynomial time.
It's been a challenge for about 30 years to find such an operator $\mathbf{T}$ that resets the permutation register to $|00\cdots0\rangle$. Then again, some people think Babai's quasi-polynomial classical algorithm makes this algorithmic exploration less attractive. Personally, for me I like to spend more time thinking about ways to instantiate the 31-year-old Simon's oracle or the 28-year-old Welded Trees oracle, etc.