All quantum operators must be unitary. Unitary means the conjugate-transpose of the operator is its inverse. In your case:
$UU^{\dagger} = \begin{bmatrix}
1 & 0 & 0 & 0\\
0 & 1 & 1 & 0\\
0 & 0 & 0 & 0\\
0 & 0 & 0 & 0
\end{bmatrix}
\begin{bmatrix}
1 & 0 & 0 & 0\\
0 & 1 & 0 & 0\\
0 & 1 & 0 & 0\\
0 & 0 & 0 & 0
\end{bmatrix} =
\begin{bmatrix}
1 & 0 & 0 & 0\\
0 & 2 & 0 & 0\\
0 & 0 & 0 & 0\\
0 & 0 & 0 & 0
\end{bmatrix}$
So it is most certainly not unitary because $UU^{\dagger} \neq \mathbb{I}_4$ (same as your second attempt).
There's a long way to construct functions like this, and a short way. The long way is to write out all inputs and outputs:
$U|000\rangle = |000\rangle$
$U|001\rangle = |001\rangle$
$U|010\rangle = |011\rangle$
$U|011\rangle = |010\rangle$
$U|100\rangle = |101\rangle$
$U|101\rangle = |100\rangle$
$U|110\rangle = |110\rangle$
$U|111\rangle = |111\rangle$
You can then pretty easily construct the operator from this.
An easier way is to use projection operators & matrix addition to implement "if-then" semantics with matrices:
$U = |00\rangle\langle00| \otimes \mathbb{I}_2 + |01\rangle\langle01|\otimes X_2 + |10\rangle\langle10| \otimes X_2 + |11\rangle\langle11|\otimes \mathbb{I}_2$
The way to read this is "if the input is $|00\rangle$ or $|11\rangle$, do not flip the third bit. If the input is $|01\rangle$ or $|10\rangle$, flip the third bit. $|\phi\rangle\langle \phi|$ is called the outer product, and is defined for example as follows:
$|0\rangle\langle0| = \begin{bmatrix} 1 \\ 0 \end{bmatrix} \begin{bmatrix} 1 & 0 \end{bmatrix} = \begin{bmatrix} 1 & 0 \\ 0 & 0 \end{bmatrix}$
which is called a projection operator. Use projection operators to only apply an operation on specific states - here, $|0\rangle$.
A huge benefit of this projectors & addition approach is that you never have to actually write out the full matrix, which can become enormous as the number of qbits increase - 3-qbit 8x8 matrices already have 64 elements! This is your first step into using symbolic rather than matrix reasoning. For example, we can use the rules of linear algebra to calculate the action of our $U$ on some input:
$U|101\rangle = (|00\rangle\langle00| \otimes \mathbb{I}_2 + |01\rangle\langle01|\otimes X_2 + |10\rangle\langle10| \otimes X_2 + |11\rangle\langle11|\otimes \mathbb{I}_2)|101\rangle$
Now, matrix multiplication distributes over addition. This means we have:
$|00\rangle\langle00| \otimes \mathbb{I}_2 |101\rangle + |01\rangle\langle01|\otimes X_2 |101\rangle + |10\rangle\langle10| \otimes X_2 |101\rangle + |11\rangle\langle11|\otimes \mathbb{I}_2 |101\rangle$
Let's apply further transformation rules. Note $|101\rangle = |10\rangle \otimes |1\rangle$, and $(U\otimes V)(|x\rangle \otimes |y\rangle) = U|x\rangle \otimes V|y\rangle$, where in our cases (for example) $U = |00\rangle\langle00|$, $V = \mathbb{I}_2$, $x=|10\rangle$ and $y=|1\rangle$:
$|00\rangle\langle00|10\rangle \otimes \mathbb{I}_2 |1\rangle + |01\rangle\langle01|10\rangle \otimes X_2 |1\rangle + |10\rangle\langle10|10\rangle \otimes X_2 |1\rangle + |11\rangle\langle11|10\rangle \otimes \mathbb{I}_2 |1\rangle$
Now, note we have the following four terms:
$\langle00|10\rangle, \langle01|10\rangle, \langle10|10\rangle, \langle11|10\rangle$
These are called inner products, or dot products and here all of them are zero except for $\langle10|10\rangle$ - the dot product of $|10\rangle$ with itself:
$\langle10|10\rangle = \begin{bmatrix} 0 & 0 & 1 & 0\end{bmatrix}
\begin{bmatrix}0 \\ 0 \\ 1 \\ 0\end{bmatrix} = 1$
Since the other terms are all zero, they all cancel out:
$\require{cancel} \cancel{|00\rangle\cdot 0 \otimes \mathbb{I}_2 |1\rangle} + \cancel{|01\rangle\cdot 0 \otimes X_2 |1\rangle} + |10\rangle\cdot 1 \otimes X_2 |1\rangle + \cancel{|11\rangle\cdot 0 \otimes \mathbb{I}_2 |1\rangle}$
So we are left with:
$|10\rangle \otimes X_2 |1\rangle$
Where of course $X_2|1\rangle = |0\rangle$, so:
$|10\rangle \otimes |0\rangle = |100\rangle$
And we calculated $U|101\rangle = |100\rangle$ as expected, without once having to write out a huge inconvenient matrix!
