I'll provide a tentative solution based on a result derived by M. K. Transtrum and J.-F. S. Van Huele, J. Math. Phys. 46, 063510 (2005). They derived a general expression for the commutator of functions $f(A,B)$ and $g(A,B)$ of noncommuting operators $A$ and $B$:
\begin{equation}
\left[f(A,B),g(A,B)\right] = \sum_{k=1}^\infty \frac{(-c)^k}{k!} \left( \frac{\partial^k g}{\partial A^k} \frac{\partial^k f}{\partial B^k} - \frac{\partial^k f}{\partial A^k} \frac{\partial^k g}{\partial B^k}\right),
\quad \text{where} \quad
c=[A,B].
\end{equation}
Tentative solution
I consider the particular case $f=f(A)$ and $g=g(B)$:
\begin{equation}
\left[f(A),g(B)\right] = - \sum_{k=1}^\infty \frac{(-c)^k}{k!} \frac{\partial^k f}{\partial A^k} \frac{\partial^k g}{\partial B^k}
= \left[ - \sum_{k=1}^\infty \frac{(-c)^k}{k!} \frac{\partial^k }{\partial A^k} \frac{\partial^k }{\partial B^k} \right] f(A)g(B),
\end{equation}
where I believe the last step is not problematic as long we understand its meaning: derivative $\partial_A\equiv\frac{\partial}{\partial_A}$ acts on $f(A)$ and derivative $\partial_B\equiv\frac{\partial}{\partial_B}$ acts on $g(B)$. Finally we simplify the result to
\begin{equation}
[f(A),g(B)] = \left( 1-e^{-c\partial_A \partial_B} \right)f(A)g(B) \quad \text{or} \quad g(B)f(A)=e^{-c\partial_A \partial_B}f(A)g(B).
\end{equation}
Heading to $[|\mathbf{\hat{x}}|,|\mathbf{\hat{p}}|]$, I'll now ommit the "hats" for simplicity and use the notation $\mathbf{x}=(x,y,z)$ and $\mathbf{p}=(p_x,p_y,p_z)$ for the position and momentum operators respectively. My approach is writing |\mathbf{x}| and |\mathbf{p}| as power series:
\begin{equation}
|\mathbf{x}| = \sum_{abc}A_{abc} x^a y^b z^c \quad \text{and} \quad |\mathbf{p}| = \sum_{uvw}B_{uvw} p_x^u p_y^v p_z^w.
\end{equation}
Then
\begin{equation}
[|\mathbf{x}|,|\mathbf{p}|] = \sum_{abc}\sum_{uvw} A_{abc}B_{uvw} (x^a y^b z^c p_x^u p_y^v p_z^w - p_x^u p_y^v p_z^w x^a y^b z^c).
\end{equation}
The last term can be recast as
\begin{equation}
p_x^u p_y^v p_z^w x^a y^b z^c = (p_x^u x^a) (p_y^v y^b) (p_z^w z^c) = (e^{-i\hbar\partial_x \partial_{p_x}} x^a p_x^u) (e^{-i\hbar\partial_y \partial_{p_y}} y^b p_y^v) (e^{-i\hbar\partial_z \partial_{p_z}} z^c p_z^w) = e^{-i\hbar(\partial_x \partial_{p_x} + \partial_y \partial_{p_y} + \partial_z \partial_{p_z})} x^a p_x^u y^b p_y^v z^c p_z^w = e^{-i\hbar\partial_\mathbf{x}\cdot\partial_\mathbf{p}} x^a p_x^u y^b p_y^v z^c p_z^w,
\end{equation}
where
\begin{equation}
\partial_\mathbf{x}\cdot\partial_\mathbf{p} \equiv \sum_i \frac{\partial}{\partial x_i} \frac{\partial}{\partial p_i}.
\end{equation}
Finally,
\begin{equation}
[|\mathbf{x}|,|\mathbf{p}|] = \sum_{abc}\sum_{uvw} A_{abc}B_{uvw} (1-e^{-i\hbar\partial_\mathbf{x}\cdot\partial_\mathbf{p}})x^a y^b z^c p_x^u p_y^v p_z^w = (1-e^{-i\hbar\partial_\mathbf{x}\cdot\partial_\mathbf{p}})|\mathbf{x}| |\mathbf{p}|
\end{equation}
or, explicitly,
\begin{equation}
[|\mathbf{x}|,|\mathbf{p}|] = - \sum_{n=1}^\infty \frac{(-i\hbar)^n}{n!}(\partial_\mathbf{x} \cdot \partial_\mathbf{p})^n |\mathbf{x}| |\mathbf{p}|.
\end{equation}
A few remarks:
Prospects:
Maybe a nice, closed form for the above result could be achieved writing $\partial_\mathbf{x} \cdot \partial_\mathbf{p}$ in spherical polar coordinates?
This derivation seems to work without modifications for any $[|\mathbf{x}|^n,|\mathbf{p}|^m]$. If the last remark is met with success, we can check if the proposed answer recovers some commutators that can be computed easily -- e.g., one with $n=m=2$.
I'm investigating these prospects and I will modify this answer accordingly.
