It is not a curl. This can be seen by expressing the curl in vector components.
$$\nabla \times \mathbf M=\begin{pmatrix}
\partial_yM_z-\partial_z M_y\\
\partial_zM_x-\partial_x M_z\\
\partial_xM_y-\partial_y M_x
\end{pmatrix}$$
Here $\partial_x$ denotes the partial derivative with respect to $x$. The quantity $\partial_x\mathbf M$ is a vector just like $\mathbf M$. It has components
$$\partial_x \mathbf M=\begin{pmatrix}
\partial_xM_x\\
\partial_xM_y\\
\partial_xM_z
\end{pmatrix}$$
Calculating the quantity $\partial_x\mathbf M\times\partial_y\mathbf M$ is then just a matter of applying the cross product.
$$\partial_x\mathbf M\times\partial_y\mathbf M=\begin{pmatrix}
\partial_xM_y\partial_yM_z-\partial_xM_z\partial_yM_y\\
\partial_xM_z\partial_yM_x-\partial_xM_x\partial_yM_z\\
\partial_xM_x\partial_yM_y-\partial_xM_y\partial_yM_x
\end{pmatrix}$$
This is a daunting expression and you probably won't get a lot of intuition from looking at the components. What you can say about it is that $\mathbf A\cdot(\mathbf B\times \mathbf C)$ forms the vector triple product. This gives the volume spanned by (the parallelepiped of) $\mathbf A,\mathbf B$ and $\mathbf C$. So the quantity you're integrating is the volume spanned by $\mathbf M,\partial_x \mathbf M$ and $\partial_y \mathbf M$.
To calculate the integral in your last equation is just a matter of plugging everything in in my last expression for $\partial_x\mathbf M\times\partial_y\mathbf M$. This is tedious but should be doable.
EDIT I'll add some more info to make the calculation less tedious. The partial derivatives can be expanded using the chain rule
$\partial_x=\frac{\partial r}{\partial x}\partial_r+\frac{\partial \alpha}{\partial x}\partial_\alpha$. These can be calculated to be
$$\partial_x=\cos\alpha\partial_r-\frac{\sin\alpha}r\partial_\alpha\\
\partial_y=\sin\alpha\partial_r+\frac{\cos\alpha}r\partial_\alpha$$
Next note that
$\partial_r\mathbf M=\frac{d\theta}{dr}\partial_\theta\mathbf M$ and
$\partial_\alpha\mathbf M=\frac{d\phi}{d\alpha}\partial_\phi\mathbf M$. If we name these partial derivative vectors
$\mathbf e_\theta=\partial_\theta\mathbf M$ and
$\mathbf e_\phi=\partial_\phi\mathbf M$ then the cross product becomes
$$\partial_x\mathbf M\times\partial_y\mathbf M=\left(\cos\alpha\frac{d\theta}{dr}\mathbf e_\theta-\frac{\sin\alpha}r\frac{d\phi}{d\alpha}\mathbf e_\phi\right)\times\left(\sin\alpha\frac{d\theta}{dr}\mathbf e_\theta + \frac{\cos\alpha}r\frac{d\phi}{d\alpha}\mathbf e_\phi\right)$$
Finally you can calculate that
$\mathbf e_\theta\times \mathbf e_\phi=\sin\theta \,\mathbf M$ and you should be able to do this calculation without explicitly calculating all the components.
And yes you should add the factor $r$ when you switch to polar coordinates like you mentioned in your comment.