You want to calculate the torque about a point B due to a general force $\vec{F}$ passing through point A. Without loss of generality, decompose the force into three components ($F_1$, $F_2$, $F_3$) one of which is along the line AB.

Now treat the torque about B as the sum of the torques from the three components.
The line of action of each component is defined by the point A and the direction chosen. For $F_3$ the line of action passes through A as you can see above (by design).
This means that $F_3$ has zero torque about AB, and the total torque at B is solely due to the perpendicular components $F_1$ and $F_2$.
Mathematically this is done using a cross product which ignores any parallel components.
Also, note that you can reverse the above construct and decompose the position vector $\vec{r}_{A/B}$ along three directions, one of which is parallel to the force $\vec{F}$. In the end the parallel component will be ignored and only the perpendicular distances survive to contribute to torque. This is a bit more difficult to visualize but entirely equivalent to the first case.

As a general rule a force vector can slide along its line of action without changing the problem at hand. In the figure above, slide the force down to meet the (1,2) plane to make the out-of-plane distance zero $d_3=0$.
It is this freedom to slide of (some) vectors that necessitate the use of perpendicular distances and cross products.
I have also posted a more general answer to the nature of torque that I recommend you read also.