How do I know if I have TE, TM, or TEM rectangular conductive waveguide? For instance, I am doing a lab where we want maximum magnetic field in the waveguide, does that mean we want the TE because $\vec{E}=0$ near boundary? Is it the material of waveguide that tells you, frequency of signal, or something else? Thank you!
BTW I am using microwave radiation.
Firstly, if your waveguide is a hollow conductor, it cannot support TEM modes. There must be at least two separate (electrically insulated from one another) conductors in the waveguide's cross section for TEM modes to propagate. The reason is that the transverse field dependence of a TEM mode is the same as that of a static field, as I explain in detail in this answer here. That is, the fields are of the form $\vec{E}(x,y) \,E_z(z \pm c\,t)$ and $\vec{H}(x,y) \,H_z(z \pm c\,t)$, where $\vec{E}(x,y) = -\nabla \phi_E(x,y)$ and $\vec{H}(x,y) = -\nabla \phi_H(x,y)$. So, at a given cross-section, a conductor must be an equipotential surface. If there were only one such conductor, in the farfield it would look like charged thread, with field lines directed radially, and so the potential in the farfield would vary like $\log r$, which is unphysical because it is divergent. There must be two conductors for field limes to separately begin and end on. Likewise, inside a hollow conductor, there can be no static field, therefore no TEM field. So TEM waveguides are things like co-axial cables (outer and centre conductor), twisted pairs and microwave strip lines with dielectric sandwiched between two conductors.
To tell whether the waveguide will be TE or TM (or indeed hybrid) you need to look at the details of the cross section and the frequency it is working at. The whole answer is that you need to look at the full boundary value problems for Maxwell's equations for the waveguide's cross section. If it is a rectangular waveguide, the propagation constant as a function of frequency is the same expression for both TE and TM modes: the one difference is that TE modes have a lower cutoff frequency so if working above TE cutoff but below TM cutoff the field will be TE. See this document:
The particular section you need (13.4) is at: http://web.mit.edu/6.013_book/www/chapter13/13.4.html
