Problem: Given Newton's second law
$$\begin{align} m\ddot{q}^j~=~&-\beta(t)\dot{q}^j-\frac{\partial V(q,t)}{\partial q^j}, \cr j~\in~&\{1,\ldots, n\}, \end{align}\tag{0.1} $$
for a non-relativistic point particle in $n$ dimensions, subjected to a friction force, and also subjected to various forces that have a total potential $V(q,t)$, which may depend explicitly on time.
I) Conventional approach: There is a non-variational formulation of Lagrange equations
$$\begin{align} \frac{d}{dt} \left(\frac{\partial L}{\partial \dot{q}^j}\right)-\frac{\partial L}{\partial q^j}~=~&Q_j, \cr j~\in~&\{1,\ldots, n\},\end{align}\tag{0.2} $$
where $Q_j$ are the generalized forces that do not have generalized potentials.
In our example (0.1), the Lagrangian in eq. (0.2) is $L=T-V$, with $T=\frac{1}{2}m\dot{q}^2$; and the force
$$ Q_j~=~-\beta(t)\dot{q}^j\tag{0.3} $$
is the friction force. It is shown in e.g. this Phys.SE post that the friction force (0.3) does not have a potential. As OP mentions, one may introduce the Rayleigh dissipative function, but this is not a genuine potential.
Conventionally, we additionally demand that the Lagrangian is of the form $L=T-U$, where $T=\frac{1}{2}m\dot{q}^2$ is related to the LHS of EOMs (0.1) (i.e. the kinematic side), while the potential $U$ is related to the RHS of EOMs (0.1) (i.e. the dynamical side).
With these additional requirements, the EOM (0.1) does not have a variational formulation of Lagrange equations, i.e. it is not of the form of Euler-Lagrange (EL) equations
$$\begin{align} \frac{d}{dt} \left(\frac{\partial L}{\partial \dot{q}^j}\right)-\frac{\partial L}{\partial q^j}
~=~&0,\cr j~\in~&\{1,\ldots, n\}.\end{align}\tag{0.4} $$
The Legendre transformation of eq. (0.2) to the Hamiltonian formulation is of the form
$$\begin{align} \frac{dq^j}{dt}
~=~&\frac{\partial H}{\partial p_j},\cr
\frac{dp_j}{dt} +\frac{\partial H}{\partial q^j}~=~&Q_j, \cr j~\in~&\{1,\ldots, n\},\end{align}\tag{0.5}$$
see this Phys.SE post for details.
II) Unconventional approaches:
Trick with an integrating factor: In our example (0.1) we can introduce an integrating factor$^1$
$$ e(t)~:=~\exp\left(\frac{1}{m}\int_{t_i}^t\!dt^{\prime}\beta(t^{\prime})\right). \tag{1.1}$$
A possible variational formulation (0.4) of Lagrange equations is then given by the Lagrangian
$$\begin{align}
L(q,\dot{q},t)~:=~&e(t)L_0(q,\dot{q},t), \cr
L_0(q,\dot{q},t)~:=~&\frac{m}{2}\dot{q}^2-V(q,t).\end{align}\tag{1.2}$$
The corresponding Hamiltonian is
$$ H(q,p,t)~:=~\frac{p^2}{2me(t)}+e(t)V(q,t).\tag{1.3}$$
One caveat is that the Hamiltonian (1.3) does not represent the traditional notion of total energy.
Another caveat is that this unconventional approach can not be generalized to the case where two coupled sectors of the theory require different integrating factors, e.g. where each coordinate $q^j$ has individual friction-over-mass-ratios $\frac{\beta_j}{m_j}$, $j\in\{1, \ldots, n\}$. For this unconventional approach to work, it is crucial that the integrating factor is an overall common multiplicative factor for the Lagrangian (1.2). This is an unnatural requirement from a physics perspective.
By the way, the action
$$ S(t_f)~=~\frac{1}{e(t_f)}\int_{t_i}^{t_f}\!dt~e(t)L_0(q,\dot{q},t),\tag{1.4} $$
is the solution to the 1st-order linear ODE
$$\begin{align}
\dot{S}~=~&\widetilde{L}(q,\dot{q},t,S)~:=~L_0(q,\dot{q},t) +\frac{\beta(t)}{m}S, \cr
S(t\!=\!t_i)~=~&0.
\end{align}\tag{1.5}$$
In eq. (1.5) we have introduced an action-dependent Lagrangian $\widetilde{L}$! This construction may be generalized in various ways, c.f. the Herglotz's variational principle.
Imposing EOMs via Lagrange multipliers $\lambda^j$: A variational principle for the EOMs (0.1) is
$$\begin{align}L ~=~& m\sum_{j=1}^n\dot{q}^j\dot{\lambda}^j\cr
&-\sum_{j=1}^n\left(\beta\dot{q}^j+\frac{\partial V(q,t)}{\partial q^j}\right)\lambda^j.\end{align}\tag{2.1}$$
(Here we have for convenience "integrated the kinetic term by parts" to avoid higher time derivatives.)
Classical Schwinger/Keldysh "in-in" formalism: The variables are doubled up. See e.g. eq. (20) in C.R. Galley, arXiv:1210.2745. Ignoring boundary terms$^2$ the Lagrangian reads
$$\begin{align}
\widetilde{L}(q_{\pm},\dot{q}_{\pm},t)
~=~&\left. L(q_1,\dot{q}_1,t)\right|_{q_1=q_+ + q_-/2}\cr
~-~&\left. L(q_2,\dot{q}_2,t)\right|_{q_2=q_+ - q_-/2}\cr
~+~&Q_j(q_+,\dot{q}_+,t)q^j_-\cr
~=~&\sum_{j=1}^n\frac{\partial L(q_+,\dot{q}_+,t)}{\partial\dot{q}_+^j}\dot{q}_-^j\cr
~+~&\sum_{j=1}^n\left(\frac{\partial L(q_+,\dot{q}_+,t)}{\partial q_+^j} +Q_j(q_+,\dot{q}_+,t)\right)q_-^j\cr
~+~&{\cal O}(q_-^3).
\end{align}\tag{3.1} $$
The EL equations for the tilde Lagrangian (3.1) wrt. $q_+$ has the physical limit solution
$$ q_-^j~=~0,\qquad j~\in~\{1, \ldots, n\}; \tag{3.2}$$
while the EL equations for the tilde Lagrangian (3.1) wrt. $q_-$
$$ \begin{align}
\frac{\partial L(q_+,\dot{q}_+,t)}{\partial q_+^j}-\frac{d}{dt}\frac{\partial L(q_+,\dot{q}_+,t)}{\partial \dot{q}_+^j} +Q_j(q_+,\dot{q}_+,t)~=~&{\cal O}(q_-^2), \cr
j~\in~&\{1, \ldots, n\},
\end{align}\tag{3.3}$$
is the sought-for Lagrange equations (0.2) [up to higher-order terms in $q_-$, which vanish in the physical limit (3.2)]. The initial conditions
$$\left\{\begin{array}{rcl} q^j_+(t_i)&=&q^j_i,\cr\dot{q}^j_+(t_i)&=&\dot{q}^j_i\end{array}\right.\tag{3.4} $$
implement the system's underlying initial values.
The final conditions
$$\begin{align}\left\{\begin{array}{rcl} q^j_-(t_f)&=&0\cr \dot{q}^j_-(t_f)&=&0 \end{array}\right. &
\cr\cr\qquad\Downarrow&\qquad\cr\cr
\left.\frac{\partial \widetilde{L}}{\partial \dot{q}^j_+}\right|_{t=t_f}~=~&0 \end{align}\tag{3.5} $$
implement the physical limit solution (3.2). The doubling trick (3.1) is often effectively the same as introducing Lagrange multipliers (2.1).
Gurtin-Tonti bi-local method: See e.g. this Phys.SE post.
$^1$ Hat tip: Valter Moretti.
$^2$ The variational problem (3.1)+(3.4)+(3.5) needs an appropriate initial term, which might not always exist! In particular, since we already imposed $4n$ boundary conditions (3.4)+(3.5), it would be too much to also impose the initial condition
$$ q^j_-(t_i)~=~0. \qquad (\leftarrow\text{Wrong!})\tag{3.6}$$
Example: If
$$L~=~\frac{1}{2}m\dot{q}^2,\qquad Q~=~0,\tag{3.7}$$
then
$$\widetilde{L}~\stackrel{(3.7)+(3.1)}{=}~m\dot{q}_+\dot{q}_-,\tag{3.8}$$
and one should add an initial term $\left. m\dot{q}_+q_-\right|_{t=t_i}$ to the action
$$\widetilde{S}~=~\left. m\dot{q}_+q_-\right|_{t=t_i}+
\int_{t_i}^{t_f}\!dt~\widetilde{L}
,\tag{3.9}$$
so that an infinitesimal variation becomes
$$ \delta \widetilde{S}~\stackrel{\rm IBP}{=}~\text{bulk terms} ~+~ \text{boundary terms},\tag{3.10}$$
where the boundary terms
$$\begin{align}\text{boundary terms}
~=~&\left. mq_-\delta\dot{q}_+ \right|_{t=t_i}
+\left. m\dot{q}_+\delta q_-\right|_{t=t_i}
+\left[m\dot{q}_{\pm}\delta q_{\mp}\right]_{t=t_i}^{t=t_f}\cr
~=~&\left. mq_-\delta\dot{q}_+ \right|_{t=t_i}
+\left. m\dot{q}_+\delta q_-\right|_{t=t_f}
+\left[m\dot{q}_-\delta q_+\right]_{t=t_i}^{t=t_f}\cr
~\stackrel{(3.4)+(3.5)}{=}&~0.\end{align}\tag{3.11}$$
vanish, as they should.
