general relativity - What is the reasoning that leads one to postulate this second form for the relativistic particle action?

The action for the free relativistic particle with worldline $\gamma : I\subset \mathbb{R}\to M$ is

$$S[\gamma]=-m\int d\lambda\sqrt{-\dot{\gamma}^a(\lambda)\dot{\gamma}_a(\lambda)}\tag{1} $$

Now, one may postulate a second action

$$S'[\gamma,\eta]=\frac{1}{2}\int d\lambda \bigg(\eta(\lambda)^{-1}\dot{\gamma}^a(\lambda)\dot{\gamma}_a(\lambda)-\eta(\lambda)m^2\bigg).\tag{2}$$

These are classically equivalent actions.

My question is: usually what we have is (1) and we have a problem both with the square root and with the massless limit. Given this, how could we think about postulating (2)? In other words, how can we reach (2)?

Usually some people answer this by saying: "it doesn't matter, actions are postulated, you postulate it, compute the equations, prove it works and its over".

Now I beg to differ. I want to know how could someone reason exactly in order to know what to postulate.

I am perfectly comfortable with computing the equations of motion. I want to know is how given (1) we would have the idea to postulate (2).

Is it some special case of some general procedure that deals with constraints?

Answer

It's a lore in physics that any Lagrangian system has an equivalent Hamiltonian formulation. If we start from OP's square root Lagrangian $$ L_0~:=~ -m\sqrt{-\dot{x}^2}, \qquad \dot{x}^2~:=~g_{\mu\nu}(x)~ \dot{x}^{\mu}\dot{x}^{\nu}~<~0,\tag{1}$$ it is natural to ponder what the Hamiltonian formulation is? The momentum reads $$ p_{\mu}~=~\frac{\partial L_0}{\partial\dot{x}^{\mu}}~\stackrel{(1)}{=}~\frac{m\dot{x}_{\mu}}{\sqrt{-\dot{x}^2}},\tag{2}$$ and the energy $$ H_0~=~p_{\mu}\dot{x}^{\mu}-L_0~\stackrel{(1)+(2)}{=}~0\tag{3}$$ vanishes, cf. e.g. this Phys.SE post. Now when we try to perform the Legendre transformation, we discover that the momenta $p_{\mu}$ are not all independent. They have to satisfy a mass-shell constraint $$ p^2+m^2~\stackrel{(2)}{\approx}~0, \qquad p^2~:=~g^{\mu\nu}(x)~ p_{\mu}p_{\nu}~<~0.\tag{4}$$ The Dirac-Bergmann prescription then tell us to impose this constraint in the Hamiltonian $$H~=~\frac{e}{2}(p^2+m^2)\tag{5}$$ via a Lagrange multiplier field $e$. It is easy to check that this is indeed the Hamiltonian formulation of a relativistic point particle. The corresponding Hamiltonian Lagrangian becomes $$ L_{H}~=~p_{\mu}\dot{x}^{\mu}-H~\stackrel{(5)}{=}~p_{\mu}\dot{x}^{\mu}-\frac{e}{2}(p^2+m^2) .\tag{6}$$ Ok, cool, but what does this have to do with OP's question?, the reader may ask. Wait for the punch-line: Now we can ask the opposite question: What happens if we perform the inverse Legendre transformation, namely eliminate/integrate out the momenta via their EL equations $$ p_{\mu}~\stackrel{(6)}{\approx}~ \frac{1}{e}~\dot{x}_{\mu}~?\tag{7}$$ Surprisingly, we don't get quite back to where we started. Instead we get OP's non-square root Lagrangian $$L~\stackrel{(6)+(7)}{=}~\frac{\dot{x}^2}{2e}-\frac{e m^2}{2}.\tag{8}$$ This is one possible answer to OP's title question. For more information about relativistic point particles, see this Phys.SE post. For the analogous question for strings, see this Phys.SE post.