hamiltonian formalism - Hamilton's equations from the action with boundary conditions involving position and momentum

Generally, when you are given the action

$$ S=\int_{t_1}^{t_2}\mathrm dt (p\dot q - \mathcal H )$$

the boundary conditions are

$$q(t_1)=q_1\quad\text{and}\quad q(t_2)=q_2.$$

This is useful because to calculate $\delta S$ we do an integration by parts with boundary term

$$ [p\delta q]_{t_1}^{t_2} = 0. $$

But suppose I give you different boundary conditions for the action, namely $$q(t_1)=q_1\quad\text{and}\quad p(t_2)=p_2.$$
Then solving $\delta S = 0$ should still give you Hamilton's equations, I think, but I'm having trouble showing this, as I get annoying boundary terms since $\delta q(t_2) \neq 0$.

Answer

I) In general, for a given choice of boundary conditions, it is important to adjust the action with compatible boundary terms/total divergence terms in order to ensure the existence of the variational/functional derivative. As OP observes, the problem is (when deriving the Euler-Lagrange expression) that the usual integration by parts argument fails if the boundary conditions (BCs) and the boundary terms (BTs) are not compatible.

II) Concretely, for the mixed BCs

$$\tag{1} q(t_i) ~=~ q_i \qquad\text{and}\qquad p(t_f) ~=~ p_f, $$

which OP considers, we need to prepare the standard Hamiltonian action

$$\tag{2} S_0[p,q] ~=~ \int_{t_i}^{t_f} \! dt ~ \left\{ p\dot{q} -H \right\} $$

with a total divergence term $-\frac{d}{dt}(p_f q)$. The new action becomes

$$\tag{3} S[p,q] ~=~ \int_{t_i}^{t_f} \! dt ~ \left\{ (p-p_f)\dot{q} -H \right\} ,$$

or what amounts to the same,

$$\tag{4} S[p,q] ~=~ \int_{t_i}^{t_f} \! dt ~ \left\{ \dot{p}(q_i-q) -H \right\}. $$

It is straightforward to use the BCs (1) to show that the actions (3) and (4) are equal.

III) Now when we vary the action (3)

$$ \tag{5} \delta S[p,q] ~=~ \left[ (p-p_f)\delta q\right]_{t_i}^{t_f}+ \int_{t_i}^{t_f} \! dt ~ \left\{ (\dot{q} -\frac{\partial H}{\partial p})\delta p -(\dot{p}+\frac{\partial H}{\partial q})\delta q\right\}, $$

the BCs (1) cancels the total derivative term

$$ \tag{6} \left[ (p-p_f)\delta q\right]_{t_i}^{t_f}~\stackrel{(1)}{=}~0, $$

so that the variation (5) only contains bulk terms. The corresponding Euler-Lagrange equations become Hamilton's equations.

IV) The above example can be generalized to other BCs. We leave it to the reader to work out the compatible BTs.