unitarity - When is a unitary operator a quantum gate?

Quantum gates we use like X, Y, Z, H, CNOT, etc. are all unitary. When can an arbitrary unitary operator be considered as a quantum gate?

Answer

Quantum gates are all unitary transformations on a state of qubits. Any unitary transformation can be considered a "gate", although the ones you mention are primitive ones from which others can be constructed. More complex ones are usually referred to as circuits. The two qubit gates, $\mathrm{H}$, $\frac{\pi}{8}$ and $\mathrm{CNOT}$ are considered universal gates, because any gate set can be constructed out of those.

You may want to take a look at these lecture notes, in particular, Lemma 12. I would also suggest getting a hold of the textbook by Nielsen & Chuang.

Addendum: I said something incorrect about the Toffoli gate, which is universal for classical computation, but as Peter Shor pointed out in the comments, will not give you complex entries (both Hadamard and Toffoli are real).