Particle

What is a particle? S. Weinberg described a particle simply as "a physical system that has no continuous degrees of freedom except for its total momentum".  Recall that the spacetime admits the so-called Poincare symmetry \begin{equation} {x'}^{\mu} = \Lambda^{\mu}_{\,\,\nu}\,x^{\nu} + a^{\mu}\,.\end{equation} By Noether's theorem, each continuous symmetry indicates a conservation. Thus, the total momentum $p^{\mu}$ is a good quantum because of the spacetime translation invariance. As we shall see below, there are other discrete degrees of freedom, denoted by $\sigma$, that are associated with the Lorentz invariance. So the quantum state of a particle is described as $|p, \sigma\rangle$. 

Under the Lorentz transformation $\Lambda$, the one-particle state $|p, \sigma\rangle$ changes to a new state $U(\Lambda)|p,\sigma\rangle$ by a unitary operator $U(\Lambda)$. Note that the Lorentz transformation only changes the reference frames, elementary particles should remain the same species. Therefore, for a given particle, all its states $|p, \sigma\rangle$ should form a subspace that is invariant under the Lorentz transformation. That is, $|p,\sigma\rangle$ is the basis of the irreducible unitary representation of the Poincare group, which can be viewed as an equivalent definition of one-particle state.

The detailed derivations of the irreducible unitary representation of the Poincare group can be found in Weinberg's QFT volume I. Here we only sketch the key steps:

1. Note that any momentum $p^{\mu}$ can be obtained from some standard reference momentum $k^{\mu}$ by some Lorentz transformation $L(p)$: \begin{equation}p^{\mu} = L^{\mu}_{\,\,\nu}(p)\, k^{\nu}\,,\tag{1}\end{equation} where the reference momentum $k$ and the corresponding Lorentz transformation $L(p)$ can be chosen as follows:
   • If $p^{\mu}p_{\mu}\equiv -M^2 < 0$, we choose \begin{equation}k^{\mu}\equiv [M, 0, 0, 0]^T\,.\end{equation} For a massive particle, one can always find an inertial frame such that the massive particle looks rest in this inertial frame. $L(p)$ in this case can be found in this post.
   • If  $p^{\mu}p_{\mu}= 0$, we choose \begin{equation}k^{\mu}\equiv [\kappa, 0, 0, \kappa]^T\,.\end{equation} A massless particle travels at the speed of light in all inertial frames. $L(p)$ in this case can be found in this post.
2. Because of the transformation (1), the one-particle state changes as \begin{equation} |p, \sigma\rangle = \sqrt{\frac{k^0}{p^0}} U\left(L(p)\right)\,|k,\sigma\rangle\,,\tag{2}\end{equation} where the normalization constant $\sqrt{k^0/p^0}$ comes from the requirements $\langle p'|p\rangle = \delta^{(3)}(\mathbf{p'}-\mathbf{p})$ and $\langle k'|k\rangle = \delta^{(3)}(\mathbf{k'}-\mathbf{k})$ as well as the Lorentz invariance $\sqrt{p^0} \delta^{(3)}(\mathbf{p'}-\mathbf{p}) = \sqrt{k^0}\delta^{(3)}(\mathbf{k'}-\mathbf{k})$.
3. By applying $U(\Lambda)$ on both sides of Eq.(2), we have 
   \begin{equation}U(\Lambda)\,|p,\sigma\rangle = \sqrt{\frac{k^0}{p^0}}\, U(L(\Lambda p)) U(W) |k,\sigma\rangle\,,\tag{3} \end{equation} in which \begin{equation}W\equiv L^{-1}(\Lambda p)\, \Lambda\,L(p)\tag{4}\end{equation} is called Wigner rotation. Here is the key observation: the Wigner rotation (4) does not change $k$ since it first changes $k$ to $p$, then to $\Lambda p$, and finally back to $k$. As a result, we can expand $U(W) |k,\sigma\rangle$ as \begin{equation}\boxed{U(W) |k,\sigma\rangle = \sum_{\sigma'} |k, \sigma'\rangle D_{\sigma'\sigma}(W)}\,.\tag{5}\end{equation} Be careful for the order of the subscripts $\sigma'$ and $\sigma$.
4. Combining Eq. (2), (4) and (5), we can show that \begin{equation}\boxed{U(\Lambda)\,|p,\sigma\rangle = \sqrt{\frac{(\Lambda p)^0}{p^0}}\sum_{\sigma'} |\Lambda p, \sigma'\rangle D_{\sigma'\sigma}(W)}\,.\tag{6}\end{equation}

The main conclusion is that the irreducible unitary representation of the full Poincare group $D_{\sigma'\sigma}(W)$, as in Eq. (6), is exactly the same as the representation of a subgroup of the Poincare group in Eq. (5) that keeps the reference momentum $k^{\mu}$ unchanged. This subgroup is called little group and all the left is to work out the little group representation $D_{\sigma'\sigma}(W)$ in Eq.(5).

Massive case

For the massive case, it is easy to see that all Lorentz transformations $W$ that keep $k^{\mu}\equiv [M, 0, 0, 0]^T$ unchanged form the 3-dimensional special rotation group SO(3).

The projective representation of SO(3) is the ordinary representation of its universal covering group SU(2) and we already learned the irreducible unitary representation of SU(2) in quantum mechanics. Here is a brief summary: 

• The basis state of the irreducible unitary representation can be written as of $|j, m\rangle$, where $j=0,\frac{1}{2},1, \frac{3}{2}, 2\cdots$ and $m =-j, -j+1, \cdots,j-1, j$.
• Let $J_1, J_2, J_3$ be the three components of the angular momentum operator, then the state $|j, m\rangle$ satisfies the relations \begin{equation}\begin{split}J^{2}\,|j,m\rangle &=&j(j+1)\,|j,m\rangle\\ J_3\,|j,m\rangle &=& m\,|j,m\rangle\\ J_{\pm}|j,m\rangle&=& \sqrt{j(j+1)-m(m\pm 1)}\,|j,m\pm 1\rangle \end{split}\,,\tag{7}\end{equation} where  $J^2\equiv J_1^2+J_2^2+J_3^2$ and $J_{\pm}\equiv J_1\pm i J_2$.
• A general rotation $R$ in ordinary three-dimensional space can be parameterized as \begin{equation}R\equiv \exp\left[-i  \sum_{l=1}^3\omega_l\,L_l\right]\tag{8}\end{equation} where $L_1,L_2,L_3$ are 3x3 matrices of three generators associated with the rotations along x, y and z directions, whose explicit forms can found in Eq. (3) in this post. Then its irreducible unitary representation within (2j+1)-dimensional subspace is \begin{equation}U(R)|j,m\rangle = \sum_{m'} |j, m'\rangle\, {\mathscr{D}}^{(j)}_{m'\,m}(R)\,,\end{equation} where \begin{equation}{\mathscr{D}}^{(j)}_{m'\,m}(R)=\langle j, m'|  \exp\left[-i  \sum_{l=1}^3\omega_l\,J_l\right] |j,m\rangle\tag{9}\end{equation} can be computed explicitly by the relations (7). Note that the only difference between (8) and (9) is on the generators $L_l$ and $J_l$. Indeed, (8) is a special case of (9) when $j=1$.

With this knowledge, for any $W\in SO(3)$, the irreducible unitary representation of little group (5) in the massive case can be written in the following form: \begin{equation}\boxed{U(W) |k,j,m\rangle = \sum_{m'=-j}^j |k, j, m'\rangle {\mathscr{D}}^{(j)}_{m'\,m}(W)}\,.\tag{10}\end{equation}Recall $k\equiv [M,0,0,0]^T$, Eq. (10) suggests that all states $|k,j,m\rangle$ with the same $M$ and $j$ form the basis of a $(2j+1)$-dimensional invariant subspace indexed by $m$, leading to the following conclusions:

• We can fully classify all massive particles in their rest mass $M$ and spin $j$.
• A massive particle with rest mass $M$ and spin $j$ has $2j+1$ internal degrees of freedom that are the spin projection along some axis. 

Massless case

For the massless case, as shown in this post, all Lorentz transformations $W$ that keep $k^{\mu}\equiv [\kappa, 0, 0, \kappa]^T$ unchanged form the 2-dimensional special Euclidean group SE(2) and the basis of the irreducible representation of SE(2) can be written as the state $|t,j\rangle$.

The case $t\neq 0$ will lead to the so-called "continuous spin representation", which will NOT be considered for the following reasons:

• Empirical experience: we have never observed such continuous degree of freedom in experiments.
• Theoretically, it brings locality issue in quantum field theory as mentioned in Chapter 4.2 in T. Banks QFT book.
• There is research on this continuous spin representation and it possibility in reality. See this top answer for a list of several papers in this area.

So we only consider the irreducible representation of SE(2) when $t=0$. As worked out in this post, the irreducible unitary representation of little group (5) in the massless case can be written in the following form:  \begin{equation}\boxed{U(W(\theta))\,|k,0, j\rangle = e^{-i\, j\,\theta}\,|k, 0, j\rangle}\tag{11}\end{equation} for any $W\in SE(2)$ and $k\equiv [\kappa,0,0,\kappa]^T$. In this degenerate representation, 

• The only effective parameter in $W$ is $\theta$, which is the rotation angle along the momentum direction $\mathbf{k}$. Other parameters in $W$ becomes the gauge degrees of freedom in the massless case.
• $j$ is called helicity, in contrast to the name of spin in the massive case. 
• The fundamental group of the Lorentz group is $\mathbb{Z}_2$, which means any closed loop traversed twice can be continuously contracted to a point. As a result, similar to the spin in the massive case, the helicity $j$ in the massless case (11) can only be integers and half-integers.

As a result, we can think of massless particles with different helicity as different species. However, the particles with exactly the opposite helicity are related by space inversion symmetry.

Conclusion

With the little group (10) and (11) in massive and massless cases,  the irreducible unitary representation of Poincare group in Eq. (6) becomes \begin{equation}U(\Lambda)\,|p,j,m\rangle = \sqrt{\frac{(\Lambda p)^0}{p^0}}\sum_{m'=-j}^j |\Lambda p, j,m'\rangle\,{\mathscr{D}}^{(j)}_{m'm}\left(W(\Lambda, p)\right)\tag{12}\end{equation} for $p^{\mu}p_{\mu}<0$, and \begin{equation}U(\Lambda)\,|p,j\rangle = \sqrt{\frac{(\Lambda p)^0}{p^0}}e^{-i\,j\,\theta(\Lambda, p)} |\Lambda p, j\rangle \tag{13}\end{equation} for $p^{\mu}p_{\mu}=0$.

 

In sum, we can classify all particles in its rest mass $M$ and $j$, where $M$ is continuous and $j$ takes values of integers and half-integers. For each particle,

• If $M>0$ (massive), there are $2j+1$ internal degrees of freedom (spin projection);
• If $M=0$ (massless), there are 2 or 1 internal degrees of freedom w/o space inversion symmetry (helicity). 

Remarks:

• There are only infinite-dimensional unitary representations of Poincare group since Poincare group is noncompact. Indeed, the number of one-particle state as in  (12) and (13) that forms the basis of irreducible unitary representation is infinite: once a state with momentum $p$ included, the state with momentum $\Lambda p$ also needs to be included.
• The Lie algebra of Lorentz group is equivalent to $su(2)\oplus su(2)$, from which one can construct the irreducible representation of the dimension $(2j_1+1)\times (2j_2+1)$. Such irreducible representation is finite, and thus can NOT be unitary.