Wilks' Theorem in the Multi-Parameter Case

Derives Lemma A21.6. Each entry of \(J_{n}\) is the average of \(n\) independent copies of \(\pp^{2}\ln f(X;\theta_{0})/\pp\theta^{a}\pp\theta^{b}\), integrable by (R4) with mean \(-I_{ab}\) by Equation (15.48), so Theorem A21.3 gives \(J_{n}\rightarrow-I\) entrywise in probability; a matrix norm is a continuous function of finitely many entries, so \(\norm{J_{n}+I}\rightarrow0\) in probability. The same theorem applied to \(M\), integrable by (R5), gives the second claim.

For the third, fix \(\vect{a}\in\R^{k}\), \(\vect{a}\neq\vect{0}\). Then \(\vect{a}\transpose\vect{Z}_{n} =n^{-1/2}\sum_{i}\vect{a}\transpose\vect{U}(\theta_{0};X_{i})\) with \(\vect{U}=\nabla\ln f\) the score, a normalised sum of independent identically distributed variables with mean zero (Lemma 15.61) and variance \(\vect{a}\transpose I\vect{a}\), finite and strictly positive by (R5). Theorem 15.46 gives \(\vect{a}\transpose\vect{Z}_{n}\rightarrow \mathcal{N}(0,\vect{a}\transpose I\vect{a})\), and Theorem A21.4 converts this into \(\vect{Z}_{n}\rightarrow\mathcal{N}(0,I)\). Boundedness in probability of each component is Equation (15.43), and \(\norm{\vect{Z}_{n}}\le\sum_{a}\abs{Z_{n}^{a}}\) is a finite sum of variables each bounded in probability.

Derives Lemma A21.7. Fix \(\theta\) and set \(g(u)=\ell_{n}(\theta_{0}+u\delta)\) for \(u\in[0,1]\); the segment lies in \(\bar B(\theta_{0},a_{0})\) by convexity of the ball, so \(g\) is three times continuously differentiable by (R4). Taylor's theorem with Lagrange remainder (Theorem 11.38) gives \(g(1)=g(0)+g'(0)+\tfrac12g''(0)+\tfrac16g'''(\xi)\) for some \(\xi\in(0,1)\). By the chain rule (Proposition 11.104),

which are the first two displayed terms, and

Each third partial derivative of \(\ell_{n}\) is a sum of \(n\) terms, one per observation, and (R5) bounds each of them by \(M(X_{i})\) throughout \(\bar B(\theta_{0},a_{0})\) — in particular at the unknown intermediate point \(\theta_{0}+\xi\delta\), which is what makes the estimate independent of \(\xi\) and hence of \(\theta\). Since \(\abs{\delta^{a}}\le\norm{\delta}\) for each component and the sum Equation (A21.6) has \(k^{3}\) terms,

and \(R_{n}(\theta)=\tfrac16g'''(\xi)\).

Derives Corollary A21.8. Put \(\delta=\vect{u}/\sqrt{n}\) in Equation (A21.5) and multiply by \(-2\):

because \(n\delta\transpose J_{n}\delta =\vect{u}\transpose J_{n}\vect{u}\) and \(\sqrt{n}\vect{Z}_{n}\transpose\delta =\vect{Z}_{n}\transpose\vect{u}\). Subtracting Equation (A21.7) leaves \(-\vect{u}\transpose\left(J_{n}+I\right)\vect{u}-2R_{n}\), whose first part is at most \(\norm{J_{n}+I}\norm{\vect{u}}^{2}\) in modulus and whose second is at most \(\tfrac13k^{3}\bar{M}_{n}\norm{\vect{u}}^{3}/\sqrt{n}\) by Equation (A21.5) with \(\norm{\delta}=\norm{\vect{u}}/ \sqrt{n}\). Both are increasing in \(\norm{\vect{u}}\), so the supremum over the ball is attained at the bound. That \(\epsilon_{n}(K)\rightarrow0\) in probability is Lemma A21.6: the first term is \(K^{2}\) times something tending to \(0\), and the second is \(\bar{M}_{n}\), bounded in probability, times \(\tfrac13k^{3}K^{3}n^{-1/2}\rightarrow0\).

Derives Lemma A21.9. Write \(\hat{\delta}=\hat{\theta}-\theta_{0}\) and \(d=\norm{\hat{\delta}}\le a_{1}\). Since \(\theta_{0}\in A\) we have \(\ell_{n}(\hat{\theta})\ge\ell_{n}(\theta_{0})\), so Equation (A21.5) gives

Bound the three terms. The first is at most \(\sqrt{n}\norm{\vect{Z}_{n}}d\) by the Cauchy–Schwarz inequality for vectors. For the second, write \(\hat{\delta}\transpose J_{n}\hat{\delta} =-\hat{\delta}\transpose I\hat{\delta} +\hat{\delta}\transpose\left(J_{n}+I\right)\hat{\delta}\); the first piece is at most \(-\lambda d^{2}\), since \(\lambda\) is the smallest eigenvalue of the positive-definite \(I\), and the second is at most \(\norm{J_{n}+I}d^{2}\). The third is at most \(\tfrac16nk^{3}\bar{M}_{n}d^{3}\). Hence

From Equations (A21.5) and (A21.10) (bound the linear term by Cauchy–Schwarz, split the Hessian term into \(-I\) plus its deviation, and insert the uniform remainder bound).

Let \(E_{n}\) be the event on which both \(\norm{J_{n}+I}\le\lambda/4\) and \(\bar{M}_{n}\le\bar{M}+1\); by Lemma A21.6, \(\Pr(E_{n})\rightarrow1\). On \(E_{n}\), and using \(d\le a_{1}\) with the choice of \(a_{1}\),

Substituting in Equation (A21.11) and cancelling \(n\lambda d^{2}/4\) from both sides leaves \(\tfrac14n\lambda d^{2}\le\sqrt{n}\norm{\vect{Z}_{n}}d\); dividing by \(\tfrac14n\lambda d\) when \(d>0\) — and noting the conclusion is trivial when \(d=0\) — gives \(\sqrt{n}\,d\le4\norm{\vect{Z}_{n}}/\lambda\), which is Equation (A21.9). Boundedness in probability follows because \(\norm{\vect{Z}_{n}}\) is bounded in probability (Lemma A21.6), and consistency because \(\sqrt{n}d=O_{P}(1)\) forces \(d\rightarrow0\) in probability.

Derives Lemma A21.10. Choose \(a_{2}\in(0,a_{1}]\) small enough that the closed ball \(\bar B(\theta_{0},2a_{2})\) lies in \(W\) and that \(\set{\vect{e}\in E\mid\norm{\vect{e}}\le2a_{2}}\subseteq V\); let \(c_{0}=\tfrac12\sup\norm{D^{2}h}\) over that compact set, finite because \(h\) is twice continuously differentiable and the supremum of a continuous function on a compact set is attained (Theorem 11.24). Taylor's theorem applied to \(u\mapsto h(u\vect{e})\) on \([0,1]\), with \(h(\vect{0})=\vect{0}\) and \(Dh(\vect{0})=0\) killing the first two terms, gives

(ii) Take \(\theta=\theta_{0}+\vect{e}+h(\vect{e})\), which lies in \(\Theta_{0}\) by Equation (A21.4); then \((\theta-\theta_{0})-\vect{e}=h(\vect{e})\) and Equation (A21.12) applies.

(i) Such a \(\theta\) lies in \(W\), so \(\theta-\theta_{0}=\vect{e}+h(\vect{e})\) for some \(\vect{e}\in V\); the two summands are orthogonal, so \(\vect{w}=\vect{e}\) and \(\norm{\vect{e}}\le\norm{\theta-\theta_{0}}\le a_{2}\). Then \((\theta-\theta_{0})-\vect{w}=h(\vect{e})\), and Equation (A21.12) together with \(\norm{\vect{e}}\le\norm{\theta-\theta_{0}}\) gives the bound.

Derives Lemma A21.11. Since \(I\) is positive definite, \(\Lambda_{n}\) is a strictly convex quadratic; completing the square with \(\vect{m}=I^{-1}\vect{Z}_{n}\),

so the unconstrained minimum is attained at \(\vect{m}\) and the minimum over the subspace \(E\) at the \(I\)-orthogonal projection \(\vect{p}\) of \(\vect{m}\) onto \(E\). Both \(\norm{\vect{m}}\) and \(\norm{\vect{p}}\) are bounded in probability, being bounded linear images of \(\vect{Z}_{n}\). Also \(\Lambda_{n}\) is Lipschitz on each ball: for \(\norm{\vect{u}},\norm{\vect{u}'}\le K\),

which is bounded in probability for fixed \(K\).

Fix \(\varepsilon>0\). By Lemma A21.9 and the boundedness just noted, there is \(K\) such that with probability at least \(1-\varepsilon\) for all large \(n\) the four vectors \(\hat{\vect{u}},\hat{\vect{u}}_{0},\vect{m},\vect{p}\) all lie in \(\bar B(\vect{0},K)\); call that event \(A_{n}\) and intersect it with the event that \(\epsilon_{n}(K+1)\le\varepsilon\), which also has probability approaching \(1\) by Corollary A21.8. Work on the intersection, and abbreviate \(D_{n}(\theta)=-2\left[\ell_{n}(\theta)-\ell_{n}(\theta_{0})\right]\), so that Equation (A21.8) reads \(\abs{D_{n}(\theta_{0}+\vect{u}/\sqrt{n})-\Lambda_{n}(\vect{u})} \le\varepsilon\) for every \(\norm{\vect{u}}\le K+1\). The extra unit of radius is there because one auxiliary point below, the image in \(\Theta_{0}\) of the tangential minimiser, misses \(\bar B(\vect{0},K)\) by \(O(n^{-1/2})\); all other points used lie in \(\bar B(\vect{0},K)\). Write \(L_{n}=L_{n}(K+1)\) for the Lipschitz constant Equation (A21.15) at that radius.

First limit. That \(\Lambda_{n}(\hat{\vect{u}})\ge \min_{\R^{k}}\Lambda_{n}\) is immediate. For the reverse, put \(\theta_{\mathrm{m}}=\theta_{0}+\vect{m}/\sqrt{n}\), which lies in \(\bar B(\theta_{0},a_{1})\) once \(K/\sqrt{n}\le a_{1}\), so \(\ell_{n}(\hat{\theta}_{n})\ge\ell_{n}(\theta_{\mathrm{m}})\) and hence \(D_{n}(\hat{\theta}_{n})\le D_{n}(\theta_{\mathrm{m}})\). Therefore

so the first difference in Equation (A21.13) is at most \(2\varepsilon\) on an event of probability approaching \(1-\varepsilon\).

Second limit. For the lower bound let \(\vect{w}\) be the orthogonal projection of \(\hat{\vect{u}}_{0}\) onto \(E\). By Lemma A21.10(i) applied to \(\theta=\hat{\theta}_{0,n}\in\Theta_{0}\), whose distance to \(\theta_{0}\) is at most \(K/\sqrt{n}\le a_{2}\) for large \(n\),

and \(\norm{\vect{w}}\le\norm{\hat{\vect{u}}_{0}}\le K\), so Equation (A21.15) gives \(\Lambda_{n}(\hat{\vect{u}}_{0})\ge\Lambda_{n}(\vect{w}) -L_{n}c_{0}K^{2}/\sqrt{n}\ge\min_{E}\Lambda_{n} -L_{n}c_{0}K^{2}/\sqrt{n}\).

For the upper bound apply Lemma A21.10(ii) to \(\vect{p}/\sqrt{n}\), of norm at most \(K/\sqrt{n}\le a_{2}\): there is \(\theta_{\mathrm{p}}\in\Theta_{0}\) with \(\norm{\left(\theta_{\mathrm{p}}-\theta_{0}\right) -\vect{p}/\sqrt{n}}\le c_{0}K^{2}/n\), hence \(\norm{\sqrt{n}\left(\theta_{\mathrm{p}}-\theta_{0}\right)-\vect{p}} \le c_{0}K^{2}/\sqrt{n}\) and \(\theta_{\mathrm{p}}\in \Theta_{0}\cap\bar B(\theta_{0},a_{1})\) for large \(n\). Since \(\hat{\theta}_{0,n}\) maximises \(\ell_{n}\) over that set, \(D_{n}(\hat{\theta}_{0,n})\le D_{n}(\theta_{\mathrm{p}})\), and running the same three-step chain as before, then moving from \(\sqrt{n}(\theta_{\mathrm{p}}-\theta_{0})\) to \(\vect{p}\) by Equation (A21.15),

Both correction terms tend to \(0\) in probability, \(L_{n}\) being bounded in probability, and \(\varepsilon>0\) was arbitrary.

Derives Theorem A21.12. Write \(D_{n}\) as in Lemma A21.11, so that \(q=D_{n}(\hat{\theta}_{0,n})-D_{n}(\hat{\theta}_{n})\) — the common reference value \(\ell_{n}(\theta_{0})\) cancels. By Corollary A21.8 each \(D_{n}\) equals the corresponding \(\Lambda_{n}\) up to \(\epsilon_{n}(K)\), and by Lemma A21.11 each \(\Lambda_{n}\) equals the corresponding minimum up to a quantity tending to zero in probability. Hence

From Equations (A21.13) and (A21.14) (replace each statistic by the minimum of the quadratic form and cancel the common constant).

The second equality is Equation (A21.14), whose additive constant \(-\vect{Z}_{n}\transpose I^{-1}\vect{Z}_{n}\) is the same in both minima and cancels.

Now substitute \(\vect{V}_{n}=I^{1/2}\vect{m}=I^{-1/2}\vect{Z}_{n}\) and \(S=I^{1/2}E\), where \(I^{1/2}\) is any invertible matrix with \(I^{1/2}\left(I^{1/2}\right)\transpose=I\), symmetric here; the existence of one is the weak form of the spectral theorem recorded in Remark 15.86. Since \(I^{1/2}\) is invertible, \(\dim S=\dim E=k-r\), and

with \(P\) the orthogonal projector onto \(S^{\perp}\), of dimension \(r\): the nearest point of a subspace to a given vector is its orthogonal projection, and what is left over is the projection onto the orthogonal complement.

By Lemma A21.6, \(\vect{Z}_{n}\rightarrow \mathcal{N}(0,I)\) in distribution; applying Theorem A21.4 to the linear map \(\vect{z}\mapsto I^{-1/2}\vect{z}\) — for each \(\vect{a}\), \(\vect{a}\transpose I^{-1/2}\vect{Z}_{n}\rightarrow \mathcal{N}(0,\vect{a}\transpose I^{-1/2}II^{-1/2}\vect{a}) =\mathcal{N}(0,\norm{\vect{a}}^{2})\) — gives \(\vect{V}_{n}\rightarrow\vect{V}\sim\mathcal{N}(0,\identity)\) in distribution. The map \(\vect{v}\mapsto\norm{P\vect{v}}^{2}\) is continuous, so the continuous-mapping clause of Theorem A21.4 gives \(\norm{P\vect{V}_{n}}^{2}\rightarrow\norm{P\vect{V}}^{2}\) in distribution, and \(\norm{P\vect{V}}^{2}\sim\chi^{2}_{r}\) by Lemma 15.83. Finally, adding an \(o_{P}(1)\) term does not change a limit in distribution, by the sum clause of Lemma 15.52. Hence \(q\rightarrow\chi^{2}_{r}\).