1
$\begingroup$

I have been struggling quite a bit with reconciling my intuitive understanding of probability distributions with the weird properties that almost all topologies on probability distributions possess.

For example, consider a mixture random variable $X_n$: pick a Gaussian centered at 0 with variance 1, and with probability $\frac{1}{n}$, add $n$ to the result. A sequence of such random variables would converge (weakly and in total variation) to a Gaussian centered at 0 with variance 1, but the mean of the $X_n$ is always $1$ and the variances converge to $+\infty$. I really don't like saying that this sequence converges because of that.

edit: $X_n$ has density

$$p_n(x) = \frac{n-1}{n} g(x) + \frac{1}{n} g(x-n)$$

where $g$ is the density of the gaussian with unit variance and mean 0

I took me quite some time to remember everything I've forgotten about topologies, but I finally figured out what was so unsatisfying to me about such examples: the limit of the sequence is not a conventional distribution. In the example above, the limit is a weird "Gaussian of mean 1 and of infinite variance". In topological terms, the set of probability distributions isn't complete under the weak (and TV, and all the other topologies I've looked at).

(note:the problem remains with probability measures)

I then face the following question:

  • does there exist a topology such that the ensemble of probability distributions is complete ?

  • If no, does that absence reflect an interesting property of the ensemble of probability distributions ? Or is it just boring ?

Original post here (crosspost crossvalidated!): https://stats.stackexchange.com/questions/186670/topologies-for-which-the-ensemble-of-probability-distributions-is-complete

Improve this question
$\endgroup$
12
  • 3
    $\begingroup$ The space of regular Borel probability measures on $\mathbb{R}$ is complete with respect to the total variation norm; by the Riesz representation theorem you can identify it with a closed subset of the dual of the Banach space $C_0(\mathbb{R})$, for instance. I'm a little confused by your example, but if the distributions really do converge in total variation then the limit is a regular Borel probability measure. $\endgroup$
    – Paul Siegel
    Dec 17, 2015 at 11:42
  • $\begingroup$ That's interesting. The sequence of $X_n$ do converge in TV (the distance is upper-bounded: $d(X_n,X_\infty)\leq 1/n$). Is being complete weirdly defined for measures ? $\endgroup$
    – Guillaume Dehaene
    Dec 17, 2015 at 15:41
  • $\begingroup$ Completeness isn't anything weird for measures, but the measures themselves can be weird. Could you elaborate on exactly what your $X_n$ are, perhaps by writing out their density functions? $\endgroup$
    – Paul Siegel
    Dec 17, 2015 at 15:51
  • 1
    $\begingroup$ What do you mean by what "the limit is really"? Under total variation, as you noted, the limit is really N(0,1). $\endgroup$
    – Yoav Kallus
    Dec 20, 2015 at 19:06
  • 3
    $\begingroup$ OK, I agree that your $X_n$ converge in TV to $N(0,1)$, and that the first two moments of $X_n$ converge to $1$ and $\infty$, respectively. But this is not an issue with completeness, it's an issue with continuity. I think you are really trying to ask: "Is there a topology on the space of probability distributions with the property that the functions $X \mapsto E(X)$ and $X \mapsto Var(X)$ defined in this space are continuous? $\endgroup$
    – Paul Siegel
    Dec 20, 2015 at 20:17

1 Answer 1

Reset to default
1
$\begingroup$

As I asserted in my comments, I think it is too much to hope for a reasonable topology on the space of random variables which makes the map $X \mapsto E(X)$ continuous. This is a bit like hoping that pointwise convergence or convergence in measure implies convergence in the $L^1$ norm; it seems reasonable, but there are simple counter-examples.

But all is not lost. In analysis one salvages the situation by making stronger assumptions: for instance, if $f_n \to f$ pointwise and $|f_n(x)| \leq g(x)$ for some integrable function $g$ and all $n$, then $\int f_n \to \int f$ (the dominated convergence theorem). There is a sort of counterpart to this in probability theory.

Definition: A sequence $X_n$ of random variables is uniformly integrable if:

  • $E(|X_n|)$ is uniformly bounded in $n$: there is a constant $K$ such that $E(|X_n|) \leq K$ for all $n$
  • For every $\varepsilon > 0$ there exists $\delta > 0$ such that $\int_A |X_n| < \varepsilon$ for all $n$ whenever $P(A) < \delta$

Uniform integrability is implied by the stronger (but more easily checked) condition that $E(|X_n|^{1 + \delta})$ is uniformly bounded for some $\delta > 0$.

Theorem: Suppose $X_n$ converges to $X$ in distribution and the sequence $|X_n|^k$ is uniformly integrable. Then $E(X_n^j) \to E(X^j)$ whenever $1 \leq j \leq k$. (Reference)

There are some other results of this flavor in the wikipedia page on convergence of random variables but this theorem is the best result that I know.

Improve this answer
$\endgroup$
4
  • $\begingroup$ Uniform integrability reminds me a little of equicontinuity. I wonder if the theorem above is a special case of a suitable generalization of the Arzela-Ascoli theorem. $\endgroup$
    – Paul Siegel
    Dec 20, 2015 at 23:25
  • 1
    $\begingroup$ I sort of agree with the basic point, but the choice of example in your first paragraph is unfortunate, if I am not mistaken; the map $f\to \int f$ is continuous as a linear functional on the normed space $L^1({\bf R})$ $\endgroup$
    – Yemon Choi
    Dec 21, 2015 at 0:45
  • $\begingroup$ @YemonChoi Arg, continuity of that map is essentially a tautology. After recovering from flashbacks to my qualifying exams in graduate school, I amended the answer. $\endgroup$
    – Paul Siegel
    Dec 21, 2015 at 1:28
  • $\begingroup$ @PaulSiegel: thank you for the answer ! However, there is a simple metric topology which I think works (though, since I've forgotten my topology, I'm not completely sure what "making the map continuous" is exactly). That's the Wasserstein-1 metric. More generally, the Wasserstein-k metric (I think) works for all statistics with at most polynomial growth $\endgroup$
    – Guillaume Dehaene
    Jan 4, 2016 at 7:36

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge that you have read and understand our privacy policy and code of conduct.

Not the answer you're looking for? Browse other questions tagged or ask your own question.