Last Time:

    -    in general context of:

    -    revisited protocol background
 

    -  Mice and Elephants graphic, with "one minute split" flows
 

    -    introduced simple model
 

    -    investigated modelling assumptions

            -    SiZer analysis:  not quite homogeneous intensity

            -    QQ analysis:  not quite exponential interarrivals
 
 
 

(Soundbite level introduction)
 
 

References:
 

Resnick, S. I. (1987) Extreme values, regular variation and point processes, Springer.
 

Leadbetter, M. R., Lindgren, G. and Rootzen, H. (1983) Extremes and related properties of random sequences and processes, Springer.
 
 
 

Context:    For i.i.d. random variables 
 

with cumulative distribution function ,
 

study the asymptotic (as )  distribution of

Comments / Corrections welcome!
 
 
 

Analog:         Central Limit Theorem
 

describes the asymptotic distribution of the sample mean, as

Intuitive ideas:

    -    sample mean  "clusters around" population mean 

    -    gets closer as sample size grows

    -    gets closer at specific rate 

    -    precise "normalization" gives limiting distribution
 
 
 

Say "  is in the domain of attraction of the distribution "
 

when:
 

      there are sequences  and , so that
 
 

      as 
 
 
 

Limiting distributions, :
 
 

1.    "Extreme Value", i.e. "Gumbel" 

    -    roughly happens for   with "exponential tails"

    -    e.g. Gaussian, Weibull, log Normal

    -    could be bounded (but "little mass near end")
 
 

2.    "Weibull" 

    -    roughly happens for   with "polynomial tails"

    -    e.g. Cauchy, Pareto
 
 

3.    "Negative Weibull" 

    -    roughly "bounded from above"

    -    e.g. Uniform, negative of Exponential, Weibull or Pareto
 
 
 

Notes:
 

    -    can apply to minima as well as maxima, since

    -    which is useful for "inter-arrival times"
 

    -    since "time to next packet" is a minimum over flows
 

    -    thus expect Weibull distribution for inter-arrivals
 

    -    not all   have such a limiting distribution for 
 

    -    there exist precise mathematical characterizations
 

    -    elegant and fun mathematics
 
 
 

(careful for next time:
    need to rethink several things here:
    -    "usual beta parametrization" is "b-1"
    -    but probably this is better? (makes "pole" more clear)
    -    should show beta densities
    -    better (for later) to reformulate in terms of the min?
    -    part about "Weibull tails" is not right (since at min not tail)
    -    can be sorted out by Weibull graphic?
    -    need to be careful about scale
    -    perhaps do "median matching"?)

An example:    the beta  density:

    note:  allows study of range of "upper bound behavior"
 
 

Can show:    domain of attraction is

with shape parameter: 
 
 

Interesting cases:
 

:

        -    small near ,

        -    Weibull shape parameter is 

        -    i.e. has lighter tail than exponential

        -    sensible, since "few observations near "
 

:

        -   constant height near ,

        -    e.g. Uniform or negative of exponential

        -    Weibull shape parameter is 

        -    i.e. domain of attraction of exponential distribution

        -    special notes:

             -   this is interarrival times of Poisson Process

             -   other cases are departures from this
 

:

        -   has a pole (infinite peak) near ,

        -    Weibull shape parameter is 

        -    i.e. has heavier tail than exponential

        -    sensible since "more observations near "
 
 

2 b.    Weibull QQ, parameters est'd by quantile matching
 

Offpeak

    -    very good fit

    -    shape parameter 0.9 "close to Poisson 1.0"?

    -    provides a workable approximation????

    -    now understand potential mechanism:

    -    but "clustered Poisson" may be more likely??

    -    since can think of such a mechanism
 
 

Recall poor Q-Q analysis for:
 
 

Peak

    -    visual impression distorted by few large values

    -    very similar shape parameter 0.9
 
 

Fix for this (boundary effect) problem:
 

consider only flows that start after 0.2 of range:
 

Boundary adjusted peak

    -    now same result as off peak

    -    Weibull, with shape parameter ~0.9

    -    shows "very large values" before were at beginning

    -    can reject hypothesis of exponential (graphic)

    -    but still maybe OK as "first pass approximation"
 
 

An Open problem:  which of the following can explain the

(Recall graphics:   Off Peak               Peak)
 
 

a.    Independent Weibull(0.9) interarrivals?
 

b.    Poisson cluster process?
 
 
 

An aside on the  SiZer  analysis:

    -    downwards slope
 

Investigate by starting only after 20% of range:

Boundary Adjusted Peak

    -    looks much better

    -    confirms problem was mostly boundary effect

    -    but could still be some non-stationarity
 
 
 

interesting "physical explanation" of

from:
 

Downey, A. B. (2000) The structural cause of file size distributions, Wellesley College Tech. Report CSD-TR25-2000, http://rocky.wellesley.edu/downey/filesize
 
 
 

Main results:

    -    Studies distributions of file sizes

    -    On individual computers

    -    Claims generally log Normal

    -    not Pareto
 
 
 

Fundamental Premise:
 
 

Copying:

Translating and filtering:

Editing:

Major assumption:

    -    easy for copying: factor of 1

    -    very sensible for translating and filtering

    -    OK for most programming and word processing?

    -    wrong for:

            -    concatenated files

            -    text files "created from scratch"
 
 

Simple stochastic model:
 

for original file size 
 

and a random modification factor 
 

new file size is 
 
 
 

Distribution of ?
 

Perhaps doesn't matter by "Central Limit Theorem argument"

Downey's conclusion:
 

Since log normal is "light tailed"  (e.g. all moments exist)
 

Then don't have:

I.e. something else must be creating the apparent LRD
 
 
 
 

Can the lognormal lead to Long Range Dependence?
 
 

Simplest answer:

Deeper question:

A precise (actually not very!) question:  for "simple model",

    for what sequences of parameters   and ,

    do we get classical long range dependence

as ????