Last Time :

    -    Large Variable Association

    -    Found way to make axes "commensurable"

    -    Main Problem:  tail indices not right?
 
 
 

Recall Main Goal (from 10/22/01 and 10/24/01) :

Find "more realistic" version of

Motivating data:    HTTP Response Data, from UNC, 2001
 

Recall log-log CCDF: graphic

Suggests classical extreme value theory "hasn't kicked in yet"
 

Surprising point:

Suggests "systematic", not "noise driven" behavior
 
 
 

A weakness of above analysis:

    (since worked with summarized data)
 

Consequence:  for n ~ 1,000,000 - 7,000,000,

Improved version:  "insert" raw data values, graphic

    -    chunkiness is gone

    -    distributional shape still surprisingly constant

    -    still suggests systematic nonlinear behavior

    -    still clear "classical tail theory" doesn't apply
 
 
 

Suggests need to update concept of:

since "classical" notions of "heavy tails" don't apply.
 

Recall earlier approach:

    -    based on difference quotients (truncated below)

    -    zero denominators caused problems

    -    but clear visual impression of "usually in heavy tail range"

    -    suggests possibility of updating above theory
 
 
 

Improved version of slopes analysis:

    -    "0 denominator" problem becomes much worse

    -    hence tried "grid based difference quotients"

    -  still "feels noise"

    -  has "smoothed out noise"

    -    general shape is same for both

    -    same lesson as before about effective tail indices

    -    still seems worth updating above theory
 
 
 

Provides new motivation for definitions of "heavy tails":

Version 1:    For some ,

Which results in:

Open Problem 1:    For the simple Model (recall from 10/1/01),

with Version 1 tailed Duration Dist'n,   is

?

(i.e. Version 1 heavy tails results in index   LRD?)
 
 
 

Note:  Version 1 is not "completely right",

    but instead only holds "most of the time"
 

Improvement:  modified definition of "heavy tailed"
 

Version 2:    Reformulate, in terms of:    have

Open Problem 2:    For the simple model (recall from 10/1/01),

with Version 2 tailed Duration Dist'n,

can we still have    (in a suitable sense)?
 

How do we modify version 2 to make this happen?
 
 
 
 

Can lognormal durations lead to Long Range Dependence?
 

Simplest (widely held) answer:

Deeper question:

Who cares?

    -    Common misperception about "infinite moments"
 

A precise question:  for a sequence of "simple models",

    with log normal (parameters   and ) durations,

    under what conditions (on   and ) does classical L.R.D.

    result, as ????
 
 
 

Hannig, J., Marron, J. S., Samorodnitsky, G. and Smith F. D.

(2001) "Log-normal durations can give long range dependence".
 

For a sequence of "simple models" from Lecture 10-1-01,

    -    indexed by 

    -    Continuous time (simpler than discrete?)

    -    Homogeneous Poisson() "starting time" (for flows)

    -    Draw independent "duration time" (for each flow)

    -    Define    =  number of "active flows"
 

Choose the durations to be log-normal (,), and choose

    -       (growing intensity)

    -          (lognormal mean goes 0)

    -        (lognormal "spreads")
 

Note that for each  ,  the durations are "light tailed"

but "compressed towards 0", and "variance expanded",

so in , get "a few large values" and "many small values"
 

Is growing intensity "objectionable"?

    -    by some standard measures

    -    but internet traffic is growing rapidly...

    -    why do asymptotics?

            -    limiting process models something?    No!

            -    because everybody always has?    No!

            -    to illuminate important, underlying structure?    Yes!
 
 
 

Now measure "Long Range Dependence" through covariance:

Main result:  Given  and  ,

for any sequence    with  :

Intuitive Idea:   for a "wide range of lags ",

the autocovariance   

has a "polynomial rate of decay"  ,

which is a "common symptom of long range dependence"
 
 
 

Recall Mice and Elephants:

    -    Aggregation of Mice    short range indepen’ce

    -    Aggregation of Elephants     long range depen’ce
 

Deep question:  which will “eventually dominate”?

    -    Cleveland:   theory for mice swamping elephants

    -    Others (e.g. me):   elephants will continue to grow
 
 
 

Future Major Players(???):

    -    Strong “Quality of Service” Demands

    -    Big $$$$ behind this
 

Industry response???
 

Short term future:

    -    Internet Service Providers:

        -    Financial incentive for over-provisioning
 

Long term future(????):  Interesting competition between:

    -    Above Overprovisioning

    -    "Differentiated service"

    -    intermediate solutions
 
 
 
 

Ancient Indian Story:

 View 1:     It is like a tree trunk …

 View 2:     It is like a rope …

 View 3:     It is like a long hose …

 View 4:     It is like a large fan …
 
 
 

Cornell Evaluations
 
 

Q1:    Html lectures, vs. Power-Point?
 

Q2:    How to encourage more student involvement?
 
 
 
 
 
 
 
 
 

Topics never covered:

Stationarity?  (scale based?)  (paper from Richard)
Cleveland & Cao and Ramanan asymptotics & mice vs. elephants
New "big trace" analysis
Shared Fate Visualization
Functional Data Analysis of Traces