Last Time:
 

    -    Finished (?) study of heavy tails

    -    Began study of "Long Range Dependence"

    -    via correlation analysis   (sensible??)

    -    in context of:

Heavy tailed durations   Long Range Dependence

Course Goals:
 

    -    Explore Internet Traffic from several viewpoints
 

    -    Highlight interesting open problems
 

    -    Promote possible joint research
 

    -    Maximize understanding by all class members
 
 
 
 
 
 
 
 
 
 

(Sound Bite)  Introduction to Time Series Analysis
 

    -    Autocorrelation

    -    ARMA process

    -    Periodogram

    -    Partial Correlation

    -    ARIMA processes

    -    Long Range dependence

    -    Fractional ARIMA processes
 
 
 
 

Question 1:    Is it really there?
 
 

    -    Early conceptions:   no

    -    Current thought:   yes
 

    -    Very recent work (Cleveland, et. al.):    not important
 

    -    Motivated zooming autocorrelation view.
 

    -    Revealed "both viewpoints correct" depending on scale
 

    -    Surprised at "how dependence comes in"?
 

    -    Expected "lump of dependence" coming in from right??
 
 
 
 
 
 

Notion of large lump on right (in autocorr.):

Caution 1:    periodicities    large lump,

Caution 2:    TCP has its periodicites

A tempting idea:

BUT:  aggregated data say something different
 
 

AN EXPLANATION:  depends on where loss occurs:

    -    loss at link where measuring?    then YES

    -    far away from measurement point?    then NO
 
 

Recall simple view of the Internet:

Current situation:
 

    -    Backbone is "over-provisioned"

    -    Loss occurs mostly at "edges"

    -    Thus aggregation of these could be independent

Observed effects due to data sparsity?
 
 

Time Series’s 2:    For increasing seq’s of 10,000 bins

    -    time scales 

    -    # obs’s / bin 

    -    total length 
 
 

Major Problem:      assumes “stationarity”
 
 
 
 
 

Zooming Autocorrelation 2:

    -    can’t distinguish from indep. at small scales

    -    strong dependence at larger scales

    -    “vertical lifting of dependence”

    -    not “coming in from right”
 

Questions:

    -    looking at too narrow a lag range?

    -    where are “times” in zooming auto-correlation?
 
 
 
 

Zooming Autocorrelation 3:

Larger lag range  &  “time markers”
 

    -   cyan bar shows old lag boundary
 

    -   yellow bars show how time zooms
 

    -    vertical lift not completely level
 

    -    but still doesn’t “move in from right”
 

    -    instead “lifts first on left”
 
 
 
 

Zooming Autocorrelation 4:    Time invariant view

    -    expect “curve follows mountains of dependence”
 

    -    from “dependence at time scale” model
 

    -    instead see “dependence increasing with scale”
 
 
 
 
 
 

Hannig, J., Marron, J. S. and Riedi, R. H. (2001)  Zooming statistics: Inference across scales, Journal of the Korean Statistical Society, 30, 327-353.  Go here to download.
 
 

Idea:  Compare autocorr’n when adjacent bins are combined:
 

Relate lag   at scale          to lag   at scale 
 

Can show:

Notes:

   -   when really uncorr’d, always stays at 0

   -   slight positive autocorr. Magnified by 2

   -   big lift for small lag one autocorr.

   -   small lift for large lag one autocorr.

   -   small scale Poisson model is not correct

   -   but still OK as a fine scale approximation???
 
 
 
 
 
 

Idea:  Study "dependence" in terms of

Recall SiZer finds "significant slopes"
 
 
 

Need for zooming:  to view wide range of scales
 
 
 
 
 
 
 

    -    settings: scatterplot smoothing and histograms
 

    -    Fossils data
 

    -    Incomes data
 

    -    Central Question:

    -    Solution Part I, Scale Space
 

    -    Solution Part II, SiZer
 
 
 
 
 

Smoothing Setting 1: Scatterplots
 

E.g.  Fossil Data
 

    -    from T. Bralower, Dept. Geological Sciences, UNC
 

    -    Strontium Ratio in fossil shells
 

    -    reflects global sea level
 

    -    surrogate for climate
 

    -    over millions of years
 
 
 
 
 

Smooths of Fossil Data (details given later)
 

    -    dotted line: undersmoothed (feels sampling variability)
 

    -    dashed line: oversmoothed (important features missed?)
 

    -    solid line: smoothed about right?
 
 
 

Central question: Which features are “really there”?
 
 
 
 
 

My scatterplot smoothing method (others disagree):

Main idea: (illustrated by toy example)

Window Width, h, is critical
 
 
 
 
 

Smoothing Setting 2: Histograms
 

Family Income Data: British Family Expenditure Survey, 1975

    -    Distribution of Incomes

    -    ~ 7000 families
 
 

Histogram Analysis:

    -    Again under- and over- smoothing issues

    -    Perhaps 2 modes in data?

    -    Histogram Problem 1: Binwidth (well known)
 
 

Central question: Which features are “really there”?

Why not use (conventional) histograms?
 
 

Histogram Problem 2: Bin shift (less well known)

    -    For same binwidth

    -    get much different impression

    -    by only “shifting grid location"
 
 

Solution to binshift problem: average over all shifts

    -    1st peak all in one bin: bimodal

    -    1st peak split between bins: unimodal
 
 

Another name: Kernel Density Estimate
 
 
 
 
 

Kernel density estimation
 
 

View 1: Smooth histogram
 
 

View 2: Distribute probability mass, according to data
 
 

E.g. Chondrite data (from how many sources?)
 
 
 
 
 
 

Kernel density estimation (cont.)
 

Central Issue: width of window, i.e. “bandwidth”,  h

Old Approach: data based bandwidth selection

Jones M. C., Marron, J. S. and Sheather, S. J. (1996) A brief survey of bandwidth selection for density estimation, Journal of the American Statistical Association, 91, 401-407.
 
 

New Approach: "scale space" (look at all of them)
 
 
 
 
 
 

“Scale Space” – idea from Computer Vision
 
 

Conceptual basis:

    -    Oversmoothing = “view from afar” (macroscopic)

    -    Undersmoothing = “zoomed in view” (microscopic)
 
 

Main idea: all smooths contain useful information,