Last Time :

    -    Asymptotic Independence, using SiZer analysis

    -    relationship between Size, Time and "Rate" = Size/Time

    -    Saw hard to explain dependencies

    -    Introduced modification: "Large Variable Association"
 
 
 
 

Idea:

        variation of "Asymptotic Dependence"

        that is well defined for finite samples,

        not just in limit as 
 

Goals:
 

    1.    Indicate whether large values are

    2.    Reduce to classical Asymptotic Independence

Approach:
 

a.  For "properly adjusted marginals"

        (adjust extreme value distributions as above?)

        (surely "make scales comparable", e.g. divide by median)
 

b.  Consider "polar coordinates" versions of ,

where  and .
 

c.  Transfrom data to "angularly equally spaced",

    by replacing sorted s with an equally spaced grid

            (essentially Probability Integral Transform)
 

d.  Study association of "large values", by density of s

        where corresponding  is large (> threshold)
 

e.  Have "large variable association" when "pile up in middle"
 

f.  Have "large variable disassociation" when "pile up at ends"
 

g.  Use SiZer to study statistical significance
 
 
 

Toy Example 1:    5,000,000 absolute values of Standard Normal

Large Variable Association movie (through thresholds)

    -    As expected looks very uniform
 

Toy Example 2:    5,000,000 independent Pareto (1.5)

Large Variable Association movie (through thresholds)

    -    As expected shows strong Large Variable Disassociation
 
 
 

Toy Example 3:    5,000,000 independent Unif(0,1)
 

Raw data scatterplot, thresholded to 10,000

    -    Just have small corner left after threhsolding

    -    Opposite effect from Pareto

    -    I.e. anticipate strong Association this time
 

Angular equal spacing scatterplot

    -    above effects diluted only slightly

    -    since full data angles not far from uniform

    -    continue to anticipate strong Association
 

Large Variable Association movie (through thresholds)

    -    As expected shows strong Large Variable Association

    -    Interesting "triangular distribution"

    -    Shape theoretically predictable?
 
 

Toy Example 4:    5,000,000 absolute correlated Gaussian
 

Raw data scatterplot, thresholded to 10,000

    -    Have outer piece of parabola after threhsolding

    -    Opposite effect from Pareto

    -    I.e. anticipate strong Association this time
 

Angular equal spacing scatterplot

    -    this time get clear "spreading effect"

    -    but continue to anticipate strong Association
 

Large Variable Association movie (through thresholds)

    -    As expected shows strong Large Variable Association

    -    have a limiting Gaussian shape?

    -    Shape theoretically predictable?
 
 

Toy Example 5:    5,000,000 independent Exponential
 

Raw data scatterplot, thresholded to 10,000

    -    Surprising shape?

    -    Symmetric with respect to L1 unit ball?

    -    More sensible to measure "distance" using L1 norm?
 

Angular equal spacing scatterplot

    -    very little change

    -    slight "pinching towards center" (as expected)
 

Large Variable Association movie (through thresholds)

    -    As expected shows no Large Variable Association

    -    have a limiting Gaussian shape?

    -    Shape theoretically predictable?
 
 

Toy Example 6:    5,000,000 independent log-normal(5.28,2.46)

    -    Parameters from earlier "Response SiZe QQ fit"
 

Raw data scatterplot, thresholded to 10,000

    -    Similar lessons to Pareto

    -    Note single large observation (> twice others)
 

Angular equal spacing scatterplot

    -    very little change, as for Pareto
 

Large Variable Association movie (through thresholds)

    -    As for Pareto, shows no Large Variable Association
 
 
 

Further notes:

    -    main difference with "Asymptotic Independence" is no

                "extreme value normalization"

    -    makes Large Variable Association more sensible??

                (at least much easier to implement)

    -    what are drawbacks of this approach?

    -    when are these the same in ???

    -    perhaps in "heavy tailed case"?

    -    could use this as a definition of "heavy tailed"??

    -    how will things go with the real data?
 
 
 

Real Data Analysis:  Response Size Data
 

Large Variable Association Movies (through thresholds)
 

1.    Time vs. Size,

    Thursday afternoon (peak)

        -    Large Value Disassociated?

        -    Too many angles near 1??
 

    Thursday evening

        -    Again Large Value Disassociated?

        -    Again too many values near 1??
 

    Sunday morning (off peak)

        -    Mostly disassociated?

        -    But some "other peaks near 0 and 1"?

        -    Again too many values near 1?
 

2.    Rate (Size / Time) vs. Size, movie

    Thursday afternoon (peak)

        -    More strongly disassociated

        -    This time too many near 0??
 

    Thursday evening

        -    Diassociated?

        -    Even more near 0
 

    Sunday morning (off peak)

        -    Strong disassociation for larger threshold

        -    Too many near 0 for smaller threshold?
 

3.    Inverse Rate (Time / Size) vs. Time, movie

    Thursday afternoon (peak)

        -    Way too many near 1?

        -    Call this disassociation?
 

    Thursday evening

        -    Similar lessons, with spike not at 1?
 

    Sunday morning (off peak)

        -    Similar lessons
 
 
 

Overall Impressions:

    -    Poor job done of making "axes commensurate"

    -    Recall used extreme value rescaling before

    -    "Angular Prob. Int. Trans." hasn't solved this

    -    Disappointed that now "everything looks independent"

    -    Recall seemed found interesting associations earlier

    -    Were those "really there"???
 
 
 

What about other time blocks?
 

For threshold fixed at 200,

1.    Time vs. Size

2.    Rate (Size / Time) vs. Size

3.    Inverse Rate (Time / Size) vs. Time

    -    Lessons similar to the above
 
 

How to make axes more commensurate?
 

There are a number of choices, e.g.

    -    Rescale (e.g. by median)

    -    Prob. Int. Trans. on Angles

    -    Apply Prob. Int. Trans. to axes (Copula trans.)

    -    Prob. Int. Trans. on radii (polar coord. Copula?)

    -    mix and match
 

Problem:  hard to keep track of (understand) all of these
 
 

Approach:  richer visualization
Setting:  Thursday afternoon
Raw Data:
    -    Scatterplot shows size numbers far bigger than time
    -    log -log scatterplot show selection bias
    -    Hard question:  how to do better???
    -    Q-Q plots show axes far from commensurate
    -    not totally surprising, since units completely different
    -    all angles piled up on 0.
Median Rescaled:
    -    Now all on time axis, but one huge size
    -    single outlier has terrible effect...
    -    log-log Q-Q crosses diagonal at median
    -    does poor job making axes commensurate
    -    no multiplicative rescaling can help
    -    angular density surprisingly uniform
Median RS & Angular PIT.:
    -    quite similar to above
    -    expected since angles already reasonably uniform
Med. RS & A. PIT & top 200:
    -    discussed above
    -    still see axes not very commensurate
Copula Transformed (PIT on both axes):
    -    axes very commensurate
    -    especially clear in Q-Q plots
    -    angular density clustered around 0.5
    -    caused by mapping to unit square
    -    have more angles in "direction of corner"
    -    preferable may be "angular version of copula"
Copula and Angular PIT:
    -    not a large change is needed for this
    -    angles not extremely far from uniform
    -    some rotation away from diagonal
    -    note little distortion in marginal Q-Q plots
Copula & A. PIT & Top 200:
    -    Suggest Strong Association!?!
    -    Opposite of above conclusion
    -    Scatterplot is "outer rim" of above scatterplot
    -    all quite close to diagonal
    -    seems to "choose wrong part of data"
    -    sensible modification???
Angular PIT:
    -    some improvement over raw data
    -    but axes still non-commensurate
    -    except perhaps on log scale
Angular PIT & Radius PIT:
    -    best job of commensurate axes
    -    looks like useful "polar coordinate copula"?
A. PIT & R. PIT & top 200:
    -    chose "outer rim" of polar coord. copula
    -    more dense in some regions than others
    -    marginal Q-Q plots show axes commensurate
    -    SiZer map shows some disassociation
    -    Too many towards 0?
    -    i.e. still dominated by "large size"?
    -    But opposite direction from above
    -    Why is R. PIT needed, if only study angles?
A. PIT & top 200:
    -    same SiZer analysis (since same angles)
    -    but axes look much less commensurate??
    -    something wrong with this "commensurate" idea????
    -    this analysis better than "median rescaled" above???