Approximate randomization tests

[Jan Šnajder]

Hi Daniël,

Wow, that's a lovely coincidence! I need to do stratified shuffling right now, wanted to do it in Haskell, and was about to implement this on my own, but then I got this. Great work, thanks! :-)

Does anyone know if it is appropriate to use stratified shuffling in the multiple-classifiers single-domain setting? I'm worried about the well-known problem associated with repetition of pairwise tests: the increased chance of committing type I error. ANOVA or Bonferroni adjustment is the common way to address this. How does one deal with this in stratified shuffling?

Best,
Jan

> Hi again,
> 
> On Apr 3, 2012, at 2:20 PM, Dani?l de Kok <me at danieldk.eu> wrote:
> > To test the significance in the evaluation of (amongst other
> > things) parse disambiguation and fluency ranking, I developed a
> > module and two utilities for performing approximate randomization
> > tests:
> > 
> > https://github.com/danieldk/approx-rand-test
> 
> Months ago, I also added support for drawing histograms that show the
> score frequencies of the randomized samples, the statistic for the
> samples and the significance boundaries:
> 
> http://danieldk.eu/Software/art-hist.png
> 
> This requires Cairo and can be enabled with:
> 
> cabal install -fwithCairo
> 
> The 'w' option can then be used to write the histogram to a file.
> There is also an option (-h) for making ASCII art histograms, which
> are a bit less pretty ;):
> 
> http://www.let.rug.nl/~dekok/misc/art-histogram.png
> 
> -- Dani?l