Ratio Scientiae
RATIO SCIENTIAE BLOG
  • Home
  • Meet the Author
  • Ratio Scientiae Blog
  • Random Science
  • Writing & News
  • Nonfiction Books
    • Science Can Be Right Because It Can Be Wrong
    • The Gift of Science
    • Random Science
  • Fiction Books
    • The Sun Zebra
    • Spirit Women
  • Science Cat (Mascot)
  • Contact

10/25/2018

It’s Statistically Significant! But, Does (Sample) Size Matter?

0 Comments

Read Now
 
Picture
How many times have you read about the results of studies that were claimed to be or not be statistically significant? How many times have you read that the frequency of some occurrence was found or not found to be statistically significantly above chance? It’s like if the phrase “statistically significant” is considered some sort of seal of legitimacy. If it is statistically significant, it must be true, no? And if it is not statistically significant, it must be false, right?
 
The truth, of course, is more complex than that. Let’s address one issue that invalidates many studies regardless of whether their results are statistically significant or not: sample size.

Studies conducted with a small sample size, if not carefully designed, are very prone to variability. A very big red flag is when the results of a small study are unexpected. Studies with a small sample size can produce false results either in favor or against the premise that they are addressing, although most of the time they fail to produce statistically significant results for an effect, even if the effect is real. This sample size issue is often not reported when the study is presented in the news or, if reported, it normally goes something like this:
 
The authors caution that this was a study with a small sample size but (there is always a “but”) if this result can be repeated in a larger study then…
 
On the one hand, the authors acknowledge that the results of the study may be bogus due to the small sample size, but on the other hand the authors nevertheless find the results worthy enough to publicize or to acquiesce to the desire of journalists to publicize them!
 
So, are all studies with a small sample size bad, and should be avoided?
 
Not necessarily. Sometimes scientists perform small studies intended to give the issue being tested a “look-see”. Sometimes studies with small sample sizes are all that is possible to do due to budgetary constraints or other constraints such a working with a rare disease. Scientists also perform small studies to gain experience in order to later design larger studies. But there is another reason a study with a small sample size may be justified. Consider the following statistical joke:
 
A scientist asks a group of 3 students to concentrate in teleporting through a solid wall. Of these 3 students one manages to accomplish this feat. Excitedly the scientist informs a colleague of the results of the experiment, but his colleague dismisses it outright arguing that with a sample size of 3 you cannot possibly obtain any statistically significant results.
 
The premise of this joke is that one person achieving such a feat would be enough. No further sample (or even statistics for that matter) is required! This is intended to illustrate that the nature of the effect being evaluated is important. If scientists are only interested in detecting a very large effect, then a study with a small sample size may be totally justified.
 
But you may argue that a study with a large sample size is always more desirable, no? After all, you can’t go wrong if you perform a study with a large sample size, right?
 
Not only are studies with large sample sizes more expensive, time-consuming, and difficult to implement, but there is a seldom discussed and surprising downside to large samples sizes illustrated in the following example:
 
A researcher spends several years performing trials to evaluate whether human subjects can mentally affect the outcome of the toss of a coin as simulated by a computer. After accumulating and analyzing hundreds of thousands of trials, the researcher finds that human subjects can influence the computer mentally to produce heads as oppose to tails with a frequency of 50.0001%, and that this effect is statistically significantly above mere chance.
 
The problem outlined in the example above, is that using a very large sample size you can detect anything, including random background fluctuations or equipment calibration imperfections, as statistically significant. Detecting such a small effect, regardless of whether it is statistically significant or not, may not only be meaningless, but is also devoid of any practical significance.
 
The dirty little secret of statistical analysis is that statistical significance cannot replace good judgement. Before a study, you have to ask what is the size of the effect that you want to detect and whether it would be of importance to detect an effect of that magnitude, if indeed it is present. These questions will lead to determining the correct sample size for the study, and in fact whether the study should be performed at all!
 
To recap and answer the question posited in the title of this post: in statistics whether sample size matters or not depends on the magnitude and importance of the effect being evaluated.
 
1) The detection of an important small effect may require a study with a large sample size.
 
2) The detection of an important large effect may be achieved with a carefully designed study employing a small sample size.
 
3) The detection of a small effect with a large sample size may be irrelevant if the effect is not important, regardless of statistical significance.

4) The detection of an effect with a small sample size in a study not carefully designed is likely to be a happenstance occurrence, regardless of statistical significance.
 
So next time you hear about whether something was statistically significant, inquire about sample size.

 
Image by Nick Youngson used here under an Attribution-ShareAlike 3.0 Unported (CC BY-SA 3.0) license.

Share

0 Comments
Details

    Categories

    All
    911
    Absence Of Evidence
    Adrenaline
    Adrenochrome
    Advances In War Medicine
    Affidavits
    Airplane On Conveyor Belt
    Alder's Razor
    Alfred Wegener
    Alzheimer's Disease
    Amyloid Theory
    Ancient Astronauts
    Animal Rights
    Animals
    Anthony Fauci
    Anthropomorphism
    Antibodies
    Anti Vaccination
    Astronauts
    Authority Figure
    Autumn
    Bambi
    Believers
    Bias
    Big Bang Theory Sitcom
    Bigfoot
    Bill Ney
    Black Death
    Black Holes
    Blind Experimental Design
    Brain
    Brains
    Building 7
    Buoyancy
    Cancer
    Cannonballs
    Carnivores
    Catoctin Mountains
    Cell Culture
    Center Of Mass
    Challenger
    Chance
    Chemical Names
    Chemical Reactions
    Chemtrails
    Chesapeake Bay
    Child Abuse
    Child Testimonies
    Chimborazo
    Citations
    Citogenesis
    Civil War
    Climate Change
    Climategate
    Climate Skeptics
    Clinical Trial
    Coelacanth
    Coincidence
    Cold-Stress
    Communicating Science
    Completeness Of Scientific Theories
    Consciousness
    Conspiracy
    Contrails
    Controls
    Coquina Rock
    Coronavirus
    Counterintuitive Facts
    COVID19
    COVID 19 Vaccine
    COVID-19 Vaccine
    Creationism
    Cross-Sectional Method
    Cryptozoology
    Dan Shechtman
    Demonic Possession
    Density
    Devotion To Science
    Dihydrogen Monoxide
    Dinosaurs
    Discrimination
    Disgust
    Dishonesty In Science
    Diversity In Methodologies
    Diversity In Scientists
    DNA
    Doctors
    Dog Experiments
    Donald Trump
    Double Slit Experiment
    Do Your Own Research
    Dreams
    Dr. Gloom's Crypt Of Curiosities
    Drugs
    Dust Bunnies
    Eclipse
    Efrain Racker
    Einstein
    Elections 2020
    End Of The World
    Erosion
    Established Science
    Everest
    Evidence
    Evolution
    Exorcism
    Experimenter Bias
    Experiments You Can Do In Your Home
    Experts
    Extinction
    Face Masks
    Facts
    Faith
    Faith Healing
    Fall
    Falsifiability
    Fantastical Claims
    Fauna
    Fear
    Feynman
    Finger Snapping
    Fosbury Flop
    Francis Crick
    Frankenstein
    Fraud
    Frederick's Municipal Forest
    Free Will
    Friction
    Fruit Fly
    Funerals
    Funny
    Galaxies
    Genes
    Genius
    Geology
    Ghost
    Global Warming
    Global Warming Denial
    God
    Gravitational Lens
    Gravitational Waves
    Gravity
    Gun Violence
    Hanlon’s Razor
    Hearing
    Hearing Aids
    Hearing Loss
    HeLa Cells
    Henrietta Lacks
    Heroic Science
    Hitchens's Razor
    Hoax
    Homosexuality
    HPV Vaccine
    Human Experimentation
    Human Folly
    Hunting
    Hurricane
    Hydroxychloroquine
    Hypothesis
    Ignaz Semmelweis
    Ignorance As Evidence
    Immunotherapy
    Infamy Or Glory
    Influenza
    Insults
    Intelligence
    Intelligence Tests
    Intelligent Design
    Intersex
    Iraq War
    Irish Washerwoman
    Isaac Asimov
    James T. Kirk
    James Watson
    Katherine Hayhoe
    Language
    Lazarus Effect
    Left Fork Rocks
    Level Of Detail
    Libet Experiment
    Lightning
    Limericks
    Limitations Of Science
    Loch Ness Monster
    Lord Howe Island Stick Insect
    Loud Music
    Luck
    Mad Scientist
    Magnus Effect
    Malaria Vaccine
    Max Planck
    Medical Risks
    Medical Terms
    Mediums
    Men
    Mental Illness
    Mice
    Microbiome
    Milgram Experiment
    Mind
    Mind In The Gutter
    Misuse Of Science
    MMR Vaccine
    Mnemonic Devices
    Moments Of Discovery
    Monty Hall Puzzle
    Moon
    Movies
    NASA
    Natural
    Nature
    Nature Of Science
    Negative Evidence
    Nerds
    Newton
    Nobel Prize
    NOMA
    N-Rays
    Obesity
    Occam's Razor
    Open Mind
    Ouija Board
    Outreau
    Oxbow Lakes
    Para-Dimethylaminobenzaldehyde
    Paranormal
    Pasteur Louis
    Peak Of Illusion
    Peers
    Perception
    Philosophy
    Phobias
    Phrenology
    Physics
    Pink Lady's Slippers
    Plague
    Politicization Of Science
    Polywater
    Popper’s Falsifiability Principle
    Possible/Impossible
    Power To The People!
    Predatory Journals
    Pregnancy
    Premonitions
    Probability
    Propaganda
    Prophesy
    Pseudoscience
    Psychic
    Publication
    Puerperal Fever
    QAnon
    Quack
    Quackery
    Quantum Mechanics
    Quartzite
    Racism
    Radical New Ideas
    Radioactivity
    Radium
    Randomness
    Ratio Sapientiae
    Ratio Scientiae
    Reality
    Reason
    Religion
    Rock Climbing
    Rosalyn Franklin
    RSV Vaccine
    Sagan's Standard
    Salem Witch Trials
    Sample Size
    Science Fiction
    Science Jokes
    Science Pranks
    Scientific Consensus
    Scientific Establishment
    Scientific Guidelines
    Scientific Journals
    Scientific Method
    Scientific Names
    Scientific Question
    Scientific Terms
    Scientific Theories
    Seashells
    Self-Experimentation
    Shape Of The Earth
    Skepticism
    Skeptics
    Skin Color
    Slavery
    Snake Oil
    Soccer
    Sonic Hedgehog
    Space
    Spanish Flu
    Spontaneous Generation
    Stanley Prusiner
    Stars
    Star Trek
    Statistical Significance
    Statistics
    Sublime/Ridiculous
    Suicide
    Sun
    Superstition
    Surface To Volume Ratio
    Swanson Conversion
    Swine Flu
    Tangier Island
    Taxonomy
    Technical Details
    The Support Of God
    Thomas Young
    Timeline Method
    Tooth Worm
    Transposable Elements
    Trusted Messenger
    Trust In Scientists
    Truth
    Tyranny Of Fantasy
    Understanding
    Universe
    Vaccine Hesitancy
    Vaccines
    Vacuum
    VAERS
    Valley Of Despair
    Values
    Video Games
    West Side Story
    West Virginia Penitentiary
    Wikipedia
    William Shatner
    Wolf Rock
    Women
    World Trade Center
    Xenophobia
    Y Chromosome
    Zinc
    Zombies

    Archives

    October 2024
    September 2024
    August 2024
    July 2024
    June 2024
    May 2024
    April 2024
    December 2023
    November 2023
    October 2023
    September 2023
    May 2023
    April 2023
    March 2023
    February 2023
    January 2023
    November 2022
    October 2022
    September 2022
    August 2022
    July 2022
    June 2022
    May 2022
    April 2022
    August 2021
    July 2021
    June 2021
    May 2021
    April 2021
    March 2021
    February 2021
    January 2021
    November 2020
    October 2020
    September 2020
    August 2020
    July 2020
    June 2020
    May 2020
    April 2020
    March 2020
    February 2020
    January 2020
    November 2019
    October 2019
    September 2019
    August 2019
    July 2019
    June 2019
    May 2019
    April 2019
    March 2019
    February 2019
    January 2019
    December 2018
    November 2018
    October 2018
    September 2018
    August 2018
    July 2018
    June 2018
    May 2018
    April 2018
    March 2018
    February 2018
    January 2018
    December 2017
    November 2017
    October 2017
    September 2017
    August 2017
    July 2017
    May 2017
    April 2017

    RSS Feed

Proudly powered by Weebly
  • Home
  • Meet the Author
  • Ratio Scientiae Blog
  • Random Science
  • Writing & News
  • Nonfiction Books
    • Science Can Be Right Because It Can Be Wrong
    • The Gift of Science
    • Random Science
  • Fiction Books
    • The Sun Zebra
    • Spirit Women
  • Science Cat (Mascot)
  • Contact