The Reproducibility Crisis in Sports Science: One Study Doesn't Settle It—How Coaches and Athletes Should Read the Literature

That Day, a Trainee Rushed in Holding Their Phone

In all my years of coaching, the question that stumps me most isn’t “Coach, how do I get stronger?”—it’s questions like this: “Coach, I read a study yesterday saying that drinking beetroot juice before a race can boost power output by 3%. Should I down a glass every day?” Or “A paper said low-carb diets can massively increase fat utilization. Should I cut out rice?”

Once, a track time trial trainee, A, excitedly forwarded me a post titled “Latest Research Confirms: This Training Method Boosts Endurance by a Shocking 20%.” He was already planning to scrap his entire training block and start over. I asked him to take a deep breath, then posed three questions: How many people were in this study? Are those people anything like you? Has any other lab produced the same result? He froze, because he’d only read the headline and that “20%.”

This article is about a topic that sounds very academic but is actually deeply relevant to anyone who trains seriously—the reproducibility crisis in sports science. This isn’t about telling you to stop trusting science and go back to training by feel. Quite the opposite: precisely because we respect evidence, we need to know how much weight a single study can bear, and how to read research as a training aid, not a shackle that yanks you around with every new headline.

What Is the “Reproducibility Crisis”

Let’s start with an event that shook the entire scientific community. In 2015, a large-scale replication project in psychology (the Reproducibility Project: Psychology) attempted to redo 100 published psychology experiments, using designs with high statistical power. What happened? In the original studies, a whopping 97% reported “statistically significant” results. But after replication, only about 36% of the replicated experiments achieved statistical significance; moreover, the effect sizes calculated from the replications were, on average, only about half of the original studies.

This is called a “crisis” because one of the foundations of science is reproducibility: if a finding is real, then others following the same methods should be able to see similar results. When a large batch of “published, seemingly certain” conclusions can’t be reproduced in others’ hands, we have to ask: how much of these conclusions is a true signal, and how much is just noise, luck, or bias in the research process itself?

Let me also clarify two terms that are often conflated. “Reproducibility” usually means: using the same raw data and running the same analysis, can you get the same numbers—this tests the transparency of analysis and reporting. “Replicability” means: with a new set of subjects and newly collected data, can you see results in the same direction—this tests whether the finding itself is robust. Both matter, and sports science faces challenges on both fronts: some studies don’t even make their raw data available (poor reproducibility), and some disappear when redone with different people (poor replicability). You don’t need to memorize these two terms; just remember the core idea: if something is real, it can withstand being tested again by others in different ways.

What About Sports Science? Don’t Think We’re Immune

Many people think: that’s a psychology problem. Sports science measures hard physiological numbers like power, heart rate, and lactate—surely it can’t have this problem? Unfortunately, sports and exercise science is not exempt either.

A recent attitude survey of sports and exercise science researchers (with 511 respondents) found that about 42% believed the field has a “significant” reproducibility crisis, and another ~36% believed there is a “slight” crisis. In other words, nearly eight in ten researchers in the field feel the problem is real.

Even more telling are the actual replication attempts. Sports and exercise science long lacked the large collaborative replication projects that psychology and cancer biology had, and only in recent years have people started doing this work. In one large replication study, after redoing the original studies, about 56% of the replication results were consistent with the original in terms of null hypothesis significance testing, but only about 28% were judged as truly “successful replications” (reaching statistical significance with compatible effect sizes).

That 28% isn’t meant to make you despair of sports science; it’s meant to build a healthy expectation: any single study—especially one with few participants and particularly “pretty” results—carries inherently high uncertainty. It’s a clue, not a verdict.

Worth noting: the people running these replication projects also hit a very practical obstacle—many original studies reported incomplete statistical information, and the raw data was unavailable; some even hit a wall when contacting the original authors for help, with them unwilling to cooperate with replication efforts. This itself reflects a cultural problem in the field where “replication” has long been undervalued—everyone wants to grab the “new discovery,” and few are willing to spend time verifying whether “old findings” actually hold up.

What This Crisis Is NOT Saying

I need to be clear so you don’t swing to the other extreme. The reproducibility crisis is not saying “sports science is all lies,” nor is it telling you to go back to pure trial-and-error by feel. Quite the opposite:

• It’s not saying all research is wrong; it’s saying the certainty of any single study is overestimated.
• It’s not telling you to distrust science; it’s telling you to trust “a body of consistent research” more than “one lone study”.
• It’s not bad news; it’s a healthy sign of science examining and correcting itself—a system that catches its own flaws is one worth trusting long-term.

Keep these three points in mind, and you won’t fall into nihilistic cynicism just because you read the word “crisis.”

Why Does This Happen? How Weak Evidence Grows

To learn how to read research, you first need to understand why studies “fail to replicate.” Here, in plain language, I’ll break down several pitfalls sports science is especially prone to.

One: Small Samples, Luck Calls the Shots

Sports science experiments are expensive and exhausting. Recruiting people to the lab, drawing blood, measuring VO₂, pushing to exhaustion—it often takes weeks. So many studies only include 8 to 15 subjects. With small numbers, one or two “super responders” can drag the group average into a pretty number, making an intervention look effective when it might just be sampling luck.

I often tell trainees an analogy: if you flip a coin 10 times and get 7 heads, you wouldn’t claim the coin is biased; but many small studies take “7 heads out of 10” as evidence and write it up as “this coin is significantly biased toward heads.”

Two: Only the “Good-Looking” Results Get Published (Publication Bias)

Journals, media, and social media all favor stories of “effective,” “significant,” “breakthrough.” An experiment that finds “no difference” often gets shoved in a drawer and never published (the file-drawer effect). The result: the studies you see online are a filtered, good-news-biased subset, while those quietly showing “it doesn’t actually work” are invisible to you.

Three: Too Much Analytical Freedom (p-hacking)

The same dataset can be sliced many ways: Should we exclude this outlier? Look at average power or peak power? Split by sex or not? If you’re willing to try enough slicing methods, one will “just happen” to produce p < 0.05. This practice of fishing for significance and stopping when you find it churns out a mountain of conclusions that look solid but are actually fragile.

Four: The Subjects Aren’t Like You

Many studies recruit “young, healthy, trained male college students.” But if you’re a 50-year-old commuter just starting cycling with mild hypertension, that beetroot juice study done on kinesiology students applies to you with far less reliability. Generalizability (whether it extends to you) is the most overlooked yet most critical link. Incidentally, past sports science research has predominantly used male subjects; women, older adults, and chronic disease populations have long been underrepresented, which means conclusions applied to these groups should inherently be treated with more caution.

Five: Measurement Itself Has Error

Even with a flawless study design, many things sports science measures are inherently “jittery.” The same person’s VO₂, lactate, and power output vary day to day due to sleep, diet, caffeine, room temperature, and mood. When the effect you’re trying to detect is only 1–2%, but the measurement tool’s day-to-day variability might be 3–5%, then the difference you see is likely measurement noise, not a real intervention effect. This is why small-sample, short-term studies are especially unreliable.

Six: “Novelty Effects” and “Hawthorne Effects”

Once people know they’re being observed, or are trying some “new method,” they unconsciously put in more effort and focus, so short-term results naturally improve. This improvement is often mistakenly credited to the “new method” when it’s really just you trying harder. This is why a study without a control group (or your own attempt without a comparison) tends to overestimate effects.

The table below is a quick reference I use to teach trainees how to identify “evidence strength.” You don’t need to be a scientist—just check a study against it when you see one.

Indicator	Weaker Evidence (Treat with Caution)	Stronger Evidence (More Trustworthy)
Number of subjects	Single digits to a dozen or so	Dozens or more, or multi-center
Study design	Single-group pre/post, no control	Randomized, placebo-controlled
Replication status	Only this one study says so	Multiple independent teams with consistent direction
Effect size	Implausibly large (e.g., “20% boost”)	Modest and reasonable (e.g., 1–3%)
Subject population	Very different from you (age/level/sex)	Population similar to yours
Data transparency	Raw data not public, vague methods	Pre-registered, data and methods open
Publication format	Press release, product placement only	Peer-reviewed journal, systematic review

So How Should I View “a Single Study”

This is the most practical part. I’ve condensed the judgment process I’ve developed over years of coaching into a set of principles. The core is one sentence: a single study “proposes a hypothesis”; multiple consistent studies “confirm a conclusion.”

Principle One: First Ask “Has Anyone Else Said This”

When you see a striking conclusion, don’t rush to change your training plan. First ask: is this an isolated case, or have many independent teams already found the same direction? If you can find a systematic review or meta-analysis on the topic, that document integrating dozens of studies carries far more weight than any single new paper. Practically, treat a meta-analysis as “the whole forest” and a single study as “one tree”—you wouldn’t declare the whole forest fallen because you saw one crooked tree.

Principle Two: The More Exaggerated the Effect, the More Skeptical You Should Be

Genuinely solid sports science interventions usually have modest effects. For endurance performance, if a training or nutrition strategy can reliably deliver a 1–3% improvement, that’s already remarkable at the competitive level. So when you see headlines like “20% boost” or “double the performance,” your first reaction shouldn’t be excitement but alarm: if the effect were that large, why hasn’t it been discovered in the past few decades? It’s likely small-sample luck or clickbait exaggeration.

Principle Three: Separate “Statistical Significance” from “Meaningful to You”

“Statistically significant (p < 0.05)” only means “this result is unlikely to be pure coincidence.” It does not tell you whether the effect is large enough or worth changing your life for. A study can produce a “significant” effect so small you’d never feel it in real training. Learn to ask: even if it’s real, is this 0.5% difference worth an extra 30 minutes a day or spending money on supplements?

Principle Four: Distinguish “Mechanistic Studies” from “Performance Studies”

Many studies are done at the cellular level, on muscle biopsies, or on acute physiological responses (mechanistic). This is valuable, but “a substance activates a signaling pathway in muscle” does not equal “your race performance will improve.” Between mechanism and real-world performance lies a vast distance. When you see “this ingredient promotes mitochondrial biogenesis,” don’t jump to “so I’ll get faster.”

The table below is what I use to communicate with trainees about how to act at different evidence levels.

Evidence Level	Typical Form	Recommended Action
Single small study / press release	A dozen people, one team, striking effect	Treat as an interesting clue; observe first, don’t change your plan
Several studies in the same direction	Multiple teams, consistent results	Can try it on a small scale, low risk
Meta-analysis / systematic review	Integrates dozens of studies, with confidence intervals	Can incorporate into training principles, still individualize
Professional organization evidence-based guidelines	Society consensus, regularly updated	Use as the backbone of default practices
Your own long-term records	Your data, your responses	Final arbiter, overrides all averages

Note the last row. This is what I want to emphasize: no matter how good the research, it gives you “group averages,” and you are a specific individual. A study saying a strategy works on average doesn’t mean it works for you; a study saying no average difference doesn’t mean it won’t work for you as a “super responder.” Your own carefully recorded training data, heart rate, feelings, and race results are first-hand evidence no paper can replace.

Reading a Study Abstract in Three Minutes

Many people get stuck on “I’m not a researcher; how can I understand a paper?” Actually, you don’t need to understand the statistical details. If you can just look at a few key fields in the abstract, you’ll have a good sense of a study’s weight. I’ve put the most practical entry points into a quick-reference table—run through it next time you open a paper.

What to Look For	Where It Usually Appears	Questions to Ask Yourself
How many people (n)	Methods section	Single digits—discount it; more is more stable
Who the subjects are	Beginning of Methods	Does age, sex, training level match me?
Whether there’s a control group	Design description in Methods	A placebo/control rules out psychological effects
How large the effect is	Results section	Is it a reasonable 1–3%, or an exaggerated big number?
Authors’ own caveats	End of Discussion	Good studies honestly list “limitations”
Conflicts of interest	Disclosure at the end	Studies funded by supplement companies deserve more caution

Special note on the last two rows. In an honest study, authors will proactively state their limitations in the Discussion (small sample, acute-only, not generalizable, etc.); if a paper presents itself as flawless with implausibly large effects, raise your guard instead. As for conflict-of-interest disclosures, this isn’t to say industry-funded studies are always fraudulent—it’s a reminder to weigh it on the “credibility scale.”

Common Mistakes and My Corrections

Over years of coaching, I’ve seen all sorts of people led astray by “one study.” Here are the most common cases and how I help correct them.

Mistake One: Chasing Headlines, Changing Plans Weekly

Case scenario: Trainee B is the information-anxious type of rider. Every new study he sees, he wants to change his training. By the end of the season, he’d never completed a single full training block; his fitness was patchwork, and his performance stagnated.

My correction: I set a rule for him—any new method must let the current plan run a full block (usually 4–8 weeks) before discussing changes. The biggest danger in training isn’t choosing the wrong method; it’s never executing any method well enough to see results. Consistency beats novelty.

Mistake Two: Treating Anecdotes as Research

Case scenario: “My friend took this supplement and set a PR!” This kind of statement is powerful because it’s vivid and emotional. But one person’s single improvement could come from training, sleep, weather, course, mood… the supplement is just one of many variables.

My correction: I teach trainees to distinguish “anecdote” from “data.” Anecdotes can be a source of inspiration, but they can’t be the basis for decisions. If you really want to try something, test it in a controlled way (see “single-person experiment” below).

Mistake Three: Confusing Mechanism with Outcome, Animals with Humans

Case scenario: A trainee read that a certain ingredient improved endurance in mice and directly extrapolated to doubling his own dose. The problem is that mouse dosages, metabolism, and physiology differ greatly from humans, and “acute physiological change” is far from “your race gets faster.”

My correction: Develop the habit that when you see words like “mice,” “cells,” or “acute response,” you automatically add in your head: “this is far from real human performance.”

Mistake Four: Ignoring Risk and Cost, Seeing Only Benefits

Many people evaluating a new strategy only count the “potential benefits” and skip the “costs and risks.” Extreme low-carb diets, heavy supplementation, aggressive weight loss—these all carry physiological and health costs, especially for people with chronic conditions (like diabetes, hypertension, or heart disease). Every intervention must put risk on the scale, and when disease is involved, you must return to your primary care physician and professional team for individualized assessment—don’t use online averages as a prescription.

Actionable Advice for Readers at Different Levels

After all these principles, I know what everyone really wants is “so what exactly should I do?” Here are recommendations by level—find yours.

For Beginners New to Training

1. Don’t rush to read the latest research. At your stage, 99% of your progress comes from fundamentals—“train consistently, sleep well, eat enough, don’t get injured”—not some cutting-edge strategy. Building the foundation pays far more than chasing new knowledge.
2. Develop a reflex: when you see a sensational headline, find the original paper and check how many people were in it and who they were. This single habit will filter out eighty percent of the noise.
3. Recording matters more than reading. Start a simple training log (what you did today, how you felt, how many hours you slept). Three months later, your own data will understand you better than any paper.

For Intermediate Enthusiasts with Some Foundation

1. Build an awareness of the “evidence pyramid.” When encountering a new method, prioritize meta-analyses and professional society guidelines over single news articles. Downgrade single studies to “clues worth noting.”
2. Want to try something new? Test it as a “single-person experiment (N=1).” Concretely: pick a quantifiable metric (e.g., power or time on a fixed climb), control other variables as much as possible (similar sleep, weather, fatigue), alternate A (without) / B (with) multiple times, and see whether the difference consistently exceeds your normal day-to-day fluctuation. A single positive result doesn’t count; it must repeat.
3. Beware of “confirmation bias.” We all tend to remember the time it “worked” and forget the time it “didn’t.” So write down your judgment criteria in advance, then honestly evaluate against the data.

What a Real Single-Person Experiment Looks Like

Abstract principles alone feel hollow, so I’ve organized the single-person experiment process I use with trainees into an example table. Suppose trainee C wants to test “the effect of a pre-race strategy on a fixed climb performance.” Here’s how I’d design it:

Step	What to Do	Why
Choose a metric	A fixed ~5 km climb with steady elevation, timed	The metric must be objective, repeatable, and minimally affected by external factors
Establish baseline	Do no intervention first, test 3–4 times, record the time range	First understand how large your “normal” natural fluctuation is
Alternate tests	Alternate A (without) and B (with) multiple times; don’t do them consecutively	Avoid fatigue or weather trends creating illusions
Control variables	Do each test at similar times of day, sleep, food, and weather	Make the difference come as much as possible from what you’re testing
Blind it a bit	If possible, have someone else randomly assign; you find out afterward	Reduce psychological expectation (placebo) contaminating results
Interpret	B must be “consistently” faster than A, with a gap clearly larger than baseline fluctuation	A single lead might be luck; it must repeat

The key insight here: if your time on the same climb normally fluctuates by 30 seconds, then a strategy making you “15 seconds faster” doesn’t count at all, because it’s still within your natural noise range. This is exactly how small studies fool people—presenting fluctuations within the noise as real effects. Learn to measure your own noise first, and you’ll have immunity against “one study.”

FAQ for Advanced Readers

Q: Should I just stop reading studies altogether, since one can’t be trusted?
A: Quite the opposite. Reading research is a good habit; you just need to read smartly. The point isn’t “believe or don’t believe,” but “give proportionate trust based on evidence strength.” Treat single small studies as clues, and consistent meta-analyses as the backbone.

Q: What is pre-registration, and why is it often mentioned?
A: It means researchers publicly register what they plan to measure and how they’ll analyze it before collecting data. This greatly reduces p-hacking—fishing for significance after the fact. When you see a study with pre-registration, its credibility usually gets a boost.

Q: Pros use this stuff, so it must work, right?
A: Not necessarily. Elite athletes are willing to gamble on any marginal method that “might help a tiny bit,” even with weak evidence, because for them 0.5% is the difference between a medal and not. But for the average enthusiast, the cost-benefit of these marginal methods often isn’t worth it; you’re better off nailing the fundamentals.

Q: When a coach’s experience conflicts with research, who do I listen to?
A: The two aren’t opposed. Good practical decisions come from the intersection of “best research evidence × coach’s professional experience × the athlete’s individual situation.” Research provides average trends; experience fills in the individual details research can’t cover. Both are indispensable.

For Coaches and Team Leaders

1. Be an “evidence gatekeeper” for your athletes. Athletes are easily swayed by social media and product marketing. Your role is to filter for them—bring them the “forest,” not every “leaf.”
2. Use conservative language and individualized decisions. Especially with athletes who have health concerns, any advice involving nutrition, supplements, injury, or disease should be referred back to professional medical and nutrition teams—don’t issue prescription-style directives yourself.
3. Build your team’s own database. The athlete data you accumulate long-term is the “local evidence” closest to your specific group, and its value rivals journal publications.

A Note for Taiwan

Finally, let me address some very practical, very local situations.

Taiwan’s summers are humid and hot. Many endurance or fueling studies conducted in cool European or American environments don’t transfer directly to Taiwan’s humid outdoor heat—heat dissipation and electrolyte loss conditions are very different, and hydration and sodium strategies often need to be more aggressive. This is a living example of “generalizability.”

Diet-wise, Taiwanese people mostly eat out. Many precision nutrition plans from abroad (weighing grams, calculating ratios) are hard to replicate in the reality of bento boxes, noodle stands, and buffet restaurants. Rather than obsessing over a paper’s perfect ratio, master the big principles first: enough carbs around training, adequate protein, sufficient water and electrolytes. The benefit of these robust principles far exceeds chasing some optimization after the decimal point.

On healthcare, Taiwan’s National Health Insurance is convenient—that’s our advantage. Instead of self-diagnosing with a study you found online, make good use of resources: see a sports medicine clinic, or find a licensed physical therapist or dietitian. If you have a chronic condition, are on medication, or are planning aggressive training changes, let professionals assess your individual situation first, rather than treating some “average effective” study as a guarantee that applies to you.

Here’s another typical local case. Trainee D saw a post online about “high-intensity intervals can massively boost VO₂ in a very short time.” Since he had long work hours and little time, he wanted to replace his regular aerobic training entirely with daily high-intensity sprints. The problem: he had mild hypertension and was training outdoors at noon in Taiwan’s summer. There are two danger points here: first, the study’s subjects were mostly healthy young people, unlike his condition (a generalizability issue); second, switching all training to high intensity in a hot, humid environment raises both cardiovascular and heatstroke risks.

My approach: first, I had him return to his doctor to assess his blood pressure control and exercise intensity limits; then, I introduced the “new method” in small doses, progressively, into his existing training—adding one session per week, scheduling it in the cooler morning hours, with hydration and electrolytes ready, and closely monitoring his heart rate and perceived exertion. The same study might be good advice for a healthy college student, but for a middle-aged person with a chronic condition training in a hot, humid environment, it must pass three checkpoints before application: medical evaluation, environmental adjustment, and gradual introduction. This is what “individualization” looks like in practice—not just a slogan.

One more common information-source issue in Taiwan: many popular “research summaries” are written to sell products or drive traffic. They deliberately cherry-pick single studies favorable to a product, amplify the effects, and omit limitations. When you see an article ending with a pitch for a specific supplement or device, automatically downgrade its credibility and go find a neutral meta-analysis or professional society position instead.

Conclusion: Respecting Science Means Not Deifying Any Single Study

Back to the trainee who rushed in holding his phone. In the end, I didn’t stop his curiosity—I encouraged him to keep reading research. But I taught him to read more slowly, more skeptically, and to value “replication” and “consistency.” A season later, he was no longer yanked around by every weekly headline. Instead, by executing one method thoroughly and fine-tuning with his own data, he broke a performance plateau he’d been stuck on for two years.

The reproducibility crisis sounds scary, but it’s actually good news: it reminds us that science is a process of continuous self-correction, not a set of conclusions carved in stone. True evidence-based thinking isn’t blindly believing any paper, nor cynically distrusting all research—it’s learning to give proportionate trust based on evidence strength: treat weak evidence cautiously, use strong evidence as the backbone, and let your own long-term data always be the final arbiter.

Next time you see “latest research confirms…” I hope you’ll take a deep breath and ask those three questions: How many people? Are the subjects like you? Has anyone else reproduced it? Just doing that already puts you ahead of most people in knowing how to get along with science.

Wrap-Up: A Pocket Checklist

Here’s the article condensed into a checklist you can keep in your head. When you see any striking conclusion, run through it in order:

1. Sample size: How many people? Single digits—discount it heavily.
2. Subjects: Does the age, sex, level, and health status of the subjects match mine?
3. Control: Is there a placebo or control group? Without one, effects may be overestimated.
4. Replication: Is it an isolated case, or have multiple independent teams found the same direction?
5. Magnitude: Is the effect a reasonable 1–3%, or implausibly exaggerated?
6. Transparency: Is it pre-registered, with open data and honest limitations?
7. Motivation: Is the person publishing this also selling something?
8. Yourself: Even if it passes all checks, ultimately verify it on yourself with a single-person experiment.

These eight questions require no statistical background, yet they’ll shield you from the vast majority of overclaims out there. Turn them into a reflex, and you can enjoy the pleasure of reading research with peace of mind, instead of being led by the nose with every new wave of headlines. The training road is long—stability, patience, and honesty with evidence are what will truly carry you to the end.

---

This article is educational content and does not replace individual diagnosis or treatment advice from a physician, physical therapist, or dietitian. If you have chronic conditions such as diabetes, hypertension, or heart disease, or are on medication, or are planning aggressive training or dietary changes, please consult your primary care physician and professional team for individualized assessment.

References

• Reproducibility Project: Psychology — Estimating the reproducibility of psychological science, Science: https://www.science.org/doi/10.1126/science.aac4716
• Survey on attitudes toward reproducibility in sports and exercise science — A Survey on the Attitudes Towards and Perception of Reproducibility and Replicability in Sports and Exercise Science, Communications in Kinesiology: https://storkjournals.org/index.php/cik/article/view/53
• Large-scale project estimating replicability in sports science — Estimating the Replicability of Sports and Exercise Science Research, Sports Medicine: https://link.springer.com/article/10.1007/s40279-025-02201-w
• Narrative review of replication concerns in sports science — Replication concerns in sports and exercise science: a narrative review, Royal Society Open Science: https://royalsocietypublishing.org/doi/10.1098/rsos.220946

Related Reading

• Introduction to Sports Science Research Methods: How to Read a Paper (A Coach Shows You Through the Words “Research Confirms”)
• Correlation Is Not Causation: A Coach Helps You Understand Sports Science Research—Stop Being Fooled by a Single Headline
• The Evidence Pyramid of Sports Supplements: Which Ones Really Work, Which Are Just Trends, and How to Read the Research
• How to Maintain Off-Season Strength Gains During the Season (Just Once a Week)