About fair tests 

Not only do unproven treatments need to be tested, but the tests also need to be fair.

Without a fair test, the findings from any research may not mean very much.

Even worse, an unfair test could give healthcare professionals and patients the wrong idea, and people may be given a treatment that doesn't work or is actually harmful. They may also not be given a treatment that could be helpful.

This page explains:

Making comparisons

If someone who's ill takes a treatment and then gets better, it could be the result of a natural recovery that would have happened anyway.

To tell if the treatment has worked, it needs to be compared with another treatment or a placebo. The two results have to be different enough to indicate a difference hasn't occurred by chance.

Comparing a treatment with a placebo

The treatment may be compared with a placebo (a dummy treatment), such as a sugar pill, that looks the same as the treatment.

If there are fewer symptoms or other problems after a treatment than after taking a placebo, this suggests the treatment works.

Comparing a treatment with a standard treatment

Where a treatment is already known to be effective from previous research, it's usually not considered right (ethical) to compare the new treatment with a placebo. The new treatment usually needs to be compared with a standard treatment that's already known to be helpful.

This makes it possible to assess whether the new treatment works better than the treatment already being used. New treatments are as likely to be worse as they are to be better than existing treatments.

Placebo effect

The placebo effect is the phenomenon of someone's symptoms improving when they've only been given a dummy treatment, or even after they've just seen a doctor.

Sometimes a doctor's or other healthcare professional's reassurance, and their confident way of communicating with people who are feeling ill, helps some people feel better. The placebo effect is a largely mysterious and fascinating effect that can be quite powerful.

If you think and believe you're going to get better, you're much more likely to. However, this doesn't work in all situations and for all conditions.

Dummy treatments may be given to people in clinical trials. A placebo medicine looks the same as the medicine being studied, so you don't know which one you're taking. Some people may feel better after taking the placebo medicine because they think they're being given real medication. This is the placebo effect.

Placebos are particularly powerful in conditions where symptoms are important. For example, people feel pain differently and respond better to treatments they think are going to work. In extreme circumstances, some people who are in severe pain respond to a placebo as well as they would to a powerful painkiller.

But placebos don't work for all conditions. High blood pressure (hypertension) can be lowered by active medicines, but placebos have no detectable effect. Similarly, placebo treatments don't lower blood cholesterol, but statin medicines do.

Sham treatments that work

Researchers have designed ways of creating placebos for complementary medicine treatments, such as acupuncture.

It's possible to carry out sham acupuncture where needles are inserted to a different depth and in different places from those used in real Chinese acupuncture. In recent trials, both types of acupuncture appeared to be better than doing nothing.

Studies have also carried out placebo surgery on people with knee pain, and these have shown the placebo surgery often has good results.

Examples of the placebo effect include:

  • four placebo tablets work better than two in gastric ulcers
  • pink dummy pills are better at maintaining concentration than blue ones
  • placebo injections are more effective than placebo pills
  • painkillers work better if they're believed to be costly than if they're believed to be cheap

(Source: Bad Science, Ben Goldacre, Fourth Estate, 2008)

Control groups

Participants in a clinical trial will usually be put into one of two groups. They may be put in a group where they're given: 

  • the unevaluated treatment being assessed 
  • an existing standard treatment, or a placebo if no proven standard treatment exists (known as the control group)

The aim is to compare what happens in these groups. Participants are randomly assigned to one of these groups.

While the treatments are different in the two groups, researchers try to keep as many of the other conditions the same as possible. For example, both groups should have people of a similar age, with a similar proportion of men and women, who are in similar overall health.

Randomisation

The best way to get similar groups is to allocate individuals to one of the groups in the trial in an unpredictable, random way. This increases the likelihood that the two groups will be similar. This process is called random allocation or randomisation.

In most trials, a computer will be used to decide which group each patient will be allocated to. This allocation will be concealed until after each eligible patient has been accepted for the trial.

These precautions mean the people who decide whether a patient is eligible to participate in the trial can't influence which treatment a patient is allocated to receive. This protects the study from conscious or unconscious bias, which would make the test unreliable.

Blinding

Many trials are set up so nobody knows who's been allocated to receive which treatment. This is known as blinding, and it helps reduce the effects of bias when comparing the outcomes of the treatments.

Many people feel better if they think they're getting a better new treatment, even if the treatment is ineffective and their underlying health problem hasn't really changed at all.

When both the medical staff organising treatment and those taking part in the trial don't know who's receiving which treatment, it's called a double-blind trial.

Blinding is easier when testing medicines, but more difficult when testing other types of treatments or methods of caring for people. For example, it may be impossible to blind a trial comparing two types of surgery.

Why blinding is important

Some clinical trials measure hard outcomes such as survival, so outcome measurement is unlikely to be biased.

However, most trials measure outcomes that are more open to biased assessment. For example, patients and researchers may have to make some sort of judgement about how bad symptoms are.

If either researchers or participants know – or think they know – who is receiving which treatment (including placebos), that knowledge may influence what they report.

Participants who think they're taking an active treatment may not want to let down the researcher, and may exaggerate benefits and minimise side effects. Researchers also may allow their hopes about a new treatment to unconsciously influence their recording of symptoms.

The result of these biases is often to overestimate how effective a treatment is. To reduce these possible sources of bias, many trials are double-blind.

Size of trials

For a trial to be a fair test, the number of people taking part needs to be large enough. For example, in a small trial of 20 people, with 10 people taking each treatment, seven people may improve on the new treatment and five on the standard treatment.

Most of us would not think of that as a fair test because, while the new treatment may be better, the finding could easily have occurred by chance.

If the trial was bigger, with 700 out of 1,000 people improving on the new treatment and 500 out of 1,000 on the standard treatment, the result means researchers can be very confident that the new treatment was better.

The degree of this confidence in the difference can be estimated. Researchers can provide ranges called confidence intervals to show you how certain their results are.

Researchers can also test how "statistically significant" a result is. This can help to show where differences between treatments are unlikely to be the result of chance.

More information

Read more about fair tests on the Testing Treatments interactive (TTi) and the James Lind Library websites, where you can also download books for free, including how to make smart health choices and understanding health statistics.

The placebo effect

Ben Goldacre explains what the placebo effect is and describes its role in medical research and in the pharmaceutical industry.

Media last reviewed: 08/07/2015

Next review due: 08/07/2017

Page last reviewed: 05/01/2015

Next review due: 05/01/2017