Journal Articles as Evidence for Alternative Therapies

This article is on human studies for alternative therapies - for information on pre-clinical studies such as animal and test tube studies, see the article on Pre-Clinical Evidence.

Although the stereotype is that there is no published data for alternative therapies, this is very far from the truth; there is published clinical data for a surprising number of these therapies and this data is one of the most important forms of evidence for or against an alternative therapy.

The best evidence is from published data in "peer reviewed" medical journals. "Peer reviewed" means that the articles are reviewed by other doctors in the field before being published. Peer review is a form of quality control, though it is certainly not a guarantee of either quality or truth of the results. Still, peer reviewed publications in reasonable journals have been scrutinized for errors and bias by knowledgable people in the field - not just anything can pass. This is why peer reviewed journals must be distinguished from informal articles intended for a mass market, or even as marketing aids, and from unreviewed case reports not published in a peer reviewed journal. It is certainly possible, and commonly claimed that articles on alternative therapies are rejected from peer reviewed publications due to prejudice rather than fundemental flaws. Yet, I believe that a researcher who wishes to publish findings in the journals who is willing to revise his article based on criticism and perhaps even collect additional data, could do it if determined. Perhaps not in a top journal but at least in some reasonable journal. Many have, in fact, done so. Dr Prudden (Bovine Cartilage) and Dr. Burzynski (Anti-Neoplastons) are two examples. So I would be extremely skeptical of alternative practitioners who say they have great results but cannot get them published in a peer reviewed journal.

You will encounter several types of articles as you look at the literature. I describe them in rough order of their weight as evidence (my listing of cases over epidemiological studies might be controversial)

Types of Studies

Prospective Clinical Trial Reports

"Prospective" means designed in advance. "Clinical Trial" means a test of a specific intervention rather than mere observation of people's behavior. Clinical trials are the standard scientific method of evaluating conventional therapies, and they can also be used to evaluate alternative therapies.

I have extensive information on the various types of clinical trials in CancerGuide, and I strongly recommend you review my article on The Clinical Trial System if you are not familiar with how cancer clinical trials work.

There are some considerations particular to clinical trials of alternative therapies. In standard medicine, the "gold standard" is a randomized clinical trial showing improvement in survival, what most to be consider the ultimate goal. However in conventional medicine, randomized trials are always preceded by other studies (such as Phase II studies) which show definitively that the treatment has a biological effect on cancer - i.e. that it shrinks tumors. In my view, randomized or especially historical control studies of alternative therapies are much weaker if there is not a background of clinical evidence for a biological effect. It is very easy for some sort of bias to creep into a randomized trial, or especially a historical controlled study, and while studies proving a biological effect on tumors don't mean that such biases haven't crept in, they do increase the plausibility of results from a randomized trial.

In randomized studies, patients are randomly assigned to one of two treatment groups, and the results from the two groups compared. In historical controlled studies, all of the patients are treated with the method under test, and the results are compared to the past results of standard treatment. The weakness here is that it is extremely easy for there to be a systematic difference between the historical control and the current patients. This might be especially true if only a fraction of the current patients completed the therapy, or if the therapy requires a lot of commitment from the patient, since this can be expected to introduce a bias in favor of stronger patients in the treatment group.

An example of an alternative clinical trial with problems of a historical control group, poor treatment compliance, and lack of prior clinical evidence of a biological effect, would be Nick Gonzalez's study of metabolic/enzyme therapy for pancreatic cancer. This compared survival in a small group of patients to historical averages, but a large number of patients didn't comply with the therapy and were not included in the final analysis, and there is no published clinical data showing a biological effect.

Reference: Gonzalez J, Isaacs L, Evaluation of Pancreatic Proteolytic Enzyme Treatment of Adenocarcinoma of the Pancreas, With Nutrition and Detoxification Support, Nutrition and Cancer 33(2), 117-124

In some cases, negative results from randomized trials of alternative therapies have been reported by researchers not associated with the normal practitioners of the therapy. There have been negative controlled trials of Vitamin C and Hydrazine Sulfate, for instance. It is important to ask whether such a trial treated patients with a type of cancer and amount of disease consistent with the original claims for the treatment, and also whether the treatment was given the same way that the original discoverers gave it, or whether there were differences in the way it was given which might account for the negative results. Merely because there is a difference or even an outright error does not mean that it is plausible that this explains the negative result but it has to be strongly considered. For example, trials of Hydrazine Sulfate have been strongly criticized by proponents for allowing incompatible drugs such as alcohol and various tranquilizers. Re-analysis of the trials after the fact showed no benefit in patients who didn't use such drugs, yet surely it was an error by the researchers to allow them. So while this after the fact analysis supports their negative result, their error certainly weakens the validity of these studies.

Case Series and Retrospective Treatment Reports

Case series are reports of treatment in a series of individual patients. What distinguishes them from clinical trials is that there was no plan designed in advance for admitting and treating patients and for evaluating the results (Incidentally such a plan is called a protocol). Instead, the results of individual cases are reported in more or less detail. It may not even be clear how many patients were treated - not all patients may be reported, and the treatment of different patients may have varied as the practioner's ideas evolved. Click on on cases for more on evaluating this type of evidence. Case series are naturally more common as the main published evidence for alternative therapies than they are as the main evidence for conventional therapies where the mainstay of evidence is usually prospective clinical trials. To the extent that there is strong positive evidence from conventional clinical trials the treatment should not be classed as "alternative" - though prejudice may classify them as such anyway. Case series may not always be labeled as such. You may have to read how the study was designed to know if you are looking at a case report or at a clinical trial.

A related type of study is a retrospective comparative study. In this type of study survival (or other measures) of patients treated in the past with the alternative therapy is compared to historical results of conventional treatment of similar patients. This is much like a historically controlled prospective study except that the data for the treatment group is also gathered by looking through records rather than by treating patients according to a defined protocol. This is a bit weaker yet than the historical controlled study because the patients and measures of success were not defined in advance of treatment, and because questions about the completeness of records for the treatment group arise. Were all patients who got the alternative treatment included? - Or were only some included? - And could there be a bias that favors inclusion of patients with better results?

Epidemiological Studies

This is a large class of studies where researchers try to correlate people's behavior with the development or progression of cancer. For instance, what they eat, where they live, or how much they exercise. In these studies, patients aren't given a treatment - instead they are observed to find out if choices they made, or characteristics they had may be associated with differences in the outcome.

Epidemiological studies are not nearly as good evidence as prospective trials, and in my (unconventional here) opinion, may not be as good as well documented case series. One problem with epidemiological studies is that, as the statisticians like to say, "Correlation is not Causation." Just because people who drink coffee get more lung cancer doesn't mean that coffee causes lung cancer! More likely, heavy coffee drinkers tend to be smokers. So quitting coffee won't reduce your risk of lung cancer. What appears to correlate to a better outcome is not always imply an effective treatment. It has been found that people who consume a lot of beta-carotene in their food get less lung cancer. But then several studies of supplementing with beta-cartotene showed this actually increased the chance of getting lung cancer in smokers. There are several possible explanations for this, but one is that people who get a lot of beta-carotene in their diets are also getting lots of other healthy plant chemicals because the way you get beta-carotene is by eating various vegetables. So beta-carotene may be a marker for consumption of other substances which do act to reduce the risk of lung cancer in smokers. If these other substances are other carotenoids (related chemicals) then adding high amounts of beta-carotene to the diet might actually block the absorption or action of these more effective carotenoids.

Often studies on prevention of cancer with various nutrients are used to claim that these substances are effective in treating cancer. Preventing cancer and treating cancer are very different things! Once cancer develops, there are already cancer cells multiplying out of control, and bringing them under control probably requires different treatment than preventing normal cells from becoming cancerous in the first place. Think of cancer as being like a forest fire. Putting out a fire is totally different from preventing one!

More promising are studies which show that patients who have already developed cancer, but who behave in certain ways, live longer or have fewer recurrences. Still, these retrospective studies are clearly weaker than randomized studies, which prospectively study the effect of that behavior.

Letters, Reviews, and Opinion

Often you will find articles reviewing some aspect of medicine, or offering new ideas based on a reading of the literature. You will also find letters to the editor commenting on other articles. What all of these have in common is that they do not present original data, so such articles cannot be primary evidence for or against a therapy. In particular, hypotheses are not data!

Reviews can be very helpful in summarizing aspects of some area of medicine and are often easier to read than primary articles. It is rare to find an unbiased review of an alternative therapy, but you may find reviews of various aspects of cancer biology which are relevant to some alternative therapy.

You may also find reviews of alternative therapies by opponents. For example, a very prominent journal, JAMA, the Journal of the American Medical Association, has for years run occasional review articles criticizing various alternative therapies. These articles include all kinds of factors other than the actual data, such as legal actions against the therapies and questions of whether the practitioner is really qualified as well as the data or lack of data. I think while these articles should be considered, one should also try to find out if there are any responses or rebuttals to the charges made - the journal itself may then refuse to publish rebuttal letters by the practitioner, so you may have to look at other sources to find the rebuttal if any. Such negative reviews, even if not necessarily the last word, are valuable in alerting you to possible problems with the evidence for an alternative therapy.

Other Important Considerations

Independent Confirmation

Ensuring that results are really reproducible is one of the most basic parts of the scientific method, and part of that is ensuring that people other than the original discoverer can get the same results when they use the same methods.

Alternative therapies are usually "questionable" in one way or another, and are often based on the work of just one practitioner or researcher. That one researcher or doctor may have published clinical studies, and this is definitely encouraging when it happens.

But it is even more encouraging if other doctors used the same methods and were also able to get positive results. Publication of positive results by people other than the original discoverer is a vastly positive piece of evidence.

Is the Data Recent?

The literature for some alternative therapies includes very old articles, from the 1930's and 40's or even the 1800's. You should be more wary of such very old articles.

Modern medicine has powerful imaging techniques such as CT scans, MRI and nuclear medicine scans for tracking the effect of cancer treatments, and we now have agreement on the importance of pathological diagnosis of cancer as well as vastly improved techniques for making that diagnosis. The increase in knowledge of most areas of biology has been simply tremendous, and so old papers may well be based on very outdated ideas.

In addition, very old articles may use different terms and techniques than current ones and thus they may be very difficult to interpret. None of this means that very old articles cannot be evidence - citing of old papers may just be evidence of a particularly scholarly mind at work - but such articles do deserve a lot of caution, particularly if they are a substantial part of the evidence.

Appropriate Citation

Informal articles purporting to support (or deride!) alternative therapies will often cite articles - first ask if the articles cited are peer reviewed journal articles or if they are other informal articles. Then ask if the use of the article is appropriate for the claims being made for it.

If something is being claimed to be a highly effective cancer treatment, and the main evidence for it are newspaper articles, epidemiological, and pre-clinical studies, there is a major problem. I commonly see this, and when I do, I lose respect for the article I am reading. On the Internet I often see multi-level marketing companies flogging pre-clinical results as proof of a miracle cure. Don't buy it! This doesn't mean that epidemiological and pre-clinical studies, and case reports can't be legitimately cited in support of a therapy, but the conclusions drawn should be appropriate to the type of evidence. Indeed I gain respect for a source which cites all the evidence but does so appropriately.

This CancerGuide Page By Steve Dunn. Copyright 1999 Steve Dunn
Last Updated Aug 24, 1999