The September 1, 2004 edition of the Journal of Clinical Oncology will contain two reviews relating to cell culture drug resistance testing (CCDRT). Electronic versions of these reviews are available (to JCO subscribers) on the web site for the Journal of Clinical Oncology. Links to the abstracts are provided below. I have read the full text versions of each, but copyright restrictions prevent me from posting the full text versions of the two papers. However, these papers are potentially important, and I would like to discuss them.

What needs to be emphasized, first of all, is that both reviews were restricted to a consideration of published studies describing the results of clinical trials in which comparisons were made between groups of patients treated with and without consideration of CCDRT test results. Neither paper considered the (voluminous) literature which describes the predictive accuracy of CCDRT (the extent to which the results of CCDRT correlate with and predict for clinical outcomes). Thus, the first paper is NOT a "systematic review" and the second paper is NOT a true "technology assessment"

The first paper (written by an insurance industry group) reviewed 11 studies and reported that 5 of the 9 studies found significantly higher response rates for patients receiving assay-guided therapy, as did 1 of 2 randomized studies, despite the fact that these were all small studies, with a low power for detecting significant differences. In the case of the assays using cell death endpoints, 4 of 5 studies showed higher response rates for assay-guided therapy, and two non-randomized studies reported survival advantages for assay-guided therapy (n.b. The one "negative" study should not even have been included, as it was a study of sub cultured cells grown in monolayers, which is not a relevant technology.)  The insurance industry authors concluded the following: "Higher response rates for assay-guided therapy cannot be clearly attributed to more effective treatment, given that nine of 11 studies were nonrandomized. In this set of studies, it is often hard to exclude selection bias and confounding as influences on the results" [sic]. Although the authors did not review the data pertaining to predictive accuracy, they quoted their own previous review showing that "a negative assay result can be false in 20% of cases, and a positive assay result can be false in 13% of cases." Firstly, the authors got their data from my own review and they got the data wrong.  Considering appoximately 2,000 peer-review published clinical correlations, the published data indicate that patients treated with assay "positive" drugs had an overall 80% response rate, while patients treated with assay "negative" drugs had an overall 13% response rate.  The authors failed to note that even the incorrect level of accuracy quoted in their own paper would still mean that there would be better than a 4:1 advantage for choosing "assay positive" drugs over "assay negative" drugs, and most patients would prefer an 87% probability of clinical benefit, over a 20% probability of clinical benefit.  The correct numbers indicate better than a 6:1 advantage (80% vs. 13%).  Numerous studies have also confirmed significantly better survival for treatment with "positive" durgs.

The second paper (written by a working group from the American Society for Clinical Oncology) was broadly similar to the first paper, and also concluded that the existing literature did not established that assay-directed therapy was superior to empiric therapy. The "summary of Literature and Recommendations for Practice" concluded that "little has changed since the review published by Cortazar and Johnson in 1999." (nb. A critique of this earlier review appears elsewhere on this web site). The authors go on to say: "The absence of a suitable [CCDRT method] for routine clinical use reflects problems in technical success and yield of the assays (nb. in our own real world experience with over 5,000 clinical specimens submitted to us for testing, we successfully reported results from well over 90% of specimens received, testing a median of 18 drugs and combinations in at least two different assay systems, at two drug concentrations), the lack of adequate prospective evaluation [of assays] in clinical trials, and the tendency of [assays] to recommend treatments which would be given empirically" (n.b. by no means true; I'll try and post representative data in various diseases on this web site in the future).

The conclusion of both sets of authors is that prospective clinical trials (first paper) "are needed" and should be (second paper) "a priority." The first paper (from the insurance industry) does not presume to tell clinical oncologists that currently-available tests should or should not be used, but the second paper (from a working group representing American medical oncologists) comes right out and says that use of the tests "is not recommended outside of the clinical trial setting." Instead, the group says, "oncologists should make chemotherapy treatment recommendations on the basis of published reports of clinical trials and a patient's health status and treatment preferences." The paper does not state what should be done in the event that the clinical trials data are inconclusive and the patient prefers to have assay-guided treatment (see discussion below).

DISCUSSION:

The problem with both of the above reviews is that they are, in effect, attempting to impose an entirely new standard for the evaluation of medical tests. The previous standard always used to evaluate any type of medical test (e.g. laboratory tests, radiographic tests, functional tests, etc.) has always been the correlative and predictive accuracy of the test. How well does an estrogen receptor assay predict for clinical benefit of tamoxifen? How well does a Her2/neu test predict for clinical benefit of Herceptin? How well does a bacterial culture and sensitivity test predict for clinical success or failure of penicillin therapy? How well does a CT scan or MRI scan or PET scan or CA-125 level or CEA level correlate with the presence and growth of a cancer? How well does a battery of immunohistochemical tests performed on tumor biopsies correlate with diagnosis and prognosis? Not only is test accuracy (not "efficacy") the established standard for evaluating every single test used in medicine, it is also the precise standard used by the FDA in approving a test kit for cell culture drug resistance testing ("chemosensitivity testing") [Link to 3 MB PDF file copy of original document based on which favorable FDA review was obtained].  The FDA didn't require proof of "efficacy" (as it has never required proof of "efficacy" for any medical tests).

What has NEVER been shown, with ANY of the above tests is whether patients treated with the benefit of test information have higher response rates, longer durations of survival, less toxicity, or improved quality of life than patients managed without the benefit of test results. And these tests are used to select therapy in cancer patients no less than are the cell culture drug resistance tests currently being considered. Should a patient receive tamoxifen or chemotherapy? Should a patient currently on chemotherapy be maintained on the same chemotherapy, or should the patient be switched to a different form of chemotherapy? Are serial CT scans or MRI scans or PET scans better or worse at making this determination than are simply histories, physical examinations, and simple laboratory and "plain film" radiographic tests? We don't really know the answer to any of those questions; yet oncologists routinely order these tests and insurance companies routinely pay for them.

The degree to which "standard therapy" has been well documented in published clinical trials to be "effective" has been often overstated (or at least over assumed). Let's take ovarian cancer as only one of scores of obvious examples. "Standard" first line therapy for ovarian cancer is carboplatin plus Taxol. This therapy is very expensive (and, it must be noted, extremely remunerative to oncologists) and often toxic. What have 25 years worth of randomized clinical trials in ovarian cancer shown? Firstly, there is no established difference between platinum combinations and single agent alkylators (such as melphalan). Secondly, there is no established advantage to carboplatin/Taxol over single agent carboplatin. So why do American oncologists use carboplatin/Taxol? First, because they just take it on faith that it is better. Second, because it pays very well. (It should be noted that, while the American Society of Clinical Oncology has consistently tried to extinguish the use of CCDRT in cancer chemotherapy, it has, at the same time, worked hard to preserve the system of reimbursement wherein medical oncologists had the impossible conflict of interest of choosing between different forms of chemotherapy with wildly differing profit margins. In the absence of CCDRT, medical oncologists are free to choose from among many otherwise therapeutically equivalent regimens, with wildly differing profit margins, while the use of CCDRT severely constrains this choice. The American Society of Clinical Oncology has never supported nor suggested clinical trials to show that patients had equivalent outcomes in the presence and absence of profit motives in choice of therapy.).

One must recall the extraordinary difficulty in proving the efficacy of chemotherapy in general and of specific drug regimens in particular in studies of non-assay-directed chemotherapy. Only with extremely large studies (and sometimes only with meta-analyses of extremely large studies and sometimes, e.g. platinum-based chemotherapy of ovarian cancer, not even then) has it been possible to document that chemotherapy of any type produces survival advantages compared to no chemotherapy at all, in many clinical situations. The quite impossible challenge of documenting the clinical standard of "efficacy" (as opposed to the heretofore traditional laboratory standard of "accuracy") with these non-proprietary, public domain technologies was, in fact, pointed out by Dr. Maurie Markman (a noted critic of Human Tumor Assays) who correctly wrote that "even if it were possible to establish the efficacy of [the assays] in a particular situation, this would do nothing at all to establish the efficacy of [the] assays in any other situation".

The challenge of "validating" a single test for a single treatment in a single disease is challenging enough (e.g. estrogen receptor in breast cancer, which has still, after 30 years, only been shown to correlate with clinical outcome and has yet to be shown to improve clinical outcome, despite the fact that Dr. Daniel F. Hayes, Director of the Georgetown U Breast Cancer Program and a member of ASCO's "Tumor Expert Guidelines Panel" has referred to Estrogen Receptor as being "the best predictive factor in oncology"). Now consider the challenge of "validating" a test for 40 different drugs which can be used in tens of thousands of combinations in hundreds of diseases. If documented clinical "efficacy" is the standard to be demanded of non-proprietary laboratory tests, then clinicians should abandon all tests currently used in their practices. It will be interesting to see which standard is applied in the future to other laboratory tests associated with the prediction of drug resistance, such as tests based gene expression patterns.

It is very easy to say "prospective clinical trials should remain a priority;" it is quite another to support such trials.  In point of fact, the cooperative oncology groups (the only organizations capable of carrying out clinical trials of the scope necessary to address the questions of interest) have consistently refused to cooperate with the proponents of cell culture drug resistance testing (CCDRT) to carry out the needed trials.  For example, the Gynecologic Oncology Group has refused to consider clinical trials of cell culture drug resistance testing with cell death endpoints (which are the assays with the greatest body of clinical validation), while supporting "Big Pharma" trials, supported by millions of dollars from "Big Pharma" which go to support both the trials and the individual investigators.  All that the proponents of CCDRT can offer the cooperative groups is donated assays and the prospects of improving the clinical results of cancer chemotherapy, but this is not enough to motivate the cooperative groups to do the clinical trials.

While evaluating the data discussed elsewhere on this web site, please consider that it has taken 25 years to amass this body of evidence in an environment of continued hostility and non-support by the academic oncology community toward work in this area and consider also the little which has been achieved in the area of empiric methods of drug selection, despite billions of dollars spent on empiric clinical trials enthusiastically supported by this same academic oncology community. In point of fact, staying with the representative example previously given, the only thing clearly established after 30 years of clinical trials in advanced ovarian cancer is that carboplatin and cisplatin are therapeutically equivalent, albeit with different toxicity profiles. The data regarding optimum first line chemotherapy remain entirely inconclusive, and there are absolutely no data to support any of the half dozen or so available drug choices for second and third line therapy over any other choice. So what is the "risk" in using currently available assays to help guide these choices?  Compared to the risk of continuing to support the conflict of interest existing when choices are made between different forms of chemotherapy with vastly differing profit margins?

Only when these assays are widely performed and used and routinely included as an integral part of clinical trials will these already promising technologies be improved and only then will their role in patient management become better defined. But this is true for all complex laboratory technologies (a good example being immunohistochemical staining for batteries of cell antigens). Absent this testing, on what basis are drugs chosen today for use in the many clinical settings in which a single "best" empiric regimen has not been well defined? An objective reviewer would admit (as discussed above) that many oncology practices would base choice of drug regimen, at least in part, on the profit "spread" between the wholesale cost of the drug(s) and the reimbursement which the third party payers provide. This is an impossible conflict of interest as well as a cost ineffective method for selecting therapy; yet it is a method which the oncology and insurance communities support every single day in their treatment and coverage decisions.

It is the loss of this "freedom to choose" and the overzealous dedication to an extremely weak clinical trials paradigm (identification of the "best" treatment to give to the average patient) which is largely behind the reluctance to introduce these technologies as an important component of current clinical trials and as a part of the process of clinical drug selection in situations where clear empiric "best regimens" have not been well defined through prior clinical trials.

The private sector laboratories offering CCDRT as a patient service have been able to make considerable progress in improving the assay technologies and in building databases which improve the interpretation of "raw" assay results. But this progress has only been possible because insurers and often patients have been willing to pay for the tests and because clinicians have wanted to have the information provided by the tests. The progress would have been much faster (and doubtless even more substantial) had the academic oncology community not done everything it could to oppose this work at every step of the way.

By raising the bar of acceptance to levels unprecedented for a laboratory test, in essence a tariff has been erected to protect the paradigm of the "best" empiric treatment for the average patient, as identified in appallingly non-productive clinical trials. This tariff also serves to protect the paradigm of drug selection with consideration of the spread between wholesale cost and reimbursement.

Finally, the tariff discourages discovery of new, effective drug regimens through the use of CCDRT to guide drug selection. Take, for example, the gemcitabine/cisplatin combination. Years before gemcitabine/cisplatin became a widely used drug regimen, CCDRT identified this as the most active regimen in a patient with pancreatic cancer metastatic to kidney, omentum, and liver, despite the poor activity of gemcitabine and cisplatin tested as single agents. This patient went on to achieve a complete remission with gemcitabine/cisplatin and remained alive with an excellent quality of life 7 years later. A second such patient was an ovarian cancer patient with primary resistance to paclitaxel/carboplatin who then underwent tandem stem cell transplant/high dose chemotherapy regimens (at a cost of more than $250,000) without ever achieving a response. At a time when she had bulky, non-cytoreducible abdominal and pleural disease, CCDRT confirmed resistance to single agent cisplatin, carboplatin, and gemcitabine, but good activity for the gemcitabine/carboplatin combination. She subsequently received gemcitabine/carboplatin as an outpatient, achieved a durable complete response, and returned to work full time as an oncology nurse, where she remained well, for four years, until an intra-pericardial relapse. Indeed, early anecdotal results of this type occurring in diseases in which there was no existing clinical trials literature accelerated clinical trials of this regimen in diseases in which assay-directed responses had been observed.

With more widespread use of these assays in clinical oncology, it is very likely that the activity of new drugs and new regimens would be identified at a much earlier time than with the current system relying exclusively on usually empiric, Phase II trials.

It is ironic that the Journal of Clinical Oncology reviews are following so closely on the heels of a seminal New England Journal of Medicine publication (August 5, 2004). The authors of this latter paper used a 96 hour suspension culture drug resistance assay with a cell death (MTT) endpoint  to define cut-offs for "sensitivity" and "resistance."  They then used these data to define gene expression patterns associated with sensitivity and resistance to each of 4 drugs commonly used in the treatment of childhood leukemia.  They were then able to show that these gene expression definitions of sensitivity and resistance were significantly and independently associated with treatment outcome on multivariate analysis.  Note that this work could not have been done without the prior work in more than a thousand CCDRT assays from children with leukemia to define sensitivity and resistance cuf-offs for each of the four drugs.  The cell culture assays, therefore, are the "Rosetta Stone" which allows for identification of clinically relevant gene expression patterns which correlate with clinical drug resistance for different drugs in specific diseases.  This further shows how short-sighted it has been for the academic and clinical oncology community not to support the development and clinical application of CCDRT.

Why is it so necessary to protect the patient from the information provided by a perfectly rational laboratory test, supported by a wealth of entirely consistent data? If used to assist in the selection of a regimen chosen from a series of otherwise reasonable alternatives, then patients will never be harmed and best available evidence strongly indicates that they will often be helped.

Think of all the objections to this testing. Now try to design all of the clinical trials which would be needed to meet all of these objections and think of how much money these would require and who is going to provide this money and how many years the studies would take and how many patients will continue to receive ineffective or sub optimum treatment in the interim. The body of information will never be sufficiently large and complete and definitive to encompass even a reasonable fraction of the situations where the information provided by the tests would be helpful. Now ask the questions: What is the potential risk? What are the potential benefits? What is the probability that these tests really do provide information which can improve the drug selection process in individual clinical situations? What is the potential cost? How does the benefit/risk ratio balance out? What is the (financial) cost as a percentage of total costs relating to management of patients on chemotherapy (including the costs of radiographic and laboratory studies performed only to determine if a given form of treatment is working or not)? What are the long term costs if drug selection always remains an empiric, one-size-fits-all, trial and error process? What would be the impact on improving existing technologies (through the attraction of more laboratory and clinical investigators into the field) and developing new technologies should these assays become more widely used?

If one wishes to see an example of an entirely rational technology advance, in a human disease crying out for precisely such a technology advance, supported by an entirely consistent body of data, where the advance continues to be held hostage to a high bar of extraordinarily difficult clinical trials which the critics have been entirely unwilling to support, in an area (laboratory testing) for which such trials would be entirely unprecedented, one need look no further.

- Larry Weisenthal
August 12, 2004