My response to ONCOLOGY's single written review provided to me:
November 30, 2001
James F. McCarthy Senior Vice President, Editorial
48 South Service Road
Melville, NY 11747
Dear Mr. McCarthy:
I am in receipt of your letter of November 29, 2001, informing me that you are unwilling to give me the opportunity to respond to the critique of the reviewer and submit a revised manuscript. While your letter of December 6, 2000, along with my agreement to prepare the manuscript, may not have constituted a contract, I do feel that we at least had a good faith covenant, and I am going to ask you and your editorial board to deal with me in good faith.
There are several relevant issues to consider:
1. In your letter of December 6, 2000, you pointed out that each paper is referred to one or two reviewers, who are asked to write a short commentary to be published alongside the paper. You stated the following: "The commentary might be entirely laudatory, highly critical, or somewhere in between. In any case it will serve to highlight areas of consensus or disagreement among people most familiar with the topic in question. The result should be both lively and informative."
Now, certainly you knew in advance of your invitation that this was a highly controversial topic. Furthermore, you certainly knew that the preponderance of clinical opinion was against the clinical use of these assays. I could have written (and still can revise the manuscript to present) a more dispassionate and less controversial discussion. I think that perhaps my (highly defendable) criticism of current drug selection methods as being unblinded as to profit margin may have struck a particularly sensitive nerve. But this point is not central to the overall scope of the paper, and it could be omitted, along with certain other controversial statements. But my point is this: given the wording of your invitation, I had every reason to believe that you wanted me to take a stand that was both honest (from my point of view) and blunt. You invited controversy, and I gave you what you asked for.
2. Let's now look at the manuscript review provided to me. You state that "the science just doesn't support the clinical recommendations," and that this is the basis of your rejection. I note once again that no one has found fault with the scope, completeness, or accuracy of my review. I worked very hard on ensuring that I was writing the most comprehensive and accurate review ever written on the subject (I have previously contributed a number of prior reviews to the literature, though none in the past 5 years, during which time there have been some very important new studies published). So you asked me to review the topic, and I still have no reason to believe that I in any way failed to do a good job in my review.
So the point of your dissatisfaction would appear to be centered around what I conclude from these data, versus what your editorial board concludes. You reject the paper because the science doesn't support the clinical recommendations. But I make no specific clinical recommendations. I just conclude that the tests "should be much more widely utilized in clinical oncology practice and as an integral component to ongoing and future clinical trials." Now, I think that the above point of view is highly defensible, but it may certainly be criticized. That is what you had in mind, I believe, when you asked me to write the paper. I review the literature. I arrive at certain conclusions. Then the reviewers write rejoinders and you publish both. If the literature do not support the current use of these assays, then this is a point which deserves to be made, because these assays ARE being rather widely used in everyday clinical oncology practice, in this country. Most of the major laboratories are located here in California, although specimens are submitted from all parts of the USA, including from many NCI-designated comprehensive cancer centers. All the California managed care organizations have approved payment for this testing and routinely do pay. The California Medicare contractor has officially approved payment, and this means that any Medicare patient anywhere in the USA can have his/her tumor sent for testing, and Medicare will and does pay for the testing. The technology has twice undergone rigorous evaluation by the nationally-respected California Blue Shield technology assessment committee and both times was unanimously approved for routine payment, which is routinely provided. Approximately 10% of all ovarian cancer patients receive assay-directed chemotherapy in the USA; I think that the number of assays currently being performed on a non-investigational basis is on the order of 1,500 per month in the USA. These are a lot of patients, and this is a lot of assay-directed chemotherapy and a lot of money. If all of this is inappropriate and even dangerous, then this should be brought to light and should not be buried.
My manuscript (which, again, is up to date and comprehensive...save for a couple additional papers published since I wrote it, which I could include in a revision) makes the best case for clinical use of the assays. If this case falls short, then why not have two excellent clinical oncologists expose the shortcomings of the data which support 1,500 assays per month and Medicare reimbursement and Blue Shield reimbursement and managed care reimbursement?
3. I would like to briefly address the issues raised by the written review you provided to me (I will spare you the point by point rebuttal, which I could certainly provide, but will not, as I want to be constructive and not contentious). The reviewer takes a very narrow point of view, looking only at ovarian cancer. In addressing the general point of view that these assays "should be much more widely utilized in clinical oncology," by focusing solely on ovarian cancer, I believe the reviewer certainly leaves out many clinical situations where the use of the assays is highly defensible, in the context of the data presented in my review.
But let's just consider ovarian cancer; as noted 10% of all ovarian cancer patients in the USA are currently being managed with regard to data provided by cell culture drug resistance testing.
1. In the reviewer's correlation of assay findings with response, he/she doesn't understand or consider the Bayesian statistics which apply to this and all other diagnostic and predictive tests, including radiographic tests. "True positive" and "true negative" rates for any test (including the use of barometric pressure as a predictor of rain), vary according to the "expected" or "pre-test" probability of something happening. This is precisely the point illustrated in the data shown in Figure 3 in the manuscript.
The cell culture drugs resistance tests are VERY accurate. Just as a barometric pressure reading with a precision $1,000 German barometer is very accurate. When the barometer says that the atmospheric pressure is rising, you can believe what it says. But how does this help us in predicting weather? It helps in the following way: absent a barometric pressure reading, all we have is historical experience. In Southern California in November, there is about a 10% chance of rain on any given day. With a rising barometer, this decreases to less than 4%. With a falling barometer, the chance of rain increases perhaps to 35%. So the instrument is: (1) "accurate" (in correctly measuring atmospheric pressure) and (2) useful (in more clearly defining the probability of rain). Note that the instrument is "accurate," but the correlation is only "useful," but not perfect. To take this analogy one step further, let's move the barometer to Boston (still in the month of November). In Boston, historical experience indicates that there is a 30% chance of rain on any given day. With a rising barometer, this falls to less than 10%. With a falling barometer, this increases to more than 70%. Note that the accuracy of the instrument in measuring barometric pressure hasn't changed, but the correlation has changed (the instrument is still "useful," as it's much better to decide to play golf on a day with a 10% chance of rain than with a 70% chance of rain, even though the correlation between barometric pressure and actual weather is not perfect.
The assays are very similar to the above analogy. They are VERY accurate in measuring tumor cell death in the laboratory. Tumor cell death in the laboratory clearly correlates with tumor cell death in the patient. Drugs are more likely to kill tumors in patients in cases where they have been found to kill tumors in the laboratory (and not to kill tumors in patients in cases where they don't kill tumors in the laboratory). But the precise correlations vary according to the underlying probability of the drugs working on average, based on historical experience, just as in the case of historical experience with probabilities of rain on any given day in either Southern California or Boston in November. Just as with barometric pressure, the correlation is about 7:1, meaning a seven-fold greater probability of clinical benefit for drugs with good activity in the laboratory, compared to drugs with poor activity in the laboratory (or a seven-fold greater probability of good weather with a rising barometer than with a falling barometer).
Let's take a few examples:
Previously-untreated breast cancer. Historical probability of clinical benefit with Adriamycin = 50%. Theoretical probability of benefit with a favorable assay = 80%. Theoretical probability of benefit with an unfavorable assay = 11%
Previously-untreated colon cancer Historical probability of clinical benefit with 5FU = 20%. Theoretical probability of benefit with a favorable assay = 38%. Theoretical probability of benefit with an unfavorable assay = 4%
Previously-treated colon cancer Historical probability of clinical benefit with irinotecan = 15% Theoretical probability of benefit with a favorable assay = 29% Theoretical probability of benefit with an unfavorable assay = 3%
And so on. The relationship between the historical "expected" probability of response to chemotherapy and the more clearly-defined "assay-predicted" probability of response is shown in the enclosed graph.
Now, the value of the above information varies with different clinical situations. In some situations, it might be of only negligible value and might not be cost effective. But in other situations, it may be of considerable value. But this holds for every laboratory and radiographic test in medicine. Some clinicians use tests wisely; others use them unwisely. But the fact that tests may be misused does not negate the value of tests in situations where their utilization is helpful.
2. Clinical trials. The most valid criticism of the tests, in my opinion, is that no one has ever completed prospective, randomized studies to show that using the tests makes a difference in treatment outcome (n.b. two controlled, non-randomized trials have been published, each showing highly positive results, as described in the review). This is a completely valid criticism. But it has to be pointed out that no laboratory or radiographic tests in clinical oncology have ever been shown to make a difference in treatment outcome. Perhaps these tests are somehow different than the use of panels of expensive immunohistochemical stains to help diagnose and classify tumors when such testing is performed for the purpose of selecting chemotherapy. If so, these arguments could again be made in the accompanying published critiques.
3. Response as a measure of benefit. The reviewer seems to agree that selection of active first line therapy is important with respect to survival, but questions the relevance of 2nd line therapy to survival. But this is not a criticism of the assays; it is a more general criticism of 2nd line therapy. Response is certainly considered important in 2nd line therapy. When response occurs, therapy is continued. When response doesn't occur, therapy is discontinued or changed. Therefore the relevance of identifying the most active drug(s) would appear not to be controversial, at all. What is a matter for debate (which is precisely what the manuscript was designed to promote) is whether or not the assays should be used to assist in drug selection.
In point of fact, there are many different forms of 2nd line treatment: platinums, topotecan, Doxil, etoposide, gemcitabine, gemcitabine combinations, tamoxifen, hexamethylmelamine, and many other potentially reasonable treatments (vinorelbine, docetaxel, 2nd line platinum combinations, etc.). A legitimate point for consideration is whether or not the whole of the published literature (considering all forms of cancer which comprise the entire published dataset), supports the current use of these assays in assisting with this selection.
4. Cost benefit analysis. As you well know, very little in clinical oncology has been subjected to rigorous cost-benefit analysis. None-the-less, the points raised by the reviewer constitute a counter-argument to the positions taken by me in my discussions of the meaning of the data presented in my review. It would be highly appropriate to publish this counter-argument; the importance of doing this is precisely because these tests are being so widely used (and reimbursed), and their use is growing in this country. If the current data do not support this, then is it not important to bring this to the attention of the readership of ONCOLOGY? (of course, I could provide counter-counter arguments to support my point of view; though your reviewers would have the last word on this).
Reviewer's concluding remarks:
The reviewer's discussion of first line treatment of ovarian cancer actually makes a defendable case for the use of the assays in this situation. The reviewer correctly points out (as did I, in my own discussion presented in the review) that "vast numbers of studies even in front line have failed to consistently prove superiority of a combination over platinums by themselves." And yet what is "standard" first line therapy of ovarian cancer in the USA? Carboplatin + paclitaxel. This is much more expensive than carboplatin alone. What about using the assays to determine if carboplatin should be given alone, or with paclitaxel, or perhaps with another drug (e.g. gemcitabine)? The arguments and counter-arguments which are relevant to this question would make for an interesting debate.
Of great practical relevance is that these arguments have been made and continue to be made on a regular basis in this country in the context of reimbursement decisions. This again speaks to the relevance and importance of a scholarly consideration of these issues in your journal.
With regard to the anecdote of gemcitabine/cisplatin, the reviewer states that the advancement of this combination would have occurred absent the use of the assays. This is no doubt true, but the point was that the clinical use of the assays did clearly advance the clinical trials of this combination, and the several clinical anecdotes provided (including the ovarian cancer patient who failed first line carboplatin/Taxol (primary resistance), then failed tandem high dose therapy with stem cell transplants (primary resistance), only to subsequently achieve a durable CR with assay-directed gemcitabine/platinum, does provide support for (though certainly, by itself, does not make the entire case for) a role for currently-available cell culture drug resistance testing in at least certain clinical situations.
Once again, you invited me to prepare a complete and accurate review of the published data, for the point of stimulating debate on whether or not these data are sufficient to support the current, rather widespread and growing, use of these assays. I believe that I did exactly what you asked me to do, and I cannot understand why you are not making a good faith effort to work with me on revising this review to meet your worthy objectives.
Larry M. Weisenthal