| Coverage Home |
Medicare Coverage ~ MCAC
Executive Committee
Human Tumor Assay Systems and AuSCT for Multiple Myeloma
Transcript of December 8, 1999 Meeting
Note: The language on this website comes directly from the transcribed testimony taken at this panel meeting. The views and opinions are those of each of the experts and not those of the Centers for Medicare & Medicaid Services. CMS does not edit these transcripts and makes no assertion as to their accuracy.
-
Chairperson:
- Harold C. Sox, M.D.
- Sharon K. Lappalainen
- Linda A. Bergthold, Ph.D.
- Randel E. Richner, M.P.H.
- Robert H. Brook, M.D., Sc.D.
- Hugh F. Hill, III, M.D., J.D.
- Thomas V. Holohan, M.A., M.D., FACP
Leslie P. Francis, J.D., Ph.D.
- John H. Ferguson, M.D.
Robert L. Murray, Ph.D.
- Alan M. Garber, M.D., Ph.D.
Michael D. Maves, M.D., M.B.A.
- David M. Eddy, M.D., Ph.D.
Frank J. Papatheofanis, M.D., Ph.D.
- Harold C. Sox, M.D.
Ronald M. Davis, M.D.
- Daisy Alford-Smith, Ph.D.
Joe W. Johnson, D.C.
-
Opening Remarks
Panel Business
HCFA Presentation - Levels of Evidence
Open Public Session
Open Committee Deliberation - Levels of Evidence
HCFA Presentation - Evaluation of New Technologies
Open Committee Deliberation
Open Public Comment
Committee Conclusions - Levels of Evidence
Review of Medical Specialty Panel
Recommendation
Review of Medical Specialty Panel
Recommendation
Open Public Comments
-
Robert A. Nagourney, M.D.
Larry Weisenthal, M.D., Ph.D.
Frank J. Kiesner, J.D.
Elizabeth Panke, M.D,, Ph.D.
MS. LAPPALAINEN: I would like to say good morning and welcome, Panel Chairperson and Committee Members and members of the audience. I am Sharon Lappalainen. I am the Executive Secretary of the Executive Committee of the Medicare Coverage Advisory Committee.
The committee is here today to hear reports from recent meetings of the Medicare Coverage Advisory Committee Medical Specialty Panels. The committee will also consider how to provide guidance to and substantive coordination among MCAC panels. For example, the committee will consider levels of evidence, types of information needed and the nature of issues that will be considered by the Medical Specialty Panels at future meetings.
For today's panel, I would also like to welcome Dr. Hugh Hill, who is with the Coverage and Analysis Group. Dr. Hill comes to us from the Johns Hopkins University. In addition to his duties, he will also serve as the HCFA liaison to the Executive Committee.
I would also like to read the conflict of interest statement to the record. The following announcement addresses conflict of interest issues associated with this meeting and is made part of the record to preclude even the appearance of an impropriety. To determine if any conflict existed, the agency reviewed the submitted agenda and all financial interests reported by the committee participants.
The conflict of interest statutes prohibit special government employees from participating in matters that could affect their or their employers' financial interests. The agency has determined that all members and consultants may participate in the matters before the committee today.
With respect to all other participants, we ask, in the interest of fairness, that all persons making statements or presentations disclose any current or previous financial involvement with any firm whose products or services they may wish to comment upon.
At this time, I would like to turn the meeting briefly over to our distinguished Chairperson, Dr. Sox, who will ask the members to introduce themselves.
DR. SOX: Thank you. Good morning. I think we will make introductions perhaps starting with Linda Bergthold. Could you say who you are, where you work and your role on the panel, please.
DR. BERGTHOLD: Sure. My name is Linda Bergthold. I am the consumer representative to the Executive Committee. I am from California.
MS. RICHNER: I am Randel Richner. I am the industry representative from Boston Scientific.
DR. FRANCIS: I am Leslie Francis. I am a professor of law and professor of philosophy at the University of Utah in Salt Lake City.
DR. HOLOHAN: I am Dr. Tom Holohan. I am trained in hematology, oncology. I am Chief of Patient Care Services for the Veterans Health Administration in the Washington, D.C. Headquarters.
DR. FERGUSON: John Ferguson. I am a neurologist and former Director of the National Institutes of Health Consensus Development Program for the last eleven years and now a public health consultant.
DR. MURRAY: I am Bob Murray. I am an attorney and biochemist at Lutheran General Hospital in Chicago, Vice Chairman of the Laboratory and Diagnostics Panel.
DR. BROOK: Hi. Robert Brook at RAND and UCLA.
DR. SOX: I am Hal Sox. I am a general internist. I chair the Department of Medicine at Dartmouth Medical School. I chair one of the panels and I guess I am also the Chair of the Executive Committee.
DR. HILL: I am Hugh Hill and I have been introduced.
DR. GARBER: I am Alan Garber. I am a professor of medicine at Stanford and staff physician at the Department of Veterans Affairs. I am Chair of the Medical and Surgical Procedures Panel.
DR. MAVES: I am Mike Maves. I am an otolaryngologist at Georgetown. I am President and CEO of the Consumer Health Care Products Association and I am the Vice Chair of the Medical and Surgical Panel.
DR. ALFORD-SMITH: I am Daisy Alford-Smith, Director of the Summit County Department of Human Services in Ohio, as well as the County Coordinator for all of its social services. I chair the DME Committee.
DR. JOHNSON: Joe Johnson, private practice, chiropractor. I am co-chair of the DME Committee.
MS. LAPPALAINEN: We have two items on the agenda for panel members. The first is to disclose the panel of the schedule of the Medicare Coverage Advisory Committee. That tentative schedule is available to you, the audience, as a handout. You can pick that up.
But I would like to remind the Executive Committee that their schedule is March 1st and 2nd, June 6th and 7th, and November 7th and 8th. I would like to note that these are tentative dates and that they could be subject to change.
The second item is the types of information that may come before MCAC, and I will defer to Dr. Hill, who will make a few remarks on this item.
Also, I have been reminded by our audiovisual, that the small gooseneck microphones are extremely sensitive, and they will pick up your voice, so you don't need to bring them too close.
DR. HILL: Thank you, Sharon, I will be brief especially since I am within striking distance of Dr. Sox's gavel.
I am just going to review a couple of things from the charter and from the Federal Register notice about the panels and about the Executive Committee, and finish with one additional thought about what we are considering as appropriate for referral to panels.
The charter calls upon the committees to review and evaluate medical literature, to review technical assessments, to examine data and information on the effectiveness and appropriateness of medical services and items.
The panels are to develop technical advice to be reviewed and ratified by the MCAC, and the Executive Committee is to do three things: provide guidance to panels, facilitate substantive coordination among panels, and review and ratify panel reports and submit the report to HCFA.
Our Federal Register notice regarding our national coverage decisionmaking process provided that a referral to an MCAC or an outside assessment will involve issues that generally are complex and controversial, often involve broad health policy concerns.
The issues may require extensive consultation with specialty societies, medical researchers, and others familiar. In general, we may refer an issue to the MCAC if it's a subject of significant scientific or medical controversy, there is a major split in opinion among researchers and clinicians regarding the medical effectiveness of the service, the appropriateness of staff or setting, or some other significant controversy that would affect whether the service is "reasonable" and "necessary" under the Act.
Two. It has the potential to have a major impact on the Medicare program, and we define that broadly.
Three. It is subject to broad public controversy.
Finally, in addition to those criteria, we are internally talking about the propriety of referring to panels, issues that are close calls. When the medical literature and the scientific evidence are clear one way or the other that something is or is not reasonable and necessary, it may be easier for us to go ahead and decide that internally without referral to a panel.
Thank you.
DR. SOX: I would like to call upon Ron Milhorn, who is a health insurance specialist.
Mr. Milhorn.
HCFA Presentation - Levels of Evidence
MR. MILHORN: Good morning.
[Slide.]
I want to briefly go over Medicare's use of medical evidence.
[Slide.]
Since its inception in '66, Medicare has used medical evidence to make its coverage decisions. In the early days, primarily due to necessity, relied to a great degree upon the informed opinion of consultants and professional societies, and those sorts of things.
Gradually, we developed a more evidence-based decisionmaking process.
[Slide.]
Over the past decade, we have increasingly stressed the need for published scientific studies in order to develop our coverage policies.
Our coverage notice, which we published on April 27th of this year, makes it clear that requesters, those who wish to investigate an issue for coverage, need to provide for us the published scientific evidence and, in some cases, unpublished scientific evidence, that is sufficient for us to review the issue and develop a covered policy whether yea or nay.
[Slide.]
Our general requirements--and I think I should point out at this juncture that we are currently working on a regulation or a proposed regulation actually that is supposed to outline the criteria to be used in making coverage decisions under Medicare--the primary purpose of this regulation is not to be the final word, but actually to provide an underpinning or a foundation for what we have called sector-specific guidance documents which will be much more narrowly focused toward drugs, devices, diagnostic imaging services, and that sort of thing.
We have, as some of you well know, made several attempts to publish a coverage regulation, 13 or 14, in the history of the program, four in the last 10 years. None of them have been successful, at least in part because there is this tendency to say, ah, that paragraph doesn't apply to me, that sentence I don't like, and you get nibbled at that by ducks.
One of the ways we hope to avoid that, and we think we can avoid it, and also in the process provide useful information to those who want to come here and get something covered, is to be very much more specific as to the particular area of medical care that we are looking at.
In addition to the general requirement, though, that we use, and we are going to continue to use unless we get or until we get--I should say "until," not "unless"--we get this coverage regulation published in final and the guidance documents prepared, is that the service in question must demonstrate by authoritative evidence that it's medically effective. That's it. That's all there is to it, but it is, of course, a great deal more complicated than that.
In addition, our program has requirements as to the appropriateness of the service, and I will go to that in just a second.
[Slide.]
Authoritative evidence is written medical or scientific results demonstrating the medical effectiveness of the service. Generally, what we are looking for here is reports of such things as controlled clinical trials and peer-reviewed literature, and so on, and so forth, and there is, of course, a vast body of articles and books and papers as to what constitutes sufficient evidence.
We are searching for evidence that demonstrates the safety, the clinical effectiveness, and the comparative benefit if there are other services to which the service in question can be compared, compared to benefit of the service.
[Slide.]
Demonstrated effectiveness. What we are looking for here is three or four things. We sometimes don't get all of them. It depends upon the service in question and, to some degree, its level of development.
We are looking, first of all, for a positive evaluation of the benefits versus the risks, and this is done on a service-by-service basis. As I mentioned before, we have to prepare or we think we should prepare in order to make ourselves a little bit more clear sector-specific guidance documents.
One of the things that is the case here is that the sector-specific guidance documents themselves will outline the degree to which you need X number of people or approximately X number of people to power the study, and so on, and so forth.
So, we always get asked the question how much evidence do you need. The short answer--and we are not being flippant--is enough, and enough tends to vary with the service in question.
We are looking also for improved health outcomes, either generally for a broad spectrum of patients or perhaps for a particular group of patients. If a service, for example, provides a substantial benefit for a very narrow range of patients, that is easy.
If it provides a less substantial benefit or perhaps no discernible benefit for a broader group of patients, one of the things we can and do do is narrow our coverage to account for those patients for whom the service has proven medical effectiveness.
As time goes by and as the service matures or is tested and refined, we may, and we often do, broaden that coverage to include additional groups of patients for whom the medical effectiveness has been demonstrated.
Finally, if applicable, FDA has determined the service is safe and efficacious. FDA's determination is one step, but by no means the only step, as those of you involved in devices well know, in getting Medicare coverage.
FDA looks primarily at does it work. The result, of course, is that to the degree that FDA has done a premarket approval as opposed to a 510(k), there is evidence of the efficacy of the service.
However, as one of our medical directors always used to say, "so what," what does it do for a patient or a particular group of patients, and that, although FDA sometimes gets into that area, that becomes then our bailiwick and the thing that we look at most carefully in many cases.
[Slide.]
Medically appropriate. We are not approving services as FDA does for marketing. We are trying to integrate these services into an existing program. Several things are very important for the Medicare program, and one of these is appropriateness of the service.
First of all, the patient has to have a suitable indication for the service. This may sound rather A, B, C, kindergarteny, but believe me, you would be amazed at some of the claims that come in.
The service is suitable for, but not in excess of, the patient's needs. In those cases where the service is in excess of the patient's needs, the classic one-hour office visit where a 15-minute visit would have been more than sufficient, we can and do refuse to cover the additional amount.
The term used in claims processing for this is downcoding, recoding the claim to a lesser service.
The service is furnished by qualified personnel. We have, in both our statute and in our regulations, long, long descriptions of who these personnel are and what they are authorized to do either by statute or by regulations.
Finally, the service is furnished in a setting that is suitable for, but not in excess of, the patient's needs. Here again, this is a service-by-service question. Of course, we have services, for example, that are only reasonably furnished in hospitals. Others may be furnished in either hospitals or ambulatory surgical or outpatient surgical centers of one sort or another, and, of course, a good many of them in physician's offices.
[Slide.]
Appropriate evidence. Not all types of evidence are appropriate for all services. The so-called gold standard of randomized controlled clinical trials obviously are not going to be done on, for example, liver transplant patients, certainly not a blind or a double-blind study on surgical patients.
This sounds again rather kindergarteny, but we do get into big fights with people about this. Most people understand that, but it is worth repeating, that the amount and kind of evidence required is going to vary due to a number of factors, the nature of the service itself, whether there are alternatives available or not.
To be very blunt about it, if we have got a condition for which there is absolutely no alternative, the amount and kind of evidence is not quite as, shall we say, stiff or serious as the amount and kind of evidence for a service for a condition for which there is more than ample alternatives that apply to every patient who has that condition or illness.
The patient population likely to require the service. In some cases, the patients, of course, are obviously in very bad shape to begin with, and the kinds of rigorous trials that might be appropriate for other patients, may not be appropriate for them, and alternatives are found to be acceptable.
[Slide.]
Here is where we get into the real cat fights. Of course, again, what kind of evidence are you looking for? As I mentioned before, we are going to try and do this on a sector-specific basis because we are constantly arm wrestling about is this the right kind of evidence or enough.
In general--and I don't think we are too far off the reservation here--we have in the past ranked the kinds of evidence that are available in terms of the most authoritative to the least authoritative, and in doing this, I think it is worth emphasizing this several times.
This is not something that HCFA made up or the government made up. Ever since the program began, we have basically looked to the medical and scientific profession and asked them, okay, how do you decide these questions, how do you make up your mind about these kinds of things, and we have followed that pretty much over the past 33 and a half years.
The most authoritative, everyone I think agrees, maybe not, at least where it is appropriate, is the controlled clinical trial. The controlled clinical trial that has been published or at least accepted for publication or a condition for publication in a peer-reviewed journal.
There are various types of controlled clinical trials. Essentially, they all have the same feature, which is there is some sort of control, some sort of group that got standard medical treatment or did not get the treatment being tested.
All kinds of variations and flavors on this - blind, double-blind, and so forth, often depending upon the type of service being studied. A number of variables can affect the persuasiveness of these kinds of studies, the number and type of patient studied.
You will see controlled clinical trials with 20 patients, not as impressive as one with 200 patients generally. The statistical methodology that is used to come up with the results of the trial, the level of noncompliance, the dropout rates.
We have seen particularly in some surgeries crossover rates, as they are called, people in the control arm are allowed to cross over to the treatment arm, sometimes within three to six months, and so the long-term effect of the surgical procedure is pretty much lost in that kind of a trial.
You know how they were doing for six months, but after that, the control group or at least numbers and members of the control group, have in effect dropped out of the control group and moved over to the other arm.
[Slide.]
I have not gotten very original with my titles here. The next six slides you are going to see have the same title.
The age and health of the patients involved in the trial. We see a lot of trials, for example, where the cutoff age is 50 years old, and for the most part, it is another 15 years before you are entitled to Medicare unless you become disabled.
The inclusion and exclusion criteria where people are allowed in the trials that have multiple problems, others are restricted to people who only have the condition which is being treated.
This is highly important for the Medicare program because obviously, most of our patients have multiple problems. Believe me, I am 58 and I have already got multiple problems. I hate to think of what I am going to have when I am 65.
Internal inconsistencies. Dr. Holohan's famous diminishing denominator, we see it, unfortunately, even in peer-reviewed published studies where they start out with 287 patients, on page 2 there are 282, by the time they get to page 5 there are 266 with no explanation of what happened to these people in the interim.
These are the kinds of things you will see, and they are the kinds of things that raise your eyebrows and you say, hmm, how much weight do I give this controlled clinical trial.
[Slide.]
Another area of evidence are the assessments that we contract with, primarily with the Agency for Health Care Policy and Research, although we do contract with private organizations, as well, either through them or directly.
An assessment is usually quite extensive, but it may be rather limited. It is used when the amount and kind of evidence is either extensive, or in some cases limited, and there are modeling or other techniques used to try and get to some conclusion, or where the evidence is in serious dispute, either as to what it demonstrates or as to whether the studies involved were properly done or properly reported.
Assessments represent an informed third party review, an evaluation of the available evidence. They are useful to us both with respect to the fact that we are able to tap into expertise we don't have here, as well as one step removed. This is not HCFA making this evaluation, this is a neutral third party who has no interest really in the outcome giving us an evaluation.
[Slide.]
Evaluations or studies initiated by Medicare contractors. We have a number of very talented folks around the country who are facing the three and a half million claims a day that this program generates, and in the process of doing so, they are required to make, not only the daily decisions as to the claim, but also to develop what they call local medical review policies, which often involve things which are very akin to the kinds of national coverage policies with which we deal.
In performing this function, they may do it alone in their particular area, they may do it together in groups, and we have a number of work groups formed of our contractor medical directors who give us very valuable advice, not only with respect to the medical and scientific evidence available, but also where the rubber meets the road in terms of how the claims processing system handles these things.
[Slide.]
Further down the list, reports on case studies that are published or accepted for publication. What we are looking for here are limited types of case studies that present treatment protocols, and these vary in terms of how comparative they are, and quite frankly, they are sort of at the bottom of the food chain in terms of medical evidence. They vary in terms of their worth in terms of making a coverage decision - case controlled studies, comparisons of a series of case histories usually, a cohort study, treatment versus no treatment comparison.
There are a little different names for these, by the way. I was reading Dr. Garber's paper, and he at one point just gave up and said, under a variety of names that are under study.
[Slide.]
What we are looking for is slides that present treatment and perhaps patient selection protocols based on the evidence developed in the study, protocols as to who would and would not benefit from this treatment, and perhaps most importantly, whether the treatment is a late or last resort, or whether it should be moved quickly into standard treatment.
[Slide.]
Studies with very small numbers. Even if they are controlled clinical trials, we have seen controlled clinical trials, honest to God, with as few as 18 people in them. Whether they are done prospectively or retrospectively, individual case reports are generally not considered good evidence by most folks.
In summary, all evidence is not "equal" even amongst clinical trials. There are good ones, there are not so good ones, and then there are a lot in the middle, and so consequently, the fact that there is a--you know, we have got 20 clinical trials, what is enough?
One good clinical trial can be enough, 200 bad clinical trials can be less than we need. It all depends.
[Slide.]
Now we get to the fact that we have got these things, how do we look at them? We asked several questions. Obviously, as I mentioned, was the study published.
The posters at the conferences, we get those a lot. They are interesting, they are informative, but good medical evidence, not really.
What issues does the study present evidence for - is it looking at the clinical effectiveness of this treatment? Is it trying to determine what good patient selection is, its appropriate use, whether it's a late resort, a last resort, and so on, and so forth? How strong was the study, how big was it?
What was the study design? Was it a multi-center study or was it done only in one place? How was it implemented, how was the analysis done, and was the analysis sufficient, and does it hold up against the actual data that was produced by the study.
[Slide.]
How did the study relate to other studies? Usually, at least you would hope that if four or five people do controlled clinical trials in different places, their results will be very similar. Unfortunately, that is not always the case.
There are sometimes conflicts, and one of the things that you often look for here is, okay, why did those conflicts happen, are there some special factors that are not apparent from the study itself? Are there certain confluences that make sense, are there certain conflicts that don't appear to make sense? In short, you know you have to do some more digging.
Finally, is the evidence sufficient to reopen a coverage decision. One of the cautions, we usually, of course, only look at things that we have not covered in the past to see if they are covered from this point on.
But under our new process--and we actually have a live case here--when we make a coverage decision, anyone with an e-mail address or an envelope and a stamp can challenge that and say, hey, I don't think you made the right decision, I don't think you assessed the evidence properly, and so on, and so forth, and we reopen it and we look at it again. Again, good clinical trials can often change our minds rather substantially.
The thing you have to keep in mind is it may change our mind in the direction that perhaps the person who furnished it to us didn't want us to go.
[Slide.]
Just a word on medical versus clinical effectiveness. One of the things that we have to look at is whether the information that is presented to us--and this is extremely difficult to do, and quite frankly, I am not going to tell you we do it very well because sometimes we don't, but we do try--is whether this service is really what we would like to say ready for prime time.
It has been demonstrated to be effective in major medical centers where everybody is watching everybody else, and you have got the very best surgeons and the very best radiologists and the very best nurses, and so on, and so forth.
One of the reasons that when we cover transplants, we are very, very careful about the facilities in which we cover them, was there was obvious evidence that if you didn't have a facility that did it a certain way on certain patients, your success rate wasn't nearly as good as it should have been.
A service may be medically effective under strict protocols. It may be medically effective when done by very talented people in major medical centers. When it diffuses out to the community, it may lose some of that effectiveness - not always and usually not, but it can happen.
So, one of the things that you look for is not only the medical effectiveness as shown by the study, but try and predict the future a little bit to the general clinical effectiveness - how is this going to work when it gets out to the average hospital, the average doctor, the average nursing staff, and so on, and so forth.
To some degree, it usually doesn't matter. The availability of the service may override any diminution in the success of the service, particularly if you are looking at an area where there is no alternative, but in some cases, it can cause us to limit the coverage of the service to people who have had certain training, to facilities that have certain ancillary services. In short, we may not let this particular service just flow out there, but have some rules or some fences we put around it.
[Slide.]
I have pretty much gone through that, haven't I. These are often called appropriateness decisions as has been mentioned before.
[Slide.]
Cautionary tales, a few things to keep in mind. As I mentioned before, these rules are not our invention. They are our read on what the medical and scientific profession itself has come up with, and I think that is a very important point to keep stressing, because I have spent a lot of time at meetings where people are screaming at me across the table saying, hey, where did you come up with that. The honest answer is I didn't; the medical and scientific profession did.
Secondly, no coverage is static, believe me. Nothing ever gets settled around here. There is no final answer. The simple fact of the matter is that as new evidence is developed, as new techniques are developed as people develop ways of doing things that didn't work 10 or 15 years ago.
Coverage may come into existence down the road, it may be limited, it may expand, whatever.
Timing is important. I think if you paid any attention to our notice in April, it is obvious that when you come in here to make a formal request, you should have sufficient evidence for us to make a decision.
If you don't, the default position is no. If you are looking at evidence, and there isn't evidence or there isn't sufficient evidence, the no answer is the one that you would naturally gravitate to.
Assessing evidence is critical, it is complex, and believe me, there is three ways you can start a real fight - discuss religion, politics, or medical evidence. At least in this context, that certainly is true.
Any questions?
DR. SOX: I just want to give an opportunity to Executive Committee members who arrived late to introduce themselves - Dr. Eddy and Dr. Davis.
Dr. Eddy, tell us where you are from, what you do, and what your status is on the committee.
DR. EDDY: I apologize for being late, a little difficulty in finding the building, believe it or not.
I am an independent researcher and writer and speaker, and so forth. I am also a senior adviser to Kaiser Permanente, Southern California. My interests are in technology assessment and coverage decisions, and a variety of things. Other experience I have that might be pertinent to this committee is that I am a chief scientist for the Blue Cross/Blue Shield Association's tech program, which does its technology assessments. I am the Chair of the Diagnostic Imaging Committee or Panel, I believe.
DR. SOX: Thank you. And Dr. Davis.
DR. DAVIS: Thank you, Dr. Sox. I am sorry for being late, too. I just caught the red eye from San Diego, so if I look a little bleary-eyed, I apologize.
I am Ron Davis I am the Director of the center for Health Promotion and Disease Prevention at the Henry Ford Health System in Detroit. I am a preventive medicine physician and epidemiologist trained in public health and epidemiology at the CDC in Atlanta, formerly Chair of the Council on Scientific Affairs at the American Medical Association, and I am also the North American editor of the British Medical Journal.
DR. SOX: Thank you very much, Dr. Eddy, and Dr. Davis.
We are now going to hear from Harry Burke from New York Medical College. Welcome.
DR. BURKE: Thank you, Dr. Sox.
I am a consultant to HCFA, and I have no conflict of interest.
[Slide.]
I would like to begin by saying I am going to try and make two points in my presentation. I will make it brief because most of the panelists I think know pretty much what I am going to say.
But the first point is, is that much of what is going to come to HCFA has not been FDA approved, approved in the formal process, so the evaluation of safety and efficacy has not occurred prior to the request coming to HCFA. So, there is an adequacy of evidence required to establish at least that threshold because if the test, device or treatment is not effective, then, there is clearly no comparative benefit that it can have.
So the adequacy of evidence must first be adduced to that if it has not received FDA formal approval, and then secondly, the adequacy of evidence must be adduced for what I call its comparative benefit.
So, with that said--
[Slide.]
So, I am going to talk about comparative clinical benefit levels of evidence and presentation of evidence.
[Slide.]
Comparative clinical benefit, also known as reasonable and necessary, can be defined as a test, treatment or device providing a measurable improvement over all the current relevant tests, treatments, or devices at a cost commensurate with the measured improvement.
So, I think what I would really like to do is frame this in a relativistic context, and that is, if you come to HCFA, you have to talk about comparative benefit, in other words, how does your test, device, or treatment do compared to what is out there now and where you have measured its comparative benefit rather than just assumed it.
[Slide.]
Now, clearly, FDA approval is prima facie evidence for safety and efficacy, but of course it is not evidence for comparative clinical benefit. The FDA makes no claims of comparative clinical benefit, and clearly, many tests, devices and some treatments are used without receiving FDA approval. Many of them are off-label uses, but those off-label uses may very well present themselves to the HCFA for payment.
[Slide.]
So, what would our comparative clinical benefit study look like? Well, generally, you would compare the test, treatment, or device to all the other relevant tests, treatments, or devices in terms of safety and efficacy if not FDA approved and in terms of clinical benefit.
So, the studies that are presented have to show safety, have to show efficacy, and have to be comparative in nature. Otherwise, how would you know?
[Slide.]
Now, an issue that always comes up I think are side effects, balancing the risks versus the benefits. Now, it seems to me there is two approaches to that. One is to look at the benefit of the intervention and weigh it to the risks associated with the intervention, but another approach I think which is out there as well is to compare the severity of the disease to the risks associated with the intervention, and I think these two weighting mechanisms are very different. I think they are almost different in kind and result in different empowerments in the process.
[Slide.]
So, who decides what the balance is between severity of illness and the risks of the intervention?
Well, I think the regulatory agencies may very well decide the magnitude of the risk if it is appropriate for the severity of the disease, but I think the patients should be empowered to decide if they are willing to accept the risk associated with the intervention.
In other words, individual patients should be allowed to decide if the clinical benefit of the intervention is commensurate with the risk given the severity of the disease which has already been taken into account in offering the test or treatment to the patient.
[Slide.]
Now, clearly, randomized prospective clinical trials that are large are optimal results, but as we know, you know, they are rare, the entry criteria limit the generalizability, there is some problems with reproducibility.
[Slide.]
So, I mean I think we all know the problems with RCTs - the cost, the length of time, how do you deal with relatively uncommon diseases, low event frequency, and ethical issues.
So, I would like to make the point that I am not sure that we can rely on large prospective RCTs for every decision that is made. We would like to. So, I would kind of like to rehabilitate retrospective evidence a little bit maybe.
[Slide.]
So, I would like to suggest that properly replicated studies by independent investigators may provide strong scientific evidence by confirming study results.
In other words, you know, if you go back to the scientific method, the scientific method relies on replication for adequacy of evidence, can the study be replicated by independent investigators, and more importantly, was the study properly done and then replicated by independent investigators.
I think the real problem with retrospective studies is that properly done, you know, dealing with the various biases that can creep into a retrospective study, but I think we are sophisticated enough today in our methodologies that I think that we can deal with many of the biases of retrospective studies.
Ten, 20 years ago, you know, when RCT--well, 30 years ago, when RCTs were coming to the fore, it was clear that our statistical and epidemiologic knowledge wasn't sufficient to deal with the retrospective evidence, but I think we have come a long way since then.
[Slide.]
So, I would like to propose three adequacy of evidence levels - strong, moderate, and weak. Now, clearly, a large properly designed, implemented and analyzed prospective randomized clinical trial is strong evidence for safety and/or efficacy and/or comparative clinical benefit.
[Slide.]
But I would like to suggest that failing that, if that is not always possible, that a large properly designed, implemented, and analyzed retrospective clinical study replicated by independent investigators also in a large properly designed and implemented, and analyzed would also be strong evidence.
[Slide.]
Of course, two, medium sized randomized prospective clinical trials where one replicates the second would, of course, be strong evidence.
[Slide.]
Now, moderate strength of evidence, a medium sized RCT I think gives us a moderate belief. We have all seen how medium sized clinical trials have not always been consistent when they have been reproduced, so we can't really say that that is terribly strong evidence, because it has been overturned relatively frequently, but still in a very small minority of cases.
[Slide.]
A large properly designed and implemented retrospective study that has not been replicated would only be moderate evidence in this setting.
[Slide.]
A medium sized RCT that is replicated I think would add strength to the findings, and would at least give the appearance of a moderate strength.
[Slide.]
Weak evidence. I think small properly designed, implemented, and analyzed RCT, I think they are now recognized as relatively weak evidence. One of the reasons for meta-analyses is simply because small RCTs just don't do the job today that we would like them to do.
[Slide.]
Any retrospective study that is not large and hasn't been replicated would, I think, be weak evidence.
[Slide.]
Insufficient evidence. Small systematic studies, I call them exploratory rather than evidence. Case series are clearly anecdotal. Any study, no matter how big or what manner in which it is done, if it's not properly designed or implemented or analyzed, it cannot be used as good evidence.
[Slide.]
Well, "large," I define as 500 subjects, medium is 250 to 500, small is less than 250.
[Slide.]
I think that in this domain of HCFA, I think that many times medical supplies and devices, they are not tests or medications, they are relatively simple to assess, for example, a wheelchair, and I think they only need to demonstrate functional equivalence and equivalence in price in order to meet some evidentiary standard.
So, I think there are two things going on. They are the tests and medications which clearly require a higher standard than simple devices like a wheelchair.
[Slide.]
There is a real problem with presentation of evidence. One of the things that I have seen here at HCFA is when people come to HCFA and present evidence for a particular position, they don't present it many times scientifically. They present an ad hoc juxtaposition of many different types of studies, all the way from raw data to abstract to various types of publications, and I think that there really has to be systematicity in the actual presentation of the evidence to HCFA, because it shouldn't be HCFA's job to try and make sense out of a hodgepodge of stuff. It should be the person who is making the proposals job to create a cogent argument.
Any questions? Yes.
DR. FRANCIS: On the Drugs, Biologics, and Therapeutics Panel, when we reviewed the myeloma studies, we were specifically told that we were not to look at cost. So, one element of the comparative clinical benefit analysis that you just put up there dropped out. I wonder if you have any comments on that.
DR. BURKE: No, I did not consider cost either in my presentation, that's correct, so that would have been beyond the scope of the science involved in my presentation and move more into the politics of the process.
DR. FRANCIS: I thought your definition of comparative clinical benefit was measurable improvement at commensurate cost.
DR. BURKE: Yes, but I am not defining the cost aspect of it at this time. I am leaving that to HCFA, the cost comparison.
DR. SOX: I would like to ask Dr. Hill to comment on the issue of cost.
DR. HILL: I can confirm what Dr. Francis is saying, that we do not consider costs as part of this equation as we are currently working the decisionmaking process.
DR. BURKE: Measurable benefit, if you can measure the improvement in something and then later on just deal with the cost.
DR. SOX: Dr. Eddy.
DR. EDDY: Thank you, Dr. Burke.
I want to make sure I understand your definition of clinical benefit. Did you say that it requires measurable improvement over the current relevant tests, and a later slide talked about all relevant alternatives?
Does that mean that if something is effective, but not quite as effective as another treatment that is already out there, then, it does not have clinical benefit by your definition?
DR. BURKE: That would be correct.
DR. EDDY: So, if TPA had come out first, then, streptokinase would not have clinical benefit?
DR. BURKE: Well, again, you are addressing the cost?
DR. EDDY: No. Let's just assume that TPA is a little bit better than streptokinase.
DR. BURKE: That is correct. Duplicating what is already out there with a less effective agent would not be a measurable benefit, that's correct.
DR. EDDY: So, the benefit is always defined in terms of a comparison with an existing technology, not in terms, by your definition, not in terms of a comparison with the natural history or the untreated condition or a placebo.
DR. BURKE: That's correct.
DR. GARBER: This is really by way of clarification. Your highlighting suggested that in the study designs that were "less than clinical trial," like the retrospective studies, the key issue is replicability, but your language earlier in the sentence says that involved replicability, said properly designed, and I am hoping that you will clarify for the committee what you meant.
I think a prime example of the problem here is the studies of hormone replacement therapy for postmenopausal women where there are a number of fairly well designed observational studies that were quite consistent and remarkable consistency among a large number of studies showing that it prevented heart disease and lowered all-cause mortality, and the first randomized clinical trial contradicted all of those.
So, perhaps you could clarify your ranking of the design issue versus replicability.
DR. BURKE: Right. I mean if, in fact, everything can be assessed by an RCT, then, I think that is clearly the way to go. Okay. But the issue I am addressing is what if it can't, what are you going to do then. Okay.
So, I am not saying that they are comparable theoretically, clearly, they are not, but in the practical world in which we deal with, it is not always possible to have a large RCT, and if it is not possible, then, what do you do, what is your adequacy of evidence at that point, and that is the issue I was stressing.
DR. FERGUSON: If I understand correctly, a well conducted randomized clinical trial with 249 people in it would be weak evidence.
DR. BURKE: I am not determining how many hairs makes a bald man. I mean the 250 is a relatively arbitrary number. What I am trying to do is make a distinction between large, medium, and small, and whatever the committee thinks those numbers should be is fine.
I was just trying to put something down, so we had some frame of reference.
DR. SOX: Dr. Eddy.
DR. EDDY: I would like to ask you a few more questions about the role of retrospective studies. As I listened to you, I didn't hear you say anything about controls in the retrospective studies, so you are including retrospective studies that have no controls, like a review of records, the clinical series.
DR. BURKE: I am suggesting it has to be properly designed, and I am not specifying the proper design. I am just suggesting that if it can't be properly designed, it shouldn't be done, and/or if it is done, it shouldn't be good evidence.
DR. EDDY: So, the issues about what the proper design of a retrospective are still open.
DR. BURKE: Exactly, and I think that is something that the committee probably has very strong thoughts about.
DR. EDDY: One more question, if I may.
Let's imagine that a randomized controlled trial could be done, but hasn't been done. It could be done, it is feasible, say, it takes two or three years or something like that, you can get the appropriate sample size, but let's say it has not been done, but let's imagine that there had been some retrospective studies of proper design, however we define that, but they are not randomized, so there are always remaining questions about random biases.
Let's imagine they had been replicated. Would you consider that to be strong evidence?
DR. BURKE: Yes, I would. In other words, if there are large, properly designed, dealt with all the potential retrospective biases that could occur in a study, was replicated by independent investigators in a separate population, okay, also properly designed and large, yes, I probably would.
DR. EDDY: I guess the question is whether you would know whether all the biases could be--
DR. BURKE: Right, and that's a judgment call, and I mean that is something that people have to actually look at the study design specifically on a study by study basis.
DR. SOX: Ms. Richner.
MS. RICHNER: The replication issue is important, too, in terms of who would ask or pay for replication of studies for evidence, for instance, for this coverage committee, et cetera, that is a question of mine, and then also the types of evidence that are going to be required, Dr. Eddy just addressed my concern, whether or not retrospective data would be--
DR. BURKE: I think that it is up to the person, as Ron pointed out, it is up to the person, the group proposing that something be paid for, that they provide adequate evidence. So, the standards for adequacy have to be out there, and they have to decide whether they believe that they meet those standards.
MS. RICHNER: And this would forego FDA requirements or be beyond what the FDA would require.
DR. BURKE: They would have to meet efficacy standards. If they have not been met, they must meet them, and this is a really critical issue because at our last meeting, we spent all our time on efficacy, and nothing on comparative benefit, okay, because it had not been established, efficacy had not been established, so that is a really, really important issue if it hasn't been established already.
DR. SOX: Dr. Holohan.
DR. HOLOHAN: Let me get back to point that Dr. Eddy had raised that I would like a little more detail on.
You had made the statement we have techniques today to deal with biases. Virtually all case series or retrospective studies have biases, unintentional or otherwise, and we have kind of glossed over how we deal with those biases.
Let me give one example. Large retrospective studies replicated, that are in fact in a different patient population, for example, evidence submitted to Medicare based on retrospective studies where the oldest patient was 55, and we know that there aren't very many Medicare beneficiaries who are that age group.
How can we extrapolate that intrinsic bias of the study and come to any conclusion as to whether, in fact, it applies to the population we are concerned with?
DR. BURKE: I don't believe that retrospective studies are the best standard of evidence, let's be clear about that, but I think that in the real world, they are going to wind up being a standard of evidence whether we like it or not, and the issue is I think 20, 30 years ago, we didn't know most of the biases that could creep into retrospective studies.
Today, we know, I think, most of the biases that can creep in, and I think we can evaluate the studies and see if they were able to deal with those biases, and if we believe that they were not successfully able to do so, then, it is not a properly designed study.
DR. SOX: I am going to ask for one more question, then, we will move on, and we will get a chance to get you up here again and ask you questions during the open session.
Daisy, why don't you go ahead.
DR. ALFORD-SMITH: I just wanted really for you to repeat what you had talked about in reference to medical supplies.
DR. SOX: Could you repeat the question?
DR. BURKE: The question is, is medical supplies and devices, and my point was, was that cotton swabs, for example, or wheelchairs, if they are functionally equivalent, I don't think that we should apply a high level of evidence requirement in order to pay for a different swab or a different wheelchair assuming that there is functional equivalency demonstrated.
DR. SOX: Dr. Garber, a brief clarification?
DR. GARBER: I will skip my real question, but just to the point of clarification. When you say "retrospective," I believe you mean any kind of observational study, not solely retrospective, including prospective registry, et cetera.
DR. BURKE: Right.
DR. GARBER: Thank you.
DR. SOX: Thank you very much.
One of our members has arrived a bit late. Would you introduce yourself, say where you are from, what you do, and whether you are a voting member or a consumer rep or manufacturer's rep.
DR. PAPATHEOFANIS: I am Frank Papatheofanis. I am a faculty member at the University of California, San Diego, in the Department of Radiology. I am the vice chair of the Diagnostic Imaging Panel, and I am a voting member.
DR. SOX: Thank you very much.
Before we move on to the open public comment section, I would like to suggest where we ought to be going. One of our jobs is to provide guidance to the panels and to coordinate the panels, and in respect to the issue of standards of evidence, I think this could mean probably two things and will.
One is some form of written expectations or principles of the type of evidence that is required in order to make a recommendation for coverage, and the second is what might be called case law. That is, it is our comments on the proposals for coverage that come to us, and I suspect that over time the combination of some sort of upfront written expectations about the standards of evidence and the so-called body of case law that we develop as a result of our comments as we will undoubtedly do this afternoon, will make up the form of guidance to the panels that will enable them to function effectively.
So, I think in general that is where we are going. Early in the afternoon, I am hoping that somebody on the panel is going to give us a written proposal that we can discuss and vote on, so that we can begin the process of helping our panels to function in a roughly similar way across all panels.
So, with those brief comments, let's move on now to an open session, and we are going to have two speakers. Each are going to speak for 10 minutes, and we will have time for questions after each speaker.
I would like to remind each speaker that they should state if they have any financial obligations with the manufacturers of any products being discussed or with their competitors.
We will start with Dr. Greg Raab from the Health Industry Manufacturers Association.
DR. RAAB: I would like to be accompanied by our counsel, Brad Thompson, from Baker & Daniels.
Thank you. My name is Greg Raab. I represent the Health Industry Manufacturers Association. It is a pleasure to speak to you this morning at the first meeting of the Executive Committee.
I am particularly pleased to be here because the Payment Committee of HIMA's Board of Directors has been discussing this week the very issue that you are considering this morning.
Before I share with you HIMA's views on Medicare coverage criteria, I want to applaud the considerable progress that HCFA has made over the past year in establishing a more transparent and predictable national coverage process.
The chartering and the establishment of the MCAC, and the publication in the Federal Register of a coverage process notice, help give the public an understanding of at least the basic rules of the road of Medicare coverage.
In addition, I want to thank MCAC for fostering an open coverage review process by providing ample time at panel meetings for beneficiaries and outside experts to speak. This assures that beneficiaries have a voice in these decisions, and HIMA encourages MCAC to continue this pattern of openness in upcoming panel meetings.
Given the credentials of this panel and the pivotal role MCAC plays in the coverage decision making process, I am most certain that you already appreciate the fundamental importance of coverage criteria.
These criteria will directly influence the capacity of many medical device companies to undertake and develop new products because HCFA's data demands will directly affect the cost, length, and likely success of the innovation process.
As you know, Medicare's coverage criteria have not been spelled out for the public, and HIMA believes that they need to be made explicit. HCFA announced in its April 27 coverage process notice that it will undertake formal rulemaking to establish coverage criteria.
Originally, we expected that a proposed regulation would be published before the first MCAC panels began meeting. This obviously did not happen, and while HIMA appreciates how acutely the MCAC panels must feel the lack of coverage criteria as they hold their first meetings, we nevertheless must stress the importance of following formal rulemaking in crafting these standards.
I want to emphasize to you that the proper forum for developing these coverage criteria for the Medicare program is the federal rulemaking process, not the deliberations of this advisory committee.
The rulemaking will allow for broader public exchange of ideas to shape an appropriate set of substantive criteria, well beyond what can occur in this room. This is particular important given the sharply divergent views that, in all likelihood, exist on this subject.
It would seem incongruous for HCFA to issue any directives at an MCAC meeting regarding the agency's views of the "proper" substantive criteria to be used without first allowing for public input pursuant to the rulemaking requirements. HCFA must be diligent to avoid tainting the public process or using the podium to announce what are, in effect, substantive rules.
In addition to the risk that HCFA might prematurely announce new rules that should first undergo rulemaking, we are concerned that HCFA is not properly using the MCAC Executive Committee.
The MCAC charter and the HCFA policy statements make clear that MCAC is set up to offer advice to HCFA on technical matters - it has not been established to recommend policy for the agency. This body should not develop, nor rule upon, the criteria that the various MCAC panels are supposed to apply.
We applaud HCFA for undertaking rulemaking to develop national Medicare coverage criteria, and we hope that the agency will allow the rulemaking's public process to go forward as intended, without extra-procedural influences that may indicate agency prejudgment.
I would like to at this point pause in my statement before I get into a few specific guidelines as Medicare coverage, what we think Medicare coverage criteria ought to be, and ask our counsel to comment on the proceedings. Brad.
MR. THOMPSON: I apologize. I am from Indiana, so I speak very bluntly, and I will speak bluntly to you now.
I think this panel is in a bit of a predicament, and that may be an understatement actually. There are a number of federal requirements that apply to how these policy issues get resolved.
Greg just explained to you that there is an ongoing rulemaking, and the rulemaking is a very public process that goes well beyond this room. That rulemaking is the place to ventilate these issues. Doing this kind of discussion before that rulemaking is completed in effect short-circuits the rulemaking, and that just isn't allowed.
To be very specific and very practical about this, the presentation that you just heard from Dr. Burke--and I am not picking on Dr. Burke here--but I assume that he is speaking on behalf of the agency, I see him listed as the HCFA presentation, and he described himself as a consultant, so I assume he has the mantle of the agency, and his remarks were in some fashion pre-cleared by the agency.
Those remarks are very problematic for two reasons, not the least of which is the Administrative Procedures Act, but more fundamentally, the fact that a lot of what he and the agency are advising you on in that presentation is legally incorrect.
His description of the federal Food, Drug, and Cosmetic Act is simply legally incorrect. His description of the impact or the proper criteria for reasonable and necessary, the requirement, for example, that new technologies be better than existing technologies is legally incorrect.
Now, if this were rulemaking, we would file elaborate written comments and cite all the law and provide you with our analysis. You take it back, you would read it, and you would study the act yourself, and you would reach a conclusion. You can't do that here. That is why the rulemaking is more suitable for resolving issues like what the criteria and the evidence ought to be.
I also heard price discussed in the context of it, and Dr. Burke clarified that he wasn't offering an opinion on cost effectiveness. It was in the remarks, it was in it a couple different places, and I would urge you to disregard cost at this juncture because that is very problematic and most likely illegal.
So, the predicament that you are in is everybody is assembled here, I know you want to get some business done, there are valuable things that you can do, but outlining--and Chairman Sox, I am referring specifically now to your objective of the day of coming up with a written concept of what the evidence ought to be--that objective is very problematic.
You can certainly share thoughts about what the different panels are doing, the issues that they are facing, trying to figure out ways to harmonize the decisions that you are making right now, but coming up with proscriptive requirements for future decisionmaking short-circuits the public process.
That's all.
MS. LAPPALAINEN: I would like to remind the speakers please state your name.
MR. THOMPSON: I apologize. Greg introduced me. I am Brad Thompson with Baker & Daniels, and I do have a financial interest because I am engaged by HIMA.
MS. LAPPALAINEN: Thank you. And if you could also please state the mission of the Health Industry Manufacturers Association?
DR. RAAB: The Health Industry Manufacturers Association is a trade association representing more than 800 manufacturers of medical devices. It is based in Washington, D.C.
MS. LAPPALAINEN: Thank you.
DR. SOX: I just want to remind you that his time counts against your time. You have got about seven more minutes.
DR. RAAB: About seven more minutes? What I would really like to do at this point is highlight a few of the key principles that HIMA believes should be included in the Medicare coverage criteria regulation. We hope that these principles will guide HCFA as it develops a proposed rule, and we think that, taken together, these principles might serve as a useful yardstick against which the proposal can be assessed.
First, HIMA believes that coverage criteria should put the patient first. This means a product's clinical effectiveness should be the determining factor for HCFA in judging whether a product is "reasonable and necessary," and covered by Medicare. We believe this because this judgment, whether or not to provide beneficiaries access to a product or service, is fundamentally a patient care decision.
Economic factors should play no rule in this decision. Coverage decisions should not be used as a way to limit overall Medicare expenditures, this budgetary role is reserved to the Congress. It is the Congress that allocates the funds for Medicare's payment systems.
Let me be emphatic on this point. We see nothing in the Medicare law giving HCFA authority to make coverage decisions based on economic information. As you go about your work advising HCFA on coverage decisions, you should be guided by the clinical effectiveness of a product or service, not its cost or its cost effectiveness. Economic factors are more appropriately considered in the context of payment.
Let me stress also that the coverage determinations you help HCFA make can be undercut if HCFA's coding and payment systems do not result in timely decisions and fair reimbursement for the technology or service in question.
We at HIMA are concerned that beneficiary access to covered services is sometimes placed in jeopardy because these technologies are not properly integrated into the Medicare program.
Second, HIMA believes that clinical evidence used in making coverage decisions should be reasonable, clinically relevant, and collaboratively developed.
HIMA believes that the evidence gathered as part of the FDA review process to demonstrate a new product's safety and effectiveness should in many cases be sufficient for Medicare coverage. This has been the case in the past, and we expect the new HCFA rule to recognize the importance of this information.
Further, we believe that Medicare should not duplicate FDA's review. HCFA should not reconsider, or otherwise challenge, an FDA determination that a product is safe and effective.
With respect to other data that may be required for coverage decisions, HIMA believes that these requests should be grounded in common sense. HIMA believes that HCFA should certainly ask for the data it needs to determine that a product or service is reasonable and necessary for patient care, but it should avoid demanding excessive or unrealistic amounts of data.
If HCFA demands more than is truly necessary, the data themselves, in a sense, become a hurdle to innovation. For this reason, HIMA recommends that specific evidentiary requirements be developed with the involvement of clinicians and product innovators. Further, these requirements should be tailored to the medical treatment, technology, or procedure under review.
Evidentiary requirements should also take into account the practical impediments, that is, the time involved, the cost, and the patient impact, to the development of this information.
With respect to hierarchy of evidence, which was discussed earlier, HIMA believes that the clinical evidence used in making coverage decisions should be marked by the same innovation and flexibility that mark the technology development process itself.
This means that technology assessments, based on peer reviewed randomized clinical trials, may not be the best way to assess the clinical merit of a new technology, and that ways must be found to solicit the input and experience of practicing physicians, the insights of medical specialty societies, and the experiences and observations of the inventors themselves.
Agency for Health Care Policy and Research Director John Eisenberg summed up this point nicely in a recent article in the Journal of the American Medical Association. I would like to quote from that article.
"Those who conduct technology assessments should be as innovative in their evaluations as the technologies themselves. There is little argument that the randomized clinical trial is an accepted high standard for testing effectiveness under ideal circumstances, but it may not be the best way to evaluate all the interventions and technologies that decision makers are considering."
Eisenberg concludes by saying that "the randomized trial is unlikely to be replaced, but it should be complemented by other designs that address questions about technology from different perspectives. Researchers need to develop and test new ways of evaluating technologies that can be accomplished quickly and can take advantage of emerging databases and information needs."
Third, HIMA believes that Medicare should amend its current policy on investigational devices subject to investigational device exemptions at the FDA by making payment for all Category B non-experimental or investigational technologies.
This would eliminate the current uncertainty that exists regarding whether or not these products--which represent incremental as opposed to breakthrough improvements--are made available to Medicare's beneficiaries.
Fourth, HIMA believes that HCFA should not make national non-coverage decisions until it has definitive information that the product or service is not effective or that it causes patient harm.
National non-coverage decisions can cut short the development of valuable clinical information.
Finally, HIMA believes that coverage restrictions, through appropriateness reviews, when necessary, should be well grounded in clinical evidence and frequently updated.
HIMA recognizes that HCFA will occasionally decide to set limits on the availability of a covered technology or procedure to ensure what the agency believes is appropriate use.
HIMA believes that HCFA should establish such limitations only if they are supported by medical evidence, and only if the restrictions are consistent with the advice of medical specialty societies. Further, HIMA believes that HCFA should make available to the public the rationale and justification for any restrictions it imposes.
Given the rapid pace of change in the technology industry and the way care is delivered, coverage restrictions must be updated frequently if they are to remain clinically relevant. Coverage limitations should be updated or revisited annually at least if they are to be kept in effect.
This concludes my presentation. Thank you for permitting me to share with you HIMA's views.
DR. SOX: We have got about a half-hour to complete the open public session. I think I will entertain about 10 minutes for questions or comments on Dr. Raab's presentation.
Dr. Hill.
DR. HILL: Thank you. I hope it is understood that our choice not to engage in debate with counsel does not imply our agreement with his assertions necessarily or our disagreement.
While I don't want to engage in that kind of interplay because it involves him personally, I would ask the Chairman's indulgence if Dr. Burke could clarify his situation and whether or not his remarks were pre-cleared and if you would go to the microphone and state whether or not this was an independent suggestion you made.
DR. BURKE: This is an independent assessment of adequacy of evidence for HCFA. I was not speaking--thank you for giving me the opportunity to clarify that--I was not speaking for HCFA when I made my remarks.
DR. HILL: Thank you. Last, if I may, would you care to offer, Dr. Raab, a suggestion as to where in the hierarchy of evidence you think these more creative or these more novel forms of evidence should lie?
DR. RAAB: Hierarchy is a difficult term. I think there is a range of evidence heights that fit and should be used. The word "hierarchy" implies that one is better than another. What might be best for a particular technology, in a particular circumstance, might not fall at one end of the scale. Items should be considered appropriate for the technology.
DR. SOX: Dr. Eddy.
DR. EDDY: I am going to ask two questions which I hope can be answered very briefly, because I want to make--I am going to start with this one.
I am looking at your fourth recommendation, which if I read it correctly, has huge implications. Basically, it flips what is the common burden of proof.
Do I hear you or do I read this correctly that something should be covered unless there is good evidence that it is ineffective or causes harm?
DR. RAAB: I don't understand your point.
DR. EDDY: Would you repeat the fourth principle for us?
DR. RAAB: HIMA believes that coverage restrictions, when necessary, should be well grounded in clinical evidence and frequently updated.
DR. EDDY: No, I am sorry. HCFA should not make a national non-coverage decision--
DR. RAAB: --should not make national non-coverage decisions unless it has definitive clinical information that the product or service is not effective or that it causes harm.
DR. EDDY: So, is that a recommendation that something would be covered unless there was evidence that it was not effective or caused harm?
DR. RAAB: It is a recommendation that HCFA view the importance of a national non-coverage decision, that there is a sense in industry that in the past, HCFA has made national non-coverage decisions which have headed off, stopped and halted the development of information, that there was no information saying that there was any harm or problem.
For instance, products may be reviewed locally. Medicare contractors could be covering costs. HCFA might understand that that is happening and issue a national non-coverage decision which would halt this coverage.
DR. EDDY: Let me rephrase it. Let's imagine that there is an intervention, and there is no evidence yet of safety or effectiveness. Are you recommending that HCFA should cover that or not cover it?
DR. RAAB: There is no information on safety and effectiveness?
DR. EDDY: There is no information on safety or effectiveness.
DR. RAAB: I think a better situation would be an FDA-cleared product that is considered locally by a Medicare contractor. It is covered there, but in absence of a national Medicare decision.
DR. EDDY: I will try once again. Let's say it is not in the jurisdiction of the FDA, so it's a device. There is no evidence of safety or effectiveness. Do you think it should be covered? I am just trying to understand the principle that you are recommending to us.
DR. RAAB: I guess I am not tracking with your question. Brad, do you understand this?
MR. THOMPSON: I think so.
MS. LAPPALAINEN: For example, what if we have an exempt device in front of us for a coverage decision. This is a device that is, by law, exempt from the FD&C Act.
DR. RAAB: There is two elements to this. One is there are exemptions to the federal Food, Drug and Cosmetic Act for products which, in the view of Congress, don't require safety and effectiveness data to be lawfully marketed, because they don't pose a risk. In that case, I would say that there is some evidence there is at least a congressional determination that this product falls into that category.
But the second and more important element to your question is remember there is three potential outcomes - there is national coverage, there is national non-coverage, and then there is hands off and allowing the local process to make the decisions.
The point of this bullet point is that if this committee doesn't have evidence which suggests that it is unsafe, it ought to allow the local process to continue to decide whether or not to cover it.
So, we are not saying it is automatically covered. We are saying this committee shouldn't elevate it to a national decision and make a national non-coverage decision. It should let it continue to percolate through the local system and let the local contractors decide whether or not to cover it.
DR. SOX: That is a pretty clear answer to your question.
DR. EDDY: I now understand the answer.
DR. SOX: Dr. Francis.
DR. FRANCIS: I want to press you on the distinction between a policy judgment, which you think requires notice and comment rulemaking, and a technical judgment. Maybe the way to do that would be to start just by asking you about some of the points in Dr. Burke's presentation.
Would, for example, the size of a randomized clinical trial, whether it's a small trial, if it's under 250, for example, be a technical or a policy question, and then maybe you could go on to the issue of some of the classifications as strong, weak, or moderate strength.
Again, are those the kinds of choices that you are advocating should be subject to notice and comment rulemaking or are they technical?
DR. SOX: I just want to remind you that you have got four minutes until the end of the discussion period.
DR. RAAB: I will be brief. In the law, the dichotomy between rulemaking and not rulemaking is not policy versus technical, so I am afraid I can't answer your question directly because that is not the legal framework.
The legal framework is whether it is a substantive rule or not. It becomes a substantive rule if it is prescriptive. So, if the agency said there must be 250 individuals in a trial, that is prescriptive, that is a rule, that requires rulemaking.
If a group of scientists are debating relative size and power of a study, and don't lay out prescriptive requirements, that is a technical discussion, and that doesn't require rulemaking. I hope that is responsive.
DR. EDDY: Can I try one more quickly?
DR. SOX: Please do.
DR. EDDY: This is a question to Mr. Thompson.
I think you said that it might be illegal for the panels to consider cost.
Did you mean that and, if so, what is the basis for that?
MR. THOMPSON: Part of the difficulty here is that this discussion is premature. My law firm is engaged right now in writing a legal brief on whether cost is a permissible item to consider under the reasonable and necessary standard.
Thus far, I can tell you, and there is thousands of pages of legislative history, thus far, I can tell you that it is our opinion that Congress did not intend cost effectiveness to be considered in the context of reasonable necessity.
I have not yet finished my legal analysis. By the time the rulemaking rolls around, I expect to have it done, but as of right now that would be my assessment, but it is a preliminary one.
DR. SOX: Briefly.
DR. EDDY: So, would this mean that--let's imagine that a technology did not meet Dr. Burke's definition of clinical benefit, that it was effective compared to no treatment at all, but wasn't quite as good as some other product that was out there, but it was much less expensive.
Would this say that we could not consider cost effectiveness?
MR. THOMPSON: Well, see, the other part of your scenario is that this comparability issue, that that is a basis for the decisionmaking, and I heard Dr. Burke say that new technologies needed to be superior to existing ones.
I would challenge that opinion, as well, on the basis of the law, that new technologies do not need to be superior, they can merely be comparable. Sometimes you can come up with a more or less equivalent mousetrap, and that equivalent mousetrap ought likewise to be covered.
So, there is a couple of different elements to your question, and I would say that if it's in the realm of clinical comparability, not superiority, it ought to be covered regardless of cost. The cost element comes in later in the equation, and the agency has a lot of tools in the payment arena to decide how much it will pay for a technology, but that is a conceptually separate issue.
I am not saying cost is irrelevant. I am just saying is occurs at a later point in the regulatory process.
DR. SOX: Thank you very much. I think we will close the discussion period from your presentation.
Linda?
DR. BERGTHOLD: I would like to get him to clarify this high level of evidence issue.
Are you saying that if there is a large randomized controlled trial, that there would be any situation under which that would not be the best level of evidence? You talk about creativity of evidence sources, but if you have a large RCT, is that not the best level?
DR. RAAB: An RCT, if it is available, should be used. The issue is in a coverage situation, to demand of a product sponsor upfront sorts of data, when other data may be available and just as good to make the decision.
DR. SOX: Thank you.
We will move on. Dr. Larry Weisenthal from the Weisenthal Cancer Group will be the next commentator. I would just remind you to disclose any financial involvement that you may have.
Larry M. Weisenthal, M.D., Ph.D.
DR. WEISENTHAL: My name is Larry Weisenthal. I am a medical oncologist in Huntington Beach, California. I am essentially in a laboratory-based private practice. I am entirely self-supported. I do have a conflict of interest in that I provide one of the services that has been under consideration by MCAC.
DR. SOX: Sir, do we understand that your remarks are going to be general now and then later--
DR. WEISENTHAL: I am going to address that in my introduction.
DR. SOX: You do have allotted time for discussion of the issue of tumor--
DR. WEISENTHAL: Yes, I discussed this with Sharon Lappalainen before making my talk, and I understand the purpose of the morning session, and I will try to stick with the spirit of that.
In his presentation, Ron Milhorn, one of his slides showed that not all types of medical evidence of medical effectiveness are appropriate for all services, and he kind of went by that quickly, but I think that is important.
The idea is that you can make these decisions very mathematical, as Dr. Burke is quite able to do, but everything needs to be considered in context. There is four specific issues that I want to cover in my remaining nine minutes.
One is the idea that you need to compare the levels of evidence relating to the new method as contrasted with the levels of evidence which exist to support the old method. In other words, it is not fair only to consider what is the evidence that supports the new, but how does that compare with the evidence that supports the old.
Secondly, it is very important to define the relevant dataset of evidence to consider, and this particularly applies to a situation in which there exists not, let's say, just a few pieces of evidence, so you are going to consider everything, but rather you have a large array, let's say, 100 different very small studies.
So, the question is which of these data are relevant, which should be included and which should be excluded. I think in the MCAC meeting earlier, all of my complaints relating to that have to do with the fact that there was not an agreement on which was the relevant dataset.
Honest people can disagree over the interpretation of data and sometimes you just can't agree, but I think that in most cases, it is possible to agree in advance on what is the relevant dataset to consider.
I think that in the meeting that we just had, we could have, with just a little preliminary communication, both sides could have agreed that these are the relevant studies which should be included, these are the irrelevant studies which would should be excluded, and then the whole process would have been much more clear to the panel and I think more satisfactory to everybody.
So, I would urge that in the future, that in advance of the meeting, that there at least be an attempt to reach an agreement on what are the relevant datasets to consider.
Thirdly, it is very important to consider conflicts of interest in those presenting evidence, and there is a natural tendency to focus in on the conflicts of interest or proponents, such as me, who might have a commercial interest, but it is important to ask that question of everyone presenting evidence is there a conflict of interest.
Lastly, it is very important to consider the need for the service being proposed and therefore, to consider the risk in not providing an opportunity for the proposed service to compete with existing services.
Now, that is what I am going to cover. I will illustrate each of those points with an example from the service that I am here in the afternoon to represent, but the purpose in doing this is not to argue the issue, and I think you will agree that I am presenting a balanced presentation, but rather just to illustrate why each of these points is important.
Firstly, comparing levels of evidence relating to the new service compared to that supporting the old service. On Thanksgiving, I went down and had Thanksgiving dinner with my friend who is a gastroenterologist, and his wife was is a physician, too, and these are excellently trained physicians, Johns Hopkins, Case Western Reserve, Tufts, Boston University, well-trained physicians.
The husband, who is one of my best friends, I was the best man at his wedding, is a gastroenterologist, and he performs a procedure called colonoscopy. Fifteen years ago, Medicare paid him $350 to perform a colonoscopy. Today, it is about $115, it has been cut down by two-thirds. $115 is about 15 percent more than a family practitioner gets to perform a flexible sigmoidoscopy. Flexible sigmoidoscopy is a very easy procedure, you only go in about 25 or 30 centimeters, whereas, a colonoscopy can be very challenging, you want to go in to 100 centimeters.
So, my friend told me something very interesting, and that is, that it used to be in the old days when they got paid $350, it was sort of a point of professional pride and responsibility that you tried to visualize the entire colon. This was very important, and if it took you 45 minutes or an hour, you did it.
Today, he tells me nobody does that. They figure they are getting 15 percent more than going in 30 centimeters, so their responsibility is to go in and do about 38 centimeters and if they encounter any problems and it takes them more than eight minutes, they just send the patient up to a barium enema.
What happens then? The patient has to undergo two bowel preps. Medicare ends up paying for two procedures, colonoscopy plus barium enema, and sometimes three procedures because the barium enema may then reveal a proximal lesion, and the endoscopist has got to go back in and rebiopsy that.
So, he was explaining this with a lot of bitterness, and he voted for Bill Clinton twice, he blames Bill Clinton somehow for this, so he says he is never going to vote for a Democrat again, but there is a lot of bitterness in his heart.
He tells me that he is in a multispecialty group and all the other specialists, the cardiologists, the infectious disease people, the endocrinologists, they are all having the same problem. They all hate medicine, they don't want their kids to go in it, and so forth.
Now, this is relevant to this consideration, because there is one exception. There is one group of specialists that he says are doing very well, and those are the oncologists. Why are they doing well?
Well, they are doing well for the following reason, that is that most chemotherapy in this country is given as an outpatient by oncologists in their office, and what happens is, is that they get reimbursed, not just for providing the service, but they get reimbursed for the drugs. The drugs are very, very expensive, and typically, with most insurance plans, they would get reimbursed by some formula relating to the average wholesale cost.
Now, those of you who know the oncology literature know that there is rarely a situation in which there is one form of therapy, and only one, which has proven effective, and particularly you get into second-line therapy, there is no situations where this is standard second-line therapy, and if you just look the PDQ, which the NCI publishes, which is supposed to be state-of-the-art treatment, you can find multiple different forms of therapy.
So, you could flip a coin and be equally well off or equally supported by the literature in choosing therapy. How do they choose therapy? It is on the basis of the spread between the average wholesale cost and what they get reimbursed, so you have got a choice of drugs, and you are in an environment where doctors are getting killed or they are having trouble making their mortgage payments, much less saving up for retirement. And you don't think that that is going to enter into their decisionmaking? It does.
So, basically, the competing paradigm, the new thing that is being proposed, is you test the biology of the tumor, and you choose the treatment based on what is tailored to that individual biology.
The old paradigm is you either flip a coin or, more insidiously, you look at the spread between wholesale cost to reimbursement, and you choose it on that basis.
Now, what does it take to support Medicare reimbursement for a therapy? Typically, two papers published in the literature, these are not randomized papers, but let's say an oncologist wants to use gemcitabine in sarcomas. He just has to usually produce one or two papers showing that, yes, gemcitabine has been used in sarcomas.
This is the level of existing evidence, and I think that when you consider the new paradigm, you have to consider the levels of evidence relating to the new paradigm compared to the levels of evidence relating to the old paradigm.
Let's see, I have only got two minutes, so I really have got to hurry up.
Dataset is very important. I don't think I need to say anything more on that. It is just that it certainly is possible in advance to agree on a relevant dataset, and I will just leave it at that.
Conflicts of interest in those presenting evidence. There is the tendency to think that anybody that is providing a service wants to have it covered for his or her own selfish purposes. You want to have the service covered, so that you can get paid for it.
Certainly, that applies to things maybe like bone marrow transplantation, but it doesn't apply to everything. I daresay that you have very few ophthalmologists writing Medicare requesting coverage for refractive surgery for doing LASIK.
This is a procedure where you get paid $5,000 cash in advance, and these guys are getting rich off of it, and they don't want--you know, I doubt that there is any of those people that really want Medicare to cover it. I think in Europe, they pay $700 for the procedure, in the U.S., it is $5,000. Why should they want to have Medicare cover it?
Likewise, with respect to providers of the laboratory service. Some laboratories provide a relatively inexpensive service and coverage would definitely help them. Others provide a very expensive service, and because they have been in business, such as me, long enough, we have no trouble getting referrals, and we just have the patients sign an advance beneficiary notice and we can then bill them whatever we want to bill them, and in the patients being in a desperate situation, will usually pay.
So, the thing to consider is, is that when you hear opinions from people that are providing the service, you know, it is not necessarily true that everybody that is providing the service wants to see it reimbursed, and there are individual reasons why someone might not want to have the service reimbursed, and you have got to look at that.
Now, when you look at people that aren't providing the service, such as, you know, that are giving testimony, such as the National Cancer Institute, universities and private practitioners, I was going to kind of take you through that and show you how the NCI had a conflict of interest in their testimony, how the universities have a conflict of interest in their testimony, and I already told you how the private practitioners certainly have a conflict of interest.
Why should they want to upset a system where they can choose the drug based on the spread, and thereby maximize their reimbursement? You know, why should they want to have a system in which, you know, they have to use a certain drug even if maybe they lose money on giving it, you know, if it appears to be best for the patient?
Then, the final thing that I want to say, and I can finish up really in 30 seconds here, and that is, the magnitude of risk in not providing an opportunity for the service to compete.
Dr. Bagley at the MCAC meeting made a statement with which I vehemently disagree, and Dr. Bagley stated that once Medicare makes a coverage decision to cover something, research stops, and there is a danger in covering it because then you just don't get any research done.
That may be true if you have something that is sort of universally accepted and everybody wants to provide. However, if you have got something that is controversial, so you have got competition of ideas and competition of technologies, the way to assure that the studies get done is actually to move the technology into prime time, so it is out there competing with existing technologies.
An example certainly would be the different ways of treating coronary artery disease, and you can do coronary artery bypass surgery, you can do percutaneous transluminal angioplasty, you can give statin drugs, you can now, I read, refer patients to certain clinics where they can be put on the Dean Ornish diet, 10 percent fat, and all of these are different ways of addressing the same problem, but there has been a lot of research done.
In other words, you know, by just approving coronary artery bypass surgery, that didn't make that the standard, I mean, so there really is a competition.
So, I think that in some situations, there is a huge need for a service, and if you don't provide coverage, you run the risk that it will never get the opportunity to compete.
The only thing specifically I will say about the service that I am a proponent of is to relate the following, and that is, that today, there is about 30 to 40 drugs available for treating cancer. Over the next 10 years, that number is going to explode.
There is fast-track FDA approval now, and what is going to be happening over the next 10 years is that you are going to have an ever-increasing supply of partially effective therapies. These therapies are very expensive, oftentimes very toxic. They are partially effective.
The budget crunch for Medicare is not in the year 2000, it is going to be in the year 2010 or 2015. There is going to be a crying need to be able to match the most appropriate therapy to the most appropriate patient, so that each patient gets the therapy that is individually the best for that patient.
These are orphan technologies, you know, they are not proprietary technologies, and I can go through all the reasons why you are just never going to have "industry" support for putting millions of dollars into the trials, however, if Medicare were to approve this, I guarantee you it would be the shot heard around the world, and it would stimulate the sort of definitive studies that everybody wants to see and for which there will be a crying need in just a few years.
Thank you very much.
DR. SOX: Thank you, Dr. Weisenthal. Sharon, do you want--
MS. LAPPALAINEN: Yes, I would like to make a point of clarification to the audience regarding the requirements for conflict of interest.
The conflict of interest statutes may be found under 18 U.S.C. and 5 U.S.C. The Federal Government and all Federal Government employees who are employed by the Federal Government must undergo conflict of interest.
This includes the special government employees who are here today on the panel. The Federal Government does not examine conflict of interest of sponsors or any non-federal employee.
Thank you.
DR. SOX: We have five minutes for discussion of Dr. Weisenthal's presentation.
Yes, Dr. Murray.
DR. MURRAY: Dr. Weisenthal, about 10 minutes ago we heard an exchange, the gist of which was that cost and expense are not to be considered in this decisionmaking. In 25 words or less, without going into detail, could you reconcile those comments with the basis of your argument, which seemed to rely heavily on cost?
DR. WEISENTHAL: My argument relies heavily on humanitarianism, you know, seeing that a desperately ill cancer patient gets the best treatment for that patient.
My own personal opinion, what I heard was that a very sophisticated legal team was studying the legality of that. I don't know anything about the legality. My own personal common sense opinion is, of course, cost counts. I mean this is the year 1999 going into 2000, cost counts in everything.
DR. SOX: Other questions? I guess maybe I could ask one.
You said that, to some degree, our standards for making a coverage decision ought to be affected somewhat by the need for the service, and I guess my question is how do you know what the need is for the service unless you have good measures of the impact of the service on patient care outcomes?
DR. WEISENTHAL: Well, I think that this can be made very objective and mathematical, but it still requires some wisdom and common sense. The only way that I can answer that is just by giving an example relevant to the service that I am promoting, and the argument that I made was in this specific case, I don't know to generalize it, you have to consider each individual case, but I think that it is kind of like giving a student a grade. You give him an A, a B, a C, and a D, and how do you define what a B is and what a C is and what a--well, if you are a good professor, you know this is an A student, this is a B student. This is one of the situations you kind of know it when you see it.
I mean in this situation, even today we have, as I said, 30 to 40 different drugs which can be put together in hundreds of combinations. You can flip a coin and pick any one of them and find some support for it, and yet, 75 percent of all chemotherapy doesn't work, 75 percent of all chemotherapy that is administered doesn't benefit the patient at all.
In the second line situation, there are no studies at all to show population benefits any chemotherapy, yet, it is given all the time, and it is only going to get worse. As I said, the fast track approval, all the new biotechnology products, and so forth, mechanistic-based drug screens that are going to be bringing lots of things on the line, and these are expensive drugs, toxic drugs, and only partially effective, and there just has to be some rational way of matching treatment to patient, so I think the need is self-evident.
DR. SOX: Thank you. There are no more questions. Thank you very much. That will end the open public session.
At the suggestion of several of my colleagues, we are going to take a break now for 15 minutes. I would remind everybody that the cafeteria closes at 10:30, so those of you who need an extra shot of high octane coffee, this is your chance.
[Break.]
DR. SOX: I would like to go ahead and proceed. Actually, I would like to call on Dr. Hill to make a couple of clarifying remarks about the role of the Executive Committee in helping the panels to consider the evidence.
DR. HILL: Thank you, Dr. Sox.
I want to point out that there is a proscriptive element in what HCFA has to do. In our making of policy, we are guided by the--controlled by the statute, which says that no payment may be made for any expenses for services that are not reasonable and necessary for the diagnosis and treatment of illness or injury.
So, we do have to deal with threshold questions of what is enough evidence for something to be considered reasonable and necessary.
If the panel can tell us what it believes is an appropriate threshold for evidence that it would consider to be technically and medically efficacious, that would be valuable advice.
We are hoping that the committee will share with us, the individual members, as well as the thoughts of the committee as a whole, about what is an appropriate ordering of evidence.
All of the committee members look at scientific evidence and critically read articles in their own professional lives, as well as in their job here, and they all have ideas about what is a good study and what is not a good study, at a very minimum sharing that with us will be helpful.
Thank you very much.
DR. SOX: We will now proceed to a presentation by Alan Garber, a member of our Executive Committee.
Open Committee Deliberation - Levels of Evidence
DR. GARBER: If nobody objects, I will remain at my seat. I don't have slides to present, and I am hoping that we can use a good portion of my time for discussion.
A document has been distributed to the Executive Committee members. I am not sure if other people have received it. It is something I wrote called Standards of Clinical Evidence and their Application, and I realize now it is necessary for me to state that this document was not even requested by HCFA. This is something that I asked to have distributed.
I did inform HCFA that I was going to be producing this and this was motivated by the recognition that several of us have had that it is rather difficult to proceed as panels without having a set of criteria by which to judge evidence.
My document actually is not intended to be prescriptive, it is not that I don't have views about what we should do, but it is intended to describe what others have done, what some of the rationale is for developing standards of evidence, how that might work, and I do mention some options although I don't clearly state which ones I would favor, and it is intended to be that way because this is intended to structure discussion rather than to come to any specific conclusions. I hope that is an outcome of today's meeting.
Basically, I am not going to go through a summary of this document, but just to say that the reasons for having evidence standards as a key component of any process to either make coverage decisions, develop clinical guidelines, decide what is investigational and what isn't, they all have in common the idea that everyone benefits by having a fairly clear idea of what kinds of evidence are needed to draw conclusions.
Some of the reasons, of course, are transparency. The more specific and clear we are about what kinds of evidence we need to draw conclusions, the easier it is for everyone to understand the reasons for any decision.
It promotes consistency. If we say sometimes clinical trials, sometimes case controlled, sometimes this, sometimes that, there is no guarantee that a slightly different panel, composed of like people in the sense that they represent the same segments of society, will come to the same conclusion, so consistency is ordinarily considered a virtue, I think, for everyone concerned.
You can actually improve health care quality, adhering to high standards of evidence, of course, means that you are better able to avoid disseminating types of treatment, types of diagnostic procedures, and so on, that are ineffective and/or harmful.
I mention, not because it is necessarily relevant to our deliberations, but, in fact, using standards of evidence can be helpful in controlling health care costs in the narrow sense that by avoiding the dissemination of ineffective treatments, you have avoided expenditures on those treatments.
It promotes research. I am not sure I agreed with the quotation attributed to Grant Bagley that once a coverage decision is made, all research stops, but I certainly agree with the sentiment behind it. As anyone who has followed the saga of high dose chemotherapy for breast cancer can testify, it is extremely difficult to recruit patients for randomized controlled clinical trials once coverage has been made and once the belief is out there that a treatment is effective.
Of course, any decision we reach will be more credible and defensible if it is based on a fairly well defined set of standards for evaluating evidence. So, I hate to belabor these points, but I realize that not everybody is in agreement necessarily that it is important to have standards.
Now, let me be clear, and I hope this comes through in the document, that believing in standards does not necessarily mean that you believe in rigidity, and, in fact, part of the art of this process is deciding when evidence is good enough and when it isn't, and, in fact, all of the speakers this morning I think alluded to the fact that sometimes something less than the so-called gold standard, the randomized controlled clinical trial, is going to be adequate and sometimes it isn't, and that's where the debate often comes.
Innovation was a word that was used in ways to analyze data, and in someone who has made a big bit of his career training people to innovate in methodologies for analyzing observational data, I believe in that very strongly, but a belief in flexibility and a belief in innovation is not the same as saying that there are no standards, and that is where the real issues become--and I don't think this is a policy issue, it is indeed a technical issue, it's a highly technical issue very often. It comes down to can you make a credible case that the biases in something that is not a randomized clinical trial are negligible.
Now, unfortunately, the ultimate answer to that are the biases significant enough to account for the result, say, a positive treatment effect, can't be known with certainty until after the fact, that is, a randomized controlled trial has been performed, and that is one of the reasons why we have so much difficulty because until we have had the randomized trial, we are going to some extent upon belief, and maybe our subjective estimates of how large biases are, but we are in a very difficult situation when we don't have a randomized trial, and I think that is what has been illustrated time and time again.
So, although we speak of a hierarchy of evidence, that may be an unfortunate use of the term because it does imply that there is one type of evidence that is always best, and although I think all of us agree that when you have a randomized controlled clinical trial that is directly involving the treatment of interest in the population of interest that is best, we rarely have that even when we have lots of randomized controlled clinical trials.
Then, we have to draw inferences from the population studied in the randomized trials to the population that will receive the treatment, and they can be very different, leaving us with some difficult decisions, and I think that we will be dealing with this very issue this afternoon.
So, a randomized trial in a narrow sense is indeed the gold standard, but rarely do we have the exact right randomized trial, so we are always dealing with evidence that falls somewhat short of perfection, and then, we, as an executive committee, and each of the panels has to deal with, well, what conclusions can we draw, when do we have adequate evidence.
The document that I have put together does not say under every circumstance what is adequate evidence, and I think we have to recognize as anybody who has participated in processes like this before, acknowledge that you really have to have some flexibility around some standards.
I had a very brief summary of types of evidence, but the types of evidence were well handled this morning, and I have asked to be distributed about a 100-page chapter from an Institute of Medicine publication about types of studies for evaluating technologies.
I would apologize for the length except that this does deal with the types of analyses and types of data that we will be confronting as panelists, and anything more comprehensive would be technical points at a minimum, so I thought this was the shortest document that would do, but I would refer that to everyone, and I hope it will be made available to the people who aren't on the panel who would like to see what has been distributed to the Executive Committee.
Now, as I said, I refrained from making any recommendations in this document that really describes what other--a big part of it is describing what other groups have done, but let me point out some areas of commonality, and for those of you who don't have copies of the document, among the groups whose approaches I tried to summarize, actually quoted directly from the documents were the Agency for Health Care Policy and Research, the U.S. Preventive Services Task Force, the Canadian Task Force on the Periodic Health Examination, American College of Cardiology, American Urological Association, the Blue Cross/Blue Shield Association, and there are many others. That is not meant to be a comprehensive list, but it is meant to be a sampling of what is out there.
One of the areas of commonality is they all have as part of their processes, some rating of the adequacy of evidence, and invariably, it is a two-step process.
One is you say is there enough evidence to draw conclusions, and the second step is what conclusions can you draw from the evidence once you have decided that the evidence is adequate to draw conclusions.
So, the first step is the rating of quality of evidence, and the second one is what are the results of your analysis of the evidence. I think it won't arouse too much controversy to say that if we are going to at least meet the standards of what everybody else is doing who has any credibility in this area, we have to at least adhere to those two steps - rating adequacy of evidence and then deciding what the evidence shows.
Let me propose in crude terms, then, and here I depart from the written document, a two-step procedure for us to follow. One is each panel should make a decision about whether the evidence is adequate to draw conclusions.
Now, I don't think that the Executive Committee should spell out in excessive detail what those standards should be. For example, we heard this morning from Harry Burke about small versus large randomized trials.
Well, as everyone knows, what is in a large enough clinical trial depends on a lot of things. It is not a specific number. Many of us think in terms of statistical power. We think about the consequences of being wrong, what's at stake, and so on. So, we can't say as a rule, trials of 500 or more, or something like that, would be adequate.
We have to recognize that rarely is the evidence perfect, and in each case, we are going to have to have a discussion given the imperfections in the data about whether the imperfections are so severe that we can't really draw conclusions, do they call the major conclusions into question.
When we have a randomized trial, is it in the wrong population? Think about the economics of randomized trials. If you want to have a small sample size, which means a less expensive trial, you are going to pick the population with the greatest propensity to benefit.
So, the question becomes--and usually that is going to be a small fraction of the clinically relevant population, so let's say that you have established efficacy, and I underlined the word "efficacy" there because that is what most trials are about, they are not about effectiveness, that is how it works in the real world, you have established efficacy in that population.
Can we conclude that that means that this treatment will be effective in the population of Medicare beneficiaries? That is a question we will be dealing with over and over again.
Sometimes an observational study will be sufficient. For example, consider a condition that is known to be fatal within three or four months from the time of diagnosis 100 percent of the time.
We are not going to propose a randomized trial for a treatment that appears to work in that situation, but there is a trap here, and the trap is that very seldom do you have that situat