| HOME | HELP | FEEDBACK | SUBSCRIPTIONS | ARCHIVE | SEARCH | TABLE OF CONTENTS |
|
|
|
|||||||
|
|||||||||
Susan H. Ehringhaus, AAMC
Participants:
Aaron J. Ciechanover, Faculty of Medicine, Technion–Israel Institute of Technology: Nobel Laureate, Chemistry, 2004
Russell A. Hulse, University of Texas at Dallas; Nobel Laureate, Physics, 1993
Chris B. Pascal, Office of Research Integrity, U.S. Department of Health and Human Services
Dr. Steven R. Goodman, University of Texas at Dallas; President of the AACBNC:
We are sponsored by both the AAMC and ORI for this particular round-table. They have awarded us $10,000, for which we are very appreciative. There are two people who will be involved in this round-table who have not yet been introduced at this meeting. The first is the moderator, Susan Ehringhaus. Susan is the general counsel for the AAMC. Previously, she was at University of North Carolina at Chapel Hill as vice chancellor as well as general counsel for many years. She is a leading expert on legal issues that relate to both universities and medical schools. The second participant you haven’t formally met yet is Chris Pascal. Chris is the Director of the Office of Research Integrity at the United States Department of Health and Human Services. So we certainly couldn’t have two better people to be involved in this particular discussion, and the discussion is: conflicts of interest, scientific conduct, fair sharing, and intellectual property in an interdisciplinary, inter-institutional consortium. And now Susan is going to moderate this discussion.
Susan H. Ehringhaus, Moderator:
Good morning. For those of you who can’t stand southern accents, this is probably a really good time for you to go get a cup of coffee. I am so happy to be here, and I want to thank all of you for the invitation. I bring greetings from AAMC. Many of you know that organization. You know who our participants are, and it was a pleasure to hear you, Russell, yesterday, and I’m so sorry, Aaron, that I wasn’t able to be here in time to hear your talk.
Let’s start with just a few environmental points, which you know better than I, about what’s changed in basic science departmental structure over the last 25 years. There have been a large number of mergers of you and your colleagues in the past decade. Further, you have seen a tremendous increase in the number of centers and institutes. And as you know, these animals are quite different in terms of organizational structure from your traditional departments. And finally, there is the lack of cohesion or a nucleus really in a center or institute that is comparable to that of the traditional academic basic science department. The number of centers that provide primary appointments to one or more faculty is a mere 22%. Almost 2/3 of centers and institutes provide no salary support to faculty. That’s just a word about how different the environment is now from how it was a couple of decades ago. I believe that these organizational characteristics you see here have something to do with responsible conduct of research. That leads me to the recent tragic story from South Korea, dealing with stem cell research. Perhaps it will spill into this country; we don’t know that yet. These are the environmental factors that I think might fairly characterize what happened in South Korea. You had a team of participants that were international in scope. There were multiple authors on papers and an overly enthusiastic environment. There was enormous enthusiasm, according to the New York Times, on the part of the South Korean government. The science was linked inextricably with getting South Korea’s economy started again. So there was an enormous push from a government in this case. That same sort of push or enthusiasm can come, and has come, in this country from private commercial sponsors. It also can come from universities being linked to economic growth, and they in fact are.
The Boston Globe reported last week about the ease with which images are now manipulated, falsely or otherwise. And that was implicated in this particular scandal. And then finally, and not coincidentally, there is, according to the San Francisco Chronicle, and many of you may agree with this, a real frenzy among the front running scientific journals: Science, Nature and others. I don’t single any out, but there is a serious desire to be the leader—the first one to publish breaking news.
How does the media see us now? Well, I’m from the biomedical research community, so this is perhaps a little more focused on my world than you would like to see, but the media thinks we leak confidential information frequently. The most recent example is the unholy partnership between academic investigators and investment analysts that has resulted in breaches of confidentiality to the investment community so it can capitalize on hot tips. These academic investigators owe more not only to their academic institutions, but especially to their colleagues.
The media thinks we hide, and I refer specifically to selective reporting, or nonreporting, of research results. Vioxx comes to mind, or the failure to show certain effects from antidepressants on kids. They think that we hype. South Korea is a good example—that we fabricate stem cell research. There was just last week a report in the New York Times, I think a couple of us read it, where a researcher in Norway is being accused of having completely fabricated data showing a risk reduction for oral cancer when taking anti-inflammatory drugs. And finally, other institutions are now are in the position of having to verify what we do. And you know, as well as or better than I do, the standards are increasing in the journals.
What does this attention tell us? It tells us the press is interested. They think the public is interested. Responsible conduct of research is clearly on the public’s mind. The public doesn’t make this fine distinction between fraud and falsification and plagiarism on the one hand, and shoddy research practices on the other. They think it’s all misconduct. We have been blessed in the academic community with the freedom to make our own standards. The U.S. federal government has let us define for ourselves how we’re going to deal with responsible conduct of research, that is, questionable research practices, how we’re going to deal with conflict of interest, and has really trusted us. Well, it looks like some of those standards aren’t working. Congressional concern is maybe the most ominous; the same Congress that writes the NIH budget, that writes the NSF budget. Congress is now interested in us in a way that I think is very unhealthy.
We need to examine ourselves. There is a short distance between marginally ethical behavior and outright misconduct. So, here’s what we’re going to talk about this morning, and I’m going to tee up the first one, conflicts of interest. And this is the question that ultimately we posed for our panelists: How well are we doing in terms of regulating conflicts of interest? The popular press doesn’t think we’re doing very well. You may have seen Jerry Kassirer’s book, and Jennifer Washburn’s book, and Marsha Angell’s book. JAMA reported three years ago that industry-sponsored research tends to draw pro-industry conclusions. Paul Krugman, just a month ago, said conflicts of interest aren’t the exception, they’re the norm. And finally our vulnerabilities: we don’t have consistent standards, and we judge ourselves. So, that’s an introduction to conflict of interest. Chris, I think you might have a couple of things to say.
Dr. Chris B. Pascal:
I just wanted to make a few introductory remarks before getting to the specific slide. There are many disciplines represented here, and I ask each of you to put on your healthcare consumer hat. We’re talking about biomedical research and how that affects you, what you need to know in order to make good decisions as a healthcare consumer. To serve this customer, the research enterprise should provide transparency, safety, and a quality product. We know it’s not an easy thing to do, so mistakes can happen. But improvements can also be made. Large financial conflicts require management or avoidance of the financial interest. I think that’s still a hot topic in the community. High-impact incidents reduce confidence in the research enterprise. The Jesse Gelsinger case from the University of Pennsylvania, involving the death of a human subject, is still uppermost in the minds of many who deal with competing financial interests in conducting research. New policies adopted by major institutions and associations do have the potential to improve the conflict of interest process, but more can be done. Studies of how institutions have managed conflicts of interest show substantial variation in practices, even within the same university system, such as the University of California. ORI funds some research on financial interests and its impact on the research community, along with the NIH. ORI believes that the research community should consider standards for transparency in managing conflicts, thus bolstering public confidence in research. For yourself and for your family members, you want to know what information you can trust. We need accurate information about new research products. I think all of us can agree on that. The research community should consider adopting specific standards for cases where conflicts of interest should be totally prohibited. This should be done on a community basis, not case by case with each institution going its separate way.
Ehringhaus:
Here’s the first question for the panel. Russell, would you like to start with how well are universities doing and how well are academic scientists doing?
Dr. Russell A. Hulse:
Thank you. Let me start by giving a little context as to where I come from in looking at this. Frankly, in my own personal professional life, I haven’t had to deal with any of this. I think the root of that is because the fields I’ve been in basically have no commercial value. It sounds funny, but I think that it’s true. I mean, first I worked in astrophysics and there’s nobody out there looking to get the patent on the next pulsar so they can make a billion dollars. And then I worked on the fusion energy program, which certainly is potentially, in the long term, of great economic value. But in the short term, the immediate term, there’s nobody who’s going to make a lot of money off of it. So a lot of what you’re talking about, or in fact everything you’re talking about here, has not been applicable to my field. The perspective I have is in fact the one you were just talking about. I am a biomedical consumer looking at what goes on out there. And certainly, from that perspective, it is a very worrying situation. I think there are several components. The high-profile cases that you mentioned are certainly very, very, very worrisome. However, I have this funny feeling that the high-profile cases are not as important to eroding public confidence as a lot of the smaller stuff. That’s because a lot of the high-profile cases tend to rapidly receive some sort of organized response from government, from the scientific community, and the bad guy is caught and exposed. And so some people may be upset about it, but it’s in some sense like a burglary in the neighborhood. It’s upsetting, but if the burglar is immediately caught and sent to jail, people don’t feel quite so bad because they feel the system works and I’m being protected. It’s the steady stream of more questionable practices, say conflicts of interest for example, where people look at drug studies and say, "Well, this is all funded by the drug companies, right, so can I believe the results?" It produces a certain erosion of confidence. There is built in here also, I think, another issue, which is that people just don’t understand how science works, so it erodes their confidence in science when they see results that are of personal importance to them change back and forth, and I just mean the usual thing like "drinking coffee is bad for you, well no—maybe it’s good for you, well, maybe it’s bad for you, maybe this and that and the other thing." I think the long-term effect is that people come to the conclusion that scientists really don’t know what they’re talking about. Then you add to that the fact that there is high-profile publicity for other situations that seem suspect even if perhaps they’re being honestly run; for instance, drug trial results are funded by the self-interested parties. You put all that together and it gradually reduces the confidence people have in science, it takes science out of its special place in our society. That’s one of the really damaging things, because, without being parochial, I think science really does deserve to have a special place in our society as a special way for seeking truth. All of this has a very strong corrosive effect not only in biomedical science, but I think in the general public’s mind also for science in general. Much of the average person’s view of science in general is formed by their opinion of medicine, because that’s the science that they see, that’s the science that matters the most directly to their lives. Unfortunately, the problem is it’s also one of the more complicated and difficult sciences to do definitively and quantitatively, due to the fundamental complexity of the human body and of biology. So they see these results changing, and they don’t understand that that’s how science works, that we iterate our way towards greater understanding. I guess the final thing I would say concerns the whole hype aspect, which connects to a society which at least in my view is overhyped everywhere. I mean, if you turn on CNN and look at the news, it’s one continuous stream of hype. And it’s done as a conscious, deliberate policy. In fact, I’ve gone so far as to say it’s a policy of lying, because they hype things to the point where you don’t know where the reality is anymore, it is all an attempt to attract viewers and readers by hitting emotional hot buttons. But anyway, I guess what I’m trying to say is that, while it’s not an excuse, all these issues with the scientific community are embedded in a culture and a society where hype has become the expected norm rather than the exception. And, unfortunately, I think it tends to drag the scientists along both in big ways as well as in little ways, for example in some tendency to insist that every one of their results is obviously a world-shattering discovery that shows that whatever went before was completely wrong. That’s something else that drives me really crazy, because again it gives the wrong impression of how science typically works, that is, in an iterative fashion, building on and refining prior results which were not wrong so much as incomplete or not fully understood.
Dr. Aaron J. Ciechanover:
Just one "cynical" personal comment. Since the time I received the prize, people have been referring to me as a world expert on numerous subjects, from Middle East politics, and on to education, religious affairs, and herbal medicine, and now about misconduct and conflict of interest in science. And I say to myself, you know, we discovered something 25 years ago. The only thing that has happened ever since is that several billion neurons in my brain have degenerated. I’m a scientist that runs my own laboratory, and even that with difficulties. So I don’t know why is it that people think that Nobel Laureates are better in any sense or think differently from others. From time to time I’m embarrassed to participate in these kinds of discussions, as I have nothing to add. Anyway, I feel that there are two issues here that are really linked to one another, the conflict of interest issue and misconduct in science. I shall relate in the first part of my comments only to conflict of interest, the influence of modern biomedicine and the development of drugs on academic research, and how directing scientific research towards more applied research is affecting academic research. It is a complicated issue, but there are solutions. When I started my journey into science and decided to become a scientist, I knew that there was no money there, and that I was not going to be a rich man. But times have changed. Now there are many openings for all of us that are tempting us to become what we did not think we were going to become when we walked into this business. We knew that we were going into science because of curiosity. We wanted to decipher the secrets of nature, and all of a sudden, some of us discovered the gold mine, unexpectedly. So for me, at least, the basic instinct is still there. I am a person that goes after my gut feeling and curiosity, though it may be very naïve. I strongly believe in Louis Pasteur’s old saying that there no basic and applied science, only good and bad science, and good science, at the end, will find its route to application. I think that the idea is to completely separate the laboratory from the more applied world outside, and that’s what I’m doing in practice. I completely separate my laboratory from any commercial influence. There is not a single penny of commercial money that funds my research. I want my students to talk to one another; I want them to go to conferences; I want them to be open; and I want them to discuss freely their unpublished data. I don’t want one student to work in the laboratory on a commercial product and as a result not to share with others what he is doing. I don’t want any assessment committees that will examine standing in milestones in my laboratory. If there is anything that is commercial—I am a member of several scientific advisory boards of companies—it has nothing to do with my laboratory. If I shall ever establish a company (I am sure I will not), it will be completely out of the university. The IP should be shared according to the rules of the university. But I completely and sharply separate between industry and my laboratory. Having said that, I deeply appreciate and see the extreme importance of the collaboration between industry and academia. It is an extremely important cornerstone of modern medicine. After all, most discoveries that led to the development of drugs were discovered in universities, and the universities should enjoy the earnings that come from these ideas. But I think that in recent years we have seen more and more penetration of the industry into universities, affecting the direction that scientists go. Even funding agencies somehow prefer more applied, disease-oriented, research. My belief is that we should keep industry completely apart, outside of the university walls—collaborate with it, cooperate with it, but keep it completely apart. It also affects our science, because instead of being discoverers, we are becoming developers. We are departing from the traditional track of science, and instead of chasing our curiosity, we are starting to run after new drugs and lead compounds. It has also to do with funding, especially in the United States. The current funding rate of the NIH, at around 10%, which is on the verge of gambling rather than of judging science by its merit, also plays a role in the change we see.
Ehringhaus:
I have to tell you a story. My brother-in-law is a journalist, and he asked me what I was doing this weekend and I said I was going to go make this speech and moderate a panel with a couple of Nobel Laureates. And he said, Well, what are you going to say? And I told him that I was going to talk about ethics, and I got kind of worked up about it and how important it was. And he suddenly says, Stop. He says, Susan, you are in no danger of getting a standing ovation. He said, on the other hand, that my panelists might. So congratulations for at least starting this out in the right direction. Chris is going to take the next question. We’re going to pause and get some audience reaction in a few minutes. You’ve just given me a reaction to what Aaron said, and I want to come back to that, but let’s get through another couple of questions.
Pascal:
A recent study has shown a high level of self-reported "questionable research practices." How many of you are familiar with that terminology? Less than half responded in the affirmative. The national academies have talked about this topic several times in Institute of Medicine reports and they have indicated that questionable research practices have been a substantial problem for quite some time. They have identified some specific practices that are troublesome, several of which deal with research data. A recent study in Nature suggested a high rate of questionable research practices such as dropping data points based on a "gut feeling" (15%), not on a scientific decision; inadequate record keeping (27%); and overlooking another’s flawed data (13%). There were a number of other problem areas as well, but we just picked some of the highlights. As a consumer of medical research, this does not give me confidence in the reported results. ORI misconduct cases also suggest this is a problem. We get a lot of misconduct allegations and don’t always find misconduct. In fact, we usually do not. However, we do see a lot of poor data practices. When we get these allegations and the institution investigates, they often report to us that some or major portions of the data are missing. So often neither the institution nor ORI can really come to any conclusions, which may benefit the respondent in the case, because we cannot find research misconduct. But the lack of data is very troublesome. The NIH does have requirements that data be maintained. If an institution is funded to do a research study and the institution does not have any data to show for it at the end of the day, the institution has a big problem. From ORI’s viewpoint, the institution should enforce that; i.e., the institution should make it clear that the principal investigator and the research group are "expected" to produce and maintain data to demonstrate that the research was carried out. Who wants to hear that after 3 years of research and a million-dollar award, there is no data? The institution ought to respond forcefully to those situations even if they can’t show research misconduct. Research institutions and other members of the research community have the authority and personal experience to address concerns regarding questionable research practices. Some large institutions do that. They adopt their own standards for their faculty and their scientists; it’s not a federal requirement, it’s something that they do on their own.
Ehringhaus:
So here’s the question to our Laureates: Do you have concerns about the real and perceived quality of research conducted and reported in the U.S., or internationally with U.S. funding? And Aaron, you drew a distinction between conflicts of interest and misconduct issues. Will you start this one?
Ciechanover:
Well, now we are into the second part, and we are moving into misconduct in science, which is an extremely complicated issue. Here we have to look at the mirror and to blame only ourselves. We have generated a system that invites misconduct in any aspect of it. It clearly has to do with education, with basic education of students and young scientists; we have to stress to them that science is all about discovery, unraveling the secrets of nature and the universe. We are discoverers of processes that have been there for many millions and more years; we are tourists in the world of creation. It has to do with our university systems, with our promotion decisions, with the journals that we are publishing, and with the funding system. Each generates a large pressure, and some—luckily, few—collapse and choose the absolute wrong way, which shows that they should not have been there to start with and/or that their education was wrong.
The journals play a special role in this tragedy, especially the leading ones, such as Cell and Nature and their daughter journals, which are basically private business, but also science that is run by the AAAS. These are high-impact-factor journals, and promotion and funding are dependent to a large extent, at least in life sciences, on publication in such journals. So many try to find their way to the narrow tip of the pyramid, as the publication space in these journals is obviously limited, which puts pressure on scientists, and some fail at this point. These days, when all the publications can be easily found in databases, I think it much less important to publish in such journals, as the importance of a finding will be determined, in the end, by its real impact on science—how many scientists find the data important, repeat them, and expand them. So these days we have better criteria to judge quality than based on where they were published, which may be misleading in many ways. I dislike the private journals because of a different reason. They use data generated in public institutions, universities and research institutes, and funded by the taxpayer’s money, and then sell it back to the same institutions for a huge amount of money which comes again from the taxpayer, and in between make a large profit out of this cycling of information. PLoS (Public Library of Science) Biology comes to respond to this problem, and I support the idea wholeheartedly. It is a high-profile journal, open to the entire public, not only to subscribers, and runs at cost.
As for misconduct in basic scientific research, it is extremely dangerous. It endangers our position as scientists in the eyes of the public, it endangers funding, and it wastes the time and effort of scientists who try to reproduce the results. So it hits us in many ways. But on the other hand, in the long term, and as far as science is concerned, it is not damaging, because science is a self-distilling process. Let’s use as an example the recent case in South Korea, the mega-fraud of Hwang in stem cell research. Since nobody could have reproduced the data, it was doomed to be discovered immediately, as it was a hot topic of research, and indeed, the lifetime of this lie was several months. The only thing which is not clear is how the researcher himself could not see it. Not that anybody should ever think of this direction, lying and making up data, but the results were so clear and obvious. Well, I am not a soul researcher. It’s clearly something to be studied by experts in the field of psychology. But his particular case has also to do with the culture of the country, Korea. Because Hwang, unlike in my country, and certainly unlike the situation in the U.S.A., had become a national hero, a friend of the president and members of the government; he got a police escort and a free ticket on Korea Air, and the country kind of expected him to repay the debt by putting Korea on the map—not the scientific map, I assume, but that of awards and honor. I remember reading that thousands of Koreans protested in the streets when the investigation started; they wanted to stop it, not to air the TV documentary showing how the fraud was exposed; they wanted to maintain him as a national hero. Nothing like that happens in my country. Nothing like that happens in the United States. Scientists are regular citizens of their countries, not national heroes. So it has also to do with a national drive and local culture. Japan announced recently that the target of the country is to win 30 Nobel Prizes in the next 50 years. It is a national target. Well, it has a positive side, because that means that the country will probably invest a tremendous amount of money in basic research. I hope that the Nobel Prize was used a symbol, a metaphor for achievements, and people will not feel obliged to repay the debt to the country by stretching their way beyond reality, and mostly beyond truth. Things are different, however, in a major way, for fraud in clinical research; here we are in a different ballgame.
Clinical studies are extremely, extremely complicated, time-consuming, and expensive, and are prone to all kinds of effects that may affect the results and conclusions. For example, we are influenced by environment, by diet, by genetic background, by ethnic origin, by age, by sex, even by religion and beliefs. So to establish a clinical study that would be coherent and cohesive, the results of which would be adopted by the clinical community as a method of treatment, is extremely complicated. Think how much it took to convince the clinical community and society in general that smoking is dangerous. Think of how many years of study it took to show a linkage between smoking and lung cancer. Think about cholesterol. Think of how many years it took to link cholesterol to cardiovascular diseases. Take a study, a huge study, a naïve study, not a fraudulent one, that recommended bone marrow transplantation after complete body irradiation to metastatic breast cancer patients; it turned out to be useless, and such a treatment does not have any advantage over other, more conservative approaches. And this is a study that was conducted on thousands of women. They have undergone the awful procedure of bone marrow transplantation. Can you imagine if such a study was fraudulent? So it is obvious that any fraudulence in clinical studies is horrendous, it causes bodily damage and may end up with the death of many, it walks on the verge of murder and can be murder. Another study is a fraudulent study that was mentioned by Susan that was discovered last week, carried out in the Oslo Radium Institute using non-steroidal anti-inflammatory drugs against oral cancer. If accepted, clinicians would have adopted it as a method of treatment. No one would ever reproduce the data; it is impossible to carry out the study again. This contrasts with basic research data that can be easily reproduced and verified. Clinicians simply adopt the results as a novel treatment modality. Data published in the New England Journal of Medicine or in The Lancet can the next day become a new treatment. And now we are taking thousands of patients and starting to treat them with a treatment that somebody knows that does not help. And he did it for promotion, for funding, for something. It is a crime. It will take years now and thousands of patients to realize that the treatment simply does not work, or works less efficiently than other known treatments. These people should not only be excommunicated forever from the clinical and scientific communities but brought to criminal court. So we should make a distinction here, I think between fraud in basic science and in clinical studies. I reiterate that education is a major factor in how scientists and clinicians grow up and how they meet the expectations from the different agencies with which they work. It has to do with our roles as mentors, how enthusiastic we are ourselves about our research, how interested we are in what is going on in our own laboratories, how much time we devote to our students and fellows, on their own research, but also discussing other, science-related issues. It is a very personal issue, apparently multifaceted, how we run our laboratories.
I am asked frequently if there is a secret to success in science. I do not think so. One has to choose carefully his or her research project, but even more importantly his or her mentor. This is critically important. Then it is important to focus, to build a career in one subject, to be identified and contribute in one area. As for the Nobel Prize, the only thing I can remember is a book that came out recently written by Rolf Zinkernagel, a Nobel Prize Laureate in Physiology or Medicine, on his seminal work in immunology. One of his secrets is to be healthy, live on healthy food, and be physically active, as the Nobel Prize is typically awarded at an old age, and one has to get to this age.
Ehringhaus:
That’s going to be a hard act to follow.
Hulse:
Indeed, that is a hard act to follow, and it was great to hear because I think those are really wonderful comments. And I agree completely with what you say. I’d like to pick up a couple of those points and maybe reinforce a couple and then add another thought. That is, any of the cases of scientific fraud, ranging down to more minor cases of misconduct, always upset me terribly because of my view of science. Truth to me is very important. Maybe that sounds corny, but that’s why I’m in science. It’s the aesthetics and the truth, and so all of this strikes deeply at the nature of what science is really about, or at least what it should be about, as a unique human enterprise and way of getting at the truth. We desperately need to make sure that we preserve public understanding and the reality that science is a way to find out the truth. I am also always amazed, as I think you said, that people engage in fraud, particularly in large-scale fraud, because you just know that they’re going to get found out. And I agree with you that this is deviant psychology. That’s where understanding of such behavior belongs. How can smart people be so dumb? It’s that sort of issue. One important point that you bring up, in terms of the way the scientific system now works, is the fact that the way science funding is given out has become almost unrealistic in terms of how science actually works. Science funding now seems to look for perfection and 100% performance. And that just isn’t the way that science works, which helps create some of the problem. The other thing I must say, in putting on my hat as a biomedical consumer, concerns when I hear about major clinical trials, where my mind says "Ah, it sounds like this should have good statistical significance, and it sounds like it was done by reputable people, and they talk about controlling for different factors, etc. etc., so the results should be very robust." But then you still end up with different studies coming up with contradictory results. To me as a scientist, I start thinking "those methods that they’re using cannot be as robust as they claim they are." If you have too many times where the results are outside the error bars that you claimed, then clearly you’re not defining your error bars correctly. I suspect it’s often due to unknown effects, things outside of the formal statistics of the problem, rather than errors in the statistical math itself. But the bottom line is, even I, as a staunch believer in the fundamental value of science as a way to find the truth, start to get cynical at times listening to some of the things I hear on the news. If four different studies are all pointing in different directions and each one says that they got the right answer with reasonable error bars, this isn’t how science is supposed to work.
Ehringhaus:
Well, that brings us to the next question, and it flows directly from what Aaron and Russell have just said, and that is the steps that we as a community can take to ensure the integrity of research. And the AAMC has been extremely active in this, particularly in the area of clinical trials because this is such a huge problem, and Aaron, I think you set the stage for that. And then Russell, you have followed up when you see four major studies coming out with different things all beginning from the same, presumably the same, hypothesis. The whole community, the consuming community, which would include Congress and the funding agencies, of course, loses confidence. So again, in Nature, about six months ago, we saw evidence about people changing design or method of research in response to pressure from a funding source. And we also see that 10% say that they have withheld data of methodology or results. These are two very disturbing findings, particularly in the area of clinical research. And then in December of 2005, the practice of selective reporting—and we know about Vioxx, and we know about the antidepressants in teenagers—is equated with scientific misconduct.
One of the environmental problems, therefore, that we think exists, that has to be addressed, is access to and reporting of data on industry-sponsored clinical trials. There are public concerns, as you know, about timely and complete reporting of clinical trial results. These concerns lead to concerns about us, because we are often prominent participants in these trials. There are widely varying standards from institution to institution, and there’s a need for an articulation of principles. So the AAMC has come up with some data integrity principles that we think will go a long way towards improving the quality of clinical trials. And Aaron, you’ll be happy to know that we are publishing it in PLoS, which is consistent with just the values he was talking about. The contracts for trials must commit adequate funding to support the prespecified analysis plan. And they should commit adequate funding for publication, even when the trial is terminated prematurely or where there are no results. One of the problems is that we never hear about negative results. And we know as a scientific community the importance of knowing when something hasn’t worked, as well as when something does work. But prominent journals don’t want to publish negative results. Trials should be registered in a publicly accessible registry—the only one that exists right now is clinicaltrials.gov in the U.S., but WHO is building a registry. Each multisite clinical trial should have a publication and analysis committee controlled by investigators, not by the sponsor. That is to ensure investigator access to the full data set. One of the major problems in our view has been investigator inability to get at the data. The sponsor has provided investigators with only data summaries. And then, obviously, we must have authorship and publication standards that are transparent. Finally, in order to ensure replication on your part, data sharing is very important. Data underlying publications, similar to NIH data sharing, should be made publicly available.
Chris, you wanted to say a couple things about data integrity.
Pascal:
ORI has a lot of experience in looking at the data, because in our research misconduct cases, that’s the main thing to look for. ORI tries to find out whether or not the data or results are accurate or were intentionally falsified. ORI believes that data is the main outcome of research and the quality and accuracy of the data are of prime importance. Accurate data is essential to the transparency of the research and the ability of doctors and health care consumers to make informed choices. For me, that may be the single most important change that I would like to see in the research enterprise: information that is accurate and easily accessible. Perhaps it could be published on a government website. How many of you remember ten or twenty years ago when you went to buy a car and you tried to negotiate the price without knowing what the true cost was for the car? I remember that. But now you can find the actual dealer’s cost, and that was a big change in helping the consumer get accurate information and fair play. The internet has certainly helped with that sort of information. That’s the type of transparency I would like to see in the research enterprise: accurate information, easily accessible, and so forth.
ORI recommends three steps to ensure quality data. First, the research institution should adopt a data policy for collecting, recording, maintaining, and reporting data. Next, the policy should be implemented at the group or lab level, with specific changes necessary to reflect the type of research conducted in the specific group or lab. Finally, the lab chief or principal investigator should provide training to the lab staff on how the data should actually be managed. This could be done through case studies, formal lecture and discussion, and regular review by the principal investigator of the lab notebooks, and discussion of any problems with how the data have been handled. If the principal investigator thinks that the data is important, and communicates that to the research team, and takes sufficient time to do so, it is likely that the lab or group will learn appropriate ways to handle the data without formal training—it just happens. If the PI has a specific way that he or she wants the data to be kept or recorded, then the PI needs to communicate that clearly to the research team. ORI encourages the research community to clarify or define the normative standards for conducting and reporting research, and then obtain community support for following those standards. ORI and the American Association of Medical Colleges have worked with scientific and academic societies for four years now funding relatively small projects with those societies and getting them to address responsible conduct of research education, guidelines, and so forth. We have had quite a few studies like that done, but the major societies really haven’t shown much interest. Of course, you know, this is a voluntary process, so it’s not a regulatory requirement. But I think that it could be very helpful if societies developed clear guidance for scientists to handle authorship credit, data management, peer review, etc. Members of the research community always say that there are ways to do it. But it’s usually not written down, and it’s not clear to everyone what that is. I mean, an individual scientist may be able to stand up and say, I can explain to you how I do authorship and how I share the data, and so forth. But there is not really a normative standard that is set out clearly for the young scientist in training to follow. I think that’s part of the problem. Certainly, some disciplines will have different standards for a good reason, and that’s OK. But I think making those standards evident and accessible would improve research in the biomedical research enterprise.
Ehringhaus:
OK, here’s a question for our panelists, and then I’d like to involve the audience. We’ve gotten some good ideas about what are the culprits here. What are the things that we can focus on, we as individuals, individual investigators, we as a scientific community, professional societies, and institutions? Russell, do you want to start?
Hulse:
Again, actually I feel a little out of my element here because it’s clear that a lot of what you’re talking about is deeply rooted in biomedical research in general and clinical trials in particular. I understand this with the current audience, but it’s not something which I personally have ever been professionally involved with. Maybe I should just take a moment for a little bit of levity. All this conversation reminds me of a funny paper that was passed around the lab many years ago. It was definitions—translations of phrases used in research papers and what they really meant. It’s one of those things I wish I’d saved because it was so wonderful, but unfortunately I didn’t. But one entry which I remember was: "Carefully prepared" with the translation "Not dropped on the floor." The other ones I particularly remember were "Is known" translated to "I think so" and "Is well known" translated to "The guy next door thinks so too." But anyway, let me make a comment about standard methods and data integrity. I do have a little experience with that, because working at a Department of Energy laboratory, we are subject to all sorts of mandates that come down the pike. And frankly, some of them, while I’m sure well intentioned, end up being a little off the mark when they end up down at the working scientist’s level. If I remember correctly, one particular situation started with the discovery of unethical human testing which occurred many, many years in the past, and associated with this there was a data preservation issue. Well, in a laboratory that never had anything to do with human experimentation, nonetheless we ended up having to respond to a plethora of orders coming down that we had to preserve all of our data, and all of our raw data, and this, that, and the other thing. The way the orders came down just didn’t correspond at all with the realities of our situation, and it created this flurry of activity and angst as to what we’re supposed to do to respond to this, and what procedures to follow about how much data we should keep. You know, it just made no sense. It’s the classic bureaucratic heavy-handed response to a problem, making life difficult for people who weren’t even in the area of where the misconduct occurred. Part of the response to these issues was the concept of working papers, which is something that you didn’t have to retain, in other words, quite sensibly, every time you write something on a scrap of paper, it doesn’t necessarily have to be fully documented. But then you can end up with the tendency to define everything as a working paper which doesn’t have to be preserved. So it just seems to me that it’s one of those things that’s very tricky to implement in a meaningful way, particularly in a one-size-fits-all sort of mandate. This doesn’t mean one shouldn’t try for better record-keeping standards and validation of the scientific process, particularly in situations which involve human health, but one has to be careful about trying to micromanage it and producing a lot of bureaucracy that doesn’t really accomplish anything. On a related subject, I have started to think that, especially in the context of what has happened in recent years, that maybe ethics courses are a good idea. Perhaps graduate students should have at least some formal exposure to the fact that as scientists we have a high responsibility for the truth, along with discussion of some case studies of people that went awry, and why, and here’s what happened to them. We could also provide students with guidance as to what they should do if they find themselves in a questionable circumstance. At least prepare them for the fact that that may happen and then give them at some pointers as to what you do—what are good ideas on how to proceed. So I think that could be very helpful. Again putting on my biomedical consumer hat, I picked up on one of the things that you talked about here. I was really struck by this report that the results from clinical trials that didn’t give the answer that was desired tend to be either not published or buried in obscure journals. I was really appalled at that. I think you made some suggestions that there should be some uniform reporting and I absolutely 100% agree with that. If someone funds a clinical trial to do some definitive study, whatever happens, even if they drop all the samples on the floor, that should be written down and publicly accessible in some common repository. There’s no doubt about that. However, I guess I should just restate a caveat on that, and make a final remark. Applying uniform rules as to what data integrity and data record keeping mean across different areas of science is very tricky. In my field, a mandate to keep all your raw data by itself really just doesn’t make any sense because the raw data—the ones and zeros by themselves—just don’t mean anything. There’s a whole context of what instrumentation the data came from, what the point of the study was, etc., etc. So it becomes much more than "Oh yeah, I’ll save that data tape." In some situations, if one is not careful, rigid application of poorly thought through rules can involve an incredible amount of effort, much of which is simply wasted effort, and irrelevant. So you have to be careful. In clinical trials, I would think it’s a lot clearer. I mean, if you set up the trial well and you define what the criteria were for the trial and what the observations you’re going to make are, then that data sounds like it should be relatively self-explanatory and therefore relatively easy to mandate that it be retained. So maybe it’s easier in clinical trials than in some other things that I’m thinking of.
Ciechanover:
Well, I really want to comment briefly, because I think we should now turn to the audience, and this is a unique audience in the sense that people here are not only responsible for educating their own students and fellows but also for fostering newly recruited young faculty, and they have an influence on the next generation in science. I kind of do not believe in artificial means. Data is very important and we can make numerous copies of the data and writing them and then keeping them in safes, in the laboratory and in the dean’s office. But these days, data can be manipulated easily. We are using phosphorimaging and we are using fluorescent microscopes, and we can control the brightness and we can control the contrast. And the student just before he enters my room with the data can flip them upside down without me even noticing what’s going on. So we are away from the old days when data were solid, when numbers came from a gamma counter, and even then samples could be prepared, and skin grafts transferred among mice of the same species, etc. I believe in deeper means, not administrative ones. On one hand, it is education. On the other, the way we manage our laboratories, repeat data, generate groups within the laboratory of students and fellows that work in teams on similar though not identical projects, so they can somehow control, or at least know, the data generated by their peers, and then alleviate pressures in publishing, funding, and promoting. That does not mean we lower criteria—on the contrary. But we assess the quality of science and not where it was published and how many papers were published. So I think that the idea is more basic. It’s more of education in the deep sense of it, of being a role model, of keeping the laboratory in the appropriate size that can be controlled, of being enthusiastic yourself. What is science? Science is discovery, it’s curiosity. We are not inventing anything, it’s all there. Ubiquitin was there for millions of years. We only discovered it. We didn’t make it. We didn’t do anything for it. We just unraveled one of its pathways that has been there forever. So it is important to elicit curiosity and to explain to students, and then to the public, what is science all about. We must teach what we are doing there in our laboratories.
Hulse:
I agree with you. Your discussion there prompted me to think of something which has always bothered me in my work with science education. When you talk to many people in corporations, in government, etc., there seems to be this one-to-one equivalence that science education is important because it is necessary for our economic development. And it is absolutely true, of course, that it is vitally important to our economic development. But this purely economic view of science completely loses sight of science as a discipline, as a calling, as something that transcends economic considerations. It feeds economic development, but it is not identically equivalent. Its entire purpose is not simply economic development. It has a larger purpose; it has a larger ethos associated with it. And our society has lost track of that.
Ciechanover:
Just one last comment about it; again, I can learn from our own case. When I started my journey into the ubiquitin system in the mid-70s, as a graduate student of Avram Hershko, we didn’t think of drugs, we didn’t think of diseases. We just were after an interesting yet unexplored problem. And then, after years, when people realized it is important indeed, they further evolved it, expanding on the basic findings, and then moved further to their expansion into diseases and drug targeting. So identification of an important yet experimentally approachable question, then curiosity, drive, possibly some cleverness and experience, and I should admit also luck, brought us to a certain point that was then further developed by others, all the way to drug application, which patients are now enjoying. And we had nothing to do with this late scientific and certainly applied and economic development. We just let it go—and the rest came by itself, will come by itself. So the idea is again to do, as Pasteur said, good science; if it is truly good, there will be some use for it.
Ehringhaus:
It’s time to hear from you. Anybody moved—I’m sure—you’ve gotten a lot to provoke you.
Dr. Vytas A. Bankaitis, University of North Carolina, Chapel Hill:
We now live in the era of large databases. All right. So if we just sort of forget the clinical trials aspect, that’s what these essentially are—this is not a good idea—because we do not know the quality of the databases. Any of them. Number one. And training our students to think that the database approach is really the way to do science is fallacy. Second of all, you know, what this is all about is whether we honor a profession or we don’t. This is really what it boils down to. And before we get to sort of the big questions, something that was raised earlier just comes to the issue of what is the enterprise coefficient in your own laboratory. Do you actually know the quality of your own data? Many people, I suspect, find it inconvenient to really check very carefully whether the results that lead them to a certain track are actually true. I think that in journals, the pressure that we feel to publish makes it very easy to go ahead and do the minimal publishable unit sort of approach. We sort of kid ourselves that well, this may actually not be exactly correct, but we’re not really wrong here, we just didn’t take it to the end. And unfortunately, there’s this culture now where large stories, publishing large stories—like Sidney Brenner did, for example, where it was an 80-page paper, a single paper, you know, which was a pretty important paper as I recall. You don’t see that any more. And I think that’s a big part of the problem. As a chair, one of the things that I’ve tried to do is allow the graduate students in my department to come into my door to talk about whatever they want. So when we talk about issues of motivation or demotivation, I have found it really amazing that one of the most demotivating issues that graduate students bring up is that their ideal of science—that this is a search for truth—that’s dashed in the first year and a half, two years of graduate school—within their own laboratories. Not so much on a large scale—they haven’t even gotten to that point. So if we just essentially worry about quality—not worry about parlaying our interests into some other issue, we’re going to be OK. But it’s going to start at home, just like most education.
Dr. James R. West, Texas A&M University System Health Science Center:
I’m convinced that the problem that we’re talking about today is actually perhaps as complicated as, and more elusive than, the science that we do. The idea that we should have more and better education in terms of ethics is crucial. But when it comes down to it, we make ethical decisions almost daily. And most of us haven’t even had the training to recognize that that’s what we’re doing. But from a basic perspective in terms of looking at ethics, it’s pretty simple. If you can borrow from another philosophical term—if there’s a categorical imperative related to ethics, it’s one should always do the right thing and strive for that. But as in many cases, it’s often much harder to do something even when you know it’s right. If I might give an example, everybody’s talking about North Carolina. Before I went to college, I was in the Marines, and going to a duty base in 1960, I got off a little bus in a little town in North Carolina, and I went to get a drink at a drinking fountain, and somebody grabbed my arm and pulled me away and said "You can’t drink from that one—that’s for the coloreds." And I looked around while I was in North Carolina, and I saw lots of churches. One would expect that there would be a lot of ethics, a lot of doing the right thing. But there wasn’t. There was lots of racism, there was lots of segregation. Later on when I did a lot of reading, I found examples of how maybe someone who tried to help African Americans—somebody would come to them and say "look, either you stop that or we’re going to fire you from your job." So that person then had to look at life in terms of—if I do what I think is right, my family’s not going to be able to eat or have a place to live. And those kinds of things have a tremendous influence on what people ultimately decide from a practical perspective. So we can sit back, those of us who have established careers, long grant histories, and things seem pretty clear. But it’s a different story for a young person with a family who if they don’t get the grant or if they don’t do things a certain way may lose their careers. If it’s an assistant professor where part of his salary comes in from grants, and you lose a grant, you lose your car payment. I think one can see how it gets a whole lot more complex and clouded for someone like that making decisions. And some things that we might just be appalled about might somehow slip by. And I guess my concern is—I don’t know how, with a class on ethics to graduate students and postdocs, we’re going to be able to get at dealing with those kinds of issues. And that’s the ones that I think are important.
Dr. Gordon I. Kaye, Albany Medical College and WR2, Inc.:
Jim has made a very important point in extending our view of this to a somewhat wider field than just biomedical science or biomedical research. I was struck this morning looking at the newspaper that the whole idea of conflict of interest and integrity is a much more broadly based societal problem, and what brought that home to me was the announcement by the Japanese prime minister that U.S. beef exports to Japan have been cut off again. Because less than two weeks after resuming the imports, contaminated beef containing bad material was shipped into Japan—under the supervision of the Food Safety and Inspection Service of the USDA. And that brings to mind the whole question of "Who are our regulators and who are they working for?" Who is protecting public health and safety? Is the FSIS working for the public? Who pays its salary, or is it working for the meat industry? Is the FDA working for the public it’s supposed to be protecting, or is it working for the pharmaceutical industry? And the very practices of integrity and conflict of interest that we have focused on this morning in a very narrow area—and I’m sure Professor Hulse has seen it working in the DOE: are the people who are in charge of the nuclear regulatory controls working for the nuclear industry, or are they working for the public health and safety? NRC tends to be a little better than some others, but there are still a lot of areas of blindness, ignoring problems, and helping the industry rather than helping the public. And I think this whole area of conflict of interest is a much, much broader problem than what we’re dealing with here today and a much more fundamental problem in our society.
Dr. John I. Clark, University of Washington School of Medicine:
I’m responding to the specific steps that can be taken by a principal investigator. I thought there were two themes that came up. One was communication, and we talked about it previously—hypothesis-driven research. I always, like Russell, was thinking that misconduct really wasn’t a problem in my field until about three years ago I was asked to participate in an investigation at the University of Washington on scientific misconduct. And the complaint came from the NIH—I don’t know where they got it. But three scientists and three attorneys were assembled to investigate this example. And when I reviewed the summary requesting my participation, I just thought it just sounded like the complaint was a competitor who was trying to slow down this lab, which I was aware of—they’re a very prominent lab. So I thought it would be a week and a half, maybe two weeks participation, and this investigation turned out to be nearly two years. The point was that when we reviewed the records, again, we felt the investigator being charged was simply the target of a competitor. We asked him to present the work in question and with a terrific presentation, very well done, until at the end he was asked what was the hypothesis. And he stumbled around; he really couldn’t define the hypothesis. And he said it was really an assay, so we asked him about the controls for the assay. And he showed us a series of controls that had error bars that were remarkably tiny, and that of course gave us our first lead into where to look in the reams of data. It was forty file boxes of information, and fifty hard drives that had been confiscated from his laboratory. But we went after the controls and we went after the hypothesis, and we realized it was the controls that had been fabricated, not the actual data. He had the data that were presented in the journals, but the controls were in error. And when we talked with him again about this, he said, well in his laboratory of 21 people, one of his individuals had given him this control data. And of course, we interviewed that person, and the person said yes, he had given it at a lab meeting, but then it just disappeared. And there was no communication even within the lab—open communication—even within the lab about where that information went, and how it was presented. Information had appeared in abstracts and in articles that the individuals who had created the data were unaware of. So the lesson I took away from that was one, it’s very important to have a hypothesis that can be tested. And two, communication really starts within the lab where the work can be exposed and evaluated and looked at carefully. Not just taken away and formulated into a publication that’s presented later. So those are two areas that I think we can address in our own laboratories—focus on hypothesis-driven research and on communication between our lab members.
Dr. Michael T. Shipley, University of Maryland School of Medicine:
So I kind of share some of the same instincts as Aaron. I like to simplify as much as possible. And it seems to me that the central question facing us not only as chairs and as scientists, but also administrators in schools of medicine and even beyond that, is—are we getting money to do science, or are we doing science to get money? A number of years ago when I came in the field, many of us came in the field, the whole point was to get some funding so you could do the research. But with the corporatization of our profession, of medical schools—and not just medical schools—whole universities, the whole process has sort of gotten reversed. We’re now doing science to get money so we can compare the length of our portfolios with other institutions so that we can go to donors, so that we can go to legislators and we can tell them how big our portfolio is. And the other thing that I don’t like is, I hate litanies of problems without solutions. So I’ve tried to at least think of a couple of potential things that might help this. And I didn’t just think of them today, I’ve been thinking of them over the years. One of them I think will be implemented. At the AAMC meeting in Salt Lake City that many of us went to last October, there was, I thought, a clear statement on the part of the NIH that one of the things that they’re going to do, at least for a few years, is to eliminate department-based NIH rankings. Now none of us mind accountability. I mean, I’m happy to have my department ranked. The problem with the rankings, as we all know, is that it depends on apples and apples. You know, are you being compared with a faculty of size X. And so, the NIH rankings never really normalize for faculty size, for the size of the budget that you’ve got. And so they were always—it was always kind of a shell game anyway. But to some people who have sat at high altitudes at medical schools for too long, they sort of tend to forget this. And so they just focus on the absolute number. So, perhaps if we can get a little bit away from this comparing at the level of absolute numbers with department ranks, it might help. I think the other thing that might help, though it’s far more difficult given historically the way our country does its budgeting process, is that I for one would be perfectly happy for the NIH to know that it’s got a 3 or 4 percent increase, even a 3% increase for 5 years down the road, rather than going through the roller coaster of doubling the budget, and then falling back, and doubling the budget. Because that’s completely out of sync with the way we do science, number one, and, as several of you mentioned, with our really problematical appointments and promotions process, number two. So you bring a lot of faculty in during this bubble because the deans, the administrators are saying wow, we’ve lost clinical income; it looks like research is the golden road to solvency. You bring a lot of people in, the NIH budget drops down, and now you’ve got a lot of problems at the level of promotion and tenure and covering salary. And that does contain the seed for unethical behavior. Because you’re absolutely right—when people’s livelihoods and incomes in large numbers are predicated on these kinds of measures and getting grants funded and publishing in Cell and Nature and this sort of thing, there will be problems. If there are large enough numbers, there are going to be some people that are going to try to push it too far. So we have control over some of it, but part of it has to do with the mindset of our administrators, and I think part of it has to do with a better relationship between we who are doing the science and the NIH which is funding the science. So if we can get off the rankings for a while and if we could somehow enjoy a mechanism of more steady funding—maybe not the more spectacular periods of growth, so that we could do some longer-range planning, things would improve. And then finally, I agree with Vytas and a number of other people—it really does begin at home. You know, in my own view, regular lab meetings where students and postdocs and other folks in labs present their data and go through the crucible of question and interrogation by their peers is important for the issues discussed today. Sometimes we share lab meetings with other groups. This is the place to really point out that it’s truth that matters and not spin. As labs grow too large and get outsize, then problems come in. But anyway, that’s it. And thank you very much for the panel and for your comments. I think you can tell from our response that we are very appreciative.
Dr. Robert D. Goldman, Northwestern University:
I just wanted to point out that last year there was a report from the National Academy, the National Research Council, and NIH and Howard Hughes called "Bridges to Independence." And what it basically summarized was—or the bottom line is—that the average age of an assistant professor now is 40 in a medical school. And the average age of the first grant is 43 to 45 because it takes three to five years to get that grant. And this was when times were good. With the recent cuts in the NIH budget, in fact we just heard that we’re taking a 3% across-the-board cut and, considering biological inflation and research inflation, this means a huge hit on individual faculty. For young faculty who manage to get a grant, this is a huge cut. And yet we still expect that they will get 50% or more of their salary from those grants. We’ve learned to become totally dependent upon federal funds. Med schools are built on federal money. And without that 50%, departments go under. They get into big trouble with the deans. So I think the AAMC in particular needs to adopt a policy which stabilizes, especially, the basic science departments. Basic sciences are very, very different from clinical departments. We have one source of income—and 90% of that is the NIH. This is then minimally supplemented by whatever we get from the dean. So with very small contributions to salaries from medical schools and the expectation that we will have to get 50 to 75 percent of our salary from grants, this puts an enormous pressure on young people. At the age of 40, most of them are married and have kids. And they’re not paid that well. And yet they have to earn their salary at our school; 60% is the expectation for them to generate after 3 years. And this is becoming an impossibility. So, whether the NIH likes it or not—or the federal government likes it or not—we’ve built up a partnership. And yet, there’s not a lot of respect between the universities and the federal government. They’re always fighting—they’re always arguing—but they’re always arguing about policy—in fact, they’re making lives miserable for those of us who do research. There’s so much micromanagement now that it has become intolerable. And it’s all based on one or maybe 0.001% of those scientists who cheat in science. Most of us are pretty good—we’re quite honest—and we’re excited about what we do. But we’ve been forced into situations that make science actually very, very unattractive as a career.
Dr. Sally S. Atherton, Medical College of Georgia:
I have either the blessing of the curse of being the responsible conduct of research course coordinator for our university. And one of the things that I require as part of the course—it’s a one-week to once-a-week course that goes through an entire semester and covers a variety of things from peer review to scientific misconduct and a variety of things—is that the students are required to write, God forbid, a paper—and to talk to a mentor, an individual at the university about a scientific situation of perhaps misconduct or questionable things. And two things—one of the things is that I was surprised at how many people actually had encountered some of that or could speak to a mentor that had. I was very pleased to see that most of these—and these are mostly first-year students—were very insightful about what they would have done to have prevented that in the future. And many of the things that we’ve spoken about and the panel has spoken about today are the same things. And so somewhere between being a student who comes in with their eyes open but yet very wise, and then perhaps into the faculty position where you’re sort of led astray by other things—I think that we as investigators and as mentors need to make sure that we keep the students and everyone apprised of that in our labs. And many of the things that we’ve talked about today I think will do that.
Dr. Joseph C. Besharse, Medical College of Wisconsin:
I particularly enjoyed a couple of the comments and would like to just say something about this relationship of our medical schools themselves creating part of the problem. I had an e-mail about a year or so ago from an assistant professor at another very prominent university asking me for some advice. He was sending me a copy of a manuscript that had just been rejected from Cell. And he was very concerned about his future career, and at the same time sent me a copy of an e-mail that he had received from his dean that apparently had been sent globally to faculty about the absolute need of that institution to publish more papers in Cell and in Nature. And the young man was obviously beside himself—what journal can I get it into, and what can I do. And of course, it was actually a very nice piece of work, and rejection from Cell meant absolutely nothing. But he was being placed in a situation in which there is really a collaboration of the administrative structure of the medical school within the creation of some of these kinds of pressures. The other one that I have frequently observed is that the young man that has simply an RO1 grant and runs a laboratory is really pretty much a nobody in the mainstream of many of the medical schools today. You have to have a big multimillion-dollar complex thing in order to get a lot of respect and clout. I want to give you one other anecdote that comes from this kind of climate in a medical school that troubles me a great deal. And this comes from a colleague—and a very good colleague—and one who has good integrity, by the way, and is a good scientist—talking over which of the roadmap initiatives we should get together and go after. We looked at one particular structure and many of us said well, this is totally ridiculous to write a grant like this. But the response to the whole thing from that, and I quote, from this individual, is "let’s just get the money and worry about it later." It’s part of an atmosphere where our promotion system is so much based on the money that getting the money is more important than getting the scientific result. And I might add to that—I think many of us have experienced situations where we see that you get the big dollars but you don’t really see the big publication in any particular time frame that can be directly related to that. And we change what those big items are so frequently that I wonder about accountability for many of these kinds of programs and what they have actually accomplished in terms of pushing science. So we have a collaborative thing going on—we collaborate with the system in order to create things that are really harming, I think, the way we conduct our scientific research.
Dr. Fred J. Roisen, University of Louisville School of Medicine:
A couple of the comments that came out today that really touched at my heart and really tug at me is the inability to publish careful but negative results. I had two students this past year—graduate students who were finishing. One was very fortunate and had exciting results. Another worked just as hard, perhaps even harder, did very careful research, and had negative results. And I said, well, we’re going to publish both. Then the student came back, "You suggested that we publish in a low-impact-factor journal." The students now—the society has so corrupted the system that they know the impact factor and not even the name of the journal. They go by impact factor. And that’s the basis—that’s the database that’s come out. It’s not—if you affect 20 people and you give them critical information, isn’t that important? Students don’t see that—they only see impact factor. One other point I wanted to add is not only are we requiring people to generate 50 to 60 percent of their salary, but at many of our institutions, and I assume most of you, we have to pay or generate so many dollars per square foot. So it’s not just salary, but in order to have the space of the department, the system is we have to generate the funds to protect that space. And that doesn’t come from the basic science chairs at all. That comes from the deans of the medical schools getting together and they hear—"oh, at this school they generate 60 percent—at another they generate 70—so Fred, if we’re asking you to generate 45, that’s not so bad." So we need to get to the root of the problem, and I don’t think it’s out in the trenches.
Dr. Barry E. Stein, Wake Forest University School of Medicine:
I’d like to change the nature of the conversation. This is very cathartic—we’re the best at complaining. I think that’s why we’ve assumed these positions. And I doubt that there would be great disagreement with the things that the panel has said or the things that some of us have said with regard to the problems that we face. Our biggest problem that we face is our inability to come up with a coherent solution to these kinds of problems. So, I think one solution was offered with PLoS to one problem. But we’re facing a crisis. The crisis is going to be precipitated by the short flow of NIH funds and the fact that we’re all leveraged and we don’t have enough money to run our operations, or anticipate that we won’t have enough money to fund our operations. And I think we have to change our perspective to be solution-oriented, and also to take into account the fact that every time we perceive that there is a problem, in this case from the Office of Research Integrity, we’re hearing about the problems with research integrity, the individual investigator—assistant professor, full professor, chair, whatever—is overwhelmed with problems—with assaults. For those of us in the biomedical sciences, we’re assaulted by requirements for promotion. The dean’s requirements for what we do, NIH requirements, requirements of the institution for getting money, the animal rights people, the office of the IACUC/IRB. Whenever we’re talking about these issues and thinking about the problems, our solutions seem to be, "Let’s tell the investigator to do something more." So we at Wake Forest, like every other school, have a compliance office there and a compliance committee. I happen to have the misfortune of being on that committee, and what we do is we hire people when there is a problem to tell other people what to do. We never hire someone to do it. We hire someone to tell the investigator what to do. So I would like to change the nature of this conversation—we might not have the time today, but in the future that instead of doing the complaining, which we do very, very well, we can do that amongst ourselves—that we start to focus on crafting a plan to deal with these issues to change the research culture. A practical plan—I don’t mean some theoretical plan—but rather a practical plan that we can discuss and morph as time goes on. Thank you.
Ciechanover:
I have just a few brief comments. One has to do with the hypothesis-driven research. I am probably the archetypical hypothesis-driven scientist, and I inherited this approach from my Ph.D. mentor. Because we had something defined in mind, and we were searching for it. But we should be very careful about it because the databases—as reliable or not reliable as they are these days—provide us with other opportunities of being fishermen—going on a fishing expedition. And it’s not that bad. From time to time we discover a protein in the laboratory and we don’t know what it is, and we go to a two-hybrid system fishing expedition and see what we are fishing without any hypothesis. And then from time to time some golden fish is caught up. And I can go to even more of an extreme and tell you that with the availability of antibodies to so many proteins—actually, there is now at the University of Stockholm a huge project that generates systematically antibodies to the entire proteome, 22,000 well-characterized polyclonal antibodies to the entire proteome, one by one. You can take all these antibodies and scatter them over 10,000 sections of prostate cancer, 10,000 sections of neuroblastoma, 10,000 of whatever you want, and then fish out antigens that change in their level in different tumors just by scanning for them—some maybe very important causative proteins—but the search for them has no hypothesis at all. And mRNAs came just from fishing the noncoding DNA database. So you may argue that this is not a type of university research—maybe we should outsource it to a company. But nevertheless, with the existence of databases and experimental tools to carry out such types of research, we have opened new avenues for excellent research that is not hypothesis-driven. But hypothesis-driven research still has its own important place. That’s just to put things in proportion. But I really want to relate to the last speaker, to the promotion and salary issue. There are no magic solutions, because we are deeply involved in this culture and it would take years to get out of it. But there are several solutions—one of them was mentioned by me, and I’m happy that people agree with it—about the advantages of diverting from the current publication culture. One has to evaluate the quality of the work, not the name of the journal where it was published. Then we have to change the payment system, so researchers will not be dependent for 100% of their salaries on their grants. This will allow more researchers to enjoy grants and will increase the funding rate to above gambling level, to 20 to 25 percent or so, from less then 10% nowadays. This will be adapting part of the European system, with all of the shortcomings of this system. In Israel, we are in a unique situation because we adapted for good some of the American system’s values—that’s competition on soft money for research; on the other hand, we are providing the tenured faculty with full salary for the rest of their lives—which is something wrong—because people after getting tenured can sit with their hands and legs crossed, doing nothing and having the same salary as active researchers. But we can adopt something which is used in the state university system in the United States: a nine-month salary. So there is an incentive to obtain extramural funding, not only for research but also for income, and the funds should come from the universities or from the government, so that faculty members, tenured and not yet tenured alike, can rely on 75% of their salary from the university. All these measures may alleviate part of the pressure under which scientists are living these days. So I think that multiple measures are needed here—it won’t be a single one. But basically, it should be our own will within the university to change the culture of the environment that we are working in—and it must be done. Because otherwise we are destroying ourselves, and possibly the progress of science, deterring young people from entering into this impossible field.
Hulse:
I was just going to put on my science education hat for a moment and pick up on a comment that was made from the floor talking about the quality of life of a scientist. I always find it remarkable when people—especially government officials—get so exercised about getting kids interested in science, and it never occurs to them, it seems, that these kids are smart enough to look at what sort of life style scientists have, and will decide if they really want to live like that. I have certainly met very bright high school students who were passionately interested in science, but who had consciously decided not to pursue science as a career because they didn’t like the career prospects. So a lot of these issues, in terms of what the life of a scientist is really like, have an impact all the way back to whether you’re going to get kids interested in becoming scientists or not.
Bankaitis:
So just a couple comments on what Aaron suggested. So actually, there are a number of universities now, non–medical-school situations, that have exactly the system that you proposed. There is a nine-month hard salary, and then there is two and a half or three months. I think if we’re really honest with ourselves, what we have to get rid of is tenure. It will be interesting to see if we have the courage to do that. Because really, the fact is, it’s our long-term, permanent 25-year financial obligations to people who do nothing that are really bringing the system under. I think if we went to a five-year annually renewable rolling contract, which society—nobody in society gets that—not even NFL coaches, we would be better off. I mean, we talk about how hard it is. My brother went to work for seven years not knowing if that day he was going to be fired—OK, he’s the regular guy. We are very privileged. So we should just give up tenure. Go to a rolling, annually renewable contract situation. There are dangers to that, I understand it. But I think the fact is that if we police quality, this is our best solution.
Ciechanover:
You are absolutely right about these 25-year commitments of the university to the staff, without knowing what gain the university is going to make out of it. There are many intermediate solutions in between. For example, those who are less active can be pushed by the dean or the department chair to do more teaching, and those that are more active in research should be exempt from teaching.
Bankaitis:
If you would cut salaries, I would go with that too. But at many places, you can’t do that.
Dr. Robert H. Singer, Albert Einstein College of Medicine:
It seems to me that there is really one essential feature to ensure integrity of research, and that’s self-criticism. And I think one of the things that we need to teach our students is to look at their data not only from the pro side but also from the con side. And I’ve always been impressed, when I was on study sections, that there’s always a section of the grant proposal which lists possible pitfalls to this approach, and where things could go wrong, and how they’re going to view the data. And that’s something about grant proposals that is much more transparent than in papers. So I was thinking while we’re talking here that there should be a section in every paper that is written as to where the possible pitfalls are. Nobody knows that better than the person that is involved in the paper. In fact, in some papers, there are sections in the discussion where there is a discussion about other possible interpretations to the data—and it’s something that we’ve kind of glossed over in trying to get published papers in high-impact journals. But I think it could even go into the supplementary materials where we kind of discuss some of the potential pitfalls of the research or where we think some of the data should be pursued further.
Ehringhaus:
Just as I suspected, the reactions from you have been extremely stimulating and provocative, and certainly have given me a number of ideas to take back to my organization. Chris and I had prepared three more questions but, I think given the time, maybe the best thing for us to do is have concluding remarks from each of us, and Chris, is there anything you would like to say? Then we’ll hear from our panelists.
Pascal:
I guess I would just like to deviate a little bit and talk about the ORI mission. Because I’m sure some of you are not familiar with ORI other than the fact that we have a regulatory responsibility for responding to alleged research misconduct. ORI really looks at its role as much broader than that. It includes promoting research integrity and responsible research and programs like this, and correcting the scientific literature. So when we have misconduct cases, we do retractions or corrections in the journals, and generally the journals have been very forthcoming in assisting with that and publishing those retractions and corrections. We feel that responsible research education is very important, and we have a number of programs for that. We have funded the Council of Graduate Schools to start up a program. We also have a number of individual products, mostly web-based, on our website that have been supported with ORI funds on conflict of interest, research misconduct, data, and so on. Overall, ORI’s view toward the research enterprise is we would like it to be the best it can be. We don’t come to the ORI mission with the idea of having some sort of punitive outlook on the research enterprise. I’m a healthcare consumer—I think it’s very important and we should continue to do the research in the best way that we can. It’s clear that conducting good research is a difficult task. It’s not easy to do—mistakes are made. And I think, frankly, the consumer is willing to accept mistakes—as long as they get transparent, accurate information. They need to feel like they can get good information from the doctor, the doctor has to get good information from the biomedical literature, and basically you need to have trust all around. Overall, I think the comments at this roundtable have been very, very useful.
Ehringhaus:
Aaron, nothing?
Hulse:
Maybe I’ll make one final comment. Certainly, this is a very, very important subject. I personally, as I said, have come into this not from a biomedical career perspective, but largely from having some concerns as a scientifically trained biomedical consumer. In some ways I am leaving more concerned than when I entered this discussion, because I have learned here that there are deeper aspects to this than I realized. On the other hand, I’d like to end on a positive note: as I sit here getting myself a little depressed about all of this, what also comes to mind is the many times that I’ve sat in research meetings back at the plasma physics lab, over the 20-some-odd years that I’ve been there, and how often I have seen endless, endless, painful arguments and discussions about error bars and recalibrations, etc. And so there are examples of scientific ethics at its best. I’m not saying that institution is unique, but rather that we need to remember that there are a lot of scientists out there who really do put their heart and soul into the pursuit of truth. Scientists for whom every last error bar matters. Indeed, the reason why I remember some of the cases is that sometimes for the particular piece of data being argued about so energetically it didn’t even really matter to the big picture whether it’s a little bit off. So I remember asking myself, "Why’s everybody getting so exercised?" And the good news is—they were getting exercised because they were good scientists and they wanted it to be right. Period.
Ehringhaus:
I searched for something to conclude with, and I repaired to an old friend of mine, the old friend being this reprint of maybe the most splendid piece I’ve ever seen on mentoring. And if you haven’t read it, you should read it. And it’s written by a splendid physician, a man named Jeremiah Barondess, who is the president of the New York Academy of Medicine. It is published in the proceedings of the American Clinical and Climatological Association, which is the oldest association of internal medicine types. He, again, wrote this piece titled "A Brief History of Mentoring," and he concludes like this: "Stephen Hawking"—and this should be meaningful to you—"from whom I have in effect plagiarized the title of this talk, has predicted that of the two possible destinies of the universe—indefinite expansion or recollapse—we are probably headed for the big crunch rather than endless increase in size. Hawking has said that in this connection, he has certain advantages over other prophets of doom, namely that whatever happens, he doesn’t expect to be around to be proved wrong. The seminal difference, of course, between the fate of the universe and medical education and training is that in connection with the former we are without any prospect of inputs of any kind, whereas in the case of the latter, we are very much the architects of what will happen; what’s more, the time frame against which this will be played out will not only not be cosmic in scale, but will be such that many of us will in fact be around to see what happens, and to live with it." Thank you very much for your attention, this has been great for us. You’ve been a fine audience.
| ||||
