BIOL398-04/S15:Class Journal Week 11

Lucia I. Ramirez

• What were the main issues with the data and analysis identified by Baggerly and Coombs? What best practices enumerated by DataONE were violated? Which of these did Dr. Baggerly claim were common issues?
  • Although papers were approved, Dr. Baggerly found many inconsistencies in the report. He found a mixture of drug labels and when trying to retrieve data, it was considered confidential so that was not possible. When they tried reproducing the data, their results did not validate the results seen in the original report. In addition, problems that Dr. Baggerly found were not explained in the paper and there was a false claim of a Rhode Scholar. Some practices enumerated by DataONE that were violated are disorganization and inconsistency. Most common mistakes were complete confounding of results and easy mistakes made on excel (e.g. scrambled labels, poor documentation).
• What recommendations does Dr. Baggerly recommend for reproducible research? How do these correspond to what DataONE recommends?
  • Dr. Baggerly recommends better documentation and have more break points for approval from someone else. Specifically, he asks someone else to rerun the program in order to pass it onto the next stage of the research.
• Do you have any further reaction to this case after viewing Dr. Baggerly's talk?
  • There is nothing else to say other than there was really poor peer editing, which was caused by poor documentation.
• Go back to the methods section of the paper you presented for journal club. Do you think there is sufficient information there to reproduce their data analysis? Why or why not?
  • There is not sufficient information to reproduce their data analysis because they focused too much on the background that in their analysis for the code that was created is only available upon request. Their method also did not really go into enough detail.

William A. C. Gendron

1. What were the main issues with the data and analysis identified by Baggerly and Coombs? What best practices enumerated by DataONE were violated? Which of these did Dr. Baggerly claim were common issues?
   • The main issues is that they did not share enough of their methods to allow another group to check them. The fact that they did not have any transparency allowed their mistakes to become published. The first mistake after not disclosing their process was that they did not make a data set. This led to them making errors such as "being of by one". The flipping of the data would have led to them possibly giving patients the wrong drug.
2. What recommendations does Dr. Baggerly recommend for reproducible research? How do these correspond to what DataONE recommends?
   • Baggerly agrees with the DataONE suggestions which is to put everything in there. The code, data, labels, description and explanation. The results should be completely repeatable and easy for people to check.
3. Do you have any further reaction to this case after viewing Dr. Baggerly's talk?
   • I believe that these studies should be heavily regulated and repeated by others in the case of clinical studies. Clinical studies should be taken with the utmost caution and should be repeated by third parties. If someone else cannot figure out what you are doing it should not put human lives at risk.
4. Go back to the methods section of the paper you presented for journal club. Do you think there is sufficient information there to reproduce their data analysis? Why or why not?
   • Probably not. They did not go into enough detail for them to be easily reproducible. Most likely I would have to do a lot of research to figure out what they did and the whole paper is somewhat scatterbrained. It seems like it would be too painful to approach reproducing. With that amount of effort, I may as well spend my time writing my own paper.

Lauren M. Magee

1. What were the main issues with the data and analysis identified by Baggerly and Coombs? What best practices enumerated by DataONE were violated? Which of these did Dr. Baggerly claim were common issues?
   • The main issue was that they didn't provide readers with enough details to reproduce their findings. As Baggerly and Coombs attempted to reproduce their results, they found that their heat maps did not look the same as those included in the paper. Upon further inspection, they saw that their was an "off by one" error in their spreadsheet, because of an oversight in their column headers. Beyond this, they had to create their own software to predict if patients would be sensitive or resistant to a specific drug, because the paper did not provide a way to access the one they had used. After using their own software, Baggerly and Coombs noted that they had flipped the number of individuals sensitive and the number of individuals resistant. So they would hypothetically be administering the drug to the only patients that were resistant to it. A lot of DataOne practices were violated, but the biggest issue I believe came from their lack of a valid data set. This step is the genesis of data analysis and if you have an "off by one" error initially, then all of your other results are going to be incorrect. Baggerly claims that the following are seen as common issues: mixing up sample labels, mixing up the gene labels, mixing up the group labels, and simply providing incomplete documentation.
2. What recommendations does Dr. Baggerly recommend for reproducible research? How do these correspond to what DataONE recommends?
   • Baggerly is actually very much in line with the DataONE recommendations, he suggests that every paper have data included and given an appropriate name, as well as provenance, code, descriptions of nonscriptable steps, and a description of planned design. He recommends for clinical trials even further detailing, because the consequences for a mistaken conclusion can put human lives in danger. These are similar to those of DataONE as they highlights the formats in which it is helpful to log data and analysis so that it is easily accessible to anyone who might be interested in reproducing it.
3. Do you have any further reaction to this case after viewing Dr. Baggerly's talk?
   • I find it surprising that the researchers in these cases don't try and reproduce their results themselves. When I complete a statistical analysis on even seemingly inconsequential data, I check and recheck all of my methods to make sure my results are accurate and make sense in the context of the experiment. It is shocking to me that the medical field isn't more selective in what they are adapting into their practice. If human lives are at risk, I think it would only be fair to double and triple check if the study supporting a specific treatment was valid.
4. Go back to the methods section of the paper you presented for journal club. Do you think there is sufficient information there to reproduce their data analysis? Why or why not?
   • I don't believe there to be sufficient information in the paper I presented on for the passed journal club, because they referenced too many different articles in their methods. I think it would have been more beneficial to outline their methods step by step themselves, instead of directing the reader into a different direction and citing five different papers. It is quite possible that I could follow all the methods from the other papers and come to the same result, but it would be challenging.

Lauren M. Magee 00:25, 7 April 2015 (EDT)

Natalie Williams

1. What were the main issues with the data and analysis identified by Baggerly and Coombs? What best practices enumerated by DataONE were violated? Which of these did Dr. Baggerly claim were common issues?
   • The main issue was that the data was not reproducible. DataONE emphasizes that documentation and data entry needs to be consistent and organized. Beggarly was having trouble with getting the same numbers and had to come up with his own methods to receive the same results. They also did not mention that the software required specific handling of the inputs to get certain outputs.
2. What recommendations does Dr. Baggerly recommend for reproducible research? How do these correspond to what DataONE recommends?
   • To have reproducible research you must have good documentation of what was done so that you are not second-guessing yourself or your methods. This corresponds with DataONE’s recommendations because the slides suggest that recorded data must be organized. The labels, formatting, and how data is recorded must be consistent.
3. Do you have any further reaction to this case after viewing Dr. Baggerly's talk?
   • I am again just surprised and appalled that they disregarded Beggarly and Coombes’ findings with the inconsistency of the data. Even though they halted production and trials, they dismissed the findings. It is sad that they wanted to believe that they were willing to overlook the inconsistencies in their results because it could help so many people. In science and mathematics, it takes only one example to disprove a theory and lead people back to the drawing board. It is said that Duke forgot this practice.
4. Go back to the methods section of the paper you presented for journal club. Do you think there is sufficient information there to reproduce their data analysis? Why or why not?
   • I think that I would be able to follow their methods to reproduce the data with the computational equations used. The research does not mention what conditions the cells were under, how the cells were treated, or the specific time points measured RNA expression. As far as data analysis goes, they mentioned five criterion, but did not state what those were. They did not talk about cell cultures or strains, but focused on the computational aspect of the study.

Natalie Williams 01:53, 7 April 2015 (EDT)

Kara M Dismuke

1. What were the main issues with the data and analysis identified by Baggerly and Coombs? What best practices enumerated by DataONE were violated? Which of these did Dr. Baggerly claim were common issues?
   • To be concise, the main issue Baggerly and Coombs found with data and analysis was that it could not be reproduced. When they attempted to reproduce the data and analysis, they discovered errors: indexing errors, reused test samples, etc. These errors manifested themselves in Baggerly and Coombs being unable to match the samples; in fact, they were able to back track to figure out what the samples should have been (instead of what they were- since they some were mislabeled). Their violation of a DataONE best practice of good documentation (consistent labels, organization, etc.) produced common issues with the initial published researched- research that was eventually redacted from publication upon it being disproved.
2. What recommendations does Dr. Baggerly recommend for reproducible research? How do these correspond to what DataONE recommends?
   • He suggests that to obtain reproducible research, researchers label columns, identify which samples are which, provide the code used, provide a description of the non-scriptable statistics, and provide a description of the planned design. He also recommends reusing templates, reporting the structure, providing appendices (with some things we commonly want to know: session info, saves, file location, etc.).
3. Do you have any further reaction to this case after viewing Dr. Baggerly's talk?
   • It is amazing to me that so much work was being done that pointed towards the invalidity of the Duke research; yet, the researchers at Duke continued to press on, rather than pausing and taking a serious look at the objections. In addition, it surprised me how many times Baggerly and Coombs submitted refutations (or at a minimum serious objections/questions) to the published research, only to time and time again not be met with a spirit of cooperation in seeking the truth. Lastly, it is surprising to me that despite all of their objections, it took the discovery of a falsification of a researcher's qualification/past for the scientific community to open their eyes to this research malpractice.
4. Go back to the methods section of the paper you presented for journal club. Do you think there is sufficient information there to reproduce their data analysis? Why or why not?
   • While the equations were all included and presented in an organized fashion, a lot was left to be desired in terms of the details of the experiment. The fact that the code to the algorithm used was not included (not even in an appendix), when the conclusions drawn were done so primarily based on the results generated by the algorithm, supports my belief that I would be unable to reproduce the experiment’s data analysis. Granted, they offer the reader the option to contact them and request further information; however, if the goal of research is to be reproducible, the authors of the paper certainly do not make it easy for the reader to try to do this.

--Kara M Dismuke 02:45, 6 April 2015 (EDT)

Alyssa N Gomes

1. What were the main issues with the data and analysis identified by Baggerly and Coombs? What best practices enumerated by DataONE were violated? Which of these did Dr. Baggerly claim were common issues?
   • The issues found in this case were that it seemed as if the numbers had been forged into what the researchers had wanted to find. Under reconstruction of the same data, the same results would not have been found, or anywhere near it. Instead, Baggerly and Coombs found the true results after their own re-evaluation. Test samples had been reused. This research ended up showing that the initial experiment could be disproved. Common issues included mislabeled samples. disorganization. DataONE policies were violated because the data should have shown the step-by-step process in discovering the results, making it easily understood and reproducible.
2. What recommendations does Dr. Baggerly recommend for reproducible research? How do these correspond to what DataONE recommends?
   • Dr. Baggerly recommended that each step is carefully detailed and able to be reproduced without complications. Each of the methods and analysis should be meticulous. This explains the necessity of a scientific journal.
3. Do you have any further reaction to this case after viewing Dr. Baggerly's talk?
   • This reminded me of science fair days in Elementary school, where you could think that it would be easy to forge numbers. After taking more science classes in High School and College, it was discovered that it is OK to have errors...as long as you admit to them. It is better for the sake of science to have errors and seek to find the reason for the error than to try to make sure your experiment matches up with your hypothesis. Relating back to the case at Duke, it is very easy to discover errors and miscalculations in experiments, as many other scientists and researchers will look back on the false information for their experiments and see something wrong with the comparison of their experiment to yours.
4. Go back to the methods section of the paper you presented for journal club. Do you think there is sufficient information there to reproduce their data analysis? Why or why not?
   • To be honest, I don't think I would be able to reproduce my methods. The excel steps, unless you are skilled in Excel, seem a bit spaced. Because after I did the tests and data analyses, I personally understood how to use these mewthods, it does not mean that any other person could. So going back, I probably would put more into my exact worksheets procedures.

Alyssa N Gomes 00:52, 7 April 2015 (EDT)

Kristen M. Horstmann

1. What were the main issues with the data and analysis identified by Baggerly and Coombs? What best practices enumerated by DataONE were violated? Which of these did Dr. Baggerly claim were common issues?
   • The primary issue Baggerly and Coombs discovered was that it was not reproduceable. While they were doing further analysis on the data, they found many errors ranging from mislabeling to purposefully molding the numbers to a specific output. DataONE specifies the need for thorough documentation throughout the experiment like being detailed and organized in order to ensure it is reproducible and others can get the same results, which was not followed. This led to Baggerly and Coombs needing to create their own model in order to even find their own data to disprove it.
2. What recommendations does Dr. Baggerly recommend for reproducible research? How do these correspond to what DataONE recommends?
   • Baggerly encourages meticulous organization of the scientific process: labeling, providing strict details, important discriptions, making note of the process, etc. This is important for both other scientists trying to reproduce your work along with making sure that you understand your own process as you go and if you need to go back to retest something. This corresponds with DataONE as DataONE also discussed how important it is to keep a detailed and consistent record.
3. Do you have any further reaction to this case after viewing Dr. Baggerly's talk?
   • While I am still disgusted and saddened by the purposeful deception of the patients through the treatment, I feel like this video reminded me how Duke was equally at fault. Regardless of the information brought forward and the proof of data falsification that was presented, the trial treatments continued, even though some had been told how it could be hurting the patients more than helping. This bothers me because I feel like it was likely due to that they didn't want to tarnish their name or deny the prestige of finding a cure for cancer instead of coming forward and being honest in order to save lives. Instead, Duke chose to deny Baggerly and Coombs' findings until it became evident that they could not deny it any longer.
4. Go back to the methods section of the paper you presented for journal club. Do you think there is sufficient information there to reproduce their data analysis? Why or why not?
   • I do not believe that I would be able to reproduce the same experimental design that the paper presented. As discussed earlier, for sake of clarity/word count, the authors left out a lot of their specifics in both the scientific growth of the yeast- it didn't even specify the temperature grown- and the mathematical modelings. Very vague language was used like "data was inputted to the algorithm" or "further information avaliable upon request" and this made it difficult to fully understand along with made any chance at reproduction impossible unless I was to reach out the the main researcher for the project.

Kristen M. Horstmann 19:03, 6 April 2015 (EDT)

Tessa A. Morris

1. What were the main issues with the data and analysis identified by Baggerly and Coombs? What best practices enumerated by DataONE were violated? Which of these did Dr. Baggerly claim were common issues?
   • The main issues with the data and analysis identified by Baggerly and Coombs was that it was not reproducible. DataONE enumerates the practice of methodical recording of data, methods, and observations, which was violated in this case. Dr. Baggerly claimed that they were not able to follow what was done, and thus had trouble performing the experiment themselves.
2. What recommendations does Dr. Baggerly recommend for reproducible research? How do these correspond to what DataONE recommends?
   • Dr. Baggerly recommends that you practice meticulous note taking. In order to perform an experiment again or have others repeat your experiment, you need to have taken extremely detailed notes that are organized, easy to follow, and thorough.
3. Do you have any further reaction to this case after viewing Dr. Baggerly's talk?
   • After viewing Dr. Baggerly's talk I am more confused about how it took so long to prove that there was a case of fraud. There was more than a sufficient amount of evidence to show that fraud took place, yet the university stuck by their employee for a long time.
4. Go back to the methods section of the paper you presented for journal club. Do you think there is sufficient information there to reproduce their data analysis? Why or why not?
   • The paper I presented on for the journal club did not have anywhere near enough information to be able to reproduce the data analysis. They added a note that they would provide more detailed information upon request so it is possible that there was a a detailed methods section that was removed in order to meet a maximum word count. But from the information that was provided, there was no chance of reproducing the data. In order to reproduce it, a step-by step guide would be needed, and the paper did not provide that

Tessa A. Morris 17:37, 6 April 2015 (EDT)

Karina Alvarez

1. What were the main issues with the data and analysis identified by Baggerly and Coombs? What best practices enumerated by DataONE were violated? Which of these did Dr. Baggerly claim were common issues?
   • The main issue in this case was that the researchers altered the data to fit what they were looking for or anticipating. When Baggerly and Coombs tried to recreate the experiment, they did not get the same results. The DataONE policy violated was when the researchers did not share their methods for the experiment so that it could be reproducible.
2. What recommendations does Dr. Baggerly recommend for reproducible research? How do these correspond to what DataONE recommends?
   • Dr. Baggerly recommends recording every step so that other researchers will be able to recreate the experiment and either corroborate or disprove your results accurately.
3. Do you have any further reaction to this case after viewing Dr. Baggerly's talk?
   • I was surprised at how long it took the university to acknowledge Baggerly and Coomb's work. Even though they proved that they were meticulous and careful in their methods, the institution ignored them because it was inconvenient. I am surprised that any entity or person would put convenience before the lives of other people.
4. Go back to the methods section of the paper you presented for journal club. Do you think there is sufficient information there to reproduce their data analysis? Why or why not?
   • I think that I might be able to recreate the experiment but I would struggle in some parts to understand exactly what tricks in Excel to use. This experiment requires Excel experience and someone without that may struggle to understand the methods.

Jeffrey Crosson

1. What were the main issues with the data and analysis identified by Baggerly and Coombs? What best practices enumerated by DataONE were violated? Which of these did Dr. Baggerly claim were common issues?
   • They weren't closely peer-reviewed. It's as if it was skimmed. Their methods weren't clear, so errors couldn't be easily seen.
2. What recommendations does Dr. Baggerly recommend for reproducible research? How do these correspond to what DataONE recommends?
   • Dr. Baggerly thinks all of the information should be put in there. DataONE has the same opinion. The results should be clear and easy to reproduce.
3. Do you have any further reaction to this case after viewing Dr. Baggerly's talk?
   • It should be critically reviewed by many people and reproduced mutlipple times before being applied to patients.
4. Go back to the methods section of the paper you presented for journal club. Do you think there is sufficient information there to reproduce their data analysis? Why or why not?
   • No, I would have to contact them before I would be able to completely reproduce it. Much of the material was pulled from other papers and it was pretty esoteric to me.

Jeffrey Crosson 02:54, 14 April 2015 (EDT)