DataONE:Protocols/Find GEO reuses - Revision history

Heather A Piwowar: add ArrayExpress variant

2010-07-20T20:38:19Z

add ArrayExpress variant

←Older revision		Revision as of 20:38, 20 July 2010
Line 99:		Line 99:
	* To Do: compare results to GEO's list of 3rd party reuses: http://www.ncbi.nlm.nih.gov/projects/geo/info/ucitations.html		* To Do: compare results to GEO's list of 3rd party reuses: http://www.ncbi.nlm.nih.gov/projects/geo/info/ucitations.html

		+	== Variants ==
		+	===Reuses of ArrayExpress datasets===
		+	* as with GEO datasets, but gather ArrayExpress accession numbers through screen scrape of ArrayExpress website (is there a better way?).
		+	** used this url: http://www.ebi.ac.uk/microarray-as/ae/browse.html with Display=500 and click "Detailed view" in the header. Warning, this is slow.
		+	** most expedient data extraction I could easily figure out: actually copy the raw data from within the frame and paste into a text file
		+	* didn't use any varients of ArrayExpress accession numbers. A quick google scholar exploration suggested that people are pretty consistent with the E-XXXX-nnnn formatting.
		+	* obviously the "NOT pmc_gds[filter]" isn't going to do much because it captures links between PMC and GEO not ArrayExpress. Left it in there anyway, since a large proportion of ArrayExpress content is pulled from GEO, might exclude some of the data creation articles
		+	* expect a higher proportion of data creation articles in resulting set, because no very effective automated filter

	==Application==		==Application==

Heather A Piwowar: /* Protocol Overview */ clean up content

2010-07-15T20:53:25Z

Protocol Overview: clean up content

←Older revision		Revision as of 20:53, 15 July 2010
Line 20:		Line 20:

	==Protocol Overview==		==Protocol Overview==
-	*Query GEO for all GDS and ~~GDS~~ accession numbers for datasets	+	*Query GEO for all GDS and GSE accession numbers for datasets deposited within specified date range
-	*Query PubMed Central for these accession numbers ~~in the~~ full text of papers available through PubMed Central	+	*Determine which GSE accession numbers are within of which GDS numbers, to estimate total number of data packages
-	*~~Enumerate~~ the PMC papers that ~~reused~~ GEO data	+	*Query PubMed Central for each of these accession numbers, using eutils to search full text of papers available through PubMed Central
-	*~~Manually exclude <<~~	+	*Exclude the PMC papers that created the GEO data, using Entrez links and guided manual inspection
-	*Estimate what percent of ~~these~~ papers ~~depended on~~ the ~~GEO~~ data for ~~their scientific contribution~~	+
		+	Optionally:
		+	*Extrapolate to all of PubMed, using yearly proportion of articles with the MeSH term "gene expression profiling" in PMC vs all of PubMed
		+	*Estimate what percent of reuse papers have authors in common with the corresponding data creation paper, using last names, institutions, and manual inspection
		+	*Estimate what percent of reuse papers use data for metaanalysis, using MeSH
		+	*Estimate what percent of reuse papers use data for tool and method validation, using MeSH and journal title keywords

	==Materials==		==Materials==

Heather A Piwowar: /* Protocol Overview */

2010-07-15T19:00:19Z

Protocol Overview

←Older revision		Revision as of 19:00, 15 July 2010
Line 21:		Line 21:
	==Protocol Overview==		==Protocol Overview==
	*Query GEO for all GDS and GDS accession numbers for datasets		*Query GEO for all GDS and GDS accession numbers for datasets
-	*Query PubMed Central for these accession numbers in the full text of ~~PMC~~ papers ~~published between 1900 and 2009~~	+	*Query PubMed Central for these accession numbers in the full text of papers available through PubMed Central
	*Enumerate the PMC papers that reused GEO data		*Enumerate the PMC papers that reused GEO data
		+	*Manually exclude <<
	*Estimate what percent of these papers depended on the GEO data for their scientific contribution		*Estimate what percent of these papers depended on the GEO data for their scientific contribution

Heather A Piwowar: /* Background */ more content

2010-07-15T18:58:29Z

Background: more content

←Older revision		Revision as of 18:58, 15 July 2010
Line 11:		Line 11:
	So what does the reuse behaviour look like for other datasets?		So what does the reuse behaviour look like for other datasets?

-	We don’t know~~. It is difficult to track reuse~~. There have been a few surveys, but they suffer from limited scope and self-reporting biases. ~~I gather that download~~ stats are poorly correlated with perceived value. So let’s track reuse in the published literature.	+	We don’t know. There have been a few surveys, but they suffer from limited scope and self-reporting biases. Download stats are poorly correlated with perceived value <<citation?>>. So let’s track reuse in the published literature.
		+
		+	Unfortunately, there are nto well-established attribution formats and standards for data to facilitate the sort of automated citation analysis that bibliomatricians perform with journal articles. Following the track of data is difficult in several additional ways: datasets do not have unambiguous identifiers, attribution is often within full text and thus difficult to query across journals and disciplines, and it is difficult to disambiguate the mention of a dataset in the context of reuse from the mention of a dataset deposit.
		+
		+	Restricting our focus to gene expression microarray data helps to address several of these issues. First, most shared gene expression microarray data is shared in once central repository: the NCBI's Gene Expression Omnibus (GEO). It is common practice to refer to datasets by their GEO accession numbers, and the GEO accession numbers have a fairly unique format. Furthermore, most creations and reuses of gene expression microarray data in the published literature are indexed by PubMed and are increasingly (as per NIH mandate) available for full-text query in PubMed Central. The coordinated Entrez databases and eUtils web service means that full-text can be queried automatically, links between articles and datasets can be monitored, and standard indexing metadata can be collected. All disciplines should be so lucky.
		+
		+	Below, then, is a protocol for using these resources to collect information on reuse. Please note the limitations section, and contribute if you have other ideas!

	==Protocol Overview==		==Protocol Overview==

Heather A Piwowar: /* Aim */ flushed out a bit

2010-07-15T18:37:01Z

Aim: flushed out a bit

←Older revision		Revision as of 18:37, 15 July 2010
Line 2:		Line 2:

	==Aim==		==Aim==
-	To collect data on the ~~uses of uses~~ of datasets in the published literature. This ~~proposal~~ focuses on reuses of gene expression microarray datasets stored in NCBI's Gene Expression Omnibus (GEO) repository and tracks reuses attributed through accession numbers.	+	The aim of this protocol is to collect data on the reuses of datasets in the published literature. This particular protocol focuses on reuses of gene expression microarray datasets stored in NCBI's Gene Expression Omnibus (GEO) repository and tracks reuses attributed through accession numbers within the full text of articles in PubMed Central.

	==Background==		==Background==

Heather A Piwowar: /* Known uses */

2010-07-14T21:17:46Z

Known uses

←Older revision		Revision as of 21:17, 14 July 2010
Line 102:		Line 102:

	===Known uses===		===Known uses===
-	* A [[DataONE:GEO_reuse_study/Phase_1\|first snapshot]] of this data is included in ~~an perspectives~~ manuscript-in-progress	+	* A [[DataONE:GEO_reuse_study/Phase_1\|first snapshot]] of this data is included in a manuscript-in-progress
	* [[Special:WhatLinksHere/DataONE:Protocols/Find_GEO_reuses\|What else on OpenWetWare links to this page]]		* [[Special:WhatLinksHere/DataONE:Protocols/Find_GEO_reuses\|What else on OpenWetWare links to this page]]

Heather A Piwowar: /* Known uses */ link to phase 1 page

2010-07-14T21:17:15Z

Known uses: link to phase 1 page

←Older revision		Revision as of 21:17, 14 July 2010
Line 102:		Line 102:

	===Known uses===		===Known uses===
-	* ~~draft~~ manuscript	+	* A [[DataONE:GEO_reuse_study/Phase_1\|first snapshot]] of this data is included in an perspectives manuscript-in-progress
	* [[Special:WhatLinksHere/DataONE:Protocols/Find_GEO_reuses\|What else on OpenWetWare links to this page]]		* [[Special:WhatLinksHere/DataONE:Protocols/Find_GEO_reuses\|What else on OpenWetWare links to this page]]

Heather A Piwowar: /* Assumptions, Limitations, and Unknowns */ tweak

2010-07-14T21:15:27Z

Assumptions, Limitations, and Unknowns: tweak

←Older revision		Revision as of 21:15, 14 July 2010
Line 106:		Line 106:

	==Assumptions, Limitations, and Unknowns==		==Assumptions, Limitations, and Unknowns==
-	This is a ~~conservative estimate~~ because:	+	This protocol captures a subset of all dataset reuses because of several limitations:
-	* Many data citations ~~not~~ attributed using accession numbers	+	* Many data citations are attributed without using accession numbers
	** don't have a good way to estimate this yet		** don't have a good way to estimate this yet
	** would require a manual inventory, similar to [[User:Sarah_Judson/Notebook/DataOne_DataCitationPractices\|Sarah's data citation inventory in DataONE summer 2010 project]]		** would require a manual inventory, similar to [[User:Sarah_Judson/Notebook/DataOne_DataCitationPractices\|Sarah's data citation inventory in DataONE summer 2010 project]]
	* Many papers are not in PubMed Central		* Many papers are not in PubMed Central
	** we can estimate what percentage are and then try to extrapolate to all papers. For example, using "gene expression profiling"[mesh] query in PMC vs PubMed over 2007-2009 suggests PMC contains 30% of all related papers in PubMed		** we can estimate what percentage are and then try to extrapolate to all papers. For example, using "gene expression profiling"[mesh] query in PMC vs PubMed over 2007-2009 suggests PMC contains 30% of all related papers in PubMed
-	* Papers in PubMed Central may not be representative
-	* Deposits into PMC not stable over time, distribution may change over time, may be skewed based on open-access uptake or NIH-funding levels in various communities
-	* our estimates do not consider reuses after our study timeframe
	** many datasets will continue to be used in the future... these reuses are obviously not continued in our estimate		** many datasets will continue to be used in the future... these reuses are obviously not continued in our estimate
	* our methods do not find studies that both create and reuse data		* our methods do not find studies that both create and reuse data
Line 120:		Line 117:
	** we don't have an estimate of how many this is, would require manual inventory		** we don't have an estimate of how many this is, would require manual inventory
	* Doesn't capture reuse outside the peer-reviewed literature (for example, reuse during training)		* Doesn't capture reuse outside the peer-reviewed literature (for example, reuse during training)
		+
		+	Furthermore, extrapolations based on this data may be biased:
		+	* Papers in PubMed Central may not be representative
		+	* Deposits into PMC not stable over time, distribution may change over time, may be skewed based on open-access uptake or NIH-funding levels in various communities
		+	* our estimates do not consider reuses after our study timeframe

	===Open Questions===		===Open Questions===

Heather A Piwowar: /* Assumptions, Limitations, and Unknowns */ tweak

2010-07-14T21:13:04Z

Assumptions, Limitations, and Unknowns: tweak

←Older revision		Revision as of 21:13, 14 July 2010
Line 107:		Line 107:
	==Assumptions, Limitations, and Unknowns==		==Assumptions, Limitations, and Unknowns==
	This is a conservative estimate because:		This is a conservative estimate because:
		+	* Many data citations not attributed using accession numbers
		+	** don't have a good way to estimate this yet
		+	** would require a manual inventory, similar to [[User:Sarah_Judson/Notebook/DataOne_DataCitationPractices\|Sarah's data citation inventory in DataONE summer 2010 project]]
		+	* Many papers are not in PubMed Central
		+	** we can estimate what percentage are and then try to extrapolate to all papers. For example, using "gene expression profiling"[mesh] query in PMC vs PubMed over 2007-2009 suggests PMC contains 30% of all related papers in PubMed
		+	* Papers in PubMed Central may not be representative
		+	* Deposits into PMC not stable over time, distribution may change over time, may be skewed based on open-access uptake or NIH-funding levels in various communities
	* our estimates do not consider reuses after our study timeframe		* our estimates do not consider reuses after our study timeframe
-	** many datasets ~~we are considering~~ will continue to be used in the future... these reuses are obviously not continued in our estimate	+	** many datasets will continue to be used in the future... these reuses are obviously not continued in our estimate
-	** could estimate this impact if we examine data deposited 7 years ago?	+
-	* Many papers not in PubMed Central	+
-	** using "gene expression profiling"[mesh] query in PMC vs PubMed over 2007-2009 suggests PMC contains 30% of all related papers in PubMed	+
	* our methods do not find studies that both create and reuse data		* our methods do not find studies that both create and reuse data
	** to narrow down our query results, we automatedly eliminate studies that create data... even though these same studies may also reuse data		** to narrow down our query results, we automatedly eliminate studies that create data... even though these same studies may also reuse data
	** we don't have an estimate of how many this is, would require manual inventory		** we don't have an estimate of how many this is, would require manual inventory
-	* Many data citations not attributed using accession numbers
-	** don't have a good way to estimate this yet
-	** would require a manual inventory, similar to Sarah's data citation inventory in DataONE summer 2010 project
-	** maybe out-of-scope to get this estimate for this project, just admit it is an underestimate
	* Doesn't capture reuse outside the peer-reviewed literature (for example, reuse during training)		* Doesn't capture reuse outside the peer-reviewed literature (for example, reuse during training)
-	* Deposits into PMC not stable over time, distribution may change over time

	===Open Questions===		===Open Questions===
-	* How to estimate what percent of these papers depended on the GEO data for their scientific contribution ~~efficiently~~?	+	* How to efficiently estimate what percent of these papers depended on the GEO data for their scientific contribution?

	==Possible Enhancements==		==Possible Enhancements==

Heather A Piwowar: /* Related references */ fixed

2010-07-14T21:06:20Z

Related references: fixed

←Older revision		Revision as of 21:06, 14 July 2010
Line 134:		Line 134:
	==Related references==		==Related references==
	<biblio>		<biblio>
-	#Piwowar-blogGauntlet [http://researchremix.wordpress.com/2010/07/05/studying-reuse-of-geo-datasets-in-the-published-literature/ blog ~~intro~~]	+	#Piwowar-blogGauntlet Piwowar, HA. Studying Reuse Of GEO Datasets In The Published Literature. Research Remix. July 5 2010. [http://researchremix.wordpress.com/2010/07/05/studying-reuse-of-geo-datasets-in-the-published-literature/ blog post]
	#Piwowar-AMIA2008 pmid=18998887		#Piwowar-AMIA2008 pmid=18998887
-	#Piwowar-~~AMIA2008~~ BioLINK ~~paper~~	+	#Piwowar-BioLINK2008 Piwowar, Wendy W Chapman (2008) Linking database submissions to primary citations with PubMed Central. BioLINK 2008, Toronto Canada. [http://www.researchremix.org/data/bioLINK2008%20Piwowar.doc Full text]
	</biblio>		</biblio>