Peer Review Simulation Project
What is this project?
The purpose of this project to use the resources of the internet, trust based social networking, filtering, and advertising to simulate the services provided by peer review journals. The intent is to use these tools to provide all those benefits without a central hierarchal control, but rather horizontal methods.
What is the purpose of this wiki?
This wiki is intended to be a place where people in the community, i.e., scientists, coders, and other interested parties, can discuss and learn about this project. Participation in this project is intended to be as open as possible, from actual implementation to comments and criticism. This is to be a community effort intended to provide a service to the community. This wiki is a work and progress.
Project Details
This project is an attempt to simulate all the services provided to the community by journals using horizontal methods in the absence of centralized control.
The arxiv is an excellent place to publish papers. Distribution is also taken care of and its freely available to all. The only problem is its hard to distinguish what papers are worth reading since they haven't been reviewed or filtered out, by a central authority for example. This can be taken care of in part by the community, by allowing members to make comments or ratings.
However, there's a lot of problems with this slashdot style commenting scheme. Manual filtering make it a pain to read comments. Comments are tedious for the reader since the reader must manually filter through comments to determine which ones are worth reading. Rating systems have their own problems, including those involving gaming. In addition, single parameter filtering has the high potential of misrepresenting the needs of readers. Scientists are picky in this regard: the want their information quick, easy, and prefiltered.
Simulation
What journals provide what these schemes do not it an understanding of who is worth using as a referee and a good knowledge of the subjects invloved. Editors know experts in the field of a papers subject. However, this is not secret knowledge. Experts of a field are well known to the community. So what we need to do is use the communities knowledge of experts to filter content for us. This is the scheme I had in mind:
- Give users tools for creating social networks
- Allow users to set up user bios and user pages
- Allow users to form groups
- Allow groups to maintain membership, manage rules, maintain a group page, bios
- Allow users (and groups) to make pages for their papers
- This would basically be a dedicated page for the paper which would contain a link to the arxiv in addition to comments on the paper by other users.
- Allow users to make comments on the paper
- Here we allow *any* person to make comments. You can break these comments into several sections: general comments, errors, criticisms, general discussion, etc. Anonymity is an option for all posters, even anonymity to all but a select group, but the system would recognize the commenter.
- Allow users to assign trust to persons and groups
- Here, the user can depend on their social network to make these assignments, or they can be manual by the user. Basically you allow the user to pick who in the network they listen to. This could be as closed as the people in their groups or as open as anybody. In addition you can give them additional tools to alter the sphere of trust: trust everyone your group members trust, trust by a connectivity measure, trust by some other algorithms or ratings, etc. Our job would be to provide them with the most effective tools for doing so.
- Allow users (and groups) to promote papers and comments
- Here you would basically let users give shoutouts to papers they think are important. People can filter what paper advertisements they see, and trust groups are one way they could filter it. They can apply other parameters and several independent filters if they wish (tools). In addition to shoutouts, you can broadcast your "top 10 papers", "headline nominees", etc. Also I think you should be able to shout out people you think others should trust and let receivers manually add those people into your trust network (if not present already). Comments also should be promotable, whether they be your own or others.
- Allow users to set up customizable filters on their user page
- This is the real power in the system. Ever user can advertise content, and each user has complete control of what content comes to them ie on their user page. They can set up customizable filters depending on what content they want, whether its papers from their in-group that are under 5 days old, comments from particular people, etc.
Summary
I believe that all these would effectively simulate many of the benefits derived from journals. In addition, extra services would be provided to the community that journals do not provide: information becomes fast, easy, free, and highly customizable. In addition potential connectivity and cooperation increase manyfold. Filters are the central theme here, but you can also play with methods of advertising comments, for instance yours or other peoples. Think of it as idea endorsement. This will only be seen by people who have their filters set to do so, by trust or other means. You could even have it so that people can comment locally (from their user page) depending on their filters. This is all up to experimentation. I think our job is to give people the tools to have easy access and distribution of content that they want. I think that's an optimal way to connect people and that participation will follow from that.
There will certainly be a lot of resistance to adopting this model. But having it up is better than the current situation. At the moment scientists have no choice but to adopt the old model because of a lack of options. Our job is to do as good of a job or better than the journals to give people at least the option to transition over (or use both). If we do a better job, which i feel like we can do, then overall migration will be positive. We are providing a service to the community, and if we can't do what the journals do then so be it, but we'll be doing things they dont do.
I would also note that the scheme could be portable, i.e., functions could be ported to a remote web page that had access to the system.
Progress
Currently we are attempting to generate interest and get the community involved. If you are interested in helping with project, please contact us or post comments here.
There is also another project I've started in same spirit with music (and potentially any information): Internet Music Task Force
If you have any comments or would like to do work on this project, please contact me at devon@phys.unm.edu or just post them here:
Comments
Steve Koch 00:14, 30 October 2007 (CDT):Hey Devon, Here are some links to relevant discussion on OWW:
Valuable comments I feel I need to post (mostly from email)
(Steve Koch): Hey Devon – I think you have a great idea. The thing that really struck me the first time I read it was the “groups” idea – most of the network-y things I am part of don’t really have that added level of grouping, and it may add a very valuable effect. It would give users both a motivation to post comments (hey look, I’m reading papers!) as well as a motivation to be careful and intelligent (oh man, I don’t want to embarrass myself in front of people who actually know me). It would also provide an outlet for local journal clubs, which often are a huge amount of thinking that goes un-recorded with no follow up. So your idea could end up being a really valuable outlet for those sessions.
Also it occurred to me that as a peer reviewer, I’d be likely to want an outlet for my criticisms when the paper is accepted and I still have some remaining questions or criticisms. Of course, this would give up my anonymity…
- Devon Hjelm: This is an astute observation. This is totally in the spirit of the project.
