Peer Review Simulation Project

What is this project?

The purpose of this project to use the resources of the internet, trust based social networking, filtering, and advertising to simulate the services provided by peer review journals. The intent is to use these tools to provide all those benefits without a central hierarchal control, but rather horizontal methods.

What is the purpose of this wiki?

This wiki is intended to be a place where people in the community, i.e., scientists, coders, and other interested parties, can discuss and learn about this project. Participation in this project is intended to be as open as possible, from actual implementation to comments and criticism. This is to be a community effort intended to provide a service to the community.

Project Details

The arxiv is an excellent place to publish papers. Distribution is also taken care of and its freely available to all. The only problem is its hard to distinguish what papers are worth reading since they haven't been reviewed or filtered out, by a central authority for example. This can be taken care of in part by the community, by allowing members to make comments or ratings.

However, there's a lot of problems with this slashdot style commenting scheme. Manual filtering make it a pain to read comments. Comments are tedious for the reader since the reader must manually filter through comments to determine which ones are worth reading. Rating systems have their own problems, including those involving gaming. In addition, single parameter filtering has the high potential of misrepresenting the needs of readers. What journals provide what these schemes do not it an understanding of who is worth using as a referee and a good knowledge of the subjects invloved. Editors know experts in the field of a papers subject. However, this is not secret knowledge. Experts of a field are well known to the community. So what we need to do is use the communities knowledge of experts to filter content for us. This is the scheme I had in mind:

1. Give users tools for creating social networks:
   • Allow users to set up user bios and user pages
   • Allow users to form groups
   • Allow groups to maintain membership, manage rules, maintain a group page, bios
2. Allow users (and groups) to make pages for their papers
   • This would basically be a dedicated page for the paper which would contain a link to the arxiv in addition to comments on the paper by other users.
3. Allow users to make comments on the paper
   • Here we allow *any* person to make comments. You can break these comments into several sections: general comments, errors, criticisms, general discussion, etc. Anonymity is an option for all posters, even anonymity to all but a select group, but the system would recognize the commenter.
4. Allow users to assign trust to persons and groups
   • Here, the user can depend on their social network to make these assignments, or they can be manual by the user. Basically you allow the user to pick who in the network they listen to. This could be as closed as the people in their groups or as open as anybody. In addition you can give them additional tools to alter the sphere of trust: trust everyone your group members trust, trust by a connectivity measure, trust by some other algorithms or ratings, etc. Our job would be to provide them with the most effective tools for doing so.
5. Allow users (and groups) to promote papers and comments.
   • Here you would basically let users give shoutouts to papers they think are important. People can filter what paper advertisements they see, and trust groups are one way they could filter it. They can apply other parameters and several independent filters if they wish (tools). In addition to shoutouts, you can broadcast your "top 10 papers", "headline nominees", etc. Also I think you should be able to shout out people you think others should trust and let receivers manually add those people into your trust network (if not present already). Comments also should be promotable, whether they be your own or others.
6. Allow users to set up customizable filters on their user page.
   • This is the real power in the system. Ever user can advertise content, and each user has complete control of what content comes to them ie on their user page. They can set up customizable filters depending on what content they want, whether its papers from their in-group that are under 5 days old, comments from particular people, etc.

I believe that all these would effectively simulate many of the benefits derived from journals. In addition, extra services would be provided to the community that journals do not proivide: information becomes fast, easy, free, and highly customizable. In addition potential connectivity and cooperation increase manyfold.

I would also note that the scheme could be portable, i.e., functions could be ported to a remote web page that had access to the system.

There is also another project I've started in same spirit with music (and potentially any information): Internet Music Task Force

If you have any comments or would like to do work on this project, please contact me at devon@phys.unm.edu or just post them here:

Comments

Steve Koch 00:14, 30 October 2007 (CDT):Hey Devon, Here are some links to relevant discussion on OWW:

• Science 2.0/Brainstorming
• Journal_Club