CoScience.net – a cooperative science forum for cooperative forking path analysis

CoScience.net is a forum designed to facilitate cooperative research projects by offering a means through which researchers can, with minimal administrative burden: share projects, recruit additional participating laboratories, and expand the use of collected data. Whereas CoScience.net was developed initially to facilitate a specific research project funded by the German Research Foundation (DFG), this platform supports additional research projects that adhere with the cooperative principles upheld by this network. Distinct from a general crowdsourcing infrastructure for various research plans (see Psychological Science Accelerator), CoScience.net brings together smaller groups of researchers to join forces for specific projects.

This is how it works: Grounded in the principles of cooperative forking path analysis (cFPA, Wacker 2018) groups of five or more researchers from different labs share the load of data collection for testing a set of registered hypotheses using an agreed-upon design. Further, the collaborative group of researchers discuss and systematically explore analysis options for the data to minimize undisclosed flexibility (researcher degrees of freedom, garden of forking paths) that leads to an increased likelihood of false-positive results (Simmons, Nelson & Simonsohn, 2011).

Following registration of a project on CoScience.net, a public abstract of the project is posted along with a list of members who have already agreed to contribute to data collection. After joining CoScience.net and accepting its ground rules, researchers may apply for membership of a specific project wherein they will contribute regularly to the joint data collection effort and subsequent cooperative data analysis. Members of CoScience.net may also request data from completed projects for additional analyses not previously registered. Both the registration of additional objectives/hypotheses by project members and data requests by other researchers are discussed among the extant project members in an open peer-review on CoScience.net.

It is our hope that, by reducing undisclosed researcher flexibility in data analysis, promoting increased statistical power, and facilitating open expert discussions of hypotheses and analysis options, CoScience.net and the principles of cFPA will be instrumental in improving (psychological) science.

Join CoScience.net

Joining CoScience.net is free for researchers who would like to contribute to projects for which they are experts. They may contribute to projects by (1) joining a project as a member, or (2) requesting data from completed projects for additional analyses not previously registered by group members. The benefits of each are outlined below. Please note that members of CoScience.net are required to adhere to our Code of Conduct and accept our Terms and Conditions.

Project members share the load of data collection, participate in cFPA and get involved in writing/submitting a manuscript led by the project member who registered the respective hypothesis. Project membership is subject to approval by the project maintainer who, together with the initial members of the project, also defines the specific requirements for project membership and (if necessary) more specific ground rules for the project (e.g. specific rules for authorship, access to data, level of expertise required).

Project members’ benefits

  • Project members will be provided with a complete data collection package by the project maintainer. This will include (if applicable) experimental control scripts for behavioral paradigms, online questionnaires and tests, and all necessary informed consent forms and written instructions for experimenters.
  • Project members may register additional objectives/hypotheses for analyses based on the project’s data while data collection remains in progress. Registration of additional hypotheses may not overlap significantly with pre-existing hypotheses and are subject to peer-review by the other project members.
  • Project members generally serve as first authors for manuscripts based on the objectives/hypotheses they registered, if they also successfully take the lead in writing and managing the paper.
  • All project members will be listed as co-authors for all manuscripts based on the project’s data in random order if, and only if, they contributed to data collection AND actively participated in discussing (alternative) data processing and analysis paths for the manuscript’s hypotheses. If one of these individuals supervises the first author and contributes to a manuscript as equally as the first author, this individual may qualify for senior or shared first authorship, with the final decision to be confirmed by all the project members.

Data requesters’ benefits

  • Following completion of data collection within a project, members of CoScience have the opportunity to gain access to the project’s data at no cost, subject to positive open peer-review of their additional objectives/hypotheses and analysis ideas by the members of the project.
  • Data requesters will generally serve as first authors of publications based on their data requests if they:
    • provide a complete analysis script based on Matlab and R code that works on a permutated random subsample of raw data (with each variable shuffled independently) within six months of gaining access to this random subsample of data,
    • submit an initial draft for a manuscript for review by all group members within six months after receiving the project results based on their analysis script from the project maintainer, and
    • take the lead for further optimization of the manuscript based on their data request during both the internal CoScience review process and the review process after (re)submission to an international peer-reviewed journal.

Join CoScience.net

Browse existing projects

Project 1: A novel collaborative approach for EEG personality neuroscience research

Project maintainer: Jan Wacker, Universität Hamburg, Germany

 

Abstract

Linking individual differences in the electroencephalogram (EEG) to variation in personality can provide important insights into the biological basis of human individuality and constitutes a prominent approach within the young and burgeoning field of Personality Neuroscience. Frequently investigated EEG measures include frontal EEG asymmetry, error and feedback-related negativities/frontal-midline theta, and the late positivities. Despite decades of research, the links between these measures and individual differences in personality remain poorly understood due to concerns of low replicability that currently thwart many areas of Psychology and Neuroscience. It has now become clear that low statistical power and undisclosed flexibility in data analysis are at the core of the current replicability crisis.

The present project aims to alleviate this unsatisfactory status quo and provide a firm basis for future EEG personality neuroscience work through initial application of a novel approach for empirical research termed “collaborative forking path analysis” (cFPA). Briefly, cFPA is based on (1) close cooperation among teams of researchers sharing the load of data collection for an a priori agreed-upon, highly standardized experimental setup in order to achieve high statistical power and (2) conducting a systematic scripted analysis (and comparison) of all defensible data analysis paths identified in joint expert meetings. The present application of the cFPA approach entails the collection of data from several paradigms allowing quantification of each of the EEG measures noted above from N = 720 participants distributed across nine laboratories. This unprecedented collaborative effort within the format of a regular individual grant will allow us to (1) probe the replicability of prominent EEG-personality associations, (2) provide initial direct tests of potential moderators for several such associations, (3) derive initial estimates for the influence of inter-laboratory variability (e.g., in technical setups and laboratory personnel) on each of the EEG measures and their associations with other variables, (4) provide a systematic investigation of the influence of analysis choices on each of the effects of interest, (5) validate several novel methods for quantifying individual differences in EEG activity, and (6) provide an easily accessible high-quality multivariate data set for further theory-based as well as more inductive EEG personality neuroscience work by researchers around the globe. Most importantly, if demonstrated to be feasible in this initial application to a methodologically relatively demanding field, the cFPA approach may serve as a blueprint for significantly changing the practice of empirical research in Personality Neuroscience and various other areas towards increased statistical power and transparency.

Project members

Jan Wacker, Universität Hamburg, Germany (project maintainer)

André Beauducel, Universität Bonn, Germany

Jürgen Hennig, Justus-Liebig-Universität Giessen, Germany

Johannes Hewig, Julius-Maximilians-Universität Würzburg, Germany

Andrea Hildebrandt, Carl von Ossietzky Universität Oldenburg, Germany

Erik Mueller, Philipps-Universität Marburg, Germany

Roman Osinsky, Osnabrück University, Germany

Jutta Stahl, Universität zu Köln, Germany

Alexander Strobel, Technische Universität Dresden, Germany

Requirements and rules for project membership

Project members are required to

  • hold a PhD in Psychology, Neuroscience or a related field,
  • have published at least three international peer-reviewed journal articles broadly relevant to the project,
  • have sufficient access to an EEG laboratory with at least 64 EEG channels plus 5 additional channels for EOG and ECG (e.g. Biosemi Active Two) and the necessary expertise to supervise data collection in this laboratory,
  • have the capacity and necessary funding to collect 80 data sets from healthy heterosexual volunteers from an age range of 18 to 30 years who speak German as their native language (4 hours per participant),
  • agree to participate in two workshops conducted in Hamburg, Germany, for group discussions of project analysis methods and results, respectively, and
  • agree to send their experimenters to a 1-day training workshop in Hamburg, Germany, before data collection.

    All members will be invited to two workshops conducted in Hamburg, Germany. Participation will be free of charge and reimbursements for travel costs are available for a limited number of members. At the first workshop, conducted while data collection is still in progress, members will collectively compile all defensible analysis paths for each of their research questions and pinpoint their own preferred approach. Following this workshop, researchers at Hamburg will apply all alternative approaches to each of the research questions and supply each member with an integrated analysis script producing parameter estimates for each condition/participant and analysis path along with a data file containing the results of the final statistical tests of the respective research question. At the second workshop, each member will present his/her project findings obtained with the preferred analysis path along with an analysis of the impact of various methodological analysis decisions and future directions/opportunities for collaborative work.

    If you consider becoming a member of this project, please send an email to the project maintainer (Jan Wacker: jan.wacker@uni-hamburg.de) stating the title of the project and confirming that you fulfill all membership requirements and agree with all membership rules listed on this page and on CoScience.net. The project maintainer will then provide you with further details concerning the project including the work programme and the hypotheses/objectives registered so far. Afterwards, you should be in a position to make an informed decision concerning your membership in this project and are expected to communicate it to the project maintainer. After final acceptance into the project, the project maintainer will provide you with the data collection package and you will also have the opportunity to submit additional objectives/hypotheses on the project’s CoScience forum for interactive open peer-review by the project members.

    Requirements and rules for data requests

    Researchers requesting project data are required to

  • hold a PhD in Psychology, Neuroscience, or a related field.
  • have published at least three international peer-reviewed journal articles broadly relevant to the project.

    If you would like to request data from this project please send an email to the project maintainer (Jan Wacker: jan.wacker@uni-hamburg.de) stating the title of the project and confirming that you fulfill all data request requirements and agree with all data request rules listed on this page and on CoScience.net. The project maintainer will then provide you with further details concerning the project including the work programme and the hypotheses/objectives registered so far. He/she will then also invite you to submit your data request on the project’s CoScience forum for interactive open peer-review by the project members.

  • Project 2: Associations among set-shifting tasks, reinforcement learning, and personality traits

    Project maintainer: Wiebke Herrmann, Universität Hamburg, Germany

    Abstract

    Work in progress

    Post a new project

    Unless further funding is acquired, CoScience.net can only handle a limited number of projects. Therefore, in the spirit of the project for which CoScience.net was originally developed, support is currently restricted to projects within Psychology that aim to employ cFPA. Initial project teams should include members from at least five different labs, each of whom is an expert in the project’s field of study (typically documented by at least three relevant international peer reviewed articles). In the absence of strong arguments for larger effect sizes, planned projects should at least secure high statistical power (> .95) to detect the effect size of rho = .20 typical for replication studies within psychology (e.g., Open Science Collaboration, 2015) with an alpha of .05. Furthermore, project teams are required to confirm, that they (1) have already acquired the necessary funding for completion of the project, (2) have the necessary analysis/scripting expertise to apply cFPA, and (3) will take on the necessary managing and reviewing duties within the project to guarantee appropriate handling of data requests.

    If you would like to post a project conforming to these requirements, please send an informative abstract of the planned project (no more than 3000 characters including spaces) together with the names and affiliations of the project members and the funding sources of the project to jan.wacker@uni-hamburg.de. A template for submission of new projects can be found here. If your project is accepted by CoScience.net, you will have the opportunity to invite project members, use the project’s closed discussion forum for communication with your group members and for the review of data requests and novel hypotheses by group members. You may also publish project details, specific ground rules, and requirements for project membership/data requests.

    Open peer-review

    Both the registration of additional objectives/hypotheses by project members and data requests by other researchers are discussed among the members of the respective project on CoScience.net. Specifically, all project members receive a review invitation and are expected to comment on hypothesis, theoretical/empirical rationale and analysis plan in an open peer-review within 21 days. Reminders are sent once after 14 days. Project members vote for either “reject“ (providing a justification), “revise“ (providing guidance for revision in the review forum), or “accept”. Authors of requests may submit up to two revised versions, after which a final decision is made. Requests “accepted“ by the majority of the current group members are fulfilled. In this instance, the project maintainer adds the final accepted version of the new hypotheses to the project description on CoScience.net and also preregisters it on the OSF.

    Suggestions of additional hypotheses/data requests should not exceed 1000 words. They should provide a sound theoretical or empirical basis and a precise statement of the hypothesis. Data requests should also include a detailed analysis plan, sketching the whole path from the raw data to the final statistical test of interest. Open peer-review will focus on whether (1) the new hypothesis is sufficiently distinct from pre-existing hypotheses/objectives, (2) the new hypothesis is sufficiently supported by the literature cited and, in the case of data requests, (3) the analysis plan is appropriate and how it can be improved. Further, researchers requesting data are required to confirm in advance that they have the necessary expertise to (1) devise a complete analysis script from the raw data to the final statistical test, and (2) take the lead in writing a manuscript for an international peer-reviewed journal.