CoScience.net – a cooperative science forum for cooperative forking path analysis

CoScience.net is a forum designed to facilitate cooperative research projects by offering a means through which researchers can, with minimal administrative burden: share projects, recruit additional participating laboratories, and expand the use of collected data. Whereas CoScience.net was developed initially to facilitate a specific research project funded by the German Research Foundation (DFG), this platform supports additional research projects that adhere with the cooperative principles upheld by this network. Distinct from a general crowdsourcing infrastructure for various research plans (see Psychological Science Accelerator), CoScience.net brings together smaller groups of researchers to join forces for specific projects.

This is how it works: Grounded in the principles of cooperative forking path analysis (cFPA, Wacker 2018) groups of five or more researchers from different labs share the load of data collection for testing a set of registered hypotheses using an agreed-upon design. Further, the collaborative group of researchers discuss and systematically explore analysis options for the data to minimize undisclosed flexibility (researcher degrees of freedom, garden of forking paths) that leads to an increased likelihood of false-positive results (Simmons, Nelson & Simonsohn, 2011).

It is our hope that, by reducing undisclosed researcher flexibility in data analysis, allowing experts access to a large a rich and carefully collected dataset and therefore promoting increased statistical power, and facilitating open expert discussions of hypotheses and analysis options, CoScience.net and the principles of cFPA will be instrumental in improving (psychological) science.

Join CoScience.net - Request Data

Joining CoScience.net is free for researchers who would like to contribute to projects for which they are experts or who want to access our data. Please note that members of CoScience.net are required to adhere to our Code of Conduct and accept our Terms and Conditions.

Data requesters’ benefits

  • Following completion of data collection within a projecrequt, members of CoScience have the opportunity to gain access to the project’s data at no cost, subject to positive open peer-review of their additional objectives/hypotheses and analysis ideas by the members of the project.
  • req
  • Data requesters will generally serve as first authors of publications based on their data requests if they:
    • provide a complete analysis script based on Matlab and R code that works on a permutated random subsample of raw data (with each variable shuffled independently) within six months of gaining access to this random subsample of data,
    • submit an initial draft for a manuscript for review by all group members within six months after receiving the project results based on their analysis script from the project maintainer, and
    • take the lead for further optimization of the manuscript based on their data request during both the internal CoScience review process and the review process after (re)submission to an international peer-reviewed journal.

    Requirements for Data Request

    These vary between projects. For detailed information please click on the corresponding project below, here below a few general criteria are listed:

  • Researchers requesting project data are required to hold a PhD in Psychology, Neuroscience, or a related field - PhD students are encourage to request data as well, but need to apply together with someone who meets this criteria.
  • Requests need to come in written form thorugh this forum, where planned analysis and relevant data parts are listed
  • Requests are approved through CoScience members when analysis are scientifically sound and reasonable , and do not overlap with existing analysis plans.
  • Researchers agree to preregister their analysis and maker their analysis (scripts) available , and to publish the obtained results at least as a pre-print
  • Importantly, if you only desire data access to varify data analysis of an existing CoScience publication, please directly contact the corresponding authors, who will grant you access.

Browse existing projects

Project 1: A novel collaborative approach for EEG personality neuroscience research

Project maintainer: Jan Wacker, Universität Hamburg, Germany

 

Abstract

Linking individual differences in the electroencephalogram (EEG) to variation in personality can provide important insights into the biological basis of human individuality and constitutes a prominent approach within the young and burgeoning field of Personality Neuroscience. Frequently investigated EEG measures include frontal EEG asymmetry, error and feedback-related negativities/frontal-midline theta, and the late positivities. Despite decades of research, the links between these measures and individual differences in personality remain poorly understood due to concerns of low replicability that currently thwart many areas of Psychology and Neuroscience. It has now become clear that low statistical power and undisclosed flexibility in data analysis are at the core of the current replicability crisis.

The present project aims to alleviate this unsatisfactory status quo and provide a firm basis for future EEG personality neuroscience work through initial application of a novel approach for empirical research termed “collaborative forking path analysis” (cFPA). Briefly, cFPA is based on (1) close cooperation among teams of researchers sharing the load of data collection for an a priori agreed-upon, highly standardized experimental setup in order to achieve high statistical power and (2) conducting a systematic scripted analysis (and comparison) of all defensible data analysis paths identified in joint expert meetings. The present application of the cFPA approach entails the collection of data from several paradigms allowing quantification of each of the EEG measures noted above from N = 720 participants distributed across nine laboratories. This unprecedented collaborative effort within the format of a regular individual grant will allow us to (1) probe the replicability of prominent EEG-personality associations, (2) provide initial direct tests of potential moderators for several such associations, (3) derive initial estimates for the influence of inter-laboratory variability (e.g., in technical setups and laboratory personnel) on each of the EEG measures and their associations with other variables, (4) provide a systematic investigation of the influence of analysis choices on each of the effects of interest, (5) validate several novel methods for quantifying individual differences in EEG activity, and (6) provide an easily accessible high-quality multivariate data set for further theory-based as well as more inductive EEG personality neuroscience work by researchers around the globe. Most importantly, if demonstrated to be feasible in this initial application to a methodologically relatively demanding field, the cFPA approach may serve as a blueprint for significantly changing the practice of empirical research in Personality Neuroscience and various other areas towards increased statistical power and transparency.

Project members

Jan Wacker, Universität Hamburg, Germany (project maintainer)

André Beauducel, Universität Bonn, Germany

Jürgen Hennig, Justus-Liebig-Universität Giessen, Germany

Johannes Hewig, Julius-Maximilians-Universität Würzburg, Germany

Andrea Hildebrandt, Carl von Ossietzky Universität Oldenburg, Germany

Erik Mueller, Philipps-Universität Marburg, Germany

Roman Osinsky, Osnabrück University, Germany

Anja Riesel, Universität Hamburg

Jutta Stahl, Universität zu Köln, Germany

Alexander Strobel, Technische Universität Dresden, Germany

Requirements and rules for data requests

Researchers requesting project data are required to

  • hold a PhD in Psychology, Neuroscience, or a related field.
  • have published at least three international peer-reviewed journal articles broadly relevant to the project.
  • provide a data request in written form (see below).
  • Data request can be prepared with the template attached. All requests are accepted if
  • the planned analysis are scientifically sound and reasonable.
  • majority of voting CoScience members accept the request.
  • content does not overlap with already preregistered analysis (embargo of 12 months).
  • As agreed in the code of conduct, after successful data request, the researchers commit to
  • preregister their analysis on a platform of their choosing and to make their analysis (scripts) available.
  • provide functioning analysis scripts within 6 months after acceptance of the data request.
  • finalize a manuscript 6 months after results were provided.
  • publish the obtained results at least as a preprint (to avoid file-drawer problems) or any appropriate peer-reviewed journal.
  • A detailed description the dataset can be found here. To request data, please send an email to the project maintainer (Jan Wacker: jan.wacker@uni-hamburg.de) with a brief (3-6 sentences) description of the data and analysis you are interested in. This description is used to check that there is no substantial overlap across data requests. Please confirm that you fulfill all data request requirements and agree with all data rules listed on this page.The project maintainer will then provide you with further details concerning the registration in this forum and will invite you to submit your data request on the project’s CoScience forum for interactive open peer-review by the project members. Thereafter, you will be granted access to the required data. Please note that members of CoScience.net (and all data-requesters) are required to adhere to our Code of Conduct and accept our Terms and Conditions. [If you want to gain access to the data to validate a publication of the CoScience group, please contact the corresponding authors directly.]

    Open peer-review

    Both the registration of additional objectives/hypotheses by project members and data requests by other researchers are discussed among the members of the respective project on CoScience.net. Specifically, all project members receive a review invitation and are expected to comment on hypothesis, theoretical/empirical rationale and analysis plan in an open peer-review within 21 days. Reminders are sent once after 14 days. Project members vote for either “reject“ (providing a justification), “revise“ (providing guidance for revision in the review forum), or “accept”. Authors of requests may submit up to two revised versions, after which a final decision is made. Requests “accepted“ by the majority of the current group members are fulfilled. In this instance, the project maintainer adds the final accepted version of the new hypotheses to the project description on CoScience.net and also preregisters it on the OSF.

    Suggestions of additional hypotheses/data requests should not exceed 1000 words. They should provide a sound theoretical or empirical basis and a precise statement of the hypothesis. Data requests should also include a detailed analysis plan, sketching the whole path from the raw data to the final statistical test of interest. Open peer-review will focus on whether (1) the new hypothesis is sufficiently distinct from pre-existing hypotheses/objectives, (2) the new hypothesis is sufficiently supported by the literature cited and, in the case of data requests, (3) the analysis plan is appropriate and how it can be improved. Further, researchers requesting data are required to confirm in advance that they have the necessary expertise to (1) devise a complete analysis script from the raw data to the final statistical test, and (2) take the lead in writing a manuscript for an international peer-reviewed journal.

    Post a new project

    Unless further funding is acquired, CoScience.net can only handle a limited number of projects. Therefore, in the spirit of the project for which CoScience.net was originally developed, support is currently restricted to projects within Psychology that aim to employ cFPA. Initial project teams should include members from at least five different labs, each of whom is an expert in the project’s field of study (typically documented by at least three relevant international peer reviewed articles). In the absence of strong arguments for larger effect sizes, planned projects should at least secure high statistical power (> .95) to detect the effect size of rho = .20 typical for replication studies within psychology (e.g., Open Science Collaboration, 2015) with an alpha of .05. Furthermore, project teams are required to confirm, that they (1) have already acquired the necessary funding for completion of the project, (2) have the necessary analysis/scripting expertise to apply cFPA, and (3) will take on the necessary managing and reviewing duties within the project to guarantee appropriate handling of data requests.

    If you would like to post a project conforming to these requirements, please send an informative abstract of the planned project (no more than 3000 characters including spaces) together with the names and affiliations of the project members and the funding sources of the project to jan.wacker@uni-hamburg.de. A template for submission of new projects can be found here. If your project is accepted by CoScience.net, you will have the opportunity to invite project members, use the project’s closed discussion forum for communication with your group members and for the review of data requests and novel hypotheses by group members. You may also publish project details, specific ground rules, and requirements for project membership/data requests.