Review scientific code

Software is created using formal languages known as programming languages. Text files written in programming languages are called source files or source code files. The instructions expressed in programming languages are then translated into a form that can be understood and executed by computer hardware. 
 
Some programming languages, such as C or Fortran, need to be translated in this way before the program can be run. These are known as compiled languages. Other programming languages, such as R and Python, are known as interpreted languages and are translated as the program runs. Program code that needs to be interpreted by a software interpreter in this way is often referred to as a script.  

Scientific software consists of digital objects that, like other data, should be assigned persistent identifiers and described thoroughly with appropriate metadata when published, so that they are findable, accessible, interoperable, and reusable (FAIR). 

Software prepared for publication may serve as documentation of workflows and analytical methods and, in some cases, may also form part of the research results. It is common to publish code together with other data and documentation in the same data publication. Making scientific software openly available can be an effective way of making workflows transparent and reproducible. 

Software can be deposited as a frozen version in a data repository such as SND CARE. However, a repository specifically designed for code, such as GitHub, is often preferable for software, as the code can be maintained and developed further there. If related software, data, and documentation are published in different locations, the creator should ensure that they refer and link to one another in a persistent way.  

Note: If software is submitted as documentation of, or to enable the reproduction of, results that have already been produced, reviewers should be cautious about asking the creator to change the names of variables, functions in the source code, etc., as this may affect the results. How software should be reviewed and described for publication varies between programming languages, so the first step in the review process is to identify the language used. 

When research data are published, exact versions are established for data files and documentation files, and the exact version of any associated code should also be specified. Code versioning is often managed using a code repository such as GitLab or GitHub. 

• Related code and data can be published together in a research data repository. 
• It is possible to include a copy of the source code in a data publication while also providing a relational link to the code repository or software publication. This ensures that the version of the software associated with the dataset remains available and preserved, but it also allows reusers to follow any subsequent developments in the software after the data were published.
• There are several options for packaging and publishing a version of the relevant code for long-term preservation so that it receives its own persistent identifier (PID). Software Heritage assigns an SWHID, while the Zenodo integration with GitHub can be used to assign a DOI to a release in a GitHub repository. 

• Software can be protected by copyright, and it may therefore be appropriate to apply a licence to the code if the depositor wishes to do so. In some cases, copyright in computer programs may transfer to the employer, so look into what rules apply to publication and licensing at the creator’s organization. 
• Applying a licence to software increases the FAIRness of the publication by making it clear whether and how the software may be reused. Licences that meet open-source requirements are to be preferred. Open source software means that the creator of the software allows the code to be read, modified, and redistributed for different forms of reuse. 

In many cases, it is appropriate to describe and/or include files showing the expected output from the software. Such output examples can either be presented in the documentation or provided as separate files. They can be used in reproducibility testing to compare generated output and also provide a means of checking that the process behaves as intended.  

If the full set of output files is too extensive or unwieldy to include in the reproducibility package, a representative, limited output example can be provided instead. 

A README file can be used to describe: 

• The purpose and use of the code, and the expected output. 
• The files included and what they do. If there are several files, describe how they relate to one another and the order in which they should be run. 
• Information on how to install and run the software. 
• Contact information for the author (e.g., e-mail, ORCID). 
• Licence information. 
• Version history. 
• Acknowledgements and citations for code snippets, packages, etc., created by others and used in the software. 

Code files should contain a header in the form of a comment that provides basic information such as: 

• Title. 
• What the code does. 
• Functions and arguments that a user needs to know about. Does the software include a help command that can provide the user with more information? 
• Where relevant, how the file relates to other files in the publication. 

Code files often contain version and contact information that may duplicate information in a README file: 

• Version. 
• Contact information for the author (e.g., e-mail, ORCID). 
• Date. 
• Licence, where applicable. 

In addition to conventional source code files, there are also files in which source code and more detailed descriptions are stored together, following a programming paradigm known as literate programming. These files are generally structured to run code in a particular sequence and to display the results of each processing step directly. Files of this kind are known as computational notebooks. Examples include Jupyter Notebook (.ipynb), R Markdown (.Rmd), and Quarto (.qmd). 

• There needs to be information on how to run the code. 
• Make sure that the execution environment is described in sufficient detail for a reuser to recreate it without difficulty. This will usually require the exact versions of all components – including every dependency on individual packages or code libraries – to be specified. 
• Even when code is published elsewhere, the documentation should contain sufficient information to support reuse.  
• If source code is to be versioned together with data or documentation, it is a good idea to ask whether the software is already under version control somewhere, or whether it may even have been published independently elsewhere. 

• Check that filenames and variable names correspond to any information provided in the README file, article manuscript, or similar documentation. It is common for code files to be revised without remembering to update external references to the files. 
• File paths should be relative. This means that when the software reads data files, it is important that the paths do not point to files stored in unique locations on the depositor’s own computer. Instead, paths should be based on the package structure and point to files stored in the same folder as the published script, or in a specified folder within the same hierarchy, regardless of on which computer the folder is used. This is a common problem that should be addressed before publication. 
• Make sure that all data files and supporting files (image files, template files, etc.) required to run the software are included in the publication and have the same names as those specified in the script. If the names of data files have been changed during curation, the filenames must also be updated in the accompanying source files. 

• Checking that a package is complete is easier if it has a well-designed folder structure in which code, input and output data are kept in separate folders. Make sure that the correct folder structure is included with the code, even if this means including empty folders. 
• Check whether output examples are specified in the documentation or included as files. If not, the depositor can be encouraged to provide files showing what the software’s output is expected to look like. If the software requires large amounts of data or substantial computational resources, you can suggest including a limited but representative output example to support reproducibility. 

• Follow any instructions provided, for example in the README file. 
• Check whether there is a particular sequence in which different scripts are intended to be run and try to follow it. The sequence may be described in the documentation files or implemented in the form of a higher-level control script. 
• If you manage to run the software successfully, compare the results with any output examples provided. File comparison tools – either command-line tools such as fc in Windows or diff in macOS/Linux, or GUI-based tools such as Meld – can be used to determine whether the software has reproduced the example output exactly, or to identify differences in text-based output.  

If the software does not work when test-run, it may be necessary to carry out a more detailed review of the source files. 

• If you encounter errors relating to files that cannot be found, examine the code to see how the files are referenced. Scripts commonly expect data files to be located in a particular place within a predefined folder structure, with folders having specific names. If this is not documented, it may be possible to work it out by examining the code or the error messages you receive. Paths should be relative to the location from which you run the script, but in some cases the creator may have used an absolute path that is likely to work only on their own computer. This prevents reuse and should be corrected. 
• Another common reason why code cannot be run is that it depends on external packages or code libraries that are either missing from the test environment or are of a version that is incompatible with the code. When a required code library is missing from the test environment, the error messages are usually fairly self-explanatory. Check whether you have overlooked any important information or whether the dependency was omitted from the documentation or environment description.  
• When a required code library is present in the test environment, but the code does not work with the available version of the library, the error messages may be more difficult to interpret. If something looks unusual or ambiguous, ask the depositor which exact dependencies and versions were used. 

If you are still unable to run the software, provide feedback to the depositor about the problems encountered. Include any error messages and ask the depositor to run the code themselves to make sure that it works as intended. If the depositor cannot reproduce the errors, suggest that they test the code on another computer or ask a colleague to run it.