Dahlquist:TRACE Documentation

Home        Research        Protocols        Notebook        People        Publications        Courses        Contact       

The purpose of this wiki page is to develop a TRACE document for GRNmap. We will first develop our documentation on this wiki page and then create the Word/PDF documents.

Preamble

This is a TRACE document (“TRAnsparent and Comprehensive model Evaludation”) which provides supporting evidence that our model presented in:

GRNmap: Gene Regulatory Network Modeling and Parameter Estimation

was thoughtfully designed, correctly implemented, thoroughly tested, well understood, and appropriately used for its intended purpose.

The rationale of this document follows:

Schmolke, A., Thorbek, P., DeAngelis, D. L., & Grimm, V. (2010). Ecological models supporting environmental decision making: a strategy for the future. Trends in ecology & evolution, 25(8), 479-486.

and uses the updated standard terminology and document structure in:

Grimm, V., Augusiak, J., Focks, A., Frank, B. M., Gabsi, F., Johnston, A. S., ... & Thorbek, P. (2014). Towards better modelling and decision support: documenting model development, testing, and analysis using TRACE. Ecological Modelling, 280, 129-139.

• TRACE template Word document
• TRACE document example 1
• TRACE document example 2
• TRACE document example 3

and

Augusiak, J., Van den Brink, P. J., & Grimm, V. (2014). Merging validation and evaluation of ecological models to ‘evaludation’: a review of terminology and a practical approach. Ecological Modelling, 280, 117-128.

1. Problem Formulation

This TRACE element provides supporting information on: The decision-making context in which the model will be used; the types of model clients or stakeholders addressed; a precise specification of the question(s) that should be answered with the model, including a specification of necessary model outputs; and a statement of the domain of applicability of the model, including the extent of acceptable extrapolations.

Summary

• Decision-making context in which the model will be used:
• Types of model clients or stakeholders addressed:
• Precise specification of the questions that should be answered with the model
  • Specification of necessary model outputs
• Statement of the domain of applicability of the model
  • Extent of acceptable extrapolations

2. Model Description

This TRACE element provides supporting information on: The model. Provide a detailed written model description. For individual/agent-based and other simulation models, the ODD protocol is recommended as standard format. For complex submodels it should include concise explanations of the underlying rationale. Model users should learn what the model is, how it works, and what guided its design.

Summary

• What the model is:
• How it works:
• What guided its design:

3. Data Evaluation

This TRACE element provides supporting information on: The quality and sources of numerical and qualitative data used to parameterize the model, both directly and inversely via calibration, and of the observed patterns that were used to design the overall model structure. This critical evaluation will allow model users to assess the scope and the uncertainty of the data and knowledge on which the model is based.

Summary

• Source of numerical and qualitative data used to parameterize the model:
  • Direct sources of data
  • Parameters estimated inversely
• Quality of the numerical and qualitative data used to parameterize the model:
• Observed patterns that were used to design the overall model structure

4. Conceptual Model Evaluation

This TRACE element provides supporting information on: The simplifying assumptions underlying a model’s design, both with regard to empirical knowledge and general, basic principles. This critical evaluation allows model users to understand that model design was not ad hoc but based on carefully scrutinized considerations.

Summary

• Simplifying assumptions underlying the model's design
  • Empirical knowledge
  • General, basic principles

5. Implementation Verification

This TRACE element provides supporting information on: (1) whether the computer code implementing the model has been thoroughly tested for programming errors, (2) whether the implemented model performs as indicated by the model description, and (3) how the software has been designed and documented to provide necessary usability tools (interfaces, automation of experiments, etc.) and to facilitate future installation, modification, and maintenance.

Summary

• Documentation of how the computer code was tested for programming errors:
• Documentation of how the model performs as indicated by the model description:
• Software design and documentation
  • Usability tools (interfaces, automation of experiments)
  • Facilitate future installation, modification, mainenance

6. Model Output Verification

This TRACE element provides supporting information on: (1) how well model output matches observations and (2) how much calibration and effects of environmental drivers were involved in obtaining good fits of model output and data.

Summary

• How well model output matches observations
• How much calibration and effects of environmental drivers were involved in obtaining good fits of model output and data

7. Model Analysis

This TRACE element provides supporting information on: (1) how sensitive model output is to changes in model parameters (sensitivity analysis), and (2) how well the emergence of model output has been understood.

Summary:

<Provide here a concise summary of this element, e.g. brief text and/or a bullet point list. See guidelines given under TRACE element 1, “Problem formulation”.>

8. Model Output Corroboration

This TRACE element provides supporting information on: How model predictions compare to independent data and patterns that were not used, and preferably not even known, while the model was developed, parameterized, and verified. By documenting model output corroboration, model users learn about evidence which, in addition to model output verification, indicates that the model is structurally realistic so that its predictions can be trusted to some degree.

Summary:

<Provide here a concise summary of this element, e.g. brief text and/or a bullet point list. See guidelines given under TRACE element 1, “Problem formulation”.>

Links to Other Standards

Note that this is not part of the TRACE standard itself, but a place to put useful links to other standards.

• BioSharing.org
• Mathematical Modeling Ongoloty (MAMO)