CCP logo



Consensus Annotation

This page was posted with the release of version 1.6 and pertains only to this and subsequent versions.

This page steps through an example of how to use the 'consensus mode' in Knowtator. The example is distributed with Knowtator and is located at:


We will refer to this directory as <example-dir>.

The example consists of a simplified version of an annotation task in which molecular transport events were annotated. Molecular transport is a process in which proteins and other macro-molecules are moved (transported) from one part of a cell to another. The annotation schema in this example provides a very simple model of molecular transport in which a class called 'transport' has slots 'origin' and 'destination' which are constrained to be of type 'cellular component'. The transport class also has slots 'transported entities' and 'transporters' which are constrained to be of type 'molecule'. The annotated examples are not necessarily biologically accurate (or even close.) They are for example purposes only.

This example picks up in the middle of an example annotation effort. The annotation schema has already been defined, the text sources selected, and assignements given to Jane and John to annotate the text sources individually. They have annotated the same 10 sentences individually and now we wish to create a consensus set of annotations in which Jane and John's annotations are merged and their differences resolved.

The <example-dir> folder has the following directories:

Before we start creating the consensus set, please open the three annotation projects in turn and observe the following:

Merge Jane and John's Annotations

The first step is to create a new annotation project from the reference project and import the annotations from Jane and John's annotation projects. Do the following:
  1. open <example-dir>/reference-project/molecular-transport.pprj
  2. Menu->File->Save Project As... (see note on using this menu option)
  3. Save as <example-dir>/consensus-annotations/consensus-annotations.pprj
  4. Menu->Knowtator->Merge annotations
  5. select <example-dir>/janes-annotations/janes-annotations.pprj
  6. select the filter labeled "Jane's annotations"
  7. select all 10 instances of "file line text source"
  8. Menu->Knowtator->Merge annotations
  9. select <example-dir>/johns-annotations/johns-annotations.pprj
  10. select the filter labeled "John's annotations"
  11. select all 10 instances of "file line text source"
  12. save, close, reopen <example-dir>/consensus-annotations/consensus-annotations.pprj. (see bug report)
Now Jane and John's annotations are in the same project sitting side by side.

Create a filter that defines the consensus set

It is possible that you could merge other annotations into this project, but we want to create a consensus set on just these annotations. To do this we need to create a filter that defines the annotations to be added to the consensus set. Do the following:
  1. Create an instance of "knowtator filter" by following the detailed instructions here.
  2. Name the filter "Jane and John's individually annotated annotations"
  3. The 'annotators' of the filter should be set to 'Jane' and 'John'
  4. The 'sets' of the filter should be set to 'Individually Annotated'
If you are following the instructions carefully, then this filter defines all of the annotations that are currently in the consensus-annotations project. However, the consensus set creation step we are about to perform will create annotations that do not satisfy this filter (by design).

NOTE: Creating and using an annotation set is important for consensus mode. For consensus in your own annotation project, you should add all of your individually annotated annotations to a set that is different than the consensus set (that is about to be created below). This can be done in batch mode as explained here.

Create a consensus set

To create the consensus set do the following:
  1. Select Menu->Knowtator->consensus mode
  2. A wizard labeled 'consensus set creation wizard' should appear if no consensus set has yet been created for this project. Click 'next'.
  3. Enter 'consensus set' and click 'next'.
  4. Click 'select filter' and select the filter labeled Jane and John's individually annotated annotations. Click 'next'.
  5. Click 'add' and select all 10 text sources. Click 'create'.
The following is a listing of all the annotations that are now in the project:

Consensus Mode UI

If you have successfully created a consensus set, then Knowtator should look like the following (partial) screenshot.

Here are a few things to observe about this image:

Working in Consensus Mode

Now Jane and John can sit together and resolve their differences (or an adjudicator can do the same). A text source is considered 'done' when all of the annotations in the consensus set have been assigned the annotator 'consensus set annotator team'. An annotation can be assigned this annotator in one of four ways: An unwanted annotation may also be deleted in which case there is one less annotation that has not yet been assigned the 'consensus set annotator team' annotator.

Because the original individually annotations are preserved, it is easy to restart consensus work for a text source by clicking the 'restart' button. This will simply recreate the original consensus set annotations for that text source.

Default annotator and annotation set

When the consensus set is first created, Knowtator's configuration is updated such that the default annotator is the consensus team annotator and the default annotation set is the consensus set. However, after that Knowtator makes no attempt to keep the configuration synchronized with the filter that is currently selected. For example, if you have the 'consensus set filter' selected and you change the filter to be "Jane's annotations", the default annotator and default annotation set will not be changed. This may not be desirable if Jane is going to spend time editing her annotations. Make sure that when you change filters that the configuration settings for the default annotator and default annotation set are what you expect them to be. To learn more about configuring Knowtator, go to here.


The following lists things to observe for each text source in our working example: