Statistical Shape Modeling Challenge - Rules

All participants obtain a set of training data as binary images, which they use to build a model of the liver. The participants are free in the choice of registration and modeling algorithm. In order to make an evaluation possible, the final model needs to be a parametric, Gaussian model (which comprises e.g. many linear models, such as.the standard PCA models). The evaluation of the model quality is performed once the model is uploaded on the virtual skeleton database.

The model will be evaluated according to the following criteria:

Evaluation

Specificity:

We will draw 1000 samples from each model and compare it to the nearest member of the training set. The average distance will be the score.

Goal: The lower the value for the specificity the better the ranking.

Compactness:

The number of parameters of the models

Goal: The lower the number of dimensions the better the ranking.

Generalization ability:

We will fit the models to a set of test images of segmented livers and compute the following measure to assess generalization ability:

Goal: The smaller the distance the better the ranking

Scoring:

We compute for each of our evaluation criteria:

a ranking of the participant according to their performance (e.g. if there are 10 participants, the best performing algorithm gets 1 point and the worst performing algorithm gets 10 point). The final ranking will be the avarage of the individual rankings.

Upload

Model format:

The model has to be provided in statismo format, which is described here: https://docs.google.com/document/d/1ySwSsD-di8ez5WnViLl4VsYeRI-ggVa-HQRxG4kLjH4/edit?usp=sharing

A first version of the validation procedure can be found on github: https://github.com/statismo/shape-challenge-2014

Model naming:

Once the system is ready to accept the statistical models we will post here the naming und uploading instructions