All participants obtain a set of training data as binary images, which they use to build a model of the liver. The participants are free in the choice of registration and modeling algorithm. In order to make an evaluation possible, the final model needs to be a parametric, Gaussian model (which comprises e.g. many linear models, such as.the standard PCA models). The evaluation of the model quality is performed once the model is uploaded on the virtual skeleton database.
The model will be evaluated according to the following criteria:
We will draw 1000 samples from each model and compare it to the nearest member of the training set. The average distance will be the score.
Goal: The lower the value for the specificity the better the ranking.
The number of parameters of the models
Goal: The lower the number of dimensions the better the ranking.
We will fit the models to a set of test images of segmented livers and compute the following measure to assess generalization ability:
Goal: The smaller the distance the better the ranking
We compute for each of our evaluation criteria:
a ranking of the participant according to their performance (e.g. if there are 10 participants, the best performing algorithm gets 1 point and the worst performing algorithm gets 10 point). The final ranking will be the avarage of the individual rankings.
The model has to be provided in statismo format, which is described here: https://docs.google.com/document/d/1ySwSsD-di8ez5WnViLl4VsYeRI-ggVa-HQRxG4kLjH4/edit?usp=sharing
A first version of the validation procedure can be found on github: https://github.com/statismo/shape-challenge-2014
Once the system is ready to accept the statistical models we will post here the naming und uploading instructions