Scalismo demo Applications - a Statistical Shape Modelling Library in Scala

Scalismo official website: scalismo.org/
Scalismo tutorials: scalismo.org/docs/#tutorials
Setting up scalismo with the IntelliJ IDE: scalismo.org/docs/ide
Scalismo library source code: github.com/unibas-gravis/scal...
Timeline:
0:06 Introduction
0:55 Scalismo website
2:32 Hello world project in Scalismo
5:12 Scalismo Demos
5:59 Non-rigid surface registration
6:33 Non-rigid image registration/segmentation
7:52 Partial femur completion
8:05 Basel Face Model (Statistical Face Model) sampling
9:37 2D to 3D image reconstruction
10:38 Face manipulation website
11:41 Game character movement/deformations
12:25 Checking out
The femur demo that was shown is described in this paper: “Probabilistic fitting of active shape models" Morel-Forster, Andreas and Gerig, Thomas and Lüthi, Marcel and Vetter, Thomas
Non-rigid registration: github.com/unibas-gravis/icp-...
Face model viewer: github.com/unibas-gravis/base...
Parametric face model generator: github.com/unibas-gravis/para...
Face pipeline (2D to 3D): github.com/unibas-gravis/base...
Face manipulation service: face-morpher.scalismo.org/
Statistical Shape Modelling course: gravis.dmi.unibas.ch/PMM/lect...
Scalismo mailing list to post Scalismo related questions: groups.google.com/g/scalismo

[@AlexeyMatushevsky]

Very interesting video. thank you for link to the course. I had idea that looks a like the program on video - create the model that will recreate the shape in 3d based on the picture of sunglasses. it looks like the same idea as - recreating the 3d face based on image

[@DennisMadsen]

Your welcome. Hope you like the course. For the sunglass idea you can indeed get the 3D face representation from a 2D photo and then place 3D glasses on the face before projecting the 3D scene back into the photo. Not sure if this is the application you are thinking about?

[@AlexeyMatushevsky]

@@DennisMadsen Hi Dennis! Yes, exactly! not sure how to start that project for now. any hint, please?

[@DennisMadsen]

Hi @@AlexeyMatushevsky. I would suggest using a publicly available face model, like the Basel Face Model (BFM) faces.dmi.unibas.ch/bfm/bfm2019.html . On github you'll find different projects that use this model to fit to an image by regressing the parameters. Alternative to optimization, then I found a probabilistic fitting tutorial online: gravis.dmi.unibas.ch/PMM/lectures/fitting/ , this approach is usually a low slower, but does not require any additional training. There is also an updated tutorial that takes occlusion into account: gravis.dmi.unibas.ch/PMM/lectures/segmentation/ . After you get the 3D representation (shape, and pose) of the 2D image, you place the glasses in the 3D scene, before finally rendering the glasses on the original image.

[@michaellennon7805]

Thank you for this excellent demo Dennis. I was able to run the hello world project and open the UI with the simple face mesh. I have completed the FutureLearn course also and am used to working with the UI used in those examples. But I note that the demo fitting the femur SSM to the target femur has a different UI. Can you advise if there is a tutorial for using this different UI as I would like to try fitting an SSM I have built using humerus to a target mesh?

[@DennisMadsen]

Hi Michael. Are you using scalismo-lab? Or do you use scalismo with an IDE like intelliJ? Many tutorials can be found on the scalismo website here: scalismo.org/docs/ I'll have more videos coming on SSM fitting in the future :) The UI that was used in the video is a basic scalismo-ui but with a few manually added buttons. Also planning a video on this to show how scalismo-ui can easily be extended for custom applications.

[@michaellennon7805]

Hi Dennis, Thank you for your response. I have progressed from scalismo-lab to now using IntelliJ IDE. I am working through the tutorials on scalismo website using IDE. I am an engineer so limited experience as a programmer but am comfortable with scala language now. However I expect it will take some time to be familiar with all of the methods in Scalismo. In the meantime the custom UI with the manually added button looks like a really cool way to automate the fitting of an SSM to a target without too much coding. I look forward to your video on extending the UI. It would be an excellent tool to demonstrate the power of SSM using scalismo to my PhD supervisors. Let me know if you think this conversation should be added to the scalismo conversation on google.

[@DennisMadsen]

Hi Michael. We can keep the conversion here as long as it is just ideas for future learning content :) If you have more specific implementation problems or are facing bugs with scalismo, then writing to the mailing list would make more sense.

[@stevena7879]

How did you go about establishing the tool for landmarking across the curve of the femur for fitting the SSM to it? Is that MCMC or did you define some other metric?

[@DennisMadsen]

Hi Steven. The fitting is indeed done using MCMC as you mention. I see that another person already had a question regarding the UI on the mailing list: groups.google.com/g/scalismo/c/FeDigbnpcwU Marcel also created a gist to show how the UI can be customized.