back home next
Ex 4.3: Using a N-classes pixel classifier • Ex 4.5: Segmentation quality control

4. Tissue segmentation and classification¶

4.4 Semi-automatic segmentation using SAM¶

SAM (Segment Anything Model) is a deep-learning architecture created by Meta (Facebook, Instagram, ...). It allows, given a bounding-box or points as user input, to segment the object theoretically present in that box.

4.4.1 Glomerulus-to-kidney ratio¶

Goals:¶

We will try to achieve the same task in this exercise as in the Ex1: Fully manual tissue segmentation but we will use it doing deep-learning rather than manual annotations.

Required data:¶

Folder Description Location License
Histology (PAS-HE-IHC) H-DAB images of kidneys into which glomerulus are visible and lungs into which alveolus are present DOI: 10.1038/s41467-023-40291-0 MIT

A. Launch the server and SAM¶

  • Open a new conda terminal and activate the environment containing SAM's server with: conda activate samapi.
  • Launch the server with the command: uvicorn samapi.main:app --workers 2.
  • In QuPath, go to "Extensions" in the top-bar menu and launch SAM. It should open a new window.
  • The extension in QuPath will communicate with SAM server using the HTTP protocol.

Warning

From here, don't forget that SAM works only with what is visible in your viewer, at your viewer's resolution. For the exact same input annotation, the result will be different if you move the viewer a little bit or zoom in/out.

B. Segment the kidney¶

  • SAM can work only with QP rect rectangle and QP points point annotations. In this exercice, we will only work with rectangles.
  • Make a rectangle around the whole kidney biopsy. The edges of the rectangle should touch the contours of the biopsy.
  • Give the Kidney class to this new rectangle.
  • Zoom out so the biopsy fits completely in the viewer (like ×0.3 display scale).
  • In SAM's window, start by choosing a model. Don't go with the "large" or "huge" models, they require pretty big PC configurations to run. You can try either "vit_b (base)" or "sam2_bp (base plus)".
  • In the output type, set the dropdown menu to "Multi-mask (best quality)". SAM produces several segmentation candidates and this option allows to set the choise criterion to keep only one. If you prefer to see all the candidates and choose manually, you can set the option to "Multi-mask (all)".
  • Uncheck all the checkboxes below the output type:
    • We don't want to assign random colors, we want to keep the color of the Kidney class.
    • We don't want to keep the input prompt (the rectangle).
    • We don't want to generate random names.
    • We don't want to display the name of new annotations.
  • With the rectangle selected, the biopsy centered on your screen and the viewer scale set correctly, you can now click on "Run for selected".
  • A little spinning wheel should appear in the upper right corner of SAM's window while the inference is running. If the wheel disappears and the rectangle is still here, it means that SAM didn't find anything.

sam spinning

  • You can make other attempts after:
    • Moving the viewer a little bit.
    • Moving the rectangle by a few pixels.
    • Scaling the rectangle.
  • Once you have a correct-looking result, lock the kidney annotation.

C. Segment the glomeruli¶

  • For each glomerulus, make a rectangle annotation englobing it with a few pixels of padding.

Tip

You can quickly place all your rectangles by using the keyboard shortcut Kenney-r. Whenever you make a new rectangle, QuPath sets you back to panning mode. Simply keep a finger on Kenney-r and the other on the left-click! This way, you can place rectangles and move afterward.

  • Once all the rectangles are made, select them all in the annotations list and give them Glomerulus class.
  • Now, iteratively for each glomerulus, you will:
    • Make its corresponding rectangle active.
    • Place it at the center of your screen with a scale allowing you to see it all with some context around (like ×1.0 display scale).
    • Click on "Run for selected".
  • If several glomeruli are close from each other, you can make their annotations active at the same time, and try to segment them both at once.

Note

This method requires constant user feedback, it is not something that we can batch over the project.