Computer vision based methods in fish farming are under active development, as automatic tracking of the animals using cameras would allow for cost-efficient monitoring and precise collection of data on fish behaviour. Methods that do not rely on costly manual annotation work are crucial for them to be applicable in real-world systems. We tried to solve this problem by leveraging the recent advances in instance segmentation foundation models, namely the state-of-the art benchmark performance reaching Segment Anything model (SAM). In instance segmentation, individual objects are identified in the image, resulting in segmentation masks that cover the pixels belonging to the corresponding objects.
To develop and test our methods on real-life data, video data was recorded on two commercial RAS farms, one farm cultivating Rainbow trout and one focusing on Atlantic salmon, using four RGB cameras positioned over the fish tanks. For the dataset, 2654 images were extracted at different times over several days’ timespan to ensure a varied dataset, and the fish individuals were manually annotated in the images using bounding boxes.
We studied how we could automatically segment fish from the images without any annotations using SAM and compared the performance to using hand-annotated prompts. We find that with carefully chosen parameters the automatic segmentation can find a considerable number of good quality individual fish masks, while typically also producing more bad quality masks than the hand-annotated input. To tackle this, we propose a classifier-based post-processing method for filtering out bad quality masks using properties of the mask contours. Using the method together with a Kalman filter based tracking-by-detection algorithm, we can track the visible fish individuals to get estimates on their swimming speed and direction. We aim to show the connection of the swimming behaviour to fish health, feeding and tank environmental conditions, combining known production data to several months of video observation data.