AI Sitefinity Image Processor

Sitefinity has a nice feature to auto generate thumbnails. Although it is working fine most of the time, there are some cases that it is not enough. For example when there are people on the image it does not look good when their heads and faces are cropped. To illustrate this I’ll use a photo of part of our team.

This is the original image:


Here is how it's cropped by the default Image Processor:

You see how the heads in the top most of the photo are cropped?

Of course we can just resize it without cropping. Then we’ll have thumbnail with size 533x300 instead of 600x300 and smaller faces, which we don’t want because this photo was taken few years ago during the Christmas Holidays and we want to focus on the festive smiles. :)

Question is: How to tackle this one without doing it manually? The answer is Computer Vision.

If you are not familiar with Computer Vision, Machine Learning and Facial Recognition, you can check some resources online:

Machine Learning

Quote from Wikipedia:

“Machine learning (ML) is the study of computer algorithms that improve automatically through experience.[1][2] It is seen as a subset of artificial intelligence. Machine learning algorithms build a mathematical model based on sample data, known as "training data", in order to make predictions or decisions without being explicitly programmed to do so.[3] Machine learning algorithms are used in a wide variety of applications, such as email filtering and computer vision, where it is difficult or infeasible to develop conventional algorithms to perform the needed tasks”

Source: https://en.wikipedia.org/wiki/Machine_learning

Computer Vision

Quote from Wikipedia:
“Computer vision is an interdisciplinary scientific field that deals with how computers can gain high-level understanding from digital images or videos. From the perspective of engineering, it seeks to understand and automate tasks that the human visual system can do.
Computer vision tasks include methods for acquiring, processing, analyzing and understanding digital images, and extraction of high-dimensional data from the real world in order to produce numerical or symbolic information, e.g. in the forms of decisions. Understanding in this context means the transformation of visual images (the input of the retina) into descriptions of the world that make sense to thought processes and can elicit appropriate action. This image understanding can be seen as the disentangling of symbolic information from image data using models constructed with the aid of geometry, physics, statistics, and learning theory.” 

Source: https://en.wikipedia.org/wiki/Computer_vision

Facial Recognition

Quote from Wikipedia:
“A facial recognition system is a technology capable of identifying or verifying a person from a digital image or a video frame from a video source. There are multiple methods in which facial recognition systems work, but in general, they work by comparing selected facial features from given image with faces within a database. It is also described as a Biometric Artificial Intelligence based application[citation needed] that can uniquely identify a person by analyzing patterns based on the person's facial textures and shape.”

Source: https://en.wikipedia.org/wiki/Facial_recognition_system

Setting up the project

We will use the Quantum project. You can get it from GitHub and there are instructions on how to set it up: https://github.com/Sitefinity/Telerik.Sitefinity.Samples.Quantum

For the computer vision part we will use the popular OpenCV which has a wrapper for .NET:
OpenCV - https://opencv.org/
OpenCvCsharp - https://github.com/shimat/opencvsharp

Install OpenCvCsharp nuget packages:
Install-Package OpenCvSharp4
Install-Package OpenCvSharp4.runtime.win

We will create a custom Image Processor for Sitefinty. For more details you can check the documentation:
https://www.progress.com/documentation/sitefinity-cms/for-developers-image-processing-api

Here is the code for our image processor:

protected void Application_Start(object sender, EventArgs e)
        {
            Bootstrapper.Initialized += Bootstrapper_Initialized;         
        }
        void Bootstrapper_Initialized(object sender, Telerik.Sitefinity.Data.ExecutedEventArgs e)
        {
            if (e.CommandName == "Bootstrapped")
            {
               ObjectFactory.Container.RegisterType<IImageProcessor, FaceRecognitionImageProcessor>(new ContainerControlledLifetimeManager());
            }
       }
    

OpenCV in action:

The resulting thumbnail:

Using our custom image processor

You ca see how all faces are visible and centered on the image. 
Bellow are thumbnails created by the built-in image processor with the same size 600x300, cropped and resized:

Note: Keep in mind that this is only an example how we can utilize Computer Vision for better image manipulation. You may need to fine tune the code or even to train your custom model to better suit your use case.