Locomotive

Digital-First Design

Play hard Back to work

Play hard

Back to all

Image analysis and machine learning with Google Vision API

Technology and research are at the core of Pilot®'s projects developed internally at Locomotive. In spring 2019, we started discussions on a new concept to develop to succeed Moment Zero, Pilot®'s first experimental project using open data and augmented reality.

Proof of concept

This time, we organize ourselves differently. We first set up our concept base and a first team is selected to define the scope of the first features to be tested. As Pilot®'s goal is to learn and push our technical limits, we start with a prototype (MVP) of each feature, then work on user-flow, user experience, script and artistic direction.

Image analysis and machine learning

The first feature we wanted to test was about image analysis. The idea of a final product will be to ask the user to take a picture of a given object (for example: a cup) and to validate the object in question.

The first prototype is simple. It is a basic interface asking the user to take a photo of an object. The result returns all the keywords that the Image Analysis API identified by percentage of probability.

We first tested a solution with Tensorflow.js. The result was satisfactory, but the Google Vision API online test gave us more information. This first method is however more practical for detecting objects in real time, with a webcam for example.

So we decided to start testing the Google Vision API and use Node.js as a server to communicate.

Initialization of Google Vision API

First of all, you need to set up your Google Cloud Platform account and follow the first 4 steps here: https://cloud.google.com/vision/docs/quickstart-client-libraries

For step 5, you can recover the json file and put it in the root of your project. In your terminal, you will have to export the environment variable.

            export GOOGLE_APPLICATION_CREDENTIALS=”url/de/votre/fichier/json”
        

Installation of the Nodejs server with express and socket.io

To facilitate the management of the node server, we use nodemon. You will also need to install expressjs and socketIO to create communication between the client side (the application) and the node server which will communicate with the Google Vision API.

            npm install nodemon -g
npm install express --save
npm install socket.io --save
        

Index.js server

At the root of the project, you can create an index.js file which will be the file executed for launching the nodejs server.

            // Express server setup
const express = require('express')
var app = express();
var http = require('http').createServer(app);
var io = require('socket.io')(http);

app.get('/', function(req, res){
    res.sendFile(__dirname + '/index.html');
});

http.listen(2222, function(){
    console.log('listening on *:2222');
});

io.on('connection', function (socket) {
    console.log(‘User connected’);
});

        

Server launch

We can now launch the server in the terminal via localhost: 2222 and the console should confirm that a user is connected.

            nodemon index.js
        

Client-side communication

For the client side, we use locomotive-boilerplate.

In the index.html file, we can add our markup and link it to our module. We can then create our Detection.js file in assets / scripts / modules / then add it to the list of modules in modules.js

             <div data-module-detection class="c-detection">
<input type="file" data-detection="input">
</div>

        
            export {default as Detection} from './modules/Detection';
        

File management

For file and field management, we will use filepond. This flexible library allows you to manage temporary uploads, as well as uploading several files in succession. It offers useful plugins to crop / resize an image, to correct the orientation when taking a photo from a smartphone, or to validate and manage the different types and sizes of files before uploading to the server. Filepond will also be useful in the future projection of the project.

For plugins we will need the preview plugin and the orientation plugin.

On the client side, you will need to install the following packages, then initialize filepond in our Detection.js file

            npm install filepond  --save
npm install filepond-plugin-image-preview --save
npm install filepond-plugin-image-exif-orientation --save

        
            import { module } from 'modujs';
import * as FilePond from 'filepond';
import FilePondPluginImagePreview from 'filepond-plugin-image-preview';
import FilePondPluginImageExifOrientation from 'filepond-plugin-image-exif-orientation';

export default class extends module {
    constructor(m) {
        super(m);
    }

    init() {
        this.$input = this.$('input')[0];

        this.socket = io();
        
        // Add plugins to Filepond
        FilePond.registerPlugin(FilePondPluginImagePreview);
        FilePond.registerPlugin(FilePondPluginImageExifOrientation);
	  
        const pond = FilePond.create(this.$input,{
            multiple: false,
            name: 'filepond',
            allowImageExifOrientation: true,
            labelIdle: 'Take a picture'
        });
    }
}

        

Socket.IO

Now let's add Socket.IO on the client side. It is possible to place the source file in our assets/scripts/vendors folder or install it with a package manager.

At the end of our init() function we will create our socket.

            this.socket = io();
        

Image recovery and upload to Google Vision API

On the server side we can now install the Google Vision API package.

Then, in the server file index.js we can create a fetch function for the call to Google Vision API which will take as parameter the url of an image. Then simply follow the information in the documentation.

We will also create an image change listener for socket.IO, which will retrieve the url of an image and send it. At the end of this fetch function, we will send to our module an array of labels that detection will have returned to us thanks to a .emit() on the socket.

            npm install --save @google-cloud/vision
        
            io.on('connection', function(socket){
    async function fetch(imageUrl) {

        // Imports the Google Cloud client library
        const vision = require('@google-cloud/vision');

        // Creates a client
        const client = new vision.ImageAnnotatorClient();

        // Performs label detection on the image file
        const [result] = await client.labelDetection({
            "image" : {
                "content" : imageUrl
            }
        });
        const labels = result.labelAnnotations;

        // Send all labels to my Detection module
        socket.emit('updatePredictions', labels);
    }

    socket.on('updateImage', function(imageUrl){
        fetch(imageUrl).catch(console.error);
    });

});

        

Note: Sending a base64 to the Google Vision API works, but you will need to remove the prefix and send it as an "image" object as above.

Retrieving the list of labels on the client side

Back in the Detection.js module, it is now time to manage the sending of the url of our image to our node server, and the reception of the list of labels.

At the end of our init() function we will therefore have: Listening to the addition of a file to filepond with a FileReader to have a base64 of our image, which will be our url sent to our final server.

            this.reader = new FileReader();

pond.on('addfile',(error, file) => {
     // get a base64 and remove the prefix
     this.reader.onload = (e) => {
          this.socket.emit('updateImage', e.target.result.split(',')[1]);
     }
     //read file to get a base64
     this.reader.readAsDataURL(file.source);
});

       
this.socket.on('updatePredictions',(labels) => {
	// Enjoy
	console.log(labels)
})

        

Conclusion

Here we are, the prototype is working and our first capsule for our second Pilot® project is ready. Further down the road, we will think about managing the Google Vision API with PHP thanks to our Charcoal CMS, but for now, we are taking the next step with the assurance that Google Vision will be able to meet our needs for this project.

Back to all
Image analysis and machine learning with Google Vision API Locomotive storms the IGA stadium and wins honors in the Idéa competition The Revolution Of The Workspace As We Know It Can I say bravo? Locomotive crowned Agency of the Year for a second year Augmented reality on web browsers, wut? The Evolution of Scrolling at Locomotive Relive Locomotive's Parté of the Year Our monthly Loco breakfast Pet therapy at the office Locomotive is crowned Agency of the Year Snow day and cross-country skiing Locomotive in flight mode How to build and maintain the best team in town? An office full of seasonal colours Summer friday's under the hot sun A dream team for Locomotive
About Locomotive About Locomotive
C H O O C H O O