The Object Detector detects 80 different common objects: bicycles, cars, motorcycles, airplanes, buses, traisn, trucks, boats, traffic lights, fire hydrants, stop signs, parking meters, benches, birds, cats, dogs, horses, sheep, cattle, elephants, bears, zebras, giraffes, backpacks, umbrellas, handbags, ties, suitcases, frisbees, skis, snowboards, sports balls, kites, baseball bats, baseball gloves, skateboards, surfboards, tennis rackets, bottles, wine glasses, cups, forks, knives, spoons, bowls, bananas, apples, sandwiches, oranges, broccoli, carrots, hot dogs, pizza, donuts, cake, chairs, couches, potted plants, beds, dining tables, toilets, tv's, laptops, computer mice, remotes, keyboards, cell phones, microwaves, ovens, toasters, sinks, refrigerators, books, clocks, vases, scissors, teddy bears, hair driers, and toothbrushes.

It uses the recent EfficientDet convolutional neural network (CNN) detector implemented in the TensorFlow Lite framework.

When it detects an object, it can be configured to send you a text message and/or upload a picture of the object to the cloud.

Begin by turning on your Vizy and pointing your browser to it. (Please refer to the getting started guide if you need help with connecting to your Vizy, etc.) Run the Object Detector application by clicking on the ☰ icon in the upper right corner and selecting Apps/examples. Then scroll over to Object Detector in Apps, then click on Run.

The Object Detector application takes several seconds to start up. You'll be presented with a screen similar to below. The image at the top of the screen is the live video feed of what Vizy sees.

To test, you can present various objects such as a bottle, cup, scissors, and yourself (person). Bear in mind that some objects (such as forks and spoons) rely to some degree on contextual cues (such as tables and plates). You can adjust the detection sensitivity (see section on Controls and Settings), depending on how many false negative detections (increase sensitivity) or false positive detections (decrease sensitivity).

You'll notice that Vizy will keep track of recently detected objects by displaying a picture and timestamp of the detected object in the media queue as shown.

Vizy keeps track of each detected object between frames and does its best to determine when objects first enter the scene and when they leave the scene. When an object leaves the scene, Vizy will pick a “good” picture and add it to the media queue. This way you can get a quick sense of recent activity by scrolling through the media queue's pictures.

The Brightness slider gives you control over the brightness of the pictures and live video feed.

Clicking on Settings will bring up the Settings dialog.

• Detection sensitivity: Increasing the sensitivity will result in more detections, but possibly more false positive detections. Decreasing the sensitivity will result in fewer detections, but possibly more false negative detections.
• Enabled classes: Check the checkbox of object classes that you're interested in and clear the checkbox of the classes that you're not interested in. The enabled classes will be logged in the media queue.
• Trigger classes: Check the checkbox of object classes that you want to trigger events – in particular, texting a picture of the detected object – and clear the checkbox of the classes that you don't want to trigger events. See the section on Texting.
• Upload to Google Photos: Check this if you want the media items in the media queue to be uploaded to Google Photos. Google services need to be configured, however. See the section on Configuring Google services.

Your Vizy can upload pictures and videos to Google Photos. In order for Vizy to upload photos to the cloud, you'll want to set up Google services.

Once you've set this up, other Vizy applications will have access to Google cloud services such as Photos, Gmail, Sheets, and Google Drive.

Vizy's texting service allows Vizy to send you updates (typically to your phone). For example, Vizy can text pictures of objects that you're interested in that are set in Trigger classes (see settings and the picture below, from the Telegram smartphone app).

Or you can ask it to show you pictures of the most recent detections (see Text commands below). One of the advantages of texting is that you can interact with your Vizy from practically anywhere as long as your Vizy has a network connection. It's also quick!

Currently, the only text command that the Object Detector supports is mrm (most recent media). For example, to get information (description and picture) of the most recent detection:

Or you can get the N most recent detections by adding a number:

For more advanced users who want to add their own custom features, the Object Detection application has handler code for various events and for text messages. The handler code is in /home/pi/vizy/apps/object_detector/handlers.py. Note, you can bring up the handler code easily from Vizy's built-in text editor by clicking on the ☰ button in the text editor and selecting handlers.py. Note also, once you change handlers.py you can simply click reload/refresh on your browser and Vizy will automatically restart the application and your code changes will take effect.

The event handler function is handle_event, which is called when an event occurs:

def handle_event(self, event):
    print(f"handle_event: {event}")
    ...

Here, the argument self is the Object Detector class object and event is a dictionary with various values depending on the event. In particular, the event_type value specifies the type of the event. The different types are listed below:

• trigger: This event indicates that a trigger class object has been identified. The image and timestamp are included. The default implementation of handle_event sends a text message with the object class, timestamp and image.
• register: This event indicates when an object has entered the scene. The objects are listed in the dets field.
• deregister: This event indicates when an object has left the scene. The objects are listed in the dets field.
• daytime: This event indicates when it has entered the “daytime” state and has enough light to reliably identify objects.
• nighttime: This event indicates when it has entered the “nighttime” state and is inactive.

The text handler function is handle_text, which is called when Vizy receives a text message via Telegram. handle_text is called when none of Vizy's text handlers know how to handle the text message.

def handle_text(self, words, sender, context):
    print(f"handle_text from {sender}: {words}, context: {context}")

Here, the argument self is the Object Detector class object, and words is the list of words in the text message. sender is the person that sent the text and context is a list of contextual strings.