André Carlucci

Skeptic .net development

Creating a Smart Mirror with an UpBoard, MagicMirror2 and a RealSense Camera

I always wanted to create a Smart Mirror to geek up my home here in Brazil. The idea is very simple: add a small board and a monitor behind a two-way mirror and code some software to provide useful information to the person in front of it.

I’ve seen on the Internet some implementations of this, but I wanted to provide personalized information to the person in front of it using a camera and facial recognition algorithms. Something like providing commuting time and next appointments for me and change the information when my wife would stand in front of it.

So, what materials did I use?

  • AOC E1670swu USB Monitor
  • Up Board running Windows 10 full
  • SR300 RealSense Camera
  • notebook box for the case
  • 25cm x 35cm Acrylic See-Through Mirror, 1mm
  • VGA cable * HDMI – VGA adapter
  • black tape, glue, small screws
  • Self-Retracting Safety Knife

Let’s create a Smart Mirror

The Software Part

Interacting with the SR300 RealSense Camera

As I wanted to use c#, I had to port the open-source library called librealsense from c++ to .net. I did that and publish the new project on github as OpenRealSense and it is available for .net users via nuget. This library is necessary to get the streams from the RealSense Camera. The basic code to get the streams is:

For the facial recognition part, I used the a .net port of the OpenCV framework called Emgu.CV. The way to convert RealSense frames to OpenCV/EmguCV ones is shown below:

To detect a face in a frame, convert the image to grayscale and use the DetecHaarCascade function:

Once you have detected a face, crop it from the image and compare it with a faces database.
If you don’t have a faces database yet, you can use the same procedure above to find the face you want and save it as a file so I you can use it for comparison in the future. As I’m not concerned with security, I will use the EigenObjectRecognizer to do the job of facial recognition:

With our label (username) in hands, it’s just the matter of using an HttpClient to update the current user in our Smart Mirror front end application. The whole code and application for face detection and recognition is open-source and it’s available on GitHub.

The Smart Mirror Front End

I found a popular open-source project called MagicMirror2 that has a really easy way to write extensibility modules for new features.

My idea was to install it and write a new module to make it multi-user aware and integrate it with our facial recognition app. The resulting architecture is something like that:

  • Red boxes were developed by me
  • Blue boxes are already existing open-source libraries

Installing the MagicMirror2 1.

  1. Download and install the latest Node.js version.
  2. Clone the repository and check out the master branch: git clone https://github.com/MichMich/MagicMirror.git
  3. Enter the repository: cd MagicMirror
  4. Install and run the app: npm install && npm start MagicMirror2 already comes with many modules. Check its documentation for enabling and configuring them.

To create a new module for Magic Mirror, you just have to add a subfolder for your module under the “modules” folder and write a javascript file with the start up code in it.

A minimum module that would not do anything fancy would be:

Our RealSense Module

Let’s create a module called MMM-RealSense. We will take advantage of a property present in every module called “classes”. We will use special words to set this property as following:

  • default: modules set if no user is detected
  • everyone: modules which should be shown for every user
  • [username]: modules which should be shown to a specific username.

The [username] is actually any label you have used in our WPF app above when you recognized yourself or another person. In my case, I will use andrecarlucci for me and robertacarlucci for my wife.

There is also a special file called node_helper.js that you can add to your module folder to add extra features. In our case, we will open an endpoint so our WPF App can send requests to it every time it detect new users.

node_helper.js

Now we have a webservice that can be reached via http://yourMagicMirrorServer:port/login?user=[username]. Our WpfApp should call exactly this address. Check the code on Github for more details.

Install the MMM-RealSense Module

  1. Enter the repository: cd MagicMirror/modules
  2. Clone the repository and check out the master branch: git clone https://github.com/andrecarlucci/MMM-RealSense.git
  3. Add the module to the list of modules in your MagicMirror/config/config.js config.js

Enter under “users” the array of labels and welcome messages so our Smart Mirror can greet them when they interact with the mirror.

And that’s it! Start the MagicMirror2, start the WpfApp and you’re all set :)

Assembling the Smart Mirror

The idea is to connect the monitor and the camera to the Up Board. The Up Board is an amazing piece of hardware with the size of a raspberry pi, but it can run Windows 10 full with no problems.

Look how awesome it is:

I got a wifi dongle, connected to it and configured to use my home network. Then, I installed the MagicMirror2, our facial recognition App and a VNC Server so I can update the board without openning the box (you can use a bluetooth dongle and a wireless keyboard/mouse if you want). If you’re looking for a good VNC server/client, a suggest this one: http://www.tightvnc.com/

The first thing is to find a monitor to use. I got an AOC e1670swu because this one is really light and it consumes only 7 Watts. The downside is that its light emission is not the strongest one, so the widgets on the mirror appear a little bit weaker than I would like. Anyway, I think it was fine for the price.

Another thing is that it has only VGA output and our board uses HDMI, so I had to buy an adapter too.

Disassemble the monitor pulling out the front frame. You can use a “flat-blade” screwdriver to help you with the process, but don’t forget: if the frame is not coming out easily, you’re doing it wrong. Don’t force it too much, be careful.

The RealSense Camera

Intel RealSense camera SR300 is the latest and most advanced front-facing camera of Intel. It uses various sensing technologies to achieve depth perception, 3D imaging, interior mapping, and feature tracking.

Using the librealsense, now ported to C#, we can get the image streams (depth, color, infrared and fisheye) and play around with them using OpenCV. The camera will be on top of the casing and be connected to the UpBoard using a USB 3.0 cable.

The casing

The next step is to find a casing for your mirror. You can use a medicine cabinet, a wooden box or any other thing that feels right to you. Luckily I found a notebook box that could fit the monitor in a perfect way.

I made a hole on the back side and fixed the acrylic see-through mirror using the black tape.

With a self-retracting safety knife, I opened a hole for the energy supply cables and for the RealSense camera.

Then I fixed the Up Board, the wifi dongle and the cables in the interior of the notebook box.

The next things are the monitor boards and the disassembled monitor. I used the black tape again.

Close the box, fix the camera on the top of it and hang it on the wall. That’s it! You’re a proud owner of a Smart Mirror!

Next Steps?

I can’t stop thinking about adding new features to the Smart Mirror.

The next one in my mind is to use the depth stream of the RealSense Camera to detect the finger of the user in front of the mirror and make it act as a pointer.

This way we could enable, disable or add custom actions to the mirror without having to use a keyboard/mouse for it. A simple way to do this would be:

  • delete all points further than one meter
  • get the closest point to the mirror that has a distance greater than 40cm (I’ll find the best number here) from the further one
  • use this point the pointer I’m guessing that the only way to achieve this configuration is if the user raise his arm in front of him/her.

It’s not perfect, but it would be simple and fast.

Well, I let you know if it works. Until then, have fun with the mirror :)