Augmented reality sandbox control box updated

This slideshow requires JavaScript.

I constructed a couple of these control boxes for my ARsandbox project using Arduino Micro and Adafruit Feather 32u4. The main goal is to make the interaction between a user and the software more intuitive. The two flashing buttons add and remove water from the simulation, which is pretty nice. I wanted to add another feature, to move the elevation up and down. Say you have a lot of sand in the box but you want to display some “water”. You have to remove a lot of sand (time consuming) from the box or edit a config file (near impossible for an average user). On the other hand, if I can change the base plane location in the calculation, I can achieve the above goal without removing sand or editing files. With help of the original developer Dr. Oliver Kreylos, I started messing with the source code. It took me a day to understand what I could do to change the base plane and how to do it. I ended up creating a tool to manipulate the plane. Now that the plane can be moved, I added a rotary encoder to the control box to emulate the keyboard keys assigned to move the plane. Voila!

The photo gallery above shows 5 shots of different elevation values. You can clearly see the “sea” on bottom right is shrinking after each turn of the knob.

Here is a short video. I was slowly raising the elevation so more and more land became above water. Then I quickly returned the land back into water:

Here is a photo of the inside of the box:

I added a breakout board for rotary encoder and buttons. It’s very tightly sandwiched between the two buttons. There is no space between the board and the buttons, not even a fraction of millimeter. I don’t know how these pieces just managed to fit. Here is the breakout board:

I made this a while ago as an interface for my phi-panels LCD backpacks. This breakout board is pretty simple. There are two encoder channels and one shaft button, plus six push buttons, which I am not using. I just connected to the encoder channels, shaft button, and the common. I had to cut off a strip on the right side of the board to fit into the box. I was using my phi_interfaces library to read the encoder and emulate a button push if the encoder is rotated one way, another button if the encoder is rotated the other way. I didn’t assign any function for the shaft button. I am thinking that I should use it to switch between elevation adjustment and water speed adjustment.

I haven’t updated the Adafruit Feature 32u4 version but will probably design a new printed circuit board so assembling will be easier, and not involving cutting off part of a board. Some code in case anyone wants to replicate it.


/*
 * Credit: Dr. John Liu
 * Purpose: This sketch emulates keyboard keys "1" and "2" with two push buttons with LEDs. It also flashes the LEDs
 * If a rotary encoder is present, it will emulate '5' and '6' if you rotate the encoder's shaft. This combined with my modification on ARsandbox source code will shift color mapping up and down.
 * This version uses p-channel mosfets and open drain with 10Kohm pull-up to 5V to control LEDs with the 3.3V Adafruit Feather32u4.
 * Notes: On Adafruit Feather 32u4, pin 9 is connected to battery sensing voltage divider.
 * 2018-01-28
 * Visit https://liudr.wordpress.com for more information
 */
#include "Keyboard.h"
#include <phi_interfaces.h>
#define USING_MICRO
#ifdef USING_MICRO
const int button_1=2; // 11 for Adafruit feather 32u4, 2 for Arduino Micro;
const int button_2=3; // 10 for Adafruit feather 32u4, 3 for Arduino Micro
const int EncoderChnA=7;
const int ChnCommon=8;
const int EncoderChnB=9;
const int ShaftBtn=10;
const int EncoderDetent=18;
#endif

#ifdef USING_FEATHER
const int button_1=11; // 11 for Adafruit feather 32u4, 2 for Arduino Micro;
const int button_2=10; // 10 for Adafruit feather 32u4, 3 for Arduino Micro
#endif

const int led_1=6;
const int led_2=5;
const unsigned long led_on_ms=300;
const unsigned long led_off_ms=1700;
const unsigned int button_1_key='1'; //KEY_LEFT_ARROW;
const unsigned int button_2_key='2'; //KEY_RIGHT_ARROW;
const unsigned int UpKeyOut='5';
const unsigned int DownKeyOut='6';

int prev_1=HIGH;
int prev_2=HIGH;
int led_stat=LOW;

unsigned long prev_1_ms=0;
unsigned long prev_2_ms=0;
unsigned long blink_timer_0_ms=0;
unsigned long blink_timer_1_ms=0;

int debounce_ms=25;

char mapping[]={'U','D'}; // This is a rotary encoder so it returns U for up and D for down on the dial.
phi_rotary_encoders MyEncoder(mapping, EncoderChnA, EncoderChnB, EncoderDetent);
//multiple_button_input* dial1=&my_encoder1;

void setup()
{
  // make pin 2 an input and turn on the
  // pullup resistor so it goes high unless
  // connected to ground:
  pinMode(button_1, INPUT_PULLUP);
  pinMode(button_2, INPUT_PULLUP);
  pinMode(led_1,OUTPUT);
  pinMode(led_2,OUTPUT);
  pinMode(ChnCommon,OUTPUT);
  digitalWrite(led_1,LOW);
  digitalWrite(led_2,LOW);
  digitalWrite(ChnCommon,LOW); // Using this pin as ground since some prototypes don't have enough gnd pins.
  Keyboard.begin();
  blink_timer_0_ms=millis();
}

void loop()
{
  unsigned char ch=MyEncoder.getKey(); // Rotary encoder emulates two buttons.
  if (ch==mapping[0])
  {
    Keyboard.write(UpKeyOut);
  }
  else if (ch==mapping[1])
  {
    Keyboard.write(DownKeyOut);
  }

  switch (led_stat)
  {
    case LOW:
    if (millis()-blink_timer_0_ms>led_on_ms)
    {
      blink_timer_0_ms=millis();
      led_stat=HIGH;
      pinMode(led_1,INPUT); // Open drain to let pull-up resistor pull drain to 5V.
      pinMode(led_2,INPUT); // Open drain to let pull-up resistor pull drain to 5V.
    }
    break;

    case HIGH:
    if (millis()-blink_timer_0_ms>led_off_ms)
    {
      blink_timer_0_ms=millis();
      led_stat=LOW;
      pinMode(led_1,OUTPUT); // Pull drain to GND.
      pinMode(led_2,OUTPUT); // Pull drain to GND
    }
    break;
  }

  if ((digitalRead(button_1) == HIGH)&&(prev_1==LOW))
  {
    if (millis()-prev_1_ms>debounce_ms)
    {
      prev_1_ms=millis();
      Keyboard.release(button_1_key);
      //Serial.println("1 released");
    }
    prev_1=HIGH;
  }

  if ((digitalRead(button_1) == LOW)&&(prev_1==HIGH))
  {
    if (millis()-prev_1_ms>debounce_ms)
    {
      prev_1_ms=millis();
      Keyboard.press(button_1_key);
      //Serial.println("1 pressed");
    }
    prev_1=LOW;
  }

  if ((digitalRead(button_2) == HIGH)&&(prev_2==LOW))
  {
    if (millis()-prev_2_ms>debounce_ms)
    {
      prev_2_ms=millis();
      Keyboard.release(button_2_key);
      //Serial.println("2 released");
    }
    prev_2=HIGH;
  }

  if ((digitalRead(button_2) == LOW)&&(prev_2==HIGH))
  {
    if (millis()-prev_2_ms>debounce_ms)
    {
      prev_2_ms=millis();
      Keyboard.press(button_2_key);
      //Serial.println("2 pressed");
    }
    prev_2=LOW;
  }
}

Augmented reality sandbox control using Arduino Feather

It’s been a while since I completed this project. I promised the Augmented reality sandbox community to write something in details so someone can replicate it. Here is the gist of it:

The Augmented reality sandbox (ARsandbox) is an awesome simulator that blends sand landscaping with augmented reality. A projector casts color-coded elevation, contours, and simulated water flow over regular sand. A 3D sensor measures the sand landscape in real time so the computer knows where sand is high and how it slopes down and renders color and contours according the the 3D data. You can rain over the landscape by pressing a button or hold out your palm over a certain area.

Here is a video of one of my builds:

This is where the control box comes in. The sand box is a 40″ by 30″ box with up to 200lb of sand. The function to rain over the augmented terrain is to press the “1” key on a keyboard. Sand and keyboard shouldn’t mix, not to mention if the system is deployed in a museum, the keyboard probably should be hidden from patrons. So a box with rain and dry buttons will be most convenient. You can make these buttons by following this forum post. You buy and assemble a USB gamepad kit and doing some scripting but it’s not that easy to follow if you are a Linux beginner. I want a plug-n-play solution so that anyone that wants to add a button box can add it without any knowledge of Linux and minimal skills in circuit assembly. In order to achieve my goals, the electronics inside the box has to emulate keyboard keys “1” and “2” for rain and dry. This ensures that the control box requires no software setup on the Linux system. As a bonus, the buttons should flash blue and orange so it would invite patrons to press and is color-coded for rain (blue) or sun (orange). Enter the ARsandbox control box:

Here is a short video of the box in action:

Features of the box:

1. Two buttons. Blue button on the left emulates the “1” key. It starts the rain in the main simulation, and is used in the projector calibration program to enter tie points, and can be assigned other functions in other programs such as extracting plane. Orange button on the right emulates the “2” key. It drys the rain in the main simulator, and can be assigned functions in other programs.

2. Both buttons have LEDs that flash to invite patrons to press. My 5-yr old son figured out what they do simply by the colors they flash because he knew the rain and dry features already but wasn’t told what these buttons would do.

3. No software setup is needed. This box works like another keyboard, with only 2 keys though.

 

Here are the parts I used, in case you want to build your own:

  1. Adafruit Arduino feather 32u4 basic board (1)
  2. LED push buttons 16mm from adafruit (2)
  3. USB-micro cable (1)
  4. USB extension cable (1)
  5. P-channel MOSEFT BS250 from mouser (2)
  6. 330 ohm resistors (2)
  7. 1K ohm resistors (2)
  8. DuPont 30cm female-female jumper wires (4)
  9. Prototype board or my printed circuit board. Click here to download (1)
  10. Enclosure SK-15 and flange kit SK-99 from polycase (1)
  11. Cable gland M3198GBH from Heiland for pre-assembled cables with USB connectors (1)
  12. Arduino sketch. Click here to download

Here is the schematic of the circuit:

Since the ATMEGA32u4 microcontroller on Adafruit Feather 32u4 is operating at 3.3V, I had to use two transistors to ensure that the LEDs inside the buttons light up with sufficient brightness. Currently the code only blinks both button LEDs at the same rate. In the future, I may update the code to blink more rapidly on the button that is depressed.

The following is a prototype that I built with the parts on a perfboard. It’s a bit messy and also took almost 2 hours to complete. I had to solder 16 wire leads and a bunch of jumper wires on the back side of the perfboard. I wouldn’t recommend this to a beginner.

Here is the printed circuit board that I designed (3 boards, two on the right showing the top, 1 on the left showing bottom):

The board is fairly simple and straightforward to solder. Only 8 wire leads and soldering the rest on the printed board, which is very easy. I also cut in half 4 30cm female-female Dupont jumper wires so I can just use the female side with male pins on the board to cut wire leads in half.

Here is a photo of the assembled guts including the microcontroller soldered to the button box board. Notice the black female Dupont connectors :

Here is the assembled board placed inside the enclosure. This enclosure is not cheap but nice and easy to use. It has various knock-outs so I didn’t have to drill a single hole.

Here is the completed box:

Although I am not aiming to sell a lot of these boxes, I think that designing the printed circuit board was the right way to go. I probably spent a few hours designing and proofing my design but I saved about one hour for each subsequent box I made. Plus, if YOU are interested in making this box, it will save YOU at least that much time.

Augmented reality sand box

I have been involved in constructing augmented reality sand box (ARSandbox) lately. It is a beautiful project created by Dr. Oliver Kreylos at UC Davis. The system uses a Microsoft XBox 360 Kinect sensor to digitize the sand in a box and then uses a projector to project color-coded elevation and contours on the sand, thus augmenting the sand with colors and contours. Here is a photo I took on a prototype that we replicated from the ARSandbox created by Dr. Kreylos:

When someone manipulates the sand, thus changing the topography, the projected colors and contours change accordingly. You can also rain over the terrain by a hand gesture over the terrain.

For those that didn’t know, the Microsoft XBox 360 Kinect sensor is a sophiscated set of sensors that include IR projector and camera for depth sensing, body movement and gesture capture, and regular RGB camera and microphone array. The software Dr. Kreylos developed takes the depth image and calculates a topography map and projects it onto the very same sand using a calibrated projector. To show you how good the simulation is, here is a photo:

Did you notice the white cone and the colors/contours on it? The contours are depicting 1cm heights and the cone is about 4cm tall. Here is a close-up of the cone placed at a different location:

It shows roughly 4cm tall and the contours are very well centered around the tip of the cone.

Here is a video:

Like it? The whole setup is not cheap. It needs an expensive video card for the simulation, especially the water. It also needs a decent desktop computer and projector, sand box, frames etc. Here is what my setup looks like:

I didn’t buy a more expensive (thus shorter throw ratio) projector so my setup is very tall even without any legs. I am hoping to develop it into a portable system so I can take a few of them to teachers’ training workshops, museums, schools, fairs etc. for basic education and outreach for water resources.

What I’m thinking about doing using my Arduino/Raspberry pi skills is to add sensors to help preserve the projector’s bulb and have kids and operators interact with the sand box without having to use the keyboard and mouse or understanding linux. Big buttons will do certain predefined things such as rain, drought, etc.

%d bloggers like this: