COSC 341: Human Computer Interaction

A3: Gesture Recognition [25 pts]

Due date: ~~Oct 29, 2017, 9:00am~~ Nov 01, 11:59pm

What to submit:

A zip folder with:

A video of your working solutions - show us the gesture and the output in the console.
Expected video length: 45 seconds (15 seconds per gesture)
Your source code in .cs, with inline documentation
A plain text file (.txt) of the console output generated by your program (one per working gesture)
A plain text file (.txt) indicating any known bugs or incomplete functions in your program

Any help you got and any code you are using from downloaded sources must be cited within your code. Online sources must have accompanying URLs.

Overview

This is a programming assignment where you will be working with C# code in the Unity environment. The purpose of this assignment is for you to implement a program that recognizes simple hand gestures. Assignment 4 will take the program you build here and add on a playful application context. In order to maintain flexibility in your code here, you may wish to read ahead on the A4 specs to see what you have to do next.

Note: You are required to code in C# in the Unity environment for this assignment.

Specific Requirements

Create a new project in Unity. Download the TextureFilter.cs and TextureScale.cs files as assets to your project. These files should not be changed.
Download the Webcam.cs file as an asset to your project. You may change this file in later steps, or add other scripts as desired.
With the 3 scripts, build and run your project. First, you will need to drag the Webcam script over the Main Camera in order to add it as a component. When you run the project, you will see two windows pop up in your Scene: one showing what's in view in your webcam, and the other showing two colours that have been tracked. Below is a sample screenshot of my desktop:

In the screenshot, the webcam shows that I've got a red pen cap and a blue pen cap taped to my thumb and index finger respectively on my left hand, and those are the colours being tracked (which you can tweak in the Inspector), and you can see those colours showing up in the enlarged window.
The program trakcs two colours. You can change those colours in the Inspector by clicking on it. I've got Threshold 1 and Threshold 2 set to 0.1. You can change those values anywhere from 0 to 1, with larger values catching more "shades" of the colours you've selected.
In the Webcam.cs script, there's a magic number "14400" which is the expected data length. If that doesn't match your machine's data length, change it.
How the Webcam.cs script works: it finds the webcam device on your machine (e.g. laptop), collects the info into a texture called "camFeed". This info is projected onto a "camWindow" so you can see exactly what's in view by your webcam. That info gets processed as a bunch of pixels in an array called "data". (Because we are talking about images, we will typically think of "data" as a 2D matrix, even though it's only stored as a 1D array.) We simplify the image so we only want to see the colours that we are interested in. Those colours we are able to track will get projected into the "pixelWindow". A diagram of this is shown below:
Once you get the scripts working, you will have a program that detects two colours. I call this the Colour Detection project. Become familiar with this Unity environment and the Webcam.cs code, so that you can build on it.

The rest of this assignment will get you to extend the program in order to recognize the following gestures:

	Swipe: Point your index finger and move sideways (to the right)
	Pinch: Spread apart your index finger and thumb, then move them towards each other until the two fingers touch
	Frame: Position your index finger and thumb into an L shape on both hands and hold them up like a picture frame

Work with the Webcam.cs code and figure out how you can track and test the recognition of these gestures. (Your code should be fairly simple.)

To keep things simple, you will assume that your index and thumb fingers have unique colours (e.g. wear felt finger covers). When your fingers move (within the webcam view), the program will know where those fingers are. Therefore, when you swipe, you will need the program to understand where your index finger is, where it's moving to, and whether the movement is generally in a horizontal pattern.
After you get swipe working, you can try pinch. This requires your program to know the position of your index and thumb, and whether the distance between those two fingers is getting smaller and smaller, until they are touching.
Finally, you will work on the frame gesture. Rather than painting your fingers different colours, you may want to cut out cardboard of two different colours in place of your hands. (This will make testing easier.) Your program will need to recognize when the right and left hands form two L-shapes in a pre-specified orientation.

Resources

Starting Unity for the first time
Read online manual, focus on these topics:
- Working in Unity: Basics
- Working in Unity: The Main Windows (briefly - know what the different windows are, and know the Console window)
In our Colour Detection project, we have scripts as assets (Project window), panels with what's shown in the webcam and drawn back out (Scene window), the Main Camera (Hierarchy window), and detailed information of game objects (Inspector window).
Basics of a script
Note: the data matrix draws the pixels onto the pixelWindow from left to right and bottom to top. That means, if you think about your x,y coordinates, the origin is at the bottom left hand corner, x increases going right, and y increases going upwards.
Unity's C# beginners video
I didn't include anymore C# references because I coded the solution as if I was writing Java code. Seriously, you don't need to be a specialist in C# to complete this assignment.

Grading Criteria

[2 pts] Getting the basic Color Detection program to work on your station
[4 pts] Videos and source code demonstrating the swipe gesture: finger moves from left to right (not just a little)
[6 pts] Videos and source code demonstrating the pinch gesture: distance calculation becomes shorter
[8 pts] Videos and source code demonstrating the frame gesture: finding two L-shapes of certain sizes
[3 pts] Console output
[2 pts] Documention within your code and known bugs if any

Note: Because color detection is really finicky (due to lighting and color shades), you must show clearly in your video that your code works.