Pictures:

^ the headset. Googly eyes are standard issue.

^ The controller and sidearm, both with AR trackers.

 

^ Unity view of game development.

 

Skills used:

Leadership, Unity3D, Autodesk Fusion 360, Additive manufacturing, AR, VR

 

Description:

With all the stuff that I do, I love pushing the boundaries of what exactly can be done with a given technology. Anyone who has been stalking this site for an extended period of time may remember my attempt to use AR to make a virtual RSS feed watch. This idea died off about a year ago, when Microsoft HoloLens swooped in and stole my idea. Thanks guys. Regardless, that didn’t really end my interest in VR/AR, and I was still interested in doing something amazing with the tech. There was just one problem: nobody in the mechanical engineering field will care at all about a virtual reality game, so being pulled 5 ways to Sunday, VR dev got pushed under the rug for almost 2 years.

Until one day, Google came to the Virginia Tech campus and teamed up with the Entrepreneur club (which I do a lot with) to make a VR video game using Google cardboard. Now, android app development is far, far, FAR, from my specialty, so of course, I just had to try it.

As a senior, I didn’t have much trouble finding freshman engineers to join in, and our team consisted of myself as team lead, a mechanical engineer, two computer engineers, a graphic designer, and a computer scientist. Aside from myself, the rest of the team consisted of freshman students.

The challenge was simple: we had six weeks to develop a virtual reality video game and demo it. Not too hard, but of course, I couldn’t make it easy. Being team lead for the additive manufacturing competition, as well as 2 hackathon teams, numerous class projects, boy scouts, leadership summits, etc. has made me a tempered leader at this point, and one of the most important leadership axioms is to pull as much effort out of your human resources as you can without expending them. If its too easy, they will get bored. Too hard, they will give up. So how do you keep freshman programmers interested in a 6 week project? You make it tough.

After some brainstorming, we decided we wanted a horror/fantasy game. Looking around at other games in the industry, there are a lot of cardboard apps, and a lot of desktop games, but the mobile games don’t really use any sort of hand tracking or controllers like the Rift and the Vive do. That sounded like an interesting question we could come up with an awnser to. Using the same library I did for the RSS watch, we made a shield blow up in virtual space, that could be manipulated by hand and used to block attacks in virtual space. As if that wasn’t enough, we hooked up an electroencephalograph in order to read the focus of the player, so they did more damage the more focused they are.

As for how the hardware actually works, it can be subdivided into 3 separate subsystems: AR, interfacing, and bluetooth.

The AR side’s operating system is pretty straightforward. Using the view from the camera, unity can calculate its orientation and distance to generate a sword or a shield on top of the icon, which can be used to interface with the world. It was a bit complicated because the library that was used, NYARtoolkit, was made and developed in japan, so much of its documentation was in Japanese, and google translate only goes so far. Why did we pick this over the English, better documented base library ARtoolkit? Well, NYARtoolkit is pure C#, and doesn’t use plugins. More on that nightmare later.

Interfacing was my specialty, because I do a fair amount of microcontroller and Arduino work, I designed and built the physical controller in Fusion 360…

… and attached knurled nut threads, a simple joystick, a battery pick, logic level shifters, an HC-05 bluetooth chip, the AR tracker icon, a microUSB connector, analog output accelerometer, and two buttons and wired them up to the Arduino. To get brainwave data, we used a mindflex headset. The actual game itself is dead at this point, but the headsets sometimes float around on ebay. We drilled a hole into the side of the headset, and tapped into the TX and GND pins of the mindflex’s Neurosky chip (Which actually transmits straight serial. Who would have guessed?). This is wired to a Female USB connector, and connected to the controller. The Arduino itself reads all of these sensors and generates a string of csv data, which looks something like:

0, 53, 34, 65, -40, 0, 1, 0, 1

The numbers are, in order, EEG connection, Focus, Meditation, x axis, y axis (from -100 to 100), trigger 1 button, trigger 2 button, button on the stick, and acceleration higher than 2g (for sword swings). Focus and Meditation are two values from the EEG produced by black magic inside the neurosky chip. They are consciously controllable, but other than that we don’t know how they are made.

All that is spit over serial to the Bluetooth HC-05 that the android phone can read, and translated into the game using a Bluetooth library. I don’t know too much about how this part was done, but it managed to be completed, so long as we know the UUID of the HC-05.

And another note on plugins: they all want to be in charge, so the VR plugin provided by google would complain about collisions with the Bluetooth plugin. Our l33t h4xx0r programming lead managed to get them all to a metastable equilibrium, so that they both worked, at least for what we were doing.

The 6 weeks blew by pretty fast. After we had the goal in mind, we progressed as fast as we could with our full schedules. We had a couple of clashes on what we all thought the game should be or do, but that’s to be expected from any development team. Of the 24 teams formed at the kickoff, only 8 remained to demo at the end.

The judges, mostly googlers, but also a few professors and grad students in the VR field came around to look at our project and critique it. What they said we mostly expected from the get-go: It was a cool idea, but no one would buy an app that uses this because the hardware itself costs around $50 to produce.

In the end, we didn’t win best overall prize ☹, but we did manage to walk away with the award for most technically challenging from google, which is no small feat.

And oh yeah, and we made it to the front of the Roanoke times!

Pros:

• For the most part, the technology worked. We ran into some trouble with lighting and AR tracking, but still usable.
• Controllers and spatial tracking for mobile devices is still in its infancy, and this really lowers the barrier of entry for new developers.
• We won “most technically challenging” from Google.

Cons:

• AR tracking was unreliable, mainly due to the lighting in the area. Perhaps a flashlight on the front?
• Chugs battery power. Drains about 1% battery every minute, and makes the processor hot enough to cook an egg.
• The game itself was basic. It would have been great to see more, but we ran out of time.
• Using the EEG requires electrolyte, which is messy and annoying.
• Game development is probably the most worthless thing on a resume for mechanical engineers.

 

Opportunities for improvement:

 

When making code or games (or any product really), the most important thing is how it feels to the user. We, as engineers, we really focused on the “how” of this, as opposed to the “who”. While all the technology did indeed work, the headset was uncomfortable, the controller gave me blisters after extended use, and the game itself was slow, and admittedly a bit buggy.  It has a long way to go before becoming a viable product, but for a proof of concept, it preformed spectacularly.

 

Conclusion:

A good experience in leadership, but since game design is so unbelievably far outside the scope of my expertise, this is a skill I feel that no company will really care about from a mechanical engineer. I mainly did this for the leadership experience, as well as the desire to see the project through to completion, as well as the interdisciplinary work that goes with it. However, the end product is far from an end product, and is more of a prototype. That being said, a leader looks after his team, and I believe I did a good job of pushing the rest of the team to do more than they thought they could, and for the CS and CPE majors, this is a huge stepping stone for them, much more than it is for me.