Dinosaur Passage
A Multi-modal Augmented Reality Museum Experience
MDes/MPS IxD Studio at Carnegie Mellon University
Instructor: Mark Baskinger, Kyuha Shim
Duration: 6 weeks
Team Member: Wu Shengzhi, Devika Singh, Scott Dombkowski
My Contribution: Research, Concept Video, AR/Hololens Prototype
Overview
Dinosaur Passage is a multi-modal AR application for Carnegie Museum of Natural History that allows users to create their own personalized dinosaur worlds by using natural gestures. The project integrates head-mounted AR goggle and mobile AR apps to forge an engaging and dynamic learning experience for visitors in the museum.
Prototype
I made a few rapid AR prototypes with unity in order to validate our design concept, and better demonstrate our design concept to our clients(Carnegie Museum of Natural History). The live demo during final presentation received very good feedback
Hololens AR Prototype
Made by Unity, Microsoft MixedRealityToolkit & Vuforia
Mobile AR Prototype
Made by Unity & Vuforia
Project Architecture
Project Architecture
Book as an analogy to merge the digital and physical
Museums are places of learning. We believe learning by doing is its most effective form. Through AR, books become a medium for a more active learning.
Books take their readers into other worlds. We play off of this and utilize our book as a passage to an alternate universe.
Interact with Natural Gestures
We believe the interaction in head-mounted AR should be natural, and a user should be able to use their intuitive hand gestures to control the digital assets as if interact with our physical worlds.
As such, during our design, we didn’t limit ourselves by the existing hand gestures provided by Microsoft Hololens, instead, we designed a series gestures like changing scale, rotation, placing, selecting from the book etc.
Change Scale
Rotate Object
Select Object
Initial Design Brief
How to improve the Carnegie Museum of Natural History visiting experience.
Through user-centered design, create novel experiences for Carnegie Museum of Natural History visitors that enable meaningful engagement and encourage a deeper connection to the museum.
A photo as a souvenir and a trigger for AR
After creating the dinosaur world, a user will receive a photo of himself in the dinosaur world. The photo does not only serve as a souvenir to help them evoke the museum memory but also become a trigger for seeing the world remotely through a cell phone. With the companion App’s AR model, the user can check his dino world anywhere and anytime, as well as share with his friends or family members.
Design Process
Secondary Research and Insights
Primary Research and Insights
End User Interview
We had conversations with visitors.
-
We heard how visits to the museum can forge deeper connections and create traditions in families.
-
We learned how local visitors have these ongoing relationships with the museum.
-
We learned how visitors see the museum as a place for learning
Primary Research
Experts Interview & Insights
We ran expert interviews.
-
We learned that the museum wants to engage individuals in ways they are engaged outside the museum.
-
We saw how the museum is interested in what people retain from their visit to the museum.
-
We learned that museums want to make their visitors aware that they are not only enjoying themselves but learning when visiting the museum.
Primary Research
Online Survey & Insights
We ran an online survey.
-
Most people go to museums to be entertained
-
People don’t enjoy reading the information on exhibits.
Key Insights:
Dynamic Learning
The optimal scenario for both the visitor and museum is one where a visitor is learning while enjoying themselves. But most of the current museum experiences are static and prohibit users to interact with the artifacts.
Family Tradition
For local citizens, visiting the museum is a family tradition. Especially a museum lasts for centuries, which allows several generations of a family visit the museum repeatedly.
Evolving Connection
Individual’s connections with museums are built over time. Particularly a person visited the museum when he was a child, and he will also bring his children to the museum. Therefore, people’s connections with the museum are unique and constantly evolving.
Lasting Memories
Museum visits often involve moments of learning and spending precious time with family/friends. Thus the memories tend to last for years and sometimes instill children a curious mindset which can significantly affect their growth.
Reframe Design Question
How can an individual’s experience at the museum allow for deeper understanding, engagement, and connection through dynamic and evolving experiences?
Opportunity Area
How might the Natural History Museum…
create more dynamic and interactive learning experiences?
encourage visitors to share their experience with family members and friends?
forge deeper and longer-lasting connections with its visitors?
become not only a collection of artifacts but also a collection of memories?
Ideation
We went through brainstorming and each of the team members drew some quick sketches to convey our ideas as well as to stimulate others' inspiration.And during our ideation, we kept asking questions like does our design solution meet our research insights and meet design criteria.
AR VR Technology Exploration
While exploring possible concepts, our team researched both AR and VR technology.We eventually settled on AR, because of its ability to merge the physical with the digital.
While still relatively undeveloped, AR technology has the ability to dramatically change the ways we interact with the world. Today, AR technology has a number of limitations from its eld of vision to the number of sensor inputs. Significant investment is being made in this eld to overcome such limitations.We can also look at more developed VR technology to see just how AR will develop
.
AR technology also suffers from its high cost.This will change in the next couple of years.You can see from the chart on the right that today the AR industry is valued around 15 million, in only 8 years analysts are predicting the industry will be around 700 million.When the market reaches that level, AR technology will cost significantly less and have significantly lower limitations. Those exploring possible implementations of this technology today will have a competitive advantage over their competitors when this technology is more mainstream and developed.
How to design a novel augmented reality user interface
We did a series research regarding AR/VR experience, specifically the new possibilities of VR/ AR User Interface.
We found two videos that are very inspiring. One is a design engineer from Leapmotion-"Designing for Hands, Not Controllers", another is Fantastic Contraption and why VR Menus Suck
What it interests me most is both of them discuss and explore how VR/ AR User Interface should be different from traditional screen-based UI, especially how VR/AR UI should connect to the physical world to make users feel more natural.
Project Reflection
Overview
When we showed the live demo to our client (Carnegie Museum of Natural History), they were immediately attracted by AR’s immersive experience. We also received positive feedback about using companion App and photos to build a connection with users when they left the museum, which could significantly strengthen the evolving relationships between the public and the museum. While they also suggested that the experience is currently limited to dinosaur exhibition, and they are willing to see the project can be extended to other exhibitions in the museum as well.
Dinosaur Passage
Dinosaur Passage is a dynamic and evolving experience that allows for deeper understanding, interaction, and connection with the Carnegie Museum of Natural History. We take advantage of alternate realities’ unique capacity to merge the digital with the physical, by bringing alive the environments museums reproduce.
Through this, visitors become creators and builders of universes existing within a museum which generates more immersive learning opportunities and extended engagements between museums and their visitors
Persona
Sarah
Age: 12 years-old
Work: Middle school student
Location: Lives in Pittsburgh, PA
She’s been to the Natural History Museum a couple times, with the last being a school field trip. She enjoyed that visit but wished it was a little more interactive.
She enjoys playing video games and staying in touch with her friends on her cell phone. Every week, she has music and art lessons that she really likes. As she has grown up, she has spent more and more time with her friends and less and less with her parents. Her parents wish that there were more activities that they could enjoy with their daughter in Pittsburgh.
Dinosaur Passage | User Experience Journey
Our concept consists of 4 key stages: the Immersion Stage, the Creation Stage, the Experience Stage, and Connection Stage
Immersion
Creation
Experience
Connection
Storyboard
Immersion
Enter Museum
Sarah visits the Natural History Museum with her parents.
Buy Ticket
She is particularly excited about the dinosaur exhibit. One of the docents introduces her to Dino Passage. She is really interested and asks her parents if they can buy a ticket for her. Her parents agree and buy her a ticket. Sarah retrieves her AR glasses and book.
Wear AR Glass
Sarah walks over to the dedicated Dino Passage section of the museum near the main dinosaur sections and finds a lone dinosaur skeleton in the room. She puts on her AR glasses when she arrives.
Butterfly Guide
With the AR glasses on, Sarah is introduced to the butterfly. The butterfly serves as the guide, personal assistant, and Carnegie Museum of Natural History representative. The butterfly is an energetic and helpful guide.
Storyboard
Creation
Create Dino Mate
The butterfly guides Sarah through the creation of her Dino Mate. Sarah is asked a number of questions concerning what she wanted out of a mate. For example, the food, habitat, and location of her mate. She becomes more invested in the creation of her universe through this process.
Open AR Book
After answering a set of questions, Sarah is instructed to open up the book by the butterfly. When she opens the book, the dedicated Dino Passage section comes to life.
Sarah’s new universe is based on her earlier answers. For instance, if Sarah wanted her dinosaur to come from a more mountainous region, her surroundings would now reflect that.
Customize Dinosaur
Once the book is opened, Sarah has the ability to select from a number of dinosaurs. The selection of dinosaurs is based on Sarah’s answers to the earlier questions. Through this process, Sarah not only finds her perfect dino
mate, but learns more about dinosaurs (personality, diet, etc...).
Once Sarah decides on a dinosaur, she places that dinosaur in the world. No longer is Sarah an observer, she is an active participant.
Add Plants & Animals
Once Sarah has customized her dinosaur, the butterfly lets her know that she can add additional dinosaurs, animals, and plants to her new universe.
Storyboard
Creation
Adding Objects
While adding additional objects to her universe, Sarah employs custom
Interactions Once her dino is in the world, Sarah has the ability to customize the color of her dinosaur and the pattern of her dinosaur’s hide.
Manipulate Position
To manipulate the position of an object, Sarah grabs the objects and adjusts its x and y position in space.
Manipulate Orientation
To manipulate the orientation of an object, Sarah scoops the object and
turns her hand. If Sarah decides she doesn’t like an object, she grabs the object and throws it behind her.
Manipulate Size
To manipulate the size of an object, Sarah grabs the object and adjusts its z position in space.
Storyboard
Experience
Experience Your World
While Sarah has her AR glasses on, she is part of a living and evolving universe she created. She walks around and appreciates her creation. She experiences the dynamics of her creation (dinosaurs interacting with dinosaurs, dinosaurs interacting with other animals, etc...)
The butterfly is always nearby. Not only providing instruction, but also tidbits of knowledge. Letting Sarah know why and when her universe is out ofbalance/harmony and potential ways to move her universe into
balance
Learn More knowledge
The butterfly can also provide direction, guiding Sarah’s experience and learning, by helping her discover areas she would not on her own.
If Sarah is interested in a special c object, she can walk up to it and open up her book and find detailed information about that object.
Storyboard
Connection
Get a Souvenir
When leaving the museum, Sarah receives a customized postcard depicting her Dino Passage universe. At home, she downloads the Dino Passage companion app. In the companion app, she takes a photo of her postcard and her Dino Passage universe becomes associated with her version of the app.
AR Companion App
Now that her exhibit is associated, Sarah can open up the companion app’s AR mode. When in AR mode, Sarah can center her phone’s camera on her postcard and see an AR representation of her universe. This allows her to see what has changed since her last visit.
Weekly Notification
The companion app also enables weekly notifications that let Sarah know of any big developments.These notifications include both educational insights and calls to action. They also preserve Sarah’s connection to her universe and make it more likely that Sarah comes back to the museum to tend to her universe.
Next step and future consideration:
-
Share: design an experience that two or more persons can create the dinosaur world together, or allows other people watching while one is creating
-
Mobile app version for accessibility: since the HMD AR isn't very accessible currently, so I believe it is worthwhile to re-design the project with a mobile app which can run on both Android and IOS devices.
-
Avatar instructor: how to make full use of the butterfly avatar instructor to provide more personalized and seamless learning experience
The advantages of HMD (Head-mounted Display) AR
Immersive: experience complex subject, theories, and concepts
Explorative: hands-on approach helps users’ retention
Self-pacing: suited to all types of learning styles;
Active vs Passive: encourages active participation;
Engagement: HMD AR experience is immediate engagement
AR Prototyping Documentation
Objective:
We aimed to prototype a Hololens app with Unity so that we could demo our concept to promote a deeper understanding of our concept within our team as well as to the potential audience
and clients.
Process:
Initially, we watched the Microsoft Mixed Reality Academy online course tutorial. The course provided a basic understanding of how Hololens and Mixed Reality Apps are created, such as gesture and gaze interactions.
Using Vuforia
Since we had the concept to create a book that allows for interaction, for example, the pulling of an object out of a book. To achieve this we explored Vuforia, a unity library, to build our prototype.
Vuforia mainly supports Augmented Reality with marker recognition, but its functionality is limited when compared with Apple ARkit or
Google ARcore.
The Vuforia development environment is more friendly to developers, and with a few online tutorials, it could be handled and used to create the App.
Constraint:
The last version of unity doesn’t support Vuforia for Hololens, which is frustrating. So we had to download the older version of Unity to make it work.
Final Solution
Even help from a student at the ETC (Entertainment Technology Center) building the app in Hololens was still hard, but we finally made it. Our IOS development was successful and we were able to create a functional AR app with Unity and Vuforia.
Obstacle:
The Microsoft Hololens Develop Environment is unfriendly for developers. While building a mixed reality app is not hard, especially with a pre-programmed AR toolkit, the setting of the program environment was much more difficult than expected. Building Hololens App also requires Windows 10 system, Visual Studio 2017, Unity withWindows. Net programming platform, and AR toolkit plugin.
Different versions of software sometimes conflict with each other, so the system has to be reset repeatedly, which is very time
consuming.
Hololens Emulator requires Hyper-V and hardware support, which is not available in Mac system. So, the emulator could not be used to preview my program on the Laptop. Remote export the program in Hololens is unstable, thus a plugin USB wire had to be used and put it on my head to test. This process is inconvenient and uncomfortable.