Project Type

Duration

Jan - Mar, 2021

Team

Team of 4

My Role

UX Research, Participant Recruitment, Test Moderation,
Data analysis and Study Report

Overview

OpenMCT (Open Mission Control Technologies) is a next-generation mission control framework for visualization of data on desktop and mobile devices. It is an open-source platform that can be used for planning and operations of any system that provides telemetry data.

Our client, Blue Origin, was interested in understanding how this tool may be improved to help space mission operators perform their tasks more efficiently.

Process

Study Proposal

Identification of Focus Area

>>

Planning and Preparing Usability Study Kit

>>

Participant Recruitment and Usability Study

>>

Data Analysis and Presentation to client

Study Proposal

Purpose

OpenMCT being an open-source data visualization tool that allows mission control teams to configure and observe telemetric data during mission operations, it is available for users to customize according to their own preferences. The goals of this usability study are to:

Test the current effectiveness of OpenMCT as a usable mission control tool
Understand how users customize their own mission control visualizations
Observe how diverse user expertise contributes to unique interactions with the tool.

Research Questions

How do users organize data on the page for productive mission activities?
How do mission control operators and other potential users uniquely customize their display layouts?
What features of the tool deserve more articulation?

Participants

Recruitment

OpenMCT’s target users are people working in mission operations teams whose goals are to monitor/control data and build their own mission control visualizations. These users aren’t always those who operate in space, but ground workers or anyone interested in telemetry. As per our discussion with the client, we reached out to systems engineers, and people working in mission operations and flight operations. We aimed to include those familiar with but not limited to:

Data visualization / interpretation
Mission planning
Remote monitoring

We reached out to probable demographic via LinkedIn and Facebook groups for Air controllers, Mission operators, etc. Participants were from Blue origin, Boeing, University of Washington Aeronautics and Astronautics Department, etc.

Profiles

Total 8 usability studies were performed and participants were categorised based on the following metric.

Highly experienced - Includes participants who have experience working with mission control software

Moderately experienced - Includes participants who were not directly but somewhat involved with mission control / similar software

Not experienced - Includes participants who were not involved with mission control / similar software

Usability Study

Usability Study

Heuristic Evaluation

We first familiarized ourselves with the software and each member did a heuristic evaluation of the system and then in group we ranked them according to severity. We identified several heuristic violations. Some are listed below:

User control and freedom (3: major usability problem, important to fix)
- Users do not have the option to resize elements such as data columns.
- Users do not have the ability to rearrange the small windows on the interface.

Help and documentation (2: minor usability problem)
- It would be nice to have more instructions on how to use the software, outside of the onboarding tutorial.
- The onboarding tutorial is extensive and is a lot to grasp in less than 5 minutes, and there doesn’t appear to be a way to revisit this tutorial after completing it the first time.

Match between system and real world (2: minor usability problem)
- We noticed a lot of symbols and iconography with little context that the user is expected to learn to get familiar with, instead of any thumbnail images. The system should speak the users’ language, appear natural and logical.

Method and Test Environment

The study will be conducted as a moderated remote one over online video calls with participants sharing their screen and trying to think-out-loud while pointing their cursor as they navigate a few tasks using the OpenMCT tool within their web browser.

Participants will need desktop or laptop computers, a connection to the internet, and webcams for an hour long study.

OpenMCT is open source and available for everyone to use under any web browser. No pre-installation required.

A total of five people will be present in the test meeting. There will be one facilitator interacting with the test participant, one moderator and two others taking notes, making observations and data logging.

Study Kit

Task List

List of pre-decided tasks that the participants were asked to perform during the study based on the research questions.

Consent Form

Signed or acknowledged over email by participants before their study.

Test Script

Script developed to be used during facilitation to maintain an uniform flow across all tests.

Pre-Study Questionnaire

Brief set of questions to ask participants prior to study to get more information about their background.

Task List

List of pre-decided tasks that the participants were asked to perform during the study based on the research questions.

Post-Task Questionnaire

Brief set of questions based on the Modified Bedford Workload Scale to ask the participants after each task is performed.

Post-Study Questionnaire

10 questions to ask participants post study to get information about the System Usability Scale (SUS).

Data Collection

Data was collected by observing the participants as they performed task by thinking out loud. Interviews were recorded on Zoom with participants' permission. 1 data-logger and 2 note-takers recorded the data in shared spreadsheet and word documents for every interview.

Data Analysis

After the user interviews were done, we went through all the data and started putting down important snippets on the sticky notes using the online Miro app. Then we used card sorting technique to do affinity mapping and find patterns in the data.

What is working well?

Being an online web-based software, it does not require installation.
Being an open-source software it can be modified according to requirement.
The initial tutorial demo is helpful and intuitive
Simple interactions make it easier to use. Eg: drag-drop functionality; mouse cursor changes when hovers over a few functionalities; zoom buttons on “Rover Surface Operations” timeline are intuitive; search bar
Easy plots and widgets make it easier to design a mission layout. Eg: Rover Surface Operations has a timeline that visualizes the data on the left
The export to JSON is useful for tinkering and prototyping

Severity Ratings

The following 1 to 4 rating scale is used to rate the severity of usability problems:

4: Unusable: The user is not able to or will not want to use a particular part of the product because of the way that the product has been designed and implemented.

3: Severe: The user will probably use or attempt to use the product here, but will be severely limited in his or her ability to do so.

2: Moderate: The user will be able to use the product in most cases, but will have to undertake some moderate effort in getting around the problem.

1: Neutral: No major usability issues detected.

Findings

The following five themes were discovered from affinity analysis of the usability test data. Below are the five categories along with the suggested recommendations.

Category 1 - Universal Commands and Hotkeys

1/2

Category 2 - In App Navigation

1/4

Category 3 - Operation Management Analysis

1/8

Category 4 - Saving and Editing of Layout

1/3

Category 5 - Understanding and Editing of Widgets

1/2

System Usability Scale (SUS) Results

SUS Score for OpenMCT is 74.6875

Open MCT software is graded as B (grade point: 3.0) according to The Sauro-Lewis Curved Grading Scale.

For reference: SUS ratings for other products are given below.

Reflections

Reflections

What went well?

Remote usability testing went smoothly thanks to screen-sharing and the tool being freely accessible on the web.

We gathered insights from a diverse set of participants who had something unique to say about mission control.

Weekly communication and feedback from the client helped us get feedback on our research methods as well as keep track of our schedules.

What was challenging?

Due to our project’s specific demographic, it took a lot of extra effort to recruit participants who were qualified and willing to participate in the study.

This research project provided an opportunity to learn how to recruit a particular niche of participants. The team had to adapt to a few real world setbacks and improvise to achieve the desired results. One of our study had to be stopped midway owing to the participant's personal emergency, another participant requested to withdraw his data from the study as he did not feel comfortable contributing to the project.

Conclusion

We presented our research findings to a team of designers, researchers and engineers from Blue Origin. The team was impressed with the process followed and findings of our research.

Through all of our usability studies, our participants posed a polarizing question.

"For a mission control tool, what matters more?

Customizing and setting up a display

Utilizing and observing the display"

Project Type

Duration

Team

My Role

Overview

Process

Identification of Focus Area

>>

Planning and Preparing Usability Study Kit

>>

Participant Recruitment and Usability Study

>>

Data Analysis and Presentation to client

Study Proposal

Purpose

Research Questions

Participants

Recruitment

Profiles

Usability Study

Heuristic Evaluation

Method and Test Environment

Study Kit

Data Analysis

What is working well?

Severity Ratings

Findings

System Usability Scale (SUS) Results

Reflections

What went well?

What was challenging?

Conclusion

Back to Home