Pilot.ly- Usability assessment

Usability/UX Research

UX Research | Team | 14 weeks


This was a project for my course SI622 Needs Assessment and Usability Evaluation. We worked with a company called Pilot.ly on its content creator dashboard to improve its usability and evaluate what the needs of potential users were. I was part of a team of 4 and my primary role was managing the client contact and also setting up interviews and user tests.

Design Problem

Pilot.ly needed to ensure that it’s new platform facing the video content creators was catering to their needs.

Design Solution

Usability evaluation, interviews and other research methods were used to evaluate if the needs of the target population were being met.

Design Process

The process followed something like this-

Interaction Map


Interaction Map

Mapping the interaction flow (a diagram of the system’s reaction to user input). This exercise helped us learn the system’s workflow and how a potential user would use it to accomplish a task.

User Interviews, Personas and Scenarios

img103img105  img107

Personas from user interviews

We interviewed five people- two remotely and three in-person. They were mostly video content creators who were potential Pilot.ly users. We gained insights into what users might want and also what they were currently using to perform the same function. We created three personas based on the interviews we conducted and created Scenarios for the three of them to have real needs and concerns and to have an understanding of how a person’s background and surrounding affect their behavior with products.

Comparative Analysis

We shortlisted some other products which performed the same functions as Pilot.ly.  We divided them across categories such as: Direct Competitors/ Analogous/ Indirect Competitors based on what their primary functions were. This gave us a sense of what features are already being provided by other products and how well they are performing those functions.



Some of the survey results

We conducted a survey for gaining insight into the user behavior and what they might want from a product such as Pilot.ly. We chose convenience sampling and received 29 responses from video content creators across America and also internationally. This gave us an understanding that the potential users are mostly young and have less than 5 years of experience in the industry. They are relatively tech savvy and need an intuitive application for providing them feedback about their content.

Heuristic Evaluation

Finding Priority Level Comments
Appropriate feedback is missing for certain tasks, and call to action is not clear throughout the system.  (1) Medium
The user is easily able to orient him or herself within certain task processes. (2) N/A This is something the system does well, so no recommendations were made.
Insufficient resources available to the user for seeking help and support. (3) High Client specifically asked us to assess the help features on the dashboard.
The use of visual elements and language makes the website inconsistent. (4) High Concerned about new users
The website does not provide enough memory aids for users. (5) Medium
The user does not have a complete idea of the workflow. (6) Low
Most tasks can be completed without completing the required or correct subtasks.



Finalized Concise Findings from Heuristic evaluation

We performed Heuristic evaluation on Pilot.ly to understand how well it followed the best practices of Usability through Neilsen’s 10 Heuristic pointers which encompass most usability problems that a digital system can have. We performed this exercise individually and divided each heuristic into applicable pointers for Plot.ly and then, rated each heuristic based on a severity scale. This gave us a very clear picture of the usability of the system. We then came together as a group and came to a conclusion about the prioritized list of problems.

Usability tests

ut1Result of Post-test question “Would you use the system frequently, at least once in 2 days?”

After a session with the client, we found that the users that the client also wanted to target were video content creators with research intent. We conducted usability tests with three video content creators cum market researchers and two in-person tests and found a lot of issues and features that Pilot.ly can fix and make the product more useful and targeted to the users they want to potentially attract. Our final recommendations were finalized from questionnaires before and after the test.

Final Video