Investigating IA Research

Bentley University User Experience Center

Introduction

This research project was conducted with a clear focus on presenting the results at Bentley University’s Face of Finance conference, a conference geared towards financial user experience. One other senior consultant and I brainstormed ideas before deciding on an information architecture (IA) research. We were supposed to collaborate on the project, but due to external circumstances, I ended up doing almost all of the work with help from our research associates.

Research objectives

The primary goal of the research was to demonstrate the value of conducting early information architecture research through tree testing. In addition, we also wanted to demonstrate the additional value that we could bring to their projects. Therefore, we analyzed three community banks using competitive testing, and used what we learned to create our own IA. By doing so, we took a leap of faith in our abilities. Would we be able to create an IA that outperformed the banks’ existing IA?

""

Introduction to Tree testing

Tree testing is a methodology for analyzing information architecture. You present the participants with a structure similar to a computer’s folder system (see below) and ask them where they would find specific information. In other words, it is a task-based methodology used to evaluate the information structure.

Image of nested folders

Building up the test

After selecting the three banks (Rockland Trust, Cambridge Trust, and Blue Hills Bank), we looked at their website and reverse engineered their information architecture into a tree testing structure. Thereafter, we had to choose tasks that would be a fair comparison since different IAs may be optimized for different tasks.

To identify fair tasks, we started by conducting informal interviews and followed these with a large-scale survey. The purpose was to uncover what information users want when they are searching for a new bank. The survey provided us with an understanding of how important each piece of information/task was relative to one another. Our initial goal was to use the top ten tasks identified as most important to the users. However, not all banks supported the top ten tasks in their primary navigation so we had to use a few lesser-valued tasks as well.

""

Data

Once we had all the IA structures and tasks identified, we launched the tree test through Optimal Workshops Treejack. Participants were recruited through social media and other public channels. We got a good number of respondents who participated in the study. In addition, we were able to keep most of the participants as we cleaned up the data.

""

Analysis

During the analysis, we primarily focused on task success. Since the task survey had provided an understanding of how important the different tasks were compared to one another, we used this understanding as a factor for the total average score. For example, if participants in the survey had ranked one task very important, and the bank’s IA performed well on the task, it would weigh heavier than if the participants ranked it as less important.

From this weighted score, we were able to see a more accurate depiction of how each of the bank’s IA performed. With this analysis, two banks switched positions in their ranking. More specifically, one of the banks had an IA that worked well for lower valued tasks but not so well for higher valued tasks.

""

Our Information Architecture

We analyzed the structures in different ways, gaining insight that we used to create our own information structure. With the new structure created, we launched a new tree test study with the same tasks as the other study. Recruiting was carried out in a similar way, but using other venues to eliminate the chance of ending up with the same participants.

""

Outcome

Our new information structure outperformed all the other bank structures and beat the best bank’s IA with 11 percentage points. By doing this, we effectively demonstrated value in the methodology and the added value we could bring to a project.

""