Lumos
Optimizing Lumos: Transforming the Data Analytics Experience
.png)
Overview
Project Lumos is a platform that offers a rich set of dynamically configurable services to increase developer productivity, improve customer experience, and derive customer insights. It includes services such as feedback service, guided tours, web analytics, announcements, and error tracker. The services can be added or configured dynamically using a self-service user interface after that with zero code change.
Lumos facilitates rapid adoption, avoids vendor outsourcing, and enables non-coder admins to configure features while improving developer productivity.
Team
2 Research Leads, 2 Researchers, 1 Product Manager, 1 Engineering lead
Role
Lead Researcher
Impact
Optimize Lumos for better features, increased telemetry adoption, improved usability, and enhanced satisfaction.
Problem Statement
The Lumos team approached us with a set of questions in mind. Despite Lumos being an in-house developed analytics solution, a significant number of teams in VMware rely on third-party alternatives for their needs. Their objectives include enhancing Lumos' adoption within VMware, improving the product for existing users, and understanding the factors leading some users to initially choose Lumos but eventually opt for alternative solutions. In essence, the team aims to unravel:
-
What pain points do current users have
-
Why do potential users choose other products over Lumos
-
What factors are dealbreakers to users when considering integrating analytics solutions
Desk Research
We started with a review of the repository for relevant internal projects and also explored our competitor's current workflows and features. This helped refine our objectives and identify what we needed in terms of data.
Research Plan
First and foremost, we initiated the research process by developing a comprehensive research plan, which was then reviewed and confirmed with the team. Our primary objective was to establish a clear understanding and alignment on key aspects, including the overarching goal of the research, the specific questions we aimed to address through the collected data, the chosen research methods, a detailed plan with specified dates, the criteria for participant recruitment, and our analysis approach. This ensured that everyone involved was on the same page and committed to a unified direction for the research endeavor.
Research Methods
To achieve the objectives of our research, we initially adopted a two-phase approach, which subsequently evolved into three phases.
Phase One: Survey
Conducting a survey was instrumental in gathering both quantitative and qualitative data, providing a robust foundation for our research. Additionally, it played a pivotal role in recruiting participants for the upcoming 1:1 interviews.
Phase Two: 1:1 Interviews
In this stage, we engaged in in-depth interviews with current Lumos users and individuals utilizing third-party tools. This allowed us to delve deeply into the strengths and weaknesses of Lumos, as well as understand the dynamics of third-party solutions used by VMware employees.
Recruitment
With a clear understanding of the problem statement and the research questions guiding our inquiry, our next step involved recruiting two distinct user groups: those currently utilizing Lumos and those relying on third-party solutions for their analytics requirements.
Collaborating closely with the contracts and procurement team, we identified teams with existing contracts for various analytics software options. Then, we initiated communication with the designated contacts linked to each purchase order, aiming to gather details about team members actively utilizing the applications. The third-party users list was then combined with the Lumos users list provided by the Product Manager.
Phase One - Survey
We sent the survey to 370 Internal telemetry users, which included Lumos users and users with accounts on other approved telemetry applications. We had a <10% response rate. We had 36 responses, which enabled us to recruit 8 users for 1:1 interviews ( 4 current Lumos users, 4 that use other third-party applications)
Setting up a survey to be sent out to Lumos users and third-party analytics users.
Survey responses helped us recruit the right participants for the 1:1 Interviews.
Research Protocols
We devised two distinct protocols—one tailored for current Lumos users and another for users of third-party analytics solutions. Both protocols encompassed questions relevant to both audiences. The protocol for current Lumos users aimed to uncover insights into their existing challenges, identify missing features, and explore whether these users would opt for Lumos if presented with alternative solutions. The third-party user protocol aimed to understand their potential transition to Lumos, facilitate a smooth switch, explore reasons for choosing other solutions, assess awareness of Lumos and its benefits, and identify critical factors influencing their choice and deal-breakers.
Affinity Mapping
We established a project space in Dovetail and implemented a tagging system, anticipating that it would significantly simplify the affinity mapping process.
Implemented a tagging system in Dovetail, streamlining the process of creating affinity maps for our interviews.
Recordings were incorporated into participant notes, allowing us to tag the interviews seamlessly using our designated tagging system.
Sensemaking Workshop
After tagging the interviews it was time for the affinity mapping with the team. We wanted to share where we are with the research and invite the team to conduct the affinity mapping together. We created a board in Miro with information on what we know so far and we laid down the research question that we have and wanted to answer.
We granted the product team access to Dovetail and assigned each team member specific users and recordings. Their task was to review notes, tags, and recordings, and contribute their findings and insights to the board in response to the research questions.
The Sensemaking workshop not only facilitated affinity mapping of the interviews but also highlighted gaps in our research. The most significant gap identified was the need for improved representation from business units (BUs) to enhance the depth of our research. This realization prompted the initiation of research phase three.
Facilitated a Sesnemaking workshop with the product team.
Phase three: Follow-up + supplementary research
As it was identified from the sensemaking workshop, we had to have a better representation of our business units, especially among the business units that greatly use telemetry. We recruited four more participants for our supplementary research, allowing us to uncover additional insights and bridge existing gaps.
Analysis Process & Findings
For the quantitative analysis process, we used Spearman correlation tests and descriptive statistics. For qualitative: affinity mapping and nested thematic grouping.
Through the three phases of research, we were able to identify many patterns and insights among our users. The most critical were:
-
31% churn rate: participants who found Lumos difficult in the beginning don't convert to regular users, contributing to churn. This was due to a lack of training, easy onboarding, and a steep learning curve as a reason to walk away.
-
Must have features missing identified: watch recorded sessions, track usage of features, heat maps
-
Must have for user experience missing identified: easy data consumption/ease of use (Data standardization, customization of reports, aligned UX to users’ mental models), onboarding and early training (Training materials, tooltips, how-to guides), performance (Stability with large-scale data, decreased loading times)
Leading the Ethnographic Research on a Top Priority Feature
With the initial research, we were able to uncover that Lumos needs to focus on the following 4 areas: Onboarding and training, Awareness, Performance and Ease of use, customization, and interaction. To identify on a more detailed level where users struggle in practice in these four areas, together with the team decided to conduct a contextual inquiry. To uncover these specific pain points and needs, we had to conduct Contextual Inquiry with Lumos' primary users: software engineers product managers, and product designers. As the “In product feedback” service was identified as a critical P0 zero service, we decided to focus on conducting Contextual Inquiry on how users onboard it, configure and then use it.
RACI Chart & Timeline Tables
As the research lead for the contextual inquiry, collaborating with three additional researchers, I recognized the significance of establishing a solid foundation to ensure the smooth and timely execution of our research. To kickstart this process, I initiated the creation of a RACI chart. This chart effectively distributed responsibilities and accountability among the team, providing clarity on who was responsible, accountable, consulted, and informed throughout the various phases of our research.
We also implemented a live team availability table, where we could track team members' schedules. This proved invaluable in enhancing our planning and ensuring more effective coordination of our work.
.png)
RACI Chart that helped the team know our tasks in every stage of the research process.
Team availability table where we would track team members schedules.
Survey to recruit the right participants
To conduct a contextual inquiry for the In-Product Feedback (IPF) service, we needed to enlist both primary users, such as product designers and product managers and those responsible for onboarding their product to the IPF, namely the software engineers. To ensure a targeted participant selection, we designed and distributed a survey to 363 Lumos users, helping us identify and recruit the specific participants required for the contextual inquiry.
Expectations vs. Reality
From our survey responses, we discovered that product designers are currently not utilizing the In-Product Feedback (IPF) tool. Instead, it is predominantly being used by software engineers for onboarding and by product managers, who serve as the primary users. Based on this insight, we decided to initially recruit product managers and software engineers.
However, we plan to expand our participant group in future studies to include product designers, the newest role integrating Lumos into their workflows.
Additionally, we observed a limited pool of participants expected to have active work on Lumos IPF that we can observe. To address this, we opted to conduct 1:1 interviews with users who have either used or been onboarded to IPF within the past month.
In total, we conducted three contextual inquiries (1 Product Manager, 2 Software Engineers) and three 1:1 interviews (1 Product Manager, 2 Software Engineers).
Affinity Mapping
A significant challenge arose when setting up the tagging system in Dovetail, especially with four researchers involved this time, three of whom were responsible for tagging the interviews. This led to the emergence of different tags for the same interpretation. To tackle this issue, we initiated two-hour debrief sessions every Thursday. During these sessions, we thoroughly reviewed the tags in our individual buckets and merged those with similar meanings.
As we delved deeper into the data, we recognized the need for both high-level insights and granular findings. Consequently, we developed a highly successful tagging system that not only provided a unified approach but also established a cohesive tagging system for all researchers involved.
The new improved tagging system, allowing to more than one individual to tag, keeping the same meaning.
The new improved tagging system that allowed us to derive insights both on a high-level and granular level.
Findings
The contextual inquiry was a huge success and helped us uncover many areas of needed improvements. Findings were split between two primary areas:
-
Feature Enhancements ( need for meta-driven segmentation, Immediate (click-through) access to analytics when reviewing IPF reports is the next step, Report templates and dashboards in both IPF and Web analytics also high priority)
-
Improving the User Experience ( Extreme cognitive challenges in workflows, due to feature discoverability issues and disruptions, Tech-heavy and domain-specific terminology and tasks hindering progress, Insufficient system feedback causing delays and inefficient workflows)