Threat Stack

After much debate about the strategic direction of the product, Threat Stack decided to re-commit itself to the platform's fledgeling Vulnerabilities feature. A complete redesign was commissioned by company leadership. As the sole UX Researcher, this was no easy task. We now had to take a feature that was originally designed by developers, and make it user centered.

We began the Vulnerabilities initiative with a design sprint. Bringing together SMEs from Engineering, Sales Engineering, Marketing and Customer Success, we explored the current state of the feature and ways in which it could be improved, eventually settling on an initial design direction.

Key Takeaways

• The current backend accuracy of Vulns is unreliable
• The new design should employ a CVE-centric approach
• Increased support for other OSes is needed
• Information should be contextual to the user's environment
• Give the vulnerability a status
• Communicate any change in vulnerability status
• A JIRA integration would be a great addition

Now that we had learned from our internal stakeholders and produced an initial concept, we set out to conduct user interviews. The goals was to simultaneously conduct generative research, while obtaining feedback on the first iteration of the wireframes. Eight participants from six customer organizations took part in the interviews.

"I don't use Vulns much - too many false positives"

"I need to at least take notes and track my vulns"

Key Findings

Active Vulnerabilities Tab

• All participants either did not see or were confused by the Queue tab
• The OS score is paramount; the CVSS score is not considered by most to be a single source of truth
• It was validated that information should be contextual to the user's environment
• Half of the participants were confused by seeing multiple packages with a CVE
• Participants indicated that filters should be added to the Active Vulnerabilities tab

Suppressed Vulnerabilities Tab

• Suppressed Vulnerabilities should include date suppressed, who suppressed and reason for suppression, in addition to summary card information
• Participants indicated that filters should be added to the Suppressed Vulnerabilities tab
• All participants stated that they would like to be notified when the blast radius of a suppression has increased
• Participants desired to be able to suppress at the server level
• Half of participants were not comfortable with Threat Stack auto-suppressing vulnerabilities 
• It was validated that a JIRA integration would be incredibly beneficial

Detailed View Page

• Half of participants remediate vulnerabilities by AMI
• Half of participants did not know what a Threat Stack tag was
• Participants overwhelmingly wanted to be able to filter within the Infrastructure Impact section
• Participants expressed a desire to add notes and an indicator that a note had been added

Other

• 2/3 of participants do not use the Vulnerabilities feature due to backend accuracy issues
• Nearly all participants liked accessing the feature via it's new position in the left-side navigation menu
• Nearly all participants wanted the ability to export information
• Customers need to be able to export over a given time period for audit purposes
• An additional tab is needed to illustrate vulnerabilities that are no longer detected
• All participants stated that usage of the feature would increase if backend accuracy was improved and the new design was implemented

With key findings in hand, I spent time with the designer to create user flows and ideate solutions.

After ideating and updating the design, we conducted moderated usability testing, with a new cohort of five customers.

Key Findings

Active Vulnerabilities Tab

1. Participants found the horizontal CVSS bar to be distracting and felt that it did not provide any value
   1. Remove the horizontal CSSS bar from the design
2. Participants were confused as to how to sort the data when prompted
   1. An affordance needs to be added to indicate that the columns are sortable
3. Participants indicated that they would like more detailed information while viewing the summary card
   1. Add the MITRE description on hover to the summary card
4. The term Blast Radius was not initially understood by all. However, those participants that did not initially understand the term, realized the meaning after giving it some thought and loved the term. 
   1. Provide a tooltip explaining Blast Radius
5. Static check boxes led to confusion, as some participants thought that it was mandatory to click on the check box, in order to be taken to the Detailed View page
   1. Individual check boxes should be made to appear on hover

Detailed View Page

1. Some participants had trouble remembering to scroll up to find the Actions button
   1. The Actions button should be larger and float in the foreground 
2. It was noted that after suppressing a vulnerability, there was no easy way to undo it
   1. A once-click un-suppress button should be added to the page 
3. Some participants struggled to find the filtering option, placed on the right side of the page
   1. Detailed View filtering should be placed on the left side of the page, as on the Active Vulnerabilities page
4. Many participants struggled to find the Add Note button on the right side of the page
   1. The Add Note button should be moved to the left side of the page, where the body of the note is

Inactive Vulnerabilities Tab

1. It was not clear to participants what the term Inactive Vulnerabilities meant
   1. Provide a tooltip explaining the term Inactive Vulnerabilities
2. Participants were confused by the fact that what was labeled as Blast Radius on the Active Vulnerabilities Tab, was then labeled Last Recorded Impact on the Inactive Vulnerabilities Tab
   1. The term Blast Radius should be used consistently and carried over to all pages

Other

1. Participants expressed frustration with the fact that after remediating a vulnerability in the current (original) experience, it is still listed as active for quite some time
   1. Provide an option for the user to initiate a scan, or provide some kind of confirmation that the vulnerability is no longer active
2. Participants expressed an eagerness to be able to share the CVE information with others on their team
   1. Add a sharing button to allow users to share the CVE via Slack, email or print
3. All participants expressed overall satisfaction with the design

Active Vulnerabilities Tab

Detailed View Page

Inactive Vulnerabilities Tab

After usability testing had concluded, I collaborated with the UX Designer and the Product Manager, to decide which features were essential to a first release and which could wait. We settled on scheduling two waves, with the first wave being robust.

The Vulnerabilities redesign initiative was a great endeavor. It was amazing to see how excited our user base was with the new design. We were repeatedly told that it was going to make a big impact on their lives. Unfortunately for our users, Threat Stack leadership unexpectedly decided to stop coding and place the initiative on hold, in order to focus on its intrusion detection capabilities.