AI Training Assistant

Summary

Problem

Advisors and operations staff had plenty of training resources available, but they often experienced difficulty locating the necessary training materials at any given time.

Solution

Provide an AI to assist advisors and operations staff with finding training materials. The AI will be accessed using a link in the main system and SSO them into the AI application.

Outcome

After researching our riskiest assumptions about how to access the AI assistant, there was a clear need to pivot the scope to promote a better adoption of the AI assistant.

Problem

Pain Points

Provided training center is vast but difficult to search
Many keywords must be attempted to find needed material

Solution Values*

Easier search functionality given the AI's knowledge base
Only have to prompt the AI once to receive relevant results

For this research, the ultimate recommendation was to pivot away from the currently defined scope. See the "Outcomes" section below for more information.

Process

My Contribution

Researched literature for AI best practices and documented risks associated with the current development plan. Translated the risks into assumptions and identified our riskiest assumptions of the development plan. Designed structured interviews and defined validation criteria. Developed scripts and protocols, including randomized ordering of the mockups. Tracked interviewee responses, interviewer notes, and validation determinations. Compiled insights, quotes, and participant-generated ideas. Presented recommendations based on the research findings.

Literature Review

After reviewing more than 10 articles related to AI and best practices, the following cautions were documented:

Not positioning the AI within the context in which it is trained can created confusion and frustration for users
Without onboarding or information of the AI's limitations the confusion and frustration can be exacerbated
Not training the AI on the context of an user/user's demographics can hinder the validity of the AI's responses and create distrust

Lack of data integrity of the materials informing the AI can lead to distrust (and feedback indicates the Training Center is outdated)
Distrust/credibility/falsity/integrity may be impacted due to unclear ownership and maintenance (e.g., updated materials) protocols
Vague and undefined error handling may also influence perceptions from users about credibility
Not knowing the prompts users will create can cause issues if the AI is equipped to handle either statement- or question-based prompts
Inability to modify a "base prompt" relinquishes the ability to modify the personality and relationship users have with the AI

Assumption Mapping

The above cautions were translated into assumptions and mapped by risk and confidence (evidence). The top right quadrant consisted of the assumptions we had the least amount of evidence to support and were the riskiest, or most likely to result in an unsuccessful launch.

Structured Interview Questions

General Questions

Where and when do you seek resources currently? Describe the last time you needed help performing a task in the past week.
How often do you visit Salesforce? The Training Center? Follow-up: How do you currently access the Training Center? (Walk me through how you get there.)

AI Questions

Have you ever used generative AI before? (Clarify if needed: Have you used tools such as ChatGPT, even in your personal life?)
In what ways would you find the AI to be helpful? (Alt: In what ways would you use the AI?)
What are examples of things you would have the AI answer? (Alt: Tell us what you would say to the AI.)

Mockups (see right)

Training Center: How might you interact with the AI in this context? (Alt: In this context, what would you expect the AI to be able to help you with?)
Salesforce: How might you interact with the AI in this context? (Alt: In this context, what would you expect the AI to be able to help you with?)
Own application: How might you interact with the AI in this context? (Alt: In this context, what would you expect the AI to be able to help you with?)
Of the concepts shown, which context for the AI makes the most sense to you? Why?
Would you want access to the AI while you’re doing a task? Where?

Wrap-Up Questions

How important is having visual elements (photos and videos) in a response from the AI?
What all systems in the tech stack do you use?
Have you completed the AI information training for advisors?

Access to the AI within the Training Center

Access to the AI within Salesforce

Link to the standalone app (right)

AI landing page from link (left)

Findings

Results

Validation result for each participant

Key Insights

Key insights related to the riskiest assumption

Other Insights

Other insights gleaned from the research

Participant-Generated Ideas

Innovative ideas generated by the participants

Solution

Given participants overwhelmingly validated the necessity of the AI to be located in the Training Center because the AI would be trained on this data, the need to pause, pivot, or kill the release of the AI was evident. Users perceived high value in having an AI assistant, but the actual functionality and capabilities of this assistant were undetermined and varied from one user to another. The research influenced the decision to pivot, but the findings revealed multiple recommendations about possible directions for the AI.

Possible Directions & Impacts

Locate the AI in the Training Center to help users understand what the AI can answer / what data the AI is trained on

Pros: helps users immediately understand the extent of the AI's capabilities; improves the search issue
Cons: might be more complicated than expected; the AI only serves as an "enhanced search function"'; the Training Center was scheduled to move locations within the next 6-12 months, requiring more data training for the AI and/or changing the AI's location

Locate the AI in Salesforce and include UI elements that convey what data the AI is trained on

Pros: easily accessible and visible within the users' daily system
Cons: requires additional design work to help user; could still leave users wondering what data the AI can help with; might be confusing if Salesforce's AI was implemented in the future

Locate the AI in its own application and include UI elements that convey what data the AI is trained on

Pros: allows the AI to have a permanent location; enables endless possibilities for new development (unrestricted by the AI's location)
Cons: requires additional design work to help user; could still leave users wondering what data the AI can help with; compels users to remember the AI is available; positions the AI as an "assistant" without the AI having all available resources, (internet) knowledge, and user information

Change the data the AI will be trained on from the Training Center to Salesforce data

Pros: perceived by participants to be the most valuable data for the AI to assist with; provides a higher perceived value for users; solves information architecture issue that cause difficulty finding information in Salesforce
Cons: more difficult to train the AI on this data; might be redundant if Salesforce's AI was implemented in the future

Include more data in the AI's training database, such as the marketing library and informational investment content

Pros: better reflects the future state of the Training Center
Cons: requires training time, data integrity assurance, and assessment of ability to train the AI on this data; obligates the AI's release to potentially be paused until the new system for all these resources is built and released

Outcomes

1	Successful validation of the riskiest assumption and clear suggestions that inform the scope.

2	The research's findings revealed information about other assumptions, in addition to the riskiest assumption.

3	Evidence from the research supported the need for the business to pivot from the original opportunity.

Recommendations

Conduct an audit of the Training Center data to ensure accuracy and data integrity
Research Salesforce's AI capabilities to discover if this AI might be more valuable
Postpone the AI's release and either (1) implement the AI with the new system for all resources and relevant data or (2) implement the AI in it's own application with more in-depth features and data (possibly including the user's demographics) If release timeline cannot be changed, locate the AI in Salesforce and include UI elements that convey what data the AI is trained on.

Reflection

Overall, the outcomes from this research provided immense value for changing directions. In addition, the research generated more ideas about how to implement an AI for advisors and operations staff. One improvement could have been to conduct generative research before formulating the scope.

Back to Projects