AI Training Assistant

Summary

Problem

Advisors and operations staff had plenty of training resources available, but they often experienced difficulty locating the necessary training materials at any given time.

Solution

Provide an AI to assist advisors and operations staff with finding training materials. The AI will be accessed using a link in the main system and SSO them into the AI application.

Outcome

After researching our riskiest assumptions about how to access the AI assistant, there was a clear need to pivot the scope to promote a better adoption of the AI assistant.

Problem

Pain Points

Provided training center is vast but difficult to search
Many keywords must be attempted to find needed material

Solution Values*

Easier search functionality given the AI's knowledge base
Only have to prompt the AI once to receive relevant results

For this research, the ultimate recommendation was to pivot away from the currently defined scope. See the "Outcomes" section below for more information.

Process

My Contribution

Researched literature for AI best practices and documented risks associated with the current development plan. Translated the risks into assumptions and identified our riskiest assumptions of the development plan. Designed structured interviews and defined validation criteria. Developed scripts and protocols, including randomized ordering of the mockups. Tracked interviewee responses, interviewer notes, and validation determinations. Compiled insights, quotes, and participant-generated ideas. Presented recommendations based on the research findings.

Literature Review

After reviewing more than 10 articles related to AI and best practices, the following cautions were documented:

Not positioning the AI within the context in which it is trained can created confusion and frustration for users
Without onboarding or information of the AI's limitations the confusion and frustration can be exacerbated
Not training the AI on the context of an user/user's demographics can hinder the validity of the AI's responses and create distrust

Lack of data integrity of the materials informing the AI can lead to distrust (and feedback indicates the Training Center is outdated)
Distrust/credibility/falsity/integrity may be impacted due to unclear ownership and maintenance (e.g., updated materials) protocols
Vague and undefined error handling may also influence perceptions from users about credibility
Not knowing the prompts users will create can cause issues if the AI is equipped to handle either statement- or question-based prompts
Inability to modify a "base prompt" relinquishes the ability to modify the personality and relationship users have with the AI

Assumption Mapping

The above cautions were translated into assumptions and mapped by risk and confidence (evidence). The top right quadrant consisted of the assumptions we had the least amount of evidence to support and were the riskiest, or most likely to result in an unsuccessful launch.

Structured Interview Questions

General Questions

Where and when do you seek resources currently? Describe the last time you needed help performing a task in the past week.
How often do you visit Salesforce? The Training Center? Follow-up: How do you currently access the Training Center? (Walk me through how you get there.)

AI Questions

Have you ever used generative AI before? (Clarify if needed: Have you used tools such as ChatGPT, even in your personal life?)
In what ways would you find the AI to be helpful? (Alt: In what ways would you use the AI?)
What are examples of things you would have the AI answer? (Alt: Tell us what you would say to the AI.)

Mockups (see right)

Training Center: How might you interact with the AI in this context? (Alt: In this context, what would you expect the AI to be able to help you with?)
Salesforce: How might you interact with the AI in this context? (Alt: In this context, what would you expect the AI to be able to help you with?)
Own application: How might you interact with the AI in this context? (Alt: In this context, what would you expect the AI to be able to help you with?)
Of the concepts shown, which context for the AI makes the most sense to you? Why?
Would you want access to the AI while you’re doing a task? Where?

Wrap-Up Questions

How important is having visual elements (photos and videos) in a response from the AI?
What all systems in the tech stack do you use?
Have you completed the AI information training for advisors?

Access to the AI within the Training Center

Access to the AI within Salesforce

Link to the standalone app (right)

AI landing page from link (left)

Findings

Results

Assumption: Advisors will find the AI more useful and relevant if it's located in the Training Center, as it provides them with better context on how to use it.

Participant 1

Participant 2

Participant 3

Participant 4

Participant 5

Participant 6

Participant 7

Participant 8

Key Insights

Location of the AI's access point influences the users' perception of what the AI can help with.

"I definitely know in [the Training Center example], I would assume that it's only trained on a center part"
"To know what kind of question to ask, it's [the Training Center example] one"

When the AI was presented outside the Training Center, 100% of participants assumed the AI would have information beyond the Training Center.

When the AI was located in Salesforce, users said they would ask:

"What are my outstanding tasks?"
"Find me five households that have more than $5 million."
"Can you show me where I can find all of my open opportunities that are on a specific stage?"
"Give me a list of upcoming and overdue client reviews."
"Take me to client [X]."
"In [an object],...what's the best way to [do this task]?"

When the AI was located in its own app, users said they would ask:

"I might talk about my feelings a little bit."
"That's really tough...I guess I'd be curious what information he'd be gleaning from this point."
"I would be assuming it is completely comprehensive and holistic with all kinds of stuff."
"From a usability standpoint, that would be a little more frustrating to me."

Other Insights

There were several references made about the AI knowing context, such as the user's role and/or their current screen.
"I don't care about [a certain custodian's] forms, so please don't give them to me."
Example prompts for the AI were both statement- and question-based.
"I want...", "I need...", "Show me...", "Find all the...", What are...?", Where can I...?"
Participants value quick and accurate information, making videos and links less valuable.
There was a preference for the AI to be easily and quickly accessible, such as within Salesforce (i.e., their daily workflow).
Participants preferred walkthrough and summary responses from the AI, with the ability to dive deeper or clarify.
Participants rely on the Training Center and other team members before reaching out to the service team or submitting a case.
More experienced partners typically us the Training Center less, although they would still use it for specific or obscure information.
Participants assumed a standalone app for the AI would have larger (internet-reaching) knowledge.
Participants seldomly mentioned accessing previous information given to them by the AI.
There is a perception that the Training Center has a good amount of information, but the information is sometimes outdated.

Participant-Generated Ideas

Creating a case when the AI doesn't quite answer what you need or when you need more help
Showing the top 10 queries the AI answers from all users
Holistic responses for all things involved in a particular task (x2)
Reminder of action items needed when performing a certain task (x2)
Creating a report in Salesforce based on certain Salesforce information a user asks for (x2)
Having the AI accessible in multiple systems and "moves" from system to system with you (x3)
Helping fill out forms or create a DocuSign document
Prep a wealth plan for client reviews (currently takes about 1 hr to prep)
Ability to convey dynamics of a family or household

Solution

Given participants overwhelmingly validated the necessity of the AI to be located in the Training Center because the AI would be trained on this data, the need to pause, pivot, or kill the release of the AI was evident. Users perceived high value in having an AI assistant, but the actual functionality and capabilities of this assistant were undetermined and varied from one user to another. The research influenced the decision to pivot, but the findings revealed multiple recommendations about possible directions for the AI.

Possible Directions & Impacts

Locate the AI in the Training Center to help users understand what the AI can answer / what data the AI is trained on

Pros: helps users immediately understand the extent of the AI's capabilities; improves the search issue
Cons: might be more complicated than expected; the AI only serves as an "enhanced search function"'; the Training Center was scheduled to move locations within the next 6-12 months, requiring more data training for the AI and/or changing the AI's location

Locate the AI in Salesforce and include UI elements that convey what data the AI is trained on

Pros: easily accessible and visible within the users' daily system
Cons: requires additional design work to help user; could still leave users wondering what data the AI can help with; might be confusing if Salesforce's AI was implemented in the future

Locate the AI in its own application and include UI elements that convey what data the AI is trained on

Pros: allows the AI to have a permanent location; enables endless possibilities for new development (unrestricted by the AI's location)
Cons: requires additional design work to help user; could still leave users wondering what data the AI can help with; compels users to remember the AI is available; positions the AI as an "assistant" without the AI having all available resources, (internet) knowledge, and user information

Change the data the AI will be trained on from the Training Center to Salesforce data

Pros: perceived by participants to be the most valuable data for the AI to assist with; provides a higher perceived value for users; solves information architecture issue that cause difficulty finding information in Salesforce
Cons: more difficult to train the AI on this data; might be redundant if Salesforce's AI was implemented in the future

Include more data in the AI's training database, such as the marketing library and informational investment content

Pros: better reflects the future state of the Training Center
Cons: requires training time, data integrity assurance, and assessment of ability to train the AI on this data; obligates the AI's release to potentially be paused until the new system for all these resources is built and released

Outcomes

1	Successful validation of the riskiest assumption and clear suggestions that inform the scope.

2	The research's findings revealed information about other assumptions, in addition to the riskiest assumption.

3	Evidence from the research supported the need for the business to pivot from the original opportunity.

Recommendations

Conduct an audit of the Training Center data to ensure accuracy and data integrity
Research Salesforce's AI capabilities to discover if this AI might be more valuable
Postpone the AI's release and either (1) implement the AI with the new system for all resources and relevant data or (2) implement the AI in it's own application with more in-depth features and data (possibly including the user's demographics) If release timeline cannot be changed, locate the AI in Salesforce and include UI elements that convey what data the AI is trained on.

Reflection

Overall, the outcomes from this research provided immense value for changing directions. In addition, the research generated more ideas about how to implement an AI for advisors and operations staff. One improvement could have been to conduct generative research before formulating the scope.

Back to Projects