Content uploaded by Edward Y Chang
Author content
All content in this area was uploaded by Edward Y Chang on Nov 25, 2023
Content may be subject to copyright.
Content uploaded by Edward Y Chang
Author content
All content in this area was uploaded by Edward Y Chang on Oct 25, 2023
Content may be subject to copyright.
Content uploaded by Edward Y Chang
Author content
All content in this area was uploaded by Edward Y Chang on Oct 24, 2023
Content may be subject to copyright.
1
Multi-Agent Reasoning with Large Language Models
for Effective Corporate Planning
Wen-Kwang Tsao
TrendMicro AILAB
spark_tsao@trendmicro.com
Abstract—Large Language Models (LLMs) have demonstrated
significant capabilities in natural language processing tasks.
In this paper, we explore the application of LLMs within a
business context. Specifically, we employ LLMs to devise a sales
strategy geared towards maximizing customer values (benefits
and satisfaction). This sales plan encompasses five iterative stages:
market landscape survey, customer profiling, product usage
analysis, sales strategy formulation, and crafting persuasive pitches
and materials. We leverage LLMs to supplement the limited data
available to the company, aiming to enhance the efficacy of
each stage and optimize KPIs, including the value-oriented sales
conversion and profitability. Due to confidentiality and trade
secret concerns, we blend artificial data with genuine data to
ensure customer anonymity and protect sales playbooks. Despite
these precautions, we effectively demonstrate our methodology of
harnessing LLMs to refine the sales planning procedure.
Index Terms—large language model, prompt template, Socratic
Synthesis.
I. INTRODUCTION
With the advent of large language models (LLMs) such
as GPT-4 [
1
,
10
,
11
,
12
], significant advancements have
been achieved in various natural language processing tasks,
including question answering, text summarization, and doc-
ument classification, among others. However, the potential
of LLMs in optimizing business intelligence efforts, such as
sales forecasting, product prioritization, inventory management,
and daily operational enhancements to improve company key
performance indicators (KPIs), remains largely unexplored.
This study focuses on applying LLMs to enhance value-
driven sales planning, as outlined in Section II. We use this
case study to illustrate how leveraging LLMs can improve
various sales process stages, including market segment identifi-
cation, customer profiling, efficient sales strategy deployment,
competitor analysis, and the creation of compelling sales
materials. We establish performance metrics for each stage
and employ LLMs to enhance these metrics, ultimately aiming
to increase profitability and improve the value-focused lead-to-
sale conversion rate.
Business intelligence augmentation requires the gathering,
analysis, and utilization of relevant sales data to formulate
concrete action plans based on the derived insights. Therefore,
the prerequisites for large language model (LLM) utilization
in this context can be distilled into three primary objectives:
1.
Data-Driven Insights Generation: LLMs can be leveraged to
analyze vast amounts of sales data, customer feedback, and
market trends, converting raw data into actionable insights
that can inform strategy.
2.
Augmented Decision Support: LLMs can be utilized to
offer real-time recommendations during sales planning
and execution, allowing for quick adjustments based on
predictive analytics and historical data.
3.
Enhanced Customer Engagement: LLMs can be employed to
craft personalized sales pitches, responses, and engagement
strategies, ensuring that customer interactions are both
relevant and compelling, thereby maximizing conversion
opportunities.
To realize these objectives, we propose SocraPlan, which
employs SocraSynth [
4
], an approach that amalgamates the
principles of “Socratic Synthesis.” SocraSynth promotes a
synergy between human moderators and LLM agents, fostering
the creation of profound questions and overcoming human
limitations in reasoning, validation, and assessment. A standard
SocraSynth ensemble [
7
,
8
] consists of both humans and
LLM agents, each championing distinct viewpoints. An agent
might be driven by LLMs like GPT-4 [
1
], LaMDA [
11
], or
Llama [
12
], ensuring a mix of diverse information sources and
perspectives. SocraSynth’s process divides into two principal
stages: generative and evaluative. During the generative phase,
LLM agents provide arguments and counterarguments, until a
conclusion emerges. The evaluative stage leverages a variety of
virtual judges, each supported by a distinct LLM, to impartially
appraise the discourse. Upon the conclusion of both stages,
human decision-makers can scrutinize the suggestions and
arguments to perform final decision making.
The deployment of multiple LLM agents in a debate context
is driven by a dual rationale [
3
]. Firstly, LLMs possess a re-
markable capacity for knowledge representation and discovery,
embodying a “polydisciplinary” character. This characteristic
blends diverse disciplines, often blurring traditional boundaries,
to yield fresh insights and solutions. Secondly, given the vast
expanse of knowledge within LLMs—potentially equivalent to
the expertise of hundreds of PhDs across various domains—
humans may encounter challenges in formulating prompts that
explore uncharted knowledge territories. Engaging LLM agents
in debates, where they formulate questions to challenge each
other repeatedly, can increase the likelihood of unearthing
information that was previously unknown to humans.
The contributions of this study are structured as follows:
Section II outlines the steps involved in creating a sales
plan for our value-drive sales approach. Section III illustrates
how SocraPlan, facilitated by SocraSynth, performs market
research, data collection, synthesis and analysis, and then
employs SocraSynth to conduct a debate to tailor existing
2
playbooks to for a potential customer. Section IV presents a
case study. Our evaluation using the Socratic method attests
that the storybook generated by SocraPlan is better in several
metrics. We conclude our study with final remarks in Section V.
II. VALUE-DRIVEN SAL ES
Most sales cycles, irrespective of the specific market or
industry, typically progress from identifying potential customers
to finalizing a sale. Although the details of each stage can differ
based on the industry, several core stages consistently manifest
in various forms across diverse sales processes.
1.
Market survey: Discerning and targeting potential market
segments, analyzing vendor offerings, evaluating market
dynamics, and identifying opportunities where customers or
clients may exhibit interest in the offered product or service.
2.
Customer profiling and engagement assessment: Gauging
the specific needs, preferences, and challenges of both
prospective and existing customers, irrespective of their
current vendor affiliations. This extends to understanding
their current usage patterns, unmet needs, and potential
value, followed by categorizing them based on alignment
and opportunity for engagement
3.
Competitor analysis: Assessing the strengths, weaknesses,
opportunities, and threats presented by competitors and
co-exist vendors within the identified market segments.
4.
Sales strategy deployment: Upon evaluating a prospect’s
interest and purchasing potential, apply the most fitting sales
“playbooks” to engage and prioritize these leads effectively.
5.
Crafting persuasive pitches and materials (proposal): De-
signing and presenting a solution tailored to the discerned
needs of the customer.
In the cybersecurity market, we believe that a value-
driven sales strategy is the most appropriate approach. This
conviction stems from the fact that defending against cyber-
attacks demands highly specialized skills, and the methods of
attack are continually evolving. To effectively address these
challenges, continuous engagement with customers is essential.
This involves understanding their usage patterns, providing
training to maximize product capabilities, and keeping them
informed about emerging threats. Such an approach maximizes
the benefits customers derive from our solutions.
In contrast to the volume-based business model, which
prioritizes quick deal closures and customer acquisition, the
value-based approach [
2
] places a strong emphasis on long-term
engagement and the cultivation of robust customer relationships.
Importantly, many significant business opportunities often arise
after the initial transaction.
Within this value-centric framework, the role of Large
Language Models (LLMs) takes on greater significance. LLMs
excel at aggregating vast amounts of data, synthesizing
knowledge, and fine-tuning sales strategies. They excel in
uncovering deeper customer needs and ensuring the delivery of
personalized and attentive service. Consequently, we introduce
SocraPlan, which leverages LLMs to initially summarize and
analyze historical data and search for up-to-date information.
Once all relevant data is collected, the computational objective
for the various sales stages revolves around optimizing key
Key Performance Indicators (KPIs), i.e., maximizing customer
values including satisfaction and benefits. This optimization
process utilizes the compiled information as input and yields
a refined and tailored sales strategy as its output.
III. GEN ER ATIV E PHA SE O F SAL ES ST RATE GI ES
SocraPlan uses SocraSynth to carry out three main tasks:
surveying the market landscape (Section
III-A
), analysing
product usage per customer (Section
III-B
), and devising
effective sales playbooks (Section
III-C
). As we discussed in
Section I, SocraSynth creates a combined team of humans
and LLM-based agents. This combination encourages in-
depth debates, aiming to discover information that might be
missed by only human analysis. One challenge for humans
is crafting questions when they don’t know what they don’t
know. However, SocraSynch, with its structured debates and
varied queries, fills this gap, fostering a strong collaboration
between humans and AI.
A. Market Study
Company GPT-4 (Pre-2022) Bard (2022/23)
Symantec (Broadcom) ✓
McAfee ✓
Palo Alto Networks ✓ ✓
Check Point Software ✓ ✓
Fortinet ✓ ✓
Cisco ✓ ✓
CrowdStrike ✓ ✓
FireEye (Mandiant) ✓
Sophos ✓ ✓
Trend Micro ✓ ✓
Kaspersky ✓
CyberArk ✓
Zscaler ✓
IBM ✓
Microsoft ✓
Broadcom ✓
Table I: Companies Listed by ChatGPT and Bard.
The impressive performance of LLMs on benchmarks like
MMLU, makes it unnecessary for smaller organizations to
develop their own document summarization tools. These LLMs
are not just large databases but also powerful tools capable
of summarization, question-answering, and translation. An
effective strategy would be to use several LLMs to get a wide
range of information sources and viewpoints. This approach
can deepen the analysis by checking sources against each other.
Prompt
#1
. Which are the leading players in the cybersecurity
industry as of the end of 2022 and Q3 2023?
To elucidate this point, we outline the prompts posed to
both ChatGPT and Bard in our market analysis, using the
cybersecurity market as an illustrative example.
As we promote our cybersecurity product, we frequently
encounter potential customers who have already implemented
multiple security solutions. To devise effective strategies, it is
crucial for us to gain insights into the prevailing cybersecurity
providers and their product offerings that have been widely
adopted by enterprises as of the conclusion of 2022 and
throughout Q3 2023.
3
Attack Names GPT4 2020-21 Bard 2022 Bard 2023
Log4j vulnerability - External Remote Services Initial Access
Lapsus$ ransomware group attacks - Ransomware Data Encrypted for Impact
Colonial Pipeline ransomware attack Phishing Ingress Tool Transfer Ransomware
JBS Foods ransomware attack Third-party Software Data Encrypted Ransomware
Kaseya ransomware attack Supply Chain Compromise Scripting Ransomware
Microsoft Hafnium hack - Exploitation for Privilege Escalation -
MGM Resorts hack - - Credential Dumping
SolarWinds Attack Supply Chain Compromise - -
Microsoft Exchange Server Vulnerabilities Exploitation for Initial Access Web Shell -
Increase in Ransomware Attacks Phishing, Spearphishing - -
Table II: Major Cybersecurity Incidents and the Attack Tactics as reported by different sources
Table I presents GPT-4’s and Bard’s list. GPT-4 provides
information up to January 2022, whereas Brad’s information
is up-to-date. Acting as a moderator, we query GPT-4 for the
oversight with Bard’s list, GPT-4 lists three reasons.
•
Zscaler, IBM, Microsoft, and Broadcom do stand out as
companies that are significant in the cybersecurity domain
and should have been mentioned.
•
Zscaler has been on the rise in recent years with its cloud
security offerings.
•
Broadcom’s acquisition of Symantec’s enterprise security
business is significant, and it’s worth noting that Symantec’s
business continues to operate under Symantec brand.
The cross-validation approach of SocraSynth between two
LLM agents provides significant benefits. In addition to
improving information retrieval accuracy, it illuminates the
dynamic cybersecurity market, highlighting details such as
mergers, acquisitions, and new entrants. Although we can
extract each company’s product profile, we have chosen not
to in this paper because of sensitivity concerns. In actual
implementation, we also integrate other knowledge databases,
not solely relying on LLM.
Prompt
#2
. Please provide their market share information
based on market segments of network, endpoint and cloud in
2022 and 2023. Additionally, any insights into the challenges
customers face with these solutions would be invaluable.
When we prompted GPT-4 for market share information
regarding top cybersecurity firms, it failed to provide any data
and suggested that we search for the information ourselves. In
contrast, Google Bard, backed by Google Search, generated a
comprehensive report. (Results omitted due to space limit.)
Prompt
#3
. List the major events or incidents in the cyberse-
curity sector during 2022 and 2023.
Bard offers detailed information on the cybersecurity attacks
that occurred in 2022 and 2023, providing a comprehensive
overview of recent incidents. In contrast, GPT-4 primarily
focuses on highlighting specific incidents from the years 2020
to 2021. Notably, Bard even includes information on the MGM
attack in October 2023, which coincides with the timeframe
of this paper’s writing.
A.3 Other Prompts
To accommodate the space limitations, we provide a list
of additional prompts that can be employed for conducting
market research:
1. “Provide an overview of the cybersecurity market trends.”
2.
“Highlight the main challenges faced by cybersecurity firms.”
3.
“Describe the emerging technologies or methodologies (ex:
AI) in the cybersecurity domain in 2022-23.”
Utilizing LLMs, we gained valuable insights. In practical
applications, this method will be enhanced by merging pro-
fessional reports, intelligence databases, and APIs with LLM
resources, offering a comprehensive market landscape analysis.
B. Data Collection and Preparation
Traditional sales playbooks, often deemed essential by
businesses, can become outdated due to their inability to keep
pace with dynamic market trends or adapt to distinct client
needs (e.g., new methods of cyber-attacks). Our methodology
addresses these challenges by seamlessly blending human
insight with the acumen of large language model (LLM) agents.
This approach is structured into three stages: data collection
and preparation, debate, and decision generation, collectively
referred to as the “D-D-D process.”
1.
Data Collection and Preparation: Data is pivotal for informed
decision-making, and its quality is paramount. We aim to
refine imperfect data by capitalizing on both human intelli-
gence and LLM capabilities. To elucidate our concept, we
use three data categories to explain our iterative preparation
process.
2.
Debate: Once a robust dataset is established, LLM-agents,
guided by human prompts, embark on formulating playbook
strategies. Our debate structure involves two LLM-agents
deliberating the potential risks and benefits of their sug-
gestions. This dialogical approach ensures that the agents’
recommendations are cross-validated by multiple LLMs, in
this case GPT-4 and Bard.
3.
Decision Generation: Post-debate, having navigated the
breadth of the decision landscape, we employ the Socratic
methods of reasoning [
5
] for the LLM agents to reach a
consensus proposal for human-executive evaluation. This
phase synthesizes AI-derived insights with the nuance of
human expertise.
To craft a potent sales strategy, our journey commences
with the acquisition of vital information. This encompasses
understanding the prospective customer, identifying suitable
product offerings, and developing an effective attack strategy
from a repertoire of sales playbooks.
B.1 Customer data:
This includes basic profiles, customer segments, size, rev-
enue, geographical location, loyalty measured in years, purchase
4
records, product utilization, login metrics, feature usage stats,
and support incidents.
B.2 Product data:
This covers target demographics, selling points, feature break-
downs, future development plans, and pricing structures. For the
anonymized cybersecurity company cyber-X, it offers a unified
security platform that provides the following capabilities:
1.
Threat investigation: This App provides a comprehensive set
of tools for investigating threats, including incident timelines,
threat context, and forensic analysis. This makes it easier
for security teams to identify the root cause of an attack
and take appropriate action.
2.
Workflow and automation: This App can be used to automate
a wide range of security tasks, such as incident response,
threat hunting, and vulnerability remediation. This can help
security teams to save time and improve their efficiency.
3.
Risk insight: This App provides a variety of risk insights
that can help security teams to identify and prioritize their
security risks. This includes insights into vulnerabilities,
threats, and attack surfaces.
4.
Threat intelligence: This App integrates with the backend
global threat intelligence network, providing security teams
with access to the latest threat information. This can help
security teams to stay ahead of the curve and proactively
protect their organizations from attack.
5.
Zero trust and security agent: This App supports zero trust
security and provides a unified view of all security agents
across the organization. This can help security teams to
improve their security posture.
B.3 Sales playbooks:
Cyber-X, like all companies, have a repertoire of sales
playbooks, which are comprehensive guides or strategies that
provide sales teams with a structured and organized approach to
selling products or services. Some are textbook like, and some
have integrated with past experiences. The goal of SocraPlan
is to tailor a customized playbook for addressing the candidate
customer to not only maximize the lead-to-sales conversion
ratio, but also enhance the engagement quality. Each playbook
provides detailed strategies, their benefits, applications, and
standard operating procedures (SOPs). An iterative method
helps pinpoint industry-specific terminologies, definitions, and
ambiguous logs. Here are six selected playbooks
Pa
to
Pf
,
that we will use in our case study. These playbook, as we will
show shortly, contain terminologies that an LLMs may not
understand. In the data processing stage, LLMs would actural
query the human moderator to provide definitions.
Pa.
New Client Acquisition Playbook for Cyber-X Compre-
hensive Protection: Strategies to attract potential clients,
highlighting Cyber-X’ extensive coverage and unique
selling points.
Pb.
Cross-Selling/Upselling Playbook: Focuses on understand-
ing existing customer usage, discovering new needs, and
promoting deeper product integration, from endpoint
sensors to executive dashboards.
Pc.
Renewal and Retention Playbook: Strategies to ensure
continuous Cyber-X utilization for risk management,
spotlighting its unique features and alignment with ZT.
Pd.
Competitive Defense Playbook for: Techniques to un-
derstand competitor solutions in use or consideration by
clients, and ways to reinforce Cyber-X’ advantages.
Pe.
Crisis Communication Playbook for Cyber-X Troubles
and Vulnerabilities: A guide for effective communication
when clients face issues with Cyber-X , emphasizing swift
and transparent responses.
Pf.
IR Alert Process Playbook: A detailed process using
Cyber-X for handling security incidents, leveraging its
features, especially in the context of NDR.
B.4 Data Processing
The data pertaining to customers, products, and sales
playbooks are typically sourced from an internal company
knowledge database. This implies that the data could be
replete with domain-specific jargon, terminology familiar to
company insiders but potentially confusing for LLM or external
stakeholders. Therefore, a procedure is necessary to pinpoint
these terms, enrich the information, and make it ready for
LLM’s decision planning.
For clarity in explaining the process, let’s assume that
customer data is housed in a tabular knowledge database,
whereas data for products and sales playbooks are stored
in document databases. Our refinement process is executed
across both data types to ensure completeness and readiness.
In the case of tabulated data, our emphasis lies on its structural
framework and column titles. For document-based data, LLM
is harnessed to peruse the entire document, produce a summary
for verification, and pose questions. The answers to these
questions are subsequently incorporated into the company’s
FAQ, enhancing the depth of comprehension.
Utilizing one sample playbook outlined in Table III, we can
elucidate the data processing procedure. This approach is not
limited to narrative content but can also be applied to structured
tabular data schemas. We input all the sample playbooks to
GPT-4, and expect it to ask us to clarify the content.
Output from GPT-4: “Please define the following terms for
clarity in your document, especially for external readers and
LLMs: ZT, NDR, SOC, Cyber-X, and C-Level.”
C. Debate: Using LLMs to Generate a Sales Plan
The central phase of the sales planning process revolves
around combining and customizing existing playbooks to effec-
tively engage, persuade, and ultimately convert a prospective
customer.
Once the objectives are clearly defined and data sufficiency
is confirmed, the subsequent phase involves exploring the
candidate decision space. This encompasses a spectrum of
potential strategies or decisions aimed at achieving the desired
business outcomes. Through ongoing evaluation, discussion,
and testing with LLM agents, SocraPlan yields what it considers
the most effective and efficient sales plan. This iterative
approach ensures continuous refinement of both the objectives
and the strategies devised to attain them. Furthermore, it fosters
5
1. Customer Analysis:
- Review existing customer usage patterns.
- Identify the number of seats currently utilized and gauge if there’s scope
for an increase.
2. Potential Product Expansion:
- Introduce customers to a variety of sensors, including endpoint sensors,
network sensors, and email sensors.
- Assess their current setups and recommend appropriate sensors that fit
their unique requirements.
3. Enhanced Management Features:
- Demonstrate the benefits of utilizing the workbench feature for a
centralized view and management.
- Highlight the advantages of automation workflows, especially for SOC
teams, to streamline processes and boost efficiency.
4. Strengthen Incident Response:
- Educate customers on Cyber-X’s threat investigation tools and threat
intelligence resources.
- Showcase how these features enhance their incident response capabilities
and overall security readiness.
5. Risk Management via Dashboards:
- Introduce executive dashboards, focusing on its capability to provide a
high-level overview of security postures.
- Emphasize its value to C-level executives and decision-makers for
informed risk management.
6. Relationship Building:
- Foster deeper ties by addressing specific customer needs & concerns.
- Avoid aggressive sales tactics; instead, focus on organic growth by
demonstrating real value and alignment with customer objectives.
Table III: Steps of the Customer Expansion Strategy.
Objective: Understanding and meeting customers’ needs.
an exhaustive examination of supporting ideas and opposing
viewpoints, culminating in well-informed and fruitful outcomes
for both our customers and sales team.
Formally, with data denoted as
d
encompassing the potential
customer’s profile and a set of
N
playbooks represented as
PN
, SocraPlan leverages the SocraSynth platform to generate a
proposal, denoted as
Ζ
. The SocraSynth committee comprises
a sales manager who serves as the moderator, along with
two LLM-based agents: Agent A (GPT-4) and Agent B
(Bard). The moderator sets the context for the meeting and
provides both data
d
and playbooks
PN
as input. Agent A
is tasked with proposing the initial version,
ΖA1
, which is
subsequently reviewed and revised by Agent B, resulting in a
revised proposal,
ΖB1
. To promote adversarial discourse, the
“contentiousness” parameter of SocraSynth is initially set to a
high level, fostering a combative debate. After a series of
n
rounds of debates, the meeting yields proposals
ΖAn
and
ΖBn
,
respectively, each tailored from the pool of
N
playbooks and
accompanied by justifications.
At this juncture, the moderator adjusts the degree of
“contentiousness” to a more consensual level and directs the two
agents to collaborate in generating a final proposal, denoted
as
Ζ
. This final proposal is then presented to the sales team
for evaluation and further refinement.
IV. CAS E STU DY
In this case study, we present SocraPlan with company H’s
profile and the six playbooks,
Pa
to
Pf
, listed in Section B.3
as input. We let the LLM agents to debate, and we present
Agent A’s opening remarks, and Agent B’s arguments in its
second round.
1. Introduction
- Begin by acknowledging the longstanding relationship with Company
and their recent adoption of our current product.
- Highlight commitment to help them fully utilize the product’s capabilities.
2. Engagement Review
- Discuss current engagement levels, focusing on the usage behavior.
- Suggest ways to increase the frequency of logins through regular check-
ins and providing useful tips and resources.
3. Risk Insights Feature
- Address the low usage of Risk Insight App.
- Emphasize the benefits of the Risk Insights feature.
- Propose additional training or resources to help them understand and
utilize this feature effectively.
4. Advanced Capabilities
- Discuss the engagement score.
- Highlight the benefits of fully utilizing advanced product capabilities.
- Offer to provide further education on these features.
5. Sensor Deployment
- Highlight the sensor deployment status.
- Identify areas for improvement in sensor deployment.
- Propose additional support and resources to help them deploy and utilize
these sensors effectively.
6. Cross-Selling/Upselling Opportunities
- Based on their specific needs and challenges, suggest other services that
could be beneficial to them.
- This includes upselling additional sensors and advanced features.
7. Objection Handling and Value Proposition
- Address common objections customers might have, such as pricing
concerns or competition comparisons. Provide detailed responses and
objection-handling techniques. - Expand on the value propositions of
your product. Clearly articulate how your product uniquely solves their
challenges with real-world examples and case studies.
8. Conclusion
- Reiterate our commitment to their success and satisfaction.
- Emphasize that our approach is not just about increasing engagement.
- Ensure they are fully aware of all the capabilities of our product.
- This will lead to a stronger relationship and potentially higher revenue.
Table IV: Tailored Sales Playbook with Enhanced Content.
A. Agent A’s Opening Round
In the initial round of the debate, Agent A selects
Pc
,
the “Renewal and Retention” playbook. Agent A supports
their proposal for company H based on historical data and
engagement metrics as follows:
Agent A recommends Pc, the "Renewal and Retention"
playbook for Company H. Here are the top reasons:
1. Company H has a long history with us, with a loyalty of 18
years. The goal with existing customers should be retention
and ensuring high satisfaction with the services.
2. Company H has a low engagement score of 29. There is room
for improvement, which playbook Pc can address.
3. Company H has three open cases and one threat case in the
last 3 months. This indicates that they might have faced
some issues with our product. Pc focuses on addressing any
usability or budgeting concerns.
4. The "Risk Insights Access score" is 5, which is medium. This
shows that Company H might not have fully utilized the Risk
Insight feature of our product. Playbook Pc can address.
B. Agent B’s Second Round
In contrast, Agent B proposes
Pb
, the cross-selling/upselling
playbook, in the second round after comparing Company H’s
metrics with similar companies. Agent B’s argument is as
follows:
6
Considering the feedback, let’s revisit some of the key metrics
for Company H and compare them with similar companies in
the Government/Public Services sector.
1. Login Score: "weekly login count" of Company H is 7.5.
Comparing this with similar companies in the dataset,
the score is relatively high, indicating good engagement.
2. Risk Insights Action Score: Company H’s "Risk Insight
Action Score" is 0, suggesting customer didnt use the advance
feature. Comparatively, other customers also have low usage,
reflecting a lack of awareness about this feature. Playbook Pb
can address this issue.
3. Advanced Score: This score for H is which is medium. Similar
companies show varying scores, indicating that the advanced
features may not be universally utilized within the sector.
This presents an upselling opportunity.
C. Tailored Playbook
After the debate has ended, the agents jointly produce
a sales playbook documented in Table IV. The SocraPlan
dialogue involves iterative feedback and idea refinement, with
Agent A eventually aligning with Agent B’s recommendation.
The moderator concludes by outlining a sales pitch agenda,
emphasizing the use of a revised version of
Pb
as the final
proposal for engaging with Company H, highlighting the
collaborative decision-making and adaptive nature of the
discussion.
D. Evaluation and Observation
Human Expert GPT Bard
Metric PbTailored PbTailored PbTailored
Alignment 6 8 9 9 7 9
Engagement 7 9 8 9 8 10
Actionable 7 10 9 10 9 9
Clarity 8 8 9 9 8 9
Quality 7 9 8 9 9 9
Table V: Playbook Evaluation Rated by Human Expert,
GPT-4, and Bard.
The debate mechanism enhances the robustness and thor-
oughness of the final proposal. This procedure is marked by
novel ideas and perspectives. AI agents are actively engaged
in conversations, tackling subjects with new viewpoints and
subtleties. Intriguingly, we discern that the LLM’s concluding
proposal also integrates elements from various provided play-
books, molding a strategy in tune with up-to-date data and
current circumstances. The tailored proposal in Table IV delves
into Company H’s current usage and existing stance in greater
detail, resulting in more engaging product tactics and precise
sales actions.
To obtain qualitative scores for the two playbooks, we asked
human experts to review and utilized CRIT [
5
] along with a
tailored prompt to evaluate both GPT-4 and Bard based on five
metrics from [
9
]: alignment with sales goals, relevance to the
target customer, actionable guidance, clarity and structure, and
content quality. The scoring ranged from 1 (the lowest) to 10
(the highest).
Table V shows that the tailored playbook outperformed
the standard version in customer engagement and actionable
guidance. Experts noted that tailored playbooks, which integrate
real data and numbers, are warmer and more actionable than
generic templates, grounding them in the customer’s actual
situation.
V. CONCLUDING REMARKS
SocraPlan emphasizes the significance of a human-agent
collaboration approach, which combines the expertise of human
analysts with the capabilities of LLM-agents. This collaborative
effort, in alignment with the SocraSynth model, fosters a
harmonious relationship between humans and AI, ensuring
that ever-evolving data landscapes and market nuances remain
central to the crafting of sales playbooks.
While we have conducted initial qualitative evaluations of
the customized sales playbook for a potential customer, the
true assessment will rely on real-world metrics such as lead-
to-customer conversion rates and revenue growth. We will
gather data, collect feedback, and share our experiences as
we implement SocraPlan. Additionally, our future work will
explore methods to fine-tune parameters of SocraPlan, enabling
us to create sales presentations that are not only logically sound
but also emotionally engaging [6].
REFERENCES
[1]
S. Bubeck, V. Chandrasekaran, R. Eldan, J. Gehrke, E. Horvitz,
E. Kamar, P. Lee, Y. T. Lee, Y. Li, S. Lundberg, H. Nori,
H. Palangi, M. T. Ribeiro, and Y. Zhang. Sparks of artificial
general intelligence: Early experiments with gpt-4, 2023.
[2]
Z. Chan, Y. Zhang, S. Han, Y. Bai, and et al. Capturing
conversion rate fluctuation during sales promotions: A novel
historical data reuse approach. In Proceedings of the 29th ACM
SIGKDD, page 3774–3784, 2023.
[3]
E. Y. Chang. Examining GPT-4: Capabilities, Implications,
and Future Directions. The
10th
International Conference
on Computational Science and Computational Intelligence,
December 2023.
[4]
E. Y. Chang. SocraSynth: Socratic synthesis for reasoning and
decision making. Stanford University InfoLab Technical Report,
September 2023.
[5]
E. Y. Chang. Prompting Large Language Models With the
Socratic Method. IEEE 13th Annual Computing and Com-
munication Workshop and Conference, March 2023. URL
https://arxiv.org/abs/2303.08769.
[6]
E. Y. Chang. CoCoMo: Computational Consciousness Modeling
for Generative and Ethical AI, 2023. URL https://arxiv.org/abs/
2304.02438.
[7]
E. Y. Chang and E. J. Chang. Discovering Insights Beyond
the Known: A Dialogue Between GPT-4 Agents from Adam
and Eve to the Nexus of Ecology, AI, and the Brain. Stanford
InfoLab Technical Report (ResearchGate), August 2023.
[8]
J. J. Chang and E. Y. Chang. SocraHealth: Enhancing Medical
Diagnosis and Correcting Historical Records. In The
10th
International Conference on Computational Science and Com-
putational Intelligence, December 2023.
[9]
J. Cleghorn, J. Lee, E. Kennedy, and R. Huey. The Sales
Playbook of Successful B2B Teams. Harvard Business Review,
August 2021.
[10] OpenAI. GPT-4 Technical Report, 2023.
[11]
R. Thoppilan, D. D. Freitas, J. Hall, and et al. Lamda: Language
models for dialog applications, 2022. URL https://arxiv.org/abs/
2201.08239.
[12]
H. Touvron, L. Martin, K. Stone, and et al. Llama 2: Open
foundation and fine-tuned chat models, 2023. URL https://arxiv.
org/abs/2307.09288.