Building Leapfrog AI: LLM powered Chat Agent for Supply Chain Design

Introduction

Leapfrog AI is a specialized LLM assistant integrated into Optilogic’s Cosmic Frog Supply Chain Design platform. It transforms complex supply chain design into natural language interactions, making sophisticated data analysis and modeling accessible to business users. Sign up for free and start exploring here!

The Supply Chain Challenge

In the dynamic world of supply chain management, decision-making depends on timely insights, accurate models, and seamless analysis. Optilogic, a pioneer in cloud-native supply chain solutions, saw an opportunity to revolutionize how supply chain designers interact with their data and models. Traditionally, extracting actionable insights required deep technical knowledge of databases and SQL queries—creating a barrier for teams looking to move faster.

These technical requirements traditionally created bottlenecks, requiring specialized knowledge of both supply chain concepts and database operations. Enter Leapfrog AI—a custom-built Large Language Model (LLM) Agent designed to simplify supply chain design workflows and enhance productivity for Cosmic Frog users.

Leapfrog AI Solution

Leapfrog AI bridges the gap between supply chain designers and their data by enabling natural language interactions, combined with expert-level SQL capabilities. Watch as Ryan Purcell showcases Leapfrog AI’s core features in this engaging demo.

Key features of Leapfrog AI v4.1 include:

Core Architecture

The Leapfrog AI is a stateless conversational Agentic system designed to process user queries, generate structured responses, and execute function calls, primarily interacting with the Postgres database on Optilogic platform. Below is a high level overview of the Agent architecture.

Key Architectural Components

Pydantic Validation

Some examples of function signatures for Pydantic validatiors defined for function classes are given below. These include validation of syntax as well as business rules.

@field_validator("statements")
def ensure_select_in_statement(cls, v: list[str])

@field_validator("statements")
def ensure_single_select_statement(cls, v: list[str])

@field_validator("statements")
def ensure_select_statement_executable(cls, v: list[str], info: ValidationInfo)

@field_validator("statements")
def ensure_insert_in_statement(cls, v: list[str])

@field_validator("statements")
def ensure_insert_statement_executable(cls, v: list[str], info: ValidationInfo)

@field_validator("statements")
def ensure_no_inserts_to_output_tables(cls, v: list[str], info: ValidationInfo)

@field_validator("statements")
def ensure_delete_in_statement(cls, v: list[str])

@field_validator("statements")
def ensure_no_deletes_to_empty_tables(cls, v: list[str], info: ValidationInfo)

@field_validator("statements")
def ensure_delete_statement_executable(cls, v: list[str], info: ValidationInfo)

@field_validator("statements")
def ensure_update_in_statement(cls, v: list[str])

@field_validator("statements")
def ensure_no_updates_to_empty_tables(cls, v: list[str], info: ValidationInfo)

@field_validator("statements")
def ensure_no_updates_to_output_tables(cls, v: list[str], info: ValidationInfo)

@field_validator("statements")
def ensure_update_statement_executable(cls, v: list[str], info: ValidationInfo)

@model_validator(mode="after")
def prevent_delete_of_baseline(self)

@field_validator("blank_model_name")
def ensure_model_name_is_unique(cls, v: str, info: ValidationInfo)

@field_validator("blank_model_name")
def ensure_model_name_does_not_contain_forbidden_chars(cls, v: str)

Dataset Creation and Fine-tuning Methodology

Data Collection and Preparation

The system employs a three-stage approach to prepare training data:

[
  {
    "prompt": "Do I have any unassigned scenario items? If so, show me scenario item names.",
    "sql_statements": [
      {
        "table_names": [
          "scenarioitemassignments",
          "scenarioitems"
        ],
        "sql": "SELECT itemname from scenarioitems where itemname not in (select distinct itemname from scenarioitemassignments)"
      }
    ]
  }
]

Training Data Structure

Each training example follows a carefully designed conversation flow:

{
    "messages": [
        {
            "role": "system",
            "content": "Supply chain assistant prompt"
        },
        {
            "role": "user",
            "content": "Business query in natural language"
        },
        {
            "role": "assistant",
            "content": "table_info/helper tool call to understand schema"
        },
        {
            "role": "function",
            "content": "Table schema and metadata or other tool call content"
        },
        {
            "role": "assistant",
            "content": "SQL/action/utility function/ChatCompletion call"
        },
        {
            "role": "assistant",
            "content": "formatted response"
        }
    ],
    "tools": [available_function_definitions
    ]
}

Fine-tuning Process

The fine-tuned model learns tool call structures, including when to call a tool, the correct parameters for tool execution and how to generate valid SQL queries.

DataModel1/
  ├── select_examples/
  ├── update_examples/
  ├── delete_examples/
  ├── insert_examples/
  └── reply_about_me_examples/
  ...

DataModel2/
  ├── select_examples/
  ├── update_examples/
  ...

The implementation follows a systematic approach:

Below are the loss and accuracy plots provided by OpenAI API.

Validation and Error Handling

The validation system operates on multiple levels, ensuring reliable and secure operation. For SQL operations, it checks syntax, verifies table and column existence, tests query executability via EXPLAIN, and compares results for equivalence. Utility calls and Action calls undergo argument type checking and parameter validation, while text responses are evaluated using semantic similarity metrics.

Core Structure

The pipeline consists of three main stages:

Error Types During Generation

Error Types During Validation

Below is a plot of the share of different error types across all use-case

Hit Criteria

Conclusion

The Leapfrog AI architecture demonstrates a robust approach to building enterprise-grade LLM applications, currently serving hundreds of users across different organizations. Through our extensive experimentation and development process, we’ve uncovered several crucial insights:

Key Learnings

Future Directions

As we continue to evolve the system, we’re focusing on:

The success of the Leapfrog AI demonstrates that with careful architecture, comprehensive validation, and thoughtful fine-tuning, LLMs can be effectively deployed in enterprise environments while maintaining both technical precision and user-friendly interaction. Get started with Leapfrog AI here!