LLM Learning Portal
Home Previous Next

LLM Agents

21/30

From Models to Agents

LLM agents are autonomous or semi-autonomous systems that use language models as a cognitive engine to perceive, decide, and act to achieve goals.

What Makes an Agent?

  • Autonomy

    Capable of independent operation with minimal supervision

  • Tool Use

    Can leverage external tools and APIs to extend capabilities

  • Planning

    Able to formulate and execute multi-step plans

  • Memory

    Maintains context across interactions and tasks

  • Adaptivity

    Can adjust strategies based on feedback and changing conditions

Evolution of LLM Agents

Basic Chat Interfaces

Simple question-answering without persistent state

Example: Early chatbots, static assistants

Tool-Augmented LLMs

LLMs with ability to call APIs and use external tools

Example: ChatGPT plugins, Function calling

Agentic Systems

Self-directed systems that can decompose and solve complex tasks

Example: AutoGPT, BabyAGI, Voyager

Multi-Agent Systems

Multiple specialized agents collaborating to solve complex problems

Example: CrewAI, AutoGen, HuggingGPT

Agent Architectures

ReAct Framework

Interleaving reasoning and action steps

Process:

  1. Thought: Reason about current state and goals
  2. Action: Decide on a specific action to take
  3. Observation: Gather information from the action
  4. Repeat until goal is achieved

Reflexion

Self-reflection to improve agent performance

Key components:

  • Verbose thinking about actions and outcomes
  • Learning from past mistakes
  • Explicitly revising strategies based on feedback

MRKL (Modular Reasoning, Knowledge and Language)

Combining neural and symbolic modules

Architecture:

  • LLM serves as router and reasoner
  • Specialized modules for different tasks (calculators, search engines, etc.)
  • Enables precise operations where LLMs struggle

Agentic Workflow

General process flow of autonomous agents

  1. Task Planning: Breaking down goals into subtasks
  2. Tool Selection: Choosing appropriate tools for each subtask
  3. Execution: Carrying out actions with tools
  4. Observation: Gathering results and feedback
  5. Reflection: Evaluating progress and adapting the plan
  6. Iteration: Continuing until the goal is accomplished

Agent Capabilities & Applications

Core Agent Capabilities

Tool Usage

  • Function calling
  • API interaction
  • Data processing tools
  • Code execution environments

Memory Systems

  • Short-term: Conversation context
  • Working: Active task state
  • Long-term: Episodic & semantic storage
  • Associative: Knowledge graphs

Planning & Reasoning

  • Goal decomposition
  • Tree of thought reasoning
  • Chain of thought planning
  • Constraint satisfaction

Feedback Systems

  • Self-evaluation
  • User feedback incorporation
  • Environment signal interpretation
  • Reward modeling

Agent Applications

Personal Assistants

Helping individuals with everyday tasks:

  • Calendar management
  • Email organization & drafting
  • Information gathering & research
  • Personal knowledge management

Examples: Claude Opus, Perplexity AI, Rabbit r1

Professional Agents

Domain-specific expert systems:

  • Coding agents (GitHub Copilot, Devin)
  • Data analysis agents
  • Legal document processing
  • Scientific research assistants

Examples: Devin, Cognition, Microsoft Copilot Studio

Autonomous Agents

Self-directed systems with minimal supervision:

  • Automated web research agents
  • Continuous learning systems
  • Complex task automation
  • Virtual characters with persistent personalities

Examples: AutoGPT, AgentGPT, BabyAGI, Voyager

Multi-Agent Systems

Agent Collaboration

Multiple specialized agents working together:

  • Role-based specialization
  • Task delegation
  • Consensus mechanisms
  • Hierarchical organization
Agent Society Architectures

Organizational patterns for agent teams:

  • Hierarchical: Manager-worker structure
  • Market-based: Task bidding systems
  • Democratic: Voting on decisions
  • Expert panel: Specialized input & critique
Emergent Capabilities

Enhanced abilities from agent teams:

  • Complex problem decomposition
  • Self-critical improvement
  • Diverse perspective integration
  • Resilience to individual agent failures
Example: Software Development Team

Product Manager

  • Requirements analysis
  • Task prioritization
  • High-level planning

Architect

  • System design
  • Technical decisions
  • API specification

Developer

  • Code implementation
  • Unit test creation
  • Documentation

QA Tester

  • Test planning
  • Bug reporting
  • Regression testing

Each agent has specialized knowledge and tasks, communicating through shared context to build a complete product.

Current challenges with agent systems include coordination overhead, computational cost, reliability issues, and ensuring safety with increased autonomy.
Previous All Slides Next