Log Analysis

On this page, we detail how Couchbase enables easy and efficient ad-hoc analysis of your logs.

Tip

Logs are also stored locally (by default) in the ./agent-activity directory and you are free to process them out of Couchbase, but we recommend leveraging a database purposed for Big Data (e.g., Couchbase) to deliver insights about your application's performance faster.

logs Collection

Agent Catalog maintains a collection of log entries whose schema is identical to that of the Pydantic model here. Agent application analysts are free to directly query the logs collection or any of the views in the following section. For completeness, we show the use of the standard couchbase package to directly query the logs collection from the Analytics Service below:

import couchbase.auth
import couchbase.cluster
import couchbase.options

auth = couchbase.auth.PasswordAuthenticator(
    username="Administrator",
    password="password"
)
cluster = couchbase.cluster.Cluster(
    "couchbase:127.0.0.1",
    options=couchbase.options.ClusterOptions(auth)
)

query = cluster.analytics_query("""
    FROM
        [MY_BUCKET].agent_activity.logs l
    SELECT
        l.*
    LIMIT 100;
""")
for result in query:
    print(result)

Views Over logs

Agent Catalog (as of date) provides two sets of non-materialized views: one for users with the Analytics Service enabled on their cluster and another for users who would prefer using the Query Service (in conjunction with a primary index). The former uses standard view syntax (e.g., FROM agent_activity.Sessions AS s) while the latter uses function syntax (e.g., FROM agent_activity.Sessions() AS s) [1].

Sessions View

All Span instances are uniquely identified by a name and a runtime identifier session. The Sessions view provides one record per session, enabling analysts to reason about their application per "run" (e.g., per conversation).

Each session record contains:

  1. the session ID sid,

  2. the catalog version cid,

  3. the span root name root,

  4. the session start time start_t,

  5. a list of content entries content, and

  6. a list of annotations grouped by span names ann.

The content field details all events that occurred during the session (e.g., the user's messages, the response to the user, the internal "thinking" performed by some agent, etc...). Below we give Python code snippets (assuming the existence of a Couchbase couchbase.cluster.Cluster instance named cluster) to access this view for both the Analytics Service and the Query Service:

bucket = "MY_BUCKET"
query = cluster.analytics_query(f"""
    FROM
        `{bucket}`.agent_activity.Sessions s
    SELECT
        s.sid,
        s.cid,
        s.root,
        s.start_t,
        s.content,
        s.ann
    LIMIT 10;
""")
for result in query:
    print(result)

bucket = "MY_BUCKET"
query = cluster.query(f"""
    FROM
        `{bucket}`.agent_activity.Sessions() s
    SELECT
        s.sid,
        s.cid,
        s.root,
        s.start_t,
        s.content,
        s.ann
    LIMIT 10;
""")
for result in query:
    print(result)

For convenience, we also provide a UDF (for both the Query Service and the Analytics Service) LastSession() that enables users to add the following to their WHERE clause:

WHERE sid = `[MY_BUCKET]`.agent_activity.LastSession()

Exchanges View

More often than not, we are interested in events that happen between a user giving some input and an assistant's response. The Exchanges view provides one record per exchange (i.e., the period between user input and an assistant's response) in a given session. Each exchange record contains:

  1. the session ID sid,

  2. the span root name root,

  3. the user's input input,

  4. an assistant's response output, and

  5. all intermediate logs content between the input and output events (e.g., the messages sent to the LLMs, the tools executed, etc...).

Below we give code snippets to access the most recent exchange for both the Analytics Service and the Query Service:

bucket = "MY_BUCKET"
query = cluster.analytics_query(f"""
    FROM
        `{bucket}`.agent_activity.Exchanges e
    SELECT
        e.sid,
        e.root,
        e.input,
        e.output,
        e.content
    ORDER BY
        e.output.timestamp DESC
    LIMIT 1;
""")
for result in query:
    print(result)

bucket = "MY_BUCKET"
query = cluster.query(f"""
    FROM
        `{bucket}`.agent_activity.Exchanges() e
    SELECT
        e.sid,
        e.root,
        e.input,
        e.output,
        e.content
    ORDER BY
        e.output.timestamp DESC
    LIMIT 1;
""")
for result in query:
    print(result)

ToolInvocations View

To view tool calls with their corresponding tool results, many frameworks will log a tool_call_id value. Other frameworks may choose to leave out this tool_call_id value, but tool result -- tool call pairs can be found by reasoning about their temporal relation to one another. The ToolInvocations view provides one record per {tool_call, tool_result} pair and takes the UNION of both aforementioned approaches. Each tool-invocation record contains:

  1. the session ID sid,

  2. the span root name root,

  3. the tool call entry tool_call, and

  4. the corresponding tool result entry tool_result.

Below we give code snippets to access the most recent tool invocation for both the Analytics Service and the Query Service:

bucket = "MY_BUCKET"
query = cluster.analytics_query(f"""
    FROM
        `{bucket}`.agent_activity.ToolInvocations ti
    SELECT
        ti.sid,
        ti.root,
        ti.tool_call,
        ti.tool_result
    ORDER BY
        ti.tool_result.timestamp DESC
    LIMIT 1;
""")
for result in query:
    print(result)

query = cluster.query(f"""
    FROM
        `{bucket}`.agent_activity.ToolInvocations() ti
    SELECT
        ti.sid,
        ti.root,
        ti.tool_call,
        ti.tool_result
    ORDER BY
        ti.tool_result.timestamp DESC
    LIMIT 1;
""")
for result in query:
    print(result)