AI Studio provides observability tools to help you understand how your organization is using AI Studio and how your agents are performing.

The following observability views are available:

​
Global usage and spend

The Consumption view is only available to organization admins.

The Consumption view in AI Studio gives you a detailed view of your usage and spend month-by-month across all agents.

To access this view, log in to AI Studio and click the Consumption tab in the left sidebar.

You can switch between a cost view and an activity view, which shows the number of tokens used for each agent type. See below for more details about the data available in the cost view and the activity view.

The image below shows the cost view for a sample organization.

​
Cost view

The cost view shows the following information:

  • Monthly usage-based spend: The total cost of usage for the month, with a comparison to the previous month.
  • Spend by agent type: A day-by-day breakdown of the cost of usage by agent type (API, Framework, and No-Code).
  • Spend breakdown:
    • LLM models: The usage cost for each LLM model.
    • Others: The usage cost for all other services, such as Knowledge Graph hosting, web access, and optical character recognition (OCR).

​
Activity view

The activity view shows the following information:

  • Monthly token usage: The total number of tokens used for the month, with a comparison to the previous month.
  • Top users by token usage: The top five users with the most token usage.
  • Top agents by token usage: The top five agents with the most token usage.
  • Token usage by agent type: A day-by-day breakdown of the token usage by agent type (API, Framework, and No-Code).
  • Token usage by model: The total number of tokens used for each model.

​
Session logs

Session logs are only available to organization admins. Logs are only available for custom no-code agents and are not yet available for Ask Writer or prebuilt agents.

Organization admins can choose to retain logs from agent sessions. When enabled, you can view the logs for an agent for the given retention period.

​
Enable session logging

To enable session logging, log in to AI Studio and navigate to the Admin tab in the left sidebar.

From there, you can enable session logging for your organization.

Once enabled, you can choose the retention period for session logs for each agent. The default is no retention. The possible retention periods are 7 days, 30 days, 90 days, or 180 days.

Whenever someone updates an agent’s session log settings, all organization admins receive an email notification.

​
View session logs

To view session logs, navigate to specific agent from the AI Studio homepage. Select the Observability tab from the top of the page and then select the Session logs view. Click on an individual session to view the logs.

​
Flagged responses

In a chat with a no-code agent, an end user can flag a response as inappropriate or inaccurate. An organization admin can then view the flagged responses within the session logs.

To flag a response in a chat, the user clicks the Flag button in the bottom right corner of the response.

In the session logs for a particular conversation, you can see any flagged responses by clicking the View flagged responses button in the top right corner of the session log.

​
Agent observability

Agent observability provides a detailed view of an agent’s engagement, performance, and usage. Anyone can view these metrics for agents in your organization. Session logs are also available, but only for organization admins if the agent’s session log settings are enabled.

These metrics are only available for deployed agents.

To view metrics for deployed agents, navigate to the specific agent from the AI Studio homepage. Select the Observability tab from the top of the page and then choose the specific metric you want to view.

  • Engagement metrics: Show the top five users for the agent over the specified time period and the total number of user interactions for the agent.
  • Performance metrics: Show the average response time as well as the P90, P95, and P99 response times for the agent over the specified time period. It also shows the response codes broken down by 200, 400, and 500 status codes.
  • Usage metrics: Show the total number of tokens and the total spend for the agent over the specified time period.