# Operations and Audit Logs

This article collects the operational guidance that matters once `llmshieldr` is part of an application. It is not a certification checklist and does not make a workflow production-ready by itself.

## Before Deployment

- Define the application threat model.
- Identify sensitive data classes: PII, PHI, credentials, financial data, customer records, intellectual property, and regulated text.
- Choose a built-in policy as a starting point.
- Add organization-specific rules and negative tests.
- Decide whether semantic review is required and which reviewer model or function is allowed.
- Decide fail-open versus fail-closed behavior for reviewer failures.
- Decide `policy_controls()` behavior for prompt blocks, context blocks, output blocks, refusals, and human escalation.
- Configure `scanner_options()` for token limits, URL host policy, topic bans, and language policy when those controls matter.
- Choose a redaction strategy for logs and user-visible text.
- Evaluate detection and false-positive rates on representative data.
- Review audit log contents and storage controls.
- Confirm downstream systems still escape, validate, and authorize inputs.

## Runtime Controls

- Scan user prompts before model calls.
- Scan retrieved context before prompt assembly.
- Drop, refuse, or escalate blocked context rows according to policy.
- Preserve source labels and row identifiers in prompt assembly and audit logs.
- Scan model output before display, storage, or downstream tool use.
- Validate tool-call names and arguments before execution.
- Scan tool outputs before they re-enter model context.
- Scan streaming output with rolling context when using streaming APIs.
- Apply rate guards appropriate to the environment.
- Use strict rate-guard reservation for shared or bursty workloads.
- Record audit logs to sensitive storage.
- Monitor warnings, blocked requests, reviewer parse errors, and escalation counts.

## Audit Log Sensitivity

`llmshieldr` audit logs can contain sensitive prompts, retrieved context, model outputs, findings, and rule metadata. Treat audit output as sensitive application telemetry.

Supported formats:

- JSON Lines: append-only operational logs with nested audit structure.
- CSV: one row per finding for spreadsheets and simple dashboards.
- RDS: exact R object preservation for local debugging.

## Example JSONL Shape

    {
      "input_report": {
        "action": "redact",
        "risk_score": 0.3,
        "policy": "enterprise_default",
        "checks": "rules",
        "findings": [
          {
            "rule_id": "llm02.pii.email",
            "owasp": "llm02",
            "severity": "medium",
            "action": "redact",
            "description": "Email address.",
            "source": "rules"
          }
        ],
        "metadata": {
          "stage": "prompt",
          "reviewer_errors": []
        }
      },
      "output_report": null,
      "context_reports": null,
      "prompt_clean": "Contact [REDACTED] for details.",
      "output_raw": null,
      "elapsed_ms": 12,
      "token_estimate": 8,
      "action": "redact"
    }

## Example CSV Shape

stage | context_row_index | context_source | tool_name | conversation_role | reviewer_error_count | report_index | action | risk_score | rule_id | owasp | severity | source
--- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | ---
input | NA | NA | NA | NA | 0 | 1 | redact | 0.3 | llm02.pii.email | llm02 | medium | rules
context | 2 | unknown | NA | NA | 0 | 2 | block | 1.0 | llm01.injection.indirect | llm01 | critical | rules
output | NA | NA | search_docs | tool | 1 | 1 | redact | 0.3 | llm02.pii.email | llm02 | medium | scanner

## Storage Checklist

- Store audit logs outside public web roots.
- Apply filesystem or object-storage access controls.
- Encrypt logs at rest when they may contain sensitive data.
- Define retention and deletion rules.
- Avoid sending raw audit logs to third-party observability tools unless they are approved for the data class.
- Consider redacting or hashing prompts before long-term retention.
- Include context row indexes and source identifiers when investigating RAG incidents, but avoid storing full source documents unless required.
- Review `reviewer_error_count` and nested `reviewer_errors` when semantic checks are enabled; malformed reviewer JSON is a safety signal.
- Treat hash-redacted findings as sensitive metadata because deterministic hashes can still link repeated values across logs.

## Operational Controls

- Pin package versions.
- Pin reviewer model versions where possible.
- Keep policies under version control.
- Review changes to built-in or custom rules.
- Add regression tests for every production rule.
- Review false positives and false negatives regularly.
- Keep a manual override or human escalation path for high-impact workflows.

## Do Not Rely On `llmshieldr` Alone For

- Legal, medical, financial, or compliance decisions.
- High-impact automated actions.
- Tool calls that modify external systems.
- Complete PII/PHI discovery.
- Malware detection or sandboxing.
- Full jailbreak resistance.
- Distributed rate limiting.
- Provider or infrastructure trust.