Home/Refreshers/Know who is using your AI models: refresher
Know who is using your AI models
Model call records on Bedrock, Vertex AI, Azure OpenAI, and OCI Generative AI, with alerts when logging or guardrails are turned off, or stolen keys go looking for models.
Remember
- Someone calls a model: a person, a pipeline, or a stolen key.
- The call is recorded: with the caller and the model, and on AWS the prompt itself.
- Changes around it are watched: logging, guardrails, keys, and new deployments.
- An alert names the move: and a saved query shows who did what else.
| Platform | Uses | The part that catches people |
|---|---|---|
| AWS | Bedrock model invocation logging to CloudWatch Logs and S3, CloudWatch metric filters on the audit trail, and a Bedrock guardrail | Invocation logging covers the Region it is set in, and only calls through bedrock-runtime. |
| Google Cloud | Data Access audit logs for Vertex AI, organisation log sinks into the security project, log-based alerts, and Model Armor floor settings | Each model call entry names the caller and the model, and carries no prompt or response text. |
| Azure | Azure OpenAI and AI Services resource logs and the subscription's activity log in Log Analytics, with scheduled and native alerts, a token alert per account, and the built-in key access policy | RequestResponse has a row per inference call with the operation, status, duration, a caller IP with its last octet masked, and, for Entra sign-ins, the caller's object ID. |
| Oracle Cloud | The Audit service counted into metrics by caller, with alarms on Generative AI endpoints, API keys, new hosting, and logging changes | Oracle Cloud keeps no record of prompts or responses from model calls, unless a Generative AI project is set to store them. |
Run the quick start
| Download providers | terraform init |
| Preview | terraform plan |
| Apply | terraform apply |
Watch out
- What each cloud records differs.
- On Google Cloud, Standard can keep prompt text through Model Armor, which screens Gemini calls and logs what it screened.
- These logs contain whatever people paste into a prompt, so treat them as sensitive data.
Check it worked
In a test account or project, run the drill in the verify output of the template you applied. Each drill changes a setting and puts it back, and the matching alert reaches the security inbox within minutes.