Skip to content
Compliance for Technical Staff

Lesson 8: Treat AI as a managed system

You will be able to: identify the safeguards an AI feature needs before release: intended use, data handling, vendor terms, evaluation, human oversight, monitoring, and change control.

About 4 min

AI features go through the same life cycle as other software, with some risks of their own. Outputs vary from run to run. Inputs can carry instructions (prompt injection), including from content the user never typed, like an email or support ticket the model is asked to summarize. Prompts can send sensitive data to a vendor, and outputs can expose it. And the model can change underneath you.

Source: OWASP Top 10 for LLM Apps, LLM01, LLM02(opens in a new tab)

  • Intended use: what the feature is for, what it must not be used for, and who it affects. Assess the impact on people, not just accuracy.
  • Data: what goes into prompts, where it is sent, how long the provider keeps it, and whether it may be used for training.
  • Vendor: contract terms, and agreements such as a data processing agreement or business associate agreement where they are required.
  • Evaluation: representative test sets and acceptance criteria, agreed before release, including adversarial inputs.
  • Human oversight: proportionate to the impact. A person should approve outputs that reach customers or affect decisions about them.
  • Monitoring and change: watch quality in production, and treat model, prompt, and provider changes as changes.

ISO/IEC 42001 organizes these into a management system: AI risk assessment, AI system impact assessment, and controls across the AI system life cycle.

Source: ISO/IEC 42001:2023(opens in a new tab)

Exercise 1 of 2: Review the release proposal

Support reply assistant: release proposal. When an agent opens a ticket, a large language model summarizes the conversation and drafts a reply.

  • Intended use: summarize tickets and draft replies for agents to edit. Not for clinical or billing decisions.
  • Owner: support tooling team.
  • Access: only support agents can see drafts.
  • Model: the vendor’s latest model alias.
  • Data: the full ticket thread, including attachments, is sent in the prompt. Tickets sometimes include health details and payment information.
  • Vendor terms: standard self-serve API terms; not yet reviewed.
  • Evaluation: the team tried it on about ten tickets, and the results “looked good.”
  • Oversight: if the agent does not act within 10 minutes, the draft is sent automatically.
  • Monitoring: none planned.

Identify the missing safeguards, then choose a path.

Which safeguards are missing? Select all that apply.
What should happen to the release?

Apply it 2 of 2: The model is changing

The assistant is live with a pinned model version. The vendor announces that the version will be retired in 30 days, and recommends a newer one.

What do you do?

Choose the best approach.

Key points

  • Define intended use, and what the feature must not be used for, before building.
  • Know what data goes into prompts, where it goes, and on what vendor terms.
  • Evaluate against agreed criteria before release; keep a human in the loop in proportion to impact.
  • Pin model versions, monitor in production, and treat model and prompt changes as changes.