AI governance in software companies centers on liability for AI product behavior, the legal basis for using customer and telemetry data to train or fine-tune models, security of prompts and retrieval stores, EU AI Act exposure for embedded features, and the license terms of open-source and third-party models. A SaaS firm shipping AI features inherits accountability for outputs, data provenance, and cross-border obligations. The governed approach makes approval gates, audit trails, and provenance metadata kernel features so every customer-facing AI output is traceable, reviewable, and defensible to a customer, board, or regulator.
Shipping AI makes you accountable for it
When a software company embeds AI in its product, it stops being a consumer of AI risk and becomes a producer of it. If a generated answer is wrong, defamatory, or leaks another tenant's data, the customer holds the vendor accountable, not the model provider. The EU AI Act, with obligations phasing in through 2025 and 2026 and penalties reaching 35 million euro or 7 percent of global turnover for prohibited uses, classifies many workplace and decision-support features as high risk, which triggers documentation, transparency, and human-oversight duties that fall on the deploying software company.
Data rights are the second exposure. Many SaaS contracts and DPAs never contemplated using customer content to train or fine-tune models. Using it anyway, even to improve a shared model, can breach the agreement and data-protection law. Meanwhile open-source and third-party model licenses vary widely: some restrict commercial use, some impose acceptable-use terms, and some carry output-ownership ambiguity. Governance is not a compliance afterthought here; it is the difference between a defensible AI feature and an unbounded liability. Security is the third exposure that software teams routinely underweight. Prompt injection lets a crafted input override system instructions, and a retrieval store that is not tenant-scoped can surface one customer's content in another customer's session. Both are breaches, not edge cases, and both demand controls at the architecture layer rather than a post-hoc filter. The practical test is simple: for any AI output that ships to a customer, a board, or a regulator, can the company reconstruct the source documents, model, prompt version, and approver on demand. If not, the feature is not governed, it is merely deployed.
Mapping AI risk domains to controls
Each governance domain needs a named owner, a control, and evidence that the control runs.
| Risk domain | Core exposure | Control |
|---|---|---|
| Product liability | Wrong or harmful AI output reaches a customer | Human approval gate plus provenance on consequential outputs |
| Data training rights | Customer data used without contractual basis | Explicit opt-in, per-tenant training flags, data contracts |
| Security | Prompt injection, retrieval store leakage | Tenant-scoped retrieval, input filtering, secrets isolation |
| EU AI Act | High-risk features lack documentation or oversight | Use-case classification, technical file, human oversight |
| Model licensing | Restricted or ambiguous open-source terms | License register, approved-model list, output-rights review |
Make governance a kernel feature
- Route every consequential AI output through a human approval checkpoint before it is marked approved, and keep drafts clearly separated from approved artifacts.
- Attach provenance to each output: source documents, retrieval IDs, model, prompt version, and assumptions, so no customer-facing recommendation is a black box.
- Set per-tenant training flags defaulting to off, and never use customer content to train or fine-tune without an explicit contractual basis recorded in a data contract.
- Classify each AI feature against the EU AI Act and maintain the technical documentation and human-oversight design for anything that lands in the high-risk band.
- Keep an approved-model register with each model's license, acceptable-use terms, and output-ownership position, and block deployment of any model not on it.
How governance fails in practice
- Assuming the model provider carries the liability, when the deploying software company is the party the customer and regulator hold accountable.
- Quietly training on customer data to improve a shared model without opt-in, creating both contract and data-protection breaches.
- Treating the EU AI Act as an EU-only issue, when serving EU users from anywhere pulls the feature into scope.
- Adopting an open-source model without reading its license, then discovering a commercial-use or output restriction after launch.
Governance signals worth tracking
- Share of customer-facing AI outputs shipped with complete provenance metadata attached, targeting 100 percent.
- Number of AI features formally classified against the EU AI Act versus total AI features in production.
- Percentage of models in production that appear on the approved-model license register.
- Approval-gate coverage: consequential outputs passing a human checkpoint before reaching customers, versus those that bypassed it.
Frequently asked questions
Can we use our customers' data to fine-tune our AI models?
Only with an explicit contractual and legal basis. Most SaaS agreements and DPAs never contemplated training, so using customer content by default can breach the contract and data-protection law. Add opt-in language and per-tenant training flags defaulting to off.
Does the EU AI Act apply if we are not based in the EU?
Yes, if you serve EU users. The Act reaches providers and deployers whose AI output is used in the EU regardless of where the company sits, and high-risk features carry documentation, transparency, and human-oversight duties with penalties up to 35 million euro or 7 percent of turnover.
Who is liable when our embedded AI feature gives a wrong answer?
The deploying software company, in practice. Customers and regulators hold the vendor that shipped the feature accountable, not the underlying model provider. That is why human approval gates and provenance on consequential outputs are non-negotiable.
Related reading
Go deeper on this sector and topic.