Standards, Audits, and Accountability

A law that says 'AI hiring systems must be fair' is nearly unenforceable without an answer to a prior question: fair by what measurable definition, verified how, by whom? This is where technical standards and auditing enter the picture. Standards translate broad legal or ethical requirements into specific, measurable, verifiable technical criteria. Audits are the process by which independent parties verify compliance. Accountability mechanisms are the institutional arrangements that ensure violations lead to consequences. Together, they give AI governance operational force.

Technical Standards: Operationalizing Requirements

A technical standard is an agreed specification that a product, process, or service is expected to meet. Standards bodies — including the International Organization for Standardization (ISO), the International Electrotechnical Commission (IEC), and national bodies like NIST in the US and BSI in the UK — produce standards through deliberative processes that involve technical experts, industry representatives, government, and sometimes civil society. For AI, relevant standards address topics including: trustworthiness properties (accuracy, robustness, reliability, security), bias and fairness measurement methodologies, explainability and transparency requirements, data quality specifications, documentation practices (model cards, datasheets for datasets), and risk assessment procedures. ISO/IEC 42001:2023, published by the joint ISO/IEC technical committee on AI (SC 42), is the first international management-system standard for AI — analogous to ISO 9001 for quality management and ISO 27001 for information security. It specifies what an organization's AI management system must include: policies, risk assessments, documentation, competence requirements, monitoring, and continual improvement processes. Organizations can seek certification against it from accredited third-party certifiers. NIST's AI Risk Management Framework (AI RMF 1.0, 2023) is not a standard in the formal ISO sense — it is a voluntary framework — but it has been widely adopted by US organizations and referenced in government procurement requirements, which gives it practical force.

Standards have important limitations. First, they can be captured by the interests that dominate their development process — if standards bodies are heavily influenced by industry, standards may reflect industry preferences over public-interest requirements. Second, standards can become outdated quickly in a fast-moving field; a standard written for 2022-era language models may be poorly suited to 2025-era multimodal systems. Third, certification against a standard proves compliance with the standard's requirements — it does not prove the underlying system is safe or beneficial in every context. A system can be ISO 42001-certified and still cause harm if the standard's requirements are insufficient for its deployment context.

Third-Party Audits

An audit is an independent examination of a system, organization, or practice against a defined standard or set of criteria. Audit is the mechanism by which 'trust but verify' becomes 'verify.' Without it, governance frameworks depend entirely on self-reporting — companies attesting to their own compliance — which is insufficient for high-stakes systems. Third-party AI auditing is still a young field. Several organizations — including algorithmic auditing firms, academic research groups, and consulting firms — have developed audit methodologies for specific AI system types. The most developed audit methodologies address hiring algorithms (testing for disparate impact on protected groups), credit-scoring algorithms (adverse-action transparency), facial recognition systems (accuracy across demographic subgroups), and content-moderation systems (consistency and due-process compliance). A well-designed AI audit typically includes: access to the model's training data and documentation; access to the model's internal logic or at minimum its inputs and outputs; structured testing against predefined fairness, accuracy, or robustness criteria; assessment of organizational processes (how is the system monitored post-deployment?); and a written report disclosing methodology, findings, and remediation recommendations. Significant obstacles remain. Many AI developers treat their models as trade secrets and resist providing auditors with meaningful access. Sandbagging is a concern — companies might optimize for audit performance on known test distributions rather than real-world performance. The absence of standardized audit methodologies means audit quality varies enormously. And audits are point-in-time assessments of a system that changes through ongoing retraining.

Accountability: Closing the Loop

Accountability is the institutional arrangement by which actors who violate rules face consequences. Without accountability, standards and audits are advisory. With it, they have force. Accountability for AI systems operates along several dimensions. Developer accountability holds the organization that built the model responsible for its design choices, training data, and documented performance. Deployer accountability holds the organization that deploys the model in a specific context responsible for that deployment decision, for human oversight, and for monitoring outcomes. User accountability addresses misuse of AI tools by individuals. The accountability gap is a real structural problem: when a patient is harmed by an AI diagnostic tool, the hospital says it was using a certified third-party system, the AI company says it was deployed outside their stated use case, the data provider says their data met the specifications requested, and the standard-setter says the system passed all required tests. Everyone has a plausible defense. The result is that harm occurs with no responsible party. Governance frameworks address this in several ways. Mandatory documentation chains — requiring that every party in the supply chain (data provider, foundation model developer, fine-tuner, deployer) record their contribution and its specifications — create an evidentiary trail. Joint and several liability doctrines can hold multiple parties collectively responsible. Designated human oversight requirements ensure there is always an identifiable human responsible for the system's operation in a specific context.