The foundations of AI governance

For many organizations, the response to the introduction of advanced AI has been as simple as it is reflexive: ban the technology and hope nobody finds their way to the platforms on their own.  

While this makes for easy-to-communicate policy, it's not very effective. Telling people not to use technology that the rest of the world – particularly the people filling their LinkedIn feeds – is telling them is essential just leads to people finding workarounds.

According to a study from Software AG, half of all employees are using “Shadow AI” (unsanctioned AI), with those that do citing a quest for productivity gains, a desire to be independent, and the fact their employers are not offering the tools they need.  

Another survey from Cyberhaven shows that 73.8% of workplace ChatGPT usage occured through public, non-corporate accounts -- and the numbers were higher for Gemini (94.4%) and Bard (95.9%).  

This pattern is reminiscent of the growth of “Shadow IT”, where employees – who in an age of App Stores have grown used to downloading and using apps in their personal lives – use unsanctioned hardware or software (most often cloud-based SaaS) to get their work done.

Like with Shadow IT, Shadow AI brings a host of drawbacks, primarily privacy and security related. Data entered into a consumer version of platforms like ChatGPT may be used to train their models. With employees seeking efficiency gains, and without a coherent policy to guide them, the risk of sensitive customer data making its way into a large language model are uncomfortably high.

So, if we’ve established you need an approach beyond “don’t use AI,” and some form of AI governance in your organization, the next question is logically: what form should it take?

In our view, AI governance goes beyond a simple policy. AI governance comes from a combination of:

• An AI governance committee responsible for oversight

• AI policies and procedures

• Data governance for AI

• AI training

Let’s dive into each of these elements.

Building an AI governance committee  

To unlock AI governance, you need to focus on inclusion over isolation, bringing your team along for the ride. Declarations from on-high are typically ineffective and can lead to alienated employees finding their own tools, putting your organization and its customers’ data at risk.  

A more effective way to build a sense of inclusion, along with oversight, alignment with regulatory requirements, and ethical AI use, is with an AI governance committee. Indeed, a well-structured AI governance committee is the backbone of responsible AI deployment.

Today, large technology companies like Microsoft and Meta are establishing internal AI committees, often referred to as "AI Ethics Boards" or similar, to review and oversee development and implementation of their AI technologies.

Key questions to ask as you build your committee

Who should we involve?

Our take: Include representatives from key functional areas of the organization with diverse experience. Consider leaders from teams like legal, ethics and compliance, privacy, information security, research & development, and product engineering and management. With innovation at a rapid pace, it truly takes a village.

How will we define AI systems?

Our take: The definitions and approaches required for Global Data Protection Regulation (GDPR) compliance can be extended to AI Governance, offering a roadmap to define AI systems. You should also keep in mind new AI regulatory frameworks like the EU AI Act as a good blueprint for the future. But the most important thing? Any data containing sensitive information or IP fed into generative AI systems poses risks and must be governed.

How will we define risk levels?

Our take: As a starting point, keep in mind that any data containing sensitive information or IP fed into generative AI systems poses risks and must be governed accordingly. There is also an AI risk classification system outlined in the EU AI Act, which is a great place to start when defining AI risk across your organization.  

How will we ensure human oversight for high-risk systems?

Our take: Prohibit banned AI systems and set processes for evaluating all other risk categories. Consider leveraging Third-Party Risk Management (TPRM) tools and AI-specific extensions to assess AI-linked risks with privacy, security, and ethics standards. Although TPRM tools are automated, human review ensures flagged risks can be properly addressed.

What is our stance on generative AI systems like ChatGPT? And how will we enable safe use of systems like ChatGPT and DeepSeek?

Our take: We support the use of proprietary AI systems, including generative AI, provided the systems undergo thorough vetting and have guardrails to mitigate known risks. Through our new AI Governance solution, we prepare and protect your data for accelerated AI system rollout. With this solution, you can power responsible AI with clean, compliant, unbiased data — ensuring you get value out of AI, safely.

Another option is building your own large language model (LLM). In a survey of 1,300 enterprise CEOs, 51% said they were planning to build their own generative AI implementations, leveraging foundational models such as ChatGPT, Claude and Llama and extending them into their particular domain, industry, and expertise.  

But this comes with significant challenges — developing a proprietary LLM requires a massive amount of data, extensive testing, leading to high costs.  

A faster, more efficient approach is using solutions that enable quick chatbot creation and deployment. That’s where Rexbot comes in. With Rexbot, you can easily build an internal chatbot for any use case, from HR to sales, connect all your data sources, and search across your entire data estate with built-in access controls. Rexbot gives you the power of AI without the complexity of building from scratch — secure, scalable, and ready to use.

AI policies and procedures

Comprehensive AI policies provide the framework for responsible AI usage within an organization.

Key areas to address in AI policies

• Acceptable AI usage – Define permissible AI use cases and restrictions

• Data management and security – Require AI models to use high-quality, compliant data

• Bias and fairness – Establish guidelines for mitigating bias and ensuring fairness

• Transparency and explainability – Require AI models to be interpretable and auditable

• Human oversight – Define human-in-the-loop requirements for AI decisions

• Third-party AI usage – Address risks associated with vendor-provided AI tools

Data governance for AI – the foundation for all AI policies and procedures

Data governance is the bedrock of AI governance. Just as data professionals follow the principle of "garbage in, garbage out," AI systems are only as reliable and secure as the data they are trained on. High-quality, well-governed data is essential for building trustworthy AI. A well-designed AI committee with clearly defined AI policies can still get into trouble without well-governed data.

Implementing key data governance principles with RecordPoint

• Data quality and integrity – Ensure AI models use compliant, safe data. RecordPoint’s intelligence engine prioritizes data integrity, offering diverse content classification options while ensuring complete security and confidentiality throughout the training process.  

• Data provenance and lineage – Track where data comes from and how it is processed.
  Powerful data discovery and classification features enable you to track the sources and origins of your data, ensuring that data is reliable .

• Privacy and compliance – Enforce data minimization, anonymization, and retention policies.‍
  Proactively manage data to ensure compliance with the GDPR, CCPA, and emerging AI regulations. Track data sources, maintain audit trails, and create structured review processes to manage risk.

• Access control and security – Restrict who can access and build safe data sets for AI.
  Apply granular access controls and enforce least privilege principles, ensuring only authorized users can access sensitive data.

AI training  

For AI governance to be effective, employees must be well-trained on AI risks, policies, and best practices. You can have the best tools, but to get the benefit organizations must go through the process of training their employees on AI governance policies.  

Who needs AI training?

• Executives and leadership – High-level governance, risk management, and ethical considerations

• Developers and data scientists – Technical compliance, bias mitigation, and explainability

• Business and end-users – Understanding AI-assisted decision-making and responsible AI use

• Legal and compliance teams – AI risk assessment, regulatory compliance, and audit readiness

Key AI training areas

The foundation for your AI trainings should come from the AI governance policies and compliance frameworks you’ve established. From there, you can move on to important issues like:

• Recognizing and mitigating AI bias

• Transparency and explainability in AI decision-making

• Incident response for AI-related risks

RecordPoint’s two-pronged AI governance solution

Many AI projects are stalled right now, or will be soon — nearly a third, according to Gartner – but they don’t have to be. The key to unlocking their potential lies in strong AI governance. By implementing the right policies, data controls, and oversight, organizations can move AI initiatives forward with confidence. Here’s how RecordPoint can help.

AI Governance

Through our new AI Governance solution, we prepare and protect your data for accelerated AI system rollout. With this solution, you can power responsible AI with clean, compliant, unbiased data,ensuring you get value out of AI, safely.  

Rexbot

We know now that developing a proprietary LLM requires massive data, extensive testing, and high costs. A faster, more efficient approach is using solutions that enable quick chatbot creation and deployment.

That’s where Rexbot comes in. With Rexbot, you can easily build an internal chatbot for any use case, from HR to sales, connect all your data sources, and search across your entire data estate with built-in access controls. Rexbot gives you the power of AI without the complexity of building from scratch — secure, scalable, and ready to use.

Learn more

A strong AI governance committee, well-defined AI policies and procedures, and comprehensive AI training are essential for responsible AI use. However, governance cannot succeed on policy alone — you also need the right tools to execute.  

Curious about the essential tools that serve as the building blocks for effective AI governance?

• Request early access to our new AI Governance solution — prepare and protect your data to speed up AI rollout

• Reach out to our team about Rexbot — harness the power of AI without the hassle of building from scratch.