AI Data Protection Challenges & Solutions

The rapid adoption of artificial intelligence technologies is leading to an equal rate of acceleration in AI-related data breaches, such as the recent attack on Cutout.Pro that exposed 20 million users’ personal information. From inference attacks to prompt injections and training data poisoning, malicious actors use a variety of techniques to compromise AI models and steal valuable data. In addition, end-user behavior, inadequate tools, and an ever-shifting regulatory landscape further complicate data privacy and security risks. This post discusses some of the most common AI data protection challenges and describes the technologies and strategies used to solve them.

AI data protection challenges & how to solve them

The table below lists and describes the biggest AI data protection challenges and solutions. Click the links to read more about each.

PII in user prompts

Users frequently include potentially identifiable information in their LLM prompts without realizing it, increasing the risk of PII exposure and compliance issues. For example, an employee using generative AI to help create an internal report may input confidential financial or personnel information that could be leaked.

Given that LLMs continuously learn from prompt inputs, it’s vital to continuously monitor both the input data and the generated outputs. This ensures that no sensitive data is inadvertently processed or produced by the LLM, maintaining ongoing compliance and security.

Inference attacks

In an inference attack, a malicious actor probes an AI model for enough PII-adjacent information about an individual that they’re eventually able to fill in the blanks to identify them. An example would be using someone’s first name and in-store shopping habits to eventually learn their full name and physical address.

Inference attacks are avoidable by limiting the amount and type of data collected from customers and other end-users, and by removing all identifiable information from data before AI ingestion. Some AI data privacy tools also replace identifiable information with synthetic data, such as fake names and phone numbers, which allows an AI model to train on realistic data while removing the risk of data inference.

Data linkage

Data linkage uses methods similar to inference attacks. It involves attempting to re-identify an individual by combining semi-anonymized data outputs from the AI model with other available information in a company’s systems. A famous, pre-AI example of a data linkage attack occurred when researchers were able to combine data from public, anonymized Netflix movie rankings with non-anonymous IMDB user profiles to identify many users.

Like an inference attack, data linkage can be avoided by reducing the amount of identifiable information contained in company data and by ensuring anonymization techniques (like redaction and synthetic data) are used for all sensitive data, not just in AI training datasets.

Prompt injections

Prompt injection attacks involve injecting malicious content into AI prompts to get the model to expose sensitive information or to manipulate the system’s behavior in a way that compromises performance or accuracy. The potential for prompt injections to harm LLMs was demonstrated by researchers in 2022 who convinced OpenAI’s GPT-3 model to ignore its original instructions and intentionally output incorrect information.

The best way to prevent prompt injections is to continuously monitor user inputs and model outputs to detect signs of manipulation.

Training data poisoning

Training data poisoning occurs when a malicious actor intentionally contaminates a training dataset to negatively affect AI performance. Attackers add, change, or delete data in such a way as to introduce vulnerabilities, biases, or errors in the AI model’s operation, rendering any of its decisions less trustworthy.

The best way to prevent poisoning is to protect training datasets with stringent, role-based access controls and data governance features like change requests and audit trails. Continuously validating training data by detecting and removing anomalous data points before ingestion is also important. Additionally, monitoring AI model performance once it’s operational can help detect signs of poisoning, like performance degradation or unexpected behavior.

False positives and negatives

While many tools can automatically detect and classify at least some sensitive data, there is a wide range of accuracy as measured by benchmark F1 scores. Low accuracy leads to false positives and/or false negatives, each of which is a significant challenge for companies attempting to preserve privacy for AI. False positives generate costly, useless work and disruption, whereas false negatives silently increase the risk of PII exposure. Without high accuracy, the whole data privacy system breaks down, especially at scale with lots of data.

he solution to this challenge is using better PII discovery tools. For example, the Granica Screen solution uses AI-powered detection algorithms that deliver state-of-the-art accuracy to reduce false positives and negatives, improving efficiency and data privacy.

Breadth of data coverage

AI models ingest training data and user prompts from all over the world and in many different languages. That means AI developers and operators must maintain compliance with all the relevant, region-specific data privacy regulations, like the European Union’s new AI Act or California’s CCPA. In addition, some PII data discovery and classification tools may struggle to detect sensitive information in certain, less commonly used languages.

New data privacy and compliance tools are evolving to incorporate global standards and automate compliance. These solutions also offer expanded named entity and language support to provide greater coverage for training data and inputs from around the world.

Maintaining compliance in cloud data lakes

Many AI data protection tools are highly compute-intensive, making them too expensive to run comprehensively on the large datasets typically found in cloud data lakes. As a result, organizations scan only small samples of data and only scan more broadly based on the sample results.

Gaining complete visibility into who/what is accessing sensitive information - which is required by many data privacy laws and regulations - poses further challenges in cloud data lake environments where organizations don’t have complete control over their infrastructure.

Granica offers the world’s first data privacy and visibility tools for cloud data lake storage, providing two solutions to these compliance challenges.

• Granica Screen discovers sensitive information in cloud data lake files such as Parquet using a novel, compute-efficient detection algorithm that can process 5-10x more data for the same infrastructure cost as other data discovery tools. As a result, organizations can reduce their sampling and increase their data privacy and compliance coverage, as well as increase the volume of safe data for use in model training and LLM fine-tuning.
• Chronicle AI provides visibility into who and what is accessing data lake files identified to contain sensitive information, using cloud IAM (identity and access management) roles rather than user names to protect anonymity. Plus, Chronicle AI analyzes file metadata without looking inside file contents, further protecting data privacy. Leveraging these tools can help security and privacy teams understand and reduce the potential exposure of sensitive data.

Streamline AI data protection with Granica

Granica’s AI data platform provides data discovery and classification services with state-of-the-art accuracy and high compute efficiency to find and mask sensitive information in training data stored in cloud data lakes as well as in LLM input prompts and output results. Granica helps organizations to preserve privacy for LLMs from training to inference to reduce the risk of data breaches associated with using AI while improving overall security and compliance.