In June 2025, security researchers at Noma Security discovered a flaw in Google Gemini Enterprise they dubbed "GeminiJack." The vulnerability was architectural, not a bug in the traditional sense. Attackers could embed malicious instructions inside shared Google Docs, calendar invitations, or emails. When an employee asked Gemini to summarize their inbox or help with a document, the AI would execute those hidden instructions and exfiltrate corporate data.
No clicks required. The targeted employee didn't have to do anything wrong. They just had to use Gemini the way it was designed to be used.
The attack worked because Gemini pulls data from across your Google Workspace: emails, documents, calendar events, chat messages. That broad access creates broad exposure. A single shared document from an external party could trigger data extraction from your entire Workspace environment.
Google fixed the vulnerability after disclosure. But the incident illustrates a fundamental reality about AI assistants: the same capabilities that make them useful create attack surfaces that didn't exist before. Every document Gemini can read is a document that could be compromised through indirect prompt injection, human review processes, or architectural flaws not yet discovered.
Understanding exactly where your data goes when you use Gemini is the first step toward using it safely.
The short version: If you need to redact sensitive documents before they reach AI systems, PaperVeil handles that layer. The rest of this article explains where it fits in the broader governance architecture.
Where Your Data Actually Goes
When you send a document or prompt to Gemini, the data follows a path that most users never consider.
Consumer Gemini (free tier, Google AI). Your prompts and documents travel to Google's servers for processing. By default, this data may be used to improve Google's AI models. The information gets stored, potentially reviewed by human annotators, and incorporated into training datasets unless you explicitly opt out.
Even with opt-out enabled, data is retained for up to 18 months in a disconnected form for abuse monitoring and safety purposes. After safety processing, conversations and related data may be retained for up to three years total. Google states this disconnected data cannot be linked back to your account, but it still exists in Google's systems.
Gemini for Google Workspace (business/enterprise). Google's enterprise tier operates under different terms. The company explicitly states that prompts are considered customer data under the Cloud Data Processing Addendum and are not used to train foundation models. Your content is not shared with other customers or used for AI training outside your domain without permission.
Enterprise prompts and responses are stored for 30 days or less for debugging and abuse detection. Workspace administrators can shorten this period or disable prompt storage entirely.
Gemini in Google Cloud. For the API and cloud-based deployments, Google states that prompts and responses are not used to train models. Enterprise controls allow organizations to configure data residency (EU or US) and limit Google support personnel access to specific regions.
The distinction matters. Consumer Gemini treats your data as training material by default. Enterprise tiers provide contractual protection but still involve data flowing through Google's infrastructure with retention windows and potential human review.
The Training Toggle Myth
Google provides settings to opt out of data training. Many users assume these toggles fully protect their privacy. They don't.
What opt-out actually does. Disabling Gemini Apps Activity prevents Google from using your specific conversations to improve their foundation models. This is meaningful. Your contract negotiation strategy or product launch plans won't end up training the next version of Gemini.
What opt-out doesn't do. The setting doesn't prevent Google from storing your data for other purposes. Conversations may still be retained for:
- Abuse monitoring and safety review (up to 18 months)
- Quality assurance and debugging (up to 30 days in enterprise)
- Legal compliance requirements
- Security incident investigation
Human reviewers may access your conversations when they're flagged by safety filters or abuse detection systems. Google states this happens through an internal governance platform and that data accessed for abuse monitoring is not used to train models. But your sensitive business documents could still be viewed by Google employees if they trigger automated flags.
The real limitation. Even with every privacy setting enabled, you cannot prevent your data from being transmitted to Google's servers, processed by their infrastructure, and potentially accessed by their personnel. The training toggle addresses one specific concern while leaving others unresolved.
In July 2024, technology executive Kevin Bankston discovered that Gemini was automatically ingesting private documents he opened in Google Docs. He couldn't locate settings to control this behavior and noted discrepancies in Google's explanations of how the integration worked. For users who expect fine-grained control over what Gemini can access, the actual capabilities may differ from expectations.
Actual Risks Ranked
Not all data exposure risks are equally likely. Here's how they stack up for Gemini users:
1. Accidental exposure through workspace integration (Highest risk). Gemini's strength is accessing your entire Google Workspace environment to provide contextual assistance. This same capability means that asking Gemini to help with an email could pull information from documents, chats, and calendar entries you didn't intend to include. The GeminiJack vulnerability demonstrated that attackers can exploit this integration through shared content.
2. Human review of flagged content (Moderate risk). When your conversations trigger safety filters, authorized Google employees may review the content. You have no notification when this happens and limited visibility into what triggers flags. Business-sensitive information that happens to match abuse patterns could be viewed by personnel outside your organization.
3. Data persistence beyond intended use (Moderate risk). Even after you delete conversations, disconnected copies may persist in Google's systems for months or years. This data theoretically cannot be reconnected to your account, but it exists in storage systems you don't control. For organizations with strict data retention policies, this creates compliance complexity.
4. Search engine indexing (Lower risk, but documented). Shortly after Gemini launched in February 2024, users discovered their prompts appearing in search engine results. The issue stemmed from indexing by Bing despite Google's robots.txt protections. Google addressed the immediate problem, but the incident demonstrated that data can leak through unexpected channels.
5. Model training data incorporation (Lower risk for enterprise). For consumer users with default settings, prompts become training data. For enterprise users with proper agreements, Google commits to not training on your data. This risk is manageable through tier selection and configuration, unlike the others on this list.
6. Direct breach of Google's systems (Lowest risk). Google maintains robust security for its infrastructure. A direct breach of Gemini's data stores would be a major security incident affecting one of the world's largest technology companies. While not impossible, this is less likely than the other exposure vectors.
Private AI Isn't the Answer
Some organizations respond to cloud AI privacy concerns by attempting to run models locally or build private AI infrastructure. This approach has significant limitations.
Cost reality. Running Gemini-class models locally requires substantial hardware investment. Enterprise-grade GPU clusters capable of running large language models start at hundreds of thousands of dollars. The operational costs for power, cooling, and maintenance add ongoing expenses that most organizations cannot justify.
Capability gap. Locally-run open-source models don't match the capabilities of Gemini, ChatGPT, or Claude. The performance difference is measurable in research benchmarks and noticeable in practical use. Organizations that switch to private AI often find their teams reverting to commercial tools because the quality difference affects productivity.
Security complexity. Running your own AI infrastructure means taking responsibility for its security. You need expertise in GPU server administration, model deployment, network security, and access controls. Many organizations that lack the resources to properly evaluate cloud AI vendors definitely lack the resources to secure their own AI infrastructure.
Update burden. Commercial AI services improve continuously. Google ships Gemini updates regularly, incorporating new capabilities and safety improvements. Self-hosted alternatives require you to track model releases, test updates, and manage deployments. Most organizations fall behind, running increasingly outdated models.
The real question. For most organizations, the choice isn't between cloud AI and private AI. It's between using cloud AI safely and using it unsafely. The organizations attempting private AI deployments often have shadow usage of cloud tools anyway because employees prefer tools that actually work.
The Approach That Works
The practical solution to Gemini privacy concerns isn't avoiding Gemini. It's controlling what data Gemini sees.
Redaction before upload. If sensitive information never reaches Google's servers, it can't be stored, reviewed, or exposed through any of the vectors described above. A document processed to remove names, account numbers, and other identifiers before Gemini sees it creates no privacy exposure regardless of Google's policies.
This approach provides several advantages:
Policy independence. You're not relying on Google's privacy settings, which can change with policy updates. You're not trusting that opt-out toggles work as documented. The protection comes from the data transformation, not from vendor promises.
Comprehensive coverage. Redaction protects against all exposure risks simultaneously. Human review of flagged content reveals nothing useful if the content contains placeholders instead of real data. Persistence in Google's systems creates no exposure if persisted data is de-identified. Even a hypothetical breach of Google's infrastructure wouldn't expose your actual sensitive information.
Auditability. You can verify that redaction occurred before data transmission. This creates a compliance artifact that demonstrates due diligence in a way that "we enabled the privacy toggle" does not.
How it works. Effective redaction identifies sensitive patterns in documents before processing:
- Named Entity Recognition detects names, organizations, and locations
- Pattern matching finds Social Security numbers, account numbers, and other formatted identifiers
- Date detection generalizes specific dates to ranges or removes them entirely
- Custom patterns catch organization-specific sensitive information
The redacted document uses consistent placeholders: [PERSON-1], [ACCOUNT-NUMBER], [DATE-RANGE]. Gemini processes the sanitized content and generates responses using these placeholders. You map the placeholders back to real values within your controlled environment after receiving the response.
Gemini never sees the actual sensitive data. The AI assistance happens, but the privacy exposure doesn't.
Workflow Integration
Implementing redaction-based privacy protection requires integrating the process into how your teams actually use Gemini.
Document Processing Workflow
Step 1: Classify incoming documents. Before any document enters a Gemini workflow, classify its sensitivity level. Documents containing personal identifiable information, financial data, health information, or proprietary business information require redaction. General reference materials may not need processing.
Step 2: Apply automated redaction. Run classified documents through detection and redaction. The output is a sanitized version with placeholders replacing sensitive elements. Maintain a secure mapping file that links placeholders to original values.
Step 3: Process through Gemini. Submit the redacted document to Gemini for whatever assistance you need: summarization, analysis, drafting responses. The AI works with the sanitized content.
Step 4: Reconstitute the response. Using your secure mapping, replace placeholders in Gemini's output with the original values. This happens within your controlled environment, not in Google's systems.
Step 5: Archive appropriately. Store both the redacted version (for audit trails showing what Gemini saw) and the reconstituted version (for business use). Maintain the mapping in secure storage with appropriate access controls.
Email and Communication Workflows
For Gemini integration with Gmail:
- Draft emails in your local environment first
- Redact before asking Gemini for help with tone, formatting, or response suggestions
- Reconstitute after receiving Gemini's assistance
- Send from your normal email workflow
This prevents the accidental exposure that occurs when Gemini pulls context from your inbox to help with a response.
Enterprise Controls
For organizations using Gemini for Workspace:
- Configure the shortest possible retention period through admin controls
- Enable audit logging to track Gemini usage across the organization
- Implement trust rules in Drive to restrict what Gemini can access
- Train users on which document types require redaction before AI processing
These controls reduce risk but don't eliminate it. Redaction remains the most reliable protection for truly sensitive information.
The Bottom Line
Google's privacy policies for Gemini provide meaningful protections for enterprise users. Your data won't be used to train foundation models. Retention periods are limited. Compliance certifications demonstrate security controls.
But these policies address some risks while leaving others unresolved. Human review happens. Data persists for months or years. Architectural vulnerabilities like GeminiJack demonstrate that the integration capabilities that make Gemini useful also create attack surfaces. And policies can change with future updates.
For documents where exposure would create real harm, relying entirely on vendor policies is insufficient. The organizations protecting sensitive information most effectively:
- Use enterprise tiers with contractual protections
- Configure the shortest possible retention periods
- Implement pre-processing redaction for sensitive documents
- Maintain audit trails of what data was shared with AI
- Train users on appropriate versus inappropriate use cases
The productivity benefits of AI assistance are real. So is the data exposure that comes with sending sensitive documents to external systems. The solution isn't choosing between productivity and privacy. It's building workflows that deliver both.
PaperVeil lets you redact sensitive information from documents before they touch any AI system. Detect and remove personal identifiers, financial data, and proprietary information automatically. Generate audit trails showing exactly what external AI systems saw. The redaction layer that makes AI document processing actually safe.