[email protected]
Security and sovereignty in RAG architectures: a deployment guide for private and hybrid cloud for enterprises.

Security and Sovereignty in RAG Architectures: A Deployment Guide for Private and Hybrid Cloud Environments

Why securing a RAG system doesn’t follow the same playbook as securing a traditional application

Quick Answer

A RAG (Retrieval-Augmented Generation) architecture introduces risks that don’t exist in a traditional application: prompt injection through retrieved documents, information leakage between users or clients when the vector index doesn’t enforce permissions strictly, and sensitive data flowing out to third-party APIs on every embedding or inference call.

Deploying the system in a private or hybrid cloud environment, with real data isolation at the retrieval layer and components the organization can actually control and audit, reduces both risks at once: the security risk and the risk of dependency on a closed vendor.

A RAG system isn’t secure because the model can be trusted. It’s secure when it’s structurally impossible to retrieve a document the user doesn’t have permission for, no matter what happens in the prompt.

The pattern shows up in a lot of enterprise RAG pilots: the demo works perfectly with twenty test documents, all at the same confidentiality level. The problem appears when it scales to production, once the index holds thousands of documents pulled from different systems (Drive, Confluence, SharePoint, the HR intranet), each with its own permissions model, and someone on the support team stumbles onto the fact that the internal chatbot can answer questions about the executive committee’s compensation document.

Why Security in RAG Architectures Is Different From a Traditional Application

A traditional application has a reasonably well-understood attack surface: the authentication layer, the authorization layer, and database queries. A RAG system adds new surface in three places:

  • Retrieved content becomes part of the prompt: an indexed document can contain instructions designed to manipulate the model’s behavior, what OWASP classifies as indirect prompt injection, something with no equivalent in a traditional SQL query.
  • Data crosses more hops, and often more vendors: every call to the embedding model and every inference call is a potential exit point for data leaving the organization’s perimeter, especially when both are external commercial services.
  • The permissions model tends to get flattened: unifying documents from very different source systems into a single vector index makes it easy to lose the permission granularity each source system used to enforce separately.

What a RAG Architecture Is and Where the Risk Points Are

A RAG architecture connects a language model to a proprietary knowledge base through a multi-stage process: ingesting and chunking documents, generating embeddings, storing them in a vector database, retrieving the most relevant chunks for a given query, and finally generating a response from the model based on those chunks.

Each stage is a potential risk point:

  • Ingestion: if the source document’s permissions aren’t synced along with its content, the chunk ends up orphaned from any access context in the index.
  • Vector database: a shared index without strict isolation by user, client, or department can return chunks the requester shouldn’t be able to see.
  • Embedding model call: if it’s an external service, the document’s content (sensitive or not) leaves the organization on every indexing pass.
  • Orchestration layer: this is where the final prompt gets assembled by combining the user’s query with the retrieved chunks, and therefore where an indirect prompt injection takes effect.
  • LLM inference endpoint: the same data-exit risk as with embeddings, made worse because both the question and the retrieved context travel through it.

Data Isolation in RAG Systems: Multi-Tenancy, Permissions, and Context Leakage

The most common and most dangerous design mistake is retrieving first and filtering afterward: the system searches the entire index for the chunks most similar to the query, and only then discards the ones the user shouldn’t see. This pattern fails for two reasons: it’s probabilistic in a system that needs deterministic guarantees, and it can also skew or thin out the relevant results, since the chunks discarded for permission reasons have already taken up space among the top matches.

The correct pattern reverses the order: first resolve the user’s identity and permission context, then restrict the vector search space to what that identity is allowed to see, and only then run the similarity search and reranking. The search never “sees” a chunk outside the authorized scope, instead of seeing it and discarding it afterward.

  • Document-level RBAC synced from the source: each chunk’s permissions are inherited from its source system (Drive, Confluence, SharePoint) and kept as metadata on the chunk itself in the index, not as a separate rules layer that has to be kept in sync by hand.
  • Isolation by silo or by metadata: depending on sensitivity level, each client or department can have its own physically separate index (silo) or share an index with strict metadata filtering by tenant ID (pool). The former is more secure; the latter is more cost- and maintenance-efficient.
  • Row-level security for agents querying structured data: when an AI agent runs queries directly against a relational database, row-level security (RLS) has to be enforced in the database itself, not by trusting the agent to build the correct query.
  • Deletion sync: when a document is deleted or loses access in the source system, its embedding must be deleted or revoked from the vector index; otherwise, the content stays retrievable even though it shouldn’t exist anymore.
  • Auditing every retrieval: logging which user retrieved which chunk and when is what lets you investigate an incident after the fact, not just prevent one beforehand.

If your RAG system’s security depends on instructing the model to “only use documents from the right department,” you don’t have security: you have a polite request to a probabilistic system.

Deploying RAG in Private and Hybrid Cloud Environments: Architectural Options

Fully Private Cloud or Dedicated VPC RAG

The embedding model, vector database, and inference model are all deployed inside the organization’s network perimeter, typically using self-hosted, open-weight models. This is the option with the most control and the smallest external exposure surface, suited to highly sensitive data or heavily regulated industries, in exchange for taking on the operational burden of the inference infrastructure.

Hybrid RAG: Commercial Models via a Private Endpoint Plus On-Premises Data

The vector index and the data stay on the organization’s own infrastructure, while inference runs against a commercial model exposed through a private endpoint, inside a cloud provider’s VPC or virtual network, without going out over the public internet. This is a common middle ground: it keeps the data under control while still taking advantage of higher-performing commercial models.

When to Avoid Closed Commercial Models Because of Vendor Lock-In Risk

When the prompt, the orchestration layer, and the embedding format all depend entirely on a single vendor’s proprietary conventions, migrating to another option later means rewriting a large part of the system, not just changing a configuration variable. This risk grows the more critical the use case is and the faster the commercial model market keeps evolving (and changing its pricing or terms).

Code Control and Vendor Lock-In Mitigation in RAG Architectures

  • Self-hosted, open-weight models (families like Llama or Mistral, among others) served through your own inference engines such as vLLM, Ollama, or TGI, removing dependency on a commercial API for the system’s most critical component.
  • Open orchestration frameworks like LangChain, LlamaIndex, or Haystack, which keep the application logic from getting locked into a single vendor’s proprietary conventions for generative AI.
  • Open-source vector databases like Weaviate, Milvus, Qdrant, or the pgvector extension on PostgreSQL, as opposed to proprietary managed services that make the index harder to port elsewhere.
  • Portable embedding formats: evaluate from the initial design what it would cost to switch embedding providers, since vectors from two different models aren’t interchangeable, and a migration can require reindexing the entire knowledge base.
  • Documentation and knowledge transfer as a deliverable, not an afterthought: the internal team needs to be able to operate, audit, and modify every piece of the architecture without depending on the vendor that built it still being around.

These risks, particularly around prompt injection and weaknesses in embedding use, are covered more broadly in the OWASP Top 10 for LLM Applications, a useful checklist when designing or auditing any RAG system headed for production.

Is your RAG system relying on the prompt to respect permissions that should be enforced before the search even runs?

At Galde, we design RAG architectures with real data isolation, code control, and deployment in private or hybrid environments matched to each organization’s sensitivity level.

Schedule a technical diagnostic session and find out whether your RAG architecture enforces permissions before or after the vector search.

How Galde Can Help Deploy Secure, Sovereign RAG Architectures

Through generative AI, we design and build RAG systems and agents with permission-aware retrieval from the initial design, avoiding the retrieve-then-filter pattern.

Through data platforms, we build the deployment infrastructure (private, hybrid, or in a dedicated VPC) and select the right open-source or self-hosted components to minimize vendor lock-in risk without sacrificing performance.

And through data governance, we make sure the RAG system’s permissions always stay in sync with the source systems, instead of living as an independent rules layer that drifts out of date over time.

Conclusion

Securing a RAG architecture isn’t about adding instructions to the prompt and hoping the model behaves: it’s about designing the system so it’s structurally impossible to retrieve what a user shouldn’t see, and deploying every component (embeddings, vector index, inference) in an environment whose level of control and sovereignty matches the actual sensitivity of the data. The alternative to closed commercial models isn’t always building everything from scratch: it’s understanding exactly what you’re giving up in exchange for the convenience of an API, and deciding that consciously instead of by default.

Frequently Asked Questions

Is it safe to use a shared vector index for multiple clients or departments?

It can be, if you apply strict metadata filtering by tenant inside the search engine itself, before the similarity search runs, rather than as a filtering step afterward. For highly sensitive data, a physically separate index per client remains the safer option.

What is indirect prompt injection in a RAG system?

It’s the manipulation of a model’s behavior through hidden instructions embedded in a document that the system retrieves and adds to the prompt, without the user having written those instructions directly. It’s one of the risks covered by the OWASP framework for LLM applications.

Is it mandatory to self-host the model to have a sovereign RAG architecture?

Not necessarily. A commercial model exposed through a private endpoint inside a VPC can be a reasonable choice in a hybrid deployment. What matters is that the data and the index stay under the organization’s control and that vendor coupling is minimized across the rest of the architecture.

How do you prevent a document deleted from the source system from staying accessible through the chatbot?

By syncing deletions and permission changes from the source system to the vector index through a continuous update process, so the corresponding embedding is deleted or marked as non-retrievable at the same moment the document stops being available or accessible at the source.

What's the advantage of a hybrid deployment over one that's fully on public cloud?

It lets you keep the most sensitive data and the vector index under the organization’s direct control, while still taking advantage of high-performing commercial models for generation, reducing both the risk of data leakage and total dependency on a single vendor.