Zama has released a working demonstration that runs parts of a GPT2 language model on fully encrypted data, using fully homomorphic encryption (FHE) to keep both user queries and model weights private. The company published code showing how to adapt Hugging Face’s transformers library so that a single attention head of GPT2 executes on encrypted inputs.
The timing is notable. Building an LLM is expensive — Zama points out GPT3 cost roughly $4.6 million to train — and on-premise deployment risks leaking model IP. Healthcare, finance, and legal industries have been locked out of cloud LLM APIs precisely because sending sensitive text to a third-party server is a non-starter. FHE sidesteps the trade-off entirely: the server never sees the user’s data, and the user never receives the model weights.
The implementation works by splitting inference between client and server. The client runs GPT2 locally up to the first attention layer, encrypts the intermediate tensors, and ships them to the server. The server performs the attention computation on encrypted values using Concrete-Python, Zama’s FHE compiler, then returns encrypted results for the client to decrypt and continue inference locally.
Getting this to work required quantizing both model weights and activations to integers, since TFHE — the underlying encryption scheme — represents values as integers and handles nonlinear operations through Programmable Bootstrapping table lookups. Zama’s experiments show 4-bit quantization holds onto 96% of the original model’s accuracy on a dataset of roughly 80 sentences. That’s a real number, not a marketing claim.
What the blog doesn’t address is latency. Programmable Bootstrapping dominates computation time in FHE circuits, and running even a single attention head this way is dramatically slower than plaintext inference. The demo proves feasibility, not practicality. Still, for industries where privacy is legally mandated, a slow-but-private LLM beats no LLM at all.
