# MOAI: Module-Optimizing Architecture for non-interactive secure transformer Inference

**Authors:** Linru Zhang, Xiangning Wang, Jun Jie Sim, Zhicong Huang, Jiahao Zhong, Huaxiong Wang, Pu Duan, Kwok-Yan Lam  
**Published:** 10 Nov 2026  
**Links:** [arXiv](https://arxiv.org/abs/2611.13641) · [PDF](https://arxiv.org/pdf/2611.13641) · [This page](https://omen-bit.github.io/ai-paper-daily/paper/2611.13641.html)

## Key numbers

- **1.7×**: MOAI uses at least 1.7x fewer HE rotations compared to the state-of-the-art works across all matrix…
- **52.8%**: we achieve a 52.8 % reduction in evaluation time compared to the state-of-the-art HE-based non-interactive secure…
- **55.7%**: …MOAI on the Powerformer's framework and achieve a 55.7 % reduction in evaluation time compared to…

## Abstract

Privacy concerns have been raised in Large Language Models (LLM) inference when models are deployed in Cloud Service Providers (CSP). Homomorphic encryption (HE) offers a promising solution by enabling secure inference directly over encrypted inputs. However, the high computational overhead of HE remains a major bottleneck. To address this challenge, we propose MOAI, an efficient HE-based, non-interactive framework for secure transformer inference. MOAI gains significant efficiency improvement from: (1) a novel evaluation flow that combines column and diagonal packing with consistent strategies across all layers, eliminating expensive format conversions. (2) rotation-free algorithms for Softmax and LayerNorm that significantly reduce the number of costly HE rotations, removing 2,448 HE rotations in BERT-base inference. (3) Column packing removes rotations in plaintext-ciphertext matrix multiplications and interleaved batching further reduces the rotations in ciphertext-ciphertext matrix multiplications. MOAI uses at least 1.7x fewer HE rotations compared to the state-of-the-art works across all matrix multiplications of BERT-base. As a result, we achieve a 52.8 % reduction in evaluation time compared to the state-of-the-art HE-based non-interactive secure transformer inference, THOR (Moon et al., CCS 25). We then apply MOAI on the Powerformer's framework and achieve a 55.7 % reduction in evaluation time compared to Powerformer (Park et al., ACL 25), which approximates Softmax and LayerNorm with simpler functions in transformer and proposes HE-based non-interactive transformer inference. We report an amortized time of 2.36 minutes per input on a single GPU environment. We show the extendibility by applying MOAI in LLaMA-3-8B. Our implementation is publicly available as open source.

---

Notes written by AI from the paper, numbers copied from the abstract. Check the paper before you rely on them. From AI Papers Daily.