---
# How does Mixture of Experts cut LLM inference costs?
**URL:** https://www.sigmoid.com/blogs/how-does-mixture-of-experts-cut-llm-inference-costs/
Date: 2026-11-13
Author: Joha Momin
Post Type: post
Summary: Key takeaways A dense model runs every parameter on every token, so doubling capability roughly doubles the cost of serving each request....Read More...
Categories: AI/ML
Tags: AI infrastructure, Dense vs MoE models, LLM inference cost, Mixture of Experts, MoE architecture
Featured Image: https://www.sigmoid.com/wp-content/uploads/2026/11/Cost-of-mixture-of-experts-model-to-run-blog.png
---
## Key takeaways
- A dense model runs every parameter on every token, so doubling capability roughly doubles the cost of serving each request.
- Mixture of Experts breaks the link between how large a model is and what each request costs, by running only a fraction of the network per token.
- Open models have settled whether it works. Mistral’s Mixtral matches the quality of a dense model several times its size at a fraction of the cost to serve.
- The saving is in computation, not hardware. Every expert stays loaded in memory, so an MoE model can need more GPU capacity than its active-parameter figure suggests.
- Expert specialization emerges during training and cannot be configured, so no setting routes a chosen category of input to a chosen expert.
- MoE pays off on high-volume, multi-domain traffic where cost per request dominates. Dense models remain the better answer for single-request latency, edge deployment, and smaller teams.
Most conversations about large language models center on what they can do. The harder question, for anyone running one in production, is what it costs to keep doing it. Training is a one-time expense. It can be budgeted, amortized, and then largely forgotten. Inference is the opposite, because it recurs on every request for the life of the deployment and grows with adoption rather than with ambition. For most of the last decade those two costs rose together with capability, which meant a better model was also a permanently more expensive one.
The reason was structural. In a dense model, every parameter participates in every forward pass. Double the parameter count and the computation needed per token roughly doubles with it, so a model ten times more capable arrives with a ten-times-longer training run and a ten-times-larger inference bill. A less obvious inefficiency sits underneath it. Answering a simple question costs exactly as much as answering a hard one, because the whole network runs either way.
Mixture of Experts is the architecture that broke that pattern, and it is now the default at the frontier. What follows covers how it changes the arithmetic, what the open-model evidence establishes, the three places where the promised savings do not land, and how to tell which workloads justify the architecture.
## What changes when only part of the model runs
Mixture of Experts is not a new idea. It was published in 1991, decades before transformers, and stayed a minor technique until models grew large enough for its economics to matter. [3] Instead of one network handling everything, an MoE model is split into sub-networks called experts, each loosely specializing in different kinds of content. A small learned layer called the router, sometimes a gating network, scores every incoming token against every expert and picks the best one or two. Only the chosen experts process that token, and the others perform no computation on it at all. Their outputs are then combined into a single result, weighted by the scores the router gave them.
The research literature calls this conditional computation. The commercial translation is shorter: the model’s total parameter count stays large, while the number of parameters switched on for any one token stays small. Cost per request scales with the number switched on, not with the total.

Fig.1: How one token gets routed
## The evidence from open models
GPT-4’s architecture has never been published, so the dependable evidence comes from open models. Mistral’s Mixtral 8x7B, released in 2023, is the clearest public case. It holds eight experts of roughly 7B parameters each and activates two per token, which works out to about 14B active parameters against 47B total. Under a third of the model performs any computation on a given token. On most benchmarks it matched or exceeded Llama 2 70B, a fully dense model five times larger in active parameters, at a fraction of the cost to serve.[2]
Dimension
Dense model
MoE model
Total parameters
Fully active
Large, for example 47B
Active parameters per token
All of them, for example 70B
A subset, for example 14B
Training cost
High
Moderate to high
Inference cost
Scales with total parameters
Scales with active parameters
Memory footprint
Proportional to total parameters
Large, since all experts stay loaded
Parallelism needs
Simpler
Requires expert parallelism
Load balancing
Not applicable
A critical engineering challenge
Interpretability
Challenging
Slightly more modular
Table 1: Dense and MoE models compared across the dimensions that affect deployment.
## The three costs the active-parameter figure hides
The 14B active-parameter figure is the number vendors quote. It is accurate about computation, and it is also the number most likely to be misread, because three real costs sit outside it.
- Memory: Memory is the first and the most expensive to get wrong. It does not shrink alongside computation. Every expert has to stay resident so the router can dispatch to any of them, which means all eight of Mixtral’s expert weight matrices occupy memory whether or not a given token ever reaches them. An MoE model therefore often needs more memory than an equally capable dense model, and provisioning follows the total, not the active count. A team that sizes hardware off the 14B figure will come up short.
- Latency: Large MoE models distribute experts across accelerators, so the router ships tokens between devices, and the Switch Transformer work identifies that all-to-all traffic as a dominant engineering problem.[1] Managed badly, network latency consumes the savings that justified the architecture. The practical consequence is that MoE improves throughput without necessarily improving response time, and a single request may come back no faster than before.
- Quality of the model: This depends on training discipline far more heavily than a dense model. A router left to its own preferences sends most traffic to a few experts while the rest sit idle and learn nothing, a failure known as expert collapse. Two mechanisms hold it off. One is an auxiliary loss that penalizes concentrating tokens on the same few experts. The other is the capacity factor introduced in the Switch Transformer work, which caps how many tokens an expert will accept per batch and reroutes or drops the overflow.[1] Without them, an organization pays to store parameters that contribute nothing. The expert count in a model specification says nothing about whether all of them learned to do useful work. That is a question about [how model quality gets measured](/blogs/what-your-llm-judge-is-measuring-and-what-it-is-not/), and no architecture diagram answers it.
## Why experts cannot be assigned to a domain
What the experts do learn is real, but it is not something an architect chooses. Research on both Switch Transformer and Mixtral found genuine structure in how tokens get routed, with certain experts preferentially handling punctuation, numbers, particular languages, or code as against prose. [1][2] The specialization is emergent, not designed. Nothing in the training objective instructs a given expert to take on French text, and the pattern appears because grouping those tokens is an efficient division of labor that training converges on.
The practical limit follows from that. There is no setting that sends one category of input to one chosen expert, because the router’s behavior came out of training and not out of configuration. A tendency measured today may not survive the next retrain either, since nothing in the objective was holding it there. Any design that needs a particular expert to handle a particular domain is resting on a pattern the model was never told to keep. For teams that have to account for how an output was produced, that is an [explainability constraint](/blogs/the-genai-adoption-triad-responsibility-ethics-and-explainability/) rather than a configuration gap.
## Which workloads justify the architecture
The costs and limits stated earlier do not rule MoE out. They decide which workloads it suits, and the answer turns on request volume, latency tolerance, and available GPU memory.
Choose MoE when
- Request volume is high enough that cost per request dominates. MoE delivers more capability for every unit of computation spent, which is why it suits throughput-bound work such as [extracting structured attributes from product labels at scale](/case-studies/intelligent-label-processing-with-ai-and-llms-improves-compliance/), where total cost matters more than how fast any single request returns. Mixtral 8x7B on two A100s often beats Llama 2 70B on a single A100 on cost per unit of quality.
- The workload spans several languages or domains. Emergent specialization lets different content types reach different parts of the model, which is where a loose division of labor pays off.
- GPU memory is available and the cost per token needs to come down. That is the direction the trade runs, and it does not work in reverse.
Stay dense when
- Latency on a single request matters more than total throughput. Routing overhead may never pay for itself in an interactive application tuned for one query at a time.
- The model has to run at the edge. Loading only the active experts on-device takes complex engineering, so a smaller dense model is usually more practical.
- The team is small. Dense models are considerably easier to debug, which matters when the alternative brings expert-parallel infrastructure, load-balancing tuning, and a wider set of failure modes.

Fig. 2: How much of each model actually runs
## Where sparse routing goes next
Fine-grained MoE replaces a handful of large experts with many small ones, taken to 128 experts in Snowflake Arctic.[4] Mixture-of-depths applies the same conditional logic to how many layers a token passes through, letting simpler inputs take shortcuts. Multimodal MoE routes vision and language tokens to different expert types altogether. All of them rest on the same observation: not all computation is equally valuable for every input.
## Conclusion
Sparse routing did not remove the cost of scale. It moved the cost somewhere less visible. Compute per request fell, while memory, distributed infrastructure, and training discipline all rose, and the figure that advertises the architecture is the same one most likely to mislead a team sizing hardware for it. That changes the question in front of an engineering leader. Affordability was the old constraint, and sparse routing largely settled it. What remains is harder to buy: an accurate picture of the traffic a model will carry, because the architecture that suits a million small requests a day is seldom the one that suits a thousand demanding ones.
## References
1. Fedus, W., Zoph, B., & Shazeer, N. (2022). Switch Transformers: Scaling to Trillion Parameter Models with Simple and Efficient Sparsity. Journal of Machine Learning Research, 23(120), 1–39. [https://arxiv.org/abs/2101.03961](https://arxiv.org/abs/2101.03961)
2. Jiang, A. Q., Sablayrolles, A., Roux, A., Mensch, A., Savary, B., Bamford, C., et al. (2024). Mixtral of Experts. arXiv preprint arXiv:2401.04088. [https://arxiv.org/abs/2401.04088](https://arxiv.org/abs/2401.04088)
3. Jacobs, R. A., Jordan, M. I., Nowlan, S. J., & Hinton, G. E. (1991). Adaptive Mixtures of Local Experts. Neural Computation, 3(1), 79–87.
4. Snowflake AI Research (2024). Snowflake Arctic: The Best LLM for Enterprise AI. [https://www.snowflake.com/blog/arctic-open-efficient-foundation-language-models-snowflake/](https://www.snowflake.com/blog/arctic-open-efficient-foundation-language-models-snowflake/)
## About the Author
**Megha Shyam Chintagunti** is an Associate Data Scientist at Sigmoid. Specializing in Generative AI, LLM-powered applications, and multi-agent systems, he builds AI solutions that translate complex business problems into production-ready products. Most recently in Sigmoid's Center of Excellence (CoE), he developed an AI-to-BI accelerator to help teams move rapidly from raw data to actionable business insights, earning a reputation for turning ambiguous challenges into reliable, measurable outcomes.
[lc_the_tags]
## Suggested readings
[View all](/blogs/)

#### What your LLM judge is measuring and what it is not
[Read blog](/blogs/what-your-llm-judge-is-measuring-and-what-it-is-not/)

#### Scaling Enterprise AI with LLMOps on Databricks
[Read blog](/blogs/scaling-enterprise-ai-with-llmops-on-databricks/)

#### Top 5 model training and validation challenges that can be addressed with MLOps
[Read blog](/blogs/model-training-and-validation-challenges-with-mlops/)
## Featured blogs
[lc_get_posts post_type="post"
posts_per_page="4" orderby="date" output_view="lc_get_posts_mycustom_view" output_number_of_columns="4"
output_wrapper_class="row" output_article_class="shadow" output_hide_elements="Excerpt"
output_excerpt_length="0" output_excerpt_text="Read More" output_heading_tag="span"
output_featured_image_format="thumbnail" output_featured_image_class="card-img-left" ]
## Share
[addtoany]
## Subscribe to get latest insights
## Talk to our experts
Get the best ROI with Sigmoid’s services in data engineering and AI
---
## Categories
- AI/ML
---
## Navigation
- [Company](/about-sigmoid)
- [Newsroom](/newsroom)
- [Life at Sigmoid](/careers)
- [Takshashila](/takshashila)
- [Contact Us](/contact-us)
- [AI Strategy Blueprint your AI advantage](/enterprise-ai-strategy/)
- [Generative AI Drive innovation with Generative AI](/generative-ai/)
- [Responsible AI Build trust with ethical AI practices](/responsible-ai-in-enterprise/)
- [Agentic AI Reshape business with scalable agentic systems](/agentic-ai-solutions/)
- [AI Managed Services Ensure reliable AI performance](/ai-managed-services/)
- [Advanced Analytics Transform your business with data-driven insights](/advanced-data-analytics-solutions/)
- [Download whitepaper](/ebooks-whitepapers/the-new-qsr-playbook-transforming-menu-strategy-into-a-growth-engine/)
- [Data Strategy Strong data foundations for scalable AI](/data-analytics-strategy/)
- [Data Management Leverage data as a strategic asset](/ai-data-management-services/)
- [DataOps Automate data for speed and quality](/data-devops/)
- [Forward Deployed Engineering Scale enterprise AI with FDEs & specialized agents](/forward-deployed-engineering/)
- [Data Engineering Deliver insights faster with scalable pipelines](/data-engineering/)
- [Cloud Transformation Modernize data to maximise efficiency](/cloud-migration/)
- [Download whitepaper](/ebooks-whitepapers/building-data-products-in-a-data-mesh-to-drive-business-value/)
- [Data Modeling Structure data for better decisions](/data-modeling-services/)
- [Data Visualization Transform data into actionable stories](/data-visualization-service/)
- [BI Migration Enhance decision making with modern BI tools](/bi-migration/)
- [Data Observability Build trust with healthy, accurate data](/data-observability/)
- [Automated Insights Make smarter decisions with auto-generated insights](/automated-insights/)
- [Download whitepaper](/ebooks-whitepapers/how-cognitive-bi-drives-an-intelligent-enterprise/)
- [CPG & Retail End-to-end analytics for planning, operations, and commercial excellence](/industries/cpg-analytics/)
- [Life Sciences Trusted intelligence across clinical, commercial, and operational workflows](/industries/life-sciences/)
- [Banking & Financial Services AI-powered analytics for risk, compliance and customer experience](/industries/banking-financial-analytics-services/)
- [Insurance Modernize core insurance platforms and unlock new revenue streams with AI](/industries/insurance-reimagined-with-ai/)
- [Download infographic](/wp-content/uploads/2026/03/Infographic_Maturity-Framework-for-Scaling-Agentic-AI.pdf)
- [Helios Turn marketing data into measurable business outcomes](/marketing-suite-helios/)
- [Stratos AI-powered platform for resilient supply chains](/supply-chain-stratos/)
- [DataGuard Agentic AI platform for data quality and governance](/data-quality-dataguard/)
- [Financial transformation with Agentic AI Agent-led automation for enterprise and GBS finance](/financial-transformation-with-agentic-ai/)
- [MediaIQ Advanced platform for in-flight marketing measurement](/accelerators/sigmoid-mediaiq-multi-touch-attribution-tool/)
- [CampaignIQ AI-driven platform for optimized campaign budget allocation](/accelerators/sigmoid-campaigniq/)
- [AssistBot GenAI email assistant that automates human-like responses](/accelerators/sigmoid-assistbot-for-ai-email-assistant/)
- [CreativeBot GenAI tool for personalized and brand-aligned creative design](/accelerators/sigmoid-creativebot/)
- [SocialBot GenAI platform to analyze digital conversations and trends](/accelerators/#marketing|socialbot)
- [DemandIQ Predict trends accurately and optimize inventory management](/accelerators/sigmoid-demandiq/)
- [NetworkIQ Track and optimize logistics operations in real-time to quickly address disruptions](/accelerators/sigmoid-networkiq/)
- [SupplyIQ End-to-end platform to optimize supply chain operations](/accelerators/sigmoid-supplyiq/)
- [ProcurementIQ Automated procurement operations for maximum savings, compliance and efficiency](/accelerators/sigmoid-procurementiq/)
- [RapidML Accelerated deployment for machine learning models](/accelerators/sigmoid-rapidml/)
- [DataGuard Comprehensive platform for proactive data quality management](/accelerators/data-quality-tool-sigmoid-dataguard/)
- [CloudPulse Cloud cost optimization platform with multi-cloud management](/accelerators/sigmoid-cloudpulse/)
- [RAPID GenAI foundation with built-in governance and cost clarity](/accelerators/sigmoid-rapid/)
- [AnalyticsBot Conversational analytics for faster business decisions](/accelerators/sigmoid-analyticsbot/)
- [DataConnect Seamlessly ingest, integrate and harmonize data from diverse sources](/accelerators/sigmoid-dataconnect/)
- [Reconica AI-powered data harmonization and reconciliation engine](/accelerators/sigmoid-reconica/)
- [ConverseBot GenAI driven insights generation for automated insights from reports](/accelerators/#sales|conversebot)
- [iNRM Cross-lever revenue growth optimization platform](/accelerators/sigmoid-inrm/)
- [AssortmentIQ Optimize shelf layouts and assortment mix at scale with AI-based insights](/accelerators/sigmoid-assortmentiq/)
- [Read whitepaper](/ebooks-whitepapers/building-agentic-ai-chatbots-for-business-process-transformation/)
- [Listen Podcast](/events/podcast/how-jack-in-the-box-is-redefining-personalization-and-supply-chain-with-ai/)
- [Blogs](/blogs/)
- [White Papers](/ebooks-whitepapers/)
- [Case Studies](/case-studies/)
- [Podcast](/events/podcast/#Podcasts)
- [Download case study](/case-studies/ai-based-dynamic-targeting-and-personalized-hcp-engagement-for-pharma-marketing/)
- [ConverseBot](/accelerators/#sales|conversebot/)
## Tags
- AI infrastructure
- Dense vs MoE models
- LLM inference cost
- Mixture of Experts
- MoE architecture
---
## Footer Links
- [Talk to our AI experts](/contact-us/)
- [AI Strategy](/enterprise-ai-strategy/)
- [Agentic AI](/agentic-ai-solutions/)
- [Generative AI](/generative-ai/)
- [AI Managed Services](/ai-managed-services/)
- [Responsible AI](/responsible-ai-in-enterprise/)
- [Advanced Analytics](/advanced-data-analytics-solutions/)
- [Data Strategy](/data-analytics-strategy//)
- [Data Engineering](/data-engineering/)
- [Data Management](/ai-data-management-services/)
- [Cloud Transformation](/cloud-transformation/)
- [DataOps](/data-devops/)
- [Forward Deployed Engineering](/forward-deployed-engineering/)
- [Data Visualization](/data-visualization-service/)
- [Automated Insights](/automated-insights/)
- [BI Migration](/bi-migration/)
- [Data Modeling](/data-modeling-services/)
- [Data Observability](/data-observability/)
- [CPG & Retail](/industries/cpg-analytics/)
- [Life Sciences](/industries/life-sciences/)
- [Banking & Financial Services](/industries/banking-financial-analytics-services/)
- [Insurance](/industries/insurance-reimagined-with-ai/)
- [Case Studies](/case-studies/)
- [Thought Leadership](/ebooks-whitepapers/)
- [Blogs](/blogs/)
- [Helios](/marketing-suite-helios/)
- [Stratos](/supply-chain-stratos/)
- [DataGuard](/data-quality-dataguard/)
- [Company](/about-sigmoid/)
- [Newsroom](/newsroom/)
- [Accelerators](/accelerators/)
- [Careers](/careers/)
- [Privacy Policy |](/privacy-policy/)
- [Cookie Policy](/cookie-policy/)