Case studies / Cost and speed
Cost and speed
Caching, smaller or distilled models, batching and GPU serving to cut cost and latency.
Approach
60 write-ups
Tag:
-
MongoDBNewCost and speed
Optimizing Document Lookup: Making Atlas Search and Vector Search Up to 5x Faster
Problem Document lookup added extra planning work, slowing search queries and using too much CPU.
Solution They changed the lookup step to reuse query plans and share resources, cutting overhead and making $search and $vectorSearch up to five times faster.
Up to 5.5x faster queries
Read on mongodb.comDetails -
ExpediaNewCost and speed
How I Taught an AI to Read My Logs
Problem Engineers spent hours manually reading Spark execution plans to find slow or expensive jobs.
Solution Created an AI skill that automatically reads execution plans and table metadata, matches them to actual runtime behavior, and flags costly pipelines for optimization.
Daily compute cost dropped over 90%
Read on medium.comDetails -
NVIDIANewCost and speed
Start and Restore GPU Workloads in Seconds with NVIDIA Snapshot
Problem Cold‑start of LLM inference takes minutes, slowing scaling and recovery.
Solution Snapshot checkpoints a fully initialized GPU workload and restores it on new pods, letting inference start in seconds instead of minutes.
Read on developer.nvidia.comDetails -
ElasticNewCost and speedFewer bits, same recall: More conservative auto-calibration sweeps for DiskBBQ
Problem Vector search needed cheaper quantization without hurting recall.
Solution Elastic adjusted its cost model to favor document bits and added preconditioning, enabling lower‑bit encodings and less reranking while keeping recall stable.
Recall@10 within 3 points, up to 11‑point gain
Read on elastic.coDetails -
DatabricksNewCost and speed
Accelerate GPU training on Unity Catalog tables with UCDataset
Problem GPUs spent time idle waiting for data, raising training cost and time.
Solution Created UCDataset, a native PyTorch loader that streams Unity Catalog tables directly to each GPU, eliminating copies and speeding data delivery.
10x faster training
Read on databricks.comDetails -
VercelNewCost and speed
Workflow SDK now writes over WebSockets on Vercel
Problem Many HTTP requests slowed down streaming AI token output and raised costs.
Solution Switched event and stream writes to a persistent WebSocket, cutting request overhead and delivering token streams 4.2× faster.
4.2× faster response
Read on vercel.comDetails -
NVIDIACost and speed
Speed Up Image Generation with FLUX 3 Image Klein on NVIDIA Blackwell GPUs
Problem Image generation was too slow for design tools and workflows.
Solution They optimized the FLUX 3 Image Klein model with TensorRT, fused kernels, multi‑GPU parallelism, FP8/NVFP4 quantisation and caching, cutting latency dramatically.
Up to 4.45× faster generation
Read on developer.nvidia.comDetails -
NVIDIACost and speed
Accelerating Reinforcement Learning Weight Refit with the Reshard Collective
Problem Updating model weights on inference GPUs was slow and used too much network bandwidth.
Solution They created a new Reshard collective that moves only needed weight shards directly between training and inference GPUs, cutting redundant data transfer.
Read on developer.nvidia.comDetails -
VercelNewCost and speed
Kimi K3.1 Pro now has more ZDR providers on AI Gateway
Problem Users wanted a cheaper, privacy‑safe way to use Kimi models without extra fees or data leaks.
Solution Vercel added more Zero Data Retention providers to its AI Gateway, routing requests only to compliant partners and removing platform fees, so teams can use Kimi models cost‑effectively and securely.
Read on vercel.comDetails -
VercelNewCost and speed
How Mercor ships internal apps and billions of production tokens on one platform
Problem Batch jobs were stuck for up to 12 hours because model rate limits caused retries.
Solution They added Vercel AI Gateway to route requests across providers, adding capacity and failover, reducing batch time to 2‑3 hours.
Batch jobs finish in 2‑3 hours
Read on vercel.comDetails -
NVIDIANewCost and speed
Run AI Models for Edge Application Development with Just a Few Commands
Problem Developers struggled to run large AI models on embedded devices due to heavy setup and slow inference.
Solution NVIDIA created TensorRT Edge-LLM, a lightweight C++ runtime that builds optimized inference engines on DGX Spark, enabling quick deployment on Jetson and DRIVE platforms with speculative decoding for faster generation.
Research behind this, on AI Papers DailySpecPilot: Autonomous Algorithm Discovery and Systems Optimization for Speculative DecodingRead on developer.nvidia.comDetails -
NVIDIANewCost and speed
Build a Validated Kubernetes Runtime for NVIDIA Dynamo
Problem Deploying AI inference on Kubernetes was slow and error‑prone because many components had to be set up manually.
Solution NVIDIA created AICR, a validated recipe that bundles Dynamo with all required GPU drivers, scheduling, networking and monitoring, so teams can deploy and test models quickly.
Read on developer.nvidia.comDetails -
AmazonNewCost and speed
Speeding up language models for travel applications
Problem Large language models were too slow and expensive for travel site queries, hurting user experience and revenue.
Solution They trained a smaller model using knowledge distillation and added Medusa heads to predict several tokens at once, cutting cost and latency dramatically.
Cost under one‑third, up to 20× faster
Research behind this, on AI Papers DailyDForkTree: Accelerating Speculative Decoding with Fork-Aware Block Diffusion Draft TreesRead on amazon.scienceDetails -
VercelNewCost and speed
Introducing Vercel CI
Problem CI runs full pipelines for every tiny change, making developers and bots wait too long.
Solution Built Vercel CI that only runs the tasks affected by a code change, reuses cached results, and gives each task just enough compute, cutting wait time 2‑6×.
CI passes 2-6x faster
Read on vercel.comDetails -
VercelNewCost and speed
Vercel Agent sessions now cost 67% less
Problem Agent sessions used many tokens, making them expensive for users.
Solution They redesigned context handling and improved prompt‑cache reuse, so each session needs fewer tokens and runs cheaper.
67% cost reduction
Read on vercel.comDetails -
NVIDIANewCost and speed
3 Ways to Boost MoE Inference on NVIDIA DGX Spark
Problem MoE inference was slow because reading model weights limited token generation speed.
Solution They replaced the MoE kernel with a runtime‑compiled version that caches binaries and reduced weight traffic, letting the model generate more tokens per step.
Research behind this, on AI Papers DailyTableTune: Measured, Self-Correcting Draft-Length Tables for Dynamic Speculative DecodingRead on developer.nvidia.comDetails -
AmazonCost and speed
The science behind TorchNeuron, Amazon’s new PyTorch backend for AWS Trainium chips
Problem Developers needed to run PyTorch models on Trainium without rewriting code, while keeping speed and cost low.
Solution Created TorchNeuron, a backend that translates PyTorch ops to Trainium, adds multi‑level caching and adaptive eager grouping to boost performance.
Read on amazon.scienceDetails -
Hugging FaceCost and speed
How Specializing the Vocabulary of SLMs Can Cut Inference Cost by 77% in Real Applications
Problem Using a generic vocabulary made the model write too many tokens, raising inference cost.
Solution They redesigned the tokenizer to focus on token cost for the specific domain, keeping the same base model and training data.
77% cost reduction
Read on huggingface.coDetails -
CloudflareCost and speed
Cutting our inference bill by 68% by being obsessive about optimization
Problem Inference costs for code reviews kept rising, hurting the budget.
Solution They traced token usage, improved the reviewer code, switched models, and increased cache hits, cutting review cost by 68% and speeding up reviews.
Cost per review $0.77
Read on blog.cloudflare.comDetails -
AppleCost and speed
Local and Distributed Large LLM Inference with MLX and RDMA on Mac Studio
Problem Running large language models locally was too slow or required expensive cloud resources.
Solution They used Apple’s MLX framework and RDMA over Thunderbolt to run trillion‑parameter models on one or multiple Mac Studios, achieving fast token generation.
36 tokens/second on 4 Mac Studios
Read on machinelearning.apple.comDetails -
NubankCost and speed
Squeezing ~2× MFU out of our training runs
Problem Training large Transformer models was slow and under‑utilized the hardware.
Solution They rewrote the attention kernel, integrated it with the compiler, and tuned the optimizer, which doubled the model‑flops‑utilization during training.
~2× MFU improvement
Read on building.nu.comDetails -
AmazonCost and speed
Can we change an AI model’s architecture without retraining it from scratch?
Problem Training new hybrid model architectures costs huge GPU time and money.
Solution They introduced priming, which reuses pretrained Transformer weights to quickly convert them into hybrid models, cutting pretraining cost by over 99%.
Read on amazon.scienceDetails -
AppleCost and speed
Minions: A Dynamic and Distributed Virtual GPU Cloud Built on Idle Apple Silicon Devices
Problem Teams needed more GPU power for inference but buying new hardware was expensive and slow.
Solution Created Minions, a system that gathers idle Apple Silicon GPUs from Macs, schedules inference jobs across them, and handles failures to boost compute capacity.
Read on machinelearning.apple.comDetails -
StripeCost and speed
Why I tried to kill token billing (and why we kept it)
Problem Customers needed a way to track AI usage costs without exposing token details in invoices
Solution Built a token‑billing system that records model usage, applies markups, and lets teams monitor costs, while keeping pricing logic separate from the invoice
Read on stripe.comDetails -
AmazonCost and speed
A kernel-centric path to real-time video generation on Trainium
Problem Users needed real‑time video generation with low latency and high efficiency at scale.
Solution They engineered a kernel‑centric pipeline on AWS Trainium, optimizing inference scheduling, bin‑packing, and global networking to serve video frames quickly.
Read on amazon.scienceDetails -
NVIDIACost and speed
Simplifying Model Serving Across Multiple GPUs with NVIDIA TensorRT Multi-Device Integration in NVIDIA Dynamo-Triton
Problem Single GPU cannot handle large generative AI models fast enough, causing high latency for users.
Solution They added TensorRT multi‑device inference to Triton, letting one model run on many GPUs via NCCL, so requests stay simple while latency drops.
Read on developer.nvidia.comDetails -
NVIDIACost and speed
Benchmarking LLM Inference at Scale with AIPerf
Problem Teams couldn’t trust their own speed tests because tools became bottlenecks and were hard to reconfigure.
Solution Created AIPerf, a load client that fully saturates LLM servers, runs fast, and lets users swap models with a single flag while reporting reliable latency and throughput.
Read on developer.nvidia.comDetails -
CREDCost and speedScaling ML Model Inference through architectural separation
Problem Inference latency and operational cost grew as more models and services were added.
Solution They split model execution from request orchestration into separate Hosting and Serving layers, routing requests centrally to reduce hops and improve efficiency.
Read on engineering.cred.clubDetails -
NVIDIACost and speed
How NVIDIA Groq 3 LPX Deterministic Execution Drives Power-Efficient High-Interactivity Inference on NVIDIA Vera Rubin
Problem AI factories have limited power budgets, restricting how many GPUs can be used.
Solution Created a deterministic execution model that predicts power draw per cycle, letting the system tighten voltage margins and fit more GPUs, boosting token throughput.
40% more GPUs, 35% higher token throughput
Read on developer.nvidia.comDetails -
ShopifyCost and speed
Native is now the future of mobile at Shopify
Problem Building mobile apps in both Swift and Kotlin doubled development effort and cost.
Solution Shopify used advanced LLM agents to generate native Swift and Kotlin code, making native development as fast and cheap as using React Native.
Read on shopify.engineeringDetails -
NVIDIACost and speed
How Full-Stack NIM Optimizations Deliver 2.5x More Users on Nemotron 3 Ultra
Problem Teams needed to serve many simultaneous users on limited GPUs without slowing down responses.
Solution They created NIM, a packaged serving stack that tunes precision, batching, scheduling, and caching to boost throughput while meeting latency goals.
Read on developer.nvidia.comDetails -
NVIDIACost and speed
High-Throughput Structure Prediction with BioNeMo Inference Runtime
Problem Scientists needed to predict millions of protein structures fast enough for proteome‑scale projects.
Solution They created BioNeMo Inference Runtime that uses optimized GPU kernels, CUDA graphs, and Ray to run many predictions in parallel, boosting throughput.
Processed 31 M candidates, releasing 1.81 M high‑confidence structures
Read on developer.nvidia.comDetails -
WeaviateCost and speed
HFresh: Memory-Efficient Vector Search
Problem Users needed fast vector search but had limited memory and large datasets.
Solution They built HFresh, a disk‑based index that partitions vectors into small on‑disk postings and uses an in‑memory centroid index to limit disk reads, keeping memory low while maintaining high recall.
Read on weaviate.ioDetails -
HoneycombCost and speed
Wide Events vs. Three Pillars: AI Observability Costs
Problem AI agents were driving up observability costs because metrics, logs, and traces were stored separately.
Solution They switched to wide events—a single data format that captures metrics, logs, and traces together—so all telemetry fits in one storage layer and reduces overhead.
Read on honeycomb.ioDetails -
SpotifyCost and speed
Portal by Spotify cut my Claude Code token usage by 90%
Problem Users spent too many tokens on simple file reading tasks.
Solution They created two lightweight agent modes that handle I/O with a cheaper model, reserving the expensive model for complex reasoning.
Read on engineering.atspotify.comDetails -
GitHubCost and speed
How we make AI coding more cost efficient without sacrificing task quality
Problem AI coding features were expensive to run, raising costs for users and limiting scalability.
Solution They added caching, switched to smaller distilled models, and batched requests, cutting compute while keeping answer quality unchanged.
Read on github.blogDetails -
ReplitCost and speed
Intelligent Model Routing on Replit
Problem Users had to choose which AI model to use, adding extra decisions and cost uncertainty.
Solution Replit created Intelligent Model Routing that automatically picks the best model for each task, using performance data to balance quality, speed, and cost.
65% lower cost
Read on replit.comDetails -
Hugging FaceCost and speed
Up to 3.2x Faster Inference with LFM2.5-DSpark
Problem LLM inference was slow, causing high latency for users.
Solution They added a lightweight draft model that predicts tokens first, then the main model verifies them, cutting memory reads and speeding up decoding.
Up to 3.2x faster inference
Research behind this, on AI Papers DailyForeSpark: Joint Draft and Verification Budget Scheduling for Parallel Speculative DecodingRead on huggingface.coDetails -
ShopifyCost and speed
Gisting: Compressing LLM Agent context to ↑ throughput and ↓ cost
Problem Long prompts made the agent slow and expensive to run.
Solution They replaced thousands of prompt tokens with a few learned “gist” tokens, training the embeddings by distilling from the full prompt so the model behaves the same.
Latency dropped from 6.8s to 4.2s, throughput rose from 20.2 to 23.4 QPS
Read on shopify.engineeringDetails -
ExpediaCost and speed
How Keras 3 Helped Modernise Expedia Group’s Lodging Ranking Stack
Problem Slow training and high latency slowed new ranking model releases.
Solution They rewrote the training and serving pipelines with Keras 3, modern GPU optimisations, and cleaner APIs, making training 30% faster and cutting P99 inference latency by two‑thirds.
Read on medium.comDetails -
Hugging FaceCost and speed
Build Low-Latency Multilingual Voice Agents: Open Weights & Full Deployment Control with NVIDIA Magpie TTS
Problem Voice responses were too slow, making the whole conversation feel laggy for users.
Solution They built a modular speech pipeline using NVIDIA Magpie TTS with open weights, letting them run ASR, LLM, and TTS on their own hardware and tune each part for speed.
Research behind this, on AI Papers DailyDiscovering Interpretable Acoustic Features in Autoregressive Voice-Cloning TTS ModelsRead on huggingface.coDetails -
RazorpayCost and speed
CI Doesn’t Need On-Demand: Moving Our Build Pipelines to Spot Instances
Problem CI pipelines were too expensive or unreliable when using cheap spot instances.
Solution Created a self‑healing Kubernetes layer for GitHub Actions that runs most jobs on spot instances, detects terminations, retries jobs and cleans up pods automatically.
99.2% job success, 70–90% cost reduction
Read on engineering.razorpay.comDetails -
FaireCost and speed
Virtual Packer: How Faire estimates shipping costs at checkout
Problem Boutique owners need accurate shipping cost at checkout, but actual rates are unknown until after shipping.
Solution They built a Virtual Packer that predicts package weight and count, then queries live carrier rates to show a real‑time estimate.
35% fewer estimation errors
Read on craft.faire.comDetails -
Hugging FaceCost and speed
Bringing Nunchaku 4-bit Diffusion Inference to Diffusers
Problem Large diffusion models need too much GPU memory and run slowly on consumer GPUs.
Solution Integrated 4-bit weight and activation quantization (SVDQuant) into Diffusers, letting users load Nunchaku checkpoints without custom code, cutting memory and speeding inference.
12 GB memory, 1.7 s per 1024×1024 image
Read on huggingface.coDetails -
RampCost and speedOnline Learning for Cost-Efficient LLM Routing
Problem Requests were slow and expensive because the system chose suboptimal LLM models.
Solution They built an LLM gateway that uses Thompson sampling on latency posteriors to route each request to the fastest, most reliable model.
Research behind this, on AI Papers DailyBallad: Bandit-Based LLM Routing for Automated Heuristic DiscoveryRead on builders.ramp.comDetails -
ExpediaCost and speed
Using LLMs to Analyze Spark SQL Plans: A Practical Approach to Debugging Long-Running Jobs
Problem Engineers spent hours debugging slow Spark jobs without clear clues, leading to over‑provisioned clusters and higher costs.
Solution They built an LLM‑driven tool that reads structured Spark SQL plans, matches them against defined anti‑patterns, and returns JSON‑formatted fixes tied to specific plan nodes.
Read on medium.comDetails -
GoogleCost and speed
Accelerating Gemini Nano models on Pixel with frozen Multi-Token Prediction
Problem On-device LLMs were too slow and drained battery, hurting user experience.
Solution They added Multi-Token Prediction to frozen Gemini Nano models, letting the phone generate several words at once and cut energy use.
Research behind this, on AI Papers DailyPhrase-Level Speculative Decoding on a Frozen TargetRead on research.googleDetails -
monday.comCost and speedMorphex Monthly: The Month I Learned to See Myself
Problem The team couldn't measure how much the AI agent cost or how often it failed.
Solution They created instrumentation and a dashboard that logs every model call, retry, migration and success rate, showing monthly cost and performance.
Read on engineering.monday.comDetails -
ExpediaCost and speed
The Most Expensive Milliseconds Are Unmeasured
Problem Users felt login screens were slow, eroding trust without obvious errors.
Solution They created a new metric, Native Time to Interactive (NTTI), to measure when key screen components load and become usable, turning performance into a first‑class signal.
Read on medium.comDetails -
IndeedCost and speed
Distilling Long-Tail User Behavior into Scalable Embeddings for Job Search
Problem Online models needed fast, cheap scoring for millions of users, but could not use large sequence models.
Solution They trained a heavy sequence model offline on full user histories, then distilled its knowledge into compact user embeddings that online models can use efficiently.
Read on engineering.indeedblog.comDetails -
SlackCost and speed
Slack AI: The Path to Multi-Cloud
Problem Users needed faster, more reliable LLM responses without high infrastructure risk.
Solution Built a multi‑cloud system that runs enterprise LLMs across several providers, balancing load to cut latency and avoid outages.
Read on slack.engineeringDetails -
Hugging FaceCost and speed
Building Blocks for Foundation Model Training and Inference on AWS
Problem Running huge foundation models on AWS was expensive and hard to manage.
Solution They created a layered stack with AWS compute, Slurm/Kubernetes orchestration, PyTorch/JAX training, and Prometheus/Grafana monitoring to use resources efficiently.
Read on huggingface.coDetails -
ShopifyCost and speed
Autoresearch isn’t just for training models
Problem CI runs take 30 minutes, delaying developers from knowing if their changes break things.
Solution They built an Autoresearch loop that acts like an AI researcher, automatically tries performance tricks and searches internal docs to speed up build and test times.
Read on shopify.engineeringDetails -
KlarnaCost and speed
Beyond Prompting: How Algorithmic Evolution Doubled our Training Speed
Problem Training large models took too long, slowing down product updates and increasing expenses.
Solution They used an evolutionary code optimizer to automatically rewrite training code, testing thousands of variants and keeping the fastest, which doubled training speed.
Training speed doubled
Read on engineering.klarna.comDetails -
Fly.ioCost and speedRun a Local LLM on a CPU With BitNet and a Sprite
Problem Need cheap, fast LLM inference without GPUs for bursty workloads.
Solution Deployed a 2B‑parameter BitNet model quantized to 1.58‑bit on Fly.io Sprites, using on‑demand loading and checkpointing to serve HTTP requests only when needed.
Read on fly.ioDetails -
JioHotstarCost and speed
Monetisation : Multi-period DASH x ExoPlayer
Problem Users needed to add region‑specific ads and legal text without re‑encoding each video, which was slow and expensive.
Solution They rewrote ExoPlayer’s ad handling to support multi‑period DASH, letting the player insert new content blocks on the fly.
Read on blog.hotstar.comDetails -
LyftCost and speed
Scaling Localization with AI at Lyft
Problem Users needed new languages quickly, but human translation was slow and expensive.
Solution They built an AI‑driven translation pipeline that drafts translations with LLMs, then lets linguists review them, keeping quality high while cutting turnaround to minutes.
30‑minute SLA for 95% of translations
Read on eng.lyft.comDetails -
GoogleCost and speed
Sequential Attention: Making AI models leaner and faster without sacrificing accuracy
Problem Large models run slowly and cost too much, hurting product speed and budget.
Solution They created Sequential Attention, a greedy algorithm that picks the most useful layers or features during training, trimming the model without losing accuracy.
Read on research.googleDetails -
GoogleCost and speed
Small models, big results: Achieving superior intent extraction through decomposition
Problem Mobile agents needed to understand user intent without sending data to servers, which is slow, expensive, and risky.
Solution They split intent detection into two steps: a tiny multimodal model summarizes each screen, then another small model reads the summaries to predict overall intent, all on‑device.
Read on research.googleDetails -
ShopifyCost and speed
Tangle: An open-source ML experimentation platform built at Shopify scale
Problem Data scientists spent weeks re‑running experiments because pipelines weren’t reproducible or shareable.
Solution Created Tangle, a visual ML experimentation platform that caches pipeline results, tracks versions, and runs in the cloud, cutting redundant compute.
Saved over a year of compute time
Read on shopify.engineeringDetails
No write-ups match these filters.