Kubernetes bills rarely arrive as one number. They arrive as compute, memory, storage, load balancers, data transfer, and logging charges, spread across teams that each believe someone else is watching. For a business owner, the useful question is which parts of this spend buy reliability and which parts buy nothing. Answer that honestly and the savings tend to follow without a late-night incident.
A Kubernetes node is a bundle of reserved capacity. When you set a container’s CPU or memory request, you’re telling the scheduler how much of that bundle to set aside for the pod. The scheduler packs pods onto nodes based on those requests, not on what the process actually uses. This is the first place money leaks.
If a service requests two CPUs and uses 0.3, you’re paying for 1.7 CPUs that no other workload can schedule onto that node. Do this across a few dozen deployments and the waste becomes a permanent line item. The opposite mistake is worse. Set requests far below real usage and the scheduler overpacks nodes. Pods compete for CPU, latency rises, and memory pressure leads to evictions or OOM kills. The bill looks great right up until customers notice.
CPU and memory behave differently, which matters for how much safety margin you need. CPU is compressible: a container that hits its CPU limit gets throttled and slows down. Memory is not compressible. A container that exceeds its memory limit gets killed. That asymmetry should shape your policy. Keep tighter CPU requests where latency can absorb some queuing, and leave more room around memory, especially for JVM services, caches, and anything with spiky allocation. The Kubernetes documentation on managing resources for containers explains how requests and limits drive scheduling, throttling, and eviction.
Aim for a configuration where requests reflect real demand plus a known buffer. The smallest requests are rarely the cheapest reliable choice. Measure before you change anything. Pull CPU and memory usage over at least two weeks, including a traffic peak, and compare the 95th percentile against current requests. A service that sits at 40 percent of its request most of the day may still need the full request at month-end close or during a product launch.
The Vertical Pod Autoscaler can produce recommendations from historical usage. Running it in recommendation-only mode is a low-risk way to find candidates. Be careful about letting it update production workloads automatically: applying new resource values usually requires restarting pods, and a restart during a traffic spike is exactly the wrong time to discover that your replica count was too low. Use recommendations as input, then make changes in reviewed batches.
A practical rule is to separate workloads by consequence. Batch jobs, internal tools, and asynchronous workers can run close to observed usage with modest buffers. Customer-facing request paths need enough headroom to survive a lost node or an unexpected surge. That difference is a business decision, not a purely technical one, because it trades compute spend against the cost of degraded service.
There are two autoscaling problems, and confusing them wastes money. The first is pod count: how many copies of a service should run right now. The Horizontal Pod Autoscaler adjusts replicas based on metrics. It depends on meaningful resource requests and a working metrics pipeline. If requests are nonsense, the autoscaler’s utilization math is nonsense too.
The second problem is node count. A node autoscaler adds or removes machines when pods cannot fit or nodes sit idle. It only helps if pods can actually be moved. That means disruption budgets, graceful termination, and enough replica spread to survive a node disappearing.
The tradeoffs here are concrete. Scale down too aggressively and you pay for cold starts, dropped connections, and a system that feels unstable every morning. Scale up too slowly and queues build before new capacity arrives. Set minimum replicas for baseline traffic, use stabilization windows to avoid flapping, and test scale-down during business hours rather than trusting the configuration. For node pools, keep a small buffer for sudden bursts, but don’t let “just in case” become the default reason for every oversized pool.
Spot or preemptible capacity can carry a steep discount, and it comes with a condition: the provider can take the machine back with short notice. That is a good trade for stateless web tiers with several replicas, CI runners, batch processing, and anything that can checkpoint or retry. It is a bad trade for a single-replica database with no failover, or for a stateful service whose recovery path has never been tested.
The architecture work is what makes the discount safe. Spread replicas across zones and instance types, handle termination signals, drain connections, and set PodDisruptionBudgets so voluntary disruptions don’t take out too much capacity at once. Priority classes can protect critical workloads from being preempted by lower-value ones, but they are not a substitute for redundancy. A priority value does not create a second replica.
Committed-use discounts and reserved capacity work in the other direction. They lower the price of the steady baseline you know you’ll run for a year, but they lock you in. Buy commitments for the floor, not the peak, and keep the flexible portion on demand or spot. If your traffic is seasonal, a three-year commitment on peak capacity is an expensive guess.
Cost reduction programs attract bad cuts. Removing image scanning, weakening network policies, disabling audit logs, or dropping centralized logging can make a bill look better while raising the chance and cost of an incident. Security controls are not free, but they are usually small compared with the downside of a breach or a compliance failure. Treat them as fixed costs and optimize the implementation instead: sample verbose logs rather than deleting them, set retention by data sensitivity, and route only what you investigate.
Multi-tenant clusters need guardrails too. Namespaces with resource quotas stop one team from consuming the shared pool. Labels for team, environment, and service make spend attributable. Without that attribution, every cost review becomes an argument, and the team with the loudest voice usually wins rather than the team with the wasteful deployment.
Delete what nobody uses: idle load balancers, unattached persistent volumes, old container images, and duplicated observability pipelines. Those costs are easy to forget because they don’t show up as a deployment.
Will lowering requests hurt performance? It can, if you cut into the headroom a service needs during peaks. Compare p95 and p99 usage against requests, change one service at a time, and watch latency and error rates. Memory needs more caution than CPU because the failure mode is a kill, not a slowdown.
Is spot capacity safe for production? For stateless or retryable workloads with multiple replicas and tested disruption handling, yes. For a single-replica stateful service without a proven failover path, the discount is not worth the risk.
Should we run VPA and HPA together? Be careful. If both act on the same resource metric, they can fight each other. A common pattern is VPA in recommendation mode while HPA handles replica count, with humans approving resource changes. Test the combination before trusting it.
How do we stop teams from over-provisioning? Make the cost visible per team, set namespace quotas, and review requests against usage monthly. Most over-provisioning is not malicious. It’s a reasonable attempt to avoid being paged at 2 a.m.
Do we have to weaken security to save money? No. The reliable savings come from matching capacity to demand, removing idle resources, and choosing the right purchase model. Cutting security controls trades a predictable bill for an unpredictable incident.
If you want a practical review of where your cluster is spending and which changes are safe to make first, talk to Kubeify. Bring your bills and your reliability constraints, and we can work through the tradeoffs together.
Kubeify's team decrease the time it takes to adopt open source technology while enabling consistent application environments across deployments... letting our developers focus on application code while improving speed and quality of our releases.
– Yaron Oren, Founder Maverick.ai (acquired by OutboundWorks)
Let us know what you are working on?
We would help you to build a
fault tolerant, secure and scalable system over kubernetes.