Running AI Models Without GPUs: The Hidden Power of Software Optimization

Running AI Models Without GPUs: The Hidden Power of Software Optimization

The Saying 'Need an H100 GPU for AI' Is Only Half True

If your team is struggling with AI performance optimization, you have probably asked, "Can we run AI without a GPU?" at least once. It’s one of the questions our team receives most often in the field of export sales operations. In reality, a single NVIDIA H100 costs over 30 million KRW, and keeping cloud GPU instances running 24/7 drains thousands of dollars. However, when we looked into actual codebases, we found something different.

Much of the bottleneck was not in the hardware but at the software optimization level—specifically in how matrix operations were implemented. The throughput gap between code calculating matrix multiplication with a simple Python loop and a BLAS implementation optimized for cache architecture can range from tens to hundreds of times. That’s on the same CPU. We will discuss honestly in this post how single-operation benchmarks differ from real-world training workloads. But first, let’s grasp the intuition of why it's possible to drastically increase performance without changing hardware.

---

The Starting Point of LLM Inference Optimization: Understanding Bottlenecks

What is GEMM, Which Accounts for Most Transformer Operations?

The internal structure of Large Language Models (LLMs) like ChatGPT, Claude, and Gemini is based on the Transformer architecture. Attention layers and FFN (Feed-Forward Network) layers all eventually boil down to matrix multiplication, or GEMM (General Matrix Multiply). The fact that 60–80% of model operations are GEMM has been repeatedly confirmed in deep learning performance analysis studies. In other words, it is no exaggeration to say that how efficiently you run a single GEMM determines the overall speed of AI inference and training.

Key Principles of AI Performance Optimization: Why Cache Misses Make GPU Investments Look Wasteful

There are three core principles of low-level software optimization:

① Cache Locality: Both CPUs and GPUs have a memory hierarchy. Reading data from the L1 cache is dozens of times faster than reading from main memory (DRAM). If your data access pattern in matrix operations isn't cache-friendly, even the most powerful hardware spends most of its time waiting for memory. Changing just one line of code to flip the access order can change your throughput by tenfold.

② SIMD (Single Instruction Multiple Data): A method of processing multiple data points with a single instruction. Intel AVX2 can process 256 bits (8 values for FP32) at once, and AVX-512 can process 512 bits in parallel. If utilized properly, the amount of data processed per loop jumps by 8x.

③ Blocking (Tiling): A technique that splits large matrices into smaller blocks that fit into the cache. Instead of reading the entire matrix from memory, you reuse data within the size limit of the cache to maximize the cache hit rate. This is exactly why high-performance numerical libraries like OpenBLAS or oneDNN have spent years refining this principle.

---

Training LLMs with Swift? — Universal Lessons from Counterintuitive Experiments

Attempts to Break the 'AI = Python' Stereotype

There are interesting experiments being conducted in the developer community recently. Examples include implementing GPT-2 in C as shown in Andrej Karpathy’s llm.c, or applying matrix multiplication optimization in Swift step-by-step to measure performance. These experiments share one central question: If you change the language or framework, how much does it change if you apply low-level optimization principles properly?

Gflop/s → Tflop/s: The Power of Low-Level Optimization Principles

When we recreated the figures reported in the Swift matrix multiplication optimization experiment in our Python environment in the same sequence, the trend was the same. Starting from a naive implementation (simple nested loops) and sequentially adding Cache Locality improvement → SIMD utilization → Blocking application leads to a dramatic spike in throughput. Our findings in a Python + NumPy environment are summarized below:

Optimization Stage	Implementation Method	Relative Performance (vs Naive)
Naive	Pure Python nested loops	1× (Baseline)
Cache Locality	Operations after transpose	~10×
BLAS utilization	NumPy dot (via OpenBLAS)	~200×
Mixed Precision	Operations after FP16 conversion	~250×

What’s more important than the absolute figures is the flow of contribution by optimization stage. The key takeaway is that you can achieve this level of disparity through code-level changes alone without changing the hardware.

However, Things to Check Before Trusting Benchmark Numbers

Let’s be honest here.

First, the Swift experimental results reported in Apple Silicon environments include characteristics where the M1/M2/M3 chips' AMX(Apple Matrix Coprocessor) units and Metal acceleration work together. Part of the performance gain might come from hardware characteristics rather than Swift code optimization. This means you need to isolate the pure contribution of software optimization.

Second, a matrix multiplication benchmark is not the same as a real-world full LLM training loop. You shouldn't assume you can achieve Tflop/s levels with GEMM optimization alone, as real training loops involve backpropagation, optimizer steps, batch pipelining, and multi-GPU communication overheads.

Third, the Swift ML ecosystem still lags significantly behind the Python ecosystem in terms of library diversity, community size, and tooling maturity.

Therefore, the real lesson of this experiment is not that 'Swift is good for AI,' but the universal fact that if you understand low-level optimization principles in any language, you can achieve dramatic performance improvements.

---

Rethinking 'Running AI Without GPUs' Under Real-World Export Constraints

There is a gap between standard AI infrastructure optimization discussions and the reality of global trade operations. We encounter three constraints repeatedly when working with our clients:

On-premise corporate servers: Often, cloud GPU instances cannot be used due to security and compliance reasons. If you must run an inference pipeline on a company server, software optimization becomes your only lever.

Lack of IT manpower: Most small to medium-sized export enterprises do not have dedicated ML engineers. Optimization with a lower barrier to entry, like changing PyTorch settings, is more realistic than complex CUDA kernel tuning.

Budget limitations: For many teams, the ROI doesn't pencil out if cloud GPU costs reach millions of KRW per month. Increasing throughput on the same hardware through software-level optimization is the fastest path to cost reduction.

If you design optimization based on these three constraints, your approach changes. The starting point shouldn't be "Which GPU should we buy?" but "Where is the bottleneck in the current server?"

---

Practical Points You Can Apply Right Now for AI Performance Optimization

LLM Inference Optimization Points for PyTorch/JAX Environments

The principles of cache locality, SIMD, and blocking work the same whether you use Python + PyTorch, C++, or JAX. We recommend checking the following checklist in order if you want to optimize AI inference/fine-tuning in an environment without a GPU or with low-end GPUs:

1. Profile first. You cannot set a direction for optimization before you measure the actual bottleneck using torch.profiler or py-spy. Don't just work on a "feeling" that it is slow; confirm it with data.
2. Utilize Mixed Precision. Using FP16 or BF16 instead of FP32 cuts memory usage in half and increases throughput on supported hardware. You can apply it in a few lines using PyTorch's torch.autocast.
3. torch.compile or XLA JIT. In PyTorch 2.0 or higher, torch.compile optimizes the computation graph to speed up execution without additional code changes.
4. Review Batch Size and Memory Layout. Simply aligning batch sizes to powers of 2 and checking tensor memory layouts (contiguous vs. non-contiguous) can make a meaningful difference.

Realistic Performance Expectations by Environment for AI Cost Reduction

To be honest about what is possible without high-end GPUs: small-model fine-tuning (7B or less), inference optimization, and prototyping are entirely feasible. Open-source projects optimized for CPU inference like llama.cpp have already reached a practical level. Conversely, pre-training models with billions of parameters is difficult to overcome with software optimization alone. Making this distinction is the first step in a realistic AI budget reduction plan.

---

Why Our Team Is Obsessed with This Problem

In the export field, the two biggest barriers to AI adoption are cost and speed. The moment the question "How much does it cost to build the infrastructure to use AI?" is asked, many teams stop the review process. We also initially approached this by trying to expand cloud GPU instances. But when we actually looked into the pipeline, the bottleneck was elsewhere. It wasn't the model size; it was the inefficiency of the code handling inference requests.

When we ran profiling on our own buyer discovery pipeline for the first time, a significant portion of inference latency was caused by data preprocessing and batch construction, not model computation. Switching to Mixed Precision and adjusting batch sizes significantly improved throughput on the same server, eliminating the need for additional cloud GPU instances. Building a habit of checking code before increasing the GPU budget is the starting point for how we design our service.

It’s in the same vein that Rinda asks "how do we execute this operation efficiently" before "where do we rent a GPU" when applying AI to export operations. Technology is the medium; the goal is to make it work within the speed and budget constraints practitioners can actually use.

---

Conclusion: If You Want Faster AI, Doubt Your Code First

The First Action You Can Take Today

The core of LLM operational costs and speed lies in software-level AI performance optimization rather than hardware specs. The principles of cache locality, SIMD, and blocking work identically across any language and stack. The first thing you can do today before getting a GPU quote is to run torch.profiler. Changing hardware without knowing where the bottleneck is like replacing your tires when the engine is the real problem.

For Further Deep Dives

If you want to follow the principles of matrix operation optimization through code, we recommend fast.ai’s From Deep Learning Foundations to Stable Diffusion course or Andrej Karpathy’s llm.c project. They provide a step-by-step walkthrough of the optimization process with comments explaining "why we implement it this way," making them excellent references for grasping the principles.

---

Author · The Rinda Team (Research Editors for Overseas Buyer Discovery & Export Sales Automation)

Based on data from the buyer discovery pipelines of 200+ Korean export companies and internal observations of the grinda.ai platform, we edit strategies and checklists for immediate use in export operations.

---

If you are curious about AI pipeline optimization examples for exports or want to discuss specific ways to reduce inference costs in on-premise environments, please feel free to reach out via Checking how Rinda reduced costs. We can look together at points to check at the software level before making any final GPU budget decisions.

---

Q. Can we run LLMs in real services without a GPU? A. It depends on the model size and request throughput. If you optimize models 7B or smaller based on llama.cpp, you can achieve practical inference speeds even in CPU environments. However, in production environments with many concurrent requests or where real-time response is required, the parallel processing power of a GPU is still necessary. It is more realistic to approach this by profiling your workload to identify your minimum requirements rather than asking "is it possible without a GPU."

Q. How much faster will it get using torch.compile? A. It depends on the model structure and input size, but there have been reports from official PyTorch benchmarks of 1.5x–2x improvements in inference speed for certain models using the Inductor backend. Note that there is a compilation overhead, so it is generally more effective in batch processing environments than in environments with frequent, short requests. We recommend measuring your actual workload with torch.profiler before applying it.

Q. What is a realistic range of AI adoption cost savings through software optimization? A. From our observations, teams that operated without profiling were able to achieve meaningful throughput improvements on the same hardware by sequentially applying batch size adjustments, mixed precision, and compile optimizations. However, the exact extent of savings depends on...