AI Inference and Serving: Integrating Machine Learning into Distributed Systems

Core Concept

• AI systems revolve around:
  • Model (trained weights)
  • Inference (runtime prediction)
• Inference is typically exposed as a REST API

Think:

Model = learned function, Inference = using it in production

Key Building Blocks

1. Model vs Training vs Inference

• Training
  • Learn from data (expensive, offline)
• Fine-tuning
  • Adapt existing model (cheaper)
• Inference
  • Run model on real user input (online)

Most systems:

• Don’t train → just use pretrained models

2. Prompt Engineering (VERY IMPORTANT)

• Input is wrapped inside a prompt
• Prompt adds:
  • Context
  • Instructions
  • Personalization

Better prompt → better output

Hosting Models

Key constraints:

• Latency
  • Users notice >250ms
• Compute
  • Requires GPUs (expensive)

Deployment options:

1. Cloud inference (easy)

• Managed APIs
• Faster to start

2. Self-hosted (control)

• Needed for:
  • Privacy
  • Security

Small Language Models (SLMs)

• Smaller, cheaper, faster
• Can run on:
  • Mobile devices

Trade-off:

• Less accuracy vs lower cost

Model Distribution

Use standard format:

• ONNX (recommended)

Framework-independent

Use model hubs:

• Example:
  • Hugging Face

Easy access to pretrained models

Optimization:

• Cache models locally
  • Avoid repeated downloads
  • Faster startup

Safe deployment (VERY IMPORTANT)

• Roll out models gradually:
  • Start small %
  • Expand over time

Same as canary deployments

Development Strategy

Problem:

• Inference is expensive

Solution:

• Use different models per environment

Use abstraction:

• REST API layer hides model details

Retrieval-Augmented Generation (RAG)

Problem:

• Models are:
  • Static
  • Generic

Solution: RAG

• Combine:
  • User prompt
  • External data (DB, APIs)

Flow:

User Query
   ↓
Fetch external data
   ↓
Augment prompt
   ↓
Send to model

Benefits:

• Personalized responses
• Up-to-date information
• Better accuracy
• Privacy-safe

Testing AI Systems (VERY DIFFERENT)

Traditional testing:

• Exact match (A == B)

AI testing:

• “Good enough” quality
• Evaluate across:
  • Many inputs
  • Statistical performance

Key idea:

Measure overall quality, not single output

Trick:

• Use AI itself to evaluate responses

Deployment & Evaluation

Must track:

• User satisfaction
• Output quality

Not just:

• Latency
• Errors

Safe rollout:

• Monitor:
  • Feedback
  • Quality metrics

Key Insights

• AI = just another distributed system component
• Cost & latency are critical constraints
• Prompt + data (RAG) often matter more than model
• Testing is probabilistic, not deterministic

Trade-offs

Pros

• Powerful capabilities
• Flexible interfaces
• Rapid development via pretrained models

Cons

• Expensive inference
• Non-deterministic outputs
• Harder testing & debugging
• Latency constraints

One-line Summary

AI inference systems serve trained models via APIs, where prompt engineering, efficient model hosting, and techniques like RAG are key to delivering accurate, scalable, and cost-effective intelligent applications.