On-device AI refers to running machine-learning model Inference directly on end-user hardware - smartphones, laptops, tablets, wearables, or embedded systems - rather than sending data to a remote cloud server. This requires the model to fit within the strict memory and compute constraints of consumer chips, which is achieved through techniques such as Quantization (reducing weight precision from 32-bit float to 8-bit or 4-bit integer), Knowledge Distillation (training a compact student model from a larger teacher), and Neural Architecture Search for hardware-aware design.

Think of it like the difference between consulting a doctor in their office versus carrying a pocket medical reference you wrote down yourself - the pocket guide is less comprehensive, but it works instantly, in any location, without anyone else seeing your question.

Apple's Core ML and Google's MediaPipe are two widely used on-device frameworks. A Core ML inference call in Swift is straightforward:

import CoreML

let config = MLModelConfiguration()
let model  = try! MyMobileModel(configuration: config)
let input  = MyMobileModelInput(text: "Is this email spam?")
let output = try! model.prediction(input: input)
print(output.label)   // "not_spam"

Because no data leaves the device, on-device AI offers strong privacy guarantees and eliminates network round-trip latency, which is essential for real-time applications like live speech recognition, face unlock, and keyboard autocorrect. Related concepts include Quantization and Knowledge Distillation (the primary compression techniques that make on-device feasible), Inference (the computational phase being localised to the device), and Foundation Model (the cloud-scale origin model from which on-device variants are derived through distillation or fine-tuning).

You will encounter on-device AI in mobile SDK documentation, privacy-focused product announcements, and any scenario where reliable offline functionality or regulatory data-residency requirements rule out cloud processing.