While most AI features on the web rely on servers, client-side AI runs directly in the user's browser. This offers many benefits, including low latency, reduced server-side costs, no API key requirements, increased user privacy, and offline access. You can implement client-side AI that works across browsers with JavaScript libraries like TensorFlow.js, Transformers.js, and MediaPipe GenAI.
Client-side AI also introduces performance challenges: users have to download more files, and their browser has to work harder. To make it work well, consider:
- Your use case. Is client-side AI the right pick for your feature? Is your feature on a critical user journey, and if so, do you have a fallback?
- Good practices for model download and usage. Keep reading to learn more.
Before model download
Mind library and model size
To implement client-side AI you'll need a model and usually a library. When choosing the library, assess its size like you would any other tool.
Model size matters, too. What's considered large for an AI model depends. 5MB can be a useful rule of thumb: it's also the 75th percentile of the median web page size. A laxer number would be 10MB.
Here are some important considerations about model size:
- Many task-specific AI models can be really small. A model like BudouX, for accurate character breaking in Asian languages, is only 9.4KB GZipped. MediaPipe's language detection model is 315KB.
- Even vision models can be reasonably sized. The Handpose model and all related resources total 13.4MB. While this is much larger than most minified frontend packages, it's comparable to the median web page, which is 2.2MB (2.6MB on desktop).
- Gen AI models can exceed the recommended size for web resources. DistilBERT, which is considered either a very small LLM or a simple NLP model (opinions vary), weighs in at 67MB. Even small LLMs, such as Gemma 2B, can reach 1.3GB. This is over 100 times the size of the median web page.
You can assess the exact download size of the models you're planning to use with your browsers' developer tools.
Optimize model size
- Compare model quality and download sizes. A smaller model may have sufficient accuracy for your use case, while being much smaller. Fine tuning and model shrinking techniques exist to significantly reduce a model's size while maintaining sufficient accuracy.
- Pick specialized models when possible. Models that are tailored to a certain task tend to be smaller. For example, if you're looking to perform specific tasks like sentiment or toxicity analysis, use models specialized in these tasks rather than a generic LLM.
While all these models perform the same task, with varying accuracy, their sizes vary widely: from 3MB to 1.5GB.
Check if the model can run
Not all devices can run AI models. Even devices that have sufficient hardware specs may struggle, if other expensive processes are running or are started while the model is in use.
Until a solution is available, here's what you can do today:
- Check for WebGPU support. Several client-side AI libraries including Transformers.js version 3 and MediaPipe use WebGPU. At the moment, some of these libraries don't automatically fallback to Wasm if WebGPU isn't supported. You can mitigate that by enclosing your AI-related code within a WebGPU feature detection check, if you know that your client-side AI library needs WebGPU.
- Rule out underpowered devices. Use Navigator.hardwareConcurrency, Navigator.deviceMemory and the Compute Pressure API to estimate device capabilities and pressure. These APIs are not supported in all browsers and are intentionally