GPU acceleration with LiteRT

Graphics Processing Units (GPUs) are commonly used for deep learning acceleration due to their massive parallel throughput compared to CPUs. LiteRT simplifies the process of using GPU acceleration by allowing users to specify the hardware acceleration as a parameter when creating a Compiled Model (CompiledModel).

With LiteRT's GPU acceleration, you can create GPU-friendly input and output buffers, achieve zero-copy with your data in GPU memory, and execute tasks asynchronously to maximize parallelism.

Get Started

• For classical ML models, see the following demo apps.

  • Image segmentation Kotlin App: CPU/GPU/NPU inference.
  • Image segmentation C++ App: CPU/GPU/NPU inference with async execution.

• For GenAI models, see the following demos and guide:

  • EmbeddingGemma semantic similarity C++ App: CPU/GPU/NPU inference.
  • Guide on running LLMs using LiteRT-LM.

Add GPU dependency

Use the following steps to add GPU dependency to your Kotlin or C++ application.

Kotlin

For Kotlin users, the GPU accelerator is built-in and does not require additional steps beyond the Get Started guide.

C++

For C++ users, you must build the dependencies of the application with LiteRT GPU acceleration. The cc_binary rule that packages the core application logic (e.g., main.cc) requires the following runtime components: