Friday, July 5, 2024

Candle - Minimalist ML framework for Rust - Step-by-Step Tutorial - Run Models Locally

 This video installs Candle locally and run local models. Candle is a minimalist ML framework for Rust with a focus on performance (including GPU support) and ease of use. 


-- Make sure nvcc shows CUDA version 12.3 and nvidia-smi shows 8.9 for compute capacity.

nvcc --version should print information about your Cuda compiler driver.
nvidia-smi --query-gpu=compute_cap --format=csv

Make sure that that openssl and cc1plus are on the following path by finding them :  sudo find / -name cc1plus

-- export PATH=$PATH:/usr/lib/gcc/x86_64-linux-gnu/11/   //This is for cc1plus  (cc1plus is the C++ compiler driver )
-- export OPENSSL_LIB_DIR=/usr/lib/x86_64-linux-gnu      //This is for openssl
-- export PATH=/usr/local/cuda/bin${PATH:+:${PATH}}
-- export LD_LIBRARY_PATH=/usr/local/cuda-12.2/lib64\${LD_LIBRARY_PATH:+:${LD_LIBRARY_PATH}}

-- If you cannot find openssl, then run following :

sudo find / -name cc1plus

sudo apt install libssl-dev
sudo apt install pkg-config

-- Create a new app and add candle-core with Cuda support.

cargo new myapp
cd myapp

cargo add --git candle-core --features "cuda"
cargo build

-- Run Inference with Phi-2 model

git clone
cd candle/candle-examples

cargo run --example phi --features cuda --release -- --model 2 --prompt "What is happiness in 3 lines?"

No comments: