silvic-serve

silvic-serve scores LightGBM 3.x and 4.x models from LightGBM or silvic. Its default float64 mode matches LightGBM’s predictions bit for bit, and its CUDA float32 mode uses approximate float32 arithmetic and output. Python exposes these predictors as silvic.serve. The serving benchmarks compare CPU and CUDA runtimes; the CPU predictor uses leaf bitvectors to evaluate rows.

import silvic.serve

predictor = silvic.serve.Predictor("model.txt")
scores = predictor.predict(candidates, num_threads=1)

cuda = silvic.serve.Predictor("model.txt", device="cuda")
scores = cuda.predict(candidates)

fast = silvic.serve.Predictor("model.txt", device="cuda", precision="float32")
scores = fast.predict(candidates)

model = silvic.serve.Model(model_file="model.txt")  # where lleaves.Model was
model.compile()
scores = model.predict(candidates, n_jobs=1)

On the benchmark’s 24,000 held-out rows, CUDA float32 had a maximum absolute error of 6.06e-6 against LightGBM float64. NDCG@10 and top-10 order were unchanged across 480 queries. The default float64 output was bitwise exact.

Public API

silvic.serve.Predictor(model, kernel=None, *, device="cpu", device_id=0, precision="float64")

Prepares a silvic.Booster, a lightgbm.Booster, model text or a file. CPU is the default; device="cuda" selects a GPU ordinal on Linux. precision="float32" uses approximate float32 thresholds, leaves, accumulation and output. device="neuron" selects a logical core through the silvic-neuron backend and requires explicit float32. Both devices reject CPU kernel selection. Device errors propagate.

predictor.predict(data, raw_score=False, num_iteration=None, num_threads=1)

One score per row of numpy (f32 or f64, C or Fortran order) or pandas, with categories mapped as at training. Output dtype follows serving precision, not input dtype. Float32 mode converts f64 inputs to float32. CPU scoring uses one thread, or every core with 0; CUDA and Neuron require num_threads=1. Releases the GIL; shared device predictors serialize calls.

silvic.serve.Model(model_file=None, model_str=None, kernel=None, *, device="cpu", device_id=0, precision="float64")

A lleaves-style wrapper. Choose precision at construction; compile() records raw_score and ignores lleaves’ tuning options. predict(data, n_jobs=None) uses every CPU core, or default CUDA/Neuron execution. Explicit nondefault thread counts are CPU-only.

Predictor::new(&model), predictor.predict(&rows, raw_score, num_trees, num_threads)

The same from Rust, on a silvic_core::Model and a DenseMatrix; predict_raw writes raw scores into a caller’s buffer.

Kernel, Predictor::with_kernel(&model, kernel), available_kernels()

neon, avx2, avx512 or portable. Kernel::detect picks NEON on aarch64 and AVX2 on x86-64 that has it, else the portable kernel.

silvic.serve.available_devices()

Lists cpu and available Linux cuda or neuron runtimes/devices. available_kernels() remains CPU-only. Neuron’s compiler and installation requirements are documented in silvic-neuron.

CudaPredictor::new(&model, ordinal), CudaPredictorF32::new(&model, ordinal)

Rust’s opt-in cuda feature. The first returns f64 scores; the second returns approximate f32 scores. Both keep the forest and reusable buffers on device. predict returns a result; predict_raw writes into a caller’s typed buffer. Device errors propagate without fallback.

Also public predictor.precision, model.precision, predictor.device, predictor.kernel, predictor.num_trees(), predictor.num_feature(), Predictor::num_walked_trees.

How it scores

CPU kernels reuse byte-sized threshold ranks and filter leaf bitvectors across SIMD rows. Large trees, NaN thresholds and categorical features with more than 64 distinct category behaviours use exact node walks. The CPU serving comparison shows the bitvector path.

CUDA walks 32-row tiles across up to eight trees concurrently. Shared leaf tiles use 2 KiB in float64 or 1 KiB in float32; the first warp adds leaves in tree order. Float32 numerical thresholds round down to preserve decisions for representable f32 values; f64 input conversion can change paths. The objective transform uses the selected precision on the host. Triton instances remain CPU-only.

Kernels

neon aarch64

64 rows per block.

avx2 x86-64

64 rows per block; the x86-64 default when supported.

avx512

64 rows per block. Opt-in by name, not selected automatically.

portable

Plain Rust, eight byte lanes to a u64 word, 32 rows per block.

cuda opt-in device

Cooperative tree walking in float64 or explicit float32, selected with device="cuda" rather than the CPU Kernel enum.

Float64 kernels are checked bitwise against LightGBM and silvic-core across layouts, missing values, categorical decisions, threshold boundaries and tree prefixes. Float32 is checked against its own arithmetic contract, not exact float64 equivalence. Binary probabilities use the host math library, whose exp can differ in its last bit across platforms.