silvic-triton

A native Triton Inference Server backend, silvic, that serves LightGBM model.txt models with silvic-serve. FP64 outputs are LightGBM’s predictions bit for bit. It reads the FIL backend’s LightGBM configs, so a FIL model directory serves with backend: "fil" changed to backend: "silvic". It needs backend API 1.10 (Triton 22.09) or newer, and has run on tritonserver 2.51.0 (NGC 24.10).

models/ranker/config.pbtxt
models/ranker/1/model.txt

backend: "silvic"
max_batch_size: 4096
input [{ name: "input__0" data_type: TYPE_FP32 dims: [ 136 ] }]
output [{ name: "output__0" data_type: TYPE_FP64 dims: [ 1 ] }]
instance_group [{ kind: KIND_CPU count: 2 }]
dynamic_batching { max_queue_delay_microseconds: 500 }

Parameters

Triton’s string parameters. Any other key fails the load and names it, except FIL’s tuning parameters, which are accepted and do nothing.

model_file model.txt

The model file in the version directory.

raw_score false

Margins instead of the objective’s transform, the sigmoid for binary, as Booster.predict.

num_iteration 0

Score with the first N trees; 0 or less, every tree.

kernel detected

portable, neon, avx2 or avx512; one the CPU does not run fails the load.

num_threads 1

Threads per scored matrix; 0 is every core. For parallelism across requests, prefer more instances.

Inputs and outputs

One input, TYPE_FP32 or TYPE_FP64, as wide as the model has features; with max_batch_size above 0 its dims are [num_features], otherwise [-1, num_features]. One output: TYPE_FP64 holds LightGBM’s predictions exactly, TYPE_FP32 their cast, as FIL clients expect; dims [1] with batching, [-1] without. With Triton’s auto-complete, a config of only backend and max_batch_size gets FIL’s names, input__0 in FP32 and output__0 in FP64; a config that gives them is checked against the model.

A contiguous CPU input is scored where Triton holds it. Requests of fewer than 256 rows that dynamic batching delivers together are scored as one matrix, so a 1-row request doesn’t pay a 64-row block alone. A bad request gets an error response and leaves the others, the instance and the server alone. Instances run on the CPU; KIND_GPU is a load error. Triton’s statistics and metrics get every request’s and execution’s timestamps.

Build and run

docker build -f deploy/triton/Dockerfile -t tritonserver-silvic .
docker run --rm -p 8001:8001 -v "$PWD/models":/models tritonserver-silvic \
    tritonserver --model-repository=/models

The image is NVIDIA’s tritonserver with libtriton_silvic.so in /opt/tritonserver/backends/silvic/. For another tritonserver, build it on Linux with cargo build --release -p silvic-triton and put it at <backend-directory>/silvic/libtriton_silvic.so.

Against FIL

3,000-row FP32 requests over gRPC, one instance and one client, in a 4-vCPU aarch64 Linux VM on an Apple M4 Pro, server and client sharing it. Not production numbers.

silvic, 1 thread per request

13.2 ms at the median, 12.3 ms of it scoring.

silvic, 4 threads per request

4.0 ms at the median.

FIL on the CPU, OpenMP on every vCPU

243 ms at the median; its outputs, computed in f32, differ from LightGBM’s by up to 2.4e-5.

FIL is built for NVIDIA GPUs; its GPU mode, and this backend on x86, have not been measured.