@hpcc-js/wasm-llama

Installation

npm

npm install @hpcc-js/wasm-llama

yarn

yarn add @hpcc-js/wasm-llama

pnpm

pnpm add @hpcc-js/wasm-llama

Quick Start

import { Llama, WebBlob } from "@hpcc-js/wasm-llama";

let llama = await Llama.load();
const model = "https://huggingface.co/CompendiumLabs/bge-base-en-v1.5-gguf/resolve/main/bge-base-en-v1.5-q4_k_m.gguf";
const webBlob: Blob = await WebBlob.create(new URL(model));

const data: ArrayBuffer = await webBlob.arrayBuffer();

const embeddings = llama.embedding("Hello and Welcome!", new Uint8Array(data));

const reply = llama.chat("Summarize this package in one sentence.", new Uint8Array(data), {
	systemPrompt: "You are a concise assistant.",
	args: ["--temp", "0.2"]
});

const session = llama.createChatSession(new Uint8Array(data), {
	systemPrompt: "You are a concise assistant.",
	args: ["--temp", "0.2"]
});

const firstReply = session.send("Summarize this package in one sentence.");
const followUp = session.send("Now restate that as three bullet points.");

const result = llama.main(["--help"]);
console.log(result.stdout);

Try it

Local GGUF modelPromptWrite one concise sentence about WebAssembly.

Resource management

Llama supports dispose() and TypeScript's using declarations (TypeScript 5.2+ with "ESNext.Disposable" in lib). Leaving the scope is equivalent to calling Llama.unload():

import { Llama } from "@hpcc-js/wasm-llama";

{
    using llama = await Llama.load();
    const embeddings = llama.embedding("Hello and Welcome!", new Uint8Array(data));
} // Llama.unload() runs here

Llama.load() returns a singleton, so disposing it unloads the module for every caller in the current JavaScript context, including any chat sessions created from it. Disposing an instance that has already been unloaded does nothing.

Classes

• LlamaChatSession
• Llama

Interfaces

• LlamaMainResult
• LlamaChatOptions
• LlamaChatMessage
• LlamaChatSessionOptions

Type Aliases

• LlamaChatRole

Reference

• API Documentation