Update 2.3.0 Release: Distributed LLM Inference Fabric for Local Clusters

1. Compact Tool Schemas & Prefill Optimization Prompt Token Reduction: Tool instructions previously inlined complete JSON input schemas into the prompt prefix of every turn, causing even short questions to incur over 3,800 tokens of fixed overhead[1]. Schemas are now reduced to compact signatures (e.g., read_file(required: path; optional: content)), cutting prompt tokens from 3,814 down to 1,047 and prefill latency from ~13.1s to ~5.9s[1][2]. Full schemas remain accessible on-demand via tool_schema_detail[3]. Selective Remote Routing: Rather than naively splitting whole layers across the network, the tensor planner (crates/ghost-link/src/tensor_plan.rs) routes Feed-Forward Network (FFN) and MoE experts remotely while pinning embeddings, attention, KV cache, and output heads (lm_head) locally[4]. Latency Safeguards: Peer routing evaluates measured RPC round-trip time (GHOSTLINK_RPC_RTT_MS) against CPU-offload penalties to automatically reject high-latency peers[4]. Persistent Memory Server: Introduced an internal SQLite-backed MCP server (crates/mcp-memory/) providing per-workspace memory storage across sessions[5]. Automated Context Injection: On the first turn of a session, Ghostlink automatically queries memory_search and rag.search to inject relevant facts, decisions, and summaries into system context before the model generates its response[6]. Workspace boundaries are stamped server-side to prevent prompt-injection scope escalations[5][6]. Tri-Budget Control: The agent execution loop now enforces independent limits for max_steps, max_tool_calls, and max_tokens to prevent stuck tool loops from burning compute[7]. Server-Side Capability Gate: Tools are classified into read, write, and exec classes in Rust (crates/ghost-link/src/capability.rs), defaulting unlisted tools to exec[8]. Approval Tray: Potentially destructive actions (write or exec) are queued into a persistent approval tray (crates/ghost-link/src/approvals.rs) for human review rather than failing immediately[9]. Speculative Decoding: Formally wired speculative decoding parameters (--spec-draft-model, --spec-draft-n-max, --spec-draft-p-min) into llama-server[2][10]. Slot Pinning & Persistent Summaries: Session IDs are mapped to stable llama-server slots for KV cache prefix reuse[11]. Overflowing conversation turns are automatically summarized and persisted to disk (session_summaries.json) so context survives process restarts[11]. Become a supporter of this podcast: https://www.spreaker.com/podcast/deep-dive-of-rwilliamspbg-ops--7123568/support . https://github.com/rwilliamspbg-ops