High-speed Large Language Model Serving for Local Deployment
-
Updated
Jan 24, 2026 - C++
#
local-inference
Here are 25 public repositories matching this topic...
[ICLR'25] Fast Inference of MoE Models with CPU-GPU Orchestration
-
Updated
Nov 18, 2024 - Python
Modern desktop application (Rust + Tauri v2 + Svelte 5 + Candle (HF)) for communicating with AI models that runs completely locally on your computer. No subscriptions, no data sent to the internet — just you and your personal AI assistant
desktop-app rust productivity machine-learning opensource privacy cross-platform offline chatbot svelte artificial-intelligence candle conversational-ai tauri ai-chat llm local-inference tauri-rust local-llm qwen
-
Updated
Jan 30, 2026 - Rust
On-device AI for iOS & Android
embeddings gemini http-server openai document-processing rag edge-ai on-device-ai local-inference anthropic llamacpp llama-cpp local-llm gguf deepseek multimodal-ai
-
Updated
Jan 30, 2026 - TypeScript
Notolog Markdown Editor
python markdown qt emacs markdown-editor onnx python-ai on-device-ai ai-assistant pyside6 python-qt local-inference llama-cpp local-llm qwen llama-cpp-python gguf phi-4 privacy-first-ai
-
Updated
Jan 31, 2026 - Python
Tool for test diferents large language models without code.
api-rest chat-application flask-api inference-api large-language-models llm local-inference llamacpp llm-inference llama2 llama-cpp-python llama2-7b mixtral-8x7b
-
Updated
Oct 18, 2025 - Python
Study Buddy is a desktop application that provides AI tutoring without requiring internet access or accounts
electron javascript css typescript desktop-application privacy-focused local-inference ai-tutor offline-application
-
Updated
Jun 25, 2025 - TypeScript
LLM chatbot example using OpenVINO with RAG (Retrieval Augmented Generation).
natural-language-processing offline chatbot intel edge-computing rag openvino huggingface edge-inference cloud-free llm local-inference langchain dolly2 retrieval-augmented-generation llama2 neural-chat
-
Updated
Jan 25, 2024 - Python
Lightweight 6GB VRAM Gradio web app with auto-installer for running AuraFlow locally — no cloud, no clutter.
python open-source image-generation webui gradio text-to-image stable-diffusion diffusers local-inference generative-ai ai-image-generator auraflow low-vram
-
Updated
Jun 7, 2025 - Python
EN: An overfitted SD prompt engine with severe "aesthetic snobbery," forcibly transforming mundane ideas into professional-grade physical rendering instructions. CN: 一个具备“审美洁癖”的过拟合提示词引擎，强行将平庸构思纠偏为具备极致物理质感的工业级渲染指令。
native cuda self-hosted python3 micro aesthetic industrial-automation echo-server lora gradio fine-tuning ai-art stable-diffusion local-inference prompt-generation qwen local-ai-agents blackwell-gpu anti-hallucination deterministic-ai
-
Updated
Jan 19, 2026 - Python
Verify claims using AI agents that debate using scraped evidence and local language models.
python scraping beautifulsoup autonomous-agents local-inference qwen lm-studio langgraph private-llm agentic-ai claim-verification phi-4-mini evidence-based-ai
-
Updated
Jun 1, 2025 - Python
Function calling and Agentic framework test app
android-development gemma litert edge-ai mediapipe local-inference function-calling agentic-ai koog litert-lm
-
Updated
Jan 31, 2026 - Kotlin
MCP server that runs local LLMs (with full access to MCP tools included). Callable by Python to chain MCP tools with local intelligence.
-
Updated
Jan 19, 2026 - Python
Audit local LLM function calling and agentic reliability. Visual tool-use benchmarking for quantized models on YOUR hardware.
open-source gradio fan-out multi-gpu ai-safety tool-use model-benchmarking llm local-inference function-calling lm-studio llm-evaluation agentic-ai open-weight low-vram constraint-compliance quantization-testing
-
Updated
Feb 1, 2026 - Python
A lightweight Python implementation of Microsoft's Phi-3 model running locally on CPU.
-
Updated
Jan 30, 2026 - Python
Nexa SDK is a comprehensive toolkit for supporting ONNX and GGML models. It supports text generation, image generation, vision-language models (VLM), auto-speech-recognition (ASR), and text-to-speech (TTS) capabilities.
-
Updated
Nov 19, 2024 - Python
Search through your thousands of photos using natural human language locally on your PC
python docker cli ai cuda transformers gemini semantic-search gemma nvidia-gpu rag huggingface vector-database vector-embeddings qdrant local-inference langchain
-
Updated
Nov 30, 2025 - Python
script which performs RAG and use a local LLM for Q&A
-
Updated
Sep 16, 2024 - Python
The Streamlit AI Content Generator + Editor is a fully interactive, web-based tool designed to assist content creators, marketers, and developers in generating high-quality blog posts, articles, and marketing copy.
python openai content-editor content-automation pdf-export content-generation streamlit markdown-export writing-assistant local-inference seo-optimizer ollama article-writer ai-content-generator
-
Updated
Aug 9, 2025 - Python
Improve this page
Add a description, image, and links to the local-inference topic page so that developers can more easily learn about it.
Add this topic to your repo
To associate your repository with the local-inference topic, visit your repo's landing page and select "manage topics."