LocalMind AI
A private, self-hosted AI platform with multi-model LLM support, voice I/O, and document RAG — all running on consumer hardware with zero cloud dependencies.
Overview
What it does
A full-stack AI system: FastAPI backend serving local LLMs via Ollama, React Native mobile app, local Whisper STT, TTS, and a PDF RAG pipeline — all offline.
Problem solved
Cloud AI services cost money per token, log conversations, and require internet. This brings the full capability stack — chat, voice, document Q&A — to your own hardware.
Built for
Privacy-conscious individuals, developers who want a personal AI assistant without subscriptions, and researchers needing air-gapped LLM inference.
LocalMind AI is a platform rather than a single tool. The FastAPI backend exposes REST and streaming endpoints that wrap Ollama's local LLM API — supporting hot-swapping between any model (Llama 3, Mistral, Gemma, Phi-3) at runtime without restarting the server. Streaming responses are sent via Server-Sent Events (SSE) so the mobile app renders tokens as they arrive.
The voice pipeline is entirely local: Whisper (small model) handles speech-to-text, and a local TTS engine (pyttsx3 or Kokoro) handles text-to-speech. The RAG module lets users upload PDF documents and ask questions that are answered with context retrieved from FAISS — all without leaving the device.
Technical Architecture
System overview
The FastAPI backend is the single integration point for all AI capabilities. Models are loaded lazily on first use and kept warm for subsequent requests. The SSE streaming endpoint allows the mobile app to display partial responses in real time — identical to how ChatGPT renders streaming output.
Key Features
Multi-model switching
Swap between Llama 3, Mistral, Gemma, or Phi-3 at runtime via the app — no server restart needed.
Local voice I/O
Whisper (small) for STT, pyttsx3 for TTS — fully hands-free conversation with any LLM, all offline.
PDF document RAG
Upload any PDF and ask questions answered from your document — FAISS-indexed, locally embedded.
Streaming SSE responses
Server-Sent Events deliver token-by-token streaming to the mobile app — feels as responsive as cloud AI.
Challenges & Solutions
Streaming LLM responses to a mobile client
Challenge: React Native doesn't natively support streaming HTTP, and waiting for the full response before rendering felt sluggish.
Solution: Used FastAPI's StreamingResponse with text/event-stream and a custom React Native SSE client that processes chunks as they arrive and appends to the message state.
Memory management for quantized models
Challenge: Loading a 7B model takes 4–6GB RAM. Switching models without freeing memory caused OOM crashes on 8GB machines.
Solution: Implemented a model lifecycle manager that explicitly unloads the current Ollama model before loading the next, and uses Q4_K_M quantization (4-bit) to halve the memory footprint with minimal quality loss.
Whisper latency for real-time feel
Challenge: Whisper (medium) took 3–5 seconds per utterance on CPU — too slow for conversational use.
Solution: Switched to Whisper (small) which transcribes in under 1 second at <5% accuracy loss for clear speech. For longer recordings, used chunked streaming transcription.
Results & Impact
8GB
Min. RAM required
$0
Ongoing cost
4+
Supported models
3
AI subsystems
LocalMind AI demonstrates that a full production-quality AI assistant stack — chat, voice, document intelligence — can run on a typical laptop without any cloud dependency. The project is actively being expanded with conversation memory, agent tool-use, and a web UI companion.
What I'd do differently
- →Persistent conversation memory from day one. The current implementation is stateless per-session. Adding a lightweight SQLite conversation store with summary compression would make it genuinely useful for ongoing work, not just demos.
- →Replace pyttsx3 with a better TTS engine. pyttsx3 is functional but the voice quality is poor. Kokoro or a small Coqui VITS model running locally produces dramatically better output — the gap in user experience is significant.
- →Decouple the mobile app from a fixed IP. The current setup assumes the server is on the local network at a known address. A mDNS-based service discovery or a QR-code provisioning flow would make it actually deployable without configuration.
Code Highlight
FastAPI SSE streaming endpoint — delivers LLM tokens to the mobile client in real time.
from fastapi import FastAPI
from fastapi.responses import StreamingResponse
import ollama, json
app = FastAPI()
@app.post("/chat/stream")
async def stream_chat(request: ChatRequest):
async def generate():
async for chunk in await ollama.AsyncClient().chat(
model=request.model,
messages=request.messages,
stream=True
):
token = chunk['message']['content']
yield f"data: {json.dumps({'text': token, 'done': False})}\n\n"
yield f"data: {json.dumps({'text': '', 'done': True})}\n\n"
return StreamingResponse(
generate(),
media_type="text/event-stream",
headers={"Cache-Control": "no-cache"}
)