🔌
One data plane in front of every major LLM provider, speaking both the OpenAI and Anthropic wire formats.
- Dispatches to OpenAI, Anthropic, Google Gemini, xAI, AWS Bedrock, local Ollama, and self-hosted llama.cpp.
- OpenAI-compatible POST /v1/chat/completions and Anthropic-compatible POST /v1/messages (plus /v1/messages/count_tokens) on the same base URL.
- GET /v1/models lists every configured model across providers; streaming via Server-Sent Events for both wire formats.
🧵
Every request — regardless of wire format — runs through the same fixed, cheapest-first stage order.
- Build order: auth → token_budget → security_in → scratchpad → summarize → knowledge → dedup → cache → routing → dispatch → security_out → meter.
- auth, token_budget, security_in, dispatch, security_out, and meter always run; memory-layer and routing stages are independently feature-flagged.
- A blocked stage short-circuits the rest of the pipeline — a filtered prompt never reaches a provider.
🛡️
A failing provider connection is isolated instead of retried into the ground, with automatic recovery.
- Each provider connection tracks its own health: closed → open → half-open → closed.
- After 3 consecutive retryable failures the connection is held out for a 5-minute cooldown, then offered exactly one half-open probe before rejoining rotation.
- Client errors (4xx) never count toward the breaker; only timeouts, rate limits, and 5xx responses do — an unavailable primary falls back or returns 503.
📊
Normalized WaddleAI tokens for billing and quotas, plus raw LLM tokens for per-provider analytics.
- Every response includes both waddleai_tokens (normalized, cross-provider) and the provider’s raw prompt/completion token counts.
- OpenTelemetry gen_ai.* span attributes emitted on every dispatch: gen_ai.system, gen_ai.request.model, gen_ai.response.model, gen_ai.usage.input_tokens/output_tokens, gen_ai.response.finish_reason.
- Usage is aggregated in-process and flushed through a bounded retry buffer, so a transient database failure never silently drops billable tokens.
💰
Token, dollar, and provider-plan budgets are evaluated together, with graduated pressure before a hard block.
- Three budget types — token (WaddleAI/raw monthly caps), dollar, and plan/usage — evaluated together; the tightest one binds.
- Graduated pressure: roughly 80% consumed raises the escalation threshold, 95% clamps to local-only models, 100% hard-blocks.
- Quota hierarchy (most restrictive wins): user quota → organization quota → system default, settable via the management API.
🔑
Scoped, expiring API keys with per-key rate limits and audit-friendly revocation.
- Keys are issued in wa-{random} format and shown only once at creation; stored hashed thereafter.
- Per-key rpm_limit, tpm_limit, and daily dollar budget, settable at creation or via PUT /keys/{id}.
- Keys are disabled rather than hard-deleted, preserving audit trails, and can be rotated in place with POST /keys/{id}/rotate.
🏢
Organizations, users, and four fixed roles govern access across both the proxy and the management API.
- Four roles — Admin, Resource Manager, Reporter, User — each mapping to a fixed permission set enforced via OIDC scopes issued by penguin-aaa.
- Admin: full system access across all organizations. Resource Manager: scoped to their own organization. Reporter: read-only analytics. User: personal virtual keys only.
- Organization-level usage endpoints (GET /organizations/{id}/usage) and per-org quota management.
🖥️
Run Ollama or llama.cpp on your own GPU nodes, alongside commercial providers, from the same Helm chart.
- Ollama deploys as a DaemonSet on GPU-labeled nodes by default, or a fixed-replica pool Deployment for mixed-GPU clusters, sharing a ReadWriteMany PVC model cache.
- llama.cpp renders one DaemonSet + Service per configured model, sharing the same PVC-backed model cache.
- Optional GPU device-plugin sub-charts for NVIDIA (with GFD + driver stack), AMD ROCm, and Intel, each disabled by default.
🧠
A mem0-compatible memory API over PostgreSQL pgvector, with access-controlled personal and org scopes.
- Mounted at /mem0 on the proxy: add, search, list, and delete memories, with embedding generation and pgvector storage handled internally — no separate vector database required.
- Personal (user) scope is the default and locked to the authenticated caller; org (shared) scope is gated behind a feature flag, default off.
- Reads return a relevance-ranked merged personal + org view; cross-organization access is always rejected with 403.
🔒
Dual-phase prompt and response filtering, purpose-built for the threat surface an LLM proxy introduces.
- Prompt injection and jailbreak detection, classified by threat type (prompt_injection, jailbreak, data_extraction, system_prompt_leak, credential_harvesting) and severity.
- Dual-phase PII/PCI content filtering on both inputs and outputs, combining regex patterns with optional Presidio-based NER.
- Retrieved memory context is wrapped with structural delimiters and treated as data, never instructions, mitigating indirect prompt injection via memory.
🤖
A multi-agent coding CLI, a VS Code Copilot Chat provider, and MCP tool servers for any MCP-compatible client.
- WaddleAI Assistant (penguincode) orchestrates Explorer, Executor, Planner, Reviewer, Tester, Debugger, Refactor, Researcher, and Docs sub-agents, runnable local, standalone, or remote (gRPC + TLS + JWT).
- The waddleai-copilot VS Code extension adds an @waddleai chat participant to Copilot Chat with workspace context awareness and token usage tracking.
- Two separate MCP server instances — /mcp (user tools) and /mcp/admin (admin tools) — built per authenticated request, so an unauthorized caller never sees an admin tool.
