May 2026 - Present
Cloud2 Labs
Software Developer Intern
Built an enterprise inference routing framework in front of LiteLLM, along with multi-tenant knowledge ingestion and observability systems for AI workflows running across multiple model providers.
Role Overview
A quick read on what the role was and the main areas I worked on before the deeper implementation details.
Built an enterprise inference routing framework in front of LiteLLM so user requests could be routed across multiple model providers based on task capability, predicted failure risk, and business constraints.
Designed a hybrid routing approach that combined a lightweight capability classifier, model-specific error maps, and deterministic business rules, improving routing accuracy by 20% while reducing average inference cost by 15%.
Built a 5,000-question validation workflow and model error-map onboarding path so new models could be added without retraining the full 50,000-question routing classifier, cutting onboarding time by 70%.
Developed a multi-tenant knowledge ingestion pipeline for 100+ organizations with Google Drive ingestion and uploads of up to 100 documents per request, while deduplication reduced storage and ingestion overhead by 5-6%.
Built observability and telemetry across 10+ AI agents and multiple workflows, making debugging failures and drift 10-15% easier through traces, latency, token, and cost visibility.
Detailed Work
The role is grouped by major workstreams, with each one kept concise enough to explain the implementation and impact without turning into a wall of text.
Enterprise Inference Routing Framework
Built the routing layer that sits in front of LiteLLM, combining ML signals and deterministic business logic so requests could be sent to the most appropriate model as providers changed over time.
Built an enterprise inference routing framework in front of LiteLLM so user requests could be routed across multiple providers based on task capability, predicted failure risk, and business constraints instead of static model selection.
Designed a hybrid routing approach that combined a lightweight capability classifier, model-specific error maps, and rule-based business checks, improving routing accuracy by 20% while reducing average inference cost by 15%.
Added pre-routing capability filtering so unsupported models were eliminated before the main router ran, which reduced unnecessary evaluation work and kept routing decisions more efficient.
Model Onboarding with Error Maps and Evals
Designed a scalable model-onboarding workflow where new models could be evaluated against a clustered validation set and attached to the router through error maps instead of forcing full retraining each time.
Trained the main routing classifier on 50,000 questions across math, coding, reasoning, science, and general knowledge to estimate model failure likelihood for incoming requests.
Built a 5,000-question clustered validation workflow that generated per-model error maps, allowing new models to be onboarded by running targeted validation instead of retraining the entire classifier.
This reduced new-model onboarding time by 70% and made the routing layer more scalable as the provider and model set expanded.
Knowledge Ingestion and AI Workflow Observability
Built the surrounding platform systems needed for enterprise AI operations, including multi-tenant knowledge ingestion and telemetry across agents, workflows, and model executions.
Developed a multi-tenant knowledge ingestion pipeline for 100+ organizations, supporting Google Drive ingestion and bulk uploads of up to 100 documents per request.
Added deduplication to reduce storage and ingestion overhead by 5-6%, which kept repeated uploads from creating unnecessary processing and storage cost.
Built observability and telemetry across 10+ AI agents and multiple workflows, making debugging failures and drift 10-15% easier through traces, latency, token-usage, and cost visibility.