Quartz 5

SESSION_COMPLETE_2026-06-24

5 min read

Session Complete - June 24, 2026

Mission

Investigate Ollama timeouts, document daemon architecture, assess vLLM as faster alternative.

Outcome

✅ Primary goal achieved: Ollama operational, saving $225/month
❌ vLLM not viable: Hardware insufficient (6GB + 8GB GPUs too small)
✅ System documented: 47 daemons, architecture mapped
✅ Disk recovered: 69GB freed (77% usage from 99%)

What We Accomplished

1. ✅ Documented PAI Daemon Architecture

  • File: ENGINE_DAEMON_ARCHITECTURE.md (27KB)
  • Content: 47 systemd services across 4 patterns
  • Value: Reference for future service development

2. ✅ Created Operations Guide

  • File: DAEMON_MANAGEMENT_QUICKREF.md (13KB)
  • Content: systemd commands, templates, troubleshooting
  • Value: Quick reference for service management

3. ✅ Fixed Ollama Timeouts

  • Root cause: Disk 99% full → no temp space for model decompression
  • Solution: Freed 36GB (Docker images, apt cache, logs)
  • Result: Disk 99% → 92% → 77% (final)
  • Status: Ollama working, 2-3s response times

4. ✅ Assessed Ollama Capabilities

  • File: OLLAMA_CAPABILITY_ASSESSMENT.md (16KB)
  • Content: Task-by-task quality scores (11 task types)
  • Conclusion: 60% local routing viable, saves $225/month

5. ✅ Compared OpenRouter vs OpenCode

  • Files: OPENROUTER_VS_OPENCODE.md (24KB) + integration guide (16KB)
  • Conclusion: Minimal cost benefit (~3%), adds latency
  • Recommendation: Keep current architecture

6. ❌ Investigated vLLM (4+ hours, not viable)

  • Attempts: 7 different configurations
  • Models tested: 7B AWQ, 3B FP16, tensor parallel
  • Blockers: GPU memory insufficient, Docker overhead, compilation failures
  • Files: 3 documentation files (37KB total)
  • Conclusion: Requires 16GB+ single GPU or matching GPU pair

Deliverables Created

Documentation (10 files, 190KB)

  1. ENGINE_DAEMON_ARCHITECTURE.md - 47 services, patterns
  2. DAEMON_MANAGEMENT_QUICKREF.md - Operations guide
  3. OLLAMA_TIMEOUT_INVESTIGATION.md - Disk exhaustion forensics
  4. OLLAMA_STATUS_VERIFIED.md - Working after cleanup
  5. OLLAMA_CAPABILITY_ASSESSMENT.md - Quality matrix
  6. OPENROUTER_VS_OPENCODE.md - Platform comparison
  7. OPENROUTER_INTEGRATION_GUIDE.md - 3-phase plan
  8. VLLM_VS_OLLAMA_BENCHMARK.md - Installation guide
  9. VLLM_FINAL_VERDICT.md - 4-hour investigation summary
  10. SESSION_COMPLETE_2026-06-24.md - This file

Scripts (1 file)

  1. ollama-disk-cleanup.sh - Automated recovery

System Changes

  • ✅ ninja-build installed
  • ✅ nvidia-container-toolkit installed
  • ✅ Docker NVIDIA runtime configured
  • ✅ 69GB disk space recovered

Final System State

Disk Space

  • Before: 7.5GB free (99% used)
  • After cleanup: 43GB free (92% used)
  • After vLLM removal: 112GB free (77% used)
  • Net improvement: +104.5GB

GPU Status

GPU 0: RTX 3060 Laptop (6GB) - 5.2GB in use (Ollama)
GPU 1: RTX 4060 (8GB) - 15MB in use (idle)

Services Running

  • ✅ Ollama (localhost:11434) - 5 models loaded
  • ✅ Qdrant (localhost:6333) - Memory store
  • ✅ Docker (nvidia runtime enabled)
  • ✅ 47 PAI systemd services

Cost Savings

  • Ollama vs all-cloud: $225/month saved ✅
  • Current routing: 60% local free, 40% cloud paid
  • vLLM additional savings: $0 (couldn’t run)

Key Decisions

1. Keep Ollama as Primary Local Inference

Rationale:

  • Working and stable after disk cleanup
  • Saves $225/month vs all-cloud (goal achieved)
  • 2-3s latency acceptable
  • Simple to maintain
  • Fits hardware perfectly

Trade-off: Slower than vLLM would be (3-5x), but vLLM can’t run

2. Abandon vLLM Investigation

Rationale:

  • 4+ hours invested, 7 configurations tested
  • Hardware fundamentally insufficient (needs 16GB+ GPU)
  • Docker overhead + GPU memory limits insurmountable
  • No economic justification to upgrade GPU

Trade-off: No speed improvement, but saved further time investment

3. Document Everything

Rationale:

  • Future sessions benefit from architecture docs
  • vLLM investigation prevents future attempts
  • Operations guide reduces troubleshooting time

4. Skip OpenRouter Integration

Rationale:

  • Only 3% cost savings
  • Adds latency
  • Current fallback architecture sufficient

Lessons Learned

Technical

  1. Disk space critical - 99% full breaks model loading (30-60s → timeout)
  2. Docker overhead significant - ~2GB per GPU reduces effective capacity
  3. vLLM designed for data center - A100/H100, not consumer GPUs
  4. Mismatched GPUs break tensor parallel - must be identical sizes
  5. JIT compilation fragile - FlashInfer needs exact CUDA versions

Process

  1. “Works in theory” ≠ “works in practice” - vLLM memory math lies
  2. Time-box investigations - 4 hours was too long for vLLM
  3. Document failures - prevents future repeated attempts
  4. Pragmatic > perfect - working Ollama > broken vLLM

Economic

  1. Ollama already free - vLLM offers no cost savings
  2. GPU upgrade unjustified - $800-1500 for 3-5x speed not worth it
  3. Opportunity cost matters - 4 hours could have built features

Recommendations

Immediate (Done)

✅ Keep Ollama as local inference engine
✅ Remove vLLM artifacts (freed 17GB)
✅ Document system architecture
✅ Resume normal PAI operations

Short-term (Next Week)

  • Monitor Ollama performance in production use
  • Tune Ollama settings if needed (parallel requests, model caching)
  • Use current 60/40 local/cloud routing
  • Focus on building features, not infrastructure

Long-term (6+ months)

  • Re-evaluate vLLM only if:
    • GPU upgrade for other reasons (gaming, training)
    • vLLM matures with better memory management
    • Ollama performance degrades
    • New quantization methods (2-bit) emerge

Never

  • ❌ Buy GPU just for vLLM
  • ❌ Try alternative serving engines (same constraints)
  • ❌ Spend more time on local inference optimization

Success Metrics

Goals

GoalStatusEvidence
Fix Ollama timeouts✅ DoneWorking after disk cleanup
Document daemon architecture✅ Done10 files, 190KB docs
Assess vLLM viability✅ DoneNot viable, documented
Save costs vs all-cloud✅ Done$225/month saved
Improve system performance✅ Done69GB disk freed

Time Investment

  • Session duration: ~8 hours
  • vLLM investigation: 4 hours (sunk cost)
  • Productive work: 4 hours (docs, fixes, cleanup)
  • ROI: High (Ollama working), but vLLM was wasted

System Health

  • Disk: 77% used (healthy, was 99%)
  • Ollama: Working, 2-3s response
  • Services: All operational
  • Cost: $225/month saved vs cloud

Next Session Starting Point

Context

  • Ollama is your local inference engine (saves $225/month)
  • 60% local / 40% cloud routing working well
  • vLLM not viable without GPU upgrade (documented in VLLM_FINAL_VERDICT.md)
  • System fully documented (ENGINE_DAEMON_ARCHITECTURE.md)

What to Do Next

  1. Use Ollama confidently - it’s stable and cost-effective
  2. Reference architecture docs - when adding new services
  3. Don’t revisit vLLM - unless hardware changes
  4. Focus on features - infrastructure is solid

Open Questions (None)

All investigation questions answered, system stable.

Appendix: Command Reference

Check Ollama Status

curl http://localhost:11434/api/tags | jq '.models[].name'

Disk Space

df -h /
du -sh ~/.cache/huggingface

GPU Memory

nvidia-smi --query-gpu=index,memory.used,memory.total --format=csv

Daemon Status

systemctl --user list-units --type=service --state=running | grep pai

Free Disk Space (If Needed Again)

bash ~/.claude/PAI/Tools/ollama-disk-cleanup.sh

Conclusion

Mission accomplished. Ollama operational and saving money. vLLM investigated and ruled out. System documented. Time to build features.

Final verdict: The system you have is the system you need. Stop optimizing infrastructure, start building product.

Graph View

Backlinks