# Error Analysis & Root Cause - Orchestrator Production Run
## May 4, 2026

---

## Executive Summary

**Masalah:** 5 dari 9 PDF gagal diproses, hanya 4 yang berhasil dibuat node Neo4j.  
**Success Rate:** 44.4% (target: 100%)  
**Root Cause:** 2 kategori error berbeda pada pipeline stages yang berbeda

| Kategori | Jumlah | Penyebab | Severity |
|----------|--------|---------|----------|
| PDF Parsing Error | 4 files | `pymupdf4llm.to_markdown()` returns None | 🔴 High |
| Pillar Extraction Error | 1 file | `response.choices[0]` subscript on None | 🔴 High |
| Success | 4 files | Complete pipeline executed | ✅ OK |

---

## Detailed Root Cause Analysis

### Category A: PDF Parsing Error ('NoneType' object is not iterable)

**Affected Files (4):**
1. `Reviving Life on the Edge...` [1/9]
2. `Retrieval-Augmented Generation for Knowledge-Intensive NLP Tasks.pdf` [6/9]
3. `SciLit Joint Literature Discovery...` [7/9]
4. `Proses Pembuatan Pupuk Kompos...` [9/9]

**Error Trace:**
```
[HybridParser] Error pada _extract_with_pymupdf4llm: 'NoneType' object is not iterable
Melewati file ... karena gagal mengekstrak Markdown/Teks.
```

**Root Cause Code (core/parser/hybrid_parser.py:78-107):**

```python
def _extract_with_pymupdf4llm(self, pdf_path: str) -> str:
    with tempfile.TemporaryDirectory() as temp_dir:
        try:
            md_text = pymupdf4llm.to_markdown(...)  # ← Line 83: Returns None in some cases
            
            for img_file in os.listdir(temp_dir):  # ← Line 107: Error here if md_text is None
                if img_file.endswith(".png"):
                    # ... image processing ...
                    
            return md_text  # ← Could be None!
```

**Why pymupdf4llm Returns None:**

1. **PDF with No Extractable Content**
   - Completely corrupted PDF
   - PDF with invalid structure (malformed)
   - Empty PDF with no text or images

2. **pymupdf4llm Configuration Issues**
   - `header=False, footer=False` might strip all content
   - `margins=(0, 72, 0, 72)` could cut important content
   - Incompatibility with specific PDF versions

3. **Silent Failure**
   - pymupdf4llm catches exceptions internally and returns None
   - No error logs provided by library
   - Makes debugging difficult

**Impact:**
- Function returns empty string "" instead of raising exception
- `generate_chunks("")` produces empty chunk list
- File skipped silently without node creation

---

### Category B: Pillar Extraction Error ('NoneType' object is not subscriptable)

**Affected File (1):**
- `Sistem Budidaya Ikan Lele...` [8/9]

**Error Trace:**
```
Chunk mentah berhasil dimasukkan ke antrean Supabase: 34 chunk
Filtering aktif: 24 chunk relevan dipakai untuk LLM...
Gagal memproses file ... di Main Thread: 'NoneType' object is not subscriptable
```

**Root Cause Code (orchestrator_ingest.py:297-300):**

```python
def extract_pillars_with_llm(text: str, client) -> dict:
    # ... prompt construction ...
    
    response = client.chat.completions.create(...)  # ← Line 297
    raw_json = response.choices[0].message.content  # ← Line 298: CRASH HERE
    
    try:
        parsed = json.loads(raw_json)
        # ...
```

**Why response is None or Missing choices:**

1. **API Response Timeout**
   - OpenRouter API takes too long (>30s timeout)
   - Client returns None or empty response

2. **Rate Limiting**
   - Free tier has 15 RPM limit
   - Even with 5s delays, bursts exceed limit
   - Returns 429 Too Many Requests → malformed response

3. **Invalid API Key or Credits**
   - API key invalid or out of credits
   - Returns 401 Unauthorized with no choices

4. **Network Error**
   - Connection interrupted mid-response
   - Partial response received, no choices array

**Impact:**
- Chunks are already queued to Supabase (no rollback)
- Node not created in Neo4j
- Orphaned chunks without parent document node
- No recovery mechanism

---

## Current System Flow (with Failures)

```
PDF File
   ↓
HybridParser.parse_pdf()
   ├─ is_gibberish() check  
   └─ _extract_with_pymupdf4llm()  ← ❌ Error: Returns None
       └─ pymupdf4llm.to_markdown()  ← Silently returns None on error
       
   Result: Empty string ""
   ↓
generate_chunks("")  ← Returns empty list
   ↓
upload_queue.put(chunk)  ← Never called
   ↓
extract_pillars_with_llm()  ← ❌ Never called (no chunks)
   ↓
Node NOT created in Neo4j
```

**vs. When Chunks Extracted but Pillar Fails:**

```
PDF File
   ↓
HybridParser.parse_pdf()
   └─ _extract_with_pymupdf4llm()  ✅ Success
   ↓
generate_chunks(md)  ✅ Success (34 chunks)
   ↓
build_llm_context_from_chunks()  ✅ Success (24 relevant)
   ↓
upload_queue.put(chunk) × 34  ✅ Queued successfully
   ↓
extract_pillars_with_llm()  ← ❌ response is None
   ├─ response.choices[0]  ← Crash: 'NoneType' object is not subscriptable
   ↓
Exception caught, file skipped
   ↓
Node NOT created, but chunks ORPHANED in Supabase
```

---

## Quality Impact Assessment

### Issue 1: Data Inconsistency
- **Problem:** Chunks in Supabase without corresponding Neo4j node
- **Risk:** Orphaned chunks → unused storage → slow queries
- **Severity:** Medium

### Issue 2: Silent Failures
- **Problem:** Errors logged but processing continues
- **Risk:** User doesn't know which files failed without parsing logs
- **Severity:** Low (logs are visible)

### Issue 3: No Retry/Recovery
- **Problem:** Failed files are not retried
- **Risk:** Single transient API error fails entire file permanently
- **Severity:** High

### Issue 4: API Rate Limiting
- **Problem:** 5-second delays insufficient for bursts
- **Risk:** Same-second API calls trigger 429 errors
- **Severity:** Medium

---

## Deep Dive: Why Specific Files Failed

### File [1/9]: Reviving Life on the Edge
**Status:** Parsing failed  
**Hypothesis:**  
- PDF structure unusual or corrupted
- Contains only scanned images (OCR config issue)
- Invalid UTF-8 in metadata

**Evidence:**
- `[HybridParser] Error pada _extract_with_pymupdf4llm: 'NoneType' object is not iterable`
- No chunks mentioned → extraction returned None

---

### File [6/9]: Retrieval-Augmented Generation for Knowledge-Intensive NLP Tasks
**Status:** Parsing failed  
**Hypothesis:**
- Same as File 1 (likely same error pattern)

---

### File [7/9]: SciLit Joint Literature Discovery
**Status:** Parsing failed  
**Hypothesis:**
- Same as File 1

---

### File [8/9]: Sistem Budidaya Ikan Lele (Clarias sp.)
**Status:** Chunks extracted (34) → Pillar extraction failed  
**Hypothesis:**
- PDF parsed successfully ✅
- Chunks generated and filtered ✅  
- Chunks queued to Supabase ✅
- **LLM API call failed:** response is None or response.choices is empty
- Most likely: API rate limit or timeout

**Evidence:**
```log
Chunk mentah berhasil dimasukkan ke antrean Supabase: 34 chunk
Filtering aktif: 24 chunk relevan dipakai untuk LLM...
Gagal memproses file ... di Main Thread: 'NoneType' object is not subscriptable
```

---

### File [9/9]: Proses Pembuatan Pupuk Kompos
**Status:** Parsing failed  
**Hypothesis:**
- Same as Files 1, 6, 7

---

### Files [2/4/5] & [3] Success (✅ 4/9)
- SemanticGraph2Vec: 42 → 28 chunks ✅
- HMGI: 77 → 26 chunks ✅
- RAGAS: 33 → 23 chunks ✅
- PaperQA: 69 → 24 chunks ✅

**Why Succeeded:**
- PDFs have standard structure
- Text extraction works with default params
- LLM API responded successfully

---

## Solution Architecture

### Fix 1: Defensive PDF Parsing (HybridParser)

**Problem:** `pymupdf4llm.to_markdown()` returns None, causing "'NoneType' object is not iterable"

**Solution Approach:**
```python
# BEFORE (Vulnerable)
md_text = pymupdf4llm.to_markdown(...)
for img_file in os.listdir(temp_dir):  # ← Crashes if md_text is None

# AFTER (Defensive)
md_text = pymupdf4llm.to_markdown(...) or ""  # Ensure string, never None
if not md_text.strip():  # Validate non-empty
    return self._full_ocr(pdf_path)  # Fallback
```

**Benefits:**
- ✅ No None propagation
- ✅ Automatic fallback to OCR
- ✅ Better error logging
- ✅ 100% string return guarantee

---

### Fix 2: Defensive LLM Response Handling (Orchestrator)

**Problem:** `response.choices[0]` fails when response is None or choices is empty

**Solution Approach:**
```python
# BEFORE (Vulnerable)
response = client.chat.completions.create(...)
raw_json = response.choices[0].message.content  # Can crash

# AFTER (Defensive)
try:
    response = client.chat.completions.create(...)
    if not response or not response.choices:  # Explicit None check
        return json.loads("{}")  # Return empty pillars
    raw_json = response.choices[0].message.content
except (AttributeError, IndexError, TypeError) as e:  # Catch subscript errors
    print(f"LLM response error: {e}")
    return json.loads("{}")
```

**Benefits:**
- ✅ Handles None responses gracefully
- ✅ Catches all subscript errors
- ✅ Clear error messages
- ✅ Pipeline continues with empty pillars

---

### Fix 3: Transactional Chunk Management

**Problem:** Chunks uploaded even if node creation fails → orphaned chunks

**Solution Approach:**
```python
# BEFORE (Non-transactional)
if upload_queue is not None and chunk_records:
    for chunk in chunk_records:
        upload_queue.put(chunk)  # ← Uploaded immediately
    
# ... extract pillars ...
# ← If this fails, chunks are orphaned

# AFTER (Transactional)
chunks_to_upload = []
for chunk in chunk_records:
    chunks_to_upload.append(chunk)  # Queue locally

# Extract pillars FIRST
pillars = extract_pillars_with_llm(raw_text, or_client)
if not pillars or all(not v for v in pillars.values()):
    return None  # Skip upload if no pillars

# ONLY THEN upload chunks
for chunk in chunks_to_upload:
    upload_queue.put(chunk)
```

**Benefits:**
- ✅ No orphaned chunks
- ✅ All-or-nothing semantics
- ✅ Consistency across Neo4j + Supabase

---

### Fix 4: Enhanced API Rate Limiting

**Problem:** 5-second delays insufficient, triggers 429 Too Many Requests

**Solution Approach:**
```python
# BEFORE
time.sleep(5)  # Too aggressive

# AFTER
# Dynamic backoff for embedding API
time.sleep(4 + random.uniform(0, 2))  # 4-6 seconds
# Or implement exponential backoff on 429 responses
```

**Benefits:**
- ✅ Better rate limit compliance
- ✅ Random jitter prevents thundering herd
- ✅ Adaptive to API behavior

---

### Fix 5: Better Error Logging

**Problem:** Generic error messages don't help debugging

**Solution Approach:**
```python
# BEFORE
except Exception as e:
    print(f"Gagal memproses file {pdf_path} di Main Thread: {e}")
    return None

# AFTER
except Exception as e:
    import traceback
    print(f"[ERROR] File processing failed: {pdf_path}")
    print(f"  Error Type: {type(e).__name__}")
    print(f"  Message: {str(e)}")
    print(f"  Traceback: {traceback.format_exc()}")
    return None
```

**Benefits:**
- ✅ Full error context for debugging
- ✅ File-specific error isolation
- ✅ Trackable error patterns

---

## Implementation Plan

### Phase 1: Immediate Fixes (Critical)
- [ ] Fix `_extract_with_pymupdf4llm()` null checks
- [ ] Fix `extract_pillars_with_llm()` response validation
- [ ] Add comprehensive error logging

**Time Estimate:** 30-45 minutes  
**Scope:** 2 files modified

### Phase 2: Quality Improvements (Important)
- [ ] Implement transactional chunk upload
- [ ] Add API rate limit backoff
- [ ] Create error recovery mechanism

**Time Estimate:** 60-90 minutes  
**Scope:** 3 files modified

### Phase 3: Testing & Validation (Essential)
- [ ] Rerun orchestrator on same 9 PDFs
- [ ] Verify 100% of well-formed PDFs process successfully
- [ ] Test error handling with edge cases

**Time Estimate:** 45 minutes  
**Scope:** Run test suite

### Phase 4: Documentation (Best Practice)
- [ ] Update walkthrough.md with error handling
- [ ] Document API rate limit requirements
- [ ] Create troubleshooting guide for operators

**Time Estimate:** 30 minutes

---

## Prevention Strategies

### 1. Input Validation
```python
# Validate PDF before processing
def validate_pdf(pdf_path):
    if not os.path.exists(pdf_path):
        return False, "File not found"
    if os.path.getsize(pdf_path) == 0:
        return False, "Empty file"
    if os.path.getsize(pdf_path) > 100 * 1024 * 1024:  # 100MB limit
        return False, "File too large"
    return True, "OK"
```

### 2. Chunking Validation
```python
# Validate chunks before upload
def validate_chunks(chunks):
    if not chunks:
        raise ValueError("No chunks generated")
    if len(chunks) > 10000:
        raise ValueError("Too many chunks")
    return True
```

### 3. Retry Logic
```python
# Exponential backoff on transient failures
def retry_with_backoff(func, max_retries=3):
    for attempt in range(max_retries):
        try:
            return func()
        except (TimeoutError, ConnectionError) as e:
            if attempt == max_retries - 1:
                raise
            wait_time = (2 ** attempt) + random.uniform(0, 1)
            print(f"Retry {attempt+1}/{max_retries} after {wait_time:.2f}s")
            time.sleep(wait_time)
```

### 4. Monitoring & Alerting
```python
# Track per-file metrics
success_rate = 4 / 9 = 44.4%
parsed_rate = 5 / 9 = 55.6%
extraction_rate = 4 / 5 = 80%

# Alert if below threshold
if success_rate < 0.8:
    logging.warning(f"Success rate below 80%: {success_rate}")
```

---

## Recommendations

### Short-term (This Week)
1. **Implement Phase 1 fixes** → Stop cascading errors
2. **Retest with same 9 PDFs** → Verify improvement
3. **Document root causes** → Knowledge base

### Medium-term (This Sprint)
1. **Implement Phase 2** → Robust error handling
2. **Add monitoring** → Detect failures early
3. **Create retry dashboard** → Operator visibility

### Long-term (This Quarter)
1. **PDF validation suite** → Reject bad PDFs early
2. **Performance tuning** → Optimize API delays
3. **Fallback mechanisms** → Alternative extraction methods

---

## Next Steps

1. ✅ **Analysis Complete** — This document
2. ⏳ **Code Implementation** — Apply fixes from Phase 1 & 2
3. ⏳ **Testing** — Rerun orchestrator
4. ⏳ **Documentation** — Update walkthrough.md

**Estimated Time to Resolution:** 2-3 hours for Phase 1+2+3

---

## Appendix: File Inspection

To understand specific PDFs better, run:

```bash
# Check PDF metadata
pdffile="Literature/SemanticGraph2Vec.pdf"
python3 -c "
import fitz
doc = fitz.open('$pdffile')
print(f'Pages: {len(doc)}')
print(f'Text length: {len(doc[0].get_text())}')
print(f'Embedded images: {len(doc[0].get_images())}')
"

# Test extraction on single file
python3 -c "
from core.parser.hybrid_parser import HybridParser
parser = HybridParser()
md = parser.parse_pdf('$pdffile')
print(f'Extraction length: {len(md) if md else 0}')
print(f'First 200 chars: {md[:200] if md else \"NONE\"}')
"
```

---

**Document Version:** 1.0  
**Last Updated:** May 4, 2026  
**Status:** Ready for Implementation
