# 🐛 BUG #1: Duplicate Code Analysis
## `clean_llm_json_string` & `sanitize_llm_field`

**Bug ID:** BUG-001  
**Severity:** 🔴 CRITICAL  
**Status:** UNRESOLVED  
**Date Found:** 2026-05-12  

---

## 📍 Lokasi Bug

### File 1: `ingestion.py`
```
Path: /home/daegir/Documents/PERCOBAAN/Presentasi/backend-sre/app/services/ingestion.py
Lines: ~170-210 (approximately)
```

### File 2: `llm_extractor.py`
```
Path: /home/daegir/Documents/PERCOBAAN/Presentasi/backend-sre/app/services/parser/llm_extractor.py
Lines: ~10-50 (approximately)
```

---

## 🔍 Evidence: Side-by-Side Comparison

### FUNCTION 1: `clean_llm_json_string`

#### ❌ INGESTION.PY (Duplicate #1)
```python
def clean_llm_json_string(raw_text: str) -> str:
    """
    LAPIS 1 (The Sweeper):
    Membersihkan teks respons LLM dari basa-basi dan markdown fences.
    Hanya mengambil teks di antara tanda kurung kurawal pertama dan terakhir.
    """
    if not raw_text:
        return ""
    
    start_idx = raw_text.find('{')
    end_idx = raw_text.rfind('}')
    
    if start_idx != -1 and end_idx != -1 and start_idx < end_idx:
        return raw_text[start_idx:end_idx+1]
    
    return raw_text
```

#### ✅ LLM_EXTRACTOR.PY (Duplicate #2)
```python
def clean_llm_json_string(raw_text: str) -> str:
    """
    LAPIS 1 (The Sweeper):
    Membersihkan teks respons LLM dari basa-basi dan markdown fences.
    Hanya mengambil teks di antara tanda kurung kurawal pertama dan terakhir.
    """
    if not raw_text:
        return ""
    
    start_idx = raw_text.find('{')
    end_idx = raw_text.rfind('}')
    
    if start_idx != -1 and end_idx != -1 and start_idx < end_idx:
        return raw_text[start_idx:end_idx+1]
    
    return raw_text
```

**Status:** ✅ **IDENTICAL** (100% match)

---

### FUNCTION 2: `sanitize_llm_field`

#### ❌ INGESTION.PY (Duplicate #1)
```python
def sanitize_llm_field(value) -> str:
    """
    LAPIS 3 (The Pydantic Armor):
    Mengoersi berbagai macam respons halusinasi LLM (None, [], int, frasa palsu)
    menjadi string murni yang aman bagi Neo4j.
    """
    if value is None:
        return ""
        
    # Coerce ke string jika berupa list atau number
    if isinstance(value, list):
        value = " ".join(str(v) for v in value)
    elif not isinstance(value, str):
        value = str(value)
        
    value = value.strip()
    
    # Filter respons penolakan LLM (Information not available)
    reject_phrases = [
        "information not available",
        "informasi tidak tersedia",
        "not specified",
        "tidak disebutkan",
        "tidak ada",
        "none"
    ]
    
    value_lower = value.lower()
    for phrase in reject_phrases:
        if phrase in value_lower and len(value) < 100:
            return ""
            
    return value
```

#### ✅ LLM_EXTRACTOR.PY (Duplicate #2)
```python
def sanitize_llm_field(value) -> str:
    """
    LAPIS 3 (The Pydantic Armor):
    Mengoersi berbagai macam respons halusinasi LLM (None, [], int, frasa palsu)
    menjadi string murni yang aman bagi Neo4j.
    """
    if value is None:
        return ""
        
    # Coerce ke string jika berupa list atau number
    if isinstance(value, list):
        value = " ".join(str(v) for v in value)
    elif not isinstance(value, str):
        value = str(value)
        
    value = value.strip()
    
    # Filter respons penolakan LLM (Information not available)
    reject_phrases = [
        "information not available",
        "informasi tidak tersedia",
        "not specified",
        "tidak disebutkan",
        "tidak ada",
        "none"
    ]
    
    value_lower = value.lower()
    for phrase in reject_phrases:
        if phrase in value_lower and len(value) < 100:
            return ""
            
    return value
```

**Status:** ✅ **IDENTICAL** (100% match)

---

## 🎯 Apa Masalahnya?

### Problem Statement

Kedua fungsi **`clean_llm_json_string`** dan **`sanitize_llm_field`** didefinisikan **persis sama** di dua file berbeda:

```
ingestion.py          llm_extractor.py
     ↓                      ↓
  [COPY]  ←→  [ORIGINAL]
```

Ini adalah **code duplication anti-pattern** yang sangat berbahaya.

---

## 💥 Impact Analysis

### 1️⃣ **Maintenance Nightmare**

Jika ada bug di `clean_llm_json_string`, harus di-fix di **2 tempat**:

**Skenario:** Ditemukan bug di logic pembersihan JSON
```python
# BUG: Tidak handle case ketika ada multiple { atau }
raw_text = "Some text {json1} more text {json2}"
result = clean_llm_json_string(raw_text)
# Current: Returns "{json1} more text {json2}" ← WRONG
# Expected: Should handle multiple JSON objects
```

**Jika hanya di-fix di `ingestion.py`:**
```python
# ingestion.py (FIXED ✅)
def clean_llm_json_string(raw_text: str) -> str:
    # ... improved logic ...

# llm_extractor.py (STILL BROKEN ❌)
def clean_llm_json_string(raw_text: str) -> str:
    # ... old buggy logic ...
```

**Result:** 
- Ketika `llm_extractor.py` dipanggil → menggunakan logic lama (BUG)
- Ketika `ingestion.py` dipanggil → menggunakan logic baru (OK)
- **Inconsistent behavior** di aplikasi

---

### 2️⃣ **Inconsistency Risk**

Developer baru mungkin update satu file tapi lupa yang lain:

```python
# Developer A: Update reject_phrases di ingestion.py
reject_phrases = [
    "information not available",
    "informasi tidak tersedia",
    "not specified",
    "tidak disebutkan",
    "tidak ada",
    "none",
    "N/A",  # ← NEW
    "tidak diketahui",  # ← NEW
]

# Developer B: Tidak tahu ada duplicate, tidak update llm_extractor.py
# llm_extractor.py masih punya list lama
```

**Result:** 
- `ingestion.py` filter "N/A" ✅
- `llm_extractor.py` tidak filter "N/A" ❌
- **Silent bug** yang sulit di-debug

---

### 3️⃣ **Version Control Nightmare**

Ketika merge branches:

```
Branch A: Update clean_llm_json_string di ingestion.py
Branch B: Update clean_llm_json_string di llm_extractor.py

Merge conflict! ⚠️
```

Developer harus manually resolve conflict di 2 tempat.

---

### 4️⃣ **Testing Complexity**

Harus test kedua fungsi di kedua file:

```python
# Test ingestion.py version
def test_clean_llm_json_string_ingestion():
    result = ingestion.clean_llm_json_string("{...}")
    assert result == "{...}"

# Test llm_extractor.py version
def test_clean_llm_json_string_extractor():
    result = llm_extractor.clean_llm_json_string("{...}")
    assert result == "{...}"

# Jika logic berbeda, test bisa fail di satu tempat tapi pass di tempat lain
```

---

### 5️⃣ **Cognitive Load**

Developer harus ingat:
- "Mana yang authoritative version?"
- "Apakah kedua file punya logic yang sama?"
- "Jika update, harus update di mana saja?"

---

## 🔄 Call Flow: Bagaimana Duplicate Ini Terjadi?

### Current Architecture (BROKEN)

```
ingestion.py
├── import clean_llm_json_string (LOCAL)
├── import sanitize_llm_field (LOCAL)
└── def extract_pillars_with_llm():
    └── uses clean_llm_json_string() ← dari ingestion.py

llm_extractor.py
├── def clean_llm_json_string() (LOCAL DUPLICATE)
├── def sanitize_llm_field() (LOCAL DUPLICATE)
└── class ExtractionResponse:
    └── uses sanitize_llm_field() ← dari llm_extractor.py
```

**Problem:** Dua source of truth untuk fungsi yang sama!

---

## ✅ Solution: Single Source of Truth

### Proposed Architecture (FIXED)

```
llm_extractor.py (AUTHORITATIVE)
├── def clean_llm_json_string() ← SINGLE DEFINITION
├── def sanitize_llm_field() ← SINGLE DEFINITION
└── class ExtractionResponse:
    └── uses sanitize_llm_field()

ingestion.py (CONSUMER)
├── from app.services.parser.llm_extractor import (
│       clean_llm_json_string,
│       sanitize_llm_field
│   )
└── def extract_pillars_with_llm():
    └── uses clean_llm_json_string() ← dari llm_extractor
```

**Benefit:**
- ✅ Single source of truth
- ✅ One place to update
- ✅ No inconsistency
- ✅ Easier testing
- ✅ Cleaner imports

---

## 🛠️ Fix Implementation

### Step 1: Verify `llm_extractor.py` has complete functions

```python
# llm_extractor.py (KEEP THIS)
def clean_llm_json_string(raw_text: str) -> str:
    """LAPIS 1 (The Sweeper)"""
    if not raw_text:
        return ""
    start_idx = raw_text.find('{')
    end_idx = raw_text.rfind('}')
    if start_idx != -1 and end_idx != -1 and start_idx < end_idx:
        return raw_text[start_idx:end_idx+1]
    return raw_text

def sanitize_llm_field(value) -> str:
    """LAPIS 3 (The Pydantic Armor)"""
    if value is None:
        return ""
    if isinstance(value, list):
        value = " ".join(str(v) for v in value)
    elif not isinstance(value, str):
        value = str(value)
    value = value.strip()
    reject_phrases = [
        "information not available",
        "informasi tidak tersedia",
        "not specified",
        "tidak disebutkan",
        "tidak ada",
        "none"
    ]
    value_lower = value.lower()
    for phrase in reject_phrases:
        if phrase in value_lower and len(value) < 100:
            return ""
    return value
```

### Step 2: Remove from `ingestion.py`

```python
# ingestion.py (DELETE THESE)
# ❌ def clean_llm_json_string(raw_text: str) -> str:
# ❌ def sanitize_llm_field(value) -> str:
```

### Step 3: Add import to `ingestion.py`

```python
# ingestion.py (ADD THIS AT TOP)
from app.services.parser.llm_extractor import (
    clean_llm_json_string,
    sanitize_llm_field,
    ExtractionResponse,
    LiteratureMetadata,
    LiteraturePillars,
)
```

### Step 4: Update all usages in `ingestion.py`

```python
# Before (using local function)
result = clean_llm_json_string(raw_response)

# After (same, but now from llm_extractor)
result = clean_llm_json_string(raw_response)  # ← imported from llm_extractor
```

---

## 📊 Before vs After

### BEFORE (BROKEN)
```
ingestion.py (54.5 KB)
├── clean_llm_json_string() [DUPLICATE]
├── sanitize_llm_field() [DUPLICATE]
├── LiteratureMetadata [DUPLICATE]
├── LiteraturePillars [DUPLICATE]
├── ExtractionResponse [DUPLICATE]
└── extract_pillars_with_llm()

llm_extractor.py (2.9 KB)
├── clean_llm_json_string() [DUPLICATE]
├── sanitize_llm_field() [DUPLICATE]
├── LiteratureMetadata [DUPLICATE]
├── LiteraturePillars [DUPLICATE]
└── ExtractionResponse [DUPLICATE]

Total: 57.4 KB (BLOATED)
```

### AFTER (FIXED)
```
ingestion.py (48 KB)
├── from llm_extractor import (
│   clean_llm_json_string,
│   sanitize_llm_field,
│   ExtractionResponse,
│   LiteratureMetadata,
│   LiteraturePillars
│ )
└── extract_pillars_with_llm()

llm_extractor.py (2.9 KB)
├── clean_llm_json_string() [AUTHORITATIVE]
├── sanitize_llm_field() [AUTHORITATIVE]
├── LiteratureMetadata [AUTHORITATIVE]
├── LiteraturePillars [AUTHORITATIVE]
└── ExtractionResponse [AUTHORITATIVE]

Total: 50.9 KB (CLEANER)
```

**Savings:** ~6.5 KB (11% reduction)  
**Benefit:** Single source of truth

---

## 🧪 Testing Strategy

### Test Case 1: Verify import works
```python
def test_import_from_llm_extractor():
    from app.services.parser.llm_extractor import clean_llm_json_string
    assert callable(clean_llm_json_string)
```

### Test Case 2: Verify ingestion uses imported function
```python
def test_ingestion_uses_imported_function():
    from app.services.ingestion import clean_llm_json_string as ing_func
    from app.services.parser.llm_extractor import clean_llm_json_string as ext_func
    
    # Should be the same function
    assert ing_func is ext_func
```

### Test Case 3: Verify behavior consistency
```python
def test_clean_llm_json_string_behavior():
    from app.services.parser.llm_extractor import clean_llm_json_string
    
    test_cases = [
        ('{"key": "value"}', '{"key": "value"}'),
        ('Some text {"key": "value"} more text', '{"key": "value"}'),
        ('No JSON here', 'No JSON here'),
        ('', ''),
    ]
    
    for input_text, expected in test_cases:
        result = clean_llm_json_string(input_text)
        assert result == expected, f"Failed for {input_text}"
```

---

## ⏱️ Effort Estimate

| Task | Time |
|------|------|
| Remove duplicate from `ingestion.py` | 5 min |
| Add import statement | 2 min |
| Update all usages (if any) | 5 min |
| Run tests | 5 min |
| Verify no regression | 10 min |
| **Total** | **27 min** |

---

## 🎯 Acceptance Criteria

- [ ] Duplicate functions removed from `ingestion.py`
- [ ] Import statement added to `ingestion.py`
- [ ] All tests pass
- [ ] No regression in functionality
- [ ] Code review approved
- [ ] Deployed to staging

---

## 📝 Summary

**BUG #1** adalah **code duplication** yang menciptakan:
- ❌ Maintenance burden (fix di 2 tempat)
- ❌ Inconsistency risk (logic bisa berbeda)
- ❌ Version control conflicts
- ❌ Testing complexity
- ❌ Cognitive load

**FIX:** Pindahkan ke `llm_extractor.py` (authoritative), import di `ingestion.py`

**Impact:** Cleaner code, single source of truth, easier maintenance

---

**Status:** Ready for implementation  
**Priority:** 🔴 CRITICAL (Blocker untuk V3)  
**Estimated Fix Time:** 27 minutes
