# 🔧 BUG #1: Solving Strategy & Implementation Plan

**Bug ID:** BUG-001  
**Title:** Duplicate Code - `clean_llm_json_string` & `sanitize_llm_field`  
**Status:** PLANNING PHASE  
**Date:** 2026-05-12  

---

## 📋 Phase 1: Analysis & Planning

### 1.1 Identifikasi Semua Duplicate Code

Dari audit, ada **5 item yang di-duplicate**:

| # | Item | ingestion.py | llm_extractor.py | Status |
|---|------|--------------|------------------|--------|
| 1 | `clean_llm_json_string()` | ✅ Ada | ✅ Ada | DUPLICATE |
| 2 | `sanitize_llm_field()` | ✅ Ada | ✅ Ada | DUPLICATE |
| 3 | `LiteratureMetadata` | ✅ Ada | ✅ Ada | DUPLICATE |
| 4 | `LiteraturePillars` | ✅ Ada | ✅ Ada | DUPLICATE |
| 5 | `ExtractionResponse` | ✅ Ada | ✅ Ada | DUPLICATE |

**Scope:** Kita fokus pada **#1 dan #2** (fungsi), sisanya adalah Pydantic models yang akan di-handle di V3 modularization.

---

### 1.2 Identifikasi Semua Usage Points

Saya perlu cari di mana `clean_llm_json_string` dan `sanitize_llm_field` **digunakan** di `ingestion.py`:

#### Usage #1: `clean_llm_json_string`
```python
# Lokasi: ingestion.py (perlu dicari di mana dipanggil)
# Kemungkinan di: extract_pillars_with_llm() atau fungsi lain yang process LLM response
```

#### Usage #2: `sanitize_llm_field`
```python
# Lokasi: ingestion.py (perlu dicari di mana dipanggil)
# Kemungkinan di: Pydantic BeforeValidator untuk LiteraturePillars
```

**Action:** Perlu scan `ingestion.py` untuk menemukan semua call sites.

---

### 1.3 Tentukan Authoritative Source

**Decision:** `llm_extractor.py` adalah **authoritative source** karena:
- ✅ Sudah terpisah (modul khusus untuk ekstraksi)
- ✅ Lebih semantically correct (fungsi ekstraksi ada di modul ekstraksi)
- ✅ Sesuai dengan V3 modularization plan
- ✅ Sudah ada Pydantic models di sana

**Implication:** 
- ✅ KEEP di `llm_extractor.py`
- ❌ DELETE dari `ingestion.py`
- ✅ IMPORT di `ingestion.py`

---

## 📍 Phase 2: Detailed Implementation Steps

### Step 1: Verify `llm_extractor.py` is Complete

**Action:** Pastikan `llm_extractor.py` punya semua yang dibutuhkan:

```python
# llm_extractor.py harus punya:
✅ clean_llm_json_string()
✅ sanitize_llm_field()
✅ LiteratureMetadata
✅ LiteraturePillars
✅ ExtractionResponse
```

**Verification Command:**
```bash
grep -n "def clean_llm_json_string\|def sanitize_llm_field\|class LiteratureMetadata\|class LiteraturePillars\|class ExtractionResponse" app/services/parser/llm_extractor.py
```

**Expected Output:**
```
10: def clean_llm_json_string(raw_text: str) -> str:
30: def sanitize_llm_field(value) -> str:
50: class LiteratureMetadata(BaseModel):
55: class LiteraturePillars(BaseModel):
65: class ExtractionResponse(BaseModel):
```

---

### Step 2: Find All Usage Points in `ingestion.py`

**Action:** Cari semua tempat di `ingestion.py` yang menggunakan fungsi duplicate:

```bash
# Cari penggunaan clean_llm_json_string
grep -n "clean_llm_json_string" app/services/ingestion.py

# Cari penggunaan sanitize_llm_field
grep -n "sanitize_llm_field" app/services/ingestion.py

# Cari penggunaan LiteratureMetadata
grep -n "LiteratureMetadata" app/services/ingestion.py

# Cari penggunaan LiteraturePillars
grep -n "LiteraturePillars" app/services/ingestion.py

# Cari penggunaan ExtractionResponse
grep -n "ExtractionResponse" app/services/ingestion.py
```

**Expected Output (contoh):**
```
170: def clean_llm_json_string(raw_text: str) -> str:
190: def sanitize_llm_field(value) -> str:
210: class LiteratureMetadata(BaseModel):
215: class LiteraturePillars(BaseModel):
225: class ExtractionResponse(BaseModel):
350: response = ExtractionResponse.model_validate(parsed_dict)
400: cleaned_json = clean_llm_json_string(raw_response)
```

---

### Step 3: Add Import Statement to `ingestion.py`

**Location:** Top of file, after existing imports

**Current State:**
```python
from app.services.parser.hybrid_parser import HybridParser
from app.services.parser.smart_chunker import build_llm_context_from_chunks, generate_chunks, generate_parent_child_chunks
from app.services.parser.semantic_retriever import SemanticRetriever
```

**New Import to Add:**
```python
from app.services.parser.llm_extractor import (
    clean_llm_json_string,
    sanitize_llm_field,
    LiteratureMetadata,
    LiteraturePillars,
    ExtractionResponse,
)
```

**Result:**
```python
from app.services.parser.hybrid_parser import HybridParser
from app.services.parser.smart_chunker import build_llm_context_from_chunks, generate_chunks, generate_parent_child_chunks
from app.services.parser.semantic_retriever import SemanticRetriever
from app.services.parser.llm_extractor import (  # ← NEW
    clean_llm_json_string,
    sanitize_llm_field,
    LiteratureMetadata,
    LiteraturePillars,
    ExtractionResponse,
)
```

---

### Step 4: Remove Duplicate Definitions from `ingestion.py`

**Action:** Delete 5 duplicate definitions:

#### Delete #1: `clean_llm_json_string` function
```python
# DELETE THESE LINES:
def clean_llm_json_string(raw_text: str) -> str:
    """
    LAPIS 1 (The Sweeper):
    Membersihkan teks respons LLM dari basa-basi dan markdown fences.
    Hanya mengambil teks di antara tanda kurung kurawal pertama dan terakhir.
    """
    if not raw_text:
        return ""
    
    start_idx = raw_text.find('{')
    end_idx = raw_text.rfind('}')
    
    if start_idx != -1 and end_idx != -1 and start_idx < end_idx:
        return raw_text[start_idx:end_idx+1]
    
    return raw_text
```

#### Delete #2: `sanitize_llm_field` function
```python
# DELETE THESE LINES:
def sanitize_llm_field(value) -> str:
    """
    LAPIS 3 (The Pydantic Armor):
    Mengoersi berbagai macam respons halusinasi LLM (None, [], int, frasa palsu)
    menjadi string murni yang aman bagi Neo4j.
    """
    if value is None:
        return ""
        
    # Coerce ke string jika berupa list atau number
    if isinstance(value, list):
        value = " ".join(str(v) for v in value)
    elif not isinstance(value, str):
        value = str(value)
        
    value = value.strip()
    
    # Filter respons penolakan LLM (Information not available)
    reject_phrases = [
        "information not available",
        "informasi tidak tersedia",
        "not specified",
        "tidak disebutkan",
        "tidak ada",
        "none"
    ]
    
    value_lower = value.lower()
    for phrase in reject_phrases:
        if phrase in value_lower and len(value) < 100:
            return ""
            
    return value
```

#### Delete #3: `LiteratureMetadata` class
```python
# DELETE THESE LINES:
class LiteratureMetadata(BaseModel):
    title: str = Field(description="Judul lengkap artikel ilmiah")
    keywords: List[str] = Field(description="Daftar 3-5 kata kunci utama penelitian")
```

#### Delete #4: `LiteraturePillars` class
```python
# DELETE THESE LINES:
class LiteraturePillars(BaseModel):
    background: Annotated[str, BeforeValidator(sanitize_llm_field)] = Field(default="", description="Latar Belakang dari jurnal")
    methodology: Annotated[str, BeforeValidator(sanitize_llm_field)] = Field(default="", description="Metodologi yang digunakan di dalam jurnal")
    gap: Annotated[str, BeforeValidator(sanitize_llm_field)] = Field(default="", description="Gap penelitan atau masalah yang belum terselesaikan")
    objective: Annotated[str, BeforeValidator(sanitize_llm_field)] = Field(default="", description="Tujuan dari penelitian")
    futurework: Annotated[str, BeforeValidator(sanitize_llm_field)] = Field(default="", description="Pekerjaan di masa depan yang diusulkan oleh penulis")
```

#### Delete #5: `ExtractionResponse` class
```python
# DELETE THESE LINES:
class ExtractionResponse(BaseModel):
    metadata: LiteratureMetadata
    pillars: LiteraturePillars
```

**Total Lines to Delete:** ~80 lines

---

### Step 5: Verify No Broken References

**Action:** Pastikan tidak ada reference yang broken setelah delete:

```bash
# Check if any undefined references remain
python -m py_compile app/services/ingestion.py
```

**Expected:** No syntax errors

---

### Step 6: Run Tests

**Action:** Jalankan test suite untuk memastikan tidak ada regression:

```bash
# Run all tests
pytest tests/ -v

# Or specific test for ingestion
pytest tests/test_ingestion.py -v
```

**Expected:** All tests pass ✅

---

### Step 7: Verify Functionality

**Action:** Manual verification bahwa fungsi masih bekerja:

```python
# Test 1: Import works
from app.services.ingestion import clean_llm_json_string
print("✅ Import successful")

# Test 2: Function works
result = clean_llm_json_string('{"key": "value"}')
assert result == '{"key": "value"}'
print("✅ Function works")

# Test 3: Pydantic models work
from app.services.ingestion import ExtractionResponse
response = ExtractionResponse(
    metadata={"title": "Test", "keywords": ["a", "b"]},
    pillars={"background": "test", "methodology": "test", "gap": "test", "objective": "test", "futurework": "test"}
)
print("✅ Pydantic models work")
```

---

## 🎯 Phase 3: Validation & Verification

### Checklist Pre-Implementation

- [ ] **Verify `llm_extractor.py` is complete**
  - [ ] Has `clean_llm_json_string()`
  - [ ] Has `sanitize_llm_field()`
  - [ ] Has `LiteratureMetadata`
  - [ ] Has `LiteraturePillars`
  - [ ] Has `ExtractionResponse`

- [ ] **Find all usage points**
  - [ ] Grep for `clean_llm_json_string` calls
  - [ ] Grep for `sanitize_llm_field` calls
  - [ ] Grep for class references

- [ ] **Backup current state**
  - [ ] Git commit current state
  - [ ] Create backup branch

### Checklist During Implementation

- [ ] **Add import statement**
  - [ ] Import added at top of file
  - [ ] All 5 items imported

- [ ] **Delete duplicate code**
  - [ ] Delete `clean_llm_json_string()` function
  - [ ] Delete `sanitize_llm_field()` function
  - [ ] Delete `LiteratureMetadata` class
  - [ ] Delete `LiteraturePillars` class
  - [ ] Delete `ExtractionResponse` class

- [ ] **Verify syntax**
  - [ ] No syntax errors
  - [ ] File compiles

### Checklist Post-Implementation

- [ ] **Run tests**
  - [ ] All unit tests pass
  - [ ] All integration tests pass
  - [ ] No regression

- [ ] **Manual verification**
  - [ ] Import works
  - [ ] Functions work
  - [ ] Pydantic models work

- [ ] **Code review**
  - [ ] Changes reviewed
  - [ ] No issues found

- [ ] **Commit & push**
  - [ ] Changes committed
  - [ ] Pushed to branch

---

## 📊 Impact Analysis

### Before (BROKEN)
```
ingestion.py (54.5 KB)
├── clean_llm_json_string() [DUPLICATE]
├── sanitize_llm_field() [DUPLICATE]
├── LiteratureMetadata [DUPLICATE]
├── LiteraturePillars [DUPLICATE]
├── ExtractionResponse [DUPLICATE]
└── ... other code ...

llm_extractor.py (2.9 KB)
├── clean_llm_json_string() [DUPLICATE]
├── sanitize_llm_field() [DUPLICATE]
├── LiteratureMetadata [DUPLICATE]
├── LiteraturePillars [DUPLICATE]
└── ExtractionResponse [DUPLICATE]

Total: 57.4 KB
Duplication: 100% (5 items)
```

### After (FIXED)
```
ingestion.py (48 KB)
├── from llm_extractor import (
│   clean_llm_json_string,
│   sanitize_llm_field,
│   LiteratureMetadata,
│   LiteraturePillars,
│   ExtractionResponse
│ )
└── ... other code ...

llm_extractor.py (2.9 KB)
├── clean_llm_json_string() [AUTHORITATIVE]
├── sanitize_llm_field() [AUTHORITATIVE]
├── LiteratureMetadata [AUTHORITATIVE]
├── LiteraturePillars [AUTHORITATIVE]
└── ExtractionResponse [AUTHORITATIVE]

Total: 50.9 KB
Duplication: 0% (Single source of truth)
```

**Improvements:**
- ✅ File size reduced: 57.4 KB → 50.9 KB (-11%)
- ✅ Duplication eliminated: 100% → 0%
- ✅ Single source of truth established
- ✅ Maintenance burden reduced

---

## ⏱️ Time Estimate

| Task | Time | Notes |
|------|------|-------|
| Verify `llm_extractor.py` | 5 min | Grep & check |
| Find usage points | 5 min | Grep & document |
| Add import statement | 2 min | Copy-paste |
| Delete duplicate code | 5 min | Delete 5 blocks |
| Verify syntax | 3 min | `py_compile` |
| Run tests | 10 min | Full test suite |
| Manual verification | 5 min | Quick smoke test |
| Code review | 5 min | Self-review |
| Commit & push | 3 min | Git operations |
| **TOTAL** | **43 min** | |

---

## 🚀 Execution Order

### Phase 1: Preparation (5 min)
1. Verify `llm_extractor.py` is complete
2. Find all usage points
3. Create backup branch

### Phase 2: Implementation (15 min)
1. Add import statement
2. Delete duplicate code
3. Verify syntax

### Phase 3: Validation (20 min)
1. Run tests
2. Manual verification
3. Code review

### Phase 4: Finalization (3 min)
1. Commit changes
2. Push to branch

---

## 🎯 Success Criteria

✅ **All of these must be true:**

1. **No duplicate code**
   - `clean_llm_json_string` hanya ada di `llm_extractor.py`
   - `sanitize_llm_field` hanya ada di `llm_extractor.py`
   - Pydantic models hanya ada di `llm_extractor.py`

2. **Proper imports**
   - `ingestion.py` import dari `llm_extractor.py`
   - Semua 5 items ter-import

3. **No broken references**
   - File compiles without errors
   - No undefined references

4. **Tests pass**
   - All unit tests pass
   - All integration tests pass
   - No regression

5. **Functionality preserved**
   - Functions work as before
   - Pydantic models work as before
   - No behavior changes

---

## 📝 Summary

**Solving Strategy untuk BUG #1:**

1. ✅ **Verify** `llm_extractor.py` punya semua yang dibutuhkan
2. ✅ **Find** semua usage points di `ingestion.py`
3. ✅ **Add** import statement ke `ingestion.py`
4. ✅ **Delete** 5 duplicate definitions dari `ingestion.py`
5. ✅ **Verify** syntax dan tidak ada broken references
6. ✅ **Test** untuk memastikan tidak ada regression
7. ✅ **Commit** dan push changes

**Outcome:**
- Single source of truth
- No duplication
- Cleaner code
- Easier maintenance

---

**Status:** Ready for discussion & approval  
**Next Step:** Diskusi dengan user tentang strategi ini sebelum implementasi
