> ## Documentation Index
> Fetch the complete documentation index at: https://developer.kodexa.ai/llms.txt
> Use this file to discover all available pages before exploring further.

# Using Data Extraction Starter

> Use the Data Extraction Starter project template to quickly set up document data extraction in Kodexa, from upload to structured output.

## Introduction

The Data Extraction Starter is a pre-built project template that you can use to get started with Kodexa. It is a great way to get started with Kodexa and can be used to extract data from a wide range of documents.

## What's the High-Level Approach?

The aim of the template is to provide a starting point for data extraction.

It provides a pre-configured project with the following features:

* A Processing Document Store for holding documents
* An Extracted Document Store for holding extracted data in a structured form
* An empty Data Definition that you can use to define the data you want to capture from documents
* An empty Data Form you can use to build a form based on your data definition
* A default task configuration that can be used with a simple task status lifecycle
* A default document status lifecycle
* A dataflow configured to use OCR, LLM-based Data Extraction, Validation and Task Creation

## What is the basic flow?

A newly uploaded document will be processed by the following steps:

1. OCR is used to extract text from the document
2. The text is then passed to an LLM-based data extraction model
3. The extracted data is then validated
4. If the data is invalid, a task is created to review the data
   * The task is then reviewed by a human
   * If the task is approved, the changed data is saved to the Extracted Document Store
   * If the task is rejected, the changed data is saved to the Extracted Document Store
5. The document is then marked as processed

```mermaid theme={null}
sequenceDiagram
    participant Doc as Document
    participant RA as Robotic Assistant
    participant Prep as Prepare Assistant
    participant AI as Kodexa AI
    participant Trans as Transformer
    participant Tasks as Tasks Assistant
    
    Note over RA: Runs every 10 minutes<br/>Cron: */10 * * * *
    
    Doc->>Prep: New PDF Document
    
    alt Processing Success
        Prep->>Doc: Status: Prepared
        Doc->>AI: Trigger: hasMixins('spatial')
        AI->>Doc: Status: Labeled
        Doc->>Trans: Trigger: hasLabel('labeled')
        Trans->>Doc: Status: Processed
        Doc->>Tasks: Trigger: hasLabel('processed')
        Tasks->>Doc: Status: Pending Review
    else Processing Failure
        Prep-->>Doc: Status: Failed
        RA->>Doc: Check for retries
        alt Retry < 3
            RA->>Prep: Retry processing
            Prep->>Doc: Retry attempt
        else Max retries reached
            RA->>Doc: Final Status: Failed
        end
    end
    
    alt Timeout Detection
        RA->>Doc: Check processing time
        Note over RA: If processing > 24 hours
        RA->>Doc: Status: Failed (Timeout)
    end
    
    alt Task Management
        Tasks->>Doc: Create review task
        Note over Tasks: Task status management
        Tasks->>Doc: Update based on task outcome
    end
```

## Document Statuses

Each document can exist in one of the following statuses:

| **Status** | **Color Badge** | **Slug** | **Stage** |
| - | - | - | - |
| **Failed** | | `failed` | Error |
| **Pending Review** | | `pending-review` | Review |
| **Reviewed** | | `reviewed` | Complete |
| **Labeled** | | `labeled` | Processing |
| **Rejected** | | `rejected` | Error |
| **Transformed** | | `transformed` | Processing |
| **Completed** | | `completed` | Complete |
| **Prepared** | | `prepared` | Processing |

## Status Flow in Processing Pipeline

```mermaid theme={null}
flowchart LR
   A[📄 New Document] --> B[🟠 Prepared]
   B --> C[🔵 Labeled]
   C --> D[🟣 Transformed]
   D --> E{Task Decision}
   E -->|Task Created| F[🟡 Pending Review]
   E -->|No Task| G[🟢 Completed]
   F --> H[🟢 Reviewed]
   F --> I[🔴 Rejected]
   
   B -.->|Error| J[🔴 Failed]
   C -.->|Error| J
   D -.->|Error| J

   style B fill:#fd7e14,color:#fff
   style C fill:#17a2b8,color:#fff
   style D fill:#6f42c1,color:#fff
   style F fill:#ffc107,color:#000
   style G fill:#20c997,color:#fff
   style H fill:#28a745,color:#fff
   style I fill:#e74c3c,color:#fff
    style J fill:#dc3545,color:#fff
```

## Default Data Flow

The default data flow is pre-configured with everything needed to get started.

![Default Data Flow](https://mintcdn.com/kodexa/fcSOqm-7T1yIp0A8/articles/data-flow.png?fit=max\&auto=format\&n=fcSOqm-7T1yIp0A8\&q=85\&s=ce2020e0e66ef2c733941d5bce6e89ef)

However, often you will want to add to the data flow. A typical example is being able to publish to external
systems based on the completion (or failure) of a document.

In order to do this we will walk through how to create two new models, and show where to add them
to the data flow.

## Adding a custom model to publish to an external system

In this example we will add a custom model to publish to an external system.

### Publishing Success

In order to publish success we can create a very simple model and add it to the Task Assistant.

You can simply create a new model and add it to the Task Assistant. See the [Python Modules](https://developer.kodexa.ai/guides/modules/python-module) guide for how to create an inference module.

Once you have created your model you can then add it to the Task Assistant at the end.

In the code you can then get the information from the document and publish it to an external system.

```python theme={null}
def infer(document: Document, project: ProjectEndpoint, pipeline_context: PipelineContext, assistant: Assistant):
    
    # Add the logic to allow you to publish a successful document
    
    return document
```

### Publishing Failure

For publishing the failure we would usually use a scheduled event model, the reason for this is that we often want to
defer publishing failures in case we are having a problem in processing.

One thing we want to be aware of is that we need to know, in Kodexa, if we have published the failure to the external system.

This can be done by adding a label to the document.

This means we would create an event handler module that runs every 10 minutes. See the [Event Handling](https://developer.kodexa.ai/concepts/event_handling_with_modules.md) guide and the [Python Modules](https://developer.kodexa.ai/guides/modules/python-module#writing-an-event-handler-module) guide for details.

Once you have created your model you can then add it to the Schedule Assistant at the end.

In the code you can then get the information from the document and publish it to an external system.

```python theme={null}
def handle_event(event: BaseEvent):
    logger.info(f"Handle event called with event: {event.type}")
    if event.type == "scheduled":
        client = KodexaClient()

        document_store: DocumentStoreEndpoint = client.get_object_by_ref("store","my-org/store-slug")

        failed_document_families = document_store.stream_filter("documentStatus.slug:'failed' and not exists(labels.label('published-failure'))")

        for failed_document_family in failed_document_families:
            
            # Add the logic to publish the failure to the external system
            logger.info(f"Publishing failure for document family: {failed_document_family.path}")


            # Then add a label to the document so we don't pick it up again
            failed_document_family.add_label("published-failure")


```


This documentation is built and hosted on [Mintlify](https://mintlify.com), a developer documentation platform.