Segmentation

Segmentation

The Segmentation pipeline segments text into semantic units.

Example

The following shows a simple example using this pipeline.

from txtai.pipeline import Segmentation

## Create and run pipeline
segment = Segmentation(sentences=True)
segment("This is a test. And another test.")

## Load third-party chunkers
segment = Segmentation(chunker="semantic")
segment("This is a test. And another test.")

The Segmentation pipeline supports segmenting sentences, lines, paragraphs and sections using a rules-based approach. Each of these modes can be set when creating the pipeline. Third-party chunkers are also supported via the chunker parameter.

Configuration-driven example

Pipelines are run with Python or configuration. Pipelines can be instantiated in configuration using the lower case name of the pipeline. Configuration-driven pipelines are run with workflows or the API.

config.yml

## Create pipeline using lower case class name
segmentation:
  sentences: true

## Run pipeline with workflow
workflow:
  segment:
    tasks:
      - action: segmentation

Run with Workflows

from txtai import Application

## Create and run pipeline with workflow
app = Application("config.yml")
list(app.workflow("segment", ["This is a test. And another test."]))

Run with API

CONFIG=config.yml uvicorn "txtai.api:app" &

curl \
  -X POST "http://localhost:8000/workflow" \
  -H "Content-Type: application/json" \
  -d '{"name":"segment", "elements":["This is a test. And another test."]}'

Methods

Python documentation for the pipeline.

__init__(sentences=False, lines=False, paragraphs=False, minlength=None, join=False, sections=False, cleantext=True, chunker=None, tuples=False, **kwargs)

Creates a new Segmentation pipeline.

Parameters:

Source code in txtai/pipeline/data/segmentation.py

31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
``` |  ```
def init(
    self,
    sentences=False,
    lines=False,
    paragraphs=False,
    minlength=None,
    join=False,
    sections=False,
    cleantext=True,
    chunker=None,
    tuples=False,
    kwargs,
):
    “””
    Creates a new Segmentation pipeline.

    Args:
        sentences: tokenize text into sentences if True, defaults to False
        lines: tokenizes text into lines if True, defaults to False
        paragraphs: tokenizes text into paragraphs if True, defaults to False
        minlength: require at least minlength characters per text element, defaults to None
        join: joins tokenized sections back together if True, defaults to False
        sections: tokenizes text into sections if True, defaults to False. Splits using section or page breaks, depending on what’s available
        cleantext: apply text cleaning rules, defaults to True
        chunker: creates a third-party chunker to tokenize text if set, defaults to None
        tuples: return (input, output) tuples, defaults to False
        kwargs: additional keyword arguments
    “””

    if not NLTK and sentences:
        raise ImportError(‘NLTK is not available - install “pipeline” extra to enable’)

    if not CHONKIE and chunker:
        raise ImportError(‘Chonkie is not available - install “pipeline” extra to enable’)

    self.sentences = sentences
    self.lines = lines
    self.paragraphs = paragraphs
    self.sections = sections
    self.minlength = minlength
    self.join = join
    self.cleantext = cleantext

    # Create a third-party chunker, if applicable
    self.chunker = self.createchunker(chunker, kwargs) if chunker else None

    # Return (input, output) tuples as output
    self.tuples = tuples

---|---

__call__(text)

Segments text into semantic units.

This method supports text as a string or a list. If the input is a string, the return type is text|list. If text is a list, a list of returned, this could be a list of text or a list of lists depending on the tokenization strategy.

Parameters:

Returns:

Source code in txtai/pipeline/data/segmentation.py

 80
 81
 82
 83
 84
 85
 86
 87
 88
 89
 90
 91
 92
 93
 94
 95
 96
 97
 98
 99
100
101
102
103
104
105
106
107
108
109
110
111
112
113
``` |  ```
def call(self, text):
    “””
    Segments text into semantic units.

    This method supports text as a string or a list. If the input is a string, the return
    type is text|list. If text is a list, a list of returned, this could be a
    list of text or a list of lists depending on the tokenization strategy.

    Args:
        text: text|list

    Returns:
        segmented text
    “””

    # Get inputs
    texts = [text] if not isinstance(text, list) else text

    # Extract text for each input file
    results = []
    for value in texts:
        # Get text
        result = self.text(value)

        # Parse and add extracted results
        result = self.parse(result)

        # Wrap as tuple
        if self.tuples:
            result = [(value, x) for x in result] if isinstance(result, list) else (value, result)

        results.append(result)

    return results[0] if isinstance(text, str) else results

---|---