Skip to content

rabbithole

Hooks to modify the RabbitHole's documents ingestion.

Here is a collection of methods to hook into the RabbitHole execution pipeline.

These hooks allow to intercept the uploaded documents at different places before they are saved into memory.

after_rabbithole_splitted_text(chunks, cat)

Hook the Document after is split.

Allows editing the list of Document right after the RabbitHole chunked them in smaller ones.

Parameters:

Name Type Description Default
chunks List[Document]

List of Langchain Document.

 required
cat CheshireCat

Cheshire Cat instance.

 required

Returns:

Name Type Description
chunks List[Document]

List of modified chunked langchain documents to be stored in the episodic memory.
Source code in cat/mad_hatter/core_plugin/hooks/rabbithole.py 
125
126
127
128
129
130
131
132
133
134
135
136
137
138
139
140
141
142
143
144
145
@hook(priority=0)
def after_rabbithole_splitted_text(chunks: List[Document], cat) -> List[Document]:
    """Hook the `Document` after is split.

    Allows editing the list of `Document` right after the *RabbitHole* chunked them in smaller ones.

    Parameters
    ----------
    chunks : List[Document]
        List of Langchain `Document`.
    cat : CheshireCat
        Cheshire Cat instance.

    Returns
    -------
    chunks : List[Document]
        List of modified chunked langchain documents to be stored in the episodic memory.

    """

    return chunks

after_rabbithole_stored_documents(source, stored_points, cat)

Hook the Document after is inserted in the vector memory.

Allows editing and enhancing the list of Document after is inserted in the vector memory.

Parameters:

Name Type Description Default
source

Name of ingested file/url

 required
docs List[PointStruct]

List of Qdrant PointStruct just inserted into the db.

 required
cat CheshireCat

Cheshire Cat instance.

 required

Returns:

Type Description
None
Source code in cat/mad_hatter/core_plugin/hooks/rabbithole.py 
179
180
181
182
183
184
185
186
187
188
189
190
191
192
193
194
195
196
197
198
199
200
@hook(priority=0)
def after_rabbithole_stored_documents(
    source, stored_points: List[PointStruct], cat
) -> None:
    """Hook the Document after is inserted in the vector memory.

    Allows editing and enhancing the list of Document after is inserted in the vector memory.

    Parameters
    ----------
    source: str
        Name of ingested file/url
    docs : List[PointStruct]
        List of Qdrant PointStruct just inserted into the db.
    cat : CheshireCat
        Cheshire Cat instance.

    Returns
    -------
    None
    """
    pass

before_rabbithole_insert_memory(doc, cat)

Hook the Document before is inserted in the vector memory.

Allows editing and enhancing a single Document before the RabbitHole add it to the declarative vector memory.

Parameters:

Name Type Description Default
doc Document

Langchain Document to be inserted in memory.

 required
cat CheshireCat

Cheshire Cat instance.

 required

Returns:

Name Type Description
doc Document

Langchain Document that is added in the declarative vector memory.
Notes

The Document has two properties::

`page_content`: the string with the text to save in memory;
`metadata`: a dictionary with at least two keys:
    `source`: where the text comes from;
    `when`: timestamp to track when it's been uploaded.
Source code in cat/mad_hatter/core_plugin/hooks/rabbithole.py 
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
91
92
93
@hook(priority=0)
def before_rabbithole_insert_memory(doc: Document, cat) -> Document:
    """Hook the `Document` before is inserted in the vector memory.

    Allows editing and enhancing a single `Document` before the *RabbitHole* add it to the declarative vector memory.

    Parameters
    ----------
    doc : Document
        Langchain `Document` to be inserted in memory.
    cat : CheshireCat
        Cheshire Cat instance.

    Returns
    -------
    doc : Document
        Langchain `Document` that is added in the declarative vector memory.

    Notes
    -----
    The `Document` has two properties::

        `page_content`: the string with the text to save in memory;
        `metadata`: a dictionary with at least two keys:
            `source`: where the text comes from;
            `when`: timestamp to track when it's been uploaded.

    """
    return doc

before_rabbithole_splits_text(docs, cat)

Hook the Documents before they are split into chunks.

Allows editing the uploaded document main Document(s) before the RabbitHole recursively splits it in shorter ones. Please note that this is a list because parsers can output one or more Document, that are afterwards splitted.

For instance, the hook allows to change the text or edit/add metadata.

Parameters:

Name Type Description Default
docs List[Document]

Langchain Documents resulted after parsing the file uploaded in the RabbitHole.

 required
cat CheshireCat

Cheshire Cat instance.

 required

Returns:

Name Type Description
docs List[Document]

Edited Langchain Documents.
Source code in cat/mad_hatter/core_plugin/hooks/rabbithole.py 
 97
 98
 99
100
101
102
103
104
105
106
107
108
109
110
111
112
113
114
115
116
117
118
119
120
@hook(priority=0)
def before_rabbithole_splits_text(docs: List[Document], cat) -> List[Document]:
    """Hook the `Documents` before they are split into chunks.

    Allows editing the uploaded document main Document(s) before the *RabbitHole* recursively splits it in shorter ones.
    Please note that this is a list because parsers can output one or more Document, that are afterwards splitted.

    For instance, the hook allows to change the text or edit/add metadata.

    Parameters
    ----------
    docs : List[Document]
        Langchain `Document`s resulted after parsing the file uploaded in the *RabbitHole*.
    cat : CheshireCat
        Cheshire Cat instance.

    Returns
    -------
    docs : List[Document]
        Edited Langchain `Document`s.

    """

    return docs

before_rabbithole_stores_documents(docs, cat)

Hook into the memory insertion pipeline.

Allows modifying how the list of Document is inserted in the vector memory.

For example, this hook is a good point to summarize the incoming documents and save both original and summarized contents. An official plugin is available to test this procedure.

Parameters:

Name Type Description Default
docs List[Document]

List of Langchain Document to be edited.

 required
cat

Cheshire Cat instance.

 required

Returns:

Name Type Description
docs List[Document]

List of edited Langchain documents.
Source code in cat/mad_hatter/core_plugin/hooks/rabbithole.py 
152
153
154
155
156
157
158
159
160
161
162
163
164
165
166
167
168
169
170
171
172
173
174
175
176
@hook(priority=0)
def before_rabbithole_stores_documents(docs: List[Document], cat) -> List[Document]:
    """Hook into the memory insertion pipeline.

    Allows modifying how the list of `Document` is inserted in the vector memory.

    For example, this hook is a good point to summarize the incoming documents and save both original and
    summarized contents.
    An official plugin is available to test this procedure.

    Parameters
    ----------
    docs : List[Document]
        List of Langchain `Document` to be edited.
    cat: CheshireCat
        Cheshire Cat instance.

    Returns
    -------
    docs : List[Document]
        List of edited Langchain documents.

    """

    return docs

rabbithole_instantiates_parsers(file_handlers, cat)

Hook the available parsers for ingesting files in the declarative memory.

Allows replacing or extending existing supported mime types and related parsers to customize the file ingestion.

Parameters:

Name Type Description Default
file_handlers dict

Keys are the supported mime types and values are the related parsers.

 required
cat CheshireCat

Cheshire Cat instance.

 required

Returns:

Name Type Description
file_handlers dict

Edited dictionary of supported mime types and related parsers.
Source code in cat/mad_hatter/core_plugin/hooks/rabbithole.py 
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
@hook(priority=0)
def rabbithole_instantiates_parsers(file_handlers: dict, cat) -> dict:
    """Hook the available parsers for ingesting files in the declarative memory.

    Allows replacing or extending existing supported mime types and related parsers to customize the file ingestion.

    Parameters
    ----------
    file_handlers : dict
        Keys are the supported mime types and values are the related parsers.
    cat : CheshireCat
        Cheshire Cat instance.

    Returns
    -------
    file_handlers : dict
        Edited dictionary of supported mime types and related parsers.
    """
    return file_handlers

rabbithole_instantiates_splitter(text_splitter, cat)

Hook the splitter used to split text in chunks.

Allows replacing the default text splitter to customize the splitting process.

Parameters:

Name Type Description Default
text_splitter TextSplitter

The text splitter used by default.

 required
cat CheshireCat

Cheshire Cat instance.

 required

Returns:

Name Type Description
text_splitter TextSplitter

An instance of a TextSplitter subclass.
Source code in cat/mad_hatter/core_plugin/hooks/rabbithole.py 
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
@hook(priority=0)
def rabbithole_instantiates_splitter(text_splitter: TextSplitter, cat) -> TextSplitter:
    """Hook the splitter used to split text in chunks.

    Allows replacing the default text splitter to customize the splitting process.

    Parameters
    ----------
    text_splitter : TextSplitter
        The text splitter used by default.
    cat : CheshireCat
        Cheshire Cat instance.

    Returns
    -------
    text_splitter : TextSplitter
        An instance of a TextSplitter subclass.
    """

    # example on how to change chunking
    # text_splitter._chunk_size = 64
    # text_splitter._chunk_overlap = 8

    return text_splitter