langchain.document_loaders.image.UnstructuredImageLoader¶
class langchain.document_loaders.image.UnstructuredImageLoader(file_path: Union[str, List[str]], mode: str = 'single', **unstructured_kwargs: Any)[source]¶
Load PNG and JPG files using Unstructured.
You can run the loader in one of two modes: “single” and “elements”.
If you use “single” mode, the document will be returned as a single
langchain Document object. If you use “elements” mode, the unstructured
library will split the document into elements such as Title and NarrativeText.
You can pass in additional unstructured kwargs after mode to apply
different unstructured settings.
Examples
from langchain.document_loaders import UnstructuredImageLoader
loader = UnstructuredImageLoader(“example.png”, mode=”elements”, strategy=”fast”,
)
docs = loader.load()
References
https://unstructured-io.github.io/unstructured/bricks.html#partition-image
Initialize with file path.
Methods
__init__(file_path[, mode])
Initialize with file path.
lazy_load()
A lazy loader for Documents.
load()
Load file.
load_and_split([text_splitter])
Load Documents and split into chunks.
__init__(file_path: Union[str, List[str]], mode: str = 'single', **unstructured_kwargs: Any)¶
Initialize with file path.
lazy_load() → Iterator[Document]¶
A lazy loader for Documents.
load() → List[Document]¶
Load file.
load_and_split(text_splitter: Optional[TextSplitter] = None) → List[Document]¶
Load Documents and split into chunks. Chunks are returned as Documents.
Parameters
text_splitter – TextSplitter instance to use for splitting documents.
Defaults to RecursiveCharacterTextSplitter.
Returns


Defaults to RecursiveCharacterTextSplitter.
Returns
List of Documents.
Examples using UnstructuredImageLoader¶
Images


langchain.document_loaders.pdf.DocumentIntelligenceLoader¶
class langchain.document_loaders.pdf.DocumentIntelligenceLoader(file_path: str, client: Any, model: str = 'prebuilt-document', headers: Optional[Dict] = None)[source]¶
Loads a PDF with Azure Document Intelligence
Initialize the object for file processing with Azure Document Intelligence
(formerly Form Recognizer).
This constructor initializes a DocumentIntelligenceParser object to be used
for parsing files using the Azure Document Intelligence API. The load method
generates a Document node including metadata (source blob and page number)
for each page.
file_pathstrThe path to the file that needs to be parsed.
client: AnyA DocumentAnalysisClient to perform the analysis of the blob
modelstrThe model name or ID to be used for form recognition in Azure.
>>> obj = DocumentIntelligenceLoader(
...     file_path="path/to/file",
...     client=client,
...     model="prebuilt-document"
... )
Attributes
source
Methods
__init__(file_path, client[, model, headers])
Initialize the object for file processing with Azure Document Intelligence (formerly Form Recognizer).
lazy_load()
Lazy load given path as pages.
load()
Load given path as pages.
load_and_split([text_splitter])
Load Documents and split into chunks.
__init__(file_path: str, client: Any, model: str = 'prebuilt-document', headers: Optional[Dict] = None) → None[source]¶
Initialize the object for file processing with Azure Document Intelligence
(formerly Form Recognizer).
This constructor initializes a DocumentIntelligenceParser object to be used
for parsing files using the Azure Document Intelligence API. The load method
generates a Document node including metadata (source blob and page number)
for each page.


generates a Document node including metadata (source blob and page number)
for each page.
file_pathstrThe path to the file that needs to be parsed.
client: AnyA DocumentAnalysisClient to perform the analysis of the blob
modelstrThe model name or ID to be used for form recognition in Azure.
>>> obj = DocumentIntelligenceLoader(
...     file_path="path/to/file",
...     client=client,
...     model="prebuilt-document"
... )
lazy_load() → Iterator[Document][source]¶
Lazy load given path as pages.
load() → List[Document][source]¶
Load given path as pages.
load_and_split(text_splitter: Optional[TextSplitter] = None) → List[Document]¶
Load Documents and split into chunks. Chunks are returned as Documents.
Parameters
text_splitter – TextSplitter instance to use for splitting documents.
Defaults to RecursiveCharacterTextSplitter.
Returns
List of Documents.
Examples using DocumentIntelligenceLoader¶
Azure Document Intelligence


langchain.document_loaders.azure_blob_storage_file.AzureBlobStorageFileLoader¶
class langchain.document_loaders.azure_blob_storage_file.AzureBlobStorageFileLoader(conn_str: str, container: str, blob_name: str)[source]¶
Load from Azure Blob Storage files.
Initialize with connection string, container and blob name.
Attributes
conn_str
Connection string for Azure Blob Storage.
container
Container name.
blob
Blob name.
Methods
__init__(conn_str, container, blob_name)
Initialize with connection string, container and blob name.
lazy_load()
A lazy loader for Documents.
load()
Load documents.
load_and_split([text_splitter])
Load Documents and split into chunks.
__init__(conn_str: str, container: str, blob_name: str)[source]¶
Initialize with connection string, container and blob name.
lazy_load() → Iterator[Document]¶
A lazy loader for Documents.
load() → List[Document][source]¶
Load documents.
load_and_split(text_splitter: Optional[TextSplitter] = None) → List[Document]¶
Load Documents and split into chunks. Chunks are returned as Documents.
Parameters
text_splitter – TextSplitter instance to use for splitting documents.
Defaults to RecursiveCharacterTextSplitter.
Returns
List of Documents.
Examples using AzureBlobStorageFileLoader¶
Azure Blob Storage
Azure Blob Storage File


langchain.document_loaders.merge.MergedDataLoader¶
class langchain.document_loaders.merge.MergedDataLoader(loaders: List)[source]¶
Merge documents from a list of loaders
Initialize with a list of loaders
Methods
__init__(loaders)
Initialize with a list of loaders
lazy_load()
Lazy load docs from each individual loader.
load()
Load docs.
load_and_split([text_splitter])
Load Documents and split into chunks.
__init__(loaders: List)[source]¶
Initialize with a list of loaders
lazy_load() → Iterator[Document][source]¶
Lazy load docs from each individual loader.
load() → List[Document][source]¶
Load docs.
load_and_split(text_splitter: Optional[TextSplitter] = None) → List[Document]¶
Load Documents and split into chunks. Chunks are returned as Documents.
Parameters
text_splitter – TextSplitter instance to use for splitting documents.
Defaults to RecursiveCharacterTextSplitter.
Returns
List of Documents.
Examples using MergedDataLoader¶
MergeDocLoader


langchain.document_loaders.pdf.PyPDFium2Loader¶
class langchain.document_loaders.pdf.PyPDFium2Loader(file_path: str, *, headers: Optional[Dict] = None, extract_images: bool = False)[source]¶
Load PDF using pypdfium2 and chunks at character level.
Initialize with a file path.
Attributes
source
Methods
__init__(file_path, *[, headers, extract_images])
Initialize with a file path.
lazy_load()
Lazy load given path as pages.
load()
Load given path as pages.
load_and_split([text_splitter])
Load Documents and split into chunks.
__init__(file_path: str, *, headers: Optional[Dict] = None, extract_images: bool = False)[source]¶
Initialize with a file path.
lazy_load() → Iterator[Document][source]¶
Lazy load given path as pages.
load() → List[Document][source]¶
Load given path as pages.
load_and_split(text_splitter: Optional[TextSplitter] = None) → List[Document]¶
Load Documents and split into chunks. Chunks are returned as Documents.
Parameters
text_splitter – TextSplitter instance to use for splitting documents.
Defaults to RecursiveCharacterTextSplitter.
Returns
List of Documents.


langchain.document_loaders.pdf.PDFMinerLoader¶
class langchain.document_loaders.pdf.PDFMinerLoader(file_path: str, *, headers: Optional[Dict] = None, extract_images: bool = False, concatenate_pages: bool = True)[source]¶
Load PDF files using PDFMiner.
Initialize with file path.
Parameters
extract_images – Whether to extract images from PDF.
concatenate_pages – If True, concatenate all PDF pages into one a single
document. Otherwise, return one document per page.
Attributes
source
Methods
__init__(file_path, *[, headers, ...])
Initialize with file path.
lazy_load()
Lazily load documents.
load()
Eagerly load the content.
load_and_split([text_splitter])
Load Documents and split into chunks.
__init__(file_path: str, *, headers: Optional[Dict] = None, extract_images: bool = False, concatenate_pages: bool = True) → None[source]¶
Initialize with file path.
Parameters
extract_images – Whether to extract images from PDF.
concatenate_pages – If True, concatenate all PDF pages into one a single
document. Otherwise, return one document per page.
lazy_load() → Iterator[Document][source]¶
Lazily load documents.
load() → List[Document][source]¶
Eagerly load the content.
load_and_split(text_splitter: Optional[TextSplitter] = None) → List[Document]¶
Load Documents and split into chunks. Chunks are returned as Documents.
Parameters
text_splitter – TextSplitter instance to use for splitting documents.
Defaults to RecursiveCharacterTextSplitter.
Returns
List of Documents.


langchain.document_loaders.discord.DiscordChatLoader¶
class langchain.document_loaders.discord.DiscordChatLoader(chat_log: pd.DataFrame, user_id_col: str = 'ID')[source]¶
Load Discord chat logs.
Initialize with a Pandas DataFrame containing chat logs.
Parameters
chat_log – Pandas DataFrame containing chat logs.
user_id_col – Name of the column containing the user ID. Defaults to “ID”.
Methods
__init__(chat_log[, user_id_col])
Initialize with a Pandas DataFrame containing chat logs.
lazy_load()
A lazy loader for Documents.
load()
Load all chat messages.
load_and_split([text_splitter])
Load Documents and split into chunks.
__init__(chat_log: pd.DataFrame, user_id_col: str = 'ID')[source]¶
Initialize with a Pandas DataFrame containing chat logs.
Parameters
chat_log – Pandas DataFrame containing chat logs.
user_id_col – Name of the column containing the user ID. Defaults to “ID”.
lazy_load() → Iterator[Document]¶
A lazy loader for Documents.
load() → List[Document][source]¶
Load all chat messages.
load_and_split(text_splitter: Optional[TextSplitter] = None) → List[Document]¶
Load Documents and split into chunks. Chunks are returned as Documents.
Parameters
text_splitter – TextSplitter instance to use for splitting documents.
Defaults to RecursiveCharacterTextSplitter.
Returns
List of Documents.
Examples using DiscordChatLoader¶
Discord


langchain.document_loaders.helpers.FileEncoding¶
class langchain.document_loaders.helpers.FileEncoding(encoding: Optional[str], confidence: float, language: Optional[str])[source]¶
File encoding as the NamedTuple.
Create new instance of FileEncoding(encoding, confidence, language)
Attributes
confidence
The confidence of the encoding.
encoding
The encoding of the file.
language
The language of the file.
Methods
__init__()
count(value, /)
Return number of occurrences of value.
index(value[, start, stop])
Return first index of value.
__init__()¶
count(value, /)¶
Return number of occurrences of value.
index(value, start=0, stop=9223372036854775807, /)¶
Return first index of value.
Raises ValueError if the value is not present.


langchain.document_loaders.directory.DirectoryLoader¶
class langchain.document_loaders.directory.DirectoryLoader(path: str, glob: str = '**/[!.]*', silent_errors: bool = False, load_hidden: bool = False, loader_cls: ~typing.Union[~typing.Type[~langchain.document_loaders.unstructured.UnstructuredFileLoader], ~typing.Type[~langchain.document_loaders.text.TextLoader], ~typing.Type[~langchain.document_loaders.html_bs.BSHTMLLoader]] = <class 'langchain.document_loaders.unstructured.UnstructuredFileLoader'>, loader_kwargs: ~typing.Optional[dict] = None, recursive: bool = False, show_progress: bool = False, use_multithreading: bool = False, max_concurrency: int = 4, *, sample_size: int = 0, randomize_sample: bool = False, sample_seed: ~typing.Optional[int] = None)[source]¶
Load from a directory.
Initialize with a path to directory and how to glob over it.
Parameters
path – Path to directory.
glob – Glob pattern to use to find files. Defaults to “**/[!.]*”
(all files except hidden).
silent_errors – Whether to silently ignore errors. Defaults to False.
load_hidden – Whether to load hidden files. Defaults to False.
loader_cls – Loader class to use for loading files.
Defaults to UnstructuredFileLoader.
loader_kwargs – Keyword arguments to pass to loader_cls. Defaults to None.
recursive – Whether to recursively search for files. Defaults to False.
show_progress – Whether to show a progress bar. Defaults to False.
use_multithreading – Whether to use multithreading. Defaults to False.
max_concurrency – The maximum number of threads to use. Defaults to 4.
sample_size – The maximum number of files you would like to load from the
directory.


sample_size – The maximum number of files you would like to load from the
directory.
randomize_sample – Suffle the files to get a random sample.
sample_seed – set the seed of the random shuffle for reporoducibility.
Methods
__init__(path[, glob, silent_errors, ...])
Initialize with a path to directory and how to glob over it.
lazy_load()
A lazy loader for Documents.
load()
Load documents.
load_and_split([text_splitter])
Load Documents and split into chunks.
load_file(item, path, docs, pbar)
Load a file.
__init__(path: str, glob: str = '**/[!.]*', silent_errors: bool = False, load_hidden: bool = False, loader_cls: ~typing.Union[~typing.Type[~langchain.document_loaders.unstructured.UnstructuredFileLoader], ~typing.Type[~langchain.document_loaders.text.TextLoader], ~typing.Type[~langchain.document_loaders.html_bs.BSHTMLLoader]] = <class 'langchain.document_loaders.unstructured.UnstructuredFileLoader'>, loader_kwargs: ~typing.Optional[dict] = None, recursive: bool = False, show_progress: bool = False, use_multithreading: bool = False, max_concurrency: int = 4, *, sample_size: int = 0, randomize_sample: bool = False, sample_seed: ~typing.Optional[int] = None)[source]¶
Initialize with a path to directory and how to glob over it.
Parameters
path – Path to directory.
glob – Glob pattern to use to find files. Defaults to “**/[!.]*”
(all files except hidden).
silent_errors – Whether to silently ignore errors. Defaults to False.
load_hidden – Whether to load hidden files. Defaults to False.


load_hidden – Whether to load hidden files. Defaults to False.
loader_cls – Loader class to use for loading files.
Defaults to UnstructuredFileLoader.
loader_kwargs – Keyword arguments to pass to loader_cls. Defaults to None.
recursive – Whether to recursively search for files. Defaults to False.
show_progress – Whether to show a progress bar. Defaults to False.
use_multithreading – Whether to use multithreading. Defaults to False.
max_concurrency – The maximum number of threads to use. Defaults to 4.
sample_size – The maximum number of files you would like to load from the
directory.
randomize_sample – Suffle the files to get a random sample.
sample_seed – set the seed of the random shuffle for reporoducibility.
lazy_load() → Iterator[Document]¶
A lazy loader for Documents.
load() → List[Document][source]¶
Load documents.
load_and_split(text_splitter: Optional[TextSplitter] = None) → List[Document]¶
Load Documents and split into chunks. Chunks are returned as Documents.
Parameters
text_splitter – TextSplitter instance to use for splitting documents.
Defaults to RecursiveCharacterTextSplitter.
Returns
List of Documents.
load_file(item: Path, path: Path, docs: List[Document], pbar: Optional[Any]) → None[source]¶
Load a file.
Parameters
item – File path.
path – Directory path.
docs – List of documents to append to.
pbar – Progress bar. Defaults to None.
Examples using DirectoryLoader¶
StarRocks


langchain.document_loaders.unstructured.UnstructuredBaseLoader¶
class langchain.document_loaders.unstructured.UnstructuredBaseLoader(mode: str = 'single', post_processors: Optional[List[Callable]] = None, **unstructured_kwargs: Any)[source]¶
Base Loader that uses Unstructured.
Initialize with file path.
Methods
__init__([mode, post_processors])
Initialize with file path.
lazy_load()
A lazy loader for Documents.
load()
Load file.
load_and_split([text_splitter])
Load Documents and split into chunks.
__init__(mode: str = 'single', post_processors: Optional[List[Callable]] = None, **unstructured_kwargs: Any)[source]¶
Initialize with file path.
lazy_load() → Iterator[Document]¶
A lazy loader for Documents.
load() → List[Document][source]¶
Load file.
load_and_split(text_splitter: Optional[TextSplitter] = None) → List[Document]¶
Load Documents and split into chunks. Chunks are returned as Documents.
Parameters
text_splitter – TextSplitter instance to use for splitting documents.
Defaults to RecursiveCharacterTextSplitter.
Returns
List of Documents.


langchain.document_loaders.embaas.EmbaasBlobLoader¶
class langchain.document_loaders.embaas.EmbaasBlobLoader[source]¶
Bases: BaseEmbaasLoader, BaseBlobParser
Load Embaas blob.
To use, you should have the
environment variable EMBAAS_API_KEY set with your API key, or pass
it as a named parameter to the constructor.
Example
# Default parsing
from langchain.document_loaders.embaas import EmbaasBlobLoader
loader = EmbaasBlobLoader()
blob = Blob.from_path(path="example.mp3")
documents = loader.parse(blob=blob)
# Custom api parameters (create embeddings automatically)
from langchain.document_loaders.embaas import EmbaasBlobLoader
loader = EmbaasBlobLoader(
    params={
        "should_embed": True,
        "model": "e5-large-v2",
        "chunk_size": 256,
        "chunk_splitter": "CharacterTextSplitter"
    }
)
blob = Blob.from_path(path="example.pdf")
documents = loader.parse(blob=blob)
Create a new model by parsing and validating input data from keyword arguments.
Raises ValidationError if the input data cannot be parsed to form a valid model.
param api_url: str = 'https://api.embaas.io/v1/document/extract-text/bytes/'¶
The URL of the Embaas document extraction API.
param embaas_api_key: Optional[str] = None¶
The API key for the Embaas document extraction API.
param params: langchain.document_loaders.embaas.EmbaasDocumentExtractionParameters = {}¶
Additional parameters to pass to the Embaas document extraction API.


Additional parameters to pass to the Embaas document extraction API.
classmethod construct(_fields_set: Optional[SetStr] = None, **values: Any) → Model¶
Creates a new model setting __dict__ and __fields_set__ from trusted or pre-validated data.
Default values are respected, but no other validation is performed.
Behaves as if Config.extra = ‘allow’ was set since it adds all passed values
copy(*, include: Optional[Union[AbstractSetIntStr, MappingIntStrAny]] = None, exclude: Optional[Union[AbstractSetIntStr, MappingIntStrAny]] = None, update: Optional[DictStrAny] = None, deep: bool = False) → Model¶
Duplicate a model, optionally choose which fields to include, exclude and change.
Parameters
include – fields to include in new model
exclude – fields to exclude from new model, as with values this takes precedence over include
update – values to change/add in the new model. Note: the data is not validated before creating
the new model: you should trust this data
deep – set to True to make a deep copy of the model
Returns
new model instance
dict(*, include: Optional[Union[AbstractSetIntStr, MappingIntStrAny]] = None, exclude: Optional[Union[AbstractSetIntStr, MappingIntStrAny]] = None, by_alias: bool = False, skip_defaults: Optional[bool] = None, exclude_unset: bool = False, exclude_defaults: bool = False, exclude_none: bool = False) → DictStrAny¶
Generate a dictionary representation of the model, optionally specifying which fields to include or exclude.
classmethod from_orm(obj: Any) → Model¶


classmethod from_orm(obj: Any) → Model¶
json(*, include: Optional[Union[AbstractSetIntStr, MappingIntStrAny]] = None, exclude: Optional[Union[AbstractSetIntStr, MappingIntStrAny]] = None, by_alias: bool = False, skip_defaults: Optional[bool] = None, exclude_unset: bool = False, exclude_defaults: bool = False, exclude_none: bool = False, encoder: Optional[Callable[[Any], Any]] = None, models_as_dict: bool = True, **dumps_kwargs: Any) → unicode¶
Generate a JSON representation of the model, include and exclude arguments as per dict().
encoder is an optional function to supply as default to json.dumps(), other arguments as per json.dumps().
lazy_parse(blob: Blob) → Iterator[Document][source]¶
Parses the blob lazily.
Parameters
blob – The blob to parse.
parse(blob: Blob) → List[Document]¶
Eagerly parse the blob into a document or documents.
This is a convenience method for interactive development environment.
Production applications should favor the lazy_parse method instead.
Subclasses should generally not over-ride this parse method.
Parameters
blob – Blob instance
Returns
List of documents
classmethod parse_file(path: Union[str, Path], *, content_type: unicode = None, encoding: unicode = 'utf8', proto: Protocol = None, allow_pickle: bool = False) → Model¶
classmethod parse_obj(obj: Any) → Model¶
classmethod parse_raw(b: Union[str, bytes], *, content_type: unicode = None, encoding: unicode = 'utf8', proto: Protocol = None, allow_pickle: bool = False) → Model¶
classmethod schema(by_alias: bool = True, ref_template: unicode = '#/definitions/{model}') → DictStrAny¶


classmethod schema_json(*, by_alias: bool = True, ref_template: unicode = '#/definitions/{model}', **dumps_kwargs: Any) → unicode¶
classmethod update_forward_refs(**localns: Any) → None¶
Try to update ForwardRefs on fields based on this Model, globalns and localns.
classmethod validate(value: Any) → Model¶
Examples using EmbaasBlobLoader¶
Embaas


langchain.document_loaders.parsers.docai.DocAIParsingResults¶
class langchain.document_loaders.parsers.docai.DocAIParsingResults(source_path: str, parsed_path: str)[source]¶
A dataclass to store Document AI parsing results.
Attributes
source_path
parsed_path
Methods
__init__(source_path, parsed_path)
__init__(source_path: str, parsed_path: str) → None¶


langchain.document_loaders.airbyte.AirbyteHubspotLoader¶
class langchain.document_loaders.airbyte.AirbyteHubspotLoader(config: Mapping[str, Any], stream_name: str, record_handler: Optional[Callable[[Any, Optional[str]], Document]] = None, state: Optional[Any] = None)[source]¶
Load from Hubspot using an Airbyte source connector.
Initializes the loader.
Parameters
config – The config to pass to the source connector.
stream_name – The name of the stream to load.
record_handler – A function that takes in a record and an optional id and
returns a Document. If None, the record will be used as the document.
Defaults to None.
state – The state to pass to the source connector. Defaults to None.
Attributes
last_state
Methods
__init__(config, stream_name[, ...])
Initializes the loader.
lazy_load()
A lazy loader for Documents.
load()
Load data into Document objects.
load_and_split([text_splitter])
Load Documents and split into chunks.
__init__(config: Mapping[str, Any], stream_name: str, record_handler: Optional[Callable[[Any, Optional[str]], Document]] = None, state: Optional[Any] = None) → None[source]¶
Initializes the loader.
Parameters
config – The config to pass to the source connector.
stream_name – The name of the stream to load.
record_handler – A function that takes in a record and an optional id and
returns a Document. If None, the record will be used as the document.
Defaults to None.
state – The state to pass to the source connector. Defaults to None.
lazy_load() → Iterator[Document]¶
A lazy loader for Documents.
load() → List[Document]¶
Load data into Document objects.


load() → List[Document]¶
Load data into Document objects.
load_and_split(text_splitter: Optional[TextSplitter] = None) → List[Document]¶
Load Documents and split into chunks. Chunks are returned as Documents.
Parameters
text_splitter – TextSplitter instance to use for splitting documents.
Defaults to RecursiveCharacterTextSplitter.
Returns
List of Documents.
Examples using AirbyteHubspotLoader¶
Airbyte Hubspot


langchain.document_loaders.pdf.PyPDFDirectoryLoader¶
class langchain.document_loaders.pdf.PyPDFDirectoryLoader(path: str, glob: str = '**/[!.]*.pdf', silent_errors: bool = False, load_hidden: bool = False, recursive: bool = False, extract_images: bool = False)[source]¶
Load a directory with PDF files using pypdf and chunks at character level.
Loader also stores page numbers in metadata.
Methods
__init__(path[, glob, silent_errors, ...])
lazy_load()
A lazy loader for Documents.
load()
Load data into Document objects.
load_and_split([text_splitter])
Load Documents and split into chunks.
__init__(path: str, glob: str = '**/[!.]*.pdf', silent_errors: bool = False, load_hidden: bool = False, recursive: bool = False, extract_images: bool = False)[source]¶
lazy_load() → Iterator[Document]¶
A lazy loader for Documents.
load() → List[Document][source]¶
Load data into Document objects.
load_and_split(text_splitter: Optional[TextSplitter] = None) → List[Document]¶
Load Documents and split into chunks. Chunks are returned as Documents.
Parameters
text_splitter – TextSplitter instance to use for splitting documents.
Defaults to RecursiveCharacterTextSplitter.
Returns
List of Documents.


langchain.document_loaders.mediawikidump.MWDumpLoader¶
class langchain.document_loaders.mediawikidump.MWDumpLoader(file_path: Union[str, Path], encoding: Optional[str] = 'utf8', namespaces: Optional[Sequence[int]] = None, skip_redirects: Optional[bool] = False, stop_on_error: Optional[bool] = True)[source]¶
Load MediaWiki dump from an XML file.
Example
from langchain.document_loaders import MWDumpLoader
loader = MWDumpLoader(
    file_path="myWiki.xml",
    encoding="utf8"
)
docs = loader.load()
from langchain.text_splitter import RecursiveCharacterTextSplitter
text_splitter = RecursiveCharacterTextSplitter(
    chunk_size=1000, chunk_overlap=0
)
texts = text_splitter.split_documents(docs)
Parameters
file_path (str) – XML local file path
encoding (str, optional) – Charset encoding, defaults to “utf8”
namespaces (List[int],optional) – The namespace of pages you want to parse.
See https://www.mediawiki.org/wiki/Help:Namespaces#Localisation
for a list of all common namespaces
skip_redirects (bool, optional) – TR=rue to skip pages that redirect to other pages,
False to keep them. False by default
stop_on_error (bool, optional) – False to skip over pages that cause parsing errors,
True to stop. True by default
Methods
__init__(file_path[, encoding, namespaces, ...])
lazy_load()
A lazy loader for Documents.
load()
Load from a file path.
load_and_split([text_splitter])
Load Documents and split into chunks.


load_and_split([text_splitter])
Load Documents and split into chunks.
__init__(file_path: Union[str, Path], encoding: Optional[str] = 'utf8', namespaces: Optional[Sequence[int]] = None, skip_redirects: Optional[bool] = False, stop_on_error: Optional[bool] = True)[source]¶
lazy_load() → Iterator[Document]¶
A lazy loader for Documents.
load() → List[Document][source]¶
Load from a file path.
load_and_split(text_splitter: Optional[TextSplitter] = None) → List[Document]¶
Load Documents and split into chunks. Chunks are returned as Documents.
Parameters
text_splitter – TextSplitter instance to use for splitting documents.
Defaults to RecursiveCharacterTextSplitter.
Returns
List of Documents.
Examples using MWDumpLoader¶
MediaWikiDump


langchain.document_loaders.obs_file.OBSFileLoader¶
class langchain.document_loaders.obs_file.OBSFileLoader(bucket: str, key: str, client: Any = None, endpoint: str = '', config: Optional[dict] = None)[source]¶
Load from the Huawei OBS file.
Initialize the OBSFileLoader with the specified settings.
Parameters
bucket (str) – The name of the OBS bucket to be used.
key (str) – The name of the object in the OBS bucket.
client (ObsClient, optional) – An instance of the ObsClient to connect to OBS.
endpoint (str, optional) – The endpoint URL of your OBS bucket. This parameter is mandatory if client is not provided.
config (dict, optional) – The parameters for connecting to OBS, provided as a dictionary. This parameter is ignored if client is provided. The dictionary could have the following keys:
- “ak” (str, optional): Your OBS access key (required if get_token_from_ecs is False and bucket policy is not public read).
- “sk” (str, optional): Your OBS secret key (required if get_token_from_ecs is False and bucket policy is not public read).
- “token” (str, optional): Your security token (required if using temporary credentials).
- “get_token_from_ecs” (bool, optional): Whether to retrieve the security token from ECS. Defaults to False if not provided. If set to True, ak, sk, and token will be ignored.
Raises
ValueError – If the esdk-obs-python package is not installed.
TypeError – If the provided client is not an instance of ObsClient.
ValueError – If client is not provided, but endpoint is missing.
Note


ValueError – If client is not provided, but endpoint is missing.
Note
Before using this class, make sure you have registered with OBS and have the necessary credentials. The ak, sk, and endpoint values are mandatory unless get_token_from_ecs is True or the bucket policy is public read. token is required when using temporary credentials.
Example
To create a new OBSFileLoader with a new client:
```
config = {
“ak”: “your-access-key”,
“sk”: “your-secret-key”
}
obs_loader = OBSFileLoader(“your-bucket-name”, “your-object-key”, config=config)
```
To create a new OBSFileLoader with an existing client:
```
from obs import ObsClient
# Assuming you have an existing ObsClient object ‘obs_client’
obs_loader = OBSFileLoader(“your-bucket-name”, “your-object-key”, client=obs_client)
```
To create a new OBSFileLoader without an existing client:
`
obs_loader = OBSFileLoader("your-bucket-name", "your-object-key", endpoint="your-endpoint-url")
`
Methods
__init__(bucket, key[, client, endpoint, config])
Initialize the OBSFileLoader with the specified settings.
lazy_load()
A lazy loader for Documents.
load()
Load documents.
load_and_split([text_splitter])
Load Documents and split into chunks.
__init__(bucket: str, key: str, client: Any = None, endpoint: str = '', config: Optional[dict] = None) → None[source]¶
Initialize the OBSFileLoader with the specified settings.
Parameters
bucket (str) – The name of the OBS bucket to be used.
key (str) – The name of the object in the OBS bucket.


key (str) – The name of the object in the OBS bucket.
client (ObsClient, optional) – An instance of the ObsClient to connect to OBS.
endpoint (str, optional) – The endpoint URL of your OBS bucket. This parameter is mandatory if client is not provided.
config (dict, optional) – The parameters for connecting to OBS, provided as a dictionary. This parameter is ignored if client is provided. The dictionary could have the following keys:
- “ak” (str, optional): Your OBS access key (required if get_token_from_ecs is False and bucket policy is not public read).
- “sk” (str, optional): Your OBS secret key (required if get_token_from_ecs is False and bucket policy is not public read).
- “token” (str, optional): Your security token (required if using temporary credentials).
- “get_token_from_ecs” (bool, optional): Whether to retrieve the security token from ECS. Defaults to False if not provided. If set to True, ak, sk, and token will be ignored.
Raises
ValueError – If the esdk-obs-python package is not installed.
TypeError – If the provided client is not an instance of ObsClient.
ValueError – If client is not provided, but endpoint is missing.
Note
Before using this class, make sure you have registered with OBS and have the necessary credentials. The ak, sk, and endpoint values are mandatory unless get_token_from_ecs is True or the bucket policy is public read. token is required when using temporary credentials.
Example
To create a new OBSFileLoader with a new client:
```
config = {
“ak”: “your-access-key”,
“sk”: “your-secret-key”
}
obs_loader = OBSFileLoader(“your-bucket-name”, “your-object-key”, config=config)
```


```
To create a new OBSFileLoader with an existing client:
```
from obs import ObsClient
# Assuming you have an existing ObsClient object ‘obs_client’
obs_loader = OBSFileLoader(“your-bucket-name”, “your-object-key”, client=obs_client)
```
To create a new OBSFileLoader without an existing client:
`
obs_loader = OBSFileLoader("your-bucket-name", "your-object-key", endpoint="your-endpoint-url")
`
lazy_load() → Iterator[Document]¶
A lazy loader for Documents.
load() → List[Document][source]¶
Load documents.
load_and_split(text_splitter: Optional[TextSplitter] = None) → List[Document]¶
Load Documents and split into chunks. Chunks are returned as Documents.
Parameters
text_splitter – TextSplitter instance to use for splitting documents.
Defaults to RecursiveCharacterTextSplitter.
Returns
List of Documents.
Examples using OBSFileLoader¶
Huawei OBS File


langchain.document_loaders.xorbits.XorbitsLoader¶
class langchain.document_loaders.xorbits.XorbitsLoader(data_frame: Any, page_content_column: str = 'text')[source]¶
Load Xorbits DataFrame.
Initialize with dataframe object.
Requirements:Must have xorbits installed. You can install with pip install xorbits.
Parameters
data_frame – Xorbits DataFrame object.
page_content_column – Name of the column containing the page content.
Defaults to “text”.
Methods
__init__(data_frame[, page_content_column])
Initialize with dataframe object.
lazy_load()
Lazy load records from dataframe.
load()
Load full dataframe.
load_and_split([text_splitter])
Load Documents and split into chunks.
__init__(data_frame: Any, page_content_column: str = 'text')[source]¶
Initialize with dataframe object.
Requirements:Must have xorbits installed. You can install with pip install xorbits.
Parameters
data_frame – Xorbits DataFrame object.
page_content_column – Name of the column containing the page content.
Defaults to “text”.
lazy_load() → Iterator[Document]¶
Lazy load records from dataframe.
load() → List[Document]¶
Load full dataframe.
load_and_split(text_splitter: Optional[TextSplitter] = None) → List[Document]¶
Load Documents and split into chunks. Chunks are returned as Documents.
Parameters
text_splitter – TextSplitter instance to use for splitting documents.
Defaults to RecursiveCharacterTextSplitter.
Returns
List of Documents.
Examples using XorbitsLoader¶
Xorbits Pandas DataFrame


langchain.document_loaders.iugu.IuguLoader¶
class langchain.document_loaders.iugu.IuguLoader(resource: str, api_token: Optional[str] = None)[source]¶
Load from IUGU.
Initialize the IUGU resource.
Parameters
resource – The name of the resource to fetch.
api_token – The IUGU API token to use.
Methods
__init__(resource[, api_token])
Initialize the IUGU resource.
lazy_load()
A lazy loader for Documents.
load()
Load data into Document objects.
load_and_split([text_splitter])
Load Documents and split into chunks.
__init__(resource: str, api_token: Optional[str] = None) → None[source]¶
Initialize the IUGU resource.
Parameters
resource – The name of the resource to fetch.
api_token – The IUGU API token to use.
lazy_load() → Iterator[Document]¶
A lazy loader for Documents.
load() → List[Document][source]¶
Load data into Document objects.
load_and_split(text_splitter: Optional[TextSplitter] = None) → List[Document]¶
Load Documents and split into chunks. Chunks are returned as Documents.
Parameters
text_splitter – TextSplitter instance to use for splitting documents.
Defaults to RecursiveCharacterTextSplitter.
Returns
List of Documents.
Examples using IuguLoader¶
Iugu


langchain.document_loaders.google_speech_to_text.GoogleSpeechToTextLoader¶
class langchain.document_loaders.google_speech_to_text.GoogleSpeechToTextLoader(project_id: str, file_path: str, location: str = 'us-central1', recognizer_id: str = '_', config: Optional[RecognitionConfig] = None, config_mask: Optional[FieldMask] = None)[source]¶
Loader for Google Cloud Speech-to-Text audio transcripts.
It uses the Google Cloud Speech-to-Text API to transcribe audio files
and loads the transcribed text into one or more Documents,
depending on the specified format.
To use, you should have the google-cloud-speech python package installed.
Audio files can be specified via a Google Cloud Storage uri or a local file path.
For a detailed explanation of Google Cloud Speech-to-Text, refer to the product
documentation.
https://cloud.google.com/speech-to-text
Initializes the GoogleSpeechToTextLoader.
Parameters
project_id – Google Cloud Project ID.
file_path – A Google Cloud Storage URI or a local file path.
location – Speech-to-Text recognizer location.
recognizer_id – Speech-to-Text recognizer id.
config – Recognition options and features.
For more information:
https://cloud.google.com/python/docs/reference/speech/latest/google.cloud.speech_v2.types.RecognitionConfig
config_mask – The list of fields in config that override the values in the
default_recognition_config of the recognizer during this
recognition request.
For more information:
https://cloud.google.com/python/docs/reference/speech/latest/google.cloud.speech_v2.types.RecognizeRequest
Methods
__init__(project_id, file_path[, location, ...])
Initializes the GoogleSpeechToTextLoader.
lazy_load()
A lazy loader for Documents.
load()


lazy_load()
A lazy loader for Documents.
load()
Transcribes the audio file and loads the transcript into documents.
load_and_split([text_splitter])
Load Documents and split into chunks.
__init__(project_id: str, file_path: str, location: str = 'us-central1', recognizer_id: str = '_', config: Optional[RecognitionConfig] = None, config_mask: Optional[FieldMask] = None)[source]¶
Initializes the GoogleSpeechToTextLoader.
Parameters
project_id – Google Cloud Project ID.
file_path – A Google Cloud Storage URI or a local file path.
location – Speech-to-Text recognizer location.
recognizer_id – Speech-to-Text recognizer id.
config – Recognition options and features.
For more information:
https://cloud.google.com/python/docs/reference/speech/latest/google.cloud.speech_v2.types.RecognitionConfig
config_mask – The list of fields in config that override the values in the
default_recognition_config of the recognizer during this
recognition request.
For more information:
https://cloud.google.com/python/docs/reference/speech/latest/google.cloud.speech_v2.types.RecognizeRequest
lazy_load() → Iterator[Document]¶
A lazy loader for Documents.
load() → List[Document][source]¶
Transcribes the audio file and loads the transcript into documents.
It uses the Google Cloud Speech-to-Text API to transcribe the audio file
and blocks until the transcription is finished.
load_and_split(text_splitter: Optional[TextSplitter] = None) → List[Document]¶
Load Documents and split into chunks. Chunks are returned as Documents.
Parameters
text_splitter – TextSplitter instance to use for splitting documents.
Defaults to RecursiveCharacterTextSplitter.
Returns
List of Documents.
Examples using GoogleSpeechToTextLoader¶


Returns
List of Documents.
Examples using GoogleSpeechToTextLoader¶
Google Cloud Speech-to-Text


langchain.document_loaders.tencent_cos_file.TencentCOSFileLoader¶
class langchain.document_loaders.tencent_cos_file.TencentCOSFileLoader(conf: Any, bucket: str, key: str)[source]¶
Load from Tencent Cloud COS file.
Initialize with COS config, bucket and key name.
:param conf(CosConfig): COS config.
:param bucket(str): COS bucket.
:param key(str): COS file key.
Methods
__init__(conf, bucket, key)
Initialize with COS config, bucket and key name.
lazy_load()
Load documents.
load()
Load data into Document objects.
load_and_split([text_splitter])
Load Documents and split into chunks.
__init__(conf: Any, bucket: str, key: str)[source]¶
Initialize with COS config, bucket and key name.
:param conf(CosConfig): COS config.
:param bucket(str): COS bucket.
:param key(str): COS file key.
lazy_load() → Iterator[Document][source]¶
Load documents.
load() → List[Document][source]¶
Load data into Document objects.
load_and_split(text_splitter: Optional[TextSplitter] = None) → List[Document]¶
Load Documents and split into chunks. Chunks are returned as Documents.
Parameters
text_splitter – TextSplitter instance to use for splitting documents.
Defaults to RecursiveCharacterTextSplitter.
Returns
List of Documents.
Examples using TencentCOSFileLoader¶
Tencent COS File


langchain.document_loaders.tensorflow_datasets.TensorflowDatasetLoader¶
class langchain.document_loaders.tensorflow_datasets.TensorflowDatasetLoader(dataset_name: str, split_name: str, load_max_docs: Optional[int] = 100, sample_to_document_function: Optional[Callable[[Dict], Document]] = None)[source]¶
Load from TensorFlow Dataset.
dataset_name¶
the name of the dataset to load
split_name¶
the name of the split to load.
load_max_docs¶
a limit to the number of loaded documents. Defaults to 100.
sample_to_document_function¶
a function that converts a dataset sample
into a Document
Example
from langchain.document_loaders import TensorflowDatasetLoader
def mlqaen_example_to_document(example: dict) -> Document:
    return Document(
        page_content=decode_to_str(example["context"]),
        metadata={
            "id": decode_to_str(example["id"]),
            "title": decode_to_str(example["title"]),
            "question": decode_to_str(example["question"]),
            "answer": decode_to_str(example["answers"]["text"][0]),
        },
    )
tsds_client = TensorflowDatasetLoader(
        dataset_name="mlqa/en",
        split_name="test",
        load_max_docs=100,
        sample_to_document_function=mlqaen_example_to_document,
    )
Initialize the TensorflowDatasetLoader.
Parameters
dataset_name – the name of the dataset to load
split_name – the name of the split to load.
load_max_docs – a limit to the number of loaded documents. Defaults to 100.
sample_to_document_function – a function that converts a dataset sample
into a Document.
Attributes
load_max_docs
The maximum number of documents to load.
sample_to_document_function
Custom function that transform a dataset sample into a Document.
Methods


sample_to_document_function
Custom function that transform a dataset sample into a Document.
Methods
__init__(dataset_name, split_name[, ...])
Initialize the TensorflowDatasetLoader.
lazy_load()
A lazy loader for Documents.
load()
Load data into Document objects.
load_and_split([text_splitter])
Load Documents and split into chunks.
__init__(dataset_name: str, split_name: str, load_max_docs: Optional[int] = 100, sample_to_document_function: Optional[Callable[[Dict], Document]] = None)[source]¶
Initialize the TensorflowDatasetLoader.
Parameters
dataset_name – the name of the dataset to load
split_name – the name of the split to load.
load_max_docs – a limit to the number of loaded documents. Defaults to 100.
sample_to_document_function – a function that converts a dataset sample
into a Document.
lazy_load() → Iterator[Document][source]¶
A lazy loader for Documents.
load() → List[Document][source]¶
Load data into Document objects.
load_and_split(text_splitter: Optional[TextSplitter] = None) → List[Document]¶
Load Documents and split into chunks. Chunks are returned as Documents.
Parameters
text_splitter – TextSplitter instance to use for splitting documents.
Defaults to RecursiveCharacterTextSplitter.
Returns
List of Documents.
Examples using TensorflowDatasetLoader¶
TensorFlow Datasets


langchain.document_loaders.rocksetdb.ColumnNotFoundError¶
class langchain.document_loaders.rocksetdb.ColumnNotFoundError(missing_key: str, query: str)[source]¶
Column not found error.


langchain.document_loaders.s3_file.S3FileLoader¶
class langchain.document_loaders.s3_file.S3FileLoader(bucket: str, key: str, *, region_name: Optional[str] = None, api_version: Optional[str] = None, use_ssl: Optional[bool] = True, verify: Union[str, bool, None] = None, endpoint_url: Optional[str] = None, aws_access_key_id: Optional[str] = None, aws_secret_access_key: Optional[str] = None, aws_session_token: Optional[str] = None, boto_config: Optional[botocore.client.Config] = None)[source]¶
Load from Amazon AWS S3 file.
Initialize with bucket and key name.
Parameters
bucket – The name of the S3 bucket.
key – The key of the S3 object.
region_name – The name of the region associated with the client.
A client is associated with a single region.
api_version – The API version to use.  By default, botocore will
use the latest API version when creating a client.  You only need
to specify this parameter if you want to use a previous API version
of the client.
use_ssl – Whether or not to use SSL.  By default, SSL is used.
Note that not all services support non-ssl connections.
verify – Whether or not to verify SSL certificates.
By default SSL certificates are verified.  You can provide the
following values:
False - do not validate SSL certificates.  SSL will still be
used (unless use_ssl is False), but SSL certificates
will not be verified.
path/to/cert/bundle.pem - A filename of the CA cert bundle to
uses.  You can specify this argument if you want to use a
different CA cert bundle than the one used by botocore.
endpoint_url – The complete URL to use for the constructed


endpoint_url – The complete URL to use for the constructed
client.  Normally, botocore will automatically construct the
appropriate URL to use when communicating with a service.  You can
specify a complete URL (including the “http/https” scheme) to
override this behavior.  If this value is provided, then
use_ssl is ignored.
aws_access_key_id – The access key to use when creating
the client.  This is entirely optional, and if not provided,
the credentials configured for the session will automatically
be used.  You only need to provide this argument if you want
to override the credentials used for this specific client.
aws_secret_access_key – The secret key to use when creating
the client.  Same semantics as aws_access_key_id above.
aws_session_token – The session token to use when creating
the client.  Same semantics as aws_access_key_id above.
boto_config (botocore.client.Config) – Advanced boto3 client configuration options. If a value
is specified in the client config, its value will take precedence
over environment variables and configuration values, but not over
a value passed explicitly to the method. If a default config
object is set on the session, the config object used when creating
the client will be the result of calling merge() on the
default config with the config provided to this call.
Methods
__init__(bucket, key, *[, region_name, ...])
Initialize with bucket and key name.
lazy_load()
A lazy loader for Documents.
load()
Load file.
load_and_split([text_splitter])
Load Documents and split into chunks.


load_and_split([text_splitter])
Load Documents and split into chunks.
__init__(bucket: str, key: str, *, region_name: Optional[str] = None, api_version: Optional[str] = None, use_ssl: Optional[bool] = True, verify: Union[str, bool, None] = None, endpoint_url: Optional[str] = None, aws_access_key_id: Optional[str] = None, aws_secret_access_key: Optional[str] = None, aws_session_token: Optional[str] = None, boto_config: Optional[botocore.client.Config] = None)[source]¶
Initialize with bucket and key name.
Parameters
bucket – The name of the S3 bucket.
key – The key of the S3 object.
region_name – The name of the region associated with the client.
A client is associated with a single region.
api_version – The API version to use.  By default, botocore will
use the latest API version when creating a client.  You only need
to specify this parameter if you want to use a previous API version
of the client.
use_ssl – Whether or not to use SSL.  By default, SSL is used.
Note that not all services support non-ssl connections.
verify – Whether or not to verify SSL certificates.
By default SSL certificates are verified.  You can provide the
following values:
False - do not validate SSL certificates.  SSL will still be
used (unless use_ssl is False), but SSL certificates
will not be verified.
path/to/cert/bundle.pem - A filename of the CA cert bundle to
uses.  You can specify this argument if you want to use a
different CA cert bundle than the one used by botocore.
endpoint_url – The complete URL to use for the constructed
client.  Normally, botocore will automatically construct the


client.  Normally, botocore will automatically construct the
appropriate URL to use when communicating with a service.  You can
specify a complete URL (including the “http/https” scheme) to
override this behavior.  If this value is provided, then
use_ssl is ignored.
aws_access_key_id – The access key to use when creating
the client.  This is entirely optional, and if not provided,
the credentials configured for the session will automatically
be used.  You only need to provide this argument if you want
to override the credentials used for this specific client.
aws_secret_access_key – The secret key to use when creating
the client.  Same semantics as aws_access_key_id above.
aws_session_token – The session token to use when creating
the client.  Same semantics as aws_access_key_id above.
boto_config (botocore.client.Config) – Advanced boto3 client configuration options. If a value
is specified in the client config, its value will take precedence
over environment variables and configuration values, but not over
a value passed explicitly to the method. If a default config
object is set on the session, the config object used when creating
the client will be the result of calling merge() on the
default config with the config provided to this call.
lazy_load() → Iterator[Document]¶
A lazy loader for Documents.
load() → List[Document]¶
Load file.
load_and_split(text_splitter: Optional[TextSplitter] = None) → List[Document]¶
Load Documents and split into chunks. Chunks are returned as Documents.
Parameters
text_splitter – TextSplitter instance to use for splitting documents.
Defaults to RecursiveCharacterTextSplitter.
Returns
List of Documents.
Examples using S3FileLoader¶
AWS S3 Directory
AWS S3 File


langchain.document_loaders.cube_semantic.CubeSemanticLoader¶
class langchain.document_loaders.cube_semantic.CubeSemanticLoader(cube_api_url: str, cube_api_token: str, load_dimension_values: bool = True, dimension_values_limit: int = 10000, dimension_values_max_retries: int = 10, dimension_values_retry_delay: int = 3)[source]¶
Load Cube semantic layer metadata.
Parameters
cube_api_url – REST API endpoint.
Use the REST API of your Cube’s deployment.
Please find out more information here:
https://cube.dev/docs/http-api/rest#configuration-base-path
cube_api_token – Cube API token.
Authentication tokens are generated based on your Cube’s API secret.
Please find out more information here:
https://cube.dev/docs/security#generating-json-web-tokens-jwt
load_dimension_values – Whether to load dimension values for every string
dimension or not.
dimension_values_limit – Maximum number of dimension values to load.
dimension_values_max_retries – Maximum number of retries to load dimension
values.
dimension_values_retry_delay – Delay between retries to load dimension values.
Methods
__init__(cube_api_url, cube_api_token[, ...])
lazy_load()
A lazy loader for Documents.
load()
Makes a call to Cube's REST API metadata endpoint.
load_and_split([text_splitter])
Load Documents and split into chunks.
__init__(cube_api_url: str, cube_api_token: str, load_dimension_values: bool = True, dimension_values_limit: int = 10000, dimension_values_max_retries: int = 10, dimension_values_retry_delay: int = 3)[source]¶
lazy_load() → Iterator[Document]¶
A lazy loader for Documents.
load() → List[Document][source]¶
Makes a call to Cube’s REST API metadata endpoint.


Makes a call to Cube’s REST API metadata endpoint.
Returns
page_content=column_title + column_description
metadata
table_name
column_name
column_data_type
column_member_type
column_title
column_description
column_values
cube_data_obj_type
Return type
A list of documents with attributes
load_and_split(text_splitter: Optional[TextSplitter] = None) → List[Document]¶
Load Documents and split into chunks. Chunks are returned as Documents.
Parameters
text_splitter – TextSplitter instance to use for splitting documents.
Defaults to RecursiveCharacterTextSplitter.
Returns
List of Documents.
Examples using CubeSemanticLoader¶
Cube Semantic Layer


langchain.document_loaders.airbyte.AirbyteGongLoader¶
class langchain.document_loaders.airbyte.AirbyteGongLoader(config: Mapping[str, Any], stream_name: str, record_handler: Optional[Callable[[Any, Optional[str]], Document]] = None, state: Optional[Any] = None)[source]¶
Load from Gong using an Airbyte source connector.
Initializes the loader.
Parameters
config – The config to pass to the source connector.
stream_name – The name of the stream to load.
record_handler – A function that takes in a record and an optional id and
returns a Document. If None, the record will be used as the document.
Defaults to None.
state – The state to pass to the source connector. Defaults to None.
Attributes
last_state
Methods
__init__(config, stream_name[, ...])
Initializes the loader.
lazy_load()
A lazy loader for Documents.
load()
Load data into Document objects.
load_and_split([text_splitter])
Load Documents and split into chunks.
__init__(config: Mapping[str, Any], stream_name: str, record_handler: Optional[Callable[[Any, Optional[str]], Document]] = None, state: Optional[Any] = None) → None[source]¶
Initializes the loader.
Parameters
config – The config to pass to the source connector.
stream_name – The name of the stream to load.
record_handler – A function that takes in a record and an optional id and
returns a Document. If None, the record will be used as the document.
Defaults to None.
state – The state to pass to the source connector. Defaults to None.
lazy_load() → Iterator[Document]¶
A lazy loader for Documents.
load() → List[Document]¶
Load data into Document objects.


load() → List[Document]¶
Load data into Document objects.
load_and_split(text_splitter: Optional[TextSplitter] = None) → List[Document]¶
Load Documents and split into chunks. Chunks are returned as Documents.
Parameters
text_splitter – TextSplitter instance to use for splitting documents.
Defaults to RecursiveCharacterTextSplitter.
Returns
List of Documents.
Examples using AirbyteGongLoader¶
Airbyte Gong


langchain.document_loaders.base_o365.O365BaseLoader¶
class langchain.document_loaders.base_o365.O365BaseLoader[source]¶
Bases: BaseLoader, BaseModel
Base class for all loaders that uses O365 Package
Create a new model by parsing and validating input data from keyword arguments.
Raises ValidationError if the input data cannot be parsed to form a valid model.
param auth_with_token: bool = False¶
Whether to authenticate with a token or not. Defaults to False.
param chunk_size: Union[int, str] = 5242880¶
Number of bytes to retrieve from each api call to the server. int or ‘auto’.
param settings: langchain.document_loaders.base_o365._O365Settings [Optional]¶
Settings for the Office365 API client.
classmethod construct(_fields_set: Optional[SetStr] = None, **values: Any) → Model¶
Creates a new model setting __dict__ and __fields_set__ from trusted or pre-validated data.
Default values are respected, but no other validation is performed.
Behaves as if Config.extra = ‘allow’ was set since it adds all passed values
copy(*, include: Optional[Union[AbstractSetIntStr, MappingIntStrAny]] = None, exclude: Optional[Union[AbstractSetIntStr, MappingIntStrAny]] = None, update: Optional[DictStrAny] = None, deep: bool = False) → Model¶
Duplicate a model, optionally choose which fields to include, exclude and change.
Parameters
include – fields to include in new model
exclude – fields to exclude from new model, as with values this takes precedence over include
update – values to change/add in the new model. Note: the data is not validated before creating
the new model: you should trust this data
deep – set to True to make a deep copy of the model
Returns


deep – set to True to make a deep copy of the model
Returns
new model instance
dict(*, include: Optional[Union[AbstractSetIntStr, MappingIntStrAny]] = None, exclude: Optional[Union[AbstractSetIntStr, MappingIntStrAny]] = None, by_alias: bool = False, skip_defaults: Optional[bool] = None, exclude_unset: bool = False, exclude_defaults: bool = False, exclude_none: bool = False) → DictStrAny¶
Generate a dictionary representation of the model, optionally specifying which fields to include or exclude.
classmethod from_orm(obj: Any) → Model¶
json(*, include: Optional[Union[AbstractSetIntStr, MappingIntStrAny]] = None, exclude: Optional[Union[AbstractSetIntStr, MappingIntStrAny]] = None, by_alias: bool = False, skip_defaults: Optional[bool] = None, exclude_unset: bool = False, exclude_defaults: bool = False, exclude_none: bool = False, encoder: Optional[Callable[[Any], Any]] = None, models_as_dict: bool = True, **dumps_kwargs: Any) → unicode¶
Generate a JSON representation of the model, include and exclude arguments as per dict().
encoder is an optional function to supply as default to json.dumps(), other arguments as per json.dumps().
lazy_load() → Iterator[Document]¶
A lazy loader for Documents.
abstract load() → List[Document]¶
Load data into Document objects.
load_and_split(text_splitter: Optional[TextSplitter] = None) → List[Document]¶
Load Documents and split into chunks. Chunks are returned as Documents.
Parameters
text_splitter – TextSplitter instance to use for splitting documents.
Defaults to RecursiveCharacterTextSplitter.
Returns
List of Documents.


Defaults to RecursiveCharacterTextSplitter.
Returns
List of Documents.
classmethod parse_file(path: Union[str, Path], *, content_type: unicode = None, encoding: unicode = 'utf8', proto: Protocol = None, allow_pickle: bool = False) → Model¶
classmethod parse_obj(obj: Any) → Model¶
classmethod parse_raw(b: Union[str, bytes], *, content_type: unicode = None, encoding: unicode = 'utf8', proto: Protocol = None, allow_pickle: bool = False) → Model¶
classmethod schema(by_alias: bool = True, ref_template: unicode = '#/definitions/{model}') → DictStrAny¶
classmethod schema_json(*, by_alias: bool = True, ref_template: unicode = '#/definitions/{model}', **dumps_kwargs: Any) → unicode¶
classmethod update_forward_refs(**localns: Any) → None¶
Try to update ForwardRefs on fields based on this Model, globalns and localns.
classmethod validate(value: Any) → Model¶


langchain.document_loaders.airbyte.AirbyteSalesforceLoader¶
class langchain.document_loaders.airbyte.AirbyteSalesforceLoader(config: Mapping[str, Any], stream_name: str, record_handler: Optional[Callable[[Any, Optional[str]], Document]] = None, state: Optional[Any] = None)[source]¶
Load from Salesforce using an Airbyte source connector.
Initializes the loader.
Parameters
config – The config to pass to the source connector.
stream_name – The name of the stream to load.
record_handler – A function that takes in a record and an optional id and
returns a Document. If None, the record will be used as the document.
Defaults to None.
state – The state to pass to the source connector. Defaults to None.
Attributes
last_state
Methods
__init__(config, stream_name[, ...])
Initializes the loader.
lazy_load()
A lazy loader for Documents.
load()
Load data into Document objects.
load_and_split([text_splitter])
Load Documents and split into chunks.
__init__(config: Mapping[str, Any], stream_name: str, record_handler: Optional[Callable[[Any, Optional[str]], Document]] = None, state: Optional[Any] = None) → None[source]¶
Initializes the loader.
Parameters
config – The config to pass to the source connector.
stream_name – The name of the stream to load.
record_handler – A function that takes in a record and an optional id and
returns a Document. If None, the record will be used as the document.
Defaults to None.
state – The state to pass to the source connector. Defaults to None.
lazy_load() → Iterator[Document]¶
A lazy loader for Documents.
load() → List[Document]¶
Load data into Document objects.


load() → List[Document]¶
Load data into Document objects.
load_and_split(text_splitter: Optional[TextSplitter] = None) → List[Document]¶
Load Documents and split into chunks. Chunks are returned as Documents.
Parameters
text_splitter – TextSplitter instance to use for splitting documents.
Defaults to RecursiveCharacterTextSplitter.
Returns
List of Documents.
Examples using AirbyteSalesforceLoader¶
Airbyte Salesforce


langchain.document_loaders.slack_directory.SlackDirectoryLoader¶
class langchain.document_loaders.slack_directory.SlackDirectoryLoader(zip_path: str, workspace_url: Optional[str] = None)[source]¶
Load from a Slack directory dump.
Initialize the SlackDirectoryLoader.
Parameters
zip_path (str) – The path to the Slack directory dump zip file.
workspace_url (Optional[str]) – The Slack workspace URL.
Including the URL will turn
sources into links. Defaults to None.
Methods
__init__(zip_path[, workspace_url])
Initialize the SlackDirectoryLoader.
lazy_load()
A lazy loader for Documents.
load()
Load and return documents from the Slack directory dump.
load_and_split([text_splitter])
Load Documents and split into chunks.
__init__(zip_path: str, workspace_url: Optional[str] = None)[source]¶
Initialize the SlackDirectoryLoader.
Parameters
zip_path (str) – The path to the Slack directory dump zip file.
workspace_url (Optional[str]) – The Slack workspace URL.
Including the URL will turn
sources into links. Defaults to None.
lazy_load() → Iterator[Document]¶
A lazy loader for Documents.
load() → List[Document][source]¶
Load and return documents from the Slack directory dump.
load_and_split(text_splitter: Optional[TextSplitter] = None) → List[Document]¶
Load Documents and split into chunks. Chunks are returned as Documents.
Parameters
text_splitter – TextSplitter instance to use for splitting documents.
Defaults to RecursiveCharacterTextSplitter.
Returns
List of Documents.
Examples using SlackDirectoryLoader¶
Slack


langchain.document_loaders.parsers.html.bs4.BS4HTMLParser¶
class langchain.document_loaders.parsers.html.bs4.BS4HTMLParser(*, features: str = 'lxml', get_text_separator: str = '', **kwargs: Any)[source]¶
Pparse HTML files using Beautiful Soup.
Initialize a bs4 based HTML parser.
Methods
__init__(*[, features, get_text_separator])
Initialize a bs4 based HTML parser.
lazy_parse(blob)
Load HTML document into document objects.
parse(blob)
Eagerly parse the blob into a document or documents.
__init__(*, features: str = 'lxml', get_text_separator: str = '', **kwargs: Any) → None[source]¶
Initialize a bs4 based HTML parser.
lazy_parse(blob: Blob) → Iterator[Document][source]¶
Load HTML document into document objects.
parse(blob: Blob) → List[Document]¶
Eagerly parse the blob into a document or documents.
This is a convenience method for interactive development environment.
Production applications should favor the lazy_parse method instead.
Subclasses should generally not over-ride this parse method.
Parameters
blob – Blob instance
Returns
List of documents


langchain.document_loaders.arcgis_loader.ArcGISLoader¶
class langchain.document_loaders.arcgis_loader.ArcGISLoader(layer: Union[str, arcgis.features.FeatureLayer], gis: Optional[arcgis.gis.GIS] = None, where: str = '1=1', out_fields: Optional[Union[List[str], str]] = None, return_geometry: bool = False, result_record_count: Optional[int] = None, lyr_desc: Optional[str] = None, **kwargs: Any)[source]¶
Load records from an ArcGIS FeatureLayer.
Methods
__init__(layer[, gis, where, out_fields, ...])
lazy_load()
Lazy load records from FeatureLayer.
load()
Load all records from FeatureLayer.
load_and_split([text_splitter])
Load Documents and split into chunks.
__init__(layer: Union[str, arcgis.features.FeatureLayer], gis: Optional[arcgis.gis.GIS] = None, where: str = '1=1', out_fields: Optional[Union[List[str], str]] = None, return_geometry: bool = False, result_record_count: Optional[int] = None, lyr_desc: Optional[str] = None, **kwargs: Any)[source]¶
lazy_load() → Iterator[Document][source]¶
Lazy load records from FeatureLayer.
load() → List[Document][source]¶
Load all records from FeatureLayer.
load_and_split(text_splitter: Optional[TextSplitter] = None) → List[Document]¶
Load Documents and split into chunks. Chunks are returned as Documents.
Parameters
text_splitter – TextSplitter instance to use for splitting documents.
Defaults to RecursiveCharacterTextSplitter.
Returns
List of Documents.
Examples using ArcGISLoader¶
ArcGIS


langchain.document_loaders.spreedly.SpreedlyLoader¶
class langchain.document_loaders.spreedly.SpreedlyLoader(access_token: str, resource: str)[source]¶
Load from Spreedly API.
Initialize with an access token and a resource.
Parameters
access_token – The access token.
resource – The resource.
Methods
__init__(access_token, resource)
Initialize with an access token and a resource.
lazy_load()
A lazy loader for Documents.
load()
Load data into Document objects.
load_and_split([text_splitter])
Load Documents and split into chunks.
__init__(access_token: str, resource: str) → None[source]¶
Initialize with an access token and a resource.
Parameters
access_token – The access token.
resource – The resource.
lazy_load() → Iterator[Document]¶
A lazy loader for Documents.
load() → List[Document][source]¶
Load data into Document objects.
load_and_split(text_splitter: Optional[TextSplitter] = None) → List[Document]¶
Load Documents and split into chunks. Chunks are returned as Documents.
Parameters
text_splitter – TextSplitter instance to use for splitting documents.
Defaults to RecursiveCharacterTextSplitter.
Returns
List of Documents.
Examples using SpreedlyLoader¶
Spreedly


langchain.document_loaders.embaas.EmbaasDocumentExtractionParameters¶
class langchain.document_loaders.embaas.EmbaasDocumentExtractionParameters[source]¶
Parameters for the embaas document extraction API.
Attributes
mime_type
The mime type of the document.
file_extension
The file extension of the document.
file_name
The file name of the document.
should_chunk
Whether to chunk the document into pages.
chunk_size
The maximum size of the text chunks.
chunk_overlap
The maximum overlap allowed between chunks.
chunk_splitter
The text splitter class name for creating chunks.
separators
The separators for chunks.
should_embed
Whether to create embeddings for the document in the response.
model
The model to pass to the Embaas document extraction API.
instruction
The instruction to pass to the Embaas document extraction API.
Methods
__init__(*args, **kwargs)
clear()
copy()
fromkeys([value])
Create a new dictionary with keys from iterable and values set to value.
get(key[, default])
Return the value for key if key is in the dictionary, else default.
items()
keys()
pop(k[,d])
If the key is not found, return the default if given; otherwise, raise a KeyError.
popitem()
Remove and return a (key, value) pair as a 2-tuple.
setdefault(key[, default])
Insert key with a value of default if key is not in the dictionary.
update([E, ]**F)


update([E, ]**F)
If E is present and has a .keys() method, then does:  for k in E: D[k] = E[k] If E is present and lacks a .keys() method, then does:  for k, v in E: D[k] = v In either case, this is followed by: for k in F:  D[k] = F[k]
values()
__init__(*args, **kwargs)¶
clear() → None.  Remove all items from D.¶
copy() → a shallow copy of D¶
fromkeys(value=None, /)¶
Create a new dictionary with keys from iterable and values set to value.
get(key, default=None, /)¶
Return the value for key if key is in the dictionary, else default.
items() → a set-like object providing a view on D's items¶
keys() → a set-like object providing a view on D's keys¶
pop(k[, d]) → v, remove specified key and return the corresponding value.¶
If the key is not found, return the default if given; otherwise,
raise a KeyError.
popitem()¶
Remove and return a (key, value) pair as a 2-tuple.
Pairs are returned in LIFO (last-in, first-out) order.
Raises KeyError if the dict is empty.
setdefault(key, default=None, /)¶
Insert key with a value of default if key is not in the dictionary.
Return the value for key if key is in the dictionary, else default.
update([E, ]**F) → None.  Update D from dict/iterable E and F.¶
If E is present and has a .keys() method, then does:  for k in E: D[k] = E[k]


If E is present and lacks a .keys() method, then does:  for k, v in E: D[k] = v
In either case, this is followed by: for k in F:  D[k] = F[k]
values() → an object providing a view on D's values¶


langchain.document_loaders.readthedocs.ReadTheDocsLoader¶
class langchain.document_loaders.readthedocs.ReadTheDocsLoader(path: Union[str, Path], encoding: Optional[str] = None, errors: Optional[str] = None, custom_html_tag: Optional[Tuple[str, dict]] = None, patterns: Sequence[str] = ('*.htm', '*.html'), exclude_links_ratio: float = 1.0, **kwargs: Optional[Any])[source]¶
Load ReadTheDocs documentation directory.
Initialize ReadTheDocsLoader
The loader loops over all files under path and extracts the actual content of
the files by retrieving main html tags. Default main html tags include
,
, and
. You
can also define your own html tags by passing custom_html_tag, e.g.
(“div”, “class=main”). The loader iterates html tags with the order of
custom html tags (if exists) and default html tags. If any of the tags is not
empty, the loop will break and retrieve the content out of that tag.
Parameters
path – The location of pulled readthedocs folder.
encoding – The encoding with which to open the documents.
errors – Specify how encoding and decoding errors are to be handled—this
cannot be used in binary mode.
custom_html_tag – Optional custom html tag to retrieve the content from
files.
patterns – The file patterns to load, passed to glob.rglob.
exclude_links_ratio – The ratio of links:content to exclude pages from.
This is to reduce the frequency at which index pages make their
way into retrieved results. Recommended: 0.5
kwargs – named arguments passed to bs4.BeautifulSoup.
Methods
__init__(path[, encoding, errors, ...])
Initialize ReadTheDocsLoader
lazy_load()


Initialize ReadTheDocsLoader
lazy_load()
A lazy loader for Documents.
load()
Load documents.
load_and_split([text_splitter])
Load Documents and split into chunks.
__init__(path: Union[str, Path], encoding: Optional[str] = None, errors: Optional[str] = None, custom_html_tag: Optional[Tuple[str, dict]] = None, patterns: Sequence[str] = ('*.htm', '*.html'), exclude_links_ratio: float = 1.0, **kwargs: Optional[Any])[source]¶
Initialize ReadTheDocsLoader
The loader loops over all files under path and extracts the actual content of
the files by retrieving main html tags. Default main html tags include
,
, and
. You
can also define your own html tags by passing custom_html_tag, e.g.
(“div”, “class=main”). The loader iterates html tags with the order of
custom html tags (if exists) and default html tags. If any of the tags is not
empty, the loop will break and retrieve the content out of that tag.
Parameters
path – The location of pulled readthedocs folder.
encoding – The encoding with which to open the documents.
errors – Specify how encoding and decoding errors are to be handled—this
cannot be used in binary mode.
custom_html_tag – Optional custom html tag to retrieve the content from
files.
patterns – The file patterns to load, passed to glob.rglob.
exclude_links_ratio – The ratio of links:content to exclude pages from.
This is to reduce the frequency at which index pages make their
way into retrieved results. Recommended: 0.5
kwargs – named arguments passed to bs4.BeautifulSoup.
lazy_load() → Iterator[Document][source]¶
A lazy loader for Documents.


lazy_load() → Iterator[Document][source]¶
A lazy loader for Documents.
load() → List[Document][source]¶
Load documents.
load_and_split(text_splitter: Optional[TextSplitter] = None) → List[Document]¶
Load Documents and split into chunks. Chunks are returned as Documents.
Parameters
text_splitter – TextSplitter instance to use for splitting documents.
Defaults to RecursiveCharacterTextSplitter.
Returns
List of Documents.
Examples using ReadTheDocsLoader¶
ReadTheDocs Documentation


langchain.document_loaders.parsers.pdf.PyPDFium2Parser¶
class langchain.document_loaders.parsers.pdf.PyPDFium2Parser(extract_images: bool = False)[source]¶
Parse PDF with PyPDFium2.
Initialize the parser.
Methods
__init__([extract_images])
Initialize the parser.
lazy_parse(blob)
Lazily parse the blob.
parse(blob)
Eagerly parse the blob into a document or documents.
__init__(extract_images: bool = False) → None[source]¶
Initialize the parser.
lazy_parse(blob: Blob) → Iterator[Document][source]¶
Lazily parse the blob.
parse(blob: Blob) → List[Document]¶
Eagerly parse the blob into a document or documents.
This is a convenience method for interactive development environment.
Production applications should favor the lazy_parse method instead.
Subclasses should generally not over-ride this parse method.
Parameters
blob – Blob instance
Returns
List of documents


langchain.document_loaders.python.PythonLoader¶
class langchain.document_loaders.python.PythonLoader(file_path: str)[source]¶
Load Python files, respecting any non-default encoding if specified.
Initialize with a file path.
Parameters
file_path – The path to the file to load.
Methods
__init__(file_path)
Initialize with a file path.
lazy_load()
A lazy loader for Documents.
load()
Load from file path.
load_and_split([text_splitter])
Load Documents and split into chunks.
__init__(file_path: str)[source]¶
Initialize with a file path.
Parameters
file_path – The path to the file to load.
lazy_load() → Iterator[Document]¶
A lazy loader for Documents.
load() → List[Document]¶
Load from file path.
load_and_split(text_splitter: Optional[TextSplitter] = None) → List[Document]¶
Load Documents and split into chunks. Chunks are returned as Documents.
Parameters
text_splitter – TextSplitter instance to use for splitting documents.
Defaults to RecursiveCharacterTextSplitter.
Returns
List of Documents.


langchain.document_loaders.telegram.TelegramChatFileLoader¶
class langchain.document_loaders.telegram.TelegramChatFileLoader(path: str)[source]¶
Load from Telegram chat dump.
Initialize with a path.
Methods
__init__(path)
Initialize with a path.
lazy_load()
A lazy loader for Documents.
load()
Load documents.
load_and_split([text_splitter])
Load Documents and split into chunks.
__init__(path: str)[source]¶
Initialize with a path.
lazy_load() → Iterator[Document]¶
A lazy loader for Documents.
load() → List[Document][source]¶
Load documents.
load_and_split(text_splitter: Optional[TextSplitter] = None) → List[Document]¶
Load Documents and split into chunks. Chunks are returned as Documents.
Parameters
text_splitter – TextSplitter instance to use for splitting documents.
Defaults to RecursiveCharacterTextSplitter.
Returns
List of Documents.
Examples using TelegramChatFileLoader¶
Telegram


langchain.document_loaders.telegram.TelegramChatApiLoader¶
class langchain.document_loaders.telegram.TelegramChatApiLoader(chat_entity: Optional[EntityLike] = None, api_id: Optional[int] = None, api_hash: Optional[str] = None, username: Optional[str] = None, file_path: str = 'telegram_data.json')[source]¶
Load Telegram chat json directory dump.
Initialize with API parameters.
Parameters
chat_entity – The chat entity to fetch data from.
api_id – The API ID.
api_hash – The API hash.
username – The username.
file_path – The file path to save the data to. Defaults to
“telegram_data.json”.
Methods
__init__([chat_entity, api_id, api_hash, ...])
Initialize with API parameters.
fetch_data_from_telegram()
Fetch data from Telegram API and save it as a JSON file.
lazy_load()
A lazy loader for Documents.
load()
Load documents.
load_and_split([text_splitter])
Load Documents and split into chunks.
__init__(chat_entity: Optional[EntityLike] = None, api_id: Optional[int] = None, api_hash: Optional[str] = None, username: Optional[str] = None, file_path: str = 'telegram_data.json')[source]¶
Initialize with API parameters.
Parameters
chat_entity – The chat entity to fetch data from.
api_id – The API ID.
api_hash – The API hash.
username – The username.
file_path – The file path to save the data to. Defaults to
“telegram_data.json”.
async fetch_data_from_telegram() → None[source]¶
Fetch data from Telegram API and save it as a JSON file.
lazy_load() → Iterator[Document]¶
A lazy loader for Documents.


lazy_load() → Iterator[Document]¶
A lazy loader for Documents.
load() → List[Document][source]¶
Load documents.
load_and_split(text_splitter: Optional[TextSplitter] = None) → List[Document]¶
Load Documents and split into chunks. Chunks are returned as Documents.
Parameters
text_splitter – TextSplitter instance to use for splitting documents.
Defaults to RecursiveCharacterTextSplitter.
Returns
List of Documents.
Examples using TelegramChatApiLoader¶
Telegram


langchain.document_loaders.text.TextLoader¶
class langchain.document_loaders.text.TextLoader(file_path: str, encoding: Optional[str] = None, autodetect_encoding: bool = False)[source]¶
Load text file.
Parameters
file_path – Path to the file to load.
encoding – File encoding to use. If None, the file will be loaded
encoding. (with the default system) –
autodetect_encoding – Whether to try to autodetect the file encoding
if the specified encoding fails.
Initialize with file path.
Methods
__init__(file_path[, encoding, ...])
Initialize with file path.
lazy_load()
A lazy loader for Documents.
load()
Load from file path.
load_and_split([text_splitter])
Load Documents and split into chunks.
__init__(file_path: str, encoding: Optional[str] = None, autodetect_encoding: bool = False)[source]¶
Initialize with file path.
lazy_load() → Iterator[Document]¶
A lazy loader for Documents.
load() → List[Document][source]¶
Load from file path.
load_and_split(text_splitter: Optional[TextSplitter] = None) → List[Document]¶
Load Documents and split into chunks. Chunks are returned as Documents.
Parameters
text_splitter – TextSplitter instance to use for splitting documents.
Defaults to RecursiveCharacterTextSplitter.
Returns
List of Documents.
Examples using TextLoader¶
Cohere Reranker
Confident
Elasticsearch
Chat Over Documents with Vectara
Vectorstore
LanceDB
sqlite-vss
Weaviate
DashVector
ScaNN
Xata
Vectara
PGVector
Rockset
DingoDB
Zilliz
SingleStoreDB
Annoy
Typesense
Atlas


Zilliz
SingleStoreDB
Annoy
Typesense
Atlas
Activeloop Deep Lake
Neo4j Vector Index
Tair
Chroma
Alibaba Cloud OpenSearch
Baidu Cloud VectorSearch
StarRocks
scikit-learn
Tencent Cloud VectorDB
DocArray HnswSearch
MyScale
ClickHouse
Qdrant
Tigris
AwaDB
Supabase (Postgres)
OpenSearch
Pinecone
BagelDB
Azure Cognitive Search
Cassandra
USearch
Milvus
Marqo
DocArray InMemorySearch
Postgres Embedding
Faiss
Epsilla
AnalyticDB
Hologres
your local model path
MongoDB Atlas
Meilisearch
Conversational Retrieval Agent
Analysis of Twitter the-algorithm source code with LangChain, GPT4 and Activeloop’s Deep Lake
Use LangChain, GPT and Activeloop’s Deep Lake to work with code base
Structure answers with OpenAI functions
QA using Activeloop’s DeepLake
Graph QA
Caching
MultiVector Retriever
Parent Document Retriever
Combine agents and vector stores
Loading from LangChainHub


langchain.document_loaders.airbyte.AirbyteStripeLoader¶
class langchain.document_loaders.airbyte.AirbyteStripeLoader(config: Mapping[str, Any], stream_name: str, record_handler: Optional[Callable[[Any, Optional[str]], Document]] = None, state: Optional[Any] = None)[source]¶
Load from Stripe using an Airbyte source connector.
Initializes the loader.
Parameters
config – The config to pass to the source connector.
stream_name – The name of the stream to load.
record_handler – A function that takes in a record and an optional id and
returns a Document. If None, the record will be used as the document.
Defaults to None.
state – The state to pass to the source connector. Defaults to None.
Attributes
last_state
Methods
__init__(config, stream_name[, ...])
Initializes the loader.
lazy_load()
A lazy loader for Documents.
load()
Load data into Document objects.
load_and_split([text_splitter])
Load Documents and split into chunks.
__init__(config: Mapping[str, Any], stream_name: str, record_handler: Optional[Callable[[Any, Optional[str]], Document]] = None, state: Optional[Any] = None) → None[source]¶
Initializes the loader.
Parameters
config – The config to pass to the source connector.
stream_name – The name of the stream to load.
record_handler – A function that takes in a record and an optional id and
returns a Document. If None, the record will be used as the document.
Defaults to None.
state – The state to pass to the source connector. Defaults to None.
lazy_load() → Iterator[Document]¶
A lazy loader for Documents.
load() → List[Document]¶
Load data into Document objects.


load() → List[Document]¶
Load data into Document objects.
load_and_split(text_splitter: Optional[TextSplitter] = None) → List[Document]¶
Load Documents and split into chunks. Chunks are returned as Documents.
Parameters
text_splitter – TextSplitter instance to use for splitting documents.
Defaults to RecursiveCharacterTextSplitter.
Returns
List of Documents.
Examples using AirbyteStripeLoader¶
Airbyte Question Answering
Airbyte Stripe


langchain.document_loaders.datadog_logs.DatadogLogsLoader¶
class langchain.document_loaders.datadog_logs.DatadogLogsLoader(query: str, api_key: str, app_key: str, from_time: Optional[int] = None, to_time: Optional[int] = None, limit: int = 100)[source]¶
Load Datadog logs.
Logs are written into the page_content and into the metadata.
Initialize Datadog document loader.
Requirements:
Must have datadog_api_client installed. Install with pip install datadog_api_client.
Parameters
query – The query to run in Datadog.
api_key – The Datadog API key.
app_key – The Datadog APP key.
from_time – Optional. The start of the time range to query.
Supports date math and regular timestamps (milliseconds) like ‘1688732708951’
Defaults to 20 minutes ago.
to_time – Optional. The end of the time range to query.
Supports date math and regular timestamps (milliseconds) like ‘1688732708951’
Defaults to now.
limit – The maximum number of logs to return.
Defaults to 100.
Methods
__init__(query, api_key, app_key[, ...])
Initialize Datadog document loader.
lazy_load()
A lazy loader for Documents.
load()
Get logs from Datadog.
load_and_split([text_splitter])
Load Documents and split into chunks.
parse_log(log)
Create Document objects from Datadog log items.
__init__(query: str, api_key: str, app_key: str, from_time: Optional[int] = None, to_time: Optional[int] = None, limit: int = 100) → None[source]¶
Initialize Datadog document loader.


Initialize Datadog document loader.
Requirements:
Must have datadog_api_client installed. Install with pip install datadog_api_client.
Parameters
query – The query to run in Datadog.
api_key – The Datadog API key.
app_key – The Datadog APP key.
from_time – Optional. The start of the time range to query.
Supports date math and regular timestamps (milliseconds) like ‘1688732708951’
Defaults to 20 minutes ago.
to_time – Optional. The end of the time range to query.
Supports date math and regular timestamps (milliseconds) like ‘1688732708951’
Defaults to now.
limit – The maximum number of logs to return.
Defaults to 100.
lazy_load() → Iterator[Document]¶
A lazy loader for Documents.
load() → List[Document][source]¶
Get logs from Datadog.
Returns
A list of Document objects.
page_content
metadata
id
service
status
tags
timestamp
load_and_split(text_splitter: Optional[TextSplitter] = None) → List[Document]¶
Load Documents and split into chunks. Chunks are returned as Documents.
Parameters
text_splitter – TextSplitter instance to use for splitting documents.
Defaults to RecursiveCharacterTextSplitter.
Returns
List of Documents.
parse_log(log: dict) → Document[source]¶
Create Document objects from Datadog log items.
Examples using DatadogLogsLoader¶
Datadog Logs


langchain.document_loaders.epub.UnstructuredEPubLoader¶
class langchain.document_loaders.epub.UnstructuredEPubLoader(file_path: Union[str, List[str]], mode: str = 'single', **unstructured_kwargs: Any)[source]¶
Load EPub files using Unstructured.
You can run the loader in one of two modes: “single” and “elements”.
If you use “single” mode, the document will be returned as a single
langchain Document object. If you use “elements” mode, the unstructured
library will split the document into elements such as Title and NarrativeText.
You can pass in additional unstructured kwargs after mode to apply
different unstructured settings.
Examples
from langchain.document_loaders import UnstructuredEPubLoader
loader = UnstructuredEPubLoader(“example.epub”, mode=”elements”, strategy=”fast”,
)
docs = loader.load()
References
https://unstructured-io.github.io/unstructured/bricks.html#partition-epub
Initialize with file path.
Methods
__init__(file_path[, mode])
Initialize with file path.
lazy_load()
A lazy loader for Documents.
load()
Load file.
load_and_split([text_splitter])
Load Documents and split into chunks.
__init__(file_path: Union[str, List[str]], mode: str = 'single', **unstructured_kwargs: Any)¶
Initialize with file path.
lazy_load() → Iterator[Document]¶
A lazy loader for Documents.
load() → List[Document]¶
Load file.
load_and_split(text_splitter: Optional[TextSplitter] = None) → List[Document]¶
Load Documents and split into chunks. Chunks are returned as Documents.
Parameters
text_splitter – TextSplitter instance to use for splitting documents.


Parameters
text_splitter – TextSplitter instance to use for splitting documents.
Defaults to RecursiveCharacterTextSplitter.
Returns
List of Documents.
Examples using UnstructuredEPubLoader¶
EPub


langchain.document_loaders.html.UnstructuredHTMLLoader¶
class langchain.document_loaders.html.UnstructuredHTMLLoader(file_path: Union[str, List[str]], mode: str = 'single', **unstructured_kwargs: Any)[source]¶
Load HTML files using Unstructured.
You can run the loader in one of two modes: “single” and “elements”.
If you use “single” mode, the document will be returned as a single
langchain Document object. If you use “elements” mode, the unstructured
library will split the document into elements such as Title and NarrativeText.
You can pass in additional unstructured kwargs after mode to apply
different unstructured settings.
Examples
from langchain.document_loaders import UnstructuredHTMLLoader
loader = UnstructuredHTMLLoader(“example.html”, mode=”elements”, strategy=”fast”,
)
docs = loader.load()
References
https://unstructured-io.github.io/unstructured/bricks.html#partition-html
Initialize with file path.
Methods
__init__(file_path[, mode])
Initialize with file path.
lazy_load()
A lazy loader for Documents.
load()
Load file.
load_and_split([text_splitter])
Load Documents and split into chunks.
__init__(file_path: Union[str, List[str]], mode: str = 'single', **unstructured_kwargs: Any)¶
Initialize with file path.
lazy_load() → Iterator[Document]¶
A lazy loader for Documents.
load() → List[Document]¶
Load file.
load_and_split(text_splitter: Optional[TextSplitter] = None) → List[Document]¶
Load Documents and split into chunks. Chunks are returned as Documents.
Parameters
text_splitter – TextSplitter instance to use for splitting documents.
Defaults to RecursiveCharacterTextSplitter.
Returns
List of Documents.


langchain.document_loaders.rtf.UnstructuredRTFLoader¶
class langchain.document_loaders.rtf.UnstructuredRTFLoader(file_path: str, mode: str = 'single', **unstructured_kwargs: Any)[source]¶
Load RTF files using Unstructured.
You can run the loader in one of two modes: “single” and “elements”.
If you use “single” mode, the document will be returned as a single
langchain Document object. If you use “elements” mode, the unstructured
library will split the document into elements such as Title and NarrativeText.
You can pass in additional unstructured kwargs after mode to apply
different unstructured settings.
Examples
from langchain.document_loaders import UnstructuredRTFLoader
loader = UnstructuredRTFLoader(“example.rtf”, mode=”elements”, strategy=”fast”,
)
docs = loader.load()
References
https://unstructured-io.github.io/unstructured/bricks.html#partition-rtf
Initialize with a file path.
Parameters
file_path – The path to the file to load.
mode – The mode to use for partitioning. See unstructured for details.
Defaults to “single”.
**unstructured_kwargs – Additional keyword arguments to pass
to unstructured.
Methods
__init__(file_path[, mode])
Initialize with a file path.
lazy_load()
A lazy loader for Documents.
load()
Load file.
load_and_split([text_splitter])
Load Documents and split into chunks.
__init__(file_path: str, mode: str = 'single', **unstructured_kwargs: Any)[source]¶
Initialize with a file path.
Parameters
file_path – The path to the file to load.
mode – The mode to use for partitioning. See unstructured for details.
Defaults to “single”.


Defaults to “single”.
**unstructured_kwargs – Additional keyword arguments to pass
to unstructured.
lazy_load() → Iterator[Document]¶
A lazy loader for Documents.
load() → List[Document]¶
Load file.
load_and_split(text_splitter: Optional[TextSplitter] = None) → List[Document]¶
Load Documents and split into chunks. Chunks are returned as Documents.
Parameters
text_splitter – TextSplitter instance to use for splitting documents.
Defaults to RecursiveCharacterTextSplitter.
Returns
List of Documents.


langchain.document_loaders.airbyte.AirbyteTypeformLoader¶
class langchain.document_loaders.airbyte.AirbyteTypeformLoader(config: Mapping[str, Any], stream_name: str, record_handler: Optional[Callable[[Any, Optional[str]], Document]] = None, state: Optional[Any] = None)[source]¶
Load from Typeform using an Airbyte source connector.
Initializes the loader.
Parameters
config – The config to pass to the source connector.
stream_name – The name of the stream to load.
record_handler – A function that takes in a record and an optional id and
returns a Document. If None, the record will be used as the document.
Defaults to None.
state – The state to pass to the source connector. Defaults to None.
Attributes
last_state
Methods
__init__(config, stream_name[, ...])
Initializes the loader.
lazy_load()
A lazy loader for Documents.
load()
Load data into Document objects.
load_and_split([text_splitter])
Load Documents and split into chunks.
__init__(config: Mapping[str, Any], stream_name: str, record_handler: Optional[Callable[[Any, Optional[str]], Document]] = None, state: Optional[Any] = None) → None[source]¶
Initializes the loader.
Parameters
config – The config to pass to the source connector.
stream_name – The name of the stream to load.
record_handler – A function that takes in a record and an optional id and
returns a Document. If None, the record will be used as the document.
Defaults to None.
state – The state to pass to the source connector. Defaults to None.
lazy_load() → Iterator[Document]¶
A lazy loader for Documents.
load() → List[Document]¶
Load data into Document objects.


load() → List[Document]¶
Load data into Document objects.
load_and_split(text_splitter: Optional[TextSplitter] = None) → List[Document]¶
Load Documents and split into chunks. Chunks are returned as Documents.
Parameters
text_splitter – TextSplitter instance to use for splitting documents.
Defaults to RecursiveCharacterTextSplitter.
Returns
List of Documents.
Examples using AirbyteTypeformLoader¶
Airbyte Typeform


langchain.document_loaders.bilibili.BiliBiliLoader¶
class langchain.document_loaders.bilibili.BiliBiliLoader(video_urls: List[str])[source]¶
Load BiliBili video transcripts.
Initialize with bilibili url.
Parameters
video_urls – List of bilibili urls.
Methods
__init__(video_urls)
Initialize with bilibili url.
lazy_load()
A lazy loader for Documents.
load()
Load Documents from bilibili url.
load_and_split([text_splitter])
Load Documents and split into chunks.
__init__(video_urls: List[str])[source]¶
Initialize with bilibili url.
Parameters
video_urls – List of bilibili urls.
lazy_load() → Iterator[Document]¶
A lazy loader for Documents.
load() → List[Document][source]¶
Load Documents from bilibili url.
load_and_split(text_splitter: Optional[TextSplitter] = None) → List[Document]¶
Load Documents and split into chunks. Chunks are returned as Documents.
Parameters
text_splitter – TextSplitter instance to use for splitting documents.
Defaults to RecursiveCharacterTextSplitter.
Returns
List of Documents.
Examples using BiliBiliLoader¶
BiliBili


langchain.document_loaders.pdf.MathpixPDFLoader¶
class langchain.document_loaders.pdf.MathpixPDFLoader(file_path: str, processed_file_format: str = 'md', max_wait_time_seconds: int = 500, should_clean_pdf: bool = False, **kwargs: Any)[source]¶
Load PDF files using Mathpix service.
Initialize with a file path.
Parameters
file_path – a file for loading.
processed_file_format – a format of the processed file. Default is “md”.
max_wait_time_seconds – a maximum time to wait for the response from
the server. Default is 500.
should_clean_pdf – a flag to clean the PDF file. Default is False.
**kwargs – additional keyword arguments.
Attributes
data
source
url
Methods
__init__(file_path[, processed_file_format, ...])
Initialize with a file path.
clean_pdf(contents)
Clean the PDF file.
get_processed_pdf(pdf_id)
lazy_load()
A lazy loader for Documents.
load()
Load data into Document objects.
load_and_split([text_splitter])
Load Documents and split into chunks.
send_pdf()
wait_for_processing(pdf_id)
Wait for processing to complete.
__init__(file_path: str, processed_file_format: str = 'md', max_wait_time_seconds: int = 500, should_clean_pdf: bool = False, **kwargs: Any) → None[source]¶
Initialize with a file path.
Parameters
file_path – a file for loading.
processed_file_format – a format of the processed file. Default is “md”.
max_wait_time_seconds – a maximum time to wait for the response from
the server. Default is 500.
should_clean_pdf – a flag to clean the PDF file. Default is False.
**kwargs – additional keyword arguments.


**kwargs – additional keyword arguments.
clean_pdf(contents: str) → str[source]¶
Clean the PDF file.
Parameters
contents – a PDF file contents.
Returns:
get_processed_pdf(pdf_id: str) → str[source]¶
lazy_load() → Iterator[Document]¶
A lazy loader for Documents.
load() → List[Document][source]¶
Load data into Document objects.
load_and_split(text_splitter: Optional[TextSplitter] = None) → List[Document]¶
Load Documents and split into chunks. Chunks are returned as Documents.
Parameters
text_splitter – TextSplitter instance to use for splitting documents.
Defaults to RecursiveCharacterTextSplitter.
Returns
List of Documents.
send_pdf() → str[source]¶
wait_for_processing(pdf_id: str) → None[source]¶
Wait for processing to complete.
Parameters
pdf_id – a PDF id.
Returns: None


langchain.document_loaders.youtube.GoogleApiClient¶
class langchain.document_loaders.youtube.GoogleApiClient(credentials_path: Path = PosixPath('/home/docs/.credentials/credentials.json'), service_account_path: Path = PosixPath('/home/docs/.credentials/credentials.json'), token_path: Path = PosixPath('/home/docs/.credentials/token.json'))[source]¶
Generic Google API Client.
To use, you should have the google_auth_oauthlib,youtube_transcript_api,google
python package installed.
As the google api expects credentials you need to set up a google account and
register your Service. “https://developers.google.com/docs/api/quickstart/python”
Example
from langchain.document_loaders import GoogleApiClient
google_api_client = GoogleApiClient(
    service_account_path=Path("path_to_your_sec_file.json")
)
Attributes
credentials_path
service_account_path
token_path
Methods
__init__([credentials_path, ...])
validate_channel_or_videoIds_is_set(values)
Validate that either folder_id or document_ids is set, but not both.
__init__(credentials_path: Path = PosixPath('/home/docs/.credentials/credentials.json'), service_account_path: Path = PosixPath('/home/docs/.credentials/credentials.json'), token_path: Path = PosixPath('/home/docs/.credentials/token.json')) → None¶
classmethod validate_channel_or_videoIds_is_set(values: Dict[str, Any]) → Dict[str, Any][source]¶
Validate that either folder_id or document_ids is set, but not both.
Examples using GoogleApiClient¶
YouTube transcripts


langchain.document_loaders.parsers.pdf.PyMuPDFParser¶
class langchain.document_loaders.parsers.pdf.PyMuPDFParser(text_kwargs: Optional[Mapping[str, Any]] = None, extract_images: bool = False)[source]¶
Parse PDF using PyMuPDF.
Initialize the parser.
Parameters
text_kwargs – Keyword arguments to pass to fitz.Page.get_text().
Methods
__init__([text_kwargs, extract_images])
Initialize the parser.
lazy_parse(blob)
Lazily parse the blob.
parse(blob)
Eagerly parse the blob into a document or documents.
__init__(text_kwargs: Optional[Mapping[str, Any]] = None, extract_images: bool = False) → None[source]¶
Initialize the parser.
Parameters
text_kwargs – Keyword arguments to pass to fitz.Page.get_text().
lazy_parse(blob: Blob) → Iterator[Document][source]¶
Lazily parse the blob.
parse(blob: Blob) → List[Document]¶
Eagerly parse the blob into a document or documents.
This is a convenience method for interactive development environment.
Production applications should favor the lazy_parse method instead.
Subclasses should generally not over-ride this parse method.
Parameters
blob – Blob instance
Returns
List of documents


langchain.document_loaders.news.NewsURLLoader¶
class langchain.document_loaders.news.NewsURLLoader(urls: List[str], text_mode: bool = True, nlp: bool = False, continue_on_failure: bool = True, show_progress_bar: bool = False, **newspaper_kwargs: Any)[source]¶
Load news articles from URLs using Unstructured.
Parameters
urls – URLs to load. Each is loaded into its own document.
text_mode – If True, extract text from URL and use that for page content.
Otherwise, extract raw HTML.
nlp – If True, perform NLP on the extracted contents, like providing a summary
and extracting keywords.
continue_on_failure – If True, continue loading documents even if
loading fails for a particular URL.
show_progress_bar – If True, use tqdm to show a loading progress bar. Requires
tqdm to be installed, pip install tqdm.
**newspaper_kwargs – Any additional named arguments to pass to
newspaper.Article().
Example
from langchain.document_loaders import NewsURLLoader
loader = NewsURLLoader(
    urls=["<url-1>", "<url-2>"],
)
docs = loader.load()
Newspaper reference:https://newspaper.readthedocs.io/en/latest/
Initialize with file path.
Methods
__init__(urls[, text_mode, nlp, ...])
Initialize with file path.
lazy_load()
A lazy loader for Documents.
load()
Load data into Document objects.
load_and_split([text_splitter])
Load Documents and split into chunks.
__init__(urls: List[str], text_mode: bool = True, nlp: bool = False, continue_on_failure: bool = True, show_progress_bar: bool = False, **newspaper_kwargs: Any) → None[source]¶


Initialize with file path.
lazy_load() → Iterator[Document][source]¶
A lazy loader for Documents.
load() → List[Document][source]¶
Load data into Document objects.
load_and_split(text_splitter: Optional[TextSplitter] = None) → List[Document]¶
Load Documents and split into chunks. Chunks are returned as Documents.
Parameters
text_splitter – TextSplitter instance to use for splitting documents.
Defaults to RecursiveCharacterTextSplitter.
Returns
List of Documents.
Examples using NewsURLLoader¶
News URL