on_tool_start(serialized, input_str, *, run_id)
Start a trace for a tool run.
__init__(run_args: Optional[WandbRunArgs] = None, **kwargs: Any) → None[source]¶
Initializes the WandbTracer.
Parameters
run_args – (dict, optional) Arguments to pass to wandb.init(). If not
provided, wandb.init() will be called with no arguments. Please
refer to the wandb.init for more details.
To use W&B to monitor all LangChain activity, add this tracer like any other
LangChain callback:
```
from wandb.integration.langchain import WandbTracer
tracer = WandbTracer()
chain = LLMChain(llm, callbacks=[tracer])
# …end of notebook / script:
tracer.finish()
```
finish() → None[source]¶
Waits for all asynchronous processes to finish and data to upload.
Proxy for wandb.finish().
on_agent_action(action: AgentAction, *, run_id: UUID, parent_run_id: Optional[UUID] = None, **kwargs: Any) → Any¶
Run on agent action.
on_agent_finish(finish: AgentFinish, *, run_id: UUID, parent_run_id: Optional[UUID] = None, **kwargs: Any) → Any¶
Run on agent end.
on_chain_end(outputs: Dict[str, Any], *, run_id: UUID, inputs: Optional[Dict[str, Any]] = None, **kwargs: Any) → Run¶
End a trace for a chain run.
on_chain_error(error: BaseException, *, inputs: Optional[Dict[str, Any]] = None, run_id: UUID, **kwargs: Any) → Run¶
Handle an error for a chain run.


Handle an error for a chain run.
on_chain_start(serialized: Dict[str, Any], inputs: Dict[str, Any], *, run_id: UUID, tags: Optional[List[str]] = None, parent_run_id: Optional[UUID] = None, metadata: Optional[Dict[str, Any]] = None, run_type: Optional[str] = None, name: Optional[str] = None, **kwargs: Any) → Run¶
Start a trace for a chain run.
on_chat_model_start(serialized: Dict[str, Any], messages: List[List[BaseMessage]], *, run_id: UUID, parent_run_id: Optional[UUID] = None, tags: Optional[List[str]] = None, metadata: Optional[Dict[str, Any]] = None, **kwargs: Any) → Any¶
Run when a chat model starts running.
on_llm_end(response: LLMResult, *, run_id: UUID, **kwargs: Any) → Run¶
End a trace for an LLM run.
on_llm_error(error: BaseException, *, run_id: UUID, **kwargs: Any) → Run¶
Handle an error for an LLM run.
on_llm_new_token(token: str, *, chunk: Optional[Union[GenerationChunk, ChatGenerationChunk]] = None, run_id: UUID, parent_run_id: Optional[UUID] = None, **kwargs: Any) → Run¶
Run on new LLM token. Only available when streaming is enabled.
on_llm_start(serialized: Dict[str, Any], prompts: List[str], *, run_id: UUID, tags: Optional[List[str]] = None, parent_run_id: Optional[UUID] = None, metadata: Optional[Dict[str, Any]] = None, name: Optional[str] = None, **kwargs: Any) → Run¶
Start a trace for an LLM run.


Start a trace for an LLM run.
on_retriever_end(documents: Sequence[Document], *, run_id: UUID, **kwargs: Any) → Run¶
Run when Retriever ends running.
on_retriever_error(error: BaseException, *, run_id: UUID, **kwargs: Any) → Run¶
Run when Retriever errors.
on_retriever_start(serialized: Dict[str, Any], query: str, *, run_id: UUID, parent_run_id: Optional[UUID] = None, tags: Optional[List[str]] = None, metadata: Optional[Dict[str, Any]] = None, name: Optional[str] = None, **kwargs: Any) → Run¶
Run when Retriever starts running.
on_retry(retry_state: RetryCallState, *, run_id: UUID, **kwargs: Any) → Run¶
Run on a retry event.
on_text(text: str, *, run_id: UUID, parent_run_id: Optional[UUID] = None, **kwargs: Any) → Any¶
Run on arbitrary text.
on_tool_end(output: str, *, run_id: UUID, **kwargs: Any) → Run¶
End a trace for a tool run.
on_tool_error(error: BaseException, *, run_id: UUID, **kwargs: Any) → Run¶
Handle an error for a tool run.
on_tool_start(serialized: Dict[str, Any], input_str: str, *, run_id: UUID, tags: Optional[List[str]] = None, parent_run_id: Optional[UUID] = None, metadata: Optional[Dict[str, Any]] = None, name: Optional[str] = None, **kwargs: Any) → Run¶
Start a trace for a tool run.


langchain.callbacks.aim_callback.import_aim¶
langchain.callbacks.aim_callback.import_aim() → Any[source]¶
Import the aim python package and raise an error if it is not installed.


langchain.callbacks.llmonitor_callback.identify¶
langchain.callbacks.llmonitor_callback.identify(user_id: str, user_props: Any = None) → UserContextManager[source]¶
Builds an LLMonitor UserContextManager
Parameters
user_id (-) – The user id.
user_props (-) – The user properties.
Returns
A context manager that sets the user context.


langchain.callbacks.context_callback.import_context¶
langchain.callbacks.context_callback.import_context() → Any[source]¶
Import the getcontext package.


langchain.callbacks.manager.get_openai_callback¶
langchain.callbacks.manager.get_openai_callback() → Generator[OpenAICallbackHandler, None, None][source]¶
Get the OpenAI callback handler in a context manager.
which conveniently exposes token and cost information.
Returns
The OpenAI callback handler.
Return type
OpenAICallbackHandler
Example
>>> with get_openai_callback() as cb:
...     # Use the OpenAI callback handler
Examples using get_openai_callback¶
Azure
Token counting
Tracking token usage
Run arbitrary functions


langchain.callbacks.tracers.wandb.RunProcessor¶
class langchain.callbacks.tracers.wandb.RunProcessor(wandb_module: Any, trace_module: Any)[source]¶
Handles the conversion of a LangChain Runs into a WBTraceTree.
Methods
__init__(wandb_module, trace_module)
build_tree(runs)
Builds a nested dictionary from a list of runs. :param runs: The list of runs to build the tree from. :return: The nested dictionary representing the langchain Run in a tree     structure compatible with WBTraceTree.
flatten_run(run)
Utility to flatten a nest run object into a list of runs.
modify_serialized_iterative(runs[, ...])
Utility to modify the serialized field of a list of runs dictionaries.
process_model(run)
Utility to process a run for wandb model_dict serialization.
process_span(run)
Converts a LangChain Run into a W&B Trace Span.
truncate_run_iterative(runs[, keep_keys])
Utility to truncate a list of runs dictionaries to only keep the specified
__init__(wandb_module: Any, trace_module: Any)[source]¶
build_tree(runs: List[Dict[str, Any]]) → Dict[str, Any][source]¶
Builds a nested dictionary from a list of runs.
:param runs: The list of runs to build the tree from.
:return: The nested dictionary representing the langchain Run in a tree
structure compatible with WBTraceTree.
flatten_run(run: Dict[str, Any]) → List[Dict[str, Any]][source]¶
Utility to flatten a nest run object into a list of runs.
:param run: The base run to flatten.
:return: The flattened list of runs.


:param run: The base run to flatten.
:return: The flattened list of runs.
modify_serialized_iterative(runs: List[Dict[str, Any]], exact_keys: Tuple[str, ...] = (), partial_keys: Tuple[str, ...] = ()) → List[Dict[str, Any]][source]¶
Utility to modify the serialized field of a list of runs dictionaries.
removes any keys that match the exact_keys and any keys that contain any of the
partial_keys.
recursively moves the dictionaries under the kwargs key to the top level.
changes the “id” field to a string “_kind” field that tells WBTraceTree how to
visualize the run. promotes the “serialized” field to the top level.
Parameters
runs – The list of runs to modify.
exact_keys – A tuple of keys to remove from the serialized field.
partial_keys – A tuple of partial keys to remove from the serialized
field.
Returns
The modified list of runs.
process_model(run: Run) → Optional[Dict[str, Any]][source]¶
Utility to process a run for wandb model_dict serialization.
:param run: The run to process.
:return: The convert model_dict to pass to WBTraceTree.
process_span(run: Run) → Optional['Span'][source]¶
Converts a LangChain Run into a W&B Trace Span.
:param run: The LangChain Run to convert.
:return: The converted W&B Trace Span.
truncate_run_iterative(runs: List[Dict[str, Any]], keep_keys: Tuple[str, ...] = ()) → List[Dict[str, Any]][source]¶
Utility to truncate a list of runs dictionaries to only keep the specifiedkeys in each run.
Parameters
runs – The list of runs to truncate.
keep_keys – The keys to keep in each run.
Returns


keep_keys – The keys to keep in each run.
Returns
The truncated list of runs.


langchain.callbacks.utils.load_json¶
langchain.callbacks.utils.load_json(json_path: Union[str, Path]) → str[source]¶
Load json file to a string.
Parameters
json_path (str) – The path to the json file.
Returns
The string representation of the json file.
Return type
(str)


langchain.callbacks.utils.import_spacy¶
langchain.callbacks.utils.import_spacy() → Any[source]¶
Import the spacy python package and raise an error if it is not installed.


langchain.callbacks.infino_callback.import_tiktoken¶
langchain.callbacks.infino_callback.import_tiktoken() → Any[source]¶
Import tiktoken for counting tokens for OpenAI models.


langchain.callbacks.comet_ml_callback.CometCallbackHandler¶
class langchain.callbacks.comet_ml_callback.CometCallbackHandler(task_type: Optional[str] = 'inference', workspace: Optional[str] = None, project_name: Optional[str] = None, tags: Optional[Sequence] = None, name: Optional[str] = None, visualizations: Optional[List[str]] = None, complexity_metrics: bool = False, custom_metrics: Optional[Callable] = None, stream_logs: bool = True)[source]¶
Callback Handler that logs to Comet.
Parameters
job_type (str) – The type of comet_ml task such as “inference”,
“testing” or “qc”
project_name (str) – The comet_ml project name
tags (list) – Tags to add to the task
task_name (str) – Name of the comet_ml task
visualize (bool) – Whether to visualize the run.
complexity_metrics (bool) – Whether to log complexity metrics
stream_logs (bool) – Whether to stream callback actions to Comet
This handler will utilize the associated callback method and formats
the input of each callback function with metadata regarding the state of LLM run,
and adds the response to the list of records for both the {method}_records and
action. It then logs the response to Comet.
Initialize callback handler.
Attributes
always_verbose
Whether to call verbose callbacks even if verbose is False.
ignore_agent
Whether to ignore agent callbacks.
ignore_chain
Whether to ignore chain callbacks.
ignore_chat_model
Whether to ignore chat model callbacks.
ignore_llm
Whether to ignore LLM callbacks.
ignore_retriever
Whether to ignore retriever callbacks.
ignore_retry
Whether to ignore retry callbacks.
raise_error
run_inline
Methods
__init__([task_type, workspace, ...])
Initialize callback handler.


__init__([task_type, workspace, ...])
Initialize callback handler.
flush_tracker([langchain_asset, task_type, ...])
Flush the tracker and setup the session.
get_custom_callback_meta()
on_agent_action(action, **kwargs)
Run on agent action.
on_agent_finish(finish, **kwargs)
Run when agent ends running.
on_chain_end(outputs, **kwargs)
Run when chain ends running.
on_chain_error(error, **kwargs)
Run when chain errors.
on_chain_start(serialized, inputs, **kwargs)
Run when chain starts running.
on_chat_model_start(serialized, messages, *, ...)
Run when a chat model starts running.
on_llm_end(response, **kwargs)
Run when LLM ends running.
on_llm_error(error, **kwargs)
Run when LLM errors.
on_llm_new_token(token, **kwargs)
Run when LLM generates a new token.
on_llm_start(serialized, prompts, **kwargs)
Run when LLM starts.
on_retriever_end(documents, *, run_id[, ...])
Run when Retriever ends running.
on_retriever_error(error, *, run_id[, ...])
Run when Retriever errors.
on_retriever_start(serialized, query, *, run_id)
Run when Retriever starts running.
on_retry(retry_state, *, run_id[, parent_run_id])
Run on a retry event.
on_text(text, **kwargs)
Run when agent is ending.
on_tool_end(output, **kwargs)
Run when tool ends running.
on_tool_error(error, **kwargs)
Run when tool errors.


on_tool_error(error, **kwargs)
Run when tool errors.
on_tool_start(serialized, input_str, **kwargs)
Run when tool starts running.
reset_callback_meta()
Reset the callback metadata.
__init__(task_type: Optional[str] = 'inference', workspace: Optional[str] = None, project_name: Optional[str] = None, tags: Optional[Sequence] = None, name: Optional[str] = None, visualizations: Optional[List[str]] = None, complexity_metrics: bool = False, custom_metrics: Optional[Callable] = None, stream_logs: bool = True) → None[source]¶
Initialize callback handler.
flush_tracker(langchain_asset: Any = None, task_type: Optional[str] = 'inference', workspace: Optional[str] = None, project_name: Optional[str] = 'comet-langchain-demo', tags: Optional[Sequence] = None, name: Optional[str] = None, visualizations: Optional[List[str]] = None, complexity_metrics: bool = False, custom_metrics: Optional[Callable] = None, finish: bool = False, reset: bool = False) → None[source]¶
Flush the tracker and setup the session.
Everything after this will be a new table.
Parameters
name – Name of the performed session so far so it is identifiable
langchain_asset – The langchain asset to save.
finish – Whether to finish the run.
Returns – None
get_custom_callback_meta() → Dict[str, Any]¶
on_agent_action(action: AgentAction, **kwargs: Any) → Any[source]¶
Run on agent action.
on_agent_finish(finish: AgentFinish, **kwargs: Any) → None[source]¶
Run when agent ends running.
on_chain_end(outputs: Dict[str, Any], **kwargs: Any) → None[source]¶
Run when chain ends running.


Run when chain ends running.
on_chain_error(error: BaseException, **kwargs: Any) → None[source]¶
Run when chain errors.
on_chain_start(serialized: Dict[str, Any], inputs: Dict[str, Any], **kwargs: Any) → None[source]¶
Run when chain starts running.
on_chat_model_start(serialized: Dict[str, Any], messages: List[List[BaseMessage]], *, run_id: UUID, parent_run_id: Optional[UUID] = None, tags: Optional[List[str]] = None, metadata: Optional[Dict[str, Any]] = None, **kwargs: Any) → Any¶
Run when a chat model starts running.
on_llm_end(response: LLMResult, **kwargs: Any) → None[source]¶
Run when LLM ends running.
on_llm_error(error: BaseException, **kwargs: Any) → None[source]¶
Run when LLM errors.
on_llm_new_token(token: str, **kwargs: Any) → None[source]¶
Run when LLM generates a new token.
on_llm_start(serialized: Dict[str, Any], prompts: List[str], **kwargs: Any) → None[source]¶
Run when LLM starts.
on_retriever_end(documents: Sequence[Document], *, run_id: UUID, parent_run_id: Optional[UUID] = None, **kwargs: Any) → Any¶
Run when Retriever ends running.
on_retriever_error(error: BaseException, *, run_id: UUID, parent_run_id: Optional[UUID] = None, **kwargs: Any) → Any¶
Run when Retriever errors.


Run when Retriever errors.
on_retriever_start(serialized: Dict[str, Any], query: str, *, run_id: UUID, parent_run_id: Optional[UUID] = None, tags: Optional[List[str]] = None, metadata: Optional[Dict[str, Any]] = None, **kwargs: Any) → Any¶
Run when Retriever starts running.
on_retry(retry_state: RetryCallState, *, run_id: UUID, parent_run_id: Optional[UUID] = None, **kwargs: Any) → Any¶
Run on a retry event.
on_text(text: str, **kwargs: Any) → None[source]¶
Run when agent is ending.
on_tool_end(output: str, **kwargs: Any) → None[source]¶
Run when tool ends running.
on_tool_error(error: BaseException, **kwargs: Any) → None[source]¶
Run when tool errors.
on_tool_start(serialized: Dict[str, Any], input_str: str, **kwargs: Any) → None[source]¶
Run when tool starts running.
reset_callback_meta() → None¶
Reset the callback metadata.
Examples using CometCallbackHandler¶
Comet


langchain.callbacks.confident_callback.DeepEvalCallbackHandler¶
class langchain.callbacks.confident_callback.DeepEvalCallbackHandler(metrics: List[Any], implementation_name: Optional[str] = None)[source]¶
Callback Handler that logs into deepeval.
Parameters
implementation_name – name of the implementation in deepeval
metrics – A list of metrics
Raises
ImportError – if the deepeval package is not installed.
Examples
>>> from langchain.llms import OpenAI
>>> from langchain.callbacks import DeepEvalCallbackHandler
>>> from deepeval.metrics import AnswerRelevancy
>>> metric = AnswerRelevancy(minimum_score=0.3)
>>> deepeval_callback = DeepEvalCallbackHandler(
...     implementation_name="exampleImplementation",
...     metrics=[metric],
... )
>>> llm = OpenAI(
...     temperature=0,
...     callbacks=[deepeval_callback],
...     verbose=True,
...     openai_api_key="API_KEY_HERE",
... )
>>> llm.generate([
...     "What is the best evaluation tool out there? (no bias at all)",
... ])
"Deepeval, no doubt about it."
Initializes the deepevalCallbackHandler.
Parameters
implementation_name – Name of the implementation you want.
metrics – What metrics do you want to track?
Raises
ImportError – if the deepeval package is not installed.
ConnectionError – if the connection to deepeval fails.
Attributes
BLOG_URL
ISSUES_URL
REPO_URL
ignore_agent
Whether to ignore agent callbacks.
ignore_chain
Whether to ignore chain callbacks.
ignore_chat_model
Whether to ignore chat model callbacks.
ignore_llm
Whether to ignore LLM callbacks.
ignore_retriever
Whether to ignore retriever callbacks.
ignore_retry
Whether to ignore retry callbacks.


Whether to ignore retriever callbacks.
ignore_retry
Whether to ignore retry callbacks.
raise_error
run_inline
Methods
__init__(metrics[, implementation_name])
Initializes the deepevalCallbackHandler.
on_agent_action(action, **kwargs)
Do nothing when agent takes a specific action.
on_agent_finish(finish, **kwargs)
Do nothing
on_chain_end(outputs, **kwargs)
Do nothing when chain ends.
on_chain_error(error, **kwargs)
Do nothing when LLM chain outputs an error.
on_chain_start(serialized, inputs, **kwargs)
Do nothing when chain starts
on_chat_model_start(serialized, messages, *, ...)
Run when a chat model starts running.
on_llm_end(response, **kwargs)
Log records to deepeval when an LLM ends.
on_llm_error(error, **kwargs)
Do nothing when LLM outputs an error.
on_llm_new_token(token, **kwargs)
Do nothing when a new token is generated.
on_llm_start(serialized, prompts, **kwargs)
Store the prompts
on_retriever_end(documents, *, run_id[, ...])
Run when Retriever ends running.
on_retriever_error(error, *, run_id[, ...])
Run when Retriever errors.
on_retriever_start(serialized, query, *, run_id)
Run when Retriever starts running.
on_retry(retry_state, *, run_id[, parent_run_id])
Run on a retry event.
on_text(text, **kwargs)
Do nothing
on_tool_end(output[, observation_prefix, ...])
Do nothing when tool ends.
on_tool_error(error, **kwargs)


Do nothing when tool ends.
on_tool_error(error, **kwargs)
Do nothing when tool outputs an error.
on_tool_start(serialized, input_str, **kwargs)
Do nothing when tool starts.
__init__(metrics: List[Any], implementation_name: Optional[str] = None) → None[source]¶
Initializes the deepevalCallbackHandler.
Parameters
implementation_name – Name of the implementation you want.
metrics – What metrics do you want to track?
Raises
ImportError – if the deepeval package is not installed.
ConnectionError – if the connection to deepeval fails.
on_agent_action(action: AgentAction, **kwargs: Any) → Any[source]¶
Do nothing when agent takes a specific action.
on_agent_finish(finish: AgentFinish, **kwargs: Any) → None[source]¶
Do nothing
on_chain_end(outputs: Dict[str, Any], **kwargs: Any) → None[source]¶
Do nothing when chain ends.
on_chain_error(error: BaseException, **kwargs: Any) → None[source]¶
Do nothing when LLM chain outputs an error.
on_chain_start(serialized: Dict[str, Any], inputs: Dict[str, Any], **kwargs: Any) → None[source]¶
Do nothing when chain starts
on_chat_model_start(serialized: Dict[str, Any], messages: List[List[BaseMessage]], *, run_id: UUID, parent_run_id: Optional[UUID] = None, tags: Optional[List[str]] = None, metadata: Optional[Dict[str, Any]] = None, **kwargs: Any) → Any¶
Run when a chat model starts running.
on_llm_end(response: LLMResult, **kwargs: Any) → None[source]¶
Log records to deepeval when an LLM ends.


Log records to deepeval when an LLM ends.
on_llm_error(error: BaseException, **kwargs: Any) → None[source]¶
Do nothing when LLM outputs an error.
on_llm_new_token(token: str, **kwargs: Any) → None[source]¶
Do nothing when a new token is generated.
on_llm_start(serialized: Dict[str, Any], prompts: List[str], **kwargs: Any) → None[source]¶
Store the prompts
on_retriever_end(documents: Sequence[Document], *, run_id: UUID, parent_run_id: Optional[UUID] = None, **kwargs: Any) → Any¶
Run when Retriever ends running.
on_retriever_error(error: BaseException, *, run_id: UUID, parent_run_id: Optional[UUID] = None, **kwargs: Any) → Any¶
Run when Retriever errors.
on_retriever_start(serialized: Dict[str, Any], query: str, *, run_id: UUID, parent_run_id: Optional[UUID] = None, tags: Optional[List[str]] = None, metadata: Optional[Dict[str, Any]] = None, **kwargs: Any) → Any¶
Run when Retriever starts running.
on_retry(retry_state: RetryCallState, *, run_id: UUID, parent_run_id: Optional[UUID] = None, **kwargs: Any) → Any¶
Run on a retry event.
on_text(text: str, **kwargs: Any) → None[source]¶
Do nothing
on_tool_end(output: str, observation_prefix: Optional[str] = None, llm_prefix: Optional[str] = None, **kwargs: Any) → None[source]¶
Do nothing when tool ends.
on_tool_error(error: BaseException, **kwargs: Any) → None[source]¶


on_tool_error(error: BaseException, **kwargs: Any) → None[source]¶
Do nothing when tool outputs an error.
on_tool_start(serialized: Dict[str, Any], input_str: str, **kwargs: Any) → None[source]¶
Do nothing when tool starts.
Examples using DeepEvalCallbackHandler¶
Confident


langchain.callbacks.flyte_callback.FlyteCallbackHandler¶
class langchain.callbacks.flyte_callback.FlyteCallbackHandler[source]¶
This callback handler that is used within a Flyte task.
Initialize callback handler.
Attributes
always_verbose
Whether to call verbose callbacks even if verbose is False.
ignore_agent
Whether to ignore agent callbacks.
ignore_chain
Whether to ignore chain callbacks.
ignore_chat_model
Whether to ignore chat model callbacks.
ignore_llm
Whether to ignore LLM callbacks.
ignore_retriever
Whether to ignore retriever callbacks.
ignore_retry
Whether to ignore retry callbacks.
raise_error
run_inline
Methods
__init__()
Initialize callback handler.
get_custom_callback_meta()
on_agent_action(action, **kwargs)
Run on agent action.
on_agent_finish(finish, **kwargs)
Run when agent ends running.
on_chain_end(outputs, **kwargs)
Run when chain ends running.
on_chain_error(error, **kwargs)
Run when chain errors.
on_chain_start(serialized, inputs, **kwargs)
Run when chain starts running.
on_chat_model_start(serialized, messages, *, ...)
Run when a chat model starts running.
on_llm_end(response, **kwargs)
Run when LLM ends running.
on_llm_error(error, **kwargs)
Run when LLM errors.
on_llm_new_token(token, **kwargs)
Run when LLM generates a new token.
on_llm_start(serialized, prompts, **kwargs)
Run when LLM starts.
on_retriever_end(documents, *, run_id[, ...])
Run when Retriever ends running.
on_retriever_error(error, *, run_id[, ...])


on_retriever_error(error, *, run_id[, ...])
Run when Retriever errors.
on_retriever_start(serialized, query, *, run_id)
Run when Retriever starts running.
on_retry(retry_state, *, run_id[, parent_run_id])
Run on a retry event.
on_text(text, **kwargs)
Run when agent is ending.
on_tool_end(output, **kwargs)
Run when tool ends running.
on_tool_error(error, **kwargs)
Run when tool errors.
on_tool_start(serialized, input_str, **kwargs)
Run when tool starts running.
reset_callback_meta()
Reset the callback metadata.
__init__() → None[source]¶
Initialize callback handler.
get_custom_callback_meta() → Dict[str, Any]¶
on_agent_action(action: AgentAction, **kwargs: Any) → Any[source]¶
Run on agent action.
on_agent_finish(finish: AgentFinish, **kwargs: Any) → None[source]¶
Run when agent ends running.
on_chain_end(outputs: Dict[str, Any], **kwargs: Any) → None[source]¶
Run when chain ends running.
on_chain_error(error: BaseException, **kwargs: Any) → None[source]¶
Run when chain errors.
on_chain_start(serialized: Dict[str, Any], inputs: Dict[str, Any], **kwargs: Any) → None[source]¶
Run when chain starts running.
on_chat_model_start(serialized: Dict[str, Any], messages: List[List[BaseMessage]], *, run_id: UUID, parent_run_id: Optional[UUID] = None, tags: Optional[List[str]] = None, metadata: Optional[Dict[str, Any]] = None, **kwargs: Any) → Any¶


Run when a chat model starts running.
on_llm_end(response: LLMResult, **kwargs: Any) → None[source]¶
Run when LLM ends running.
on_llm_error(error: BaseException, **kwargs: Any) → None[source]¶
Run when LLM errors.
on_llm_new_token(token: str, **kwargs: Any) → None[source]¶
Run when LLM generates a new token.
on_llm_start(serialized: Dict[str, Any], prompts: List[str], **kwargs: Any) → None[source]¶
Run when LLM starts.
on_retriever_end(documents: Sequence[Document], *, run_id: UUID, parent_run_id: Optional[UUID] = None, **kwargs: Any) → Any¶
Run when Retriever ends running.
on_retriever_error(error: BaseException, *, run_id: UUID, parent_run_id: Optional[UUID] = None, **kwargs: Any) → Any¶
Run when Retriever errors.
on_retriever_start(serialized: Dict[str, Any], query: str, *, run_id: UUID, parent_run_id: Optional[UUID] = None, tags: Optional[List[str]] = None, metadata: Optional[Dict[str, Any]] = None, **kwargs: Any) → Any¶
Run when Retriever starts running.
on_retry(retry_state: RetryCallState, *, run_id: UUID, parent_run_id: Optional[UUID] = None, **kwargs: Any) → Any¶
Run on a retry event.
on_text(text: str, **kwargs: Any) → None[source]¶
Run when agent is ending.
on_tool_end(output: str, **kwargs: Any) → None[source]¶
Run when tool ends running.


Run when tool ends running.
on_tool_error(error: BaseException, **kwargs: Any) → None[source]¶
Run when tool errors.
on_tool_start(serialized: Dict[str, Any], input_str: str, **kwargs: Any) → None[source]¶
Run when tool starts running.
reset_callback_meta() → None¶
Reset the callback metadata.
Examples using FlyteCallbackHandler¶
Flyte


langchain.callbacks.streamlit.streamlit_callback_handler.ToolRecord¶
class langchain.callbacks.streamlit.streamlit_callback_handler.ToolRecord(name: str, input_str: str)[source]¶
The tool record as a NamedTuple.
Create new instance of ToolRecord(name, input_str)
Attributes
input_str
Alias for field number 1
name
Alias for field number 0
Methods
__init__()
count(value, /)
Return number of occurrences of value.
index(value[, start, stop])
Return first index of value.
__init__()¶
count(value, /)¶
Return number of occurrences of value.
index(value, start=0, stop=9223372036854775807, /)¶
Return first index of value.
Raises ValueError if the value is not present.


langchain.callbacks.whylabs_callback.WhyLabsCallbackHandler¶
class langchain.callbacks.whylabs_callback.WhyLabsCallbackHandler(logger: Logger, handler: Any)[source]¶
Callback Handler for logging to WhyLabs. This callback handler utilizes
langkit to extract features from the prompts & responses when interacting with
an LLM. These features can be used to guardrail, evaluate, and observe interactions
over time to detect issues relating to hallucinations, prompt engineering,
or output validation. LangKit is an LLM monitoring toolkit developed by WhyLabs.
Here are some examples of what can be monitored with LangKit:
* Text Quality
readability score
complexity and grade scores
Text Relevance
- Similarity scores between prompt/responses
- Similarity scores against user-defined themes
- Topic classification
Security and Privacy
- patterns - count of strings matching a user-defined regex pattern group
- jailbreaks - similarity scores with respect to known jailbreak attempts
- prompt injection - similarity scores with respect to known prompt attacks
- refusals - similarity scores with respect to known LLM refusal responses
Sentiment and Toxicity
- sentiment analysis
- toxicity analysis
For more information, see https://docs.whylabs.ai/docs/language-model-monitoring
or check out the LangKit repo here: https://github.com/whylabs/langkit
—
:param api_key: WhyLabs API key. Optional because the preferred
way to specify the API key is with environment variable
WHYLABS_API_KEY.
Parameters
org_id (Optional[str]) – WhyLabs organization id to write profiles to.
Optional because the preferred way to specify the organization id is
with environment variable WHYLABS_DEFAULT_ORG_ID.
dataset_id (Optional[str]) – WhyLabs dataset id to write profiles to.
Optional because the preferred way to specify the dataset id is


Optional because the preferred way to specify the dataset id is
with environment variable WHYLABS_DEFAULT_DATASET_ID.
sentiment (bool) – Whether to enable sentiment analysis. Defaults to False.
toxicity (bool) – Whether to enable toxicity analysis. Defaults to False.
themes (bool) – Whether to enable theme analysis. Defaults to False.
Initiate the rolling logger.
Attributes
ignore_agent
Whether to ignore agent callbacks.
ignore_chain
Whether to ignore chain callbacks.
ignore_chat_model
Whether to ignore chat model callbacks.
ignore_llm
Whether to ignore LLM callbacks.
ignore_retriever
Whether to ignore retriever callbacks.
ignore_retry
Whether to ignore retry callbacks.
raise_error
run_inline
Methods
__init__(logger, handler)
Initiate the rolling logger.
close()
Close any loggers to allow writing out of any profiles before exiting.
flush()
Explicitly write current profile if using a rolling logger.
from_params(*[, api_key, org_id, ...])
Instantiate whylogs Logger from params.
on_agent_action(action, *, run_id[, ...])
Run on agent action.
on_agent_finish(finish, *, run_id[, ...])
Run on agent end.
on_chain_end(outputs, *, run_id[, parent_run_id])
Run when chain ends running.
on_chain_error(error, *, run_id[, parent_run_id])
Run when chain errors.
on_chain_start(serialized, inputs, *, run_id)
Run when chain starts running.
on_chat_model_start(serialized, messages, *, ...)
Run when a chat model starts running.
on_llm_end(response, *, run_id[, parent_run_id])
Run when LLM ends running.


Run when LLM ends running.
on_llm_error(error, *, run_id[, parent_run_id])
Run when LLM errors.
on_llm_new_token(token, *[, chunk, ...])
Run on new LLM token.
on_llm_start(serialized, prompts, *, run_id)
Run when LLM starts running.
on_retriever_end(documents, *, run_id[, ...])
Run when Retriever ends running.
on_retriever_error(error, *, run_id[, ...])
Run when Retriever errors.
on_retriever_start(serialized, query, *, run_id)
Run when Retriever starts running.
on_retry(retry_state, *, run_id[, parent_run_id])
Run on a retry event.
on_text(text, *, run_id[, parent_run_id])
Run on arbitrary text.
on_tool_end(output, *, run_id[, parent_run_id])
Run when tool ends running.
on_tool_error(error, *, run_id[, parent_run_id])
Run when tool errors.
on_tool_start(serialized, input_str, *, run_id)
Run when tool starts running.
__init__(logger: Logger, handler: Any)[source]¶
Initiate the rolling logger.
close() → None[source]¶
Close any loggers to allow writing out of any profiles before exiting.
flush() → None[source]¶
Explicitly write current profile if using a rolling logger.


flush() → None[source]¶
Explicitly write current profile if using a rolling logger.
classmethod from_params(*, api_key: Optional[str] = None, org_id: Optional[str] = None, dataset_id: Optional[str] = None, sentiment: bool = False, toxicity: bool = False, themes: bool = False, logger: Optional[Logger] = None) → WhyLabsCallbackHandler[source]¶
Instantiate whylogs Logger from params.
Parameters
api_key (Optional[str]) – WhyLabs API key. Optional because the preferred
way to specify the API key is with environment variable
WHYLABS_API_KEY.
org_id (Optional[str]) – WhyLabs organization id to write profiles to.
If not set must be specified in environment variable
WHYLABS_DEFAULT_ORG_ID.
dataset_id (Optional[str]) – The model or dataset this callback is gathering
telemetry for. If not set must be specified in environment variable
WHYLABS_DEFAULT_DATASET_ID.
sentiment (bool) – If True will initialize a model to perform
sentiment analysis compound score. Defaults to False and will not gather
this metric.
toxicity (bool) – If True will initialize a model to score
toxicity. Defaults to False and will not gather this metric.
themes (bool) – If True will initialize a model to calculate
distance to configured themes. Defaults to None and will not gather this
metric.
logger (Optional[Logger]) – If specified will bind the configured logger as
the telemetry gathering agent. Defaults to LangKit schema with periodic
WhyLabs writer.
on_agent_action(action: AgentAction, *, run_id: UUID, parent_run_id: Optional[UUID] = None, **kwargs: Any) → Any¶
Run on agent action.


Run on agent action.
on_agent_finish(finish: AgentFinish, *, run_id: UUID, parent_run_id: Optional[UUID] = None, **kwargs: Any) → Any¶
Run on agent end.
on_chain_end(outputs: Dict[str, Any], *, run_id: UUID, parent_run_id: Optional[UUID] = None, **kwargs: Any) → Any¶
Run when chain ends running.
on_chain_error(error: BaseException, *, run_id: UUID, parent_run_id: Optional[UUID] = None, **kwargs: Any) → Any¶
Run when chain errors.
on_chain_start(serialized: Dict[str, Any], inputs: Dict[str, Any], *, run_id: UUID, parent_run_id: Optional[UUID] = None, tags: Optional[List[str]] = None, metadata: Optional[Dict[str, Any]] = None, **kwargs: Any) → Any¶
Run when chain starts running.
on_chat_model_start(serialized: Dict[str, Any], messages: List[List[BaseMessage]], *, run_id: UUID, parent_run_id: Optional[UUID] = None, tags: Optional[List[str]] = None, metadata: Optional[Dict[str, Any]] = None, **kwargs: Any) → Any¶
Run when a chat model starts running.
on_llm_end(response: LLMResult, *, run_id: UUID, parent_run_id: Optional[UUID] = None, **kwargs: Any) → Any¶
Run when LLM ends running.
on_llm_error(error: BaseException, *, run_id: UUID, parent_run_id: Optional[UUID] = None, **kwargs: Any) → Any¶
Run when LLM errors.


Run when LLM errors.
on_llm_new_token(token: str, *, chunk: Optional[Union[GenerationChunk, ChatGenerationChunk]] = None, run_id: UUID, parent_run_id: Optional[UUID] = None, **kwargs: Any) → Any¶
Run on new LLM token. Only available when streaming is enabled.
Parameters
token (str) – The new token.
chunk (GenerationChunk | ChatGenerationChunk) – The new generated chunk,
information. (containing content and other) –
on_llm_start(serialized: Dict[str, Any], prompts: List[str], *, run_id: UUID, parent_run_id: Optional[UUID] = None, tags: Optional[List[str]] = None, metadata: Optional[Dict[str, Any]] = None, **kwargs: Any) → Any¶
Run when LLM starts running.
on_retriever_end(documents: Sequence[Document], *, run_id: UUID, parent_run_id: Optional[UUID] = None, **kwargs: Any) → Any¶
Run when Retriever ends running.
on_retriever_error(error: BaseException, *, run_id: UUID, parent_run_id: Optional[UUID] = None, **kwargs: Any) → Any¶
Run when Retriever errors.
on_retriever_start(serialized: Dict[str, Any], query: str, *, run_id: UUID, parent_run_id: Optional[UUID] = None, tags: Optional[List[str]] = None, metadata: Optional[Dict[str, Any]] = None, **kwargs: Any) → Any¶
Run when Retriever starts running.
on_retry(retry_state: RetryCallState, *, run_id: UUID, parent_run_id: Optional[UUID] = None, **kwargs: Any) → Any¶
Run on a retry event.


Run on a retry event.
on_text(text: str, *, run_id: UUID, parent_run_id: Optional[UUID] = None, **kwargs: Any) → Any¶
Run on arbitrary text.
on_tool_end(output: str, *, run_id: UUID, parent_run_id: Optional[UUID] = None, **kwargs: Any) → Any¶
Run when tool ends running.
on_tool_error(error: BaseException, *, run_id: UUID, parent_run_id: Optional[UUID] = None, **kwargs: Any) → Any¶
Run when tool errors.
on_tool_start(serialized: Dict[str, Any], input_str: str, *, run_id: UUID, parent_run_id: Optional[UUID] = None, tags: Optional[List[str]] = None, metadata: Optional[Dict[str, Any]] = None, **kwargs: Any) → Any¶
Run when tool starts running.
Examples using WhyLabsCallbackHandler¶
WhyLabs


langchain.callbacks.arize_callback.ArizeCallbackHandler¶
class langchain.callbacks.arize_callback.ArizeCallbackHandler(model_id: Optional[str] = None, model_version: Optional[str] = None, SPACE_KEY: Optional[str] = None, API_KEY: Optional[str] = None)[source]¶
Callback Handler that logs to Arize.
Initialize callback handler.
Attributes
ignore_agent
Whether to ignore agent callbacks.
ignore_chain
Whether to ignore chain callbacks.
ignore_chat_model
Whether to ignore chat model callbacks.
ignore_llm
Whether to ignore LLM callbacks.
ignore_retriever
Whether to ignore retriever callbacks.
ignore_retry
Whether to ignore retry callbacks.
raise_error
run_inline
Methods
__init__([model_id, model_version, ...])
Initialize callback handler.
on_agent_action(action, **kwargs)
Do nothing.
on_agent_finish(finish, **kwargs)
Run on agent end.
on_chain_end(outputs, **kwargs)
Do nothing.
on_chain_error(error, **kwargs)
Do nothing.
on_chain_start(serialized, inputs, **kwargs)
Run when chain starts running.
on_chat_model_start(serialized, messages, *, ...)
Run when a chat model starts running.
on_llm_end(response, **kwargs)
Run when LLM ends running.
on_llm_error(error, **kwargs)
Do nothing.
on_llm_new_token(token, **kwargs)
Do nothing.
on_llm_start(serialized, prompts, **kwargs)
Run when LLM starts running.
on_retriever_end(documents, *, run_id[, ...])
Run when Retriever ends running.
on_retriever_error(error, *, run_id[, ...])


on_retriever_error(error, *, run_id[, ...])
Run when Retriever errors.
on_retriever_start(serialized, query, *, run_id)
Run when Retriever starts running.
on_retry(retry_state, *, run_id[, parent_run_id])
Run on a retry event.
on_text(text, **kwargs)
Run on arbitrary text.
on_tool_end(output[, observation_prefix, ...])
Run when tool ends running.
on_tool_error(error, **kwargs)
Run when tool errors.
on_tool_start(serialized, input_str, **kwargs)
Run when tool starts running.
__init__(model_id: Optional[str] = None, model_version: Optional[str] = None, SPACE_KEY: Optional[str] = None, API_KEY: Optional[str] = None) → None[source]¶
Initialize callback handler.
on_agent_action(action: AgentAction, **kwargs: Any) → Any[source]¶
Do nothing.
on_agent_finish(finish: AgentFinish, **kwargs: Any) → None[source]¶
Run on agent end.
on_chain_end(outputs: Dict[str, Any], **kwargs: Any) → None[source]¶
Do nothing.
on_chain_error(error: BaseException, **kwargs: Any) → None[source]¶
Do nothing.
on_chain_start(serialized: Dict[str, Any], inputs: Dict[str, Any], **kwargs: Any) → None[source]¶
Run when chain starts running.
on_chat_model_start(serialized: Dict[str, Any], messages: List[List[BaseMessage]], *, run_id: UUID, parent_run_id: Optional[UUID] = None, tags: Optional[List[str]] = None, metadata: Optional[Dict[str, Any]] = None, **kwargs: Any) → Any¶


Run when a chat model starts running.
on_llm_end(response: LLMResult, **kwargs: Any) → None[source]¶
Run when LLM ends running.
on_llm_error(error: BaseException, **kwargs: Any) → None[source]¶
Do nothing.
on_llm_new_token(token: str, **kwargs: Any) → None[source]¶
Do nothing.
on_llm_start(serialized: Dict[str, Any], prompts: List[str], **kwargs: Any) → None[source]¶
Run when LLM starts running.
on_retriever_end(documents: Sequence[Document], *, run_id: UUID, parent_run_id: Optional[UUID] = None, **kwargs: Any) → Any¶
Run when Retriever ends running.
on_retriever_error(error: BaseException, *, run_id: UUID, parent_run_id: Optional[UUID] = None, **kwargs: Any) → Any¶
Run when Retriever errors.
on_retriever_start(serialized: Dict[str, Any], query: str, *, run_id: UUID, parent_run_id: Optional[UUID] = None, tags: Optional[List[str]] = None, metadata: Optional[Dict[str, Any]] = None, **kwargs: Any) → Any¶
Run when Retriever starts running.
on_retry(retry_state: RetryCallState, *, run_id: UUID, parent_run_id: Optional[UUID] = None, **kwargs: Any) → Any¶
Run on a retry event.
on_text(text: str, **kwargs: Any) → None[source]¶
Run on arbitrary text.
on_tool_end(output: str, observation_prefix: Optional[str] = None, llm_prefix: Optional[str] = None, **kwargs: Any) → None[source]¶
Run when tool ends running.


Run when tool ends running.
on_tool_error(error: BaseException, **kwargs: Any) → None[source]¶
Run when tool errors.
on_tool_start(serialized: Dict[str, Any], input_str: str, **kwargs: Any) → None[source]¶
Run when tool starts running.


langchain.callbacks.utils.flatten_dict¶
langchain.callbacks.utils.flatten_dict(nested_dict: Dict[str, Any], parent_key: str = '', sep: str = '_') → Dict[str, Any][source]¶
Flattens a nested dictionary into a flat dictionary.
Parameters
nested_dict (dict) – The nested dictionary to flatten.
parent_key (str) – The prefix to prepend to the keys of the flattened dict.
sep (str) – The separator to use between the parent key and the key of the
flattened dictionary.
Returns
A flat dictionary.
Return type
(dict)


langchain.callbacks.wandb_callback.load_json_to_dict¶
langchain.callbacks.wandb_callback.load_json_to_dict(json_path: Union[str, Path]) → dict[source]¶
Load json file to a dictionary.
Parameters
json_path (str) – The path to the json file.
Returns
The dictionary representation of the json file.
Return type
(dict)


langchain.callbacks.sagemaker_callback.save_json¶
langchain.callbacks.sagemaker_callback.save_json(data: dict, file_path: str) → None[source]¶
Save dict to local file path.
Parameters
data (dict) – The dictionary to be saved.
file_path (str) – Local file path.


langchain.callbacks.labelstudio_callback.get_default_label_configs¶
langchain.callbacks.labelstudio_callback.get_default_label_configs(mode: Union[str, LabelStudioMode]) → Tuple[str, LabelStudioMode][source]¶
Get default Label Studio configs for the given mode.
Parameters
mode – Label Studio mode (“prompt” or “chat”)
Returns: Tuple of Label Studio config and mode


langchain.callbacks.streamlit.mutable_expander.ChildRecord¶
class langchain.callbacks.streamlit.mutable_expander.ChildRecord(type: ChildType, kwargs: Dict[str, Any], dg: DeltaGenerator)[source]¶
The child record as a NamedTuple.
Create new instance of ChildRecord(type, kwargs, dg)
Attributes
dg
Alias for field number 2
kwargs
Alias for field number 1
type
Alias for field number 0
Methods
__init__()
count(value, /)
Return number of occurrences of value.
index(value[, start, stop])
Return first index of value.
__init__()¶
count(value, /)¶
Return number of occurrences of value.
index(value, start=0, stop=9223372036854775807, /)¶
Return first index of value.
Raises ValueError if the value is not present.


langchain.callbacks.flyte_callback.analyze_text¶
langchain.callbacks.flyte_callback.analyze_text(text: str, nlp: Any = None, textstat: Any = None) → dict[source]¶
Analyze text using textstat and spacy.
Parameters
text (str) – The text to analyze.
nlp (spacy.lang) – The spacy language model to use for visualization.
Returns
A dictionary containing the complexity metrics and visualizationfiles serialized to HTML string.
Return type
(dict)


langchain.callbacks.mlflow_callback.analyze_text¶
langchain.callbacks.mlflow_callback.analyze_text(text: str, nlp: Any = None) → dict[source]¶
Analyze text using textstat and spacy.
Parameters
text (str) – The text to analyze.
nlp (spacy.lang) – The spacy language model to use for visualization.
Returns
A dictionary containing the complexity metrics and visualizationfiles serialized to  HTML string.
Return type
(dict)


langchain.callbacks.streamlit.streamlit_callback_handler.LLMThought¶
class langchain.callbacks.streamlit.streamlit_callback_handler.LLMThought(parent_container: DeltaGenerator, labeler: LLMThoughtLabeler, expanded: bool, collapse_on_complete: bool)[source]¶
A thought in the LLM’s thought stream.
Initialize the LLMThought.
Parameters
parent_container – The container we’re writing into.
labeler – The labeler to use for this thought.
expanded – Whether the thought should be expanded by default.
collapse_on_complete – Whether the thought should be collapsed.
Attributes
container
The container we're writing into.
last_tool
The last tool executed by this thought
Methods
__init__(parent_container, labeler, ...)
Initialize the LLMThought.
clear()
Remove the thought from the screen.
complete([final_label])
Finish the thought.
on_agent_action(action[, color])
on_llm_end(response, **kwargs)
on_llm_error(error, **kwargs)
on_llm_new_token(token, **kwargs)
on_llm_start(serialized, prompts)
on_tool_end(output[, color, ...])
on_tool_error(error, **kwargs)
on_tool_start(serialized, input_str, **kwargs)
__init__(parent_container: DeltaGenerator, labeler: LLMThoughtLabeler, expanded: bool, collapse_on_complete: bool)[source]¶
Initialize the LLMThought.
Parameters
parent_container – The container we’re writing into.
labeler – The labeler to use for this thought.
expanded – Whether the thought should be expanded by default.
collapse_on_complete – Whether the thought should be collapsed.
clear() → None[source]¶
Remove the thought from the screen. A cleared thought can’t be reused.


Remove the thought from the screen. A cleared thought can’t be reused.
complete(final_label: Optional[str] = None) → None[source]¶
Finish the thought.
on_agent_action(action: AgentAction, color: Optional[str] = None, **kwargs: Any) → Any[source]¶
on_llm_end(response: LLMResult, **kwargs: Any) → None[source]¶
on_llm_error(error: BaseException, **kwargs: Any) → None[source]¶
on_llm_new_token(token: str, **kwargs: Any) → None[source]¶
on_llm_start(serialized: Dict[str, Any], prompts: List[str]) → None[source]¶
on_tool_end(output: str, color: Optional[str] = None, observation_prefix: Optional[str] = None, llm_prefix: Optional[str] = None, **kwargs: Any) → None[source]¶
on_tool_error(error: BaseException, **kwargs: Any) → None[source]¶
on_tool_start(serialized: Dict[str, Any], input_str: str, **kwargs: Any) → None[source]¶


langchain.callbacks.wandb_callback.construct_html_from_prompt_and_generation¶
langchain.callbacks.wandb_callback.construct_html_from_prompt_and_generation(prompt: str, generation: str) → Any[source]¶
Construct an html element from a prompt and a generation.
Parameters
prompt (str) – The prompt.
generation (str) – The generation.
Returns
The html element.
Return type
(wandb.Html)


langchain.callbacks.clearml_callback.ClearMLCallbackHandler¶
class langchain.callbacks.clearml_callback.ClearMLCallbackHandler(task_type: Optional[str] = 'inference', project_name: Optional[str] = 'langchain_callback_demo', tags: Optional[Sequence] = None, task_name: Optional[str] = None, visualize: bool = False, complexity_metrics: bool = False, stream_logs: bool = False)[source]¶
Callback Handler that logs to ClearML.
Parameters
job_type (str) – The type of clearml task such as “inference”, “testing” or “qc”
project_name (str) – The clearml project name
tags (list) – Tags to add to the task
task_name (str) – Name of the clearml task
visualize (bool) – Whether to visualize the run.
complexity_metrics (bool) – Whether to log complexity metrics
stream_logs (bool) – Whether to stream callback actions to ClearML
This handler will utilize the associated callback method and formats
the input of each callback function with metadata regarding the state of LLM run,
and adds the response to the list of records for both the {method}_records and
action. It then logs the response to the ClearML console.
Initialize callback handler.
Attributes
always_verbose
Whether to call verbose callbacks even if verbose is False.
ignore_agent
Whether to ignore agent callbacks.
ignore_chain
Whether to ignore chain callbacks.
ignore_chat_model
Whether to ignore chat model callbacks.
ignore_llm
Whether to ignore LLM callbacks.
ignore_retriever
Whether to ignore retriever callbacks.
ignore_retry
Whether to ignore retry callbacks.
raise_error
run_inline
Methods
__init__([task_type, project_name, tags, ...])
Initialize callback handler.
analyze_text(text)
Analyze text using textstat and spacy.


analyze_text(text)
Analyze text using textstat and spacy.
flush_tracker([name, langchain_asset, finish])
Flush the tracker and setup the session.
get_custom_callback_meta()
on_agent_action(action, **kwargs)
Run on agent action.
on_agent_finish(finish, **kwargs)
Run when agent ends running.
on_chain_end(outputs, **kwargs)
Run when chain ends running.
on_chain_error(error, **kwargs)
Run when chain errors.
on_chain_start(serialized, inputs, **kwargs)
Run when chain starts running.
on_chat_model_start(serialized, messages, *, ...)
Run when a chat model starts running.
on_llm_end(response, **kwargs)
Run when LLM ends running.
on_llm_error(error, **kwargs)
Run when LLM errors.
on_llm_new_token(token, **kwargs)
Run when LLM generates a new token.
on_llm_start(serialized, prompts, **kwargs)
Run when LLM starts.
on_retriever_end(documents, *, run_id[, ...])
Run when Retriever ends running.
on_retriever_error(error, *, run_id[, ...])
Run when Retriever errors.
on_retriever_start(serialized, query, *, run_id)
Run when Retriever starts running.
on_retry(retry_state, *, run_id[, parent_run_id])
Run on a retry event.
on_text(text, **kwargs)
Run when agent is ending.
on_tool_end(output, **kwargs)
Run when tool ends running.
on_tool_error(error, **kwargs)
Run when tool errors.


on_tool_error(error, **kwargs)
Run when tool errors.
on_tool_start(serialized, input_str, **kwargs)
Run when tool starts running.
reset_callback_meta()
Reset the callback metadata.
__init__(task_type: Optional[str] = 'inference', project_name: Optional[str] = 'langchain_callback_demo', tags: Optional[Sequence] = None, task_name: Optional[str] = None, visualize: bool = False, complexity_metrics: bool = False, stream_logs: bool = False) → None[source]¶
Initialize callback handler.
analyze_text(text: str) → dict[source]¶
Analyze text using textstat and spacy.
Parameters
text (str) – The text to analyze.
Returns
A dictionary containing the complexity metrics.
Return type
(dict)
flush_tracker(name: Optional[str] = None, langchain_asset: Any = None, finish: bool = False) → None[source]¶
Flush the tracker and setup the session.
Everything after this will be a new table.
Parameters
name – Name of the performed session so far so it is identifiable
langchain_asset – The langchain asset to save.
finish – Whether to finish the run.
Returns – None
get_custom_callback_meta() → Dict[str, Any]¶
on_agent_action(action: AgentAction, **kwargs: Any) → Any[source]¶
Run on agent action.
on_agent_finish(finish: AgentFinish, **kwargs: Any) → None[source]¶
Run when agent ends running.
on_chain_end(outputs: Dict[str, Any], **kwargs: Any) → None[source]¶
Run when chain ends running.
on_chain_error(error: BaseException, **kwargs: Any) → None[source]¶
Run when chain errors.


Run when chain errors.
on_chain_start(serialized: Dict[str, Any], inputs: Dict[str, Any], **kwargs: Any) → None[source]¶
Run when chain starts running.
on_chat_model_start(serialized: Dict[str, Any], messages: List[List[BaseMessage]], *, run_id: UUID, parent_run_id: Optional[UUID] = None, tags: Optional[List[str]] = None, metadata: Optional[Dict[str, Any]] = None, **kwargs: Any) → Any¶
Run when a chat model starts running.
on_llm_end(response: LLMResult, **kwargs: Any) → None[source]¶
Run when LLM ends running.
on_llm_error(error: BaseException, **kwargs: Any) → None[source]¶
Run when LLM errors.
on_llm_new_token(token: str, **kwargs: Any) → None[source]¶
Run when LLM generates a new token.
on_llm_start(serialized: Dict[str, Any], prompts: List[str], **kwargs: Any) → None[source]¶
Run when LLM starts.
on_retriever_end(documents: Sequence[Document], *, run_id: UUID, parent_run_id: Optional[UUID] = None, **kwargs: Any) → Any¶
Run when Retriever ends running.
on_retriever_error(error: BaseException, *, run_id: UUID, parent_run_id: Optional[UUID] = None, **kwargs: Any) → Any¶
Run when Retriever errors.
on_retriever_start(serialized: Dict[str, Any], query: str, *, run_id: UUID, parent_run_id: Optional[UUID] = None, tags: Optional[List[str]] = None, metadata: Optional[Dict[str, Any]] = None, **kwargs: Any) → Any¶


Run when Retriever starts running.
on_retry(retry_state: RetryCallState, *, run_id: UUID, parent_run_id: Optional[UUID] = None, **kwargs: Any) → Any¶
Run on a retry event.
on_text(text: str, **kwargs: Any) → None[source]¶
Run when agent is ending.
on_tool_end(output: str, **kwargs: Any) → None[source]¶
Run when tool ends running.
on_tool_error(error: BaseException, **kwargs: Any) → None[source]¶
Run when tool errors.
on_tool_start(serialized: Dict[str, Any], input_str: str, **kwargs: Any) → None[source]¶
Run when tool starts running.
reset_callback_meta() → None¶
Reset the callback metadata.
Examples using ClearMLCallbackHandler¶
ClearML


Source code for typing
"""
The typing module: Support for gradual typing as defined by PEP 484 and subsequent PEPs.
Among other things, the module includes the following:
* Generic, Protocol, and internal machinery to support generic aliases.
  All subscripted types like X[int], Union[int, str] are generic aliases.
* Various "special forms" that have unique meanings in type annotations:
  NoReturn, Never, ClassVar, Self, Concatenate, Unpack, and others.
* Classes whose instances can be type arguments to generic classes and functions:
  TypeVar, ParamSpec, TypeVarTuple.
* Public helper functions: get_type_hints, overload, cast, final, and others.
* Several protocols to support duck-typing:
  SupportsFloat, SupportsIndex, SupportsAbs, and others.
* Special types: NewType, NamedTuple, TypedDict.
* Deprecated wrapper submodules for re and io related types.
* Deprecated aliases for builtin types and collections.abc ABCs.
Any name not present in __all__ is an implementation detail
that may be changed without notice. Use at your own risk!
"""
from abc import abstractmethod, ABCMeta
import collections
from collections import defaultdict
import collections.abc
import contextlib
import functools
import operator
import re as stdlib_re  # Avoid confusion with the re we export.
import sys
import types
import warnings
from types import WrapperDescriptorType, MethodWrapperType, MethodDescriptorType, GenericAlias
try:
    from _typing import _idfunc
except ImportError:
    def _idfunc(_, x):
        return x
# Please keep __all__ alphabetized within each category.
__all__ = [
    # Super-special typing primitives.
    'Annotated',
    'Any',
    'Callable',


'Annotated',
    'Any',
    'Callable',
    'ClassVar',
    'Concatenate',
    'Final',
    'ForwardRef',
    'Generic',
    'Literal',
    'Optional',
    'ParamSpec',
    'Protocol',
    'Tuple',
    'Type',
    'TypeVar',
    'TypeVarTuple',
    'Union',
    # ABCs (from collections.abc).
    'AbstractSet',  # collections.abc.Set.
    'ByteString',
    'Container',
    'ContextManager',
    'Hashable',
    'ItemsView',
    'Iterable',
    'Iterator',
    'KeysView',
    'Mapping',
    'MappingView',
    'MutableMapping',
    'MutableSequence',
    'MutableSet',
    'Sequence',
    'Sized',
    'ValuesView',
    'Awaitable',
    'AsyncIterator',
    'AsyncIterable',
    'Coroutine',
    'Collection',
    'AsyncGenerator',
    'AsyncContextManager',
    # Structural checks, a.k.a. protocols.
    'Reversible',
    'SupportsAbs',
    'SupportsBytes',
    'SupportsComplex',
    'SupportsFloat',
    'SupportsIndex',
    'SupportsInt',
    'SupportsRound',
    # Concrete collection types.
    'ChainMap',
    'Counter',
    'Deque',
    'Dict',
    'DefaultDict',
    'List',
    'OrderedDict',
    'Set',
    'FrozenSet',
    'NamedTuple',  # Not really a type.
    'TypedDict',  # Not really a type.


'TypedDict',  # Not really a type.
    'Generator',
    # Other concrete types.
    'BinaryIO',
    'IO',
    'Match',
    'Pattern',
    'TextIO',
    # One-off things.
    'AnyStr',
    'assert_type',
    'assert_never',
    'cast',
    'clear_overloads',
    'dataclass_transform',
    'final',
    'get_args',
    'get_origin',
    'get_overloads',
    'get_type_hints',
    'is_typeddict',
    'LiteralString',
    'Never',
    'NewType',
    'no_type_check',
    'no_type_check_decorator',
    'NoReturn',
    'NotRequired',
    'overload',
    'ParamSpecArgs',
    'ParamSpecKwargs',
    'Required',
    'reveal_type',
    'runtime_checkable',
    'Self',
    'Text',
    'TYPE_CHECKING',
    'TypeAlias',
    'TypeGuard',
    'Unpack',
]
# The pseudo-submodules 're' and 'io' are part of the public
# namespace, but excluded from __all__ because they might stomp on
# legitimate imports of those modules.
def _type_convert(arg, module=None, *, allow_special_forms=False):
    """For converting None to type(None), and strings to ForwardRef."""
    if arg is None:
        return type(None)
    if isinstance(arg, str):
        return ForwardRef(arg, module=module, is_class=allow_special_forms)
    return arg
def _type_check(arg, msg, is_argument=True, module=None, *, allow_special_forms=False):


"""Check that the argument is a type, and return it (internal helper).
    As a special case, accept None and return type(None) instead. Also wrap strings
    into ForwardRef instances. Consider several corner cases, for example plain
    special forms like Union are not valid, while Union[int, str] is OK, etc.
    The msg argument is a human-readable error message, e.g.::
        "Union[arg, ...]: arg should be a type."
    We append the repr() of the actual value (truncated to 100 chars).
    """
    invalid_generic_forms = (Generic, Protocol)
    if not allow_special_forms:
        invalid_generic_forms += (ClassVar,)
        if is_argument:
            invalid_generic_forms += (Final,)
    arg = _type_convert(arg, module=module, allow_special_forms=allow_special_forms)
    if (isinstance(arg, _GenericAlias) and
            arg.__origin__ in invalid_generic_forms):
        raise TypeError(f"{arg} is not valid as type argument")
    if arg in (Any, LiteralString, NoReturn, Never, Self, TypeAlias):
        return arg
    if allow_special_forms and arg in (ClassVar, Final):
        return arg
    if isinstance(arg, _SpecialForm) or arg in (Generic, Protocol):
        raise TypeError(f"Plain {arg} is not valid as type argument")
    if type(arg) is tuple:
        raise TypeError(f"{msg} Got {arg!r:.100}.")
    return arg
def _is_param_expr(arg):
    return arg is ... or isinstance(arg,
            (tuple, list, ParamSpec, _ConcatenateGenericAlias))
def _should_unflatten_callable_args(typ, args):


def _should_unflatten_callable_args(typ, args):
    """Internal helper for munging collections.abc.Callable's __args__.
    The canonical representation for a Callable's __args__ flattens the
    argument types, see https://github.com/python/cpython/issues/86361.
    For example::
        assert collections.abc.Callable[[int, int], str].__args__ == (int, int, str)
        assert collections.abc.Callable[ParamSpec, str].__args__ == (ParamSpec, str)
    As a result, if we need to reconstruct the Callable from its __args__,
    we need to unflatten it.
    """
    return (
        typ.__origin__ is collections.abc.Callable
        and not (len(args) == 2 and _is_param_expr(args[0]))
    )
def _type_repr(obj):
    """Return the repr() of an object, special-casing types (internal helper).
    If obj is a type, we return a shorter version than the default
    type.__repr__, based on the module and qualified name, which is
    typically enough to uniquely identify a type.  For everything
    else, we fall back on repr(obj).
    """
    if isinstance(obj, types.GenericAlias):
        return repr(obj)
    if isinstance(obj, type):
        if obj.__module__ == 'builtins':
            return obj.__qualname__
        return f'{obj.__module__}.{obj.__qualname__}'
    if obj is ...:
        return('...')
    if isinstance(obj, types.FunctionType):
        return obj.__name__
    return repr(obj)
def _collect_parameters(args):
    """Collect all type variables and parameter specifications in args
    in order of first appearance (lexicographic order).


in order of first appearance (lexicographic order).
    For example::
        assert _collect_parameters((T, Callable[P, T])) == (T, P)
    """
    parameters = []
    for t in args:
        if isinstance(t, type):
            # We don't want __parameters__ descriptor of a bare Python class.
            pass
        elif isinstance(t, tuple):
            # `t` might be a tuple, when `ParamSpec` is substituted with
            # `[T, int]`, or `[int, *Ts]`, etc.
            for x in t:
                for collected in _collect_parameters([x]):
                    if collected not in parameters:
                        parameters.append(collected)
        elif hasattr(t, '__typing_subst__'):
            if t not in parameters:
                parameters.append(t)
        else:
            for x in getattr(t, '__parameters__', ()):
                if x not in parameters:
                    parameters.append(x)
    return tuple(parameters)
def _check_generic(cls, parameters, elen):
    """Check correct count for parameters of a generic cls (internal helper).
    This gives a nice error message in case of count mismatch.
    """
    if not elen:
        raise TypeError(f"{cls} is not a generic class")
    alen = len(parameters)
    if alen != elen:
        raise TypeError(f"Too {'many' if alen > elen else 'few'} arguments for {cls};"
                        f" actual {alen}, expected {elen}")
def _unpack_args(args):
    newargs = []
    for arg in args:
        subargs = getattr(arg, '__typing_unpacked_tuple_args__', None)


subargs = getattr(arg, '__typing_unpacked_tuple_args__', None)
        if subargs is not None and not (subargs and subargs[-1] is ...):
            newargs.extend(subargs)
        else:
            newargs.append(arg)
    return newargs
def _deduplicate(params):
    # Weed out strict duplicates, preserving the first of each occurrence.
    all_params = set(params)
    if len(all_params) < len(params):
        new_params = []
        for t in params:
            if t in all_params:
                new_params.append(t)
                all_params.remove(t)
        params = new_params
        assert not all_params, all_params
    return params
def _remove_dups_flatten(parameters):
    """Internal helper for Union creation and substitution.
    Flatten Unions among parameters, then remove duplicates.
    """
    # Flatten out Union[Union[...], ...].
    params = []
    for p in parameters:
        if isinstance(p, (_UnionGenericAlias, types.UnionType)):
            params.extend(p.__args__)
        else:
            params.append(p)
    return tuple(_deduplicate(params))
def _flatten_literal_params(parameters):
    """Internal helper for Literal creation: flatten Literals among parameters."""
    params = []
    for p in parameters:
        if isinstance(p, _LiteralGenericAlias):
            params.extend(p.__args__)
        else:
            params.append(p)
    return tuple(params)
_cleanups = []
def _tp_cache(func=None, /, *, typed=False):
    """Internal wrapper caching __getitem__ of generic types.
    For non-hashable arguments, the original function is used as a fallback.
    """
    def decorator(func):


"""
    def decorator(func):
        cached = functools.lru_cache(typed=typed)(func)
        _cleanups.append(cached.cache_clear)
        @functools.wraps(func)
        def inner(*args, **kwds):
            try:
                return cached(*args, **kwds)
            except TypeError:
                pass  # All real errors (not unhashable args) are raised below.
            return func(*args, **kwds)
        return inner
    if func is not None:
        return decorator(func)
    return decorator
def _eval_type(t, globalns, localns, recursive_guard=frozenset()):
    """Evaluate all forward references in the given type t.
    For use of globalns and localns see the docstring for get_type_hints().
    recursive_guard is used to prevent infinite recursion with a recursive
    ForwardRef.
    """
    if isinstance(t, ForwardRef):
        return t._evaluate(globalns, localns, recursive_guard)
    if isinstance(t, (_GenericAlias, GenericAlias, types.UnionType)):
        if isinstance(t, GenericAlias):
            args = tuple(
                ForwardRef(arg) if isinstance(arg, str) else arg
                for arg in t.__args__
            )
            is_unpacked = t.__unpacked__
            if _should_unflatten_callable_args(t, args):
                t = t.__origin__[(args[:-1], args[-1])]
            else:
                t = t.__origin__[args]
            if is_unpacked:
                t = Unpack[t]
        ev_args = tuple(_eval_type(a, globalns, localns, recursive_guard) for a in t.__args__)
        if ev_args == t.__args__:
            return t


if ev_args == t.__args__:
            return t
        if isinstance(t, GenericAlias):
            return GenericAlias(t.__origin__, ev_args)
        if isinstance(t, types.UnionType):
            return functools.reduce(operator.or_, ev_args)
        else:
            return t.copy_with(ev_args)
    return t
class _Final:
    """Mixin to prohibit subclassing."""
    __slots__ = ('__weakref__',)
    def __init_subclass__(cls, /, *args, **kwds):
        if '_root' not in kwds:
            raise TypeError("Cannot subclass special typing classes")
class _Immutable:
    """Mixin to indicate that object should not be copied."""
    __slots__ = ()
    def __copy__(self):
        return self
    def __deepcopy__(self, memo):
        return self
class _NotIterable:
    """Mixin to prevent iteration, without being compatible with Iterable.
    That is, we could do::
        def __iter__(self): raise TypeError()
    But this would make users of this mixin duck type-compatible with
    collections.abc.Iterable - isinstance(foo, Iterable) would be True.
    Luckily, we can instead prevent iteration by setting __iter__ to None, which
    is treated specially.
    """
    __slots__ = ()
    __iter__ = None
# Internal indicator of special typing constructs.
# See __doc__ instance attribute for specific docs.
class _SpecialForm(_Final, _NotIterable, _root=True):
    __slots__ = ('_name', '__doc__', '_getitem')
    def __init__(self, getitem):
        self._getitem = getitem
        self._name = getitem.__name__
        self.__doc__ = getitem.__doc__


self.__doc__ = getitem.__doc__
    def __getattr__(self, item):
        if item in {'__name__', '__qualname__'}:
            return self._name
        raise AttributeError(item)
    def __mro_entries__(self, bases):
        raise TypeError(f"Cannot subclass {self!r}")
    def __repr__(self):
        return 'typing.' + self._name
    def __reduce__(self):
        return self._name
    def __call__(self, *args, **kwds):
        raise TypeError(f"Cannot instantiate {self!r}")
    def __or__(self, other):
        return Union[self, other]
    def __ror__(self, other):
        return Union[other, self]
    def __instancecheck__(self, obj):
        raise TypeError(f"{self} cannot be used with isinstance()")
    def __subclasscheck__(self, cls):
        raise TypeError(f"{self} cannot be used with issubclass()")
    @_tp_cache
    def __getitem__(self, parameters):
        return self._getitem(self, parameters)
class _LiteralSpecialForm(_SpecialForm, _root=True):
    def __getitem__(self, parameters):
        if not isinstance(parameters, tuple):
            parameters = (parameters,)
        return self._getitem(self, *parameters)
class _AnyMeta(type):
    def __instancecheck__(self, obj):
        if self is Any:
            raise TypeError("typing.Any cannot be used with isinstance()")
        return super().__instancecheck__(obj)
    def __repr__(self):
        if self is Any:
            return "typing.Any"
        return super().__repr__()  # respect to subclasses
class Any(metaclass=_AnyMeta):


class Any(metaclass=_AnyMeta):
    """Special type indicating an unconstrained type.
    - Any is compatible with every type.
    - Any assumed to have all methods.
    - All values assumed to be instances of Any.
    Note that all the above statements are true from the point of view of
    static type checkers. At runtime, Any should not be used with instance
    checks.
    """
    def __new__(cls, *args, **kwargs):
        if cls is Any:
            raise TypeError("Any cannot be instantiated")
        return super().__new__(cls, *args, **kwargs)
@_SpecialForm
def NoReturn(self, parameters):
    """Special type indicating functions that never return.
    Example::
        from typing import NoReturn
        def stop() -> NoReturn:
            raise Exception('no way')
    NoReturn can also be used as a bottom type, a type that
    has no values. Starting in Python 3.11, the Never type should
    be used for this concept instead. Type checkers should treat the two
    equivalently.
    """
    raise TypeError(f"{self} is not subscriptable")
# This is semantically identical to NoReturn, but it is implemented
# separately so that type checkers can distinguish between the two
# if they want.
@_SpecialForm
def Never(self, parameters):
    """The bottom type, a type that has no members.
    This can be used to define a function that should never be
    called, or a function that never returns::
        from typing import Never
        def never_call_me(arg: Never) -> None:
            pass
        def int_or_str(arg: int | str) -> None:
            never_call_me(arg)  # type checker error


never_call_me(arg)  # type checker error
            match arg:
                case int():
                    print("It's an int")
                case str():
                    print("It's a str")
                case _:
                    never_call_me(arg)  # OK, arg is of type Never
    """
    raise TypeError(f"{self} is not subscriptable")
@_SpecialForm
def Self(self, parameters):
    """Used to spell the type of "self" in classes.
    Example::
        from typing import Self
        class Foo:
            def return_self(self) -> Self:
                ...
                return self
    This is especially useful for:
        - classmethods that are used as alternative constructors
        - annotating an `__enter__` method which returns self
    """
    raise TypeError(f"{self} is not subscriptable")
@_SpecialForm
def LiteralString(self, parameters):
    """Represents an arbitrary literal string.
    Example::
        from typing import LiteralString
        def run_query(sql: LiteralString) -> None:
            ...
        def caller(arbitrary_string: str, literal_string: LiteralString) -> None:
            run_query("SELECT * FROM students")  # OK
            run_query(literal_string)  # OK
            run_query("SELECT * FROM " + literal_string)  # OK
            run_query(arbitrary_string)  # type checker error
            run_query(  # type checker error
                f"SELECT * FROM students WHERE name = {arbitrary_string}"
            )
    Only string literals and other LiteralStrings are compatible
    with LiteralString. This provides a tool to help prevent
    security issues such as SQL injection.
    """
    raise TypeError(f"{self} is not subscriptable")
@_SpecialForm


raise TypeError(f"{self} is not subscriptable")
@_SpecialForm
def ClassVar(self, parameters):
    """Special type construct to mark class variables.
    An annotation wrapped in ClassVar indicates that a given
    attribute is intended to be used as a class variable and
    should not be set on instances of that class.
    Usage::
        class Starship:
            stats: ClassVar[dict[str, int]] = {} # class variable
            damage: int = 10                     # instance variable
    ClassVar accepts only types and cannot be further subscribed.
    Note that ClassVar is not a class itself, and should not
    be used with isinstance() or issubclass().
    """
    item = _type_check(parameters, f'{self} accepts only single type.')
    return _GenericAlias(self, (item,))
@_SpecialForm
def Final(self, parameters):
    """Special typing construct to indicate final names to type checkers.
    A final name cannot be re-assigned or overridden in a subclass.
    For example::
        MAX_SIZE: Final = 9000
        MAX_SIZE += 1  # Error reported by type checker
        class Connection:
            TIMEOUT: Final[int] = 10
        class FastConnector(Connection):
            TIMEOUT = 1  # Error reported by type checker
    There is no runtime checking of these properties.
    """
    item = _type_check(parameters, f'{self} accepts only single type.')
    return _GenericAlias(self, (item,))
@_SpecialForm
def Union(self, parameters):
    """Union type; Union[X, Y] means either X or Y.
    On Python 3.10 and higher, the | operator
    can also be used to denote unions;


can also be used to denote unions;
    X | Y means the same thing to the type checker as Union[X, Y].
    To define a union, use e.g. Union[int, str]. Details:
    - The arguments must be types and there must be at least one.
    - None as an argument is a special case and is replaced by
      type(None).
    - Unions of unions are flattened, e.g.::
        assert Union[Union[int, str], float] == Union[int, str, float]
    - Unions of a single argument vanish, e.g.::
        assert Union[int] == int  # The constructor actually returns int
    - Redundant arguments are skipped, e.g.::
        assert Union[int, str, int] == Union[int, str]
    - When comparing unions, the argument order is ignored, e.g.::
        assert Union[int, str] == Union[str, int]
    - You cannot subclass or instantiate a union.
    - You can use Optional[X] as a shorthand for Union[X, None].
    """
    if parameters == ():
        raise TypeError("Cannot take a Union of no types.")
    if not isinstance(parameters, tuple):
        parameters = (parameters,)
    msg = "Union[arg, ...]: each arg must be a type."
    parameters = tuple(_type_check(p, msg) for p in parameters)
    parameters = _remove_dups_flatten(parameters)
    if len(parameters) == 1:
        return parameters[0]
    if len(parameters) == 2 and type(None) in parameters:
        return _UnionGenericAlias(self, parameters, name="Optional")
    return _UnionGenericAlias(self, parameters)
@_SpecialForm
def Optional(self, parameters):


@_SpecialForm
def Optional(self, parameters):
    """Optional[X] is equivalent to Union[X, None]."""
    arg = _type_check(parameters, f"{self} requires a single type.")
    return Union[arg, type(None)]
@_LiteralSpecialForm
@_tp_cache(typed=True)
def Literal(self, *parameters):
    """Special typing form to define literal types (a.k.a. value types).
    This form can be used to indicate to type checkers that the corresponding
    variable or function parameter has a value equivalent to the provided
    literal (or one of several literals)::
        def validate_simple(data: Any) -> Literal[True]:  # always returns True
            ...
        MODE = Literal['r', 'rb', 'w', 'wb']
        def open_helper(file: str, mode: MODE) -> str:
            ...
        open_helper('/some/path', 'r')  # Passes type check
        open_helper('/other/path', 'typo')  # Error in type checker
    Literal[...] cannot be subclassed. At runtime, an arbitrary value
    is allowed as type argument to Literal[...], but type checkers may
    impose restrictions.
    """
    # There is no '_type_check' call because arguments to Literal[...] are
    # values, not types.
    parameters = _flatten_literal_params(parameters)
    try:
        parameters = tuple(p for p, _ in _deduplicate(list(_value_and_type_iter(parameters))))
    except TypeError:  # unhashable parameters
        pass
    return _LiteralGenericAlias(self, parameters)
@_SpecialForm
def TypeAlias(self, parameters):
    """Special form for marking type aliases.
    Use TypeAlias to indicate that an assignment should


Use TypeAlias to indicate that an assignment should
    be recognized as a proper type alias definition by type
    checkers.
    For example::
        Predicate: TypeAlias = Callable[..., bool]
    It's invalid when used anywhere except as in the example above.
    """
    raise TypeError(f"{self} is not subscriptable")
@_SpecialForm
def Concatenate(self, parameters):
    """Special form for annotating higher-order functions.
    ``Concatenate`` can be used in conjunction with ``ParamSpec`` and
    ``Callable`` to represent a higher-order function which adds, removes or
    transforms the parameters of a callable.
    For example::
        Callable[Concatenate[int, P], int]
    See PEP 612 for detailed information.
    """
    if parameters == ():
        raise TypeError("Cannot take a Concatenate of no types.")
    if not isinstance(parameters, tuple):
        parameters = (parameters,)
    if not (parameters[-1] is ... or isinstance(parameters[-1], ParamSpec)):
        raise TypeError("The last parameter to Concatenate should be a "
                        "ParamSpec variable or ellipsis.")
    msg = "Concatenate[arg, ...]: each arg must be a type."
    parameters = (*(_type_check(p, msg) for p in parameters[:-1]), parameters[-1])
    return _ConcatenateGenericAlias(self, parameters,
                                    _paramspec_tvars=True)
@_SpecialForm
def TypeGuard(self, parameters):
    """Special typing construct for marking user-defined type guard functions.
    ``TypeGuard`` can be used to annotate the return type of a user-defined
    type guard function.  ``TypeGuard`` only accepts a single type argument.
    At runtime, functions marked this way should return a boolean.


At runtime, functions marked this way should return a boolean.
    ``TypeGuard`` aims to benefit *type narrowing* -- a technique used by static
    type checkers to determine a more precise type of an expression within a
    program's code flow.  Usually type narrowing is done by analyzing
    conditional code flow and applying the narrowing to a block of code.  The
    conditional expression here is sometimes referred to as a "type guard".
    Sometimes it would be convenient to use a user-defined boolean function
    as a type guard.  Such a function should use ``TypeGuard[...]`` as its
    return type to alert static type checkers to this intention.
    Using  ``-> TypeGuard`` tells the static type checker that for a given
    function:
    1. The return value is a boolean.
    2. If the return value is ``True``, the type of its argument
       is the type inside ``TypeGuard``.
       For example::
           def is_str(val: Union[str, float]):
               # "isinstance" type guard
               if isinstance(val, str):
                   # Type of ``val`` is narrowed to ``str``
                   ...
               else:
                   # Else, type of ``val`` is narrowed to ``float``.
                   ...
    Strict type narrowing is not enforced -- ``TypeB`` need not be a narrower
    form of ``TypeA`` (it can even be a wider form) and this may lead to
    type-unsafe results.  The main reason is to allow for things like
    narrowing ``List[object]`` to ``List[str]`` even though the latter is not
    a subtype of the former, since ``List`` is invariant.  The responsibility of
    writing type-safe type guards is left to the user.


writing type-safe type guards is left to the user.
    ``TypeGuard`` also works with type variables.  For more information, see
    PEP 647 (User-Defined Type Guards).
    """
    item = _type_check(parameters, f'{self} accepts only single type.')
    return _GenericAlias(self, (item,))
class ForwardRef(_Final, _root=True):
    """Internal wrapper to hold a forward reference."""
    __slots__ = ('__forward_arg__', '__forward_code__',
                 '__forward_evaluated__', '__forward_value__',
                 '__forward_is_argument__', '__forward_is_class__',
                 '__forward_module__')
    def __init__(self, arg, is_argument=True, module=None, *, is_class=False):
        if not isinstance(arg, str):
            raise TypeError(f"Forward reference must be a string -- got {arg!r}")
        # If we do `def f(*args: *Ts)`, then we'll have `arg = '*Ts'`.
        # Unfortunately, this isn't a valid expression on its own, so we
        # do the unpacking manually.
        if arg[0] == '*':
            arg_to_compile = f'({arg},)[0]'  # E.g. (*Ts,)[0] or (*tuple[int, int],)[0]
        else:
            arg_to_compile = arg
        try:
            code = compile(arg_to_compile, '<string>', 'eval')
        except SyntaxError:
            raise SyntaxError(f"Forward reference must be an expression -- got {arg!r}")
        self.__forward_arg__ = arg
        self.__forward_code__ = code
        self.__forward_evaluated__ = False
        self.__forward_value__ = None
        self.__forward_is_argument__ = is_argument


self.__forward_value__ = None
        self.__forward_is_argument__ = is_argument
        self.__forward_is_class__ = is_class
        self.__forward_module__ = module
    def _evaluate(self, globalns, localns, recursive_guard):
        if self.__forward_arg__ in recursive_guard:
            return self
        if not self.__forward_evaluated__ or localns is not globalns:
            if globalns is None and localns is None:
                globalns = localns = {}
            elif globalns is None:
                globalns = localns
            elif localns is None:
                localns = globalns
            if self.__forward_module__ is not None:
                globalns = getattr(
                    sys.modules.get(self.__forward_module__, None), '__dict__', globalns
                )
            type_ = _type_check(
                eval(self.__forward_code__, globalns, localns),
                "Forward references must evaluate to types.",
                is_argument=self.__forward_is_argument__,
                allow_special_forms=self.__forward_is_class__,
            )
            self.__forward_value__ = _eval_type(
                type_, globalns, localns, recursive_guard | {self.__forward_arg__}
            )
            self.__forward_evaluated__ = True
        return self.__forward_value__
    def __eq__(self, other):
        if not isinstance(other, ForwardRef):
            return NotImplemented
        if self.__forward_evaluated__ and other.__forward_evaluated__:
            return (self.__forward_arg__ == other.__forward_arg__ and
                    self.__forward_value__ == other.__forward_value__)
        return (self.__forward_arg__ == other.__forward_arg__ and
                self.__forward_module__ == other.__forward_module__)
    def __hash__(self):


def __hash__(self):
        return hash((self.__forward_arg__, self.__forward_module__))
    def __or__(self, other):
        return Union[self, other]
    def __ror__(self, other):
        return Union[other, self]
    def __repr__(self):
        if self.__forward_module__ is None:
            module_repr = ''
        else:
            module_repr = f', module={self.__forward_module__!r}'
        return f'ForwardRef({self.__forward_arg__!r}{module_repr})'
def _is_unpacked_typevartuple(x: Any) -> bool:
    return ((not isinstance(x, type)) and
            getattr(x, '__typing_is_unpacked_typevartuple__', False))
def _is_typevar_like(x: Any) -> bool:
    return isinstance(x, (TypeVar, ParamSpec)) or _is_unpacked_typevartuple(x)
class _PickleUsingNameMixin:
    """Mixin enabling pickling based on self.__name__."""
    def __reduce__(self):
        return self.__name__
class _BoundVarianceMixin:
    """Mixin giving __init__ bound and variance arguments.
    This is used by TypeVar and ParamSpec, which both employ the notions of
    a type 'bound' (restricting type arguments to be a subtype of some
    specified type) and type 'variance' (determining subtype relations between
    generic types).
    """
    def __init__(self, bound, covariant, contravariant):
        """Used to setup TypeVars and ParamSpec's bound, covariant and
        contravariant attributes.
        """
        if covariant and contravariant:
            raise ValueError("Bivariant types are not supported.")


raise ValueError("Bivariant types are not supported.")
        self.__covariant__ = bool(covariant)
        self.__contravariant__ = bool(contravariant)
        if bound:
            self.__bound__ = _type_check(bound, "Bound must be a type.")
        else:
            self.__bound__ = None
    def __or__(self, right):
        return Union[self, right]
    def __ror__(self, left):
        return Union[left, self]
    def __repr__(self):
        if self.__covariant__:
            prefix = '+'
        elif self.__contravariant__:
            prefix = '-'
        else:
            prefix = '~'
        return prefix + self.__name__
class TypeVar(_Final, _Immutable, _BoundVarianceMixin, _PickleUsingNameMixin,
              _root=True):
    """Type variable.
    Usage::
      T = TypeVar('T')  # Can be anything
      A = TypeVar('A', str, bytes)  # Must be str or bytes
    Type variables exist primarily for the benefit of static type
    checkers.  They serve as the parameters for generic types as well
    as for generic function definitions.  See class Generic for more
    information on generic types.  Generic functions work as follows:
      def repeat(x: T, n: int) -> List[T]:
          '''Return a list containing n references to x.'''
          return [x]*n
      def longest(x: A, y: A) -> A:
          '''Return the longest of two strings.'''
          return x if len(x) >= len(y) else y
    The latter example's signature is essentially the overloading


The latter example's signature is essentially the overloading
    of (str, str) -> str and (bytes, bytes) -> bytes.  Also note
    that if the arguments are instances of some subclass of str,
    the return type is still plain str.
    At runtime, isinstance(x, T) and issubclass(C, T) will raise TypeError.
    Type variables defined with covariant=True or contravariant=True
    can be used to declare covariant or contravariant generic types.
    See PEP 484 for more details. By default generic types are invariant
    in all type variables.
    Type variables can be introspected. e.g.:
      T.__name__ == 'T'
      T.__constraints__ == ()
      T.__covariant__ == False
      T.__contravariant__ = False
      A.__constraints__ == (str, bytes)
    Note that only type variables defined in global scope can be pickled.
    """
    def __init__(self, name, *constraints, bound=None,
                 covariant=False, contravariant=False):
        self.__name__ = name
        super().__init__(bound, covariant, contravariant)
        if constraints and bound is not None:
            raise TypeError("Constraints cannot be combined with bound=...")
        if constraints and len(constraints) == 1:
            raise TypeError("A single constraint is not allowed")
        msg = "TypeVar(name, constraint, ...): constraints must be types."
        self.__constraints__ = tuple(_type_check(t, msg) for t in constraints)
        def_mod = _caller()
        if def_mod != 'typing':
            self.__module__ = def_mod
    def __typing_subst__(self, arg):


def __typing_subst__(self, arg):
        msg = "Parameters to generic types must be types."
        arg = _type_check(arg, msg, is_argument=True)
        if ((isinstance(arg, _GenericAlias) and arg.__origin__ is Unpack) or
            (isinstance(arg, GenericAlias) and getattr(arg, '__unpacked__', False))):
            raise TypeError(f"{arg} is not valid as type argument")
        return arg
class TypeVarTuple(_Final, _Immutable, _PickleUsingNameMixin, _root=True):
    """Type variable tuple.
    Usage:
      Ts = TypeVarTuple('Ts')  # Can be given any name
    Just as a TypeVar (type variable) is a placeholder for a single type,
    a TypeVarTuple is a placeholder for an *arbitrary* number of types. For
    example, if we define a generic class using a TypeVarTuple:
      class C(Generic[*Ts]): ...
    Then we can parameterize that class with an arbitrary number of type
    arguments:
      C[int]       # Fine
      C[int, str]  # Also fine
      C[()]        # Even this is fine
    For more details, see PEP 646.
    Note that only TypeVarTuples defined in global scope can be pickled.
    """
    def __init__(self, name):
        self.__name__ = name
        # Used for pickling.
        def_mod = _caller()
        if def_mod != 'typing':
            self.__module__ = def_mod
    def __iter__(self):
        yield Unpack[self]
    def __repr__(self):
        return self.__name__
    def __typing_subst__(self, arg):


return self.__name__
    def __typing_subst__(self, arg):
        raise TypeError("Substitution of bare TypeVarTuple is not supported")
    def __typing_prepare_subst__(self, alias, args):
        params = alias.__parameters__
        typevartuple_index = params.index(self)
        for param in params[typevartuple_index + 1:]:
            if isinstance(param, TypeVarTuple):
                raise TypeError(f"More than one TypeVarTuple parameter in {alias}")
        alen = len(args)
        plen = len(params)
        left = typevartuple_index
        right = plen - typevartuple_index - 1
        var_tuple_index = None
        fillarg = None
        for k, arg in enumerate(args):
            if not isinstance(arg, type):
                subargs = getattr(arg, '__typing_unpacked_tuple_args__', None)
                if subargs and len(subargs) == 2 and subargs[-1] is ...:
                    if var_tuple_index is not None:
                        raise TypeError("More than one unpacked arbitrary-length tuple argument")
                    var_tuple_index = k
                    fillarg = subargs[0]
        if var_tuple_index is not None:
            left = min(left, var_tuple_index)
            right = min(right, alen - var_tuple_index - 1)
        elif left + right > alen:
            raise TypeError(f"Too few arguments for {alias};"
                            f" actual {alen}, expected at least {plen-1}")
        return (
            *args[:left],
            *([fillarg]*(typevartuple_index - left)),
            tuple(args[left: alen - right]),


tuple(args[left: alen - right]),
            *([fillarg]*(plen - right - left - typevartuple_index - 1)),
            *args[alen - right:],
        )
class ParamSpecArgs(_Final, _Immutable, _root=True):
    """The args for a ParamSpec object.
    Given a ParamSpec object P, P.args is an instance of ParamSpecArgs.
    ParamSpecArgs objects have a reference back to their ParamSpec:
       P.args.__origin__ is P
    This type is meant for runtime introspection and has no special meaning to
    static type checkers.
    """
    def __init__(self, origin):
        self.__origin__ = origin
    def __repr__(self):
        return f"{self.__origin__.__name__}.args"
    def __eq__(self, other):
        if not isinstance(other, ParamSpecArgs):
            return NotImplemented
        return self.__origin__ == other.__origin__
class ParamSpecKwargs(_Final, _Immutable, _root=True):
    """The kwargs for a ParamSpec object.
    Given a ParamSpec object P, P.kwargs is an instance of ParamSpecKwargs.
    ParamSpecKwargs objects have a reference back to their ParamSpec:
       P.kwargs.__origin__ is P
    This type is meant for runtime introspection and has no special meaning to
    static type checkers.
    """
    def __init__(self, origin):
        self.__origin__ = origin
    def __repr__(self):
        return f"{self.__origin__.__name__}.kwargs"
    def __eq__(self, other):
        if not isinstance(other, ParamSpecKwargs):
            return NotImplemented
        return self.__origin__ == other.__origin__


return NotImplemented
        return self.__origin__ == other.__origin__
class ParamSpec(_Final, _Immutable, _BoundVarianceMixin, _PickleUsingNameMixin,
                _root=True):
    """Parameter specification variable.
    Usage::
       P = ParamSpec('P')
    Parameter specification variables exist primarily for the benefit of static
    type checkers.  They are used to forward the parameter types of one
    callable to another callable, a pattern commonly found in higher order
    functions and decorators.  They are only valid when used in ``Concatenate``,
    or as the first argument to ``Callable``, or as parameters for user-defined
    Generics.  See class Generic for more information on generic types.  An
    example for annotating a decorator::
       T = TypeVar('T')
       P = ParamSpec('P')
       def add_logging(f: Callable[P, T]) -> Callable[P, T]:
           '''A type-safe decorator to add logging to a function.'''
           def inner(*args: P.args, **kwargs: P.kwargs) -> T:
               logging.info(f'{f.__name__} was called')
               return f(*args, **kwargs)
           return inner
       @add_logging
       def add_two(x: float, y: float) -> float:
           '''Add two numbers together.'''
           return x + y
    Parameter specification variables can be introspected. e.g.:
       P.__name__ == 'P'
    Note that only parameter specification variables defined in global scope can
    be pickled.
    """
    @property
    def args(self):
        return ParamSpecArgs(self)
    @property
    def kwargs(self):
        return ParamSpecKwargs(self)


@property
    def kwargs(self):
        return ParamSpecKwargs(self)
    def __init__(self, name, *, bound=None, covariant=False, contravariant=False):
        self.__name__ = name
        super().__init__(bound, covariant, contravariant)
        def_mod = _caller()
        if def_mod != 'typing':
            self.__module__ = def_mod
    def __typing_subst__(self, arg):
        if isinstance(arg, (list, tuple)):
            arg = tuple(_type_check(a, "Expected a type.") for a in arg)
        elif not _is_param_expr(arg):
            raise TypeError(f"Expected a list of types, an ellipsis, "
                            f"ParamSpec, or Concatenate. Got {arg}")
        return arg
    def __typing_prepare_subst__(self, alias, args):
        params = alias.__parameters__
        i = params.index(self)
        if i >= len(args):
            raise TypeError(f"Too few arguments for {alias}")
        # Special case where Z[[int, str, bool]] == Z[int, str, bool] in PEP 612.
        if len(params) == 1 and not _is_param_expr(args[0]):
            assert i == 0
            args = (args,)
        # Convert lists to tuples to help other libraries cache the results.
        elif isinstance(args[i], list):
            args = (*args[:i], tuple(args[i]), *args[i+1:])
        return args
def _is_dunder(attr):
    return attr.startswith('__') and attr.endswith('__')
class _BaseGenericAlias(_Final, _root=True):
    """The central part of the internal API.


"""The central part of the internal API.
    This represents a generic version of type 'origin' with type arguments 'params'.
    There are two kind of these aliases: user defined and special. The special ones
    are wrappers around builtin collections and ABCs in collections.abc. These must
    have 'name' always set. If 'inst' is False, then the alias can't be instantiated;
    this is used by e.g. typing.List and typing.Dict.
    """
    def __init__(self, origin, *, inst=True, name=None):
        self._inst = inst
        self._name = name
        self.__origin__ = origin
        self.__slots__ = None  # This is not documented.
    def __call__(self, *args, **kwargs):
        if not self._inst:
            raise TypeError(f"Type {self._name} cannot be instantiated; "
                            f"use {self.__origin__.__name__}() instead")
        result = self.__origin__(*args, **kwargs)
        try:
            result.__orig_class__ = self
        except AttributeError:
            pass
        return result
    def __mro_entries__(self, bases):
        res = []
        if self.__origin__ not in bases:
            res.append(self.__origin__)
        i = bases.index(self)
        for b in bases[i+1:]:
            if isinstance(b, _BaseGenericAlias) or issubclass(b, Generic):
                break
        else:
            res.append(Generic)
        return tuple(res)
    def __getattr__(self, attr):
        if attr in {'__name__', '__qualname__'}:
            return self._name or self.__origin__.__name__
        # We are careful for copy and pickle.


# We are careful for copy and pickle.
        # Also for simplicity we don't relay any dunder names
        if '__origin__' in self.__dict__ and not _is_dunder(attr):
            return getattr(self.__origin__, attr)
        raise AttributeError(attr)
    def __setattr__(self, attr, val):
        if _is_dunder(attr) or attr in {'_name', '_inst', '_nparams',
                                        '_paramspec_tvars'}:
            super().__setattr__(attr, val)
        else:
            setattr(self.__origin__, attr, val)
    def __instancecheck__(self, obj):
        return self.__subclasscheck__(type(obj))
    def __subclasscheck__(self, cls):
        raise TypeError("Subscripted generics cannot be used with"
                        " class and instance checks")
    def __dir__(self):
        return list(set(super().__dir__()
                + [attr for attr in dir(self.__origin__) if not _is_dunder(attr)]))
# Special typing constructs Union, Optional, Generic, Callable and Tuple
# use three special attributes for internal bookkeeping of generic types:
# * __parameters__ is a tuple of unique free type parameters of a generic
#   type, for example, Dict[T, T].__parameters__ == (T,);
# * __origin__ keeps a reference to a type that was subscripted,
#   e.g., Union[T, int].__origin__ == Union, or the non-generic version of
#   the type.
# * __args__ is a tuple of all arguments used in subscripting,
#   e.g., Dict[T, int].__args__ == (T, int).
class _GenericAlias(_BaseGenericAlias, _root=True):
    # The type of parameterized generics.
    #


# The type of parameterized generics.
    #
    # That is, for example, `type(List[int])` is `_GenericAlias`.
    #
    # Objects which are instances of this class include:
    # * Parameterized container types, e.g. `Tuple[int]`, `List[int]`.
    #  * Note that native container types, e.g. `tuple`, `list`, use
    #    `types.GenericAlias` instead.
    # * Parameterized classes:
    #     T = TypeVar('T')
    #     class C(Generic[T]): pass
    #     # C[int] is a _GenericAlias
    # * `Callable` aliases, generic `Callable` aliases, and
    #   parameterized `Callable` aliases:
    #     T = TypeVar('T')
    #     # _CallableGenericAlias inherits from _GenericAlias.
    #     A = Callable[[], None]  # _CallableGenericAlias
    #     B = Callable[[T], None]  # _CallableGenericAlias
    #     C = B[int]  # _CallableGenericAlias
    # * Parameterized `Final`, `ClassVar` and `TypeGuard`:
    #     # All _GenericAlias
    #     Final[int]
    #     ClassVar[float]
    #     TypeVar[bool]
    def __init__(self, origin, args, *, inst=True, name=None,
                 _paramspec_tvars=False):
        super().__init__(origin, inst=inst, name=name)
        if not isinstance(args, tuple):
            args = (args,)
        self.__args__ = tuple(... if a is _TypingEllipsis else
                              a for a in args)
        self.__parameters__ = _collect_parameters(args)