Python API Reference

The Python API lets you build, run, and optimize DocETL pipelines with chainable operations.

import docetl

Deprecated: the typed Pipeline class

The older object API (from docetl.api import Pipeline with MapOp, ReduceOp, etc.) is deprecated in favor of the Frame API documented on this page.

Quick start

import docetl

docetl.default_model = "gpt-4o-mini"

results = (
    docetl.read_json("input.json")
    .map(
        prompt="Classify this document: {{ input.text }}",
        output={"schema": {"category": "string"}},
    )
    .filter(prompt="Is this document about technology? {{ input.text }}")
    .reduce(
        reduce_key="category",
        prompt="Summarize these documents: {% for item in inputs %}{{ item.text }}{% endfor %}",
        output={"schema": {"summary": "string"}},
    )
    .collect()
)

Configuration

Set global defaults as module-level attributes:

Attribute	Type	Default	Description
`docetl.default_model`	`str`	`None`	Default LLM model for all operations
`docetl.agent_model`	`str`	`None`	Model for optimizer rewrites
`docetl.max_threads`	`int`	`cpu_count * 4`	Concurrent threads
`docetl.bypass_cache`	`bool`	`False`	Skip LLM cache
`docetl.intermediate_dir`	`str`	`None`	Directory for intermediate results (a relative path resolves against the working directory at run time)
`docetl.rate_limits`	`dict`	`None`	Rate limits per model
`docetl.fallback_models`	`list[str]`	`None`	Fallback chain on failure
`docetl.fallback_embedding_models`	`list[str]`	`None`	Fallback embedding models
`docetl.system_prompt`	`dict`	`None`	`{"dataset_description": ..., "persona": ...}` applied to all operations

Precedence. Settings layer from most to least specific: a per-operation parameter (e.g. model= on .map()) beats per-pipeline settings carried by a Frame (set when loading a YAML via Frame.from_yaml), which beat the module-level docetl.* globals above, which beat built-in defaults. Loading a YAML never changes the globals — its settings travel with that Frame only.

Reading Data

Function	Description
`docetl.read_json(path)`	Load from JSON file
`docetl.read_csv(path)`	Load from CSV file
`docetl.read_parquet(path)`	Load from Parquet file
`docetl.read_dir(path)`	One row per file in a directory (recursive): `text` is the file's content, plus `filename` and `path`. PDF/Word/PowerPoint/Excel are converted to text; other files read as UTF-8
`docetl.from_list(data)`	Load from a list of dicts
`docetl.Frame.from_yaml(path)`	Load from a YAML pipeline config

All return a Frame.

Frame

A lazy pipeline — operations are recorded but not executed until a terminal action is called. Frames are immutable; every operation returns a new Frame.

LLM Operations

`.map()`

Applies an LLM prompt to each document independently.

frame.map(
    prompt="Classify: {{ input.text }}",
    output={"schema": {"category": "str"}},
)

Parameter	Type	Default	Description
`prompt`	`str`	—	Jinja2 template. Access fields via `{{ input.key }}`.
`output`	`dict`	—	Output schema, e.g. `{"schema": {"field": "str"}}`
`model`	`str`	`None`	Override default model
`validate`	`list[str \\| callable]`	`None`	Validators: expression strings over `output`, or callables taking the output dict (callables can't be exported to YAML)
`num_retries_on_validate_failure`	`int`	`None`	Retries on validation failure
`sample`	`int`	`None`	Process only N documents
`agent`	`docetl.Agent`	`None`	Tool-equipped agent using the OpenAI Agents SDK
`drop_keys`	`list[str]`	`None`	Keys to remove from output
`timeout`	`int`	`None`	Timeout per LLM call (seconds)
`max_batch_size`	`int`	`None`	Batch size for batch processing
`batch_prompt`	`str`	`None`	Jinja2 template for batch mode
`retriever`	`Retriever`	`None`	Retriever for context augmentation
`optimize`	`bool`	`None`	Mark for optimization
`limit`	`int`	`None`	Max documents to process

`.filter()`

Keeps or removes documents based on an LLM prompt. Output schema must have one boolean field.

frame.filter(
    prompt="Is this about technology? {{ input.text }}",
    output={"schema": {"is_tech": "bool"}},
)

Parameter	Type	Default	Description
`prompt`	`str`	—	Jinja2 template
`output`	`dict`	—	Schema with one boolean field
`model`	`str`	`None`	Override default model
`validate`	`list[str \\| callable]`	`None`	Validators: expression strings or callables over the output dict
`agent`	`docetl.Agent`	`None`	Tool-equipped agent using the OpenAI Agents SDK
`retriever`	`Retriever`	`None`	Retriever for context augmentation
`cascade`	`dict`	`None`	Model cascade: run a cheap proxy (chat or embedding model) on all items and escalate only uncertain ones, with a statistical guarantee. Also available on `resolve` and `equijoin`.

frame.filter(
    prompt="Is this review about shipping problems? {{ input.text }}",
    output={"schema": {"keep": "bool"}},
    cascade={"proxy_model": "text-embedding-3-small", "guarantee": "recall",
             "target": 0.9, "label_budget": 120},
)

`.reduce()`

Groups documents by key and reduces each group with an LLM.

frame.reduce(
    reduce_key="category",
    prompt="Summarize: {% for item in inputs %}{{ item.text }}{% endfor %}",
    output={"schema": {"summary": "str"}},
)

Parameter	Type	Default	Description
`reduce_key`	`str \\| list[str]`	—	Key(s) to group by. Use `"_all"` for one group.
`prompt`	`str`	—	Jinja2 template. Iterate with `{% for item in inputs %}`.
`output`	`dict`	—	Output schema
`fold_prompt`	`str`	`None`	Prompt for incremental folding
`fold_batch_size`	`int`	`None`	Items per fold iteration
`merge_prompt`	`str`	`None`	Prompt for merging fold results
`pass_through`	`bool`	`None`	Pass through non-reduced keys
`associative`	`bool`	`None`	Enable parallel reduction
`agent`	`docetl.Agent`	`None`	Tool-equipped agent using the OpenAI Agents SDK
`retriever`	`Retriever`	`None`	Retriever for context augmentation

Tool-equipped map/filter/reduce

Use docetl.Agent when an operation should call tools over multiple turns before returning DocETL's structured output. DocETL adapts Python functions into OpenAI Agents SDK tools and routes the operation's LiteLLM model through the SDK's LiteLLM integration.

import docetl

@docetl.tool
def lookup_sla(customer_tier: str) -> dict[str, str | int]:
    """Return support entitlements for a customer tier."""
    return {
        "enterprise": {"response_hours": 1, "escalation": "page-on-call"},
        "growth": {"response_hours": 4, "escalation": "queue-lead"},
        "free": {"response_hours": 48, "escalation": "self-serve"},
    }.get(customer_tier.lower(), {"response_hours": 24, "escalation": "standard"})

agent = docetl.Agent(tools=[lookup_sla], max_turns=5, max_tool_calls=3)

rows = (
    docetl.from_list([
        {
            "ticket": "Production API latency is above the SLO.",
            "customer_tier": "enterprise",
        }
    ])
    .map(
        prompt=(
            "Use lookup_sla to classify this ticket and explain the next action: "
            "{{ input.ticket }} / tier={{ input.customer_tier }}"
        ),
        output={"schema": {"priority": "str", "next_action": "str"}},
        model="azure/gpt-4o-mini",
        agent=agent,
    )
    .collect()
)

The model is still specified on the operation (model=) or through docetl.default_model; use LiteLLM model names such as azure/gpt-4o-mini, anthropic/..., or together_ai/.... The selected model/provider must work with the OpenAI Agents SDK LiteLLM integration and the requested tools. Plain Python function tools execute as trusted Python in your process; OpenAI Agents SDK sandbox/native tools may provide isolation where that SDK/backend supports it. Agent configs are Python-only and cannot be exported to YAML. See OpenAI's Agents SDK guide and tools guide for hosted tool behavior.

Agents can also expose specialist agents as tools. Use this when a manager agent should keep ownership of the DocETL output while delegating a bounded task:

specialist = docetl.Agent(
    instructions="Extract numeric evidence from the supplied text.",
    max_turns=4,
)

manager = docetl.Agent(
    tools=[
        specialist.as_tool(
            name="extract_numeric_evidence",
            description="Extract numeric evidence for the manager agent.",
        )
    ],
    max_turns=6,
)

The specialist uses the same operation-level LiteLLM model as the manager unless you pass an SDK-native tool object with its own model configuration. This follows OpenAI's agents-as-tools orchestration pattern.

For a persistent shell/sandbox filesystem on OpenAI, create a hosted container once and bind shell tools to it:

sandbox = docetl.tools.Sandbox.create(
    name="docetl-research",
    network="disabled",
    memory_limit="1g",
)
bash = sandbox.bash()

agent = docetl.Agent(
    tools=[bash],
    max_turns=4,
    max_tool_calls=6,
)

Reuse bash across a manager and its specialist subagents when they should read and write the same hosted filesystem. If you only need an ephemeral shell tool, use docetl.tools.bash(...), which uses container_auto. Pass durable data between DocETL operations through output schemas, not only through hidden sandbox files.

docetl.tools.Sandbox.create(...) is OpenAI-specific: it calls the OpenAI Containers API and returns shell tools using container_reference. It is not portable to Claude, Together, or other LiteLLM providers. For provider-portable tooling, use @docetl.tool Python functions, MCP tools, or provider-native SDK tools. See OpenAI's sandbox agents guide.

For an end-to-end example, see the Tool-Equipped Agents tutorial.

`.resolve()`

Deduplicates entities by pairwise LLM comparison.

frame.resolve(
    comparison_prompt="Same person? {{ input1.name }} vs {{ input2.name }}",
    resolution_prompt="Canonical name: {% for e in inputs %}{{ e.name }}{% endfor %}",
    output={"schema": {"name": "str"}},
)

Parameter	Type	Default	Description
`comparison_prompt`	`str`	—	Jinja2 template comparing `{{ input1 }}` and `{{ input2 }}`
`resolution_prompt`	`str`	`None`	Prompt for resolving matched groups
`output`	`dict`	`None`	Output schema
`blocking_keys`	`list[str]`	`None`	Keys for blocking
`blocking_threshold`	`float`	`None`	Similarity threshold
`embedding_model`	`str`	`None`	Model for blocking embeddings
`optimize`	`bool`	`None`	Mark for optimization

`.extract()`

Extracts information from documents with line-level precision.

frame.extract(
    prompt="Extract key findings from this paper.",
    document_keys=["content"],
)

Parameter	Type	Default	Description
`prompt`	`str`	—	Extraction prompt
`document_keys`	`list[str]`	—	Keys containing document text
`retriever`	`Retriever`	`None`	Retriever for context augmentation

`.parallel_map()`

Runs multiple prompts on each document in parallel.

frame.parallel_map(
    prompts=[{"prompt": "...", "output_keys": ["field1"]}, ...],
    output={"schema": {"field1": "str", "field2": "str"}},
)

`.equijoin()`

Joins two datasets by LLM comparison.

right = docetl.read_json("other.json")
frame.equijoin(right, comparison_prompt="Are these related? {{ left.x }} {{ right.y }}")

Structural Operations (no LLM calls)

Method	Description
`.split(split_key, method, method_kwargs)`	Split documents into chunks
`.gather(content_key, doc_id_key, order_key)`	Add surrounding context to chunks
`.unnest(unnest_key)`	Flatten a list field into separate rows
`.cluster(embedding_keys)`	Cluster documents by embedding similarity
`.sample(samples, method)`	Sample a subset of documents

Code Operations (no LLM calls)

code takes any callable, e.g., a lambda, or a string of Python source defining a transform function (the YAML form).

Method	Description
`.code_map(code=lambda doc: {...})`	Per-document Python transform
`.code_filter(code=lambda doc: bool(...))`	Per-document Python filter (return bool)
`.code_reduce(reduce_key, code=lambda items: {...})`	Per-group Python aggregation

Retriever

Augment LLM operations with retrieved context from a LanceDB index.

retriever = docetl.Retriever(
    dataset="knowledge_base",
    index_dir="./lance_index",
    index_types=["fts", "embedding"],
    fts={"index_phrase": "{{ input.text }}", "query_phrase": "{{ input.question }}"},
    embedding={"model": "text-embedding-3-small", "index_phrase": "...", "query_phrase": "..."},
    query={"mode": "hybrid", "top_k": 5},
)

Parameter	Type	Default	Description
`data`	`str \\| list[dict]`	—	File path or in-memory records to index (alternative to `dataset`)
`dataset`	`str`	—	Name of an existing pipeline dataset or step output to index
`index_dir`	`str`	—	Directory for the LanceDB index
`index_types`	`list[str]`	—	`["fts"]`, `["embedding"]`, or both
`fts`	`dict`	`None`	Full-text search config (`index_phrase`, `query_phrase`)
`embedding`	`dict`	`None`	Embedding config (`model`, `index_phrase`, `query_phrase`)
`query`	`dict`	`None`	Query config (`mode`, `top_k`)
`build_index`	`str`	`"if_missing"`	`"if_missing"`, `"always"`, or `"never"`

Pass to any LLM operation via retriever=. Retrieved context is available as {{ retrieval_context }} in prompts.

Pass exactly one of data (a file path or list of dicts to index) or dataset (the name of an existing pipeline dataset — the frame's own input or a previous step's output, step_<operation_name>). See the Retrievers guide.

Inspection (no execution)

Method	Return Type	Description
`frame.schema()`	`dict[str, str]`	Output schema from operation definitions, including structural ops (split/unnest/gather/extract). Best-effort: `code_*` op outputs can't be known statically
`frame.count()`	`int`	Input count (no ops) or output count (executes if ops present)
`frame.to_yaml()`	`str`	Export pipeline as YAML config
`frame.to_yaml(path)`	`str`	Also write YAML to file
`frame.to_python()`	`str`	Export as Python source code

Terminal Actions

Method	Return Type	Description
`frame.show(max=5)`	`DataFrame`	Run on a sample and print results. Works on bare datasets too.
`frame.collect()`	`list[dict]`	Execute the pipeline, return the result rows
`frame.to_pandas()`	`DataFrame`	Execute the pipeline, return a pandas DataFrame
`frame.write_json(path)`	`None`	Execute and write to JSON
`frame.write_csv(path)`	`None`	Execute and write to CSV
`frame.write_parquet(path)`	`None`	Execute and write to Parquet

Terminal actions are memoized on the Frame: repeated calls with an unchanged configuration reuse the previous result instead of re-running the pipeline. Changing ops, in-memory data, or docetl.* settings invalidates the memo; edits to input files between calls are not detected.

Cost & Token Tracking

rows = frame.collect()

print(f"Cost: ${frame.total_cost:.4f}")
print(f"Tokens: {frame.token_usage}")

df = frame.to_pandas()
print(f"Cost: ${df.attrs['_total_cost']:.4f}")  # also on the DataFrame

Optimization

@docetl.register_eval
def evaluate(results):
    correct = sum(1 for r in results if r.get("correct"))
    return {"score": correct}

optimized = frame.optimize(
    eval_fn=evaluate,
    metric_key="score",
)

Parameter	Type	Default	Description
`eval_fn`	`Callable`	required	Scores pipeline output. Returns a dict of metrics.
`metric_key`	`str`	required	Key in eval_fn's return dict to optimize
`models`	`list[str]`	Auto-detect	LiteLLM model names to explore
`agent_model`	`str`	Auto-select	Model for rewrite agent
`max_iterations`	`int`	`20`	Search budget
`save_dir`	`str`	Temp dir	Where to save results
`exploration_weight`	`float`	`1.414`	UCB exploration constant
`dataset_path`	`str`	Pipeline's dataset	Sample dataset for optimization
`max_threads`	`int`	`None`	Max concurrent LLM calls per run
`max_concurrent_agents`	`int`	`3`	Parallel MCTS search agents

Returns an optimized Frame. Access search results via optimized.search_results:

Method / Property	Return Type	Description
`.best()`	`OptimizedPipeline`	Highest-accuracy solution
`.cheapest()`	`OptimizedPipeline`	Lowest-cost solution
`.frontier`	`list[OptimizedPipeline]`	All Pareto-optimal solutions
`.to_df()`	`DataFrame`	All explored plans