Feature Request: register_replacement_scan in Python API

[teserak]

TL;DR

Allow registering a Python callback (e.g. register_replacement_scan(my_callback)) that can dynamically resolve table names into actual data sources (RecordBatchReader, DataFrame, etc.), similar to replacement scans in the C++ API - without modifying local/global scope.

Description

I would like to have a function like
register_replacement_scan(my_callback)
available in the Python API.

The idea is that my_callback receives a table name and returns whatever type is accepted by register() (e.g. a DataFrame, RecordBatchReader, etc.) or None if it doesn’t apply.

Motivation

I want to pass arguments to my PyArrow RecordBatch/Dataset dynamically.

This feature would serve as a lightweight alternative to table-valued functions - easier to implement, since the current workaround already exists but relies on manipulating the local/global scope.

Current workaround

For example, running this query:

SELECT * FROM 'somefile.myformat';

requires the following workaround in Python:

def my_sql(query):
    # Extract custom table names (e.g. ending with .myformat)
    table_names = parse_query(query, ".myformat")  # regex-based parser
    # Example: for query "select * from 'somefile.myformat'"
    # table_names = ['somefile.myformat']

    for table_name in table_names:
        # batch_reader_factory returns a pa.RecordBatchReader
        locals().update({table_name: batch_reader_factory(table_name)})

    return duckdb_conn.sql(query)

Proposed solution

The same query:

SELECT * FROM 'somefile.myformat';

could be handled much more cleanly with:

def my_callback(table_name):
    if table_name.endswith('.myformat'):
        # Could return a RecordBatchReader, DataFrame, Dataset, etc.
        return batch_reader_factory(table_name)
    return None

duckdb_conn.register_replacement_scan(my_callback)
duckdb_conn.sql(query)

This would allow transparent handling of custom sources or file formats without pre-registering them manually.

Extended proposal

An additional improvement would be to introduce a "reader registry", allowing registration of callable read functions that accept parameters.

This extends the idea of register_replacement_scan() by enabling parameterized SQL functions that internally map to Python callbacks.

Example

def myformat_reader(filename, param_int_1=0, param_bool_2=True):
    return batch_reader_factory(filename, param_int_1, param_bool_2)

duckdb_conn.register_reader('myformat', myformat_reader)

Then the corresponding SQL query could look like:

SELECT * FROM read_myformat('myfile.myformat', param_int_1=2137, param_bool_2=True);

Sumary

Adding register_replacement_scan() and optionally extending it with a reader registry - would make it much easier to:

• handle custom file formats dynamically,
• inject runtime parameters,
• and integrate with data sources like PyArrow without touching the local/global namespace.

[Tishj]

An idea that was mentioned a while back related to this was using a dunder function.

So we'd check for something like __duckdb_replacement__ on an object referenced in the replacement scan path, and call that method

You can then create a wrapper class around your object that contains the parameters+object you want to return

The other idea sounds too far out imo, that would be better reserved for a table udf

[teserak]

Got it, thanks for clarifying.