Pre-UDF table logic creates unnecessary copies

[rlamy]

The pre-UDF logic in

datachain/src/datachain/query/dataset.py

Lines 589 to 598 in ee43fd1

	def process_input_query(self, query: Select) -> tuple[Select, list["Table"]]:
	if os.getenv("DATACHAIN_DISABLE_QUERY_CACHE", "") not in ("", "0"):
	return query, []
	table = self.catalog.warehouse.create_pre_udf_table(query)
	q: Select = sqlalchemy.select(*table.c)
	if query._order_by_clauses:
	# we are adding ordering only if it's explicitly added by user in
	# query part before adding signals
	q = q.order_by(table.c.sys__id)
	return q, [table]

unconditionally copies the input query into a new table, which is expensive and useless in most cases.

For context, this was introduced in https://github.com/iterative/dvcx/pull/1068

[ilongin]

How will we overcome the issue that was solved in that https://github.com/iterative/dvcx/pull/1068 you mentioned if we are not going to copy query to temp table?

[rlamy]

IIUC, that issue only happens when the result-set of the query is non-deterministic, i.e. only when there's a non-deterministic filter on the rows (such as LIMIT). So I can see 2 solutions:

• Detect deterministic cases and avoid copying in that case.
• Rely on row ids to avoid running the filter twice: create the UDF table based on the original query, and then inner-join on row ids with an unfiltered version of the query.