Skip to content

Metadata filters silently ignore $in and $like operators, returning wrong results #1687

Description

@ariel-formance

Metadata filters accept $in and $like, but every resource handler discards the operator and builds a JSONB containment predicate as if it were $match. Both silently return wrong results — no error, no warning.

Spotted while reviewing #1685; this is pre-existing on main and unrelated to that PR.

Reproduction

Two transactions, metadata.wallet = "w-alpha" and metadata.wallet = "w-beta", queried through Transactions().Paginate on main (c24250b0e):

filter rows expected emitted SQL
$match "w-alpha" 1 1 metadata @> '{"wallet":"w-alpha"}'
$in ["w-alpha","w-beta"] 0 2 metadata @> '{"wallet":["w-alpha","w-beta"]}'
$like "w-%" 0 2 metadata @> '{"wallet":"w-%"}'
  • $in builds containment against a JSON array. A row's metadata value is a string, and a string never contains an array, so the filter matches nothing — for any input.
  • $like puts the pattern in as a literal, turning it into an exact match on w-%. It returns rows only if a value is literally w-%.

Cause

ResolveFilter ignores its operator argument in the metadata[...] branch:

(line numbers on main @ c24250b0e)

  • internal/storage/ledger/resource_transactions.go:147-152
  • internal/storage/ledger/resource_accounts.go:106-111
  • internal/storage/ledger/resource_volumes.go:164-173
  • internal/storage/ledger/resource_aggregated_balances.go:132-141

All four build metadata @> ? from map[string]any{key: value} regardless of operator. I reproduced the transactions case; the other three are the same code shape, so I'd expect them to behave identically.

The operators are reachable because metadata is a NewStringMapField(), and TypeString.Operators() (internal/queries/field.go:130) advertises $match, $like and $in. So validateFilters accepts them and they reach the handler intact.

Suggested fix

Give each operator a correct SQL form in the metadata branch, e.g.

-- $like
metadata ->> 'key' LIKE ?
-- $in
metadata ->> 'key' = ANY(?)

Note both need the same NULL-safety treatment as the indexed path in #1685: metadata ->> 'key' is NULL for absent keys, so under $not a bare comparison excludes rows that NOT (metadata @> ...) includes. Prefixing metadata ? 'key' fixes it — the AND short-circuits to false rather than NULL.

If these operators aren't meant to be supported on metadata, the alternative is to reject them at validation time so callers get an error instead of a silently empty page. Either is better than the current behaviour, but the silent-wrong-answer part should go.

Impact

Any caller filtering metadata with $in or $like gets a silently empty or wrong result set. $in in particular looks like a natural way to fetch several wallets in one query and always returns nothing.

Activity

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

Labels

Type

No type

Projects

No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions