Skip to content

fix: 674 fix dataset naming - #23

Merged
daveomri merged 9 commits into
apify:fix/668-pipedream-retest-fixesfrom
daveomri:fix/674-fix-dataset-naming
Aug 18, 2026
Merged

fix: 674 fix dataset naming#23
daveomri merged 9 commits into
apify:fix/668-pipedream-retest-fixesfrom
daveomri:fix/674-fix-dataset-naming

Conversation

@daveomri

@daveomri daveomri commented Jul 20, 2026

Copy link
Copy Markdown
Collaborator

WHY

The Get Dataset Items action couldn't reference a dataset by name, silently accepted invalid limit values, showed misleading placeholder defaults, and paginated inefficiently.

Closes 674

Bug A: Dataset ID couldn't be referenced by name

The Dataset ID field was described only as "The ID of the dataset", and the dropdown passed the raw ID. The Apify API path accepts Dataset ID or username~dataset-name, but the tilde form was undocumented and unintuitive.

Now:

  • Verified against the [Apify docs](https://docs.apify.com/api/v2/dataset-items-get) that the path accepts both a raw ID and username~dataset-name, and that apify-client forwards the value as-is.
  • The description advertises username/dataset-name, and run() normalizes /~, so all three forms work: raw ID, username/dataset-name, and username~dataset-name.

Bug B: Limit accepted 0 and negatives; misleading placeholder defaults

limit had no lower bound (0 or negative silently returned nothing), and both limit (default: 100) and offset (default: 0) rendered as pre-filled values that looked intentional.

Now:

  • limit has min: 1; offset has min: 0 defensively.
  • Removed the defaults so both fields render empty; descriptions state that empty limit = all items and empty offset = start from the first item.

Bug C: Inefficient pagination (latent)

The fetch loop hardcoded page size to LIMIT (user's limit was a post-hoc cap), and while (total) did a gratuitous extra empty fetch before terminating.

Now:

  • run() rewritten: offset empty → 0; limit set → paginate in pages of min(LIMIT, remaining) capped at the limit; limit empty → return all, terminating on a short or empty page. No post-hoc truncation, no extra empty fetch, no infinite loop.

What changed

  • components/apify/apify.app.mjs: datasetId description advertises username/dataset-name; limit gets min: 1 and loses default: LIMIT; offset gets min: 0 and loses default: 0; descriptions reworded for empty = all / empty = first. (get-dataset-items is the only consumer.)
  • components/apify/actions/get-dataset-items/get-dataset-items.mjs: separator normalization (/~), pagination loop rewritten for optional limit/offset; 0.0.6 → 0.0.7.
  • components/apify/actions/get-dataset-items/get-dataset-items.test.mjs: 12 unit tests covering pagination (limit cap, limit > dataset, empty-limit paginate-all, exact-multiple termination, empty dataset), offset passthrough and empty default, dataset ID normalization, clean/fields/omit forwarding, $summary wording, and propDef metadata.

Testing

  • eslint clean.
  • 12 unit tests pass (via a local jest override — the root jest.config.js is broken pre-existing).
  • Verified via pd dev/pd publish against a live Apify account: username/dataset-name resolves; limit of 1 returns one item; empty limit paginates past 100 and returns all; offset skips correctly.

Before

obrazek

Now

obrazek

@daveomri
daveomri marked this pull request as draft July 21, 2026 07:35
@daveomri
daveomri marked this pull request as ready for review July 29, 2026 16:54
@daveomri
daveomri changed the base branch from develop to fix/668-pipedream-retest-fixes August 18, 2026 12:33
@daveomri daveomri self-assigned this Aug 18, 2026
@daveomri
daveomri merged commit e74ad6a into apify:fix/668-pipedream-retest-fixes Aug 18, 2026
drobnikj pushed a commit that referenced this pull request Sep 3, 2026
…edreamHQ#21491)

* feat(gmail): eval-driven fixes to the AI-optimized Gmail actions

Iterated against the MCP eval suite (pd-connect-eval-monster/evals/gmail);
this ships the fixes those evals surfaced. Confirming run: 29/31 pass^2 on
Sonnet 5 (multi-step 12/12), and a Haiku 4.5 + Sonnet 5 sweep flags no failure
as a tool problem — every remaining failure passes on at least one model.

- send-email [patch]: eval #16 ("reply so both messages stay in one
  conversation") returned HTTP 400 "Recipient address required" on EVERY reply
  that passed inReplyToMessageId without an explicit `to`. getOptionsToSendEmail
  only populated opts.to on the replyAll branch, so the documented "only the
  original sender is addressed" behavior was never implemented. Now defaults to
  the replied-to message's Reply-To/From. Also covers a replyAll self-thread
  case where filtering the user's own address left zero recipients, and stops
  a missing To/Cc header throwing. #16 now passes.

- create-draft [patch]: same defect via the shared app file. #17 ("draft a
  reply to Hammond") now passes.

- find-email [minor]: eval #23 ("catch me up on this week's mail and email me
  the summary") produced a confidently wrong digest — it described an offline
  fence sensor as healthy. The agent had correctly asked for format:"full" plus
  payload; the 42k-char response exceeded the MCP client's 25k-token result
  ceiling and was spilled to a file, and the in-tool budget then stripped the
  requested field, leaving 197-char snippets. The needed sentence sits at
  character 670 of a 934-character body, so no snippet could contain it and
  nothing signalled that it had been cut. Adds `bodyText` (decoded plain text,
  ~half the size of raw payload), a `bodyChars` cap with bodyTruncated /
  bodyTotalChars, snippetTruncated on metadata results, lowers the response cap
  to 30k chars, and inverts the budget rule: a field the caller explicitly named
  is never stripped — whole messages are dropped instead and the note says how
  many of how many are shown. #23 now passes; largest tool result is 45% of the
  ceiling. All new props are optional; omitting them reproduces prior output.

- list-thread-messages [minor]: same result-ceiling risk; response cap lowered
  to 30k chars. It already degraded full -> metadata -> tail, so it needed the
  cap rather than a fields projection.

- modify-labels [patch]: three description defects the evals surfaced — the
  agent skipped the tool in 20% of tasks that required it, trashed an entire
  label's worth of mail with no confirmation (PipedreamHQ#31), and believed it could create
  filter rules for future mail (#27). Adds "use this whenever the user says
  star / archive / file / label / trash / mark read", a trash-is-destructive
  confirmation rule, and a "not for filters or future mail" exclusion. Tool
  recall across the suite went 0.88 -> 1.00.

App package.json bumped 2.1.0 -> 2.2.0 (largest segment applied: minor).

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>

* chore(gmail): bump every component that bundles the changed shared code

The "Ensure component commits modify component versions" check flagged this and
it is correct: gmail.app.mjs and common/utils.mjs are bundled into every gmail
component at publish time, so each one has to be revved for the registry to pick
up the new code — even where its own behavior is unchanged.

Patch bump for the 12 actions and 5 sources that were not already versioned in
the previous commit. No functional change in any of them; the reply-recipient
fix in gmail.app.mjs only reaches send-email and create-draft (already bumped),
and the utils additions are new functions with no existing call sites touched.

package.json stays at 2.2.0 — the largest applied segment is still the minor
bump from find-email / list-thread-messages.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>

* fix(gmail): address review findings on the eval-driven changes

- find-email: requesting `fields: ["payload"]` without also setting
  `format: "full"` fetched metadata, whose payload carries only headers — no
  `parts`, no `body.attachmentId`. The caller got a payload-shaped object with
  none of the attachment data it asked for. `fetchFull` now keys off
  `wantsPayload`, so either route to a payload forces the full fetch. Verified:
  that call now fetches full and returns the attachment id; default,
  `bodyText`, and `format: "full"` paths are unchanged.

- gmail.app.mjs: a present-but-blank `Reply-To` defeated the reply recipient
  fallback. `??` only guards null/undefined, so an empty header left
  originalSender = "" and the fallback dead — reintroducing the HTTP 400
  "Recipient address required" that fallback exists to prevent. The header
  helper now treats blank/whitespace-only values as absent. Verified across
  blank, whitespace, valid, and missing Reply-To.

- gmail.app.mjs: reply-all Cc'd the authenticated user back on their own reply.
  `opts.to` had a self-filter, `opts.cc` did not. Extracted the existing filter
  and applied it to both.

- utils.mjs: MAX_RESPONSE_CHARS was declared identically in find-email and
  list-thread-messages. It is a property of the MCP client, not of either
  action, so both copies had to move in lockstep. Now exported once from
  common/utils.mjs and imported by both.

Not changed: reply-all still propagates the original message's Bcc. That
predates this PR, only fires on replies to a sent copy (Gmail strips Bcc from
received mail), and removing it is a behavior change to published semantics
with no eval covering either side — it belongs in its own change, not a
review fix-up. Flagged for the reviewer.

No version bumps: every touched component is already bumped relative to master
in this PR and none of these versions has been released.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>

---------

Co-authored-by: Dylan Sather <Dylan Sather>
Co-authored-by: Claude Opus 5 (1M context) <noreply@anthropic.com>
Co-authored-by: michelle0927 <michelle0927@users.noreply.github.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

4 participants