Skip to content
Open
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
7 changes: 4 additions & 3 deletions .gitattributes
Original file line number Diff line number Diff line change
@@ -1,7 +1,8 @@
# Check every text file out with LF everywhere. All tracked files are already LF, so this changes nothing today.
* text=auto eol=lf

# `pnpm generate:types` writes this file with LF, so a CRLF checkout would make every regeneration rewrite all 20k
# lines. `linguist-generated` also collapses it in diffs: what a reviewer reads is `src/models.ts`, which declares
# the published models on top of it.
# `pnpm generate:models` writes these files with LF, so a CRLF checkout would make every regeneration rewrite all
# 20k lines. `linguist-generated` also collapses them in diffs: what a reviewer reads is `src/models.ts` and
# `src/schemas.ts`, which declare the published models and the response schemas on top of them.
src/generated/api.ts linguist-generated=true
src/generated/schemas.ts linguist-generated=true
Original file line number Diff line number Diff line change
@@ -1,7 +1,7 @@
# Keeps `src/generated/api.ts` in sync with the published OpenAPI specification: every night it regenerates the
# types and opens a pull request when the result differs from master. Renovate cannot do this -- the specification
# is a live document, not an npm dependency -- so without this job the generated types never move and none of the
# drift guards in `src/spec_guards.ts` can ever fire.
# Keeps `src/generated/api.ts` and `src/generated/schemas.ts` in sync with the published OpenAPI specification: every
# night it regenerates both and opens a pull request when the result differs from master. Renovate cannot do this --
# the specification is a live document, not an npm dependency -- so without this job the generated models never move
# and none of the drift guards in `src/spec_guards.ts` can ever fire.
#
# Two invariants make it safe to run unattended:
# 1. Generation always happens on master, so the output follows the current specification and the current codegen
Expand All @@ -16,7 +16,7 @@
# renamed, a documented spec gap was filled, or a shared `@apify/consts` enum diverged -- and each of those needs a
# human to decide.

name: Regenerate types
name: Regenerate models

on:
workflow_dispatch:
Expand All @@ -25,7 +25,7 @@ on:
- cron: '0 2 * * *'

concurrency:
group: regenerate-types
group: regenerate-models
cancel-in-progress: false

# Writes go through the service account token below, not through `GITHUB_TOKEN`.
Expand All @@ -34,19 +34,19 @@ permissions:

env:
NODE_VERSION: 24
BRANCH_NAME: ci/regenerate-types
BRANCH_NAME: ci/regenerate-models
# Valid Conventional Commits, so the pull request is mergeable as-is; reviewers retitle it to `fix:`/`feat:`
# when the diff is user-facing.
PR_TITLE: 'chore: regenerate types from the published OpenAPI spec'
PR_TITLE: 'chore: regenerate models from the published OpenAPI spec'
ASSIGNEE: vdusek
LABEL: t-tooling
# The only file the gate looks at. The recorded specification version is committed alongside it but
# The only files the gate looks at. The recorded specification version is committed alongside them but
# deliberately excluded: that stamp moves on every apify-docs deploy regardless of client impact.
GENERATED_FILE: src/generated/api.ts
GENERATED_FILES: src/generated/api.ts src/generated/schemas.ts

jobs:
regenerate-types:
name: Regenerate types
regenerate-models:
name: Regenerate models
runs-on: ubuntu-latest

steps:
Expand Down Expand Up @@ -75,27 +75,28 @@ jobs:
echo "version=$version" >> "$GITHUB_OUTPUT"

# Downloads the specification, generates from it, and records its version in `package.json`.
- name: Regenerate types
run: pnpm generate:types
- name: Regenerate models
run: pnpm generate:models

# Gate on the generated types, not on the specification version: that stamp is a coarse marker
# Gate on the generated models, not on the specification version: that stamp is a coarse marker
# rather than a content identity, for the reasons `scripts/openapi_spec.mts` sets out. Compared
# against HEAD rather than the index, so nothing staged earlier in the job can hide a change.
- name: Check for type changes
- name: Check for model changes
id: changes
run: |
if git diff --quiet HEAD -- "$GENERATED_FILE"; then
echo "Types are already up to date with the published specification."
# Unquoted on purpose: the variable holds a space-separated list of paths.
if git diff --quiet HEAD -- $GENERATED_FILES; then
echo "Models are already up to date with the published specification."
echo "has-changes=false" >> "$GITHUB_OUTPUT"
else
git diff --stat HEAD -- "$GENERATED_FILE"
git diff --stat HEAD -- $GENERATED_FILES
echo "has-changes=true" >> "$GITHUB_OUTPUT"
fi

# A previous run may already have these exact types up for review; leave it alone instead of churning
# an open pull request. Only the generated file is compared - the branch being behind master says
# nothing about whether the types on it are still the right ones.
- name: Check whether the types are already up for review
# A previous run may already have these exact models up for review; leave it alone instead of churning
# an open pull request. Only the generated files are compared - the branch being behind master says
# nothing about whether the models on it are still the right ones.
- name: Check whether the models are already up for review
id: review
if: steps.changes.outputs.has-changes == 'true'
env:
Expand Down Expand Up @@ -131,8 +132,8 @@ jobs:
fi

git fetch origin "$BRANCH_NAME"
if git diff --quiet FETCH_HEAD -- "$GENERATED_FILE"; then
echo "The open pull request already carries these types - nothing to do."
if git diff --quiet FETCH_HEAD -- $GENERATED_FILES; then
echo "The open pull request already carries these models - nothing to do."
echo "is-new=false" >> "$GITHUB_OUTPUT"
else
echo "is-new=true" >> "$GITHUB_OUTPUT"
Expand All @@ -156,17 +157,17 @@ jobs:
echo "Deleted leftover branch $BRANCH_NAME (no open pull request)."
fi

# Creates the branch at the current master commit and lands the regenerated file as one signed
# Creates the branch at the current master commit and lands the regenerated files as one signed
# ("Verified") commit, via GitHub's createCommitOnBranch mutation.
- name: Commit the regenerated types
- name: Commit the regenerated models
id: commit
if: steps.changes.outputs.has-changes == 'true' && steps.review.outputs.is-new == 'true'
uses: apify/actions/signed-commit@v1.4.1
with:
message: ${{ env.PR_TITLE }}
add: >-
package.json
${{ env.GENERATED_FILE }}
${{ env.GENERATED_FILES }}
github-token: ${{ secrets.APIFY_SERVICE_ACCOUNT_GITHUB_TOKEN }}
branch: ${{ env.BRANCH_NAME }}
create-branch: 'true'
Expand All @@ -175,7 +176,7 @@ jobs:
if: steps.commit.outputs.committed == 'true'
env:
GH_TOKEN: ${{ secrets.APIFY_SERVICE_ACCOUNT_GITHUB_TOKEN }}
WORKFLOW_URL: ${{ github.server_url }}/${{ github.repository }}/actions/workflows/regenerate_types.yaml
WORKFLOW_URL: ${{ github.server_url }}/${{ github.repository }}/actions/workflows/regenerate_models.yaml
PREVIOUS_SPEC_VERSION: ${{ steps.previous-spec.outputs.version }}
run: |
SPEC_VERSION=$(node scripts/openapi_spec.mts recorded-version)
Expand All @@ -189,16 +190,16 @@ jobs:
fi

BODY=$(printf '%s\n' \
"- Regenerates \`src/generated/api.ts\` from the [published OpenAPI specification](https://docs.apify.com/api/openapi.json), and records its version in \`package.json\`." \
"- Regenerates \`src/generated/api.ts\` and \`src/generated/schemas.ts\` from the [published OpenAPI specification](https://docs.apify.com/api/openapi.json), and records its version in \`package.json\`." \
"${SPEC_LINE}" \
"" \
"> [!IMPORTANT]" \
"> Retitle this pull request to \`fix:\` or \`feat:\` when the diff is user-facing, so that it lands in the changelog and triggers a release - \`chore:\` does neither." \
"" \
"> [!NOTE]" \
"> A red check here is a guard in \`src/spec_guards.ts\` doing its job: a field an override block replaces was dropped or renamed, a documented spec gap was filled and its \`*SpecGaps\` entry is now stale, or a shared \`@apify/consts\` enum diverged. Each of those is a decision, not a mechanical update." \
"> A red check here is a guard in \`src/spec_guards.ts\` doing its job: a field an override block replaces was dropped or renamed, a documented spec gap was filled and its \`*SpecGaps\` entry is now stale, a shared \`@apify/consts\` enum diverged, or a generated zod schema stopped accepting its generated type. Each of those is a decision, not a mechanical update. A failing \`fixtures.test.ts\` means the specification now requires something the mock fixtures in \`test/mock_server/fixtures.ts\` do not carry." \
"" \
"> Generated by the [Regenerate types](${WORKFLOW_URL}) workflow.")
"> Generated by the [Regenerate models](${WORKFLOW_URL}) workflow.")

gh pr create \
--title "$PR_TITLE" \
Expand All @@ -212,7 +213,7 @@ jobs:
# and no release waits on this workflow. Skipped on manual dispatch so ad-hoc triggers don't spam the channel.
notify_on_failure:
name: Notify Slack on failure
needs: regenerate-types
needs: regenerate-models
if: failure() && github.event_name == 'schedule'
runs-on: ubuntu-latest

Expand All @@ -221,7 +222,7 @@ jobs:
env:
REPO: ${{ github.repository }}
WORKFLOW_URL: ${{ github.server_url }}/${{ github.repository }}/actions/runs/${{ github.run_id }}
HEADING: ':red_circle: Nightly type regeneration failed'
HEADING: ':red_circle: Nightly model regeneration failed'
run: |
jq -n \
--arg repo "${REPO}" \
Expand All @@ -243,7 +244,7 @@ jobs:
},
{
type: "section",
text: { type: "mrkdwn", text: "The generated API types are no longer being kept in sync with the published OpenAPI specification." }
text: { type: "mrkdwn", text: "The generated API models are no longer being kept in sync with the published OpenAPI specification." }
}
]
}' > slack-payload.json
Expand Down
17 changes: 14 additions & 3 deletions CONTRIBUTING.md
Original file line number Diff line number Diff line change
Expand Up @@ -52,9 +52,15 @@ src/
├── apify_client.ts # Main client class (entry point)
├── http_client.ts # Low-level HTTP layer (Axios-based)
├── apify_api_error.ts # Custom error class
├── argument_validation_error.ts # Thrown for arguments that fail their zod schema
├── response_validation_error.ts # Thrown for API responses that fail their zod schema
├── utils.ts # Utility functions
├── models.ts # Published output types, declared on top of the generated ones
├── schemas.ts # Response schemas the clients validate with, on top of the generated ones
├── spec_guards.ts # Compile-time checks tying the hand-written layers to the generated ones
├── generated/
│ └── api.ts # Types generated from the OpenAPI specification (do not edit)
│ ├── api.ts # Types generated from the OpenAPI specification (do not edit)
│ └── schemas.ts # Zod schemas generated from the OpenAPI specification (do not edit)
├── base/
│ ├── api_client.ts # Base for all clients
│ ├── resource_client.ts # Base for single-resource clients
Expand All @@ -71,12 +77,15 @@ test/
├── _helper.ts # Test utilities
└── mock_server/ # Mock API server for testing
├── server.ts
├── fixtures.ts # Spec-shaped response bodies the routes answer with
└── routes/ # Mock API routes

scripts/
├── openapi_spec.mts # Downloads the published OpenAPI specification
├── generate_types.mts # Generates src/generated/api.ts from it
└── spec_transform.mts # Spec postprocessing the generator applies
├── generate_schemas.mts # Generates src/generated/schemas.ts from it
├── schema_emitter.mts # The OpenAPI-to-zod emitter behind generate_schemas.mts
└── spec_transform.mts # Spec postprocessing both generators apply
```

### Key Patterns
Expand All @@ -92,6 +101,8 @@ scripts/

**Date Parsing**: Fields ending in "At" are automatically converted to Date objects.

**Response Validation**: Every response a resource method turns into a typed value is validated against the zod schema generated for it in `src/generated/schemas.ts` (via `src/schemas.ts`, which widens the few the API is known to deviate on). Unknown fields and unknown enum values pass through; anything else that does not match throws a `ResponseValidationError`. The mock server therefore answers with spec-shaped bodies from `test/mock_server/fixtures.ts`, and `fixtures.test.ts` keeps those in step with the schemas.

## Development Workflow

### Available Scripts
Expand All @@ -111,7 +122,7 @@ pnpm tsc-check-tests # TypeScript check test files
pnpm tsc-check-scripts # TypeScript check maintainer scripts

# API specification (needs Node 22.18+, for native TypeScript support)
pnpm generate:types # Regenerate src/generated/api.ts from the published specification
pnpm generate:models # Regenerate src/generated/{api,schemas}.ts from the published specification
pnpm spec:fetch # Only download the specification, into git-ignored tmp/

# Linting & Formatting
Expand Down
25 changes: 25 additions & 0 deletions docs/02_concepts/02_error-handling.md
Original file line number Diff line number Diff line change
Expand Up @@ -45,6 +45,31 @@ try {
}
```

## Invalid responses

After a request succeeds, the client validates the response against a [zod](https://zod.dev) schema generated from the [Apify OpenAPI specification](https://docs.apify.com/api/v2) before returning it. A response that doesn't match what the specification describes throws a <ApiLink to="class/ResponseValidationError">`ResponseValidationError`</ApiLink>. Its `message` names the request and every offending field, and `issues` and `cause` carry the structured detail, the same way `ArgumentValidationError` does.

Fields the specification doesn't describe and enum values it doesn't list pass through, so the client keeps working when the API grows. What the check catches is the API and its specification disagreeing: a missing required field, a different type, or a value outside the documented range. Such a mismatch would otherwise surface later as an `undefined` where the types promise a value. If you run into one, please [report it](https://github.com/apify/apify-client-js/issues).

```js
import { ApifyClient, ResponseValidationError } from 'apify-client';

const client = new ApifyClient({ token: 'MY-APIFY-TOKEN' });

try {
await client.actor('my-actor').get();
} catch (error) {
if (error instanceof ResponseValidationError) {
// Response from GET https://api.apify.com/v2/acts/my-actor does not match the API schema:
// Invalid input: expected string, received null at `name`
console.log(error.message);
console.log(error.issues);
}
}
```

Bodies the specification leaves to you aren't validated: dataset items, key-value store records and logs are returned as they are.

## Retries with exponential backoff

The client automatically retries requests that fail due to network errors, Apify API internal errors (HTTP 500+), or rate limit errors (HTTP 429). By default, the client retries up to 8 times with exponential backoff starting at 500ms.
Expand Down
30 changes: 30 additions & 0 deletions docs/04_upgrading/upgrading_v3.md
Original file line number Diff line number Diff line change
Expand Up @@ -135,3 +135,33 @@ Submitting a request is unchanged: <ApiLink to="class/RequestQueueClient#addRequ
`ScheduleActionRunActorTask.input` was typed as a `string`, and is now the object the specification describes. The same type backs <ApiLink to="class/ScheduleClient#update">`update()`</ApiLink>, so an action that passed its input as a JSON string has to pass the parsed object instead.

For the full per-resource breakdown of what became optional, nullable, newly exposed, or dropped, see the `BREAKING CHANGE` commit footer of [#985](https://github.com/apify/apify-client-js/pull/985).

## Responses are validated against the OpenAPI specification

Every response the client turns into a typed value is now checked against a [zod](https://zod.dev) schema generated from the same specification the types come from, the way the Python client validates its responses with pydantic. A response that doesn't match, whether a missing required field, a different type, or a value outside the documented range, throws a new <ApiLink to="class/ResponseValidationError">`ResponseValidationError`</ApiLink> (exported from `apify-client`) instead of being handed on as if it were what the type claims.

The check is deliberately lenient about growth: fields the specification doesn't describe pass through untouched, and an enum value it doesn't list is accepted too, so a new field or status on the API side isn't an error. What it catches is the API and its specification disagreeing, which previously surfaced as an `undefined` somewhere down the line. If you hit one, the specification is wrong or the API changed, so please [report it](https://github.com/apify/apify-client-js/issues).

```js
import { ApifyClient, ResponseValidationError } from 'apify-client';

const client = new ApifyClient({ token: 'MY-APIFY-TOKEN' });

try {
await client.actor('my-actor').get();
} catch (error) {
if (error instanceof ResponseValidationError) {
console.log(error.message);
// Response from GET https://api.apify.com/v2/acts/my-actor does not match the API schema:
// Invalid input: expected string, received null at `name`
console.log(error.issues); // [{ code: 'invalid_type', expected: 'string', path: ['name'], ... }]
}
}
```

Bodies the specification leaves to you aren't validated: dataset items, key-value store records and logs are returned as before.

Two return types change as a result of describing what the endpoints really return:

- <ApiLink to="class/ScheduleClient#getLog">`ScheduleClient.getLog()`</ApiLink> was typed as a `string`, even though the endpoint returns the log as a list of entries. It's now typed as <ApiLink to="interface/ScheduleInvoked">`ScheduleInvoked[]`</ApiLink>, each entry carrying `message`, `level` and `createdAt`.
- <ApiLink to="interface/TaskPublicConfig">`TaskPublicConfig`</ApiLink> now follows the specification: `publishedAt` is optional and read-only, and `categorization`, which the specification doesn't describe, is gone from the type.
Loading
Loading