From 408678ef1daaf2fb6418ebf01f69f0ee88b8a93a Mon Sep 17 00:00:00 2001 From: =?UTF-8?q?Adri=C3=A1n=20Chaves?= Date: Mon, 4 Aug 2025 10:35:58 +0200 Subject: [PATCH] WIP --- CHANGES.rst | 19 +++++++++++++++++++ docs/reference/settings.rst | 36 +++++++++++++++++++++++++++++++----- docs/usage/session.rst | 2 +- 3 files changed, 51 insertions(+), 6 deletions(-) diff --git a/CHANGES.rst b/CHANGES.rst index a0ad3c87..fd749999 100644 --- a/CHANGES.rst +++ b/CHANGES.rst @@ -1,6 +1,25 @@ Changes ======= +0.31.0 (unreleased) +------------------- + +- Added a minimum delay between reuses of any given :ref:`managed session + `. + + It is 1 second by default. Use :setting:`ZYTE_API_SESSION_DELAY` to change + that, :setting:`ZYTE_API_SESSION_POOLS` to override it for specific + :setting:`session pools `. + +- :attr:`SessionConfig.pool ` can now + return a dictionary instead of a string, allowing session configs to + override :setting:`ZYTE_API_SESSION_POOLS` and + :setting:`ZYTE_API_SESSION_DELAY` for the corresponding pool. + +- Deprecated the ``ZYTE_API_SESSION_POOL_SIZES`` setting in favor of the new + :setting:`ZYTE_API_SESSION_POOLS` setting, where you can set ``"size"``. + + 0.30.0 (2025-05-13) ------------------- diff --git a/docs/reference/settings.rst b/docs/reference/settings.rst index 304ece35..a019dbad 100644 --- a/docs/reference/settings.rst +++ b/docs/reference/settings.rst @@ -407,6 +407,24 @@ object, for example to read settings: ZYTE_API_SESSION_CHECKER = MySessionChecker +.. setting:: ZYTE_API_SESSION_DELAY + +ZYTE_API_SESSION_DELAY +====================== + +Default: ``1.0`` + +Minimum number of seconds to wait before reusing a :ref:`scrapy-zyte-api +session `. + +To override this value for specific pools, use +:setting:`ZYTE_API_SESSION_POOLS`. + +Increasing this number can reduce the number of ban-related session +expirations, hence increasing the lifetime of each session. See +:ref:`optimize-sessions`. + + .. setting:: ZYTE_API_SESSION_ENABLED ZYTE_API_SESSION_ENABLED @@ -535,22 +553,30 @@ The maximum number of active :ref:`scrapy-zyte-api sessions ` to keep per :ref:`pool `. To override this value for specific pools, use -:setting:`ZYTE_API_SESSION_POOL_SIZES`. +:setting:`ZYTE_API_SESSION_POOLS`. Increase this number to lower the frequency with which requests are sent through each session, which on some websites may increase the lifetime of each session. See :ref:`optimize-sessions`. -.. setting:: ZYTE_API_SESSION_POOL_SIZES +.. setting:: ZYTE_API_SESSION_POOLS -ZYTE_API_SESSION_POOL_SIZES -=========================== +ZYTE_API_SESSION_POOLS +====================== Default: ``{}`` :class:`dict` where keys are :ref:`pool ` IDs and values are -overrides of :setting:`ZYTE_API_SESSION_POOL_SIZE` for those pools. +dicts with any combination of the following keys that override the +corresponding setting for that pool: + +- ``"delay"`` overrides :setting:`ZYTE_API_SESSION_DELAY`. + +- ``"size"`` overrides :setting:`ZYTE_API_SESSION_POOL_SIZE`. + +These override take precedence over :attr:`SessionConfig.pool +`. .. setting:: ZYTE_API_SESSION_QUEUE_MAX_ATTEMPTS diff --git a/docs/usage/session.rst b/docs/usage/session.rst index f0b2de28..247352a0 100644 --- a/docs/usage/session.rst +++ b/docs/usage/session.rst @@ -247,7 +247,7 @@ overrides `. The :setting:`ZYTE_API_SESSION_POOL_SIZE` setting determines the desired number of concurrent, active, working sessions per pool. The -:setting:`ZYTE_API_SESSION_POOL_SIZES` setting allows defining different values +:setting:`ZYTE_API_SESSION_POOLS` setting allows defining different values for specific pools. .. _pool-size: