Feat: Implement server-side pagination and advanced database filtering by wolgwang1729 · Pull Request #431 · shopstr-eng/shopstr

wolgwang1729 · 2026-04-18T18:18:28Z

Description

Added server side pagination and filtering to improve marketplace performance and user experience.

Server-Side Pagination: Implemented limit and offset logic in fetchAllProductsFromDb and the /api/db/fetch-products endpoint to load products in batches.
Reporting Total Counts: The API now returns a total field alongside the product events, enabling accurate pagination UI and "Load More" behavior.
Advanced Filtering: Integrated search, categories, location, and pubkey filters directly into database queries for faster results and reduced client-side processing.
Hybrid Fetching Strategy: Refactored fetchProducts to immediately display results from the database while fetching fresh data from Nostr relays in the background.
Database Caching Updates: Updated the caching policy to include kind 1 (zapsnag) events tagged as products. This ensures that the server-side total product count remains accurate for all listing types, which is critical for the reliable behavior of the "Load More" button and pagination UI.
Improved UI Performance: Reduced initial load times by fetching smaller, paginated batches instead of entire datasets.
Optimal Layout Limit: Increased the default page limit from 42 to 60. This change ensures perfectly filled rows across all screen sizes (serving as the lowest common multiple for 2, 3, 4, 5, and 6 column layouts), as I noted in my previous comment.

Affirmation

My code follows the CONTRIBUTING.md guidelines

…base queries and relay fetches

vercel · 2026-04-18T18:18:33Z

@wolgwang1729 is attempting to deploy a commit to the shopstr-eng Team on Vercel.

A member of the Team first needs to authorize it.

wolgwang1729 · 2026-04-18T19:17:57Z

I have performed a benchmark test comparing the legacy database-fetching logic against the new server-side pagination and filtering implementation.

Scenario	Legacy (Before)	New (After)	Impact
Initial Load (Default 500)	296ms	356ms	+60ms latency
Small Page (Limit 50)	51ms	84ms	+33ms latency
Filtered Search	45ms (Returned unfiltered)	50ms (Returned accurate)	Correct filtering
Category Filter	47ms (Returned unfiltered)	56ms (Returned accurate)	Correct filtering
High Offset (Paginated)	N/A (Returned 0)	91ms (Returned accurate)	Pagination support

The minor increase in response time (averaging 30-60ms) is primarily due to the introduction of server-side data validation and the new requirement for reliable pagination metrics. Specifically, the database now performs parallel EXISTS checks to filter out listings without images or prices ensuring higher data quality, while also calculating a total count of unique products across both kind 30402 and kind 1 types. While this adds some server-side overhead, it significantly improves the discovery of valid products (finding approximately 60 listings where only 42 were previously visible) and eliminates the need for the browser to fetch and filter massive, potentially broken datasets.

GautamBytes

LGTM NOW!!

wolgwang1729 · 2026-04-20T09:37:10Z

Curious about the hardcoded block for 3da2082b7aa5b76a8f0c134deab3f7848c3b5e3a3079c65947d88422b69c1755 in display-products.tsx. Was this added to handle a specific spam issue?"

GautamBytes · 2026-04-20T10:31:59Z

Curious about the hardcoded block for 3da2082b7aa5b76a8f0c134deab3f7848c3b5e3a3079c65947d88422b69c1755 in display-products.tsx. Was this added to handle a specific spam issue?"

It was already in that file’s client-side filter, and I turned it into a named constant in the same file so the new server-filter path could reuse the exact same rule. Check line around 158 in this file

wolgwang1729 · 2026-04-20T12:38:40Z

Curious about the hardcoded block for 3da2082b7aa5b76a8f0c134deab3f7848c3b5e3a3079c65947d88422b69c1755 in display-products.tsx. Was this added to handle a specific spam issue?"

It was already in that file’s client-side filter, and I turned it into a named constant in the same file so the new server-filter path could reuse the exact same rule. Check line around 158 in this file

I meant why it was added in the first place. But I've figured it out. It was added in the commit 95221de as a temp fix to prevent adult content from coming up on the feed.

wolgwang1729 · 2026-04-20T13:26:59Z

TODO(after the PR is merged):

The pagination count display ("Showing X to Y of Z") stops updating after the initial page load (e.g., "Showing 1 to 60 of 497"). Previously, with client-side relay loading, the count would progressively increase as more products streamed in. With server-side pagination, we should consider updating the UI to always reflect totalEvents as the total without implying all products are loaded locally.
Fix the issue of "loading more products" flicker as demonstrated in the following video.

loading.more.products.flicker.mp4

Copilot

Pull request overview

Note

Copilot was unable to run its full agentic suite in this review.

Implements server-side pagination, total counts, and advanced filtering for marketplace product listings, while refactoring the fetch flow to show cached DB results immediately and refresh from relays in the background.

Changes:

Added shared filter types and propagated filter/total-count support through context, UI, and fetch service.
Updated DB querying to support paging + filtering and added a total-count query used by /api/db/fetch-products.
Expanded caching/schema rules to include kind 1 (“zapsnag”) events as products.

Reviewed changes

Copilot reviewed 10 out of 10 changed files in this pull request and generated 9 comments.

Show a summary per file

File	Description
utils/types/types.ts	Adds `FilterParams` type for shared filter plumbing
utils/nostr/fetch-service.ts	Hybrid DB-first + relay refresh fetching, plus filter forwarding
utils/db/db-service.ts	Adds filtered/paginated product query + total count helper; updates kind handling
utils/db/cache-event-policy.ts	Caches kind `1` events
utils/context/context.ts	Extends ProductContext with totals + paging/refresh methods
pages/api/db/fetch-products.ts	Adds filter parsing and returns `{ events, total }`
pages/_app.tsx	Implements `loadMoreProducts` / `refreshProducts` and total tracking
components/display-products.tsx	Uses server totals and triggers server-side refresh/load-more
components/tests/display-products.test.tsx	Updates mocked ProductContext shape
tests/pages/api/db/fetch-products.test.ts	Adds API handler coverage for parsing + response shape

💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.

+        let apiUrl = "/api/db/fetch-products";
+        const queryParams = new URLSearchParams();
+        if (until) {
+          queryParams.append("until", until.toString());
        }
-        return event.id;
-      };
+        if (filters?.search) {
+          queryParams.append("search", filters.search);
+        }
+        if (filters?.categories && filters.categories.length > 0) {
+          queryParams.append("categories", filters.categories.join(","));
+        }
+        if (filters?.location) {
+          queryParams.append("location", filters.location);
+        }
+        appendFilterParam(queryParams, "pubkey", filters?.pubkey);
+        appendFilterParam(
+          queryParams,
+          "excludePubkeys",
+          filters?.excludePubkeys
+        );


+      const requestedAuthors =
+        filters?.pubkey === undefined
+          ? undefined
+          : Array.isArray(filters.pubkey)
+            ? filters.pubkey
+            : [filters.pubkey];
+      const excludePubkeys = new Set(filters?.excludePubkeys ?? []);
+      const shouldFetchRelayEvents =
+        requestedAuthors === undefined || requestedAuthors.length > 0;


+    // Migration: Update product_events_kind_check constraint
+    await client.query(`
+      ALTER TABLE product_events DROP CONSTRAINT IF EXISTS product_events_kind_check;
+      ALTER TABLE product_events ADD CONSTRAINT product_events_kind_check CHECK (kind IN (30402, 1));


+        const oldestEvent = [...prev.productEvents].sort(
+          (a, b) => a.created_at - b.created_at
+        )[0];
+        if (!oldestEvent) return prev;


+                Showing {(currentPage - 1) * itemsPerPage + 1} to{" "}
+                {Math.min(totalEvents, currentPage * itemsPerPage)} of{" "}
+                {totalEvents} products


+      const dbEventKeys = new Set<string>();
+
+      for (const event of productArrayFromDb) {
+        if (event && event.id) {
+          const key = getEventKey(event);
+          dbEventKeys.add(key);


+  }
+
+  if (filters?.search) {
+    whereClause += ` AND (${alias}.content ILIKE $${paramIndex} OR ${alias}.tags::text ILIKE $${paramIndex++})`;


+      const filter30402: any = {
+        kinds: [30402], // Listings
+        limit: 500,
      };

-      const zapsnagFilter: Filter = {
-        kinds: [1],
-        "#t": ["shopstr-zapsnag", "zapsnag"],
+      const filter1: any = {
+        kinds: [1], // Zapsnag notes
+        "#t": ["zapsnag", "shopstr-zapsnag"],
+        limit: 500,
      };


+  // Refresh products when filters change (server-side)
+  useEffect(() => {
+    if (!isInitialized) return;
+    const filters = getServerFilters();
+    if (!filters) return;
+
+    if (Array.isArray(filters.pubkey) && filters.pubkey.length === 0) {
+      productEventContext.setProductEvents([], 0);
+      return;
+    }
+
+    productEventContext.refreshProducts(filters);
+  }, [


wolgwang1729 and others added 3 commits April 18, 2026 17:34

feat: implement server-side filtering and pagination for product data…

7831d7c

…base queries and relay fetches

Merge branch 'shopstr-eng:main' into add/products-pagination

5705e50

prettier

54101ff

wolgwang1729 marked this pull request as draft April 18, 2026 18:32

wolgwang1729 marked this pull request as ready for review April 18, 2026 19:18

fix: align paginated marketplace filters

5a46abf

GautamBytes approved these changes Apr 19, 2026

View reviewed changes

GautamBytes added the merge-ready label Apr 19, 2026

prettier

2246d3b

Copilot AI review requested due to automatic review settings May 2, 2026 03:45

Copilot AI reviewed May 2, 2026

View reviewed changes

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Feat: Implement server-side pagination and advanced database filtering#431

Feat: Implement server-side pagination and advanced database filtering#431
wolgwang1729 wants to merge 5 commits into
shopstr-eng:mainfrom
wolgwang1729:add/products-pagination

wolgwang1729 commented Apr 18, 2026

Uh oh!

vercel Bot commented Apr 18, 2026

Uh oh!

wolgwang1729 commented Apr 18, 2026

Uh oh!

GautamBytes left a comment

Uh oh!

wolgwang1729 commented Apr 20, 2026

Uh oh!

GautamBytes commented Apr 20, 2026

Uh oh!

wolgwang1729 commented Apr 20, 2026

Uh oh!

wolgwang1729 commented Apr 20, 2026

Uh oh!

Copilot AI left a comment

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

Conversation

wolgwang1729 commented Apr 18, 2026

Description

Affirmation

Uh oh!

vercel Bot commented Apr 18, 2026

Uh oh!

wolgwang1729 commented Apr 18, 2026

Uh oh!

GautamBytes left a comment

Choose a reason for hiding this comment

Uh oh!

wolgwang1729 commented Apr 20, 2026

Uh oh!

GautamBytes commented Apr 20, 2026

Uh oh!

wolgwang1729 commented Apr 20, 2026

Uh oh!

wolgwang1729 commented Apr 20, 2026

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

Pull request overview

Reviewed changes

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants