Skip to content

Conversation

benjaminwil
Copy link
Contributor

@benjaminwil benjaminwil commented Oct 11, 2024

Summary

This pull request proposes a replacement to the default Spree::OrderUpdater that has new and improved functionality:

  • Increased performance due to the updater making fewer write calls to the database.
  • A built-in update simulator, so that changes to an order can be previewed before persisting them.

There may be some beneficial side-effects that come out of this new order updater implementation:

  • Significantly faster order factories, and thus a significantly faster test suite for Solidus gems and Solidus applications.

We don't expect this to be the default order updater implementation in the next minor version of Solidus, but we would like to propose it as the default for the next major version of Solidus.

Note: The commits on this pull request have a long list of co-authors, as the Super Good team is approaching this as a collaborative mob programming exercise.

Milestones

For this order updater, we intend to achieve the following during updates:

  1. Don't perform writes to the database.
  2. Preload associated records to eliminate reads required.

We appreciate that there is a lot of complexity to achieving these goals (dealing with active promotions, for example).

Notes

  • We should provide more context about performance gains in this PR description. It would be insightful to include the actual number of writes the current Spree::OrderUpdater makes on a typical recalculate.
  • We should further explain why a production store may want to use an order update simulator so it's clear why this feature is worth having. For now, I'll just say that we have worked with stores who could benefit from this feature. Sometimes admins must make significant changes to completed orders, and it would be valuable for them to preview a set of changes before submitting them and causing updates on many order-associated tables.

Todo

  • Add additional cases to item_total_updater_spec (doesn't currently account for included adjustments)
  • Consider Sofia's recommendation to break this class into POROs to simplify testing
  • Add test coverage for recalculate_item_total when line item totals change
  • Scope handling of tax adjustments in InMemoryOrderUpdater to not marked for destruction
  • Scope handling of tax adjustments in OrderUpdater to not marked for destruction
  • Ensure order-level tax adjustments (like Colorado delivery) are scoped out of tax total and adjustment total calculations
  • Handle persistence in update_taxes
  • Write the InMemoryOrderAdjuster (also, should we rename this to InMemoryOrderPromotionAdjuster)
  • Fix CI failures from previous session (if any)
  • Add high-level integration test for legacy promotion system
  • Verify if it is a breaking change to call (Mar 28, 2025)
    LineItem#set_required_attributes in after_initialize instead of
    before_validation. We think this may be a breaking change that requires
    further investigation. (We investigated and don't believe it is a breaking change.)
  • Add high level test for manipulative queries around new Promotion system
    • (Optional) Add shared examples that could be used in both promotion system gems to
      ensure the above
  • (In Progress) Handle persistence in all implementations of promotions.order_adjuster_class
    • Follow up on any failing test relating to change in promotion chooser
    • Ensure adjustments are marked for destructions instead of destroyed
    • Continue on with new promotion system similar change
      • DiscountOrder
  • Investigate whether we accidentally created a breaking change to Spree::LineItem#set_required_attributes
  • Investigate if any promotion actions/benefits write to the database when calling compute_amount. We know the create quantity adjustments does. This action persists when compute_amount is called.
    • Promotion Actions (Legacy Promotion System)
    • Benefits (New Promotion System)
  • (In Progress) Add the ability to raise an error (or log it) if something tries to persist data during a call to the in-memory order updater with persist: false

Checklist

Check out our PR guidelines for more details.

The following are mandatory for all PRs:

The following are not always needed:

  • 📖 I have updated the README to account for my changes.
  • 📑 I have documented new code with YARD.
  • 🛣️ I have opened a PR to update the guides.
  • ✅ I have added automated tests to cover my changes.
  • 📸 I have attached screenshots to demo visual changes.

@github-actions github-actions bot added the changelog:solidus_core Changes to the solidus_core gem label Oct 11, 2024
@benjaminwil benjaminwil changed the title In memory order updater In-memory order updater Oct 11, 2024
@benjaminwil benjaminwil force-pushed the in-memory-order-updater branch 2 times, most recently from 27f19a8 to 340032c Compare October 11, 2024 22:19
@forkata forkata force-pushed the in-memory-order-updater branch from d22210b to d244646 Compare November 8, 2024 22:34
@jarednorman
Copy link
Member

Just making a note that we are waiting on Alistair to rebase #6026 against this.

@AlistairNorman could you just make sure that gets done at some point before the 20th? That way we can have it for our next session.

@AlistairNorman AlistairNorman force-pushed the in-memory-order-updater branch from a3bb1fc to 06e3a2a Compare December 19, 2024 18:27
@stewart stewart force-pushed the in-memory-order-updater branch from 9fd5c8d to 27e4988 Compare December 20, 2024 21:30
@jarednorman jarednorman force-pushed the in-memory-order-updater branch from 4a0f623 to 90f5420 Compare January 17, 2025 22:03
jarednorman added a commit to SuperGoodSoft/solidus that referenced this pull request Jan 17, 2025
While working on the in-memory updater in solidusio#5872, we found the need to
change how item totals were being calculated, so that we could mark
adjustments for destruction without actually destroying them, while
still keeping tax adjustments intact. This change is completely
backwards-compatible with the current OrderUpdater, so to reduce the
scope of our PR, we wanted to make this change separately.

Since the OrderUpdater is already very large, this helps reduce its
responsibilities and makes it easier to test this behaviour. We don't
see it as necessary to make this a configurable class, but this change
leaves that option open in the future.

Co-authored-by: Adam Mueller <[email protected]>
Co-authored-by: Alistair Norman <[email protected]>
Co-authored-by: Chris Todorov <[email protected]>
Co-authored-by: Harmony Bouvier <[email protected]>
Co-authored-by: Sofia Besenski <[email protected]>
Co-authored-by: benjamin wil <[email protected]>
jarednorman added a commit to SuperGoodSoft/solidus that referenced this pull request Jan 17, 2025
While working on the in-memory updater in solidusio#5872, we found the need to
change how item totals were being calculated, so that we could mark
adjustments for destruction without actually destroying them, while
still keeping tax adjustments intact. This change is completely
backwards-compatible with the current OrderUpdater, so to reduce the
scope of our PR, we wanted to make this change separately.

Since the OrderUpdater is already very large, this helps reduce its
responsibilities and makes it easier to test this behaviour. We don't
see it as necessary to make this a configurable class, but this change
leaves that option open in the future.

Co-authored-by: Adam Mueller <[email protected]>
Co-authored-by: Alistair Norman <[email protected]>
Co-authored-by: Chris Todorov <[email protected]>
Co-authored-by: Harmony Bouvier <[email protected]>
Co-authored-by: Sofia Besenski <[email protected]>
Co-authored-by: benjamin wil <[email protected]>
adammathys added a commit to SuperGoodSoft/solidus that referenced this pull request Jan 30, 2025
While working on the in-memory updater in solidusio#5872, we found the need to
change how item totals were being calculated, so that we could mark
adjustments for destruction without actually destroying them, while
still keeping tax adjustments intact. This change is completely
backwards-compatible with the current OrderUpdater, so to reduce the
scope of our PR, we wanted to make this change separately.

Since the OrderUpdater is already very large, this helps reduce its
responsibilities and makes it easier to test this behaviour. We don't
see it as necessary to make this a configurable class, but this change
leaves that option open in the future.

Co-authored-by: Adam Mueller <[email protected]>
Co-authored-by: Alistair Norman <[email protected]>
Co-authored-by: Chris Todorov <[email protected]>
Co-authored-by: Harmony Bouvier <[email protected]>
Co-authored-by: Sofia Besenski <[email protected]>
Co-authored-by: benjamin wil <[email protected]>
adammathys added a commit to SuperGoodSoft/solidus that referenced this pull request Jan 31, 2025
While working on the in-memory updater in solidusio#5872, we found the need to
change how item totals were being calculated, so that we could mark
adjustments for destruction without actually destroying them, while
still keeping tax adjustments intact. This change is completely
backwards-compatible with the current OrderUpdater, so to reduce the
scope of our PR, we wanted to make this change separately.

Since the OrderUpdater is already very large, this helps reduce its
responsibilities and makes it easier to test this behaviour. We don't
see it as necessary to make this a configurable class, but this change
leaves that option open in the future.

Co-authored-by: Adam Mueller <[email protected]>
Co-authored-by: Alistair Norman <[email protected]>
Co-authored-by: Chris Todorov <[email protected]>
Co-authored-by: Harmony Bouvier <[email protected]>
Co-authored-by: Sofia Besenski <[email protected]>
Co-authored-by: benjamin wil <[email protected]>
@adammathys adammathys force-pushed the in-memory-order-updater branch from 9ecbd13 to 92fc679 Compare January 31, 2025 22:42
@github-actions github-actions bot added changelog:solidus_legacy_promotions Changes to the solidus_legacy_promotions gem changelog:solidus_promotions Changes to the solidus_promotions gem labels Jan 31, 2025
Copy link

codecov bot commented Jan 31, 2025

Codecov Report

❌ Patch coverage is 83.78378% with 6 lines in your changes missing coverage. Please review.
✅ Project coverage is 92.71%. Comparing base (839752f) to head (e9809e2).

Files with missing lines Patch % Lines
..._promotions/spree_in_memory_order_updater_patch.rb 72.22% 5 Missing ⚠️
...dus_legacy_promotions/spree_order_updater_patch.rb 80.00% 1 Missing ⚠️
Additional details and impacted files
@@            Coverage Diff             @@
##             main    #5872      +/-   ##
==========================================
+ Coverage   89.35%   92.71%   +3.35%     
==========================================
  Files         961      399     -562     
  Lines       20195     8101   -12094     
==========================================
- Hits        18046     7511   -10535     
+ Misses       2149      590    -1559     

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:
  • ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.
  • 📦 JS Bundle Analysis: Save yourself from yourself by tracking and limiting bundle sizes in JS merges.

@AlistairNorman AlistairNorman force-pushed the in-memory-order-updater branch from 92fc679 to e7faef3 Compare February 7, 2025 22:23
@senemsoy senemsoy force-pushed the in-memory-order-updater branch from f3d92a4 to a5ed6c6 Compare February 28, 2025 22:28
@stewart stewart force-pushed the in-memory-order-updater branch 2 times, most recently from d0322b2 to 07add2a Compare March 7, 2025 22:28
@forkata forkata force-pushed the in-memory-order-updater branch 2 times, most recently from d22dbf5 to 770e527 Compare March 14, 2025 21:31
@nvandoorn nvandoorn force-pushed the in-memory-order-updater branch from 770e527 to c8e434e Compare March 28, 2025 20:22
@forkata forkata force-pushed the in-memory-order-updater branch from 7bf4adb to 2d497bd Compare April 3, 2025 18:16
@benjaminwil benjaminwil force-pushed the in-memory-order-updater branch from 67186f3 to defde7e Compare April 10, 2025 17:36
@AlistairNorman AlistairNorman force-pushed the in-memory-order-updater branch from 5423a0e to 62209cf Compare April 17, 2025 19:05
@benjaminwil benjaminwil force-pushed the in-memory-order-updater branch from 2b60e3b to 083fafe Compare May 8, 2025 19:03
AlistairNorman and others added 24 commits September 11, 2025 12:25
This puts all the update and recalculate methods together.

Co-Authored-By: Adam Mueller <[email protected]>
Co-Authored-By: benjamin wil <[email protected]>
Co-Authored-By: Andrew Stewart <[email protected]>
Co-Authored-By: Harmony Bouvier <[email protected]>
Co-Authored-By: Jared Norman <[email protected]>
Co-Authored-By: Kendra Riga <[email protected]>
Co-Authored-By: Sofia Besenski <[email protected]>
Co-Authored-By: Chris Todorov <[email protected]>
Co-Authored-By: Tom Van Manen <[email protected]>
Co-Authored-By: Noah Silvera <[email protected]>
Co-authored-by: Adam Mueller <[email protected]>
Co-authored-by: Andrew Stewart <[email protected]>
Co-authored-by: Jared Norman <[email protected]>
Co-authored-by: Noah Silvera <[email protected]>
Co-authored-by: benjamin wil <[email protected]>
Co-authored-by: Alistair Norman <[email protected]>
Includes a small refactor to the internal recalculate method to simplify
the code while maintaining the existing logic around only persisting
when the values have changed.

We'll use this persist flag to eventually only save changes to the DB
when requested. Allowing us to use this adjuster to update the order
in-memory.

Co-authored-by: An Stewart <[email protected]>
Co-authored-by: Harmony Evangelina <[email protected]>
Co-authored-by: Kendra Riga <[email protected]>
Co-authored-by: Nick Van Doorn <[email protected]>
Co-authored-by: Sofia Besenski <[email protected]>
Co-authored-by: benjamin wil <[email protected]>
Similar to the previous change. We're now passing the persist flag down
to all promotion order adjusters. This does not implement the logic
within the individual adjuster classes to skip persistance when
required, only ensures the flag is pass down from our in-memory order
updater.

Co-Authored-By: Adam Mueller <[email protected]>
Co-Authored-By: Andrew Stewart <[email protected]>
Co-Authored-By: Harmony Evangelina <[email protected]>
Co-Authored-By: Jared Norman <[email protected]>
Co-Authored-By: Kendra Riga <[email protected]>
Co-Authored-By: Senem Soy <[email protected]>
Co-Authored-By: benjamin wil <[email protected]>
Previously this would update the eligible column. We now only assign the
value and then save if persist is true.

Co-Authored-By: Adam Mueller <[email protected]>
Co-Authored-By: Andrew Stewart <[email protected]>
Co-Authored-By: Harmony Evangelina <[email protected]>
Co-Authored-By: Jared Norman <[email protected]>
Co-Authored-By: Kendra Riga <[email protected]>
Co-Authored-By: benjamin wil <[email protected]>
Co-Authored-By: Chris Todorov <[email protected]>
Co-Authored-By: Nick Van Doorn <[email protected]>
Co-Authored-By: Sofia Besenski <[email protected]>
We were missing the whole path of doing order level adjustments, e.g. The CreateDiscountedItem benefit.
This is in service of the in-memory order updater.

Co-authored-by: Adam Mueller <[email protected]>
Co-authored-by: Alistair Norman <[email protected]>
Co-authored-by: Andrew Stewart <[email protected]>
Co-authored-by: benjamin wil<[email protected]>
Co-authored-by: Chris Todorov  <[email protected]>
Co-authored-by: Harmony Bouvier <[email protected]>
Co-authored-by: Jared Norman <[email protected]>
Co-authored-by: Nick Van Doorn <[email protected]>
Co-authored-by: Noah Silvera <[email protected]>
Co-authored-by: Senem Soy <[email protected]>
Co-authored-by: Sofia Besenski <[email protected]>
Instead of destroying.

See code comments and spec changes. This is similar to how we handled
this in the legacy promotion system.

Co-authored-by: Adam Mueller <[email protected]>
Co-authored-by: Alistair Norman <[email protected]>
Co-authored-by: An Stewart <[email protected]>
Co-authored-by: benjamin wil <[email protected]>
Co-authored-by: Harmony Evangelina <[email protected]>
Co-authored-by: Jared Norman <[email protected]>
Co-authored-by: Nick Van Doorn <[email protected]>
Co-authored-by: Senem Soy <[email protected]>
Co-authored-by: Sofia Besenski <[email protected]>
Integration level test using the InMemoryOrderUpdater to ensure we are
not persisting changes during the promotion recalculations.

Co-authored-by: Chris Todorov <[email protected]>
Co-authored-by: benjamin wil <[email protected]>
Co-authored-by: An Stewart <[email protected]>
Co-authored-by: Jared Norman <[email protected]>
Integration level test using the InMemoryOrderUpdater to ensure we are
not persisting changes during the promotion recalculations.

Co-authored-by: Chris Todorov <[email protected]>
Co-authored-by: benjamin wil <[email protected]>
Co-authored-by: An Stewart <[email protected]>
Co-authored-by: Jared Norman <[email protected]>
We want to protect against manipulative database queries in the legacy promotion system
We also want to ensure that objects in memory
have their attributes changed correctly but not persisted

Co-authored-by: Jared Norman <[email protected]>
Co-authored-by: Kendra Riga <[email protected]>
Co-authored-by: Adam Mueller <[email protected]>
Co-authored-by: Alistair Norman <[email protected]>
Co-authored-by: Senem Soy <[email protected]>
Co-authored-by: Sofia Besenski <[email protected]>
Co-authored-by: Noah Silvera <[email protected]>
In an associated change we made the `CreateDiscountedItem` benefit
create in-memory line items. That causes an issue during the
recalculation of the item totals.

Co-authored-by: benjamin wil <[email protected]>
Co-authored-by: Kendra Riga <[email protected]>
Co-authored-by: Noah Silvera <[email protected]>
Co-authored-by: Jared Norman <[email protected]>
This will be used by the in-memory order updater to log manipulative
queries when the persist flag is set to false. This will inform
application developers whether their application is complying with the
in-memory order updater's "no writes" philosophy.

Previously the `db-query-matchers` gem we use to check for manipulative
queries was only used in tests. However, we now use it in application
code so the dependency must be moved to the gemspec.

Co-authored-by: Alistair Norman <[email protected]>
Co-authored-by: Sofia Besenski <[email protected]>
Co-authored-by: Adam Mueller <[email protected]>
Co-authored-by: Chris Todorov <[email protected]>
Co-authored-by: benjamin wil <[email protected]>
Enabling manipulative query logging will show the number of manipulative
queries on each call to Spree::InMemoryOrderUpdater#recalculate.

Co-authored-by: Chris Todorov <[email protected]>
Co-authored-by: Adam Mueller <[email protected]>
Co-authored-by: Alistair Norman <[email protected]>
Co-authored-by: Senem Soy <[email protected]>
Co-authored-by: benjamin wil <[email protected]>
This method will be used in the in memory order updater. We only want to
persist to the database once we perform the action, not when we are
computing the amount.
When computing item totals after a line item benefit is removed, we need
to ensure we aren't using the marked for destruction items.

We also needed to make a change to the legacy promotion order updater
patch because it is incorrectly being used in our tests even when using
the new promotion system.

Co-authored-by: Jared Norman <[email protected]>
Co-authored-by: Noah Silvera <[email protected]>
We want to make sure we don't persist anything when computing amounts
becuase that will cause issues with the in-memory order updater.
Our legacy promotion order updater patch did not have updated logic to
ensure it ignored adjustments that were marked for desctruction.

Co-authored-by: Jared Norman <[email protected]>
Co-authored-by: Noah Silvera <[email protected]>
Co-authored-by: benjamin wil <[email protected]>
Co-authored-by: Chris Todorov <[email protected]>
Instead of calling update_columns, we should just use assign_attributes
everywhere and rely on our order persistance bubbling down to the line
item when appropriate.

This avoids issues with the in-memory order updater when attempting to
update values on newly created promotional items that won't get
persisted until later in the recalculation flow.

Co-authored-by: Jared Norman <[email protected]>
Co-authored-by: Noah Silvera <[email protected]>
Sets up some shared examples so that we can run the full promotion
integration suite against both the existing order updater and the
in-memory order updater.

Co-authored-by: Jared Norman <[email protected]>
Co-authored-by: Noah Silvera <[email protected]>
Now that we are going to be using parts of this gem in our monitoring
class, we need it to be required everywhere. This change also ensures
the configuration is shared across the Solidus gems, which previously
was not the case.

Co-authored-by: benjamin wil <[email protected]>
Co-authored-by: Sofia Besenski <[email protected]>
We no longer want to have individual methods use a persist: flag,
instead we can save at the end if needed.

Co-authored-by: Chris Todorov <[email protected]>
Co-authored-by: Senem Soy <[email protected]>
Co-authored-by: benjamin wil <[email protected]>
We no longer want to have individual methods use a persist: flag,
instead we can save at the end if needed.
@benjaminwil benjaminwil force-pushed the in-memory-order-updater branch from 2d124da to 6fc0879 Compare September 11, 2025 19:28
adammathys and others added 3 commits September 18, 2025 11:49
We don't want our in-memory order updater to have to persist shipment
state.

Because we no longer rely on `Shipment#update_state` which uses
`update_column`, we are now logging any additional `Shipment#state`
changes when the order is saved. The changes to the specs reflect this
change in behaviour.

Co-authored-by: Jared Norman <[email protected]>
Co-authored-by: Noah Silvera <[email protected]>
Co-authored-by: Alistair Norman <[email protected]>
Due to upstream changes after a rebase, these tests started failing
because the `order` being set up (`Spree::Order.create`) did not have
proper shipments or inventory units created for it. The new tests use
factories to more realistically set up orders, shipments, and inventory
units in various states to simulate order updater functionality.

Co-authored-by: Alistair Norman <[email protected]>
Co-authored-by: Kendra Riga <[email protected]>
Co-authored-by: Chris Todorov <[email protected]>
Switch to in-memory order updater and log queries to separate file.

This is a temporary change to allow us to produce a log of the manipulative
queries triggered by running the test suite on CI.

Co-authored-by: Benjamin Willems <[email protected]>
Co-authored-by: Sofia Besenski <[email protected]>
@forkata forkata force-pushed the in-memory-order-updater branch from 6fc0879 to ae58cce Compare September 18, 2025 18:52
We want state change logging for shipment and payment state changes.
We got to the point of a failing spec - more jobs (and not the ones we
want) are being enqueued than we want currently. I left the empty
context in for 'logs a state change for the shipment' so it doesn't get
forgotten about, but we didn't quite start on it yet.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
changelog:solidus_admin changelog:solidus_core Changes to the solidus_core gem changelog:solidus_legacy_promotions Changes to the solidus_legacy_promotions gem changelog:solidus_promotions Changes to the solidus_promotions gem
Projects
Status: No status
Development

Successfully merging this pull request may close these issues.

9 participants