[FR] Full build/dynamic configuration - Marlin "klipperization"

[vovodroid]

As we near the end of 2024, more and more new printers (at least in the mid-price range) are coming with Klipper pre-installed. This suggests that Marlin might be losing its market share. With Raspberry Pi prices expected to decrease and the possible emergence of dedicated Klipper single-board solutions, this trend could accelerate even further.

From my perspective, Marlin already has all the functionality needed for 3D printing, including modern features like Input Shaping, Linear Advance, and more. However, its compile-time configuration and the need to rebuild/flash the firmware for every change can be intimidating for users and inconvenient even for experienced developers. For example, I recently had to rebuild Marlin multiple times just to adjust the endstop position and the extruder microsteps.

Years ago, it was impossible for 8-bit boards to handle all features due to flash and RAM limitations, forcing users to make choices at build time. However, the situation has changed with modern 32-bit boards, which can support the entire codebase. This opens the door to moving the configuration process to an INI file, similar to how Klipper operates.

Suggested Steps for "Klipperization" of Marlin on 32-Bit Boards:

1. Build-Time Options for Maximum Flexibility:
   Introduce an option to build Marlin with as much of the existing codebase as possible for a specific platform/boar), with SD card/EEPROM access pre-configured. This would include currently conflicting options that cause errors in Sanity.h, as these could be enabled or disabled at runtime.
   This approach would allow distributing pre-built images for platforms rather than specific printers.

2. Dynamic Configuration:
   Replace the current Config*.h system with a dynamic INI/EEPROM-based configuration. On startup, Marlin would use the build-time configuration only to locate a Marlin.ini file on the root of the SD card and load its settings. If no file is found, Marlin would fall back to EEPROM. The Marlin.ini file could be stored in EEPROM.

3. Dynamic Allocation of Large Variables:
   To save RAM, large static variables (e.g., Input Shaping buffers with low SHAPING_MIN_FREQ) could be dynamically allocated. Since no free/delete operations are required, this would allow for a very simple memory allocation manager.

---

With these changes, I believe Marlin could become more attractive and gain a competitive edge against Klipper.

Anyway even without full build allowing to set option in INI file already will make Marlin much more useful.

[robherc]

So what you're proposing is that we make Marlin the new "ROM chokepoint" firmware, worse than RRF, that only works on the 2% of boards with the most storage included?

[vovodroid]

As I said for low end boards some code blocks could be skipped. But at least dynamic config could be implemented.

[ThomasToka]

I did it already in my 2.1.x-bugfix fork for the e3s1pro/plus and its nothing that not could be done.
i have already:

• dynamic mesh size (5,7,10) for ubl and abl but everything is possible, i only limited it cause my dwin set is out of space for more sites
• dynamic probing margin for x and y, based on probe offsets, max_size, automatic min margin accounting etc.
• dynamic probe count
• dynamic probing points (scaling with margins) for assited tramming and manual tramming

everything changeable without reboot or recompilation and saved in eeprom.

[thinkyhead]

This feature request is too broad, and of course it is already on the TODO list, in piecemeal fashion. It's just not possible to make Everything into a variable in one go.

When we make a thing configurable that wasn't configurable before we just add an option for that, so not all builds have to use up RAM if they don't need the configurability.

If anyone has code ready for (after) 2.1.3 that makes some things configurable that weren't configurable before, submit a PR and make sure to implement it everywhere, for all displays that have UI we can modify. For things that affect one leveling system, endeavor to make it work for all.

[vovodroid]

When we make a thing configurable that wasn't configurable before we just add an option for that

So we need M... code for all parameter?

so not all builds have to use up RAM if they don't need the configurability.

If build excludes some feature, config variables also will be not present. And usage some int instead #define doesn't use too much.

If anyone has code ready for (after) 2.1.3 that makes some things configurable that weren't configurable before, submit a PR and make sure to implement it everywhere, for all displays that have UI we can modify

That's the point - I didn't talk about make all parameters configurable via UI or M... codes. But to have one generic INI based config, probably with editing in GUI. So no need in UI modification for each parameter. It removes another headache.

[thinkyhead]

So we need M... code for all parameter?

Yes. Any parameter that can be modified at runtime and saved to EEPROM should have an M-code.

As part of the next generation of Marlin, in addition to the LCD pins refactor I have a project defined for a Feature class, and this would move the ad hoc data currently floating around in various feature implementations under a class tied to the EEPROM and variable parameters. So keep that in mind as we discuss adding another configuration layer into Marlin.

If build excludes some feature, config variables also will be not present. And usage some int instead #define doesn't use too much.

If we are comparing to Klipper or RepRapFirmware, all features and kinematics are always available because everything is compiled in. So I presume when we talk about "Klipper-izing" Marlin we are also talking about making it possible to do a build with every feature available, completely configurable at runtime.

If we are just talking about replacing the output of M503 with more "human friendly" option names, reading an INI file from the SD Card at boot, etc., that is fine. We just need to be more clear about what we mean by "Klipper-ization" and the benefits of INI files over M-codes.

But to have one generic INI based config, probably with editing in GUI. So no need in UI modification for each parameter.

Well, we still have to have the code for the UI, so this just adds another thing. It doesn't replace it.

It removes another headache.

Strings are one of the more bloating things so this feature would not be fully implementable on small boards. We can of course just store the hashes to make it work on medium-sized boards. If we want to make the INI exportable from Marlin using M503 I then we do have to store the full names of all INI parameters, so that would be the most bloated version.

If you have a proposal for an INI-based configuration assigning descriptive names to all the parameters currently modified using M-codes, that would be a fine PR to submit. I'm sure you can whip the basics of that together in a few hours, and I would not discourage you from doing so.

I'm not personally convinced that using an INI file in place of the M-codes we share with other Marlin-derived firmwares is especially great. I presume the INI file would be modified by M500 instead of saving to EEPROM? Keep the Feature class proposal in mind.

[vovodroid]

Any parameter that can be modified at runtime and saved to EEPROM should have an M-code.

That's what take memory and development effort.

I have a project defined for a Feature class

Great! Is there [FR] to subscribe?

we are also talking about making it possible to do a build with every feature available, completely configurable at runtime.

May be not all, but as much as possible within board limitations.

and the benefits of INI files over M-codes.

Simple generic mechanism, that shouldn't be modified, and much easier to use. Also it allows easily to change parameters on startup, for instance for LCD/serial connections.

we still have to have the code for the UI, so this just adds another thing. It doesn't replace it.

Yes, but it's written only once. No need menu modification for new options.

Strings are one of the more bloating things so this feature would not be fully implementable on small boards

Strings themselves yes, but with M... mechanism you should have code for each of them. So you save string memory, but consume a lot of code memory. Take for instance my PR Configure extruder fan speed in menu #26839. Besides very small code change I needed to add menu code (for my platform only, so other platforms don't work I guess). And if we add new M.. code handling it will eat much more memory than just to add extruder_fan_speed=80 to config strings.

I'm sure you can whip the basics of that together in a few hours

Not really. We at least need some INI parser.

[thinkyhead]

It will certainly be interesting to have the option of using an INI file to set some things in a more-or-less readable form, and I of course wish you well on your journey to make this happen. As you can imagine our TODO list is very long, we are in the midst of a major release cycle, and so this work cannot go at the top for me.

See the "Projects" tab at the top of this page for the proposals I mentioned.

[github-actions]

This issue has been automatically locked since there has not been any recent activity after it was closed. Please open a new issue for related bugs.