Device selection criteria for usecase-driven scenarios

[fdwr]

WebNN has been moving away from explicit device selection (explicitly stating the device kind - GPU/NPU/CPU) more in favor describing the intent and expectations of the workload (currently limited to MLPowerPreference and whether to request an MLContextOptions::accelerated device), but there may be more complex scenarios where we want to be more explicit, or cases where we want to be less explicit to let WebNN or the backends choose the device based on intent.

For example, some independent attributes that could contribute to device selection include aspects like: how big a workload is, how immediately you want the result (a GPU can finish a large load faster than the CPU, but when desiring small latency with small workloads, the CPU is often better), whether it's continuous repeating work or single shot, desired power usage...

• acceptable latency: low, medium, high
• continuity: continuous, moderate, single shot
• workload size: small, medium, large
• power usage: low, medium, high
• ...

e.g.

App scenarios	preferred latency	workload size	continuity	ideal device
Realtime audio filtering	low	small	continuous	CPU/NPU (not GPU due to overhead of readback time)
Audio to text	ignorable	medium	continuous	Any
Realtime video processing	low	large	continuous	GPU (since likely to be displayed afterward anyway)
Offline video processing	ignorable	large	continuous	GPU/NPU (since large continuous workload)
Image generation	medium	large	multiple executions	GPU/NPU (much faster than CPU)
Image recognition	ignorable	small	single execution	Any (CPU is fast enough for simple recognition)
Code editor prediction	low	small	single execution	CPU since small workload and want to avoid pegging the GPU while typing

(conceptually it's kind of like the font selection problem - family & weight & width & slope ...)

Seeding this discussion @anssiko for user feedback... 📋

[fdwr]

@mtavenrath - copying your post here for followup:

@fdwr I really like that list.

Still I am curious how to really determine the perfect device and latency requirements? What if someone wants to launch a low latency, high compute intensive audio network? How is the WebNN backend supposed to know which dGPU to use if there are multiple dGPUs? What if the dGPU is slower than the iGPU, but has just been added to add certain display features like full sync or 4/8/16 displays? What if the dGPU is used by a WebGPU rendering process which would affect everything? What if a website wants to run 2 large models at once on two different dGPU?

Do we want WebNN to be limited to the simple use cases only or do we want to allow the complex use cases as well? There are ISVs in the professional space who port their software to the web and might be interested to choose a specific device. For those use cases it'd be great if it's possible to do an explicit device selection.

This leaves the question, how to not make such an API available to generic fingerprinting? With the risk of slightly lowering usability, would it make sense to make this feature available upon user-request only like webcams, audio or the GPU position?

By default only the simple API is available. If a more complex web application requires the advanced API it could request access and make the user aware that he is exposing more information from his system.

[mwyrzykowski]

If we reported MLOpSupportLimits per processing unit, then could a sophisticated website / web app make the determination of which model to run such that a UA / underlying OS has the potential to schedule the workloads on the most optional computing units? This also allows UAs which choose to expose less potential fingerprinting information to group capabilities in buckets. Or perhaps the UA wants to prohibit CPU device execution via WebNN or limit CPU execution to very simple models, then it can intentionally underreport capabilities to achieve that.

A UA which is less concerned with fingerprinting or, for instance, determines they are running on non consumer class hardware (or perhaps very 'high end' hardware) could choose to expose MLOpSupportLimits which indicate to the website / web app that they can run certain models.

I think this would also support the multiple discrete GPU case as capabilities could be reported for each GPU present.

[zolkis]

(Commenting as an IE from now on - no affiliations.)

I salute the proposed attributes (some of them, like latency have been mentioned in previous discussions).
We could add these as context creation options?

As agreed here about #897 (see some refs there): a context could be seen as "all app provided hints and prefs" which a UA can internally translate to a prioritized list of execution devices. Then, when a graph is built, the workload is scheduled more exactly and #854 investigates how to expose that information. Also, #901 introduces a new option to how graphs may be built. A lot of terms and relationships to be re-discussed.

(I don't know how useful would be at this point, but we could rework the device selection explainer to focus on this arc.)

Then,

If we reported MLOpSupportLimits per processing unit

This was mentioned in the explainer as next possible level for device selection, and would complement the "simple" device selection method (i.e. by context options).

I think it's time to illustrate the use cases with code, both for the context options and op support limits. Would you @mwyrzykowski provide some examples on how do you imagine this being used on UA's with varying internal policies?

[mwyrzykowski]

We discussed a bit more in the WG/CG meeting and I agree hints seem reasonable along with per compute unit capabilities.

E.g., for a device which only reports float16 support via dataTypes, if there is to be any possibility of the model running on that compute unit, then all operations must be performed in half precision float.

[handellm]

+1, great if you could propose an interface @mwyrzykowski. From the WebRTC perspective I think it would also be good to be able to make a distinction between audio/video capture and render since these cases have different latency requirements.

Also related is that there are use cases where Worker performance does not meet expectations and I'm going to propose adding "quality-of-service" hints which overlap the scenario list in #902 (comment). I think a model where such Worker QoS influences latency of APIs like WebNN, WebGPU etc would be ideal and also be in line with how platforms work. Wdyt?