[hls-fuzzer] Add concept of weak dependencies

[zero9178]

In some circumstances, such as in the limit type system, the transfer functions of an AST node might not care about the specific order of how elements are generated, but rather just if a sub element has been generated previously. This is e.g. the case in the limit type system where the total number of statements does not care about whether the sub-statementlist or its statement is generated first, only which has been generated first.

This PR therefore adds the new concept of a "weak" dependency to transfer functions which are merely used to affect the transfer function signature: A possibly null pointer and an optional AST node are passed to the transfer function iff that sub-element has been generated already.

[Jiahui17]

I can't think of why a nullptr or nullopt AST node would be passed to the input of the transferFn; conceptually, how does the topological sort work on these weak edges?

[zero9178]

The topological sort simply ignores weak edges entirely. That means it might be null or null opt because the given sub element hasn't been generated yet.

The advantage is that you can have cyclic dependencies using weak edges. This way the limit type system does not force an order on statement generation, but still counts the number of edges correctly

[Jiahui17]

struct LimitTypingContext {
  std::size_t expressionDepth{};
  std::size_t totalNumberOfStatements{};
};

So it is not an issue for depth but for the total statement count

Maybe if there are no other major use cases for weak deps, we could make the total statements a reference to a global value (it is a global property of the AST instead of a locally derived one)

[zero9178]

struct LimitTypingContext {
  std::size_t expressionDepth{};
  std::size_t totalNumberOfStatements{};
};

So it is not an issue for depth but for the total statement count

Maybe if there are no other major use cases for weak deps, we could make the total statements a reference to a global value (it is a global property of the AST instead of a locally derived one)

The way it is used right now does have strictly more information and that it gives the total number of statements generated so far at all times.

There are also other use cases where this would not work. Eg the dynamatic type system also currently forces an order in which binary expressions are generated when it has actually no need to (just to avoid cycles).

I would avoid globals when possible, best to always keep things local when possible

[Jiahui17]

There are also other use cases where this would not work. Eg the dynamatic type system also currently forces an order in which binary expressions are generated when it has actually no need to (just to avoid cycles).
I would avoid globals when possible, best to always keep things local when possible

Ok, I see the point

---

Could we introduce the concept of "the last generated AST node + its context?"

BTW "last" could also be the parent AST node + its context

We can specify that a TransferFn requires such an argument if we need to have some accumulated history (e.g., the number of edges that we need here), but not necessary from a specific node.

This looks also useful for the bitwidth thingie

[zero9178]

There are also other use cases where this would not work. Eg the dynamatic type system also currently forces an order in which binary expressions are generated when it has actually no need to (just to avoid cycles).
I would avoid globals when possible, best to always keep things local when possible

Ok, I see the point

Could we introduce the concept of "the last generated AST node + its context?"

BTW "last" could also be the parent AST node + its context

We can specify that a TransferFn requires such an argument if we need to have some accumulated history (e.g., the number of edges that we need here), but not necessary from a specific node.

This looks also useful for the bitwidth thingie

I can see that being useful yeah. I think what you'd then probably want though is that it shouldn't actually be "the last generated sub-element" but "the last generated node from a set of sub-elements that we care about".

Technically speaking, what weak does here already makes this implementable in some type systems: Since the context in this case (and the other cases) happens to be monotonic, you can tell 1) which AST nodes are present and 2) based on the state of their contexts tell which was generated in which order.

While something like an accumulated history is doable, I'd probably require much more effort and I am not sure its necessary for our use-cases.

[Jiahui17]

okay i see

---

For the counting statement use case, I guess it is possible that we count the statement before the subelem has been generated? How do we deal with this case?

[zero9178]

The increment only happens directly in the transfer functions of the respective statements and is then propagated back up through the output context later.
This means that the increment only happens after the discard method has already returned false.
A statement may still not be generated if a sub element fails to be generated, but in that case the original input context is unmodified.
This is a general invariant that contexts are only propagated if generation succeeded. Otherwise it'd be a bug.

Example trace for the last case:

• for statement is supposed to be generated with input context count = 2
• for statement transfer function forwards count = 3 to its statement list
• for statement fails to be generated for some reason
• generator now generates an array assignment statement with the exact same input context count = 2 as it originally attempted for the for statement. No wrong increment occurred.

[Jiahui17]

Oh I just realized that copyFirstOf does exactly "choosing the first generated valid context from among XXX"

Are the users supposed to mark weak edges directly or only use the convenient functions like copyFirstOf?

[Jiahui17]

If those weak edges are only used internally, maybe it is better to put them in the detail:: namespace?

[zero9178]

Oh I just realized that copyFirstOf does exactly "choosing the first generated valid context from among XXX"

Are the users supposed to mark weak edges directly or only use the convenient functions like copyFirstOf?

Users are meant to use this directly if they need to yes. I don't think copyFirstOf will cover all use cases (just one common one). I'd call it one of those features one can usually ignore until you need it and you'll know when you need it

[Jiahui17]

I feel like it is better to limit a bit how the user could use this feature (just for safety). For instance, let's say the user specify a copyFirstOf between many AST elems, it would be good to throw a runtime error when none of the candidates are valid at the time when the transferFn is evaluated

[zero9178]

I feel like it is better to limit a bit how the user could use this feature (just for safety). For instance, let's say the user specify a copyFirstOf between many AST elems, it would be good to throw a runtime error when none of the candidates are valid at the time when the transferFn is evaluated

I made it such that copyFirstOf now simply takes a list of indices, meaning callers must use weak themselves. This way users can also specify a fallback non-weak dependency that will always be present.
A static_assert catches if the user exclusively uses weak dependencies. The runtime error you described is therefore now impossible

[Jiahui17]

The last remark here could be a new paragraph.
Maybe also add:
In this case, the transferFn with weak dep at its input will receive an empty context and nullptr for that dependency, and it must ignore that input and/or fall back to analyzing other output contexts or the input context

[zero9178]

I've reformatted the paragraph but refrained from adding that sentence since I don't want to prescribe what users can or cannot do. The fact that its an optional and pointer that are empty and nullptr already signals that they must ignore it/cannot do anything with these values.
Especially since this is a stark contrast to the const-references that are passed for non-weak dependencies.