Distribution issue: solve_order prevents other fields from being swizzled

[cshafer-tt]

Capturing more distribution issues from #240 here.

Scaling up a bit and I'm still seeing distribution issues (unrelated to solve order, that seems to work perfectly now). Can you take a look at this one as well? Let me know if you would rather I file another issue instead.

For this one, I have added a third variable that is basically just an enable/disable that is explicitly disallowed for some formats. I have the constraint below:

  @vsc.constraint
  def c_feature_enable(self):
    with vsc.if_then(self.output_format == FormatType.Float16):
      self.feature_enable == 0

For some reason, this constraint impacts the distribution when output_format is not Float16. I would expect all other formats to have a ~50/50 distribution, but there is significant bias towards disable. Here are the results with and without the above constraint enabled:

With:

feature_enable:
  0: 8895
  1: 1105

Without:

feature_enable:
  0: 5070
  1: 4930

I have attached the updated file.

pyvsc_test.py

Originally posted by @cshafer-tt in #240

@cshafer-tt Yeah, you should probably start a new issue for this.

Looking at the current randomizing method it looks like only the fields listed in the solve_order actually get swizzled.
What you're seeing is two variables getting randomized, and the third one is getting whatever the SMT solver came up with, no randomization. So it could very well be the same value each time if that's what the solver goes to.

pyvsc/src/vsc/model/solvegroup_swizzler_partsel.py

Lines 42 to 49 in add71d8

	if rs.rand_order_l is not None:
	# Perform an ordered randomization
	if self.debug > 0: print(" following solve-order constraints")
	for ro_l in rs.rand_order_l:
	swizzled_field |= self.swizzle_field_l(ro_l, rs, bound_m, btor)
	else:
	if self.debug > 0: print(" following random field order")
	swizzled_field |= self.swizzle_field_l(rs.rand_fields(), rs, bound_m, btor)

Plus it looks like there's a max of 4 fields that get randomized/swizzled. I think that's there for performance reasons.

pyvsc/src/vsc/model/solvegroup_swizzler_partsel.py

Lines 67 to 70 in add71d8

	max_swizzle = 4
	# Select up to `max_swizzle` fields to swizzle
	for i in range(max_swizzle):

If you explicitly include self.feature_enable in the solve_order to then it will get swizzled.

vsc.solve_order(self.output_format, [self.input_format, self.feature_enable])

With that modified solve_order:

With:
feature_enable:
  0: 534
  1: 466

Without:
feature_enable:
  1: 511
  0: 489

Feel free to copy this info into a new issue.
Edit: Oh, there's even a "Reference this in a new issue" button!

Originally posted by @alwilson in #240

[cshafer-tt]

Just out of curiosity, can you explain the significance of this statement:

Plus it looks like there's a max of 4 fields that get randomized/swizzled. I think that's there for performance reasons.

Wondering what scope this "max of 4 fields" is applied to, is it a single constraint object? Is the implication that if you have more than 4 fields in an object, the randomization will behave unexpectedly?

If it's truly only there for performance reasons, I'm wondering if I just need to remove/increase it.

[alwilson]

Wondering what scope this "max of 4 fields" is applied to, is it a single constraint object?

When randomize gets called it splits out related variables and constraints into their own separate sets. If a set doesn't have a solve_order constraint then it picks 4 random fields within it to randomize, one by one. The randomize(debug=1) will show you how many rand_sets it split the problem into and which fields within them it chose to swizzle.

Is the implication that if you have more than 4 fields in an object, the randomization will behave unexpectedly?

I think the idea was that randomly picking 4 fields within a set would constrain the remaining fields enough that there's not much to left to randomize, so time and solver calls get wasted. Plus the fact that it'll be a different arrangement of 4 fields each time.

In this case the solve_order causes it to take a different path that doesn't try 4 random fields. It might help in your other cases though. I think the fix here is to do all the ordered fields first, and then pick N remaining fields to randomize. It might be worth changing that max to something like max(4, num_fields / 4) or something. Or add a keyword arg to the randomize() call to let users specify a new max.

[cshafer-tt]

Between this and the other fix we are getting much better results now. Thanks a lot for the help