Add composite relu - #3316
Conversation
j2kun
left a comment
j2kun
left a comment
There was a problem hiding this comment.
Fantastic work! I will try to carve out some time in the future to do a more detailed analysis of the tradeoff between the composite method and the vanilla approximation.
|
|
||
| #map = affine_map<(d0, d1) -> (d0, d1)> | ||
|
|
||
| // A fused elementwise generic can carry more than one ReLU. Every maximumf must |
There was a problem hiding this comment.
In your opinion, would it make sense in the future to aim for a general handler of linalg.generic for canoncalizing activations?
There was a problem hiding this comment.
Just to make sure we are on the same page, you mean grouping operations that belong to the same activation function together, so they can be treated as a unit. I think that would indeed make sense since we typically want to treat each activation function as a unit (since we think in terms of layers in the neural network)
There was a problem hiding this comment.
Right, I was mainly surprised that the linalg produced by torch-mlir can produce generics with multiple ops operating on different operands within.
j2kun
added
the
pull_ready
|
I did notice that enabling this flag for the HE-LRM model actually causes a segfault, but I think I will not have the time to investigate until after the blog deadline + my 2 weeks of vacation. However, next week you will be able to see the exact invocation and I can tell you where I flipped this flag and saw an error |
2 participants
Ports the 3 stage RELU approximation from the Orion paper. It is hidden behind a flag.