Rules for `copysign` are missing?

[RainerHeintzmann]

copysign is used in ChainRules.jl in various places, which may have to do with adherence to the IEEE float standard.
Yet, copysign throws a, probably intended error:

julia> Zygote._pullback(copysign, 3.33, 4.4)[2](1)
ERROR: Non-differentiable function Core.Intrinsics.copysign_float
Stacktrace:
 [1] error(s::String)
   @ Base .\error.jl:33
 [2] macro expansion
   @ ~\.julia\packages\Zygote\TaBlo\src\compiler\interface2.jl:0 [inlined]
 [3] (::typeof(∂(λ)))(Δ::Int64)
   @ Zygote ~\.julia\packages\Zygote\TaBlo\src\compiler\interface2.jl:43
 [4] Pullback
   @ .\floatfuncs.jl:5 [inlined]
 [5] (::typeof(∂(copysign)))(Δ::Int64)
   @ Zygote ~\.julia\packages\Zygote\TaBlo\src\compiler\interface2.jl:0
 [6] top-level scope
   @ REPL[56]:1

, but the question is whether this is really required for the pullback rule. E.g. a function like abs works as expected but is also, strictly speaking, not differentiable at the origin. Maybe this could work:

@scalar_rule copysign(y, x) (sign(x)*y, 0)

or is there a fundamental reason that copysign should not have an allowed gradient, but abs should?

[willtebbutt]

Nope, this would be a good addition -- I think the rule you propose is probably what we want. If you have the time, a PR would be very welcome.

[RainerHeintzmann]

Sure, can do so. Maybe I should wait till my other pull request (sincospi) is merged and then add this?

[willtebbutt]

Sounds good 👍

[RainerHeintzmann]

Hmm, interestingly the sign() function yields a gradient of nothing:

julia> Zygote.gradient((x) -> sign(x), 1.2)
(nothing,)

should this not be 0? Bug or feature?

[willtebbutt]

I think this is probably a Zygote thing. For historical reasons, Zygote represents zero tangents by nothing, whereas ChainRules uses either 0 or ZeroTangent(). I would hope that if you called rrule directly you would obtain a 0.

[RainerHeintzmann]

Yes, you are right:

julia> rrule(sign,-7.0)[2](1)
(NoTangent(), ZeroTangent())

indicating that ZeroTangent() may be more appropriate than 0 in my rule.
But then sign seems to also be defined for complex input, yielding a non-zero pullback:

julia> rrule(sign,-7.0)[2](1im)
(NoTangent(), 0.0 + 0.14285714285714285im)

Luckily copysign() is not allowed for complex inputs (except for Unitful.AbstractQuantity, as the first argument, which may be unintended), so maybe I can stick to the stated rule.

[RainerHeintzmann]

... The unitful version also (luckily) fails for copysign.

[willtebbutt]

Hmmm I'm slightly surprised that sign is giving you a ZeroTangent rather than 0 -- we usually go with 0 in scalar rules for performance-related reasons. @mcabbott @oxinabox @mzgubic any thoughts?

[mcabbott]

For sign, this doesn't look good, and is the reason to prefer 0.0:

julia> gradient.(x -> x^2>1 ? sign(x) : x, (-3, -2, -1, 0, 1, 2, 3))  # super-crude tanh
((nothing,), (nothing,), (1,), (1,), (1,), (nothing,), (nothing,))

julia> gradient.(x -> ifelse(x^2>1, sign(x), x), (-3, -2, -1, 0, 1, 2, 3))  # weirdly ifelse fixes it?
((0,), (0,), (1,), (1,), (1,), (0,), (0,))

julia> gradient.(x -> clamp(x, -1, 1), (-3, -2, -1, 0, 1, 2, 3))  # better?
((0,), (0,), (1,), (1,), (1,), (0,), (0,))

julia> gradient.(x -> 0<x<1, (-3, -2, -1, 0, 1, 2, 3))  # this is always correctly categorical
((nothing,), (nothing,), (nothing,), (nothing,), (nothing,), (nothing,), (nothing,))

Unlike signbit, both sign and copysign preserve type, and thus are plausible things to use in stepwise functions like that:

julia> sign.((-33f0, -0f0, 0f0, 42f0))
(-1.0f0, -0.0f0, 0.0f0, 1.0f0)

julia> signbit.((-33f0, -0f0, 0f0, 42f0))
(true, true, false, false)

julia> copysign(2f0, big(-3.0))
-2.0f0

But I'm not sure the 2nd argument of copysign shouldn't be NoTangent. It does seem like it's the input to something like x>0. Maybe that's safe?

[RainerHeintzmann]

I see. So for type stability reason, 0 is clearly the better choice. Of course the rrule of sign can be easily fixed to return 0 instead of nothing as well. The clamp idea is probably not a good choice at least for real numbers, if you think about -0.5 as input ;-)

[mcabbott]

No, I think I convinced myself of the opposite -- that the 2nd argument of copysign should be NoTangent(), like the 1st argument of ifelse. Unless someone can invent an example where this goes wrong.

I'm not sure why Zygote is giving nothing for sign above, BTW. The rule here doesn't seem to fix it:

ChainRules.jl/src/rulesets/Base/fastmath_able.jl

Line 201 in e5c5338

return (NoTangent(), ∂x)

although it does give a complex answer, and I don't see a more specific rule in here:

https://github.com/JuliaDiff/ChainRules.jl/blob/master/src/rulesets/Base/nondiff.jl

[oxinabox]

Zygote might have a ZygoteRules.@no_grad sign

I agree we should use NoTangent for the (co)tangent wrt the send argument.
Unless we find a real performance case where 0 is better.

Can always change in follow up.

Hmmm I'm slightly surprised that sign is giving you a ZeroTangent rather than 0 -- we usually go with 0 in scalar rules for performance-related reasons.

Either works.
ZeroTangent is conceptually better.
0 is a compromise we make for performance.
When required.
(Esp while we wait for a feature to be fixed in Julia for narrow union promotion with singleton. Like nothings and missings have).

(I am leaving this PR to someone else reviewing to approve and merge.)

[mcabbott]

Zygote might have a @nograd sign

I wondered this, but don't see anything: https://github.com/FluxML/Zygote.jl/search?q=sign&type=code

[mzgubic]

Closed by #502