Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Comment on that all nonlinear operators can be shifted to augmented primal #587

Open
wants to merge 6 commits into
base: main
Choose a base branch
from
Open

Comment on that all nonlinear operators can be shifted to augmented primal #587

Changes from all commits
Commits
Show all changes
6 commits
Select commit Hold shift + click to select a range
db42494
Comment on that all nonlinear operators can be shifted to augmented p…
oxinabox Sep 20, 2022
60c5e60
Update docs/src/design/changing_the_primal.md
oxinabox Dec 8, 2022
41ee9af
Update docs/src/design/changing_the_primal.md
oxinabox Dec 8, 2022
00a7a78
Apply suggestions from code review
oxinabox Dec 8, 2022
028516b
Update docs/src/design/changing_the_primal.md
oxinabox Dec 8, 2022
7e63c27
Update docs/src/design/changing_the_primal.md
oxinabox Feb 27, 2023
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
10 changes: 10 additions & 0 deletions docs/src/design/changing_the_primal.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -469,6 +469,16 @@ We don't have this in ChainRules.jl yet, because Julia is missing some definitio
We have been promised them for Julia v1.7 though.
You can see what the code would look like in [PR #302](https://github.com/JuliaDiff/ChainRules.jl/pull/302).

## What things can be taken out of the pullback?
At this point you might wonder, is there a rule for what can be taken out of the pullback and computed in the augmented primal?
We can deduce one, or in fact two.
The first and most practical one is that any computation that depends only on the primal input or output can be shifted to the augmented primal.
The second, perhaps more insightful, rule is that all nonlinear parts can be moved out.
This is a weaker but more interesting statement.
We know this because pullbacks are linear operators -- linear w.r.t. the tangent they are pulling back.
This means they are in turn composed only of functions that are linear operators w.r.t. the tangent.
The minimal pullback function only calls linear operators -- the nonlinear parts can all be shifted to the augmented primal.

## Conclusion
This document has explained why [`rrule`](@ref) is the way it is.
In particular it has highlighted why the primal computation is able to be changed from simply calling the function.
Expand Down