In the definition of anchor():

An anchor() function representing a valid anchor function resolves at computed value time (using style & layout interleaving) to the that would align the edge of the positioned boxes' inset-modified containing block corresponding to the property the function appears in with the specified border edge of the target anchor element, assuming that all scroll containers between the target anchor element and the positioned box’s containing block are scrolled to their initial scroll position (but see § 3.4 Taking Scroll Into Account).

But then, in "Taking Scroll Into Account":

All anchor references are calculated as if all scroll containers were at their initial scroll position, and then have their associated remembered scroll offset added to them.

Not far above, it says:

When a positioned element is first displayed, or when it changes fallbacks, its position is correctly calculated according to the up-to-date position of all anchor references.

So I'd think ultimately anchor() should reflect position of scrollers when it toggles in and out of display: none or fallback changes? Comment #10999 seems to match this as well.