Rewrite JS/HTML using AST-based approach

[flotwig]

• Closes Replace x.top, x.parent accessors with correct reference (at runtime) #5271 - replacing x.top, x.parent with dynamic interceptors
• Closes Rewrite 3rd party JS code to enforce that parent + top references are always correct #1467 - rewrite code to enforce x.top, x.parent correctness
• Closes Cypress cannot test sites that implement SRI #2393 - SRI stripping
• Closes Incorrect redirect to malformed url within AUT - adding _/ in url #3975 - relative location.href setter does not work as expected
• Closes window.location.replace not working - inserts __ into url #3994 - relative location.replace does not work as expected
• Closes Can't use window.parent.frames in my iframes  #2664 - x.frames reference incorrectly replaced

User facing changelog

Experimental fixes

To access these fixes, enable experimental source rewriting with the config value experimentalSourceRewriting: true:

• Fixed issues where use of window.top and window.parent could cause the AUT to break out of the Cypress iframe. Fixes Replace x.top, x.parent accessors with correct reference (at runtime) #5271 and Rewrite 3rd party JS code to enforce that parent + top references are always correct #1467.
• Fixed an issue where window.frames, window.parent.frames, and other frames accesses could point to the wrong reference after being proxied through Cypress. Fixes Can't use window.parent.frames in my iframes  #2664.
• Fixed an issue where scripts using the integrity attribute for sub-resource integrity (SRI) would not load after being proxied through Cypress. Fixes Cypress cannot test sites that implement SRI #2393.
• Fixed an issue where setting location or location.href to a relative href, or using location.replace or location.assign with a relative href, could result in the wrong URL being navigated to. Fixes Incorrect redirect to malformed url within AUT - adding _/ in url #3975 and window.location.replace not working - inserts __ into url #3994.

Additional details

• Adds AST-based rewriting behind experimental flag experimentalSourceRewriting:
  
• Explore serving source maps so users can still debug their original source code: https://github.com/benjamn/recast#source-maps
  • may possibly conflict with Error Improvements #6724 go-to-line functionality error improvements only consume sourcemaps for specfiles, not AUT JS (yet) - so we can get away with simply inlining sourcemaps for now
  • sourcemaps will be served over HTTP, we don't want the overhead of generating them for every single JS loaded - unfortunately this requires some caching of JS
  • also intercept SourceMap and X-SourceMap headers
    • order of preference: inlined sourcemappingurl, sourcemap header, x-sourcemap header
• Strip script integrity parameters - fixes Cypress cannot test sites that implement SRI #2393
  • SRI integrity attributes in <script type="text/javascript"> tags will be rewritten to cypress:stripped-integrity attributes - <script type="text/javascript" integrity="foo"> becomes <script type="text/javascript" cypress:stripped-integrity="foo">
• Cache rewritten code by ETags and potentially with a hash so it only needs 1 rewrite per session
  • HTTP caching semantics are hard, and existing cache mechanisms will still work inside a single browser session, so this can be implemented later
• Fix SIGSEGV when exiting with threads active: SIGABRT when calling process.exit with worker_threads active electron/electron#23366
  • fixed with an ugly hack on intercepting process.exit, 😢
• Throw error in driver if AST rewriting fails, since this is a bug in Cypress.

Note: This PR is huge because it does not replace the regex-based rewriting (yet), so there are some duplicate code paths and tests.

Checklist before removing "experimental" status: #7297

How has the user experience changed?

More user sites will work OOTB with Cypress 🎉

PR Tasks

• Have tests been added/updated?
• Has a PR for user-facing changes been opened in cypress-documentation? Document experimentalSourceRewriting cypress-documentation#2787
• Have API changes been updated in the type definitions?
• Have new configuration options been added to the cypress.schema.json?
• Issues tagged for release?

[cypress]

---

Test summary

7402 • 0 • 102 • 0

---

Run details

Project	cypress
Status	Passed
Commit	5877e66
Started	May 11, 2020 4:26 PM
Ended	May 11, 2020 4:32 PM
Duration	06:05 💡
OS	Linux Debian - 10.1
Browser	Multiple

View run in Cypress Dashboard ➡️

---

This comment has been generated by cypress-bot as a result of this project's GitHub integration settings. You can manage this integration in this project's settings in the Cypress Dashboard

[brian-mann]

@flotwig can we also fix this issue with this PR? #5099

[brian-mann]

@flotwig another important note - if we apply these rules to the entire proxy then we will be rewriting virtually all JS files - which will break <script> tags using the integrity attribute.

We'll need to strip those too. https://developer.mozilla.org/en-US/docs/Web/Security/Subresource_Integrity

[brian-mann]

@flotwig another note - we should capture when window.location setters are used to navigate the page in JS such as window.location = '...' and window.location.href = '...' - so we can notify Cypress and capture the stack trace as well.

[flotwig]

@flotwig another important note - if we apply these rules to the entire proxy then we will be rewriting virtually all JS files - which will break <script> tags using the integrity attribute.

We'll need to strip those too. developer.mozilla.org/en-US/docs/Web/Security/Subresource_Integrity

This uses hyntax to do streaming parsing of HTML, so this should be easy to add. It will be the first transform that we do directly to the HTML.

[jennifer-shehane]

Will this also address this issue? #3994

[flotwig]

@jennifer-shehane it will enable us to build a workaround for that issue, I believe, by making it so we can rewrite the use of window.location.(href|replace)

[kuceb]

made suggested changes as commit

[drumbeg]

Early testing looking very promising on #4576.

[jennifer-shehane]

@drumbeg This fix is available starting in 4.6.0 as an experiment which you can access by setting this config option in your cypress.json or elsewhere:

{
	"experimentalSourceRewriting": true
}

The fix is experimental, so there may be some situations where the this is not fixed.

If you're still this issue while setting the experimentalSourceRewriting to true in 4.6.0 - open a new issue with a reproducible example + screenshots, etc - filling out our issue template.