I have a heavy JavaScript file on the server (> 3MB). I want to load the page fast and show a loading progress bar to the user. Currently, I am using fetch and WritableStream to download the data and to track the download progress as:
let resource = await fetch('heavy_file.js')
resource.clone().body.pipeTo(new WritableStream({
write(t) { on_receive(t.length) }
}))
And then I am using the Function to evaluate it. This has several problems. How can I:
Load the script preferably using fetch (I'm using the same method to load WASM files, I want to track their download progress as well, and the WebAssembly.compileStreaming API requires the usage of fetch).
Track the download progress in a way that would work across modern browsers nowadays.
Be able to use this solution without enabling script-src 'unsafe-eval' in Content-Security-Policy?
PS.
Of course, currently, we need to use script-src 'wasm-eval' in Chrome when loading WASM files until the bug is fixed.
Option 1
If you can calculate the hash of the script you're loading in advance (if it's not something generated dynamically), then a simple option to avoid
enabling script-src 'unsafe-eval' in Content-Security-Policy
would be to add only a hash of that specific script into the CSP header - this will still ensure that you're not executing any untrusted code, while allowing you to load and execute script manually.
MDN has some more examples of implementing such CSP policies here.
As for loading itself, you have two different paths from here:
Option 1.1
Combine the hash with a CSP3 policy unsafe-hashes which will allow you to keep using Function or eval like you currently, while still limiting code only to trusted.
For example, if you have a script like
alert('Hello, world.');
then your CSP header should contain
Content-Security-Policy: script-src 'unsafe-hashes' 'sha256-qznLcsROx4GACP2dm0UCKCzCG-HiZ1guq6ZZDob_Tng='
Unfortunately, CSP level 3, or, at least, this option is supported only in Chromium at the moment of writing.
Option 1.2
Instead of using Function or eval, you can dynamically create a script tag from JavaScript, populate its textContent with your response content and inject it into the DOM:
let resource = await fetch('heavy_file.js')
resource.clone().body.pipeTo(new WritableStream({
write(t) { on_receive(t.length) }
}));
resource.text().then(res => {
let s = document.createElement('script');
s.textContent = res;
document.head.appendChild(s);
});
In this case you only need to add the hash of the script to the CSP and it will work across all browsers:
Content-Security-Policy: script-src 'sha256-qznLcsROx4GACP2dm0UCKCzCG-HiZ1guq6ZZDob_Tng='
Option 2
If you do want to support dynamically generated scripts, then your only option might be to move your progress-tracking code to a Service Worker, and then use Client.postMessage to communicate progress to a script on the page.
This will work for any content served from your origin, but only once Service Worker is installed - usually on a subsequent page load, and so might not help you if the large script you're loading is part of the page user visits first on your website.
Related
I have a Chrome extension that adds a panel to the page in the floating iframe (on extension button click). There's certain JS code that is downloaded from 3rd party host and needs to be executed on that page. Obviously there's XSS issue and extension needs to comply with content security policies for that page.
Previously I had to deal with CSP directives that are passed via request headers, and was able to override those via setting a hook in chrome.webRequest.onHeadersReceived. There I was adding my host URL to content-security-policy headers. It worked. Headers were replaced, new directives applied to the page, all good.
Now I discovered websites that set the CSP directives via <meta> tag, they don't use request headers. For example, app pages in iTunes https://itunes.apple.com/us/app/olympics/id808794344?mt=8 have such. There is also an additional meta tag with name web-experience-app/config/environment (?) that somewhat duplicates the values that are set in content of tag with http-equiv="Content-Security-Policy".
This time I am trying to add my host name into meta tag inside chrome.webNavigation.onCommitted or onCompleted events listeners (JS vanilla via chrome.tabs.executeScript). I also experimented with running the same code from the webrequest's onCompleted listener (at the last step of lifecycle according to https://developer.mozilla.org/en-US/Add-ons/WebExtensions/API/webRequest).
When I inspect the page after its load - I see the meta tags have changed. But when I click on my extension to start loading iframe and execute JS - console prints the following errors:
Refused to frame 'https://myhost.com' because it violates the following Content Security Policy directive: "frame-src 'self' *.apple.com itmss: itms-appss: itms-bookss: itms-itunesus: itms-messagess: itms-podcasts: itms-watchs: macappstores: musics: apple-musics:".
I.e. my tag update was not effective.
I have several questions: first, do I do it right? Am I doing the update at the proper event? When is the data content from meta tags being read in the page lifecycle? Will it be auto-applied after tag content change?
As of March 2018 Chromium doesn't allow to modify the responseBody of the request. https://bugs.chromium.org/p/chromium/issues/detail?id=487422#c29
"WebRequest API: allow extensions to read response body" is a ticket from 2015. It is not on a path of getting to be resolved and needs some work/help.
--
Firefox has the webRequest filter implementation that allows to modify the response body before the page's meta directives are applied.
https://developer.mozilla.org/en-US/Add-ons/WebExtensions/API/webRequest/filterResponseData
BUT, my problem is focused on fixing the Chrome extension. Maybe Chrome picks this up one day.
--
In general, the Chrome's extension building framework seems like not a reliable path of building a long term living software; with browser vendors changing the rules frequently, reacting to newly discovered threats, having no up-to-date supported cross-browser standard.
--
In my case, the possible way around this issue can be to throw all the JS code into the extension's source base. Such that there's no 3-rd party to connect to fetch and execute the JS (and conflict with/violate the CSP rules). Haven't explored this yet, as I expected to reuse the code & interactive components I am using in my main browser application.
I've been interested in the same things and here are a few aspects that perhaps could help:
chrome.debugger extension API with Fetch (or Network) domain can be used to modify responseBody: https://chromedevtools.github.io/devtools-protocol/tot/Fetch/
This is an example implementation: https://github.com/mr-yt12/Debugger-API-Fetch-example-Chrome-Extension
However, I'm facing a problem of Fetch.requestPaused not firing on the first page load: Chrome Extension Debugger API, Fetch domain attaches/enables too late for the response body to be intercepted
And I haven't found a solution to this yet, besides first redirecting the request to 'http://google.com/gen_204' and then updating the tab. But this creates flicker and also I'm not sure if it's possible to redirect the request like this with manifest v3.
When using debugger API, Chrome shows a warning at the top, which changes the page's size and doesn't go away (perhaps it goes away 5 seconds after the debugger is detached and also if the user clicks "cancel"). This means that it's mostly only good for personal use or when distributing as a developer mode extension. Using --silent-debugger-extension-api flag with Chrome disables this warning.
I've tried injecting my script at document_start (when the meta tag is not yet created). Then I used mutationObserver (also tried other methods) to wait for the meta tag, to modify it before it's applied or fully created. Somehow it succeeded once or a few times, but could be a coincidence or a wrong interpretation of results. Perhaps it's worth experimenting with it.
Another idea (but I think I didn't make it work, but perhaps it's possible) is to use window.stop() at document_start and then rewrite the html content programmatically. This needs more researching.
It seems that meta tag CSP is applied once the meta tag is created (or while it's being created), and then there is no way to cancel what's been applied. It should be researched more on how to prevent it from applying or modifying it before it's fully created or applied.
I want to create a custom profiler for Javascript as a Chrome DevTools Extension. To do so, I'd have to instrument all Javascript code of a website (parse to AST, inject hooks, generate new source). This should've been easily possible using chrome.devtools.inspectedWindow.reload() and its parameter preprocessorScript described here: https://developer.chrome.com/extensions/devtools_inspectedWindow.
Unfortunately, this feature has been removed (https://bugs.chromium.org/p/chromium/issues/detail?id=438626) because nobody was using it.
Do you know of any other way I could achieve the same thing with a Chrome Extension? Is there any other way I can replace an incoming Javascript source with a changed version? This question is very specific to Chrome Extensions (and maybe extensions to other browsers), I'm asking this as a last resort before going a different route (e.g. dedicated app).
Use the Chrome Debugging Protocol.
First, use DOMDebugger.setInstrumentationBreakpoint with eventName: "scriptFirstStatement" as a parameter to add a break-point to the first statement of each script.
Second, in the Debugger Domain, there is an event called scriptParsed. Listen to it and if called, use Debugger.setScriptSource to change the source.
Finally, call Debugger.resume each time after you edited a source file with setScriptSource.
Example in semi-pseudo-code:
// Prevent code being executed
cdp.sendCommand("DOMDebugger.setInstrumentationBreakpoint", {
eventName: "scriptFirstStatement"
});
// Enable Debugger domain to receive its events
cdp.sendCommand("Debugger.enable");
cdp.addListener("message", (event, method, params) => {
// Script is ready to be edited
if (method === "Debugger.scriptParsed") {
cdp.sendCommand("Debugger.setScriptSource", {
scriptId: params.scriptId,
scriptSource: `console.log("edited script ${params.url}");`
}, (err, msg) => {
// After editing, resume code execution.
cdg.sendCommand("Debugger.resume");
});
}
});
The implementation above is not ideal. It should probably listen to the breakpoint event, get to the script using the associated event data, edit the script and then resume. Listening to scriptParsed and then resuming the debugger are two things that shouldn't be together, it could create problems. It makes for a simpler example, though.
On HTTP you can use the chrome.webRequest API to redirect requests for JS code to data URLs containing the processed JavaScript code.
However, this won't work for inline script tags. It also won't work on HTTPS, since the data URLs are considered unsafe. And data URLs are can't be longer than 2MB in Chrome, so you won't be able to redirect to large JS files.
If the exact order of execution of each script isn't important you could cancel the script requests and then later send a message with the script content to the page. This would make it work on HTTPS.
To address both issues you could redirect the HTML page itself to a data URL, in order to gain more control. That has a few negative consequences though:
Can't reload page because URL is fixed to data URL
Need to add or update <base> tag to make sure stylesheet/image URLs go to the correct URL
Breaks ajax requests that require cookies/authentication (not sure if this can be fixed)
No support for localStorage on data URLs
Not sure if this works: in order to fix #1 and #4 you could consider setting up an HTML page within your Chrome extension and then using that as the base page instead of a data URL.
Another idea that may or may not work: Use chrome.debugger to modify the source code.
I'm using a service that automatically constructs and hosts launch pages. In the body of their code, they have a call to jquery:
<script type="text/javascript" src="//ajax.googleapis.com/ajax/libs/jquery/2.1.1/jquery.min.js?cache=2015-09-22-09">
Unfortunately, since I'm in China, googleapis.com is blocked and the page has to wait for this code to timeout before it will render. This portion is autogenerated as part of the template and I can't change it. However I can insert custom javascript in the head and the body.
Are there any ways I can prevent this code from making the request to googleapis.com or to force it to abort after it has already made the request?
-EDIT-
Here's a screen cap of the network tab when I try to load the page. As you can see, the call to googleapis.com hangs for 1.4 mins until it times out, at which point DomContentLoaded triggers and the entire page loads.
Right, if you are able to put html in the head of the document, not just execute javascript you could use a meta tag to block external script loading:
<meta http-equiv="Content-Security-Policy" content="script-src 'self'">
From the Mozilla Content-Security-Policy Meta Tag Docs:
Authors are strongly encouraged to place meta elements as early in the document as possible, because policies in meta elements are not applied to content which preceds them. In particular, note that resources fetched or prefetched using the Link HTTP response header field, and resources fetched or prefetched using link and script elements which precede a meta-delivered policy will not be blocked.
So the meta tag will only work in the head, certainly before the script which loads jQuery. You can whitelist URL's in the tag by adding them into the content parameter in the meta tag too.
If you can only execute javascript, you can add the meta tag dynamically. Unfortunately it is likely the browser has probably decided on it's policies by the time it is added. Nevertheless, it can be added with
var meta = document.createElement('meta');
meta.httpEquiv = "Content-Security-Policy";
meta.content = "script-src 'self'";
document.getElementsByTagName('head')[0].appendChild(meta);
More Interesting reading material homework for solving the 'Prevent an external js request' mystery:
Use JavaScript to prevent a later `<script>` tag from being evaluated?
Good Luck!
Consider using a Content Security Policy. It would be an unusual use case, but with the CSP you can tell the browser that it is not allowed to access googleapis.com before your government even gets a say in the matter. In this way, the browser won't even try to load it, and the page will not hang.
Yeah. #Niet's suggestion seems nice. To add to his answer, here's how you can block rendering of googleapi domain using CSP:
Content-Security-Policy: script-src 'self';
This code would instruct the browser to only execute YOUR own domain's scripts.
If you have access to the web server and its windows you could add an entry in the Hosts file to redirect the google address to the local server web application to download the javascript from there? (if you match the folder structure of the javascript link)
c:\windows\system32\drivers\etc\hosts
ajax.googleapis.com 127.0.0.1
Content Script can be injected programatically or permanently by declaring in Extension manifest file. Programatic injection require host permission, which is generally grant by browser or page action.
In my use case, I want to inject gmail, outlook.com and yahoo mail web site without user action. I can do by declaring all of them manifest, but by doing so require all data access to those account. Some use may want to grant only outlook.com, but not gmail. Programatic injection does not work because I need to know when to inject. Using tabs permission is also require another permission.
Is there any good way to optionally inject web site?
You cannot run code on a site without the appropriate permissions. Fortunately, you can add the host permissions to optional_permissions in the manifest file to declare them optional and still allow the extension to use them.
In response to a user gesture, you can use chrome.permission.request to request additional permissions. This API can only be used in extension pages (background page, popup page, options page, ...). As of Chrome 36.0.1957.0, the required user gesture also carries over from content scripts, so if you want to, you could add a click event listener from a content script and use chrome.runtime.sendMessage to send the request to the background page, which in turn calls chrome.permissions.request.
Optional code execution in tabs
After obtaining the host permissions (optional or mandatory), you have to somehow inject the content script (or CSS style) in the matching pages. There are a few options, in order of my preference:
Use the chrome.declarativeContent.RequestContentScript action to insert a content script in the page. Read the documentation if you want to learn how to use this API.
Use the webNavigation API (e.g. chrome.webNavigation.onCommitted) to detect when the user has navigated to the page, then use chrome.tabs.executeScript to insert the content script in the tab (or chrome.tabs.insertCSS to insert styles).
Use the tabs API (chrome.tabs.onUpdated) to detect that a page might have changed, and insert a content script in the page using chrome.tabs.executeScript.
I strongly recommend option 1, because it was specifically designed for this use case. Note: This API was added in Chrome 38, but only worked with optional permissions since Chrome 39. Despite the "WARNING: This action is still experimental and is not supported on stable builds of Chrome." in the documentation, the API is actually supported on stable. Initially the idea was to wait for a review before publishing the API on stable, but that review never came and so now this API has been working fine for almost two years.
The second and third options are similar. The difference between the two is that using the webNavigation API adds an additional permission warning ("Read your browsing history"). For this warning, you get an API that can efficiently filter the navigations, so the number of chrome.tabs.executeScript calls can be minimized.
If you don't want to put this extra permission warning in your permission dialog, then you could blindly try to inject on every tab. If your extension has the permission, then the injection will succeed. Otherwise, it fails. This doesn't sound very efficient, and it is not... ...on the bright side, this method does not require any additional permissions.
By using either of the latter two methods, your content script must be designed in such a way that it can handle multiple insertions (e.g. with a guard). Inserting in frames is also supported (allFrames:true), but only if your extension is allowed to access the tab's URL (or the frame's URL if frameId is set).
I advise against using declarativeContent APIs because they're deprecated and buggy with CSS, as described by the last comment on https://bugs.chromium.org/p/chromium/issues/detail?id=708115.
Use the new content script registration APIs instead. Here's what you need, in two parts:
Programmatic script injection
There's a new contentScripts.register() API which can programmatically register content scripts and they'll be loaded exactly like content_scripts defined in the manifest:
browser.contentScripts.register({
matches: ['https://your-dynamic-domain.example.com/*'],
js: [{file: 'content.js'}]
});
This API is only available in Firefox but there's a Chrome polyfill you can use. If you're using Manifest v3, there's the native chrome.scripting.registerContentScript which does the same thing but slightly differently.
Acquiring new permissions
By using chrome.permissions.request you can add new domains on which you can inject content scripts. An example would be:
// In a content script or options page
document.querySelector('button').addEventListener('click', () => {
chrome.permissions.request({
origins: ['https://your-dynamic-domain.example.com/*']
}, granted => {
if (granted) {
/* Use contentScripts.register */
}
});
});
And you'll have to add optional_permissions in your manifest.json to allow new origins to be requested:
{
"optional_permissions": [
"*://*/*"
]
}
In Manifest v3 this property was renamed to optional_host_permissions.
I also wrote some tools to further simplify this for you and for the end user, such as
webext-domain-permission-toggle and webext-dynamic-content-scripts. They will automatically register your scripts in the next browser launches and allow the user the remove the new permissions and scripts.
Since the existing answer is now a few years old, optional injection is now much easier and is described here. It says that to inject a new file conditionally, you can use the following code:
// The lines I have commented are in the documentation, but the uncommented
// lines are the important part
//chrome.runtime.onMessage.addListener((message, callback) => {
// if (message == “runContentScript”){
chrome.tabs.executeScript({
file: 'contentScript.js'
});
// }
//});
You will need the Active Tab Permission to do this.
I want to be able to access /robots.txt from a variety of sites using JavaScript. This is for a side project that tests the availability of sites, not all of which are under my control. I've tried this:
$.get(robotsUrl, function() {
console.log('success!');
}, "text")
.fail(function() {
console.log('failed :(');
});
However, this fails with
XMLHttpRequest cannot load https://my.test.url/robots.txt. Origin http://localhost:8000 is not allowed by Access-Control-Allow-Origin
MDN's page on Same-Origin-Policy says that it's possible to embed content with some elements, such as <script>, <iframe> <embed>. Could I load /robots.txt from an arbitrary site with any of these? Is there any other way I can access this file on other domains?
You could load it with any of them, you just won't be able to make the data available to JavaScript. That's rather the point of the Same Origin Policy.
If you want to get arbitrary data from arbitrary sites, you need to do it server side.
To get around a same origin policy, you need to either have control over the host site and set the allow-origin (not an option here), or load it by a method other than JavaScript (which JSONP does; it is loaded as a standard script).
That means you could display the robots.txt in an iframe, for example, by just setting its src attribute.
If you want to manipulate the contents in JavaScript, that won't work (even after you load the content in an iframe, you're still not allowed to interact with it). Your final option is to set up a proxy. Have a script on your server which when called will load the relevant file and redirect the content. It's not hard to do, but means your server will have higher traffic (and you'll need to lock it down so that it isn't used maliciously).
iframes won't let you peek at the content. You could show it to your user, but I'm guessing you want to analyze it with code.
You could do it on your server. Even if you just have a /cors/robots/domain.tld handler (and others for other files you need to access). This is probably the best way, if it's feasible for your situation.
AnyOrigin, is a free service allows you to make cross-origin requests.
$.getJSON('http://anyorigin.com/get?url=google.com/robots.txt&callback=?', function(data){
console.log(data.contents); // contents of Robots.txt
});
Pretty sure this is possible with Chrome by runnning the browser with the Same Origin Policy disabled: Disable same origin policy in Chrome.
It may be preferable to do something like this outside the context of a browser however, on the command line perhaps using something like CURL?