webRequest API: How to get the requestId of a new request? - javascript

The chrome.webRequest API has the concept of a request ID (source: Chrome webRequest documention):
Request IDs
Each request is identified by a request ID. This ID is unique within a browser session and the context of an extension. It remains constant during the the life cycle of a request and can be used to match events for the same request. Note that several HTTP requests are mapped to one web request in case of HTTP redirection or HTTP authentication.
You can use it to correlate the requests even across redirects. But how do you initially get hold off the id when start a new request with fetch or XMLHttpRequest?
So far, I have not found anything better than to use the URL of the request as a way to make the initial link between the new request and the requestId. However, if there are overlapping requests to the same resource, this is not reliable.
Questions:
If you make a new request (either with fetch or XMLHttpRequest), how do you reliably get access to the requestId?
Does the fetch API or XMLHttpRequest API allow access to the requestId?
What I want to do is to use the functionality provided by the webRequest API to modify a single request, but I want to make sure that I do not accidentally modify other pending requests.

To the best of my knowledge, there is no direct support in the fetch or XHMLHttpRequest API. Also I'm not aware of completely reliable way to get hold of the requestId.
What I ended up doing was installing a onBeforeRequest listener, storing the requestId, and then immediately removing the listener again. For instance, it could look like this:
function makeSomeRequest(url) {
let listener;
const removeListener = () => {
if (listener) {
chrome.webRequest.onBeforeRequest.removeListener(listener);
listener = null;
}
};
let requestId;
listener = (details) => {
if (!requestId && urlMatches(details.url, url)) {
requestId = details.requestId;
removeListener();
}
};
chrome.webRequest.onBeforeRequest.addListener(listener, { urls: ['<all_urls>'] });
// install other listeners, which can then use the stored "requestId"
// ...
// finally, start the actual request, for instance
const promise = fetch(url).then(doSomething);
// and make sure to always clean up the listener
promise.then(removeListener, removeLister);
}
It is not perfect, and matching the URL is a detail that I left open. You could simply compare whether the details.url is identical to url:
function urlMatches(url1, url2) {
return url1 === url2;
}
Note that it is not guaranteed that you see the identical URL, for instance, if make a request against http://some.domain.test, you will see http://some.domain.test/ in your listener (see my other question about the details). Or http:// could have been replaced by https:// (here I'm not sure, but it could be because of other extensions like HTTPS Everywhere).
That is why the code above should only be seen as a sketch of the idea. It seems to work good enough in practice, as long as you do not start multiple requests to the identical URL. Still, I would be interested in learning about a better way to approach the problem.

Related

perform event just before page change

I have a React app. In that react app, I have certain button click events which are triggered just before page changes (analytics event)
sendEvent('xyz')
let win = window.open(`${PAGE_PATHS.dashboard}`, "_self");
if (win !== null) {
win.focus();
}
Where event is
async event(name, data) {
await this.sendEvent(name, data)
}
Where sendEvent() is
async sendEvent(name, data) {
// some code
axios.post(url, payload, userConfig)
return
}
Now, because of the current structure, the events sometimes don't get logged. There are two things which I could do here.
Use await but I don't want to do it because sending event and getting response might take some time (user experience) and I don't care about the response
Use setTimeout.
For some reason I don't like either one of the approach. Is there a way, I can have the task to execute (if it isn't completed) even after the webpage have changed it's href? maybe using service or webworker
What you are doing is commonly called "sending a beacon", and there is a method in the Web standards especially for this case: Navigator.sendBeacon().
This method will allow your script to send a request to the web server, even after the page has been killed.
I am not an axios ninja, so I can't tell you how it should be rewritten to perform the same, but certainly it can.
The basic usage is:
navigator.sendBeacon(url, data);
where data can be an ArrayBuffer, a TypedArray, a Blob, a DOMString, FormData or an URLSearchParams object.

How can we use electron.protocol.interceptFileProtocol with only certain paths, leaving other requests untouched?

I'd like to intercept certain HTTP requests and replace them with files. So I thought I could use electron.protocol.interceptFileProtocol like so:
protocol.interceptFileProtocol('http', (request, callback) => {
// intercept only requests to "http://example.com"
if (request.url.startsWith("http://example.com")) {
callback("/path/to/file")
}
// otherwise, let the HTTP request behave like normal.
// But how?
})
How do we allow other http requests other than http://example.com to continue working as normal?
When using protocol.interceptXXXXProtocol(scheme, handler), we are intercepting scheme protocol and uses handler as the protocol’s new handler which sends a new XXXX request as a response, as said in the doc here.
However, doing so totally breaks the initial handler for this specific protocol, which we would need after handling the callback execution. Thus, we just need to restore it back to its initial state, so that it can continue working as normal :)
Let's use: protocol.uninterceptProptocol(scheme)
protocol.interceptFileProtocol('http', (request, callback) => {
// intercept only requests to "http://example.com"
if (request.url.startsWith("http://example.com")) {
callback("/path/to/file")
}
// otherwise, let the HTTP request behave like normal.
protocol.uninterceptProtocol('http');
})
Not sure if there is a way to do this exactly? but I did something similar which is to use session.defaultSession.webRequest.onBeforeRequest
See: https://developer.mozilla.org/en-US/docs/Mozilla/Add-ons/WebExtensions/API/webRequest
something like
session.defaultSession.webRequest.onBeforeRequest({urls: ['http://example.com']}, function(details, callback) {
callback({
redirectURL: 'file://' + this.getUrl(details.url)
});
});
If you need more than a redirect you could redirect to your own custom protocol (eg. a url like mycustomprotocol://...). You can implement your own protocol handler with protocol.registerStringProtocol, etc.
I used both onBeforeRequest and registerStringProtocol separately in electron without issues so far but never both together - should work together though I geuss.

v-on:change method only triggering a get request once (Vue.js)

I am using Vue.js and Choices.js javascript plugin and I have to dynamically populate values of two select fields via ajax.
What I am trying achieve is initate a get request at page load and populate the universities select, and after a value in universities select is chosen start a new getrequest to populate the faculties select.
What is happening is that when I pick the university for the first time, everything will work normally. For example if I pick a university option with value="1" an ajax get request will be sent to /faculties?university_id=1.The console log will print onChange startedso we are sure the method is running correctly; the appropriate v-model="selectedUniversity"is updating too.
If I now change the value of the select field again, the ajax function won't be called anymore and no additional requests will be done to the server. The console.logwill still run, and the v-modelis still being updated. Does anyone understand what is going on here?
var Choices = require('choices.js');
module.exports = {
data: function() {
return {
selectedUniversity: '',
selectedFaculty: '',
universities: {},
faculties: {}
}
},
mounted: function () {
var self = this;
var universitySelect = new Choices(document.getElementById('university'));
universitySelect.ajax(function(callback) {
fetch('/universities')
.then(function(response) {
response.json().then(function(data) {
callback(data, 'id', 'name');
self.universities = data;
});
})
.catch(function(error) {
console.log(error);
});
});
},
methods: {
onChange: function () {
console.log("onChange started");
var self = this;
var url = '/faculties?university_id=' + self.selectedUniversity;
var facultySelect = new Choices(document.getElementById('faculty'));
//This part below only runs the first time when the university select is selected
facultySelect.ajax(function(callback) {
fetch(url)
.then(function(response) {
response.json().then(function(data) {
callback(data, 'id', 'name');
self.faculties = data;
});
})
.catch(function(error) {
console.log(error);
});
});
}
}
}
The Headers are set like this:
I think your request URL /faculties?university_id=1 is cached and that's why it worked on first time and second time, the response is coming from the cached response.
In your fetch API, set cache mode to ignore the cached response,
fetch(url, {cache: "no-store"}).then(....)
For complete list of cache modes for fetch() API,
https://hacks.mozilla.org/2016/03/referrer-and-cache-control-apis-for-fetch/
In case if above link is unavailable,
Fetch cache control APIs
The idea behind this API is specifying a caching policy for fetch to explicitly indicate how and when the browser HTTP cache should be consulted. It’s important to have a good understanding of the HTTP caching semantics in order to use these most effectively. There are many good articles on the web such as this one that describe these semantics in detail. There are currently five different policies that you can choose from.
“default” means use the default behavior of browsers when downloading resources. The browser first looks inside the HTTP cache to see if there is a matching request. If there is, and it is fresh, it will be returned from fetch(). If it exists but is stale, a conditional request is made to the remote server and if the server indicates that the response has not changed, it will be read from the HTTP cache. Otherwise it will be downloaded from the network, and the HTTP cache will be updated with the new response.
“no-store” means bypass the HTTP cache completely. This will make the browser not look into the HTTP cache on the way to the network, and never store the resulting response in the HTTP cache. Using this cache mode, fetch() will behave as if no HTTP cache exists.
“reload” means bypass the HTTP cache on the way to the network, but update it with the newly downloaded response. This will cause the browser to never look inside the HTTP cache on the way to the network, but update the HTTP cache with the downloaded response. Future requests can use that updated response if appropriate.
“no-cache” means always validate a response that is in the HTTP cache even if the browser thinks that it’s fresh. This will cause the browser to look for a matching request in the HTTP cache on the way to the network. If such a request is found, the browser always creates a conditional request to validate it even if it thinks that the response should be fresh. If a matching cached entry is not found, a normal request will be made. After a response has been downloaded, the HTTP cache will always be updated with that response.
“force-cache” means that the browser will always use a cached response if a matching entry is found in the cache, ignoring the validity of the response. Thus even if a really old version of the response is found in the cache, it will always be used without validation. If a matching entry is not found in the cache, the browser will make a normal request, and will update the HTTP cache with the downloaded response.
Let’s look at a few examples of how you can use these cache modes.
// Download a resource with cache busting, to bypass the cache
// completely.
fetch("some.json", {cache: "no-store"})
.then(function(response) { /* consume the response */ });
// Download a resource with cache busting, but update the HTTP
// cache with the downloaded resource.
fetch("some.json", {cache: "reload"})
.then(function(response) { /* consume the response */ });
// Download a resource with cache busting when dealing with a
// properly configured server that will send the correct ETag
// and Date headers and properly handle If-Modified-Since and
// If-None-Match request headers, therefore we can rely on the
// validation to guarantee a fresh response.
fetch("some.json", {cache: "no-cache"})
.then(function(response) { /* consume the response */ });
// Download a resource with economics in mind! Prefer a cached
// albeit stale response to conserve as much bandwidth as possible.
fetch("some.json", {cache: "force-cache"})
.then(function(response) { /* consume the response */ });

How to read http headers of last response? [duplicate]

How do I access a page's HTTP response headers via JavaScript?
Related to this question, which was modified to ask about accessing two specific HTTP headers.
Related:
How do I access the HTTP request header fields via JavaScript?
It's not possible to read the current headers. You could make another request to the same URL and read its headers, but there is no guarantee that the headers are exactly equal to the current.
Use the following JavaScript code to get all the HTTP headers by performing a get request:
var req = new XMLHttpRequest();
req.open('GET', document.location, false);
req.send(null);
var headers = req.getAllResponseHeaders().toLowerCase();
alert(headers);
Unfortunately, there isn't an API to give you the HTTP response headers for your initial page request. That was the original question posted here. It has been repeatedly asked, too, because some people would like to get the actual response headers of the original page request without issuing another one.
For AJAX Requests:
If an HTTP request is made over AJAX, it is possible to get the response headers with the getAllResponseHeaders() method. It's part of the XMLHttpRequest API. To see how this can be applied, check out the fetchSimilarHeaders() function below. Note that this is a work-around to the problem that won't be reliable for some applications.
myXMLHttpRequest.getAllResponseHeaders();
The API was specified in the following candidate recommendation for XMLHttpRequest: XMLHttpRequest - W3C Candidate Recommendation 3 August 2010
Specifically, the getAllResponseHeaders() method was specified in the following section: w3.org: XMLHttpRequest: the getallresponseheaders() method
The MDN documentation is good, too: developer.mozilla.org: XMLHttpRequest.
This will not give you information about the original page request's HTTP response headers, but it could be used to make educated guesses about what those headers were. More on that is described next.
Getting header values from the Initial Page Request:
This question was first asked several years ago, asking specifically about how to get at the original HTTP response headers for the current page (i.e. the same page inside of which the javascript was running). This is quite a different question than simply getting the response headers for any HTTP request. For the initial page request, the headers aren't readily available to javascript. Whether the header values you need will be reliably and sufficiently consistent if you request the same page again via AJAX will depend on your particular application.
The following are a few suggestions for getting around that problem.
1. Requests on Resources which are largely static
If the response is largely static and the headers are not expected to change much between requests, you could make an AJAX request for the same page you're currently on and assume that they're they are the same values which were part of the page's HTTP response. This could allow you to access the headers you need using the nice XMLHttpRequest API described above.
function fetchSimilarHeaders (callback) {
var request = new XMLHttpRequest();
request.onreadystatechange = function () {
if (request.readyState === XMLHttpRequest.DONE) {
//
// The following headers may often be similar
// to those of the original page request...
//
if (callback && typeof callback === 'function') {
callback(request.getAllResponseHeaders());
}
}
};
//
// Re-request the same page (document.location)
// We hope to get the same or similar response headers to those which
// came with the current page, but we have no guarantee.
// Since we are only after the headers, a HEAD request may be sufficient.
//
request.open('HEAD', document.location, true);
request.send(null);
}
This approach will be problematic if you truly have to rely on the values being consistent between requests, since you can't fully guarantee that they are the same. It's going to depend on your specific application and whether you know that the value you need is something that won't be changing from one request to the next.
2. Make Inferences
There are some BOM properties (Browser Object Model) which the browser determines by looking at the headers. Some of these properties reflect HTTP headers directly (e.g. navigator.userAgent is set to the value of the HTTP User-Agent header field). By sniffing around the available properties you might be able to find what you need, or some clues to indicate what the HTTP response contained.
3. Stash them
If you control the server side, you can access any header you like as you construct the full response. Values could be passed to the client with the page, stashed in some markup or perhaps in an inlined JSON structure. If you wanted to have every HTTP request header available to your javascript, you could iterate through them on the server and send them back as hidden values in the markup. It's probably not ideal to send header values this way, but you could certainly do it for the specific value you need. This solution is arguably inefficient, too, but it would do the job if you needed it.
Using XmlHttpRequest you can pull up the current page and then examine the http headers of the response.
Best case is to just do a HEAD request and then examine the headers.
For some examples of doing this have a look at http://www.jibbering.com/2002/4/httprequest.html
Just my 2 cents.
A solution with Service Workers
Service workers are able to access network information, which includes headers. The good part is that it works on any kind of request, not just XMLHttpRequest.
How it works:
Add a service worker on your website.
Watch every request that's being sent.
Make the service worker fetch the request with the respondWith function.
When the response arrives, read the headers.
Send the headers from the service worker to the page with the postMessage function.
Working example:
Service workers are a bit complicated to understand, so I've built a small library that does all this. It is available on github: https://github.com/gmetais/sw-get-headers.
Limitations:
the website needs to be on HTTPS
the browser needs to support the Service Workers API
the same-domain/cross-domain policies are in action, just like on XMLHttpRequest
Another way to send header information to JavaScript would be through cookies. The server can extract whatever data it needs from the request headers and send them back inside a Set-Cookie response header — and cookies can be read in JavaScript. As keparo says, though, it's best to do this for just one or two headers, rather than for all of them.
(2021) An answer without additional HTTP call
While it's not possible in general to read arbitrary HTTP response headers of the top-level HTML navigation, if you control the server (or middleboxes on the way) and want to expose some info to JavaScript that can't be exposed easily in any other way than via a header:
You may use Server-Timing header to expose arbitrary key-value data, and it will be readable by JavaScript.
(*in supported browsers: Firefox 61, Chrome 65, Edge 79; no Safari yet and no immediate plans for shipping as of 2021.09; no IE)
Example:
server-timing: key;desc="value"
You can use this header multiple times for multiple pieces of data:
server-timing: key1;desc="value1"
server-timing: key2;desc="value2"
or use its compact version where you expose multiple pieces of data in one header, comma-separated.
server-timing: key1;desc="value1", key2;desc="value2"
Example of how Wikipedia uses this header to expose info about cache hit/miss:
Code example (need to account for lack of browser support in Safari and IE):
if (window.performance && performance.getEntriesByType) { // avoid error in Safari 10, IE9- and other old browsers
let navTiming = performance.getEntriesByType('navigation')
if (navTiming.length > 0) { // still not supported as of Safari 14...
let serverTiming = navTiming[0].serverTiming
if (serverTiming && serverTiming.length > 0) {
for (let i=0; i<serverTiming.length; i++) {
console.log(`${serverTiming[i].name} = ${serverTiming[i].description}`)
}
}
}
}
This logs cache = hit-front in supported browsers.
Notes:
as mentioned on MDN, the API is only supported over HTTPS
if your JS is served from another domain, you have to add Timing-Allow-Origin response header to make the data readable to JS (Timing-Allow-Origin: * or Timing-Allow-Origin: https://www.example.com)
Server-Timing headers support also dur(header) field, readable as duration on JS side, but it's optional and defaults to 0 in JS if not passed
regarding Safari support: see bug 1 and bug 2 and bug 3
You can read more on server-timing in this blog post
Note that performance entries buffers might get cleaned by JS on the page (via an API call), or by the browser, if the page issues too many calls for subresources. For that reason, you should capture the data as soon as possible, and/or use PerformanceObserver API instead. See the blog post for details.
For those looking for a way to parse all HTTP headers into an object that can be accessed as a dictionary headers["content-type"], I've created a function parseHttpHeaders:
function parseHttpHeaders(httpHeaders) {
return httpHeaders.split("\n")
.map(x=>x.split(/: */,2))
.filter(x=>x[0])
.reduce((ac, x)=>{ac[x[0]] = x[1];return ac;}, {});
}
var req = new XMLHttpRequest();
req.open('GET', document.location, false);
req.send(null);
var headers = parseHttpHeaders(req.getAllResponseHeaders());
// Now we can do: headers["content-type"]
You can't access the http headers, but some of the information provided in them is available in the DOM. For example, if you want to see the http referer (sic), use document.referrer. There may be others like this for other http headers. Try googling the specific thing you want, like "http referer javascript".
I know this should be obvious, but I kept searching for stuff like "http headers javascript" when all I really wanted was the referer, and didn't get any useful results. I don't know how I didn't realize I could make a more specific query.
Like many people I've been digging the net with no real answer :(
I've nevertheless find out a bypass that could help others. In my case I fully control my web server. In fact it is part of my application (see end reference). It is easy for me to add a script to my http response. I modified my httpd server to inject a small script within every html pages. I only push a extra 'js script' line right after my header construction, that set an existing variable from my document within my browser [I choose location], but any other option is possible. While my server is written in nodejs, I've no doubt that the same technique can be use from PHP or others.
case ".html":
response.setHeader("Content-Type", "text/html");
response.write ("<script>location['GPSD_HTTP_AJAX']=true</script>")
// process the real contend of my page
Now every html pages loaded from my server, have this script executed by the browser at reception. I can then easily check from JavaScript if the variable exist or not. In my usecase I need to know if I should use JSON or JSON-P profile to avoid CORS issue, but the same technique can be used for other purposes [ie: choose in between development/production server, get from server a REST/API key, etc ....]
On the browser you just need to check variable directly from JavaScript as in my example, where I use it to select my Json/JQuery profile
// Select direct Ajax/Json profile if using GpsdTracking/HttpAjax server otherwise use JsonP
var corsbypass = true;
if (location['GPSD_HTTP_AJAX']) corsbypass = false;
if (corsbypass) { // Json & html served from two different web servers
var gpsdApi = "http://localhost:4080/geojson.rest?jsoncallback=?";
} else { // Json & html served from same web server [no ?jsoncallback=]
var gpsdApi = "geojson.rest?";
}
var gpsdRqt =
{key :123456789 // user authentication key
,cmd :'list' // rest command
,group :'all' // group to retreive
,round : true // ask server to round numbers
};
$.getJSON(gpsdApi,gpsdRqt, DevListCB);
For who ever would like to check my code:
https://www.npmjs.org/package/gpsdtracking
Allain Lalonde's link made my day.
Just adding some simple working html code here.
Works with any reasonable browser since ages plus IE9+ and Presto-Opera 12.
<!DOCTYPE html>
<title>(XHR) Show all response headers</title>
<h1>All Response Headers with XHR</h1>
<script>
var X= new XMLHttpRequest();
X.open("HEAD", location);
X.send();
X.onload= function() {
document.body.appendChild(document.createElement("pre")).textContent= X.getAllResponseHeaders();
}
</script>
Note: You get headers of a second request, the result may differ from the initial request.
Another way is the more modern fetch() API
https://developer.mozilla.org/en-US/docs/Web/API/WindowOrWorkerGlobalScope/fetch
Per caniuse.com it's supported by Firefox 40, Chrome 42, Edge 14, Safari 11
Working example code:
<!DOCTYPE html>
<title>fetch() all Response Headers</title>
<h1>All Response Headers with fetch()</h1>
<script>
var x= "";
if(window.fetch)
fetch(location, {method:'HEAD'})
.then(function(r) {
r.headers.forEach(
function(Value, Header) { x= x + Header + "\n" + Value + "\n\n"; }
);
})
.then(function() {
document.body.appendChild(document.createElement("pre")).textContent= x;
});
else
document.write("This does not work in your browser - no support for fetch API");
</script>
If we're talking about Request headers, you can create your own headers when doing XmlHttpRequests.
var request = new XMLHttpRequest();
request.setRequestHeader("X-Requested-With", "XMLHttpRequest");
request.open("GET", path, true);
request.send(null);
To get the headers as an object which is handier (improvement of Raja's answer):
var req = new XMLHttpRequest();
req.open('GET', document.location, false);
req.send(null);
var headers = req.getAllResponseHeaders().toLowerCase();
headers = headers.split(/\n|\r|\r\n/g).reduce(function(a, b) {
if (b.length) {
var [ key, value ] = b.split(': ');
a[key] = value;
}
return a;
}, {});
I've just tested, and this works for me using Chrome Version 28.0.1500.95.
I was needing to download a file and read the file name. The file name is in the header so I did the following:
var xhr = new XMLHttpRequest();
xhr.open('POST', url, true);
xhr.responseType = "blob";
xhr.onreadystatechange = function () {
if (xhr.readyState == 4) {
success(xhr.response); // the function to proccess the response
console.log("++++++ reading headers ++++++++");
var headers = xhr.getAllResponseHeaders();
console.log(headers);
console.log("++++++ reading headers end ++++++++");
}
};
Output:
Date: Fri, 16 Aug 2013 16:21:33 GMT
Content-Disposition: attachment;filename=testFileName.doc
Content-Length: 20
Server: Apache-Coyote/1.1
Content-Type: application/octet-stream
This is my script to get all the response headers:
var url = "< URL >";
var req = new XMLHttpRequest();
req.open('HEAD', url, false);
req.send(null);
var headers = req.getAllResponseHeaders();
//Show alert with response headers.
alert(headers);
Having as a result the response headers.
This is a comparison test using Hurl.it:
Using mootools, you can use this.xhr.getAllResponseHeaders()
This is an old question. Not sure when support became more broad, but getAllResponseHeaders() and getResponseHeader() appear to now be fairly standard: http://www.w3schools.com/xml/dom_http.asp
As has already been mentioned, if you control the server side then it should be possible to send the initial request headers back to the client in the initial response.
In Express, for example, the following works:
app.get('/somepage', (req, res) => {
res.render('somepage.hbs', {headers: req.headers});
})
The headers are then available within the template, so could be hidden visually but included in the markup and read by clientside javascript.
I think the question went in the wrong way,
If you want to take the Request header from JQuery/JavaScript the answer is simply No. The other solutions is create a aspx page or jsp page then we can easily access the request header.
Take all the request in aspx page and put into a session/cookies then you can access the cookies in JavaScript page..

How do I intercept an API call and display data from it using a UserScript?

There's a webapp that makes a request (let's call it /api/item). This request returns a json body with a field called itemData which is normally hidden from the user, but I want to make that shown.
So how do I make a userscript that listens for the request at /api/item and displays the itemData field?
For reference the way the webapp is making the request is:
return Promise.resolve(new Request(e,r)).then(sendCookies).then(addLangParam).then(addCacheParam).then(addXsrfKey).then(checkZeroRating).then(function(e) {
return fetch(e)
}).then(checkStatus).then(checkApiVersionMismatch).then(checkApiResponse)
Most of that is irrelevant, but the important part is Request (I think).
This webapp is not using XMLHttpRequest, but the Fetch API.
You can use the fetch-intercept npm module to intercept fetch requests. Example code:
import fetchIntercept from 'fetch-intercept'
fetchIntercept.register({
response(response) {
console.log(response)
return response
}
})
Do you have access to the promise returned ?
If so, then you may add another "then".
Otherwise, you may overwrite "checkApiResponse"

Categories