We Keep Coding

Xero webhooks with Node Red; OK, than not ok

I have been dealing with cryptography craziness since yesterday, I 've literally lost my sleep over this.
I am implementing a node red solution to get webhooks from Xero to be written in a custom app. I have experienced a lot of issues with the payload and how it needs to be stringified and how needs to be hashed, but eventually figured it out thanks to a Github fellow that posted this code to get the body to its 'raw' state
let msgPayloadRaw = JSON.stringify(msg.payload).split(':').join(': ').split(': [').join(':[').split(',"entropy"').join(', "entropy"');
I then create a sha256 base64 hash to check against the header value using the following js code
var cryptojs = context.global.cryptojs;
const webhookKey = 'MyWebhookKeyHere';
let msgPayloadRaw = JSON.stringify(msg.payload).split(':').join(': ').split(': [').join(':[').split(',"entropy"').join(', "entropy"');
let bdata = new Buffer(msgPayloadRaw).toString();
let ciphertext = cryptojs.HmacSHA256(bdata, webhookKey );
let base64encoded = cryptojs.enc.Base64.stringify(ciphertext);
msg.payload = base64encoded;
return msg;
Now everything should work great, but I get a crazy result showcased in this recording, where the web hooks intent status turns to 'OK', and some seconds later returns to this error:
Retry
We haven’t received a successful response on the most recent delivery attempt and will retry sending with decreasing frequency for an overall period of 24 hours.
Response not 200. Learn more
Last sent at 2022-06-22 11:48:28 UTC
What's the problem ?
The problem relies in the http input, where node red parsed the body.
Body needs to be hashed. The body in the http request is like that
{"events":[],"firstEventSequence": 0,"lastEventSequence": 0, "entropy": "IVMMHNWPBAZYRZJRCUAQ"}
Notice the spaces after each :
Node Red converts that body to JSON object. When I do JSON.stringify(msg.payload); I will get the following
{"events":[],"firstEventSequence":0,"lastEventSequence":0, "entropy":"IVMMHNWPBAZYRZJRCUAQ"}
which is obviously the same, but technically it is not (due to spaces) and when hashed it generates a different hash value.
The GitHub fellow did that that walkround
JSON.stringify(msg.payload).split(':').join(': ').split(': [').join(':[').split(',"entropy"').join(', "entropy"');
So in order to solve this, I need to find a way to get the raw http input, instead of the parsed one that node red is providing.
Any ideas how to get the raw input ?

Sort answer: you don't.
The raw body is not available if the Content-Type header is set to application/json the bodyParser will kick in and generate the matching JSON object that is passed as the msg.payload.
the httpNodeMiddleware is attached after the bodyParser so the body has already been changed.

How to get fixed amount of documents from mongodb?

Now my code is returning all the news related to a company. But I want only the first 15 elements. How to do this? Here is the code which returns all the news for a company. google-news-json is a npm package.
export default async function handler(req, res) {
try {
let news = await googleNewsAPI.getNews(googleNewsAPI.SEARCH, req.body.companyName, 'en-US')
res.status(200).json(news)
} catch (err) {
res.status(500).json({ error: 'Failed to fetch news' })
}
}

You need to take a close look at the API reference of google news functionality.
There you can find a query parameter ( something like ?q=15 or ?count=no.of.items ) after your API key to return the count of object that you may want to receive in the fetch url

Typically you'd want to do what #iceweasel suggests in their answer. However, since this package isn't supported by Google (but rather seems to be scraping the HTML produced by directly querying Google News and returning the results as JSON), it doesn't support many of the typical options you'd expect (in particular, limiting the number of returned elements).
If you don't mind the network cost of downloading the full 100 items (which is what your function returns by default), you take the first 15 results from the API's response with:
let top15 = news.items.slice(0, 15)

Cardano-wallet malformed tx payload from Submit External Transaction

I’m trying to submit an already signed tx from cardano-cli using the cardano-wallet endpoint: https://localhost:8090/v2/proxy/transactions
the signed transaction look like this:
txBody = {
"type": "Tx MaryEra",
"description": "",
"cborHex": "83a400818258202d7928a59fcba5bf71c40fe6428a301ffda4d2fa681e5357051970436462b89400018282583900c0e88694ab569f42453eb950fb4ec14cb50f4d5d26ac83fdec2c505d818bcebf1df51c84239805b8a330d68fdbc3c047c12bb4c3172cb9391a002b335f825839003d2d9ceb1a47bc1b62b7498ca496b16a7b4bbcc6d97ede81ba8621ebd6d947875fcf4845ef3a5f08dd5522581cf6de7b9c065379cbb3754d1a001e8480021a00029361031a01672b7ea1008182582073dd733cc50d594cb89d3ac67287b02dae00982fc800e9c9c9d1bb282b56122558404d0cb4e4f1cc415ddcf546871f075d0ca6e0c2620cd784b06c21c9b86e4403cb7a115038487576dcb20e7820e9d0dc93ab2a737ed9d0a71a77bc1e12f7c4dd0ef6"
}
I just don’t know how to pass it to the endpoint using Content-Type application/octet-stream. The API doc says the payload should be:
string <binary>
Signed transaction message binary blob.
I’m using javascript for this and have tried passing the cborHex directly, using Buffer.from(txBody.cborHex).toString('base64') and the whole json Buffer.from(JSON.stringify(txBody)).toString('base64') but always got the same response:
{
"code": "malformed_tx_payload",
"message": "I couldn't verify that the payload has the c…node. Please check the format and try again."
}
Also I’ve noticed from the swagger specification that the endpoint support a JSON payload and taking a look to the cardano-wallet's source code here:
newtype PostExternalTransactionData = PostExternalTransactionData
{ payload :: ByteString
} deriving (Eq, Generic, Show)
I thought the structure should be some like this:
{
"payload": ?// some binary blob here that I can't find. I've tried with:
// Buffer.from(txBody.cborHex).toString('base64') and
// Buffer.from(JSON.stringify(txBody)).toString('base64')
}
Any idea how to construct the payload and pass the signed tx?

This code tells me that when decoding external (sealed) transaction the wallet tries Base 16 encoding first and if that fails then it tries Base 64:
instance FromJSON (ApiT SealedTx) where
parseJSON v = do
tx <- parseSealedTxBytes #'Base16 v <|> parseSealedTxBytes #'Base64 v
pure $ ApiT tx
After that BytesString is passed to this function https://github.com/input-output-hk/cardano-node/blob/5faa1d2bb85ae806ec51fa4c576dec2670c67c7a/cardano-api/src/Cardano/Api/SerialiseCBOR.hs#L32 together with the currentNodeEra that node is running.
(Each era has a different way of decoding)
It could be (I am not sure) that node is running say Alonzo but you're submitting a Mary-encoded Tx. In which case the decoding might fail.
I hope this helps.

Reply to an email in Gmail with AppScript with changed recipients ends up in a new thread

I have an email in my mailbox and I want the AppScript program to reply to it with just me and a special google group as the recipients. The purpose of this is communication of the program with me as the program replies to the message once it has processed it with necessary details about the processing in the reply body. There might also be other recipients apart from me in the original message and I don't want the program to send the reply to them.
So I need to reply with a changed set of recipients. When I do it in the Gmail GUI it works just fine, I hit reply, change the recipients, send the message and the reply ends up in the original thread. However when I do it in the script the reply always ends up in a new thread. Originally I thought Gmail decides based on the subject of the email but it seems there's more to it (perhaps it has recently changed as I think it used to work that way).
I tried multitude of slightly different approached, one of them being:
var messageBody = "foo";
var newRecipients = "me#gmail.com, my-group#gmail.com";
var messageToReplyTo = ...;
var advancedParams = {from : "my-alias#gmail.com"};
var replyDraft = messageToReplyTo.createDraftReply(messageBody);
var replySubject = replyDraft.getMessage().getSubject();
var replyBody = replyDraft.getMessage().getBody();
replyDraft.update(newRecipients, replySubject, replyBody, advancedParams);
replyDraft.send();

There are a couple fun things you need to do in order to achieve this, but you can do it without too much trouble. You should definitely review the guide to Drafts.
Per the API spec:
In order to be part of a thread, a message or draft must meet the following criteria:
The requested threadId must be specified on the Message or Draft.Message you supply with your request.
The References and In-Reply-To headers must be set in compliance with the RFC 2822 standard.
The Subject headers must match.
To start, you need to get a reference to the draft you want to update. This is probably simplest by using GmailApp:
const thread = /** get the thread somehow */;
const newBody = /** your plaintext here */;
const reply = thread.createDraftReply(newBody);
The primary issue with Gmail & Drafts is that a Draft is an immutable message to server resources. If you change any of it, you change all of it. Thus, to change a header value such as the recipient address, you need to completely rebuild the message. This is why using the GmailApp methods to update a draft fail to maintain the existing thread information - you can't specify it as one of the advanced options for building the new message. Thus, you must use the Gmail REST API for this task:
const rawMsg = Gmail.Users.Drafts.get("me", reply.getId(), {format: "raw"}).message;
To update a draft, you need to supply an RFC 2822 formatted message encoded in base64. If you are comfortable converting the rich format message parts into such a valid string, by all means work with the non-raw format, as you have direct access to the headers in the message.payload.
To work with the raw message, know that Apps Script casts the described base64 encoded string to a byte array in the above call. The leap is then to treat that byte array as string bytes, specifically, charCodes:
const msg_string = rawMsg.raw.reduce(function (acc, b) { return acc + String.fromCharCode(b); }, "");
console.log({message: "Converted byte[] to str", bytes: rawMsg.raw, str: msg_string});
Once you have the message as a string, you can use regular expressions to update your desired headers:
const pattern = /^To: .+$/m;
var new_msg_string = msg_string.replace(pattern, "To: <....>");
// new_msg_string += ....
Since the Gmail API endpoint to update a Draft expects a base64 web-safe encoded string, you can compute that:
const encoded_msg = Utilities.base64EncodeWebSafe(new_msg_string);
And the only remaining bit is to perform the call (and/or send the updated draft).
const resource = {
id: <draft id>, // e.g. reply.getId()
message: {
threadId: <thread id>, // e.g. thread.getId()
raw: encoded_msg
}
}
const resp = Gmail.Users.Drafts.update(resource, "me", reply.getId());
const sent_msg = Gmail.Users.Drafts.send({id: resp.id}, "me");
console.log({message: "Sent the draft", msg: sent_msg});
I don't claim that the handling of the Byte array returned from the Message.raw property is 100% correct, only that it seems correct and didn't result in any errors in the test message I sent. There may also be an easier approach, as the Apps Script service has a Drafts.update endpoint which accepts a Blob input and I have not investigated how one would use that.

Struggling to build a JS/PHP validation function for my app

I have a web service that returns a JSON object when the web service is queried and a match is found, an example of a successful return is below:
{"terms":[{"term":{"termName":"Focus Puller","definition":"A focus puller or 1st assistant camera..."}}]}
If the query does not produce a match it returns:
Errant query: SELECT termName, definition FROM terms WHERE termID = xxx
Now, when I access this through my Win 8 Metro app I parson the JSON notation object using the following code to get a JS object:
var searchTerm = JSON.parse(Result.responseText)
I then have code that processes searchTerm and binds the returned values to the app page control. If I enter in a successful query that finds match in the DB everything works great.
What I can't work out is a way of validating a bad query. I want to test the value that is returned by var searchTerm = JSON.parse(Result.responseText) and continue doing what I'm doing now if it is a successful result, but then handle the result differently on failure. What check should I make to test this? I am happy to implement additional validation either in my app or in the web service, any advice is appreciated.
Thanks!

There are a couple of different ways to approach this.
One approach would be to utilize the HTTP response headers to relay information about the query (i.e. HTTP 200 status for a found record, 404 for a record that is not found, 400 for a bad request, etc.). You could then inspect the response code to determine what you need to do. The pro of this approach is that this would not require any change to the response message format. The con might be that you then have to modify the headers being returned. This is more typical of the approach used with true RESTful services.
Another approach might be to return success/error messaging as part of the structured JSON response. Such that your JSON might look like:
{
"result":"found",
"message":
{
"terms":[{"term":{"termName":"Focus Puller","definition":"A focus puller or 1st assistant camera..."}}]}
}
}
You could obviously change the value of result in the data to return an error and place the error message in message.
The pros here is that you don't have to worry about header modification, and that your returned data would always be parse-able via JSON.parse(). The con is that now you have extra verbosity in your response messaging.