I have an app that takes a New York Times recipe URL, and converts the list of ingredients into a shopping to-do list.
Because the New York Times uses React, none of the data is available via standard scraping - the index.html is mostly blank. I have to use a library like NightmareJS, which uses an Electron browser to fully construct the DOM (including the Javascript) so that I can then scrape that constructed-DOM for data.
But this doesn't seem to work. Here's the code I have included in my /functions/index.js file:
// The Cloud Functions for Firebase SDK to create Cloud Functions and setup triggers.
const functions = require('firebase-functions')
// The Firebase Admin SDK to access the Firebase Realtime Database.
const admin = require('firebase-admin')
admin.initializeApp(functions.config().firebase)
const Nightmare = require('nightmare')
const Actions = require('nightmare-react-utils').Actions
exports.listify = functions.https.onRequest((req, res) => {
console.log("YOU ARE NOW INSIDE THE LISTIFY FUNCTION!")
Nightmare.action(...Actions)
const nightmare = new Nightmare({ show: false })
const selector = 'ul.recipe-ingredients'
const queryUrl = req.query.url
nightmare
.goto(queryUrl)
.wait()
.evaluate((selector) => {
console.log("YOU ARE NOW INSIDE THE EVALUATE!")
const recipeIngredientsObject = document.querySelector(selector).children
const result = []
const ingredientKeys = Object.keys(recipeIngredientsObject)
ingredientKeys.forEach((key) => {
const ingredientObject = recipeIngredientsObject[key]
const quantityAndIngredient = ingredientObject.children
result.push({
"quantity": quantityAndIngredient[0].innerText,
"ingredient": quantityAndIngredient[1].innerText
})
})
return result
}, selector)
})
When I call this Function from my front-end, I see the first console log in my Firebase logs - "YOU ARE NOW INSIDE THE LISTIFY FUNCTION!" - but I do not see the second message: "YOU ARE NOW INSIDE THE EVALUATE!"
Can I not use NightmareJS with Firebase Functions?
The console.log message will never appear. When you run evaluate, that function is executed inside the context of the headless browser, so will not log to terminal.
Try something like...
.evaluate((selector) => {
return document.querySelector(selector)
}, selector)
.end()
.then(console.log)
To see if it's working at all.
Related
I'm using sveltekit and trying to understand all the new features added after retiring Sapper. One of those new features is hooks.js which runs on the server and not accessible to the frontend. It makes dealing with db safe. So I created a connection to my mongodb to retrieve user's data before I use the db results in my getSession function. It works but I noticed that it access my database TWICE. Here is my hooks.js code:
import * as cookie from 'cookie';
import { connectToDatabase } from '$lib/mongodb.js';
export const handle = async ({event, resolve})=>{
const dbConnection = await connectToDatabase();
const db = dbConnection.db;
const userinfo = await db.collection('users').findOne({ username: "a" });
console.log("db user is :" , userinfo) //username : John
const response = await resolve(event)
response.headers.set(
'set-cookie', cookie.serialize("cookiewithjwt", "sticksafterrefresh")
)
return response
}
export const getSession = (event)=>{
return {
user : {
name : "whatever"
}
}
}
The console.log you see here returns the user data twice. One as soon as I fire up my app at localhost:3000 with npm run dev and then less than a second, it prints another console log with the same information
db user is : John
a second later without clicking on anything a second console.log prints
db user is : John
So my understanding from the sveltekit doc is that hooks.js runs every time SvelteKit receives a request. I removed all prerender and prefetch from my code. I made sure I only have the index.svelte in my app but still it prints twice. My connection code I copied from an online post has the following:
/**
* Global is used here to maintain a cached connection across hot reloads
* in development. This prevents connections growing exponentially
* during API Route usage.
*/
Here is my connection code:
import { MongoClient } from 'mongodb';
const mongoURI ="mongodb+srv://xxx:xxx#cluster0.qjeag.mongodb.net/xxxxdb?retryWrites=true&w=majority";
const mongoDB = "xxxxdb"
export const MONGODB_URI = mongoURI;
export const MONGODB_DB = mongoDB;
if (!MONGODB_URI) {
throw new Error('Please define the mongoURI property inside config/default.json');
}
if (!MONGODB_DB) {
throw new Error('Please define the mongoDB property inside config/default.json');
}
/**
* Global is used here to maintain a cached connection across hot reloads
* in development. This prevents connections growing exponentially
* during API Route usage.
*/
let cached = global.mongo;
if (!cached) {
cached = global.mongo = { conn: null, promise: null };
}
export const connectToDatabase = async() => {
if (cached.conn) {
return cached.conn;
}
if (!cached.promise) {
const opts = {
useNewUrlParser: true,
useUnifiedTopology: true
};
cached.promise = MongoClient.connect(MONGODB_URI).then((client) => {
return {
client,
db: client.db(MONGODB_DB)
};
});
}
cached.conn = await cached.promise;
return cached.conn;
So my question is : is hooks.js runs twice all the time, one time on the server and one time on the front? If not, then why the hooks.js running/printing twice the db results in my case?
Anyone?
I am trying to implement single index searching using Algoliasearch for my iOS mobile app. I have about 110 users on my application. However, when I upload their data to Algolia search's index the function times out before uploading all users. Instead it throws an Error message in the http browser and declares a timeout in the Firestore console.
Firestore console:
sendCollectionToAlgolia
Function execution took 60044 ms, finished with status: 'timeout'
I created the function using this tutorial:
https://medium.com/#soares.rfarias/how-to-set-up-firestore-and-algolia-319fcf2c0d37
Although i have ran into some complications, I highly recommend that tutorial if you have your app using swiftUI iOS platform and implement cloud functions using Typescript.
Heres my function:
import * as functions from 'firebase-functions';
import * as admin from 'firebase-admin';
import algoliasearch from 'algoliasearch';
admin.initializeApp();
const db = admin.firestore();
const algoliaClient = algoliasearch(functions.config().algolia.appid, functions.config().algolia.apikey)
const collectionIndexName = functions.config().projectId === 'PROJECT-XXXX' ? 'prod_SEARCH' : 'dev_SEARCH';
const collectionIndex = algoliaClient.initIndex(collectionIndexName);
//rename to uploadUsersToAlgolia
export const sendCollectionToAlgolia = functions.https.onRequest(async (req, res) => {
const algoliaRecords: any[] = [];
const querySnapshot = await db.collection('users').get();
querySnapshot.docs.forEach(doc => {
const document = doc.data();
const record = {
objectID: doc.id,
fullname: document.fullname,
bio: document.bio,
username: document.username,
uid: document.uid,
profileImageURL: document.profileImageURL,
backgroundImageURL: document.backgroundImageURL,
fcmToken: document.fcmToken,
accountCreated: document.accountCreated,
inspirationCount: document.inspriationCount,
BucketListCount: document.BucketListCount,
CompletedBucketListCount: document.CompletedBucketListCount,
FriendsCount: document.FriendsCount
};
algoliaRecords.push(record);
});
// After all records are created, we save them to
collectionIndex.saveObjects(algoliaRecords, (_error: any, content: any) => {
res.status(200).send("users collection was indexed to Algolia successfully.");
});
});
If you just want to change the default 1 minute timeout, you can do that when you configure the function.
functions.runWith({timeoutSeconds: X}).https.onRequest(async (req, res)
Increasing the timeout won't help if your function doesn't end up sending a response, so you should also add some logging/debugging to figure out if the final call to res.send() is actually happening. If the function never sends a response, it will definitely time out no matter what happens.
I am trying to do a content upload progress bar using Firebase storage, but I am having some problems returning the task from my function.
I have implemented a Firebase Singleton, using React Context API. In the Firebase component I have multiples functions, one of them called 'uploadContent'
Here is the code:
uploadContent = async (postInfo) => {
const { uri, description, location, tags } = postInfo;
// Post UUID
const postId = uuid();
// Upload to firestore
const data = {
id: postId,
description,
location,
tags,
time: firestore.Timestamp.fromDate(new Date()), // The time when the image is uploaded
likes: [], // At the first time, when a post is created, zero users has liked it
comments: [], // Also, there aren't any comments
};
await this.db
.collection("posts")
.doc(this.auth.currentUser.uid)
.collection("userPosts")
.add(data);
// Create a storage referece
const storageRef = firebase.storage().ref("photos").child(postId);
// Uri to Blob
const response = await fetch(uri);
const blob = await response.blob();
// Upload to storage
const task = storageRef.put(blob);
return task;
};
The thing is, that when I call this function from my uploader component, and try to use one of the returned object functions I get "[Unhandled promise rejection: TypeError: undefined is not a function (near '...task.on...')]", and I don't know how to solve this problem.
Pd: If I call this function inside the "uploadContent" method (where I create the task), it works fine, but I need to return the task...
Here is the code of the function where I call my firebase method:
const upload = async () => {
const { firebase, navigation } = props;
console.log("Uploading...");
// Prepare post information
const postInfo = {
uri: photo.uri,
description: descriptionInput.current.props.value,
location: locationName, // TODO - Object with the location coords too
tags: [], // TODO - users tagged
};
// Upload to firebase
const task = await firebase.uploadContent(postInfo);
task.on("state_changed", (taskSnapshot) => {
console.log(
`${taskSnapshot.bytesTransferred} transferred out of ${taskSnapshot.totalBytes}`
);
});
// navigation.navigate("Profile"); // TODO: route params -> task
};
Any ideas? Thanks.
I wasted so much time on a similar problem, but solved it!
In this part of the code, you are resolving the task (that is implemented with promise) into the value undefined.
// Upload to firebase
const task = await firebase.uploadContent(postInfo);
Just remove the await to use the task itself.
I am trying to write a Firebase Cloud function which would write the current time inside the database whenever called:
const admin = require('firebase-admin');
const functions = require('firebase-functions');
admin.initializeApp(functions.config().firebase);
exports.pushDateOfCall = functions.https.onRequest((req, res) => {
const currentTime = new Date();
return admin.database().ref('/dates').push({currentTime: currentTime}).then((snapshot) => {
return res.send("Complete");
}).catch((error) => res.send("Something went wrong"));
});
After deploying the function and calling it from the function's URL, nothing is written inside the database.
Output of firebase functions logs:
Function execution took 1358 ms, finished with status code: 304
P.S. I am running the link from incognito since I wish that whoever calls the link (both authorised and unauthorised) is able to use it.
const currentTime = new Date();
Here currentTime is an object. If you want to store the String of the date, use String(currentTime) as
return admin.database().ref('/dates').push({currentTime: String(currentTime)})
#hkchakladar is right, changing to {currentTime: String(currentTime)} will solve the problem.
However, note that you don't need to return res.send() nor to return the promise returned by the asynchronous push() method. This is shown in the official Firebase video about HTTP Cloud Function, see https://www.youtube.com/watch?v=7IkUgCLr5oA
So your code may be as follows:
exports.pushDateOfCall = functions.https.onRequest((req, res) => {
const currentTime = new Date();
admin
.database()
.ref('dates')
.push({ currentTime: String(currentTime) })
.then(ref => {
res.send('Complete');
})
.catch(error => res.status(500).send('Something went wrong'));
});
I am building an Amazon Alexa skill that gets data async from Google's Firebase.
When I run the Lambda function locally and call it from my Alexa skill all intents work as expected.
However when I zip up the files (not the folder) and move it to AWS lambda the function times out even thought the data has been received and the response object created as expected.
The built in intents are all working as expected too
My code is on GitHub here
The error log and console.log outputs
I have tried to find any solutions via here and google but no luck. It could be that I have been searching the wrong things or this is a specific problem
Resource
When using firebase with lambda it would appear that you need to initalize and then delete the instance for the response to be returned.
this is a code snippit that I got to work
const Alexa = require("ask-sdk");
const firebase = require("firebase");
var config = {
...
};
const GetOrderIntent = {
canHandle(handlerInput) {
const request = handlerInput.requestEnvelope.request;
return (
request.type === "IntentRequest" &&
request.intent.name === "GetOrderIntent"
);
},
async handle(handlerInput) {
firebase.initializeApp(config);
try {
const store = await firebase
.database()
.ref(`teams/${team}`)
.once("value");
// ANY OTHER CODE HERE
} catch (error) {
// HANDLE ERROR
}
// CLOSE THE CONNECTION
await firebase.app("[DEFAULT]").delete();
return handlerInput.responseBuilder
.speak(speechOutput)
.withSimpleCard(SKILL_NAME, speechOutput)
.getResponse();
}
};