Why is my data import to Algolia search using the API script timing out - javascript

I am trying to implement single index searching using Algoliasearch for my iOS mobile app. I have about 110 users on my application. However, when I upload their data to Algolia search's index the function times out before uploading all users. Instead it throws an Error message in the http browser and declares a timeout in the Firestore console.
Firestore console:
sendCollectionToAlgolia
Function execution took 60044 ms, finished with status: 'timeout'
I created the function using this tutorial:
https://medium.com/#soares.rfarias/how-to-set-up-firestore-and-algolia-319fcf2c0d37
Although i have ran into some complications, I highly recommend that tutorial if you have your app using swiftUI iOS platform and implement cloud functions using Typescript.
Heres my function:
import * as functions from 'firebase-functions';
import * as admin from 'firebase-admin';
import algoliasearch from 'algoliasearch';
admin.initializeApp();
const db = admin.firestore();
const algoliaClient = algoliasearch(functions.config().algolia.appid, functions.config().algolia.apikey)
const collectionIndexName = functions.config().projectId === 'PROJECT-XXXX' ? 'prod_SEARCH' : 'dev_SEARCH';
const collectionIndex = algoliaClient.initIndex(collectionIndexName);
//rename to uploadUsersToAlgolia
export const sendCollectionToAlgolia = functions.https.onRequest(async (req, res) => {
const algoliaRecords: any[] = [];
const querySnapshot = await db.collection('users').get();
querySnapshot.docs.forEach(doc => {
const document = doc.data();
const record = {
objectID: doc.id,
fullname: document.fullname,
bio: document.bio,
username: document.username,
uid: document.uid,
profileImageURL: document.profileImageURL,
backgroundImageURL: document.backgroundImageURL,
fcmToken: document.fcmToken,
accountCreated: document.accountCreated,
inspirationCount: document.inspriationCount,
BucketListCount: document.BucketListCount,
CompletedBucketListCount: document.CompletedBucketListCount,
FriendsCount: document.FriendsCount
};
algoliaRecords.push(record);
});
// After all records are created, we save them to
collectionIndex.saveObjects(algoliaRecords, (_error: any, content: any) => {
res.status(200).send("users collection was indexed to Algolia successfully.");
});
});

If you just want to change the default 1 minute timeout, you can do that when you configure the function.
functions.runWith({timeoutSeconds: X}).https.onRequest(async (req, res)
Increasing the timeout won't help if your function doesn't end up sending a response, so you should also add some logging/debugging to figure out if the final call to res.send() is actually happening. If the function never sends a response, it will definitely time out no matter what happens.

Related

Update 'profiles' on Supabase with RLS

I'm currently attempting to use Supabase's JavaScript API to update a row in my 'profiles' database, which has RLS on, via my backend.
This is being done following Stripe sending me a webhook indicating a payment has been successful.
I won't put the full API call in, but here is my Supabase code:
const supabaseUrl = process.env.REACT_APP_SUPABASE_URL
const supabaseAnonKey = process.env.REACT_APP_SUPABASE_ANON_KEY
const supabase = createClient(supabaseUrl, supabaseAnonKey)
module.exports = async (req, res) => {
if (event.type === "checkout.session.completed") {
const userId = String(event.data.object.client_reference_id)
const { error } = await supabase.from('profiles').update({ premium: 'true' }).eq('id', userId)
if (error) {
console.log(error)
}
}
}
However, every time I try to run this, I get a 404 error. This seems to be because I have RLS on.
As a result, I have two questions:
Is it safe for me to turn RLS off?
How can I adjust my code / apply a new database policy to allow this to be accepted?

Hooks.js running the db connection and results twice in sveltekit

I'm using sveltekit and trying to understand all the new features added after retiring Sapper. One of those new features is hooks.js which runs on the server and not accessible to the frontend. It makes dealing with db safe. So I created a connection to my mongodb to retrieve user's data before I use the db results in my getSession function. It works but I noticed that it access my database TWICE. Here is my hooks.js code:
import * as cookie from 'cookie';
import { connectToDatabase } from '$lib/mongodb.js';
export const handle = async ({event, resolve})=>{
const dbConnection = await connectToDatabase();
const db = dbConnection.db;
const userinfo = await db.collection('users').findOne({ username: "a" });
console.log("db user is :" , userinfo) //username : John
const response = await resolve(event)
response.headers.set(
'set-cookie', cookie.serialize("cookiewithjwt", "sticksafterrefresh")
)
return response
}
export const getSession = (event)=>{
return {
user : {
name : "whatever"
}
}
}
The console.log you see here returns the user data twice. One as soon as I fire up my app at localhost:3000 with npm run dev and then less than a second, it prints another console log with the same information
db user is : John
a second later without clicking on anything a second console.log prints
db user is : John
So my understanding from the sveltekit doc is that hooks.js runs every time SvelteKit receives a request. I removed all prerender and prefetch from my code. I made sure I only have the index.svelte in my app but still it prints twice. My connection code I copied from an online post has the following:
/**
* Global is used here to maintain a cached connection across hot reloads
* in development. This prevents connections growing exponentially
* during API Route usage.
*/
Here is my connection code:
import { MongoClient } from 'mongodb';
const mongoURI ="mongodb+srv://xxx:xxx#cluster0.qjeag.mongodb.net/xxxxdb?retryWrites=true&w=majority";
const mongoDB = "xxxxdb"
export const MONGODB_URI = mongoURI;
export const MONGODB_DB = mongoDB;
if (!MONGODB_URI) {
throw new Error('Please define the mongoURI property inside config/default.json');
}
if (!MONGODB_DB) {
throw new Error('Please define the mongoDB property inside config/default.json');
}
/**
* Global is used here to maintain a cached connection across hot reloads
* in development. This prevents connections growing exponentially
* during API Route usage.
*/
let cached = global.mongo;
if (!cached) {
cached = global.mongo = { conn: null, promise: null };
}
export const connectToDatabase = async() => {
if (cached.conn) {
return cached.conn;
}
if (!cached.promise) {
const opts = {
useNewUrlParser: true,
useUnifiedTopology: true
};
cached.promise = MongoClient.connect(MONGODB_URI).then((client) => {
return {
client,
db: client.db(MONGODB_DB)
};
});
}
cached.conn = await cached.promise;
return cached.conn;
So my question is : is hooks.js runs twice all the time, one time on the server and one time on the front? If not, then why the hooks.js running/printing twice the db results in my case?
Anyone?

Firebase Storage Task (Progress bar)

I am trying to do a content upload progress bar using Firebase storage, but I am having some problems returning the task from my function.
I have implemented a Firebase Singleton, using React Context API. In the Firebase component I have multiples functions, one of them called 'uploadContent'
Here is the code:
uploadContent = async (postInfo) => {
const { uri, description, location, tags } = postInfo;
// Post UUID
const postId = uuid();
// Upload to firestore
const data = {
id: postId,
description,
location,
tags,
time: firestore.Timestamp.fromDate(new Date()), // The time when the image is uploaded
likes: [], // At the first time, when a post is created, zero users has liked it
comments: [], // Also, there aren't any comments
};
await this.db
.collection("posts")
.doc(this.auth.currentUser.uid)
.collection("userPosts")
.add(data);
// Create a storage referece
const storageRef = firebase.storage().ref("photos").child(postId);
// Uri to Blob
const response = await fetch(uri);
const blob = await response.blob();
// Upload to storage
const task = storageRef.put(blob);
return task;
};
The thing is, that when I call this function from my uploader component, and try to use one of the returned object functions I get "[Unhandled promise rejection: TypeError: undefined is not a function (near '...task.on...')]", and I don't know how to solve this problem.
Pd: If I call this function inside the "uploadContent" method (where I create the task), it works fine, but I need to return the task...
Here is the code of the function where I call my firebase method:
const upload = async () => {
const { firebase, navigation } = props;
console.log("Uploading...");
// Prepare post information
const postInfo = {
uri: photo.uri,
description: descriptionInput.current.props.value,
location: locationName, // TODO - Object with the location coords too
tags: [], // TODO - users tagged
};
// Upload to firebase
const task = await firebase.uploadContent(postInfo);
task.on("state_changed", (taskSnapshot) => {
console.log(
`${taskSnapshot.bytesTransferred} transferred out of ${taskSnapshot.totalBytes}`
);
});
// navigation.navigate("Profile"); // TODO: route params -> task
};
Any ideas? Thanks.
I wasted so much time on a similar problem, but solved it!
In this part of the code, you are resolving the task (that is implemented with promise) into the value undefined.
// Upload to firebase
const task = await firebase.uploadContent(postInfo);
Just remove the await to use the task itself.

Firestore Cloud Function empty collection

I have a problem that's bugging me for days. I am trying to create a Firebase Cloud function that reads from the Firestore database.
My Firestore DB looks like this:
Problem is that I cannot list users like this:
db.collection('users').get().then((snapshot) => snapshot.forEach(...));
If I try to do this I get empty response, like there are no users in my users collection.
But I try to access user directly it works:
await db.collection('users/5CZxgu8nmNXu2TgplwOUdOIt8e33/receipts').get()
My complete code:
import * as functions from 'firebase-functions';
const admin = require('firebase-admin');
admin.initializeApp(functions.config().firebase);
exports.cat = functions.https.onRequest(async (req, res) => {
const receiptList: any = [];
const db: Firestore = admin.firestore();
const usersRef = await db.collection('users').get();
console.log(usersRef.empty); // Returns true
const receiptsRef = await db
.collection('users/5CZxgu8nmNXu2TgplwOUdOIt8e33/receipts')
.get();
receiptsRef.forEach((receipt: any) => {
console.log(receipt);
receiptList.push(receipt);
// Here I can access data
});
res.send(receiptList);
return '';
});
Does anyone have any idea what I'm doing wrong? Thank you!
Your users collection is actually empty. See how the document IDs are shown in italics? That means there is not actually a document in its place, however, there are subcollections with documents organized underneath them.
When you query a collection, you only get the documents that are immediately within that collection. A query will not pick up documents organized in subcollections. In this respect, queries are said to be "shallow". As you've seen, you need to reach deeper into the subcollection to get its documents.
Bottom line is that the queries you're showing are doing exactly what they're supposed to do.
Thanks again Doug for your help.
I manage to solve my problem. Here is my complete solution.
import * as functions from 'firebase-functions';
import {
Firestore
} from '#google-cloud/firestore';
const admin = require('firebase-admin');
admin.initializeApp(functions.config().firebase);
exports.cat = functions.https.onRequest(async (req, res) => {
const receiptList: any = [];
const db: Firestore = admin.firestore();
const receipts = await db.collectionGroup('receipts').get();
receipts.forEach((doc: any) => {
console.log(doc.id, ' => ', doc.data());
receiptList.push(doc.data());
});
res.send(receiptList);
return '';
});
.get() gets all documents. In your case those documents are empty therefore .get() doesn't consider them.
The simplest solution that I found for this is to replace .get() with .listDocuments(). Now you could read each doc entry like you would a doc.

Is NightmareJS (electron browser) compatible Firebase Functions?

I have an app that takes a New York Times recipe URL, and converts the list of ingredients into a shopping to-do list.
Because the New York Times uses React, none of the data is available via standard scraping - the index.html is mostly blank. I have to use a library like NightmareJS, which uses an Electron browser to fully construct the DOM (including the Javascript) so that I can then scrape that constructed-DOM for data.
But this doesn't seem to work. Here's the code I have included in my /functions/index.js file:
// The Cloud Functions for Firebase SDK to create Cloud Functions and setup triggers.
const functions = require('firebase-functions')
// The Firebase Admin SDK to access the Firebase Realtime Database.
const admin = require('firebase-admin')
admin.initializeApp(functions.config().firebase)
const Nightmare = require('nightmare')
const Actions = require('nightmare-react-utils').Actions
exports.listify = functions.https.onRequest((req, res) => {
console.log("YOU ARE NOW INSIDE THE LISTIFY FUNCTION!")
Nightmare.action(...Actions)
const nightmare = new Nightmare({ show: false })
const selector = 'ul.recipe-ingredients'
const queryUrl = req.query.url
nightmare
.goto(queryUrl)
.wait()
.evaluate((selector) => {
console.log("YOU ARE NOW INSIDE THE EVALUATE!")
const recipeIngredientsObject = document.querySelector(selector).children
const result = []
const ingredientKeys = Object.keys(recipeIngredientsObject)
ingredientKeys.forEach((key) => {
const ingredientObject = recipeIngredientsObject[key]
const quantityAndIngredient = ingredientObject.children
result.push({
"quantity": quantityAndIngredient[0].innerText,
"ingredient": quantityAndIngredient[1].innerText
})
})
return result
}, selector)
})
When I call this Function from my front-end, I see the first console log in my Firebase logs - "YOU ARE NOW INSIDE THE LISTIFY FUNCTION!" - but I do not see the second message: "YOU ARE NOW INSIDE THE EVALUATE!"
Can I not use NightmareJS with Firebase Functions?
The console.log message will never appear. When you run evaluate, that function is executed inside the context of the headless browser, so will not log to terminal.
Try something like...
.evaluate((selector) => {
return document.querySelector(selector)
}, selector)
.end()
.then(console.log)
To see if it's working at all.

Categories