Goal:
To develop an script that will check the last time my external pop accounts were checked by google -- while not being logged in. If the time exceeds some amount, then check the pop account.
My Reason:
I use an offline client. I don't want to be logged into gmail and I want all my external emails to flow thru gmail. Sometimes an important email comes in and I have to log into gmail, go to the account section, and then click "check email". This is incredibly annoying. I wish they had the ability to poll for pop account at a specified frequency. Instead they use an algorithm that can range from 1 minute to 1 hour.
My Approaches so far:
So I can log into gmail using curl. I can scrape the pages. The problem is that google uses javascript/ajax goodness so curl does gets the html version of gmail and that version does not have the info that I am looking for. It's only available on the ajax version of gmail.
I can use selenium, but essentially I have to have firefox open. I don't want that. I want a solution that can run in the background that will check every 10 minutes.
My suspicions on how to go about this:
I've seen several posts about using headless browsers with javascript capabilities. Apparently some of these can be controlled using python. However, this seems quite complicated.
Thus, my questions
What is the best way to solve my problem? My preference is to use python, but I am open to other languages as well. Will I have to use javascript to accomplish this task? Is a headless browser necessary or are there other alternatives?
Thank you.
Probably http://www.phantomjs.org/ is going to be the best tool for this job. They have lots of examples in their github repository for how to do this type of thing. People have had good success with complex scraping tasks.
Related
The title pretty much sums it up.
I'm sorry for asking this question here but i have been going around and around in circles for about 2h now trying to figure out if it is possible to access a calendar's events (simple get, read-only) without the user having to login to a google account?
I am trying to display the events of one calendar to use as an internal thing. But i want this to work without necessarily asking the user to log-in manually.
I have a service account set-up, but for the life of me cannot figure out an example in javascript where this works. And i'm wondering if it is even possible in JS. I know it is possible in java and php. But no examples using javascript seem to exist.
I also saw a few similar cases on stackOverflow, but those cases were looking for refresh tokens. I would like to abstract all authentication from the user.
So, is this possible in javascript?
(Angular.js client-side/Node.js server-side)
Edit for TL;DR
I want to fetch and display events from a google calendar from a node.js server without ANY user intervention. Is this possible?
HAHA! I found something that works.
Install it, create google service-account, download the .p12, convert, enter the info and you're good. Detailed on his github. Calendar stays "private" (only shared to the service account), no user intervention is necessary and it fetches the data fine
(Mostly... Having a problem deploying my js code on a different machine but it works fine on the Windows I used for testing and it's probably a mistake on my part.)
Is there a way to know the total time one spends on a page opened in a browser using a python script? For example if one uses gmail, and is currently using it (i.e the page is non idle) can we know the total time for which the page was active?
Further explaining what I intend to do:
By active means I am actually using the page, be it reading it,
doing some typing or doing some mouse work.
It would be great if there would somehow be a way to exclude the
time spent in breaks, perhaps introduce some error?
I am not sure whether javascript would be apt for it, and am open to suggestions!
Short answer: I don't think so.
This isn't really an issue of whether Python can do it, it's to do with whether your browser exposes that information in a way that an external program/script can query.
There's also the issue of how you define/determine which page is "active". Is it sufficient that the browser window is currently the active window and the page is on the selected tab? Or would you expect some kind of interaction? What if I'm reading a long text and so am not making interacting with the page for a period of time, does that still count as active?
The fact that it's hard to detect activity even from the website point of view, doing so from a 3rd party application/script be it written in Python or any other language would be even trickier.
If you wish to explore this further, I'd say your best bet would be to write a browser extension/plugin. In fact, there may already be existing ones that may meet your needs.
That sounds highly browser and platform-specific.
So, in general "no".
If the browser has some kind of interface or "hook" support then it might be possible.
This isn't really Python specific.
The best I can come up with is leave something like "Google Analytics" to do all that sort of stuff for you, then using your choice of language, get the data you need from the API
Of course that might not be appropriate, and the rules for how long it deems the page viewed, etc... may not work, but at least it'll be consistent.
Currently developing an extension for Gmail that inserts a couple of buttons, using jquery.
I'm relying on the canvas_frameiframe and a couple of more div referrenced with their class names, not by their ids.
How reliable is this? How frequent does Gmail gets their class names changed?
As a side note, I thought of an small javascript file to be pulled from a server constantly, that refreshes class names in case Gmail changes them. I don't like this approach (to change the code) since, this is a security extension, and this will probably request more permissions to the user.
As this is pretty much a hack, there is no guarantee when Google might change the underlying HTML structure.
This reminds me of Remember The Milk for Gmail as Browser Add-on. They did the add-on not via Gmail API but by pretty much hacking the Gmail HTML. It breaks really frequently, due to Google "upgrading" Gmail.
To give you an idea how often Gmail is upgraded, take a look at the list of Gmail versions that RTM team is keeping track of. Gmail is basically updated every few days!
Kudos to the RTM engineers though, they usually fix their add-on to support the latest Gmail relatively quickly whenever it breaks.
That being said, in short, there is no telling how fast <div id="canvas_frame"> might change, but at least now you have an idea how fast Gmail is changing.
Question: What precautions should I take when I let clients add custom JS scripts to their pages?
IF you want more details:
I am working on a custom CMS like project for a company, The CMS has number of "groups" that each subscriber "owns" where they do their own thing.
The new requirements is that some groups want to add google analytics to see how they are doing. So I naturally added a column in the table and made code adjustements so if there is some data in that column, I just use the following line in master page to set the script out:
ScriptManager.RegisterClientScriptBlock(Page, typeof(Page), "CustomJs", CustomJs, true);
It works just fine, only, It got me thinking...
It's really easy for someone with good knowledge of how to access cookies etc from from js. Sure, each group is moderated and only super admin can add this javascript, sure, they wouldn't be silly enough to hack their own group. Each group has their own code so its not possible to hack other groups BUT STILL
I am not really comfortable in letting user's add their own javascript codes.
I could monitor each group myself, but the groups are growing really quick and I will hit a time when I will no longer be able to do that.
So, to brief it up: What precautions should I take to avoid any mishaps ?
ps: did try to google, no convincing answers anywhere.
Instead of allowing the users to add their own Javascript files, and given that the only requirement here is for google analytics, why not just let them put their analytics ID into the CMS and if it's present, output the relevant Google Analytics code?
This way you fulfill the users requirement and also avoid the need to protect against malicious scripting.
Letting users use Javascript is in general, a very bad idea. Don't do it unless you have to.
I once I had a problem where I need to let clients use Javascript, but, the clients weren't necessarily trusted, so, I modified cofeescript so that only a small subset was compilable to javascript, and it worked pretty well. This may be waaaay too overkill for you.
You should not let your users access cookies, that's always a pain. Also, no localStorage or webSQL if you're one of the HTML5 people, and, no document.write() because that's another form of eval as JSLint tells you.
And, the problem with letting people have javascript is that even if you believe you have trusted users, someone may get a password, and you don't want that person to get access to all the other accounts in the group.
Automatically recognizing whether some JavaScript code is malicious or sandboxing it is close to impossible. If you don't want to allow hacking your site you are left with only few options:
Don't allow users to add JavaScript at all.
Only allow predefined JavaScript code, e.g. for Google Analytics.
Have all custom JavaScript inspected by a human before it is allowed to display on the site. Never trust scripts loaded from third party sites - these can change from one day to another and turn malicious.
If you have no other choice, you may consider separating path/domain of user javascripts (and cookies).
For example your user have page:
user1.server.com
and you keep user pages at
user1.server.com
So, if you set session cookies to the user1.server.com, it'll render them unobtainable for user scripts from other domains (e.g. user2.server.com).
Another option may be executing all user's javascript at server JS engine (thus controlling all it's I/O and limiting access to browser resources).
There is no simple and easy solution anyway, so better consider using options from other answers (e.g. predifined script API, human inspection).
I am looking for a tool that lets you monitor/log page rendering time on client machines. I am not looking for firebug/yslow because i want to know the following type of things:
How does fast do my pages load when the user is in russia?
How long does it take for javascript to run on some pages for everyone who accesses those pages?
So, i actually care what my site feels like to the people who use it. Do there exist tools that already do this?
I should add that my website is a software as a service website, not accessible publicly.
I've never heard of any way to do this. One solution, which may be terrible, might be to log the time yourself. At the top of your page have an inline script tag with a global variable called start that creates a new date. Then, have an onload listener that calls a function once the page is finished loading. In this function, get the difference between the start time and current time and send that back to your server. This is by no means that accurate, but might give you some idea. You could also log their IP address for geolocation when you send back the data.
I recommend https://www.atatus.com/. Atatus helps you to visualise page load time across pages, browsers and countries. It also has AJAX monitoring and Transaction monitoring.
There is not really a super easy way to do this effectively. but you can definitely fake the geo-location thing by using a proxy (which would actually give you N*2, time length) and get a pretty good idea at what it's like to browse your site.
As for JavaScript, you can profile it with the profiler in FireBug, this will give you an idea of what functions you should refactor and whatnot.
In my opinion I'd determine what most of your users are using or what the general demographic makeup they are, are they 75 year-old guys? If that is the case maybe they aren't up on the newer faster browsers, or for that matter don't care. If they are cool hipster designers in San Francisco, they its Safari 4.0... anyway this is just a way to determine the meat of the users, I think the best way is just grab an older laptop with Windows XP on it and just browse your site, you can use FireBug lite on browsers besides Firefox
I like to run Dynatrace AJAX edition from UI automation tests. This easily allows you to monitor performance deterioration and improvement over time. There's an article on how to do this on the Dynatrace website.