Google not allowing me to download data (...network may be sending automated queries...)

virosoft

Hey!
I've been having an issue with my Portfolio Slicer updating lately.
Come to find out from the logs - Google has blocked me from extracting web data. I've tried rebooting and clearing cookies but it won't let me download data. Does anyone know a fix?


Portfolio Slicer\Scripts\Log\GetQuotesGoogleWeb.txt
Requesting url: https://www.google.com/finance/historical?q=CHKR&num=30&start=0&startdate=Nov+25,+2017
CHKR - Not Found (web err)

If I try the link manually in my browser I can see why it won't extract any information:
-----------------------------
We're sorry...

... but your computer or network may be sending automated queries. To protect our users, we can't process your request right now.
See Google Help for more information.
-----------------------------


Thanks!
Rene

virosoft

Nevermind, I've just replaced my files with the ones under this download:

Download v2.3.2 (2017-Dec-01): External data Download and Management script set (for majority of users).

Works now, sorry about that - but if anyone has something similar now you know!

deepakhemadri

Am having this issue for the last 2 days and am not able to download Google quotes. Any help please? Thanks.

jcm

Yeah, the Google scripts aren't working and they make the batch file hang. At this point I've commented them out of the batch file. Probably Google finally killed the old interface the scripts were using.

jagmanvdp

same problem here....newest scripts do not allow me to download data from googleweb
If anyone has a work around...would love to know

deepakhemadri

The googleweb scripts are still not working. Also, for some reason, Yahoo is missing days of data for Feb and March for India markets. Are others seeing the same issues?

VidasMatelis

Hi Deepakhemadri - I will investigate this over the weekend. Can you provide one symbol example that you have issues with.

deepakhemadri

Thanks for replying Vidas. This is on Yahoo - you can try MENNPIS.BO.

deepakhemadri

Also, I think there are no quotes on Yahoo for 14th and 15th of March - both trading days. (I know you can't do much about it if Yahoo doesn't provide quotes)

buddyb

I see that Google wants me to prove I'm not a Robot when my PC tries to connect to it's website. You can see the message when you try to connect "manually", using this shortcut link .... https://www.google.com/finance?q=TSE:CCO Best thing to do is check back next week after Vidas is back from his holidays. I've also looked at some of my Yahoo downloads and I'm seeing entries from March 14 and 15, so For me, Yahoo looks to be OK. (I'm looking at the Yahoo symbol "XLU").

VidasMatelis

deepakhemadri, I investigated symbol MENNPIS.BO and from what I see - yahoo stopped providing data for that symbol. If you go to link https://finance.yahoo.com/quote/MENNPIS.BO/history?p=MENNPIS.BO and click on "Download Data", you will see that just last days data is available, but historical data is not available anymore. So it appears that that problem is for specific symbols, but actual Yahoo script works as expected for other symbols.
Solution - I would wait at least a few more business days and see if Yahoo will bring back historical data for that symbol. Or you can try to see if there are other sources that you could get quotes for that symbol. Please read this page about quote sources available and how to switch between them: http://www.portfolioslicer.com/scripts/quotes-datasource.html

VidasMatelis

BuddyB,

If Google wants to to prove that you are not a robot, that most likely mean that you submitted too many queries and google suspect that you might be robot. That link you provided works for me without any problems.
There is one known issue that Maxim told me about and send me fix, but it is not included in the latest files. In file GetQuotes-GoogleWeb.ps1 line 38 should be changed from:

        catch { $reqFailed++; "  " + $symbol + " - Not Found (web err) `r`n" | Out-File $logFile -Encoding OEM -Append; continue;} # if attempt to get webpage failed go to next symbol

To:
catch { $reqFailed++; " " + $symbol + " - Not Found (web err) `r`n" | Out-File $logFile -Encoding OEM -Append; $pg=100; continue;} # if attempt to get webpage failed go to next symbol

When in GoogleWeb you have listed Symbol that Google does not recognize, it returns error page. But previously script was re-submitting that request 100 times. So you either can review GoogleWeb section in psConfig.txt and remove symbols that GoogleWeb does not recognize, or you can update code. To see if you are affected by this issue - please check Scripts Log folder file 'GetQuotes-GoogleWeb.txt" and see if you see multiple lines with "Not Found (web err)" in there.
To make google believe that you are not robot stop running script for a while and visit pages on your browser and keep answering captchas or other challenges that Google ask you to do to prove you are not a robot and you should be OK after a bit.

hkts

I also found that the Google quotes are no longer working.
I have switched to the Yahoo quotes and also found that the Mar 15th quotes for quite a number of symbols in Hong Kong are missing. I believe it is the problem of Yahoo.

deepakhemadri

Vidas, making that change to line 38 worked. Thanks so much for the quick help.

deepakhemadri

Sorry, Vidas. This problem is back again. Google-web script shows a web-err message and trying to open the URL gives the above "...automated queries" message. Is it just for me or others too?

Also, I have not been sending these intra-day refresh requests more than 5-6 times a day. Yet this issue comes up. Any thoughts? Thanks a lot.

Deepak

qosmioamit

Deepak, There is a new set of scripts on the Download page. Please use them. As of yesterday, Google does not allow using their site for currency exchange anymore.