Server returned HTTP response code: 429 for URL: https://query1.finance.yahoo.com/v1/test/getcrumb

[zytan89]

I am having issue when run code line
CrumbManager.getCrumb();
#210

java.io.IOException: Server returned HTTP response code: 429 for URL: https://query1.finance.yahoo.com/v1/test/getcrumb
at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method)
at sun.reflect.NativeConstructorAccessorImpl.newInstance(Unknown Source)
at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(Unknown Source)
at java.lang.reflect.Constructor.newInstance(Unknown Source)
at sun.net.www.protocol.http.HttpURLConnection$10.run(Unknown Source)
at sun.net.www.protocol.http.HttpURLConnection$10.run(Unknown Source)
at java.security.AccessController.doPrivileged(Native Method)
at sun.net.www.protocol.http.HttpURLConnection.getChainedException(Unknown Source)
at sun.net.www.protocol.http.HttpURLConnection.getInputStream0(Unknown Source)
at sun.net.www.protocol.http.HttpURLConnection.getInputStream(Unknown Source)
at sun.net.www.protocol.https.HttpsURLConnectionImpl.getInputStream(Unknown Source)
at yahoofinance.histquotes2.CrumbManager.setCrumb(CrumbManager.java:167)
at yahoofinance.histquotes2.CrumbManager.refresh(CrumbManager.java:182)
at yahoofinance.histquotes2.CrumbManager.getCrumb(CrumbManager.java:187)
at yahoofinance.quotes.query1v7.QuotesRequest.getResult(QuotesRequest.java:65)
at yahoofinance.YahooFinance.getQuotes(YahooFinance.java:381)
at yahoofinance.YahooFinance.get(YahooFinance.java:98)
at yahoofinance.YahooFinance.get(YahooFinance.java:82)
at stock.api.StockApi.getHistoryData(StockApi.java:21)
at monitor.Screener.getData(Screener.java:194)
at monitor.Screener.screen(Screener.java:103)
at monitor.Screener.run(Screener.java:77)
at monitor.MonitorPanel.lambda$1(MonitorPanel.java:285)
at java.lang.Thread.run(Unknown Source)

[lumirodri]

@zytan89 Getting the same issue as of 2 days ago.

Apparently, they're now applying a rate limit, throttling the number of requests - which in turn results in the error code 429 - "Too Many Requests".

This seems like it's device and geo-based. I have users reporting that they don't get any data, and others saying it's working as before, although if I do test https://query1.finance.yahoo.com/v1/test/getcrumb on my browser, I always get a crumb.

If I fire up Android Studio and do the same request through the emulator, I get the error you described.

I'm not sure how, or if it's even possible to fix this issue.

[pktczwd]

You guys can try to add a "user-agent" header to the getcrumb request, then it can work as before.

[lumirodri]

You guys can try to add a "user-agent" header to the getcrumb request, then it can work as before.

I've wasted a couple hours fiddling with the code and I can say this indeed works (if you also change some other stuff). Here are the steps I've used:

• Step A:

  • FILE - CookieAndCrumbManager.java
  • Get a specific user-agent (e.g. access the website via your PC https://www.whatsmyua.info/ and retrieve the value.
  • On cookie request, add:
    • connection.addRequestProperty("User-Agent", YOUR_VALUE_HERE)
  • On crumb request, add:
    • connection.setRequestProperty("Cookie", cookie);
    • connection.addRequestProperty("Cookie", cookie);
    • connection.addRequestProperty("User-Agent", YOUR_VALUE_HERE)
    • connection.setRequestMethod("GET");

• Step B:

  • FILE - QuotesRequest.java
  • Change the type of connection from RedirectableRequest to HttpURLConnection (as per CookieAndCrumbManager)
  • On get result, add
    • connection.setRequestProperty("Cookie", cookie);
    • connection.addRequestProperty("Cookie", cookie);
    • connection.addRequestProperty("User-Agent", YOUR_VALUE_HERE)

Thanks for the tip! @pktczwd

DISCLAIMER: this should only be used for testing purposes, obviously. You shouldn't spoof user-agent!

[pktczwd]

You guys can try to add a "user-agent" header to the getcrumb request, then it can work as before.

I've wasted a couple hours fiddling with the code and I can say this indeed works (if you also change some other stuff). Here are the steps I've used:

• Step A:

  • FILE - CookieAndCrumbManager.java

  • Get a specific user-agent (e.g. access the website via your PC https://www.whatsmyua.info/ and retrieve the value.

  • On cookie request, add:

    • connection.addRequestProperty("User-Agent", YOUR_VALUE_HERE)

  • On crumb request, add:

    • connection.setRequestProperty("Cookie", cookie);
    • connection.addRequestProperty("Cookie", cookie);
    • connection.addRequestProperty("User-Agent", YOUR_VALUE_HERE)
    • connection.setRequestMethod("GET");

• Step B:

  • FILE - QuotesRequest.java

  • Change the type of connection from RedirectableRequest to HttpURLConnection (as per CookieAndCrumbManager)

  • On get result, add

    • connection.setRequestProperty("Cookie", cookie);
    • connection.addRequestProperty("Cookie", cookie);
    • connection.addRequestProperty("User-Agent", YOUR_VALUE_HERE)

Thanks for the tip! @pktczwd

DISCLAIMER: this should only be used for testing purposes, obviously. You shouldn't spoof user-agent!

Yes, I also wasted many hours on it before.

1. No need to add user-agent to the cookie request, only the cookie and user-agent are mandatory.
2. Remove the CookieHandler in yahoofinance.util.RedirectableRequest (and manage the cookie mannually in yahoofinance.histquotes2.CrumbManager).
3. Add a Accept header with value text/html to the get cookie request.
4. Remove the pattern limitation in setCookie method, and just store all the cookies.
5. Do not forget include the crumb parameter in your query reqeust.

[playinlab]

Hi,

I have the same issue (HTTP response code: 429).

Is it possible to download a working YahooFinanceAPI-3.18.0-SNAPSHOT.jar somewhere?

Best regards,
Primož

[playinlab]

Hi,

Can you at least attach the files which were changed in your working solution please (CrumbManager.java, QuoteRequest.java...).

I tried many different things but I can not get data from yahoo. I always get an error:

java.io.IOException: Server returned HTTP response code: 429 for URL: https://query1.finance.yahoo.com/v1/test/getcrumb
at java.base/jdk.internal.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method) ~[na:na]
at java.base/jdk.internal.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:62) ~[na:na]
at java.base/jdk.internal.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45) ~[na:na]
at java.base/java.lang.reflect.Constructor.newInstance(Constructor.java:490) ~[na:na]

Best regards

[playinlab]

Hi guys,

I solved the problem. Thanks @pktczwd for your hint.

Best regards,
Primož

[SamSambuka]

Hi,
unfortunately i can't get it to work.
can please someone write a step by step tutorial. or create a pull request with code changes.
thanks!

[kpavan536]

@pktczwd and @playinlab followed same steps but getting java.lang.IllegalStateException: Already connected
at java.base/sun.net.www.protocol.http.HttpURLConnection.setRequestProperty(HttpURLConnection.java:3316)
at java.base/sun.net.www.protocol.https.HttpsURLConnectionImpl.setRequestProperty(HttpsURLConnectionImpl.java:286)
at yahoofinance.histquotes2.CrumbManager.setCookie(CrumbManager.java:51)
at yahoofinance.histquotes2.CrumbManager.refresh(CrumbManager.java:216)
at yahoofinance.histquotes2.CrumbManager.getCrumb(CrumbManager.java:222)
at yahoofinance.quotes.query1v7.QuotesRequest.getResult(QuotesRequest.java:63)
at yahoofinance.YahooFinance.getQuotes(YahooFinance.java:381)

[kpavan536]

with below peace of code in get result. it is working
//URLConnection connection = redirectableRequest.openConnection();
HttpURLConnection connection = (HttpURLConnection) request.openConnection();
connection.setRequestProperty("User-Agent", "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7) AppleWebKit/605.1.15 (KHTML, like Gecko) Version/16.5.2 Safari/605.1.15");
connection.setRequestProperty("Cookie",CrumbManager.getCookie());
InputStreamReader is = new InputStreamReader(connection.getInputStream());

[SamSambuka]

@kpavan536 can you please provide code from CrumbManager.java and QuotesRequest.java ?

[code-monkey-101]

@SamSambuka This is how I did it. Did not have to change QuotesRequest to make it work.

Big thanks to @pktczwd and everybody else who provided assistance!

[ghost]

@code-monkey-101 could you show your code for the cookie portion as well?

[code-monkey-101]

@code-monkey-101 could you show your code for the cookie portion as well?

I didn't (have to) make any changes to setCookie. This is the only change I made for error 429.

However, there was a similar issue (error 401) not too long ago for which I posted a fix here: #206

[SamSambuka]

Hi, works for me with this changes:
develop...v3rtumnus:yahoofinance-api:develop

[moritz31]

@SamSambuka i tried your code but i am still stuck at 401. Based in eu and crumb on webpage gets me something, while the application it is always empty