Hi guys,

This week I've been getting a lot of reports from the users of my third party app that say they are suddenly unable to use the Comic Vine API to search for a couple of specific volumes. So I investigated, and I found that when you use the Search API to search for those two volumes, the http request will always time out and you'll get either a 500 or a 502 (bad gateway) error.

This is a new behaviour. It might indicate that some kind of infinite loop or something similar was recently introduced on the server side of Comic Vine's public API. What follows is a specific description of the problem. Note: you will have to substitute your own API key into the URL's I've provided.

-------

Say I'm searching for "Superman" volumes. The API query might look like:

https://comicvine.gamespot.com/api/search/?api_key=[YOUR-API-KEY]&format=xml&limit=20&query=superman&resources=volume

But as of this moment, that query always times out. If you change the search term to something else that will return a lot of volumes ('spiderman' or 'batman') it still works fine and returns results quickly.

What about if I'm searching for "Cyborg"? As of this writing, there's only 26 volumes that should be returned by this http request:

https://comicvine.gamespot.com/api/search/?api_key=[YOUR-API-KEY]&format=xml&limit=100&query=cyborg&resources=volume

But this request times out too! Note that limit=100 and query=cyborg.

If I switch to limit results to 20 volumes instead, it works fine:

https://comicvine.gamespot.com/api/search/?api_key=[YOUR-API-KEY]&format=xml&limit=20&query=cyborg&resources=volume

In fact, if I try anything less than 20 issues, it also works fine. But 21 issues or more, and it always times out. Also, for the first query I mentioned (superman), it times out no matter what I set the limit to.

-------

Hopefully this is enough info to help you guys track down the bug. It is probably related to changes that were made in the last week or so, since my app has a lot of users and they all started noticing problems on Wednesday. The problems have all been related to the same queries I've described above ('cyborg' and 'superman', limit=100).

If I can help verify a fix or provide more information, just let me know.

Here's a link to the Cyborg #18 page, another issue that is causing problems. It can't even be viewed in a browser, I get a "Something's Wrong T_T" error. . https://comicvine.gamespot.com/cyborg-18-singularity-aftermath-the-end/4000-634508/ so this isn't just an API or 3rd party program problem.

@solidus0079: page is viewable now, the site was having some outage problems through out the week.

Thanks, pkahyper -

But now something else seems to be going on. Scraping any comics at all seems to result in an IP ban. I'm posting via the Tor browser at the moment.

I'm also seeing an IP ban whenever attempting an API call. Attempting to reach the first url in op (https://comicvine.gamespot.com/api/search/?api_key=[YOUR-API-KEY]&format=xml&limit=20&query=superman&resources=volume) resulted in a page saying site undergoing maintenance... is there an ongoing issue with the API service?

I am also getting IP banned for 30+ minutes for running any API query.

Seems resolved for now at least. :)

@solidus0079: Yup, looks good on for me too. I wonder what happened....

still having issues with Superman. Although the message I get is different "The Comic Vine online database could not be reached. This may be a temporary technical error with the database, or there may be a problem with your internet connection."

I am able to get info for other comics right after this error. Prior to today I was unable to get info for any book, even though I was able to seemy current api usage showing I was within the limits. BTW this started 3 days ago in the middle of a request for 34 book updates. Let me know if there is any other info you need.

@admjim I was able to scrape that issue just fine after whatever fix was fixed.

Is the outage problem solved? I am still having trouble scraping. Not only superman and cyborg, but anything past 20 queries give or take.

@romsnesrom: I just did over 400 scrapes, including the problematic Superman and Cyborg issues. Not one hiccup.

Just to clarify, the original bug I reported in this thread is still occurring (i.e. can't search for Superman or Cyborg with the API).

People have started discussing a separate, unrelated bug (IP bans) in this thread as well.

Just to clarify my issue is still going on I cannot scrape the Superman Rebirth issues 30-34 however the message I get is as stated earlier it times out and then gives "The Comic Vine online database could not be reached. This may be a temporary technical error with the database, or there may be a problem with your internet connection." I attempted to scrape each issue one at a time and as a group with other books, each time it failed on the Superman Book.

I do not get a either a "500 or a 502 (bad gateway) error.

I am able to scrape other Superman issues from other years as well as other books not related to Superman. Allof this is within the last 15 minutes.

I have not tested the Cyborg issue so I cannot comment on that.

That error you are getting in the Comic Vine Scraper application (which is not created by nor directly supported by the comicvine.com) is actually being caused by 500 and 502 errors that Comic Vine Scraper encounters when it tries to access the Comic Vine API.

My original bug report is just showing those failing API calls directly (and taking Comic Vine Scraper out of the equation) but we are actually reporting the same problem.

This should now be resolved, thanks for all the reports. We believe it was being caused by Comic Vine's own network provider misidentifying the CVS app as a threat. They have removed the signature causing the issue.

Let me know if its still a problem.

I have been having severe problems using the Comic Vine scraping script for Comicrack for the past few days. I've been getting the error "The Comic Vine online database could not be reached. This may be due to a temporary technical error with the database, or there may be a problem with your internet connection.". This problem has cleared up today, except for scraping comics with very generic names. For example, I have a copy of "Marvels" that I can't scrape for. It acts like it's working, and when it gets to 500 entries it stops, and gives me the "Cannot Access Online Database" error.

The problem persists for me, Steve. If there is another data I could provide to help diagnose the situation, just ask.

@romsnesrom What are your specific symptoms? The more detail the better.

@chrishadley We found the issue. Its unrelated to the issue we already fixed, but would have the same timeout symptoms. We will get a fix out asap and I will report back. (Thanks for the detail about results after 500 being the issue, that was the magic detail that narrowed it down for us).

@romsnesrom This might be the source of your issue as well.

Thanks for all the help of late, Steve.

Superman, Cyborg, and 'The Order', here, still not working.

I'm still having the issue with items with very generic names, like 'Marvels', 'Marvel Super-Heroes', 'The Cat', and also, amusingly enough, 'Cyborg' and 'The Order'. It looks like it is hitting a limit of number of responses and locking up. I understand there may be a limit of responses on the API? Is there some way to rate-limit Comic Vine Scraper? I found no way to do that without rewriting it. It will let you pause between comics, but there's no way to limit the number of responses on a single comic. Thank you so much for your help so far!

Edit: Also, it's getting to 600 responses before dying now :)

I am happy to report that most of the problem is gone. I am still having problem with the series everyone is already complaining: "Cyborg", "Superman", "The Order", "Armies".

@cbanack: or @steveeasley As of right now (Sunday 11/19/17 11:15am CT) I am still having issues with Superman 30-34 (2017) and Cyborg 17 &18 (2017). I have been using the script on and off for the last 3 days and have never been able to get these books to scrape yet I am able to get 100's of others to wrok with now issue's. Please let me know if there is something I can do to help trace the issue. Maybe setup a Date/Time to test. I know what to do at this point to break it ;)

Hi all - I'm having the same issues as @admjim. I can't scrape Cyborg 18 and Superman 34, 35. Just did a bunch of other scraping. I tried linking to those addresses in post #1 with my API key and got the 502 error.

Can you guys try again? We just released the fix I mentioned above related to some timeouts.

It looks like all of the ones that scraping was failing on are now working, I've cleared my whole backlog successfully. You rock!

Same here - everything scraped with no issues. Thank so much!