Dashboard slowness/timeouts related to Warnings for Add-ons List Manager
complete
Zack Jenkins
Currently, GiveWP core plugin relies on an API call to givewp.com to check things, and if that call is delayed, it's causing major issues of locking down sites. Givewp.com experienced some intermittent downtime over the past few days which resulted in problems on customer's sites.
Here's the error displayed either on the dashboard directly or in PHP error logs:
Warning: Invalid argument supplied for foreach() in /home/customer/www/keywest.garden/public_html/wp-content/plugins/give/src/License/PremiumAddonsListManager.php on line 42
Jason Adams
complete
This fix for this was just released in GiveWP 2.11.3.
We sincerely apologize for the inconvenience this issue has caused to our customers. We hope that our immediate response to the problem has communicated our commitment to serving our customers and maintaining high standards.
To recap, the Issue first began when our server could not handle the flood of incoming requests for the list of products — to determine which add-ons you have installed are premium and which are free. A setting was enabled in Cloudflare (which we use for security and speed optimizations), which implicitly disabled another rule we had in place which caches the product list every day. With that in place, the response is typically almost instant and doesn't even touch our server. The second issue was the fact that, after givewp.com went down, GiveWP core wasn't properly handling this. It should quietly move along, and try again later. But instead it was making multiple requests at once, each one timing out after 5 seconds. So 6 add-ons would cause a 30 second delay, timing out the host.
The fix in this patch uses a few strategies to make sure there is only ever 1 request, that times out after 3 seconds, per hour. So if givewp.com goes down, there will be at most one slightly longer request (not nearly enough to time out the host). A successful request to givewp.com caches for a day, and an unsuccessful attempt caches for an hour. This will prevent givewp.com from causing any downtime to customer sites in the future.
GiveWP core is an open-source project, so you are welcome to learn more about the in-depth details of the problem on the public Issue and Pull Request
As always, thank you for bringing this to our attention so we could resolve it for you, and as well for being awesome customers. We appreciate you and value your cause!
Jason Adams
complete
This fix for this was just released in GiveWP 2.11.3.
We sincerely apologize for the inconvenience this issue has caused to our customers. We hope that our immediate response to the problem has communicated our commitment to serving our customers and maintaining high standards.
To recap, the Issue first began when our server could not handle the flood of incoming requests for the list of products — to determine which add-ons you have installed are premium and which are free. A setting was enabled in Cloudflare (which we use for security and speed optimizations), which implicitly disabled another rule we had in place which caches the product list every day. With that in place, the response is typically almost instant and doesn't even touch our server. The second issue was the fact that, after givewp.com went down, GiveWP core wasn't properly handling this. It should quietly move along, and try again later. But instead it was making multiple requests at once, each one timing out after 5 seconds. So 6 add-ons would cause a 30 second delay, timing out the host.
The fix in this patch uses a few strategies to make sure there is only ever 1 request, that times out after 3 seconds, per hour. So if givewp.com goes down, there will be at most one slightly longer request (not nearly enough to time out the host). A successful request to givewp.com caches for a day, and an unsuccessful attempt caches for an hour. This will prevent givewp.com from causing any downtime to customer sites in the future.
GiveWP core is an open-source project, so you are welcome to learn more about the in-depth details of the problem on the public Issue and Pull Request
As always, thank you for bringing this to our attention so we could resolve it for you, and as well for being awesome customers. We appreciate you and value your cause!
D
D O
Jason Adams: Thank you guys (and gals) for taking care of the problem so quickly, and all the communication throughout. I'd say you definitely deserve our money for your software (as if you didn't already). :)
Jason Adams
in progress
As Ben noted, this issue has two aspects:
- We need to make sure givewp.com stays up.
- Even if givewp.com goes that, that should NOT effect customer sites.
At this point, givewp.com is back up and a patch for preventing it affecting sites in the future is finished and in testing. It will go out today.
L
Lars Kemmann
Jason Adams: Good to hear! For those of us who applied John Sundberg's temporary fix below, what steps will we need to take to get the patch? (Assuming everyone else can just auto-update the GiveWP plugin, but we may need to do something more?) Can you let us know the version number for the patch when it's released? And does the patch do anything to address things more broadly than the workaround did?
Jason Adams
Lars Kemmann: I will absolutely update this when it's released. It will be going out as GiveWP 2.11.3.
John's fix below simply prevents the request causing issues. The patch I just finished adds caching layers in place so if a failed request happens, it will only take 3 seconds and happen once per hour. You can see the fix and read more about the solution in detail here: https://github.com/impress-org/givewp/pull/5863
D
Darren Russ
Jason Adams: Realise you are under the pump to confirm the resolution, regression test, etc - but was wondering if you had an approximate timeframe you believe the new version will be available for update? After the fact - may I also request a published review of how GiveWP will be architected or updated to mitigate or prevent issues of this nature adversely affecting non-GiveWP site functionality?
Jason Adams
Darren Russ: I just gave a nice, length response outlining the release — the underlying issue, the solution, and the release version. Please review this and let us know if you have any further questions or concerns. 😃
D
Darren Russ
Jason Adams: Thank you for your detailed response. I'm encouraged by your work to identify and resolve the error quickly (have just installed the update); thank you. It was very helpful to understand the root cause of the issue and to also provide other links and information for me to gain further insight and understanding about GiveWP by exploring the repo.
D
D O
Lars Kemmann: if you commented line 34 in /plugins/give/src/License/PremiumAddonsListManager.php, just edit that file again and remove the comment; then install the GiveWP Core update. That's what I just did- no probs.
D
D O
Unofficial temporary fix (credit to John Sundberg (@bhwebworks))
For a temporary fix until GiveWP sends out an update, comment out line 34 in the file listed in the Warning message – /plugins/give/src/License/PremiumAddonsListManager.php
Line 34 should be this line:
$response = wp_remote_get( self::PRODUCTS_LIST_API_URL . '?number=-1' );
Comment that line by adding // in front of it.
It looks like this error didn’t show for a while due to the plugin using a cached transient that must have ended recently.
Ben Meredith
under review
What's going on here is that our site (at givewp.com) is experiencing some issues loading a specific API endpoint, and that problem is being "passed along" to users of our plugin in the form of slow loading time specifically in the admin.
I am not currently seeing anything that is affecting the front end load times but either way, I've escalated this to the team to have a look, urgently.
This is priority 1 for us, at this point, to resolve this issue. Thanks for all of the reports of this.
D
D O
Ben Meredith: thanks for the update! A question comes to mind that I think might be of community interest:
How do we run a licensed copy of GiveWP completely independent of the GiveWP platform itself? Take today’s scenario as a good example, where something is happening external to sites running GiveWP that is negatively affecting all sites. It seems this problem has to do with updates. If true, why is there not an option to enable/disable automatic updates for GiveWP? In other words, how do we manually stop this problem on a site-by-site basis?
Justin Freid
That is a great question. Further, the plugin has to be completely reengineered such that an API issues doesn't bring all member dashboards to a complete crawl.
D
D O
Definitely. Just asked these via a priority support request; I'll post the reply for community visibility so we can all benefit from the answer:
- What specific information is being sent from my site to yours or anyone else's, with regard to GiveWP, and when does it occur?
- Today's problem has shown a scenario where the capabilities of my site can be [inadvertently] disrupted by you. What, specifically, do I need to do modify the GiveWP plugin or its add-ons to allow GiveWP to operate without such dependency? If today's problem is update-related, which it seems like it might be, I would be ok forcibly disabling GiveWP updates in order to ensure uptime on my end regardless of your server status. Edit: temporary fix, thank you John Sundberg @bhwebworks, posted over at wordpress.org at https://wordpress.org/support/topic/crashed-site-33/, commenting out line 34 in the offending file did the trick.
Ben Meredith
Justin Freid: Our team is working on the fix here, and it's not going to be a "complete reengineering" but you are correct that an outage or any slowness at our website should not affect plugin users. That's going to be the top priority for the team, and we of course will look into making sure there are not other places where this type of thing can occur. Thanks!
Ben Meredith
D O: Your questions are valid here, for sure. The short answer is that nothing is being sent to our website without your consent. If you opt into sharing telemetry data with us, anonymized data like what other plugins are installed, what version of WordPress you are running, etc is sent to us.
In this specific case a call is being sent to retrieve data
from
GiveWP.com, namely the list of premium add-ons available at GiveWP.com. As to your other question, there's not a way to completely isolate your site from all dependency on third party sites. In this specific case we are definitely going to make it so that GiveWP.com is not the cause of issues, but our plugin functionally causes dependency on some third party services (like the payment gateway, for example), and there's no way around that.
L
Lars Kemmann
Ben Meredith: I think what I'd like to know is why on earth that list of premium add-ons is being retrieved when users visit our homepage or donation form. 😝 Totally understand dependencies etc., but that seems like it could easily be optimized out to only show on the single purpose-built page for it that already exists in the admin panel.
Ben Meredith
Lars Kemmann: Totally agree with you there. Its my understanding that it is only happening on admin pages, though, not on the front end. Our team is working on patching things, either way.