Status: The Present

Statuses current as of 0 seconds ago.
Next refresh in 0 seconds.
Server time on pageload: 2012-01-28 02:55:29 -0800
 

Status: The Future

There are currently no scheduled maintenance windows.

Status: The Past

 

Tuesday, Jan 24th

2012-01-24 6:00am10:00am -07:00 (PST) 2012-01-24 13:002012-01-24 17:00 +00:00 (GMT)

v0.6.0 Subscriptions Processing Delay

Impacted Systems: app

There was a bug this morning that prevented subscriptions for stores on FoxyCart v0.6.0 from processing as normal. We have identified the issue and are working on a fix, both to run the subscriptions today as normal (albeit some hours delayed) and to prevent this from happening in the future. We’ll update here when that fix is made and the subscriptions are processed.

Update 10am PST: Resolved. All subscriptions have been processed as normal.

 

Monday, Jan 23rd

2012-01-23 12:00am1:35pm -07:00 (PST) 2012-01-23 07:002012-01-23 20:35 +00:00 (GMT)

USPS First Class rates not being returned

Impacted Systems: app

We are currently working on an issue that’s preventing USPS First Class rates from being returned. This is our top priority until it is resolved. We will be providing updates on our forum here and here, and will close this status alert when it’s fully resolved.

UPDATE 13:35 PST: This has been fully resolved. Please see the above linked forum threads for details.

 

Saturday, Jan 14th

2012-01-14 2:00am2:30am -07:00 (PST) 2012-01-14 09:002012-01-14 09:30 +00:00 (GMT)

Maintenance window on app1

Impacted Systems: app, admin

We are performing scheduled maintenance on our primary application server to upgrade some operating system packages. FoxyCart Stores may be offline for a minute or two during this window.

Update January 14, 2012 02:17 PST: Maintenance window completed. All systems normal.

 

Wednesday, Jan 11th

2012-01-11 12:00pm4:00pm -07:00 (PST) 2012-01-11 19:002012-01-12 23:00 +00:00 (GMT)

Unusually High Load on our Primary APP Server

Impacted Systems: app, admin

We’re currently experiencing high loads, resulting in slow pageloads. We’re working with our hosting provider to determine if this is a localized issue and will update this status once we know more.

Update January 11, 2012 14:08 PST: We have the loads under control and are continuing to monitor the situation. We found a memory leak in our web server processes and were able to resolve it by forcibly restarting them.

Update January 11, 2012 15:50 PST: Loads continue to fluctuate. We are scheduling an emergency maintenance window for this evening to take the server offline and reboot it.

 

Wednesday, Jan 11th

2012-01-11 9:00pm10:00pm -07:00 (PST) 2012-01-12 04:002012-01-12 05:00 +00:00 (GMT)

Emergency Maintenance: app1

Impacted Systems: app

As part of diagnosing and resolving the high load issue we are taking our primary application server offline for a brief reboot. Your FoxyCart Store will be unavailable for 5-10 minutes during this window. We apologize for any inconvenience and endeavor to get things back to normal as quickly as possible.

Update January 12, 2012 21:08 PST: Our maintenance window has been completed successfully and all systems are normal at this time.

 

Tuesday, Jan 10th

2012-01-10 9:00am10:00am -07:00 (PST) 2012-01-10 16:002012-01-10 17:00 +00:00 (GMT)

Mail delay

Impacted Systems: app, admin

Outbound mail was delayed during a 10 minute window this morning due to an issue with our email provider. At this time mails are once again delivering as rapidly as normal.

 

Sunday, Jan 8th

2012-01-08 10:00pm11:00pm -07:00 (PST) 2012-01-09 05:002012-01-09 06:00 +00:00 (GMT)

Emergency Maintenance: www

Impacted Systems: www, forum, wiki, affiliate

We have finished provisioning a more powerful server stack to replace our venerable www server. This maintenance window is complete the switchover to the new servers. During this time www, forum, and our wiki will be unavailable. Your FoxyCart store will not be affected by this transition.

Update January 7, 2012 11:10 PM: Our maintenance has been completed successfully.

 

Sunday, Jan 8th

2012-01-08 12:00pm2:00pm -07:00 (PST) 2012-01-08 19:002012-01-08 21:00 +00:00 (GMT)

Planned Maintenance: Forum upgrade

Impacted Systems: forum

We traced our www server’s performance problems to our forum software. We’ve tested out a new version and will be upgrading the forum this weekend. Forums will be unavailable during this maintenance window but FoxyCart stores and documentation will continue to function as normal.

Update January 8, 2012 12:32 PST: Our new forums are rocking and rolling! Your local DNS will update shortly.

 

Saturday, Jan 7th

2012-01-07 12:31pm3:31pm -07:00 (PST) 2012-01-07 19:312012-01-07 22:31 +00:00 (GMT)

More high loads on www

Impacted Systems: www, forum, wiki, affiliate

We are currently experiencing additional database and load problems in our www site. We are working to fix them. FoxyCart stores are unaffected.

Update January 7, 2012 13:09 PST: We are rebooting the affected server and provisioning new hosting to reduce load on our primary www server.

Update January 7, 2012 13:22 PST: The reboot has completed on www. DNS has begun to fail back for forum, wiki, and www.

 

Friday, Jan 6th

2012-01-06 10:10pm2:50am -07:00 (PST) 2012-01-07 05:102012-01-07 09:50 +00:00 (GMT)

Wiki, WWW server issues

Impacted Systems: www, wiki

Shortly after 10pm PST, the server that hosts our wiki and www sites started experiencing trouble. DNS failed over, and we are currently working to figure out the cause.

Our application environment (ie. your stores) was not impacted.

 

Friday, Jan 6th

2012-01-06 12:17pm1:17pm -07:00 (PST) 2012-01-06 19:172012-01-06 20:17 +00:00 (GMT)

High loads on www server

Impacted Systems: www, forum, wiki

High loads caused a DNS failover on www. FoxyCart stores are unaffected.

 

Wednesday, Jan 4th

2012-01-04 12:30pm4:00pm -07:00 (PST) 2012-01-04 19:302012-01-04 23:00 +00:00 (GMT)

High loads on the FoxyCart forum

Impacted Systems: www, forum, wiki

We’re currently experiencing high loads on our www server and are provisioning more resources to handle the traffic. Our forum is currently offline while we get things back under control. FoxyCart stores are unaffected by this outage.

Update January 4, 2012 14:04 PST: We have a new database server for the forums on standby and will bring it up as soon as possible.

Update January 4, 2012 14:52 PST: The new database server is up. DNS failover may still be in effect for some users, which will cause the forums to appear offline until the DNS update propagates.

 

Tuesday, Dec 13th

2011-12-13 7:30am12:30pm -07:00 (PST) 2011-12-13 14:302011-12-13 19:30 +00:00 (GMT)

Brief mail delays

Impacted Systems: app

Our mail provider seems to be having longer-than-usual delays when accepting new messages from FoxyCart. We’ve adjusted some settings on our end to increase the speed of mail delivery and, as usual, we will monitor to make sure that your store’s mail arrives as rapidly as usual.

 

Monday, Dec 12th

2011-12-12 12:20pm12:26pm -07:00 (PST) 2011-12-12 19:202011-12-12 19:26 +00:00 (GMT)

www under high load

Impacted Systems: www, forum, wiki, affiliate

Our www site is currently under high load and we’re working on fixing it. Update soon.

Update December 12, 2011 12:27 PST: Service has been restored on www.foxycart.com. FoxyCart stores were unaffected by this issue.

 

Wednesday, Nov 30th

2011-11-30 9:45am1:51pm -07:00 (PST) 2011-11-30 16:452011-11-30 20:51 +00:00 (GMT)

AUX Server Load Issues

Impacted Systems: www, forum, wiki, affiliate

We are experiencing issues with our the system housing our WWW, FORUM, and WIKI subdomains. While these issues are very brief, and we’re looking into them, they are tripping our DNS failover system, so if you see this page please check back in just a few minutes.

We sincerely apologize for the inconvenience. Please note that this does not impact our application environment.

 

Wednesday, Nov 30th

2011-11-30 8:28am8:32am -07:00 (PST) 2011-11-30 15:282011-11-30 15:32 +00:00 (GMT)

Service Interruption on www, forum, and wiki

Impacted Systems: www, forum, wiki, affiliate

We experienced high memory usage on our AUX server (which hosts our www site, forum, and wiki), which caused a brief DNS failover. FoxyCart stores were not affected.

 

Wednesday, Nov 9th

2011-11-09 3:30am4:45am -07:00 (PST) 2011-11-09 10:302011-11-09 11:45 +00:00 (GMT)

CDN Outage

Impacted Systems: app

The Edegcast Content Delivery Network (CDN) provider is currently unavailable, which may result in unacceptably slow page load times. We’re contacting Edgecast directly and rerouting our DNS entry for now.

You may be unable to update your store templates and images at this time.

Update November 9, 2011 03:42 DNS has been updated to temporarily disable the CDN. We are testing to ensure that this fix won’t adversely affect your stores.

Update November 9, 2011 04:45 Edgecast is back up and we have rerouted DNS. All systems normal.

 

Monday, Nov 7th

2011-11-07 2:25pm2:35pm -07:00 (PST) 2011-11-07 21:252011-11-07 21:35 +00:00 (GMT)

Network outage at primary datacenter

Impacted Systems: app, admin

Our primary datacenter is currently offline. We are awaiting updates from our hosting provider and will resolve this as soon as possible. Our apologies.

Update November 7, 2011 16:32 We’re back up. Still awaiting an explanation.
Update November 7, 2011 22:00 Our host provides this explanation: “We experienced a network event with their internet core. Traffic was quickly diverted and trouble area has been isolated and is under review.”

We now consider this case closed and apologize again for any inconvenience.

 

Wednesday, Nov 2nd

2011-11-02 12:45pm12:49pm -07:00 (PST) 2011-11-02 19:452011-11-02 19:49 +00:00 (GMT)

Service interruption on www

Impacted Systems: www

Our web server for our forum, help desk, and main www site ran low on memory, triggering a temporary DNS failover for that server only. FoxyCart stores were not affected.

 

Tuesday, Nov 1st

2011-11-01 11:00pm11:10pm -07:00 (PST) 2011-11-02 06:002011-11-02 06:10 +00:00 (GMT)

Database maintenance tonight

Impacted Systems: app, admin

We will be making a quick change to our database configuration to give us better insight into our database performance. Your FoxyCart store may be momentarily unavailable during the maintenance window but downtime will be minimal.

Update 23:02 PST: Maintenance has been completed successfully. All systems normal.