1.877.414.4678

PhoenixNap Data Center Power Failure

General summary:

  • Power failures at the datacenter caused two network outages, one for 5 minutes, and one for 22 minutes.
  • We've since resolved the redundancy issues which allowed this to impact our service, and the datacenter has since resolved the underlying power problems.
  • Physical servers and virtual machines were not taken down, but network access was lost during this time.
  • No data was lost or corrupted.
PhoenixNap Status Alerts

Click here to read more

Unfortunately, our datacenter, PhoenixNAP, experienced very serious power failures throughout the evening of 11-13-18. These issues have been resolved by the datacenter. While we don't own this datacenter, these issues were widespread and affected our cage and equipment.

General summary:

  • Power failures at the datacenter caused two network outages, one for 5 minutes, and one for 22 minutes.
  • We've since resolved the redundancy issues which allowed this to impact our service, and the datacenter has since resolved the underlying power problems.
  • Physical servers and virtual machines were not taken down, but network access was lost during this time.
  • No data was lost or corrupted.

At 9:15 PM (Arizona Local Time -700), the primary power at our datacenter experienced a catastrophic failure, but it quickly recovered in 5 minutes. Our router went down as a result of this, though none of our servers went down. After investigating this remotely, and seeing that such a critical device went down to what appeared to only be a failure on our "A Power" (we have both A and B power for our equipment), we went to the datacenter to see what happened in person.

While at the datacenter, the same circuits experienced much more damaging failure, and we were able to determine that the redundancy for the router had failed under the increased load, and after replacing the appropriate equipment, we brought our network back online. This outage lasted 22 minutes, starting at 11:20 PM (-700). While we could have swapped that part out relatively quickly, the primary power affected the "man traps" in the datacenter, so the datacenter security had to bring the many frustrated customers through a back door.

The datacenter continued to experience very severe power issues; however, all of our physical servers maintained at least one working PSU during these power issues. While the power issues lasted close to 3 hours, we were fortunate to mitigate that down to 27 minutes (the combined outage for the two incidents mentioned above). We've added additional redundancy in the network in place so that if this sort of issue does occur again, we shouldn't have any network downtime.

While the power issues damaged a lot of equipment (power supplies in particular), at this time we don't expect any further downtime. We will be replacing many power supplies that were destroyed during the datacenter's power failures, as well as other affected redundant equipment.

The datacenter itself will be providing an official "RFO" detailing the cause of these issues. If you're interested in that, we can pass it along once they have made it available. They will be doing an investigation on their end, but hopefully they will have that within several weeks.

In addition, we relied on our datacenter's IP services for our support system. Unfortunately we had issues with their network during these power failures as well. This caused our support system to be unreachable intermittently during the power failure event. In the short term, we're going to be moving that system to our redundant network, so that this sort of outage won't affect that system. Soon after that we'll setup a high-availability solution, to ensure those tools for communication with us are always available.

We're very sorry for these issues. We've never had this severe of a power issue at this particular datacenter in the 6 years we've been there. While things could have been worse, we always want perfection for our customers.

More info: PhoenixNap Status Alerts

[Resolved] Litespeed zero-day attack causing sites to be unresponsive

Starting at roughly 11:30AM Eastern Time, a Russian Network began exploiting a zero-day vulnerability in the Litespeed Web service. This allowed the attackers to cause the web service become unresponsive to requests; however, there is no evidence to suggest the vulnerability had any implications beyond creating downtime for customer websites.

Litespeed full disclosure:
- http://blog.litespeedtech.com/2017/02/17/ddos-attacks-taken-offline-by-litespeed-enterprise-5-1-13/

Click here to read more

Crucial Hosting worked directly with Litespeed to provide feed-back and help identify the vulnerability, as well as assist in identifying the offending networks. We provided this information to other companies on the internet, so they could also mitigate this denial of service attack. We were the first company to publicly identify the attacking networks, and we began preventing those networks from continuing their attack on our servers almost immediately.

Since this was a zero-day attack, it affected all known version of Litespeed. The only solution to the problem was ultimately waiting for Litespeed to patch the code which allowed the denial of service to take place.

Some customers may have experienced intermittent access to their websites, between 11:30 AM Eastern Time and roughly 2:15PM Eastern Time. Most traffic continued to work fine, though some sites, in particular, were targeted especially for this denial of service. We were actively working to stop this malicious activity throughout the process, and as a result of that, we mitigated the vast majority of downtime.

We have continued to monitor all of our servers for downtime, and while we have not seen this issue since patching all of our systems, we will continue to monitor for issues going forward.

- Crucial Administration

[COMPLETED] Power maintenance affecting NS3 and our support desk

The datacenter we lease equipment from will be performing a required power maintenance in the early morning on April 3rd between 1AM and 4AM EDT. They expect the maintenance to last roughly 20 to 30 minutes, and during that time our support desk and ns3.crucialwebhost.com will be unavailable.

Click here to read more

While we do our best to avoid any outages affecting important systems of ours, physical server migrations are a reality of web hosting and running a datacenter.

While almost all of our infrastructure is in our own physical control at our Phoenix datacenter, allowing us to mitigate downtime to just a few minutes, our support desk and nameservers are strategically on different networks from our main one. This is intended to diversify our availability in the event of a network outage.

We expect very little impact from this maintenance, as most browsers will automatically use one of our other two available nameservers. NS1 and NS2 will not be impacted by this maintenance.

We apologize for any inconvenience this may cause.

Magento 2.x Security Upgrade Available

Upgrade Your Magento 2.x Site Now

Today, Magento has made a new upgrade available that improves the security and functionality of Magento 2.x sites. The new release, Magento 2.0.3, is available for both Magento Enterprise Edition and Community Edition, and contains several security improvements, including:

Click here to read more

  • Prevents anonymous access to web APIs by default so that private information about the store, such as pricing, stock details, and upcoming promotions, are not disclosed without authentication. Merchants can still configure their APIs to support anonymous access if it is required by certain extensions. More information is available here.

  • Sets limits on the number of Admin and Customer Token Access API password attempts allowed to help prevent brute force attempts to guess passwords.

  • Fully resolves a previous issue with cross-site scripting so that attackers cannot enter an email address with malicious JavaScript code during customer registration on the storefront.

  • Fixes multiple parameters in the Authorize.net payment module that were vulnerable to reflected cross-site scripting attacks.

In addition, Magento 2.0.3 includes performance improvements and functional enhancements to the Orders API, Google Tag Manager, permissions, and other areas. Full details on the functional enhancements are included in the release notes. More information on the security updates can be found on the Magento Security Center.

All Magento 2.x users are strongly encouraged to update immediately.

UPDATE

Magento has released Magento Enterprise Edition and Community Edition 2.0.4 to address a packaging issue with yesterday's release. If you have already installed Magento Enterprise Edition or Community Edition 2.0.3, you must replace it with the new version to ensure that you receive all security enhancements.

[COMPLETED] Scheduled Maintenance Window 03-25-16

In our continuing effort to improve the Crucial Hosting network infrastructure and further reduce chance of an unplanned outage, we have scheduled the following Network Maintenance windows. During this maintenance there will be a short service outage as the maintenance is performed which we expect to last 5-20 minutes. Servers will be gracefully powered down and rebooted during the maintenance.

Click here to read more

Network Technicians will be on site as this maintenance is performed to ensure that downtime is limited to the shortest amount of time possible.

If you are experiencing an outage during this maintenance window your services will be available in a short time as the maintenance is completed.

Thank you for your patience as this mandatory maintenance is performed.

Crucial Administration

===================================================
Scheduled Network Maintenance Window 1
Date: Friday, March 25, 2016 (03/25/2016)
Start Time: 08:00 PM EST
End Time: 02:00 AM EST (03/26/2016)
Overflow: 03/26/2016, 8PM EST - 02:00AM (03/27/2016)
Duration: 5 - 20 minutes
Hosts affected:

phx-1310.split-dedicated.com
phx-1344.split-dedicated.com
phx-1348.split-dedicated.com
phx-1349.split-dedicated.com
phx-1350.split-dedicated.com
phx-1351.split-dedicated.com
phx-1352.split-dedicated.com
phx-1354.split-dedicated.com
phx-1355.split-dedicated.com
phx-1356.split-dedicated.com
phx-1357.split-dedicated.com
phx-1358.split-dedicated.com
phx-1360.split-dedicated.com
phx-1362.split-dedicated.com
phx-1365.split-dedicated.com
phx-1375.split-dedicated.com
phx-1389.split-dedicated.com
phx-1430.split-dedicated.com

===================================================

Scheduled Network Maintenance Window 2
Date: Saturday, March 26, 2016 (03/26/2016)
Start Time: 08:00 PM EST
End Time: 02:00 AM EST (03/27/2016)
Overflow: 03/27/2016, 8PM EST - 02:00AM (03/28/2016)
Duration: 5 - 20 minutes
Hosts affected:

phx-1239.split-dedicated.com
phx-1266.split-dedicated.com
phx-1267.split-dedicated.com
phx-1299.split-dedicated.com
phx-1316.split-dedicated.com
phx-1320.split-dedicated.com
phx-1321.split-dedicated.com
phx-1322.split-dedicated.com
phx-1323.split-dedicated.com
phx-1324.split-dedicated.com
phx-1326.split-dedicated.com
phx-1327.split-dedicated.com
phx-1328.split-dedicated.com
phx-1329.split-dedicated.com
phx-1330.split-dedicated.com
phx-1331.split-dedicated.com
phx-1332.split-dedicated.com
phx-1335.split-dedicated.com
phx-1336.split-dedicated.com
phx-1337.split-dedicated.com
phx-1338.split-dedicated.com
phx-1339.split-dedicated.com
phx-1340.split-dedicated.com
phx-1342.split-dedicated.com
phx-1345.split-dedicated.com
phx-1346.split-dedicated.com
phx-1347.split-dedicated.com
phx-1359.split-dedicated.com
phx-1378.split-dedicated.com
phx-1382.split-dedicated.com
phx-1384.split-dedicated.com
phx-1396.split-dedicated.com
phx-1404.split-dedicated.com
phx-1405.split-dedicated.com
phx-1407.split-dedicated.com
phx-1409.split-dedicated.com

===================================================

[COMPLETED] Scheduled Maintenance Window 03-18-16

In our continuing effort to improve the Crucial Hosting network infrastructure and further reduce chance of an unplanned outage, we have scheduled the following Network Maintenance windows. During this maintenance there will be a short service outage as the maintenance is performed which we expect to last 5-20 minutes. Servers will be gracefully powered down and rebooted during the maintenance.

Click here to read more

Network Technicians will be on site as this maintenance is performed to ensure that downtime is limited to the shortest amount of time possible.

If you are experiencing an outage during this maintenance window your services will be available in a short time as the maintenance is completed.

Thank you for your patience as this mandatory maintenance is performed.

Crucial Administration

===================================================
Scheduled Network Maintenance Window 1
Date: Friday, March 18, 2016 (03/18/2016)
Start Time: 08:00 PM EST
End Time: 02:00 AM EST (03/19/2016)
Overflow: 03/19/2016, 8PM EST - 02:00AM (03/20/2016)
Duration: 5 - 20 minutes
Hosts affected:

  • phx-1212.split-shared.com
  • phx-1214.split-shared.com
  • phx-1218.split-dedicated.com
  • phx-1221.split-dedicated.com
  • phx-1222.split-dedicated.com
  • phx-1258.split-dedicated.com
  • phx-1388.split-dedicated.com
  • phx-1390.split-dedicated.com
  • phx-1391.split-dedicated.com
  • phx-1392.split-dedicated.com
  • phx-1393.split-dedicated.com
  • phx-1394.split-dedicated.com
  • phx-1395.split-dedicated.com
  • phx-1397.split-dedicated.com
  • phx-1398.split-dedicated.com
  • phx-1399.split-dedicated.com
  • phx-1401.split-dedicated.com
  • phx-1403.split-dedicated.com
  • phx-1408.split-dedicated.com
  • phx-1418.split-dedicated.com
  • phx-1428.split-dedicated.com
  • phx-1429.split-dedicated.com

===================================================

===================================================
Scheduled Network Maintenance Window 2
Date: Saturday, March 19, 2016 (03/19/2016)
Start Time: 08:00 PM EST
End Time: 02:00 AM EST (03/20/2016)
Overflow: 03/20/2016, 8PM EST - 02:00AM (03/21/2016)
Duration: 5 - 20 minutes
Hosts affected:

  • phx-1208.split-shared.com
  • phx-1210.split-shared.com
  • phx-1211.split-shared.com
  • phx-1301.split-dedicated.com
  • phx-1319.split-dedicated.com
  • phx-1361.split-dedicated.com
  • phx-1366.split-dedicated.com
  • phx-1367.split-dedicated.com
  • phx-1368.split-dedicated.com
  • phx-1369.split-dedicated.com
  • phx-1370.split-dedicated.com
  • phx-1371.split-dedicated.com
  • phx-1372.split-dedicated.com
  • phx-1373.split-dedicated.com
  • phx-1374.split-dedicated.com
  • phx-1376.split-dedicated.com
  • phx-1380.split-dedicated.com
  • phx-1381.split-dedicated.com

===================================================

[COMPLETED] Scheduled Maintenance Window 02-16-16

In our continuing effort to improve the Crucial Hosting network infrastructure and further reduce chance of an unplanned outage, we have scheduled the following Network Maintenance windows. During this maintenance there will be a short service outage as the maintenance is performed which we expect to last 5-20 minutes. Servers will be gracefully powered down and rebooted during the maintenance.

Click here to read more

Network Technicians will be on site as this maintenance is performed to ensure that downtime is limited to the shortest amount of time possible.

If you are experiencing an outage during this maintenance window your services will be available in a short time as the maintenance is completed.

Thank you for your patience as this mandatory maintenance is performed.

Crucial Administration

===================================================
[COMPLETED] Scheduled Network Maintenance Window 1
Date: Friday, February 19, 2016 (02/19/2016)
Start Time: 08:00 PM EST
End Time: 01:00 AM EST (02/20/2016)
Overflow: 02/20/2016, 8PM EST - 01:00AM (02/21/2016)
Duration: 5 - 20 minutes
Hosts affected:

  • phx-1205.split-shared.com
  • phx-1206.split-shared.com
  • phx-1211.split-dedicated.com
  • phx-1213.split-dedicated.com
  • phx-1214.split-dedicated.com
  • phx-1215.split-dedicated.com
  • phx-1217.split-dedicated.com
  • phx-1219.split-dedicated.com
  • phx-1223.split-dedicated.com
  • phx-1229.split-dedicated.com
  • phx-1231.split-dedicated.com
  • phx-1235.split-dedicated.com
  • phx-1236.split-dedicated.com
  • phx-1244.split-dedicated.com
  • phx-1245.split-dedicated.com
  • phx-1246.split-dedicated.com
  • phx-1247.split-dedicated.com
  • phx-1249.split-dedicated.com
  • phx-1250.split-dedicated.com
  • phx-1251.split-dedicated.com
  • phx-1253.split-dedicated.com
  • phx-1254.split-dedicated.com
  • phx-1255.split-dedicated.com
  • phx-1259.split-dedicated.com
  • phx-1260.split-dedicated.com
  • phx-1261.split-dedicated.com
  • phx-1262.split-dedicated.com
  • phx-1264.split-dedicated.com
  • phx-1265.split-dedicated.com
  • phx-1297.split-dedicated.com
  • phx-1307.split-dedicated.com
  • phx-1325.split-dedicated.com
  • phx-1377.split-dedicated.com
  • phx-1379.split-dedicated.com
  • phx-1385.split-dedicated.com
  • phx-1406.split-dedicated.com
  • phx-1435.split-dedicated.com

===================================================

===================================================
[COMPLETED] Scheduled Network Maintenance Window 2
Date: Saturday, February 20, 2016 (02/20/2016)
Start Time: 08:00 PM EST
End Time: 01:00 AM EST (02/21/2016)
Overflow: 02/21/2016, 8PM EST - 01:00AM (02/22/2016)
Duration: 5 - 20 minutes
Hosts affected:

  • phx-1268.split-dedicated.com
  • phx-1269.split-dedicated.com
  • phx-1270.split-dedicated.com
  • phx-1272.split-dedicated.com
  • phx-1273.split-dedicated.com
  • phx-1274.split-dedicated.com
  • phx-1275.split-dedicated.com
  • phx-1276.split-dedicated.com
  • phx-1281.split-dedicated.com
  • phx-1282.split-dedicated.com
  • phx-1283.split-dedicated.com
  • phx-1284.split-dedicated.com
  • phx-1285.split-dedicated.com
  • phx-1286.split-dedicated.com
  • phx-1287.split-dedicated.com
  • phx-1288.split-dedicated.com
  • phx-1289.split-dedicated.com
  • phx-1290.split-dedicated.com
  • phx-1291.split-dedicated.com
  • phx-1293.split-dedicated.com
  • phx-1295.split-dedicated.com
  • phx-1296.split-dedicated.com
  • phx-1298.split-dedicated.com
  • phx-1300.split-dedicated.com
  • phx-1302.split-dedicated.com
  • phx-1303.split-dedicated.com
  • phx-1304.split-dedicated.com
  • phx-1305.split-dedicated.com
  • phx-1306.split-dedicated.com
  • phx-1308.split-dedicated.com
  • phx-1311.split-dedicated.com
  • phx-1312.split-dedicated.com
  • phx-1313.split-dedicated.com
  • phx-1314.split-dedicated.com
  • phx-1315.split-dedicated.com
  • phx-1333.split-dedicated.com
  • phx-1341.split-dedicated.com
  • phx-1343.split-dedicated.com
  • phx-1364.split-dedicated.com
  • phx-1400.split-dedicated.com
  • phx-1410.split-dedicated.com
  • phx-1436.split-dedicated.com

===================================================

Patch or Upgrade Your Magento Now

Magento has made new releases and patches available to improve the security and functionality of all Magento sites. While there are no confirmed attacks related to the security issues, certain vulnerabilities can potentially be exploited to access customer information or take over administrator sessions.

Click here to read more

The security issues vary across products and all versions of Magento are affected. Full articles about the Magento 1.x and Magento 2.x issues are posted in the Magento Security Center. Additionally, all new releases and a separate USPS patch support recent USPS changes.

The Magento Community Edition 2.0.1 release also contains several important functional updates, including official support for PHP7.0.2, which provides dramatic performance improvements, drastically reduces memory consumption, and supports brand-new PHP language features. These updates are detailed in the release notes.

Ransomware Strikes Magento Websites

Ransomware is one of the most insidious types of malware that one can come across. These infections will encrypt all files on the target account. Magento e-commerce sites seem to be a big target of this new website ransomware, but it is by no means limited to them.

Click here to read more

The Security Specialists at Sucuri.net have released an article on a new type of exploit that is taking special aim at vulnerable Magento eCommerce installations.

The malware was first discovered last November by Russian security company Dr.Web. Unsurprisingly, vulnerabilities in outdated software seem to be the main method of distribution. Remember web admins: Update, update update!

Ransomware Now Targeting Websites

Usually websites are used to spread PC-based ransomware; visiting the infected website will launch the payload onto the end-user machine, encrypting its contents. Earlier this year we documented a widespread campaign affecting WordPress sites that redirected visitors to exploit kits distributing ransomware. It’s also commonly distributed through tech support scams and bogus e-mails, but now we’ve seen website admins fall victim as well. Magento e-commerce sites seem to be a big target of this new website ransomware, but it is by no means limited to them.

You can read the full article here. There is no 'fix' for this one other than to restore from backup - Make sure your backup strategy is in place or check out the Crucial Enterprise Backup Product available in your cPanel.

We've launched our new website

It's been a long time coming, but we've finally launched our new website. We're excited to finally share what we've been working on with all of our visitors and customers.

Click here to read more

With our new site comes some major changes. We've launched a new affiliate system which allows our affiliates more flexibility in how they send us business. We've also launched our new custom dedicated server offering, which you can sign up for immediately. If you notice anything on our new site that doesn't work properly or you think could be improved, please don't hesitate to contact us. We'd love to hear any feedback you have to offer.