RSS

MageCoreInc Blog

Magento CE 1.9 vs Magento CE 2.0 Performance Comparison

Introduction

With our first Magento 2 article, we started investigating the newest version of the most popular eCommerce platform. The goal of Magento 2 is to provide a new , modern design and improve User Experience with many enhancements that Magento 1.x users have been asking for over time. It also provides an advanced system architecture that is designed for large deployments, better performance, scalability, and modularity.

In this article we are focusing specifically on the application performance. We evaluated Magento performance using some of the most typical scenarios of Magento deployment: average database, Amazon EC2 instance for web and database servers, and Varnish for page cache. To emulate different user scenarios and performance, we are using Gatling. In the following paragraphs, we will explain the set up and database size of specific servers, along with our testing approach and results.

This article provides a wealth of technical detail and load test results. Based on these test results detailed below we do not recommend to use Magento 2 over Magento 1, on the premise of improved performance. The test results clearly indicates that Magento 1 performs much better compared to Magento 2 when running either on a smaller or larger server with an average front-end user load and database.

We contacted the technical team of Magento and shared and discussed the results of our testing prior to publishing this blog post. Magento engineers acknowledged and were well aware of the Magento 2 application’s inferior performance and mentioned that they plan to tackle these issues in future releases.

Configuration

Hardware

  • EC2 C4.XLARGE (CPU 4 cores, RAM 8 Gb) – first test. We will call it E1 for simplicity.
  • EC2 C4.8XLARGE (CPU 36 cores, RAM 60 Gb) – final tests. We will call it E2 for simplicity.

The test was performed on two Amazon EC2 instances. Initially, we used E1, which is considered sufficient for our planned load scenario with Magento 1.x and 20 concurrent users. We were expecting to see at least the same if not better results for Magento 2.0 as well. However, during the first load test we received very poor results for Magento 2.0 installation deployed on this E1 instance. This test proved that Magento 2 requires more CPU to work effectively. As a solution we moved both testing instances (Magento CE 1.9 and Magento 2.0) to the more powerful E2 server environment.

The load test results for the smaller instance are included in this article, however the summary and the actual test was done on the larger instance.

Software

The most up-to-date software was used for both deployments.

  • OS: Amazon Linux AMI 2015.03
  • Web server: nginx/1.8.0 + php-fpm
  • PHP: 5.5.30
  • Varnish: 3.0.5
  • Redis: 3.0.5
  • Database: MySQL 5.6.27

Magento Catalog

To emulate typical Magento usage for a standard Magento site, two similar databases with the following data were created for Magento 1 and Magento 2:

  • Homepage as static CMS page
  • 2 level categories: 8 top categories and subcategories – 88 categories in total
  • 40 product attributes: 20 products with filterable attributes in layered navigation with 20-90 values each, and 20 products with no filterable attributes
  • 8 attribute sets, each using 10 filterable attributes and 10 non-filterable attributes
  • Products in each category used the same attribute set
  • 20,000 Simple products: 3% disabled, 7% out of stock, 25% having special prices, 25% having tier prices, which mirrors a typical implementation. Customer Group price was not created with the load data set, as Magento 2 does not support this feature yet
  • 5,000 configurable products with one or two configurable attributes (creating variations) and anywhere from 5 to 20 child products for each configurable. 3% configurable products were disabled and 7% were out of stock, again mirroring Magento in the wild
  • Each product has 1 to 4 images
  • All product attributes of text type (title, description, etc.) are unique and have actual content (no dummy text)

Cache Settings

Magento 2

Magento 2 Community Edition supports page cache out of the box with configured Varnish and it was used with the load test.

Magento 1

Magento Community Edition 1.x doesn’t support page cache out of the box and so we decided to use Phoenix integration with Varnish adapted and enabled by our MageCore architects. The Phoenix_VarnishCache 4.0.5 extension allows us to load all page dynamic data with AJAX calls.

Testing Methodology

Gatling software was used for load testing, allowing us to emulate different front-end user behavior. Configured user scenarios emulate actual user behavior and support building dynamic requests to a server (including AJAX calls). This is important when testing page cache performance with dynamic blocks (for example, cart summary, user compared products list, etc.).

In tests, we wanted to measure the response time from the server for the following base requests that were implemented for the project using created data sets:

  • Open Homepage
  • Open Random Category
  • Open Random Category with Layer Filter
  • Open Search result
  • Open Next Search result page
  • Open Random Simple Product
  • Add Random Simple Product to Shopping Cart
  • Open Random Grouped Product
  • Add Random Grouped Product to Shopping Cart
  • Open Random Configurable Product
  • Add Random Configurable Product to Shopping Cart
  • Open Shopping Cart
  • Checkout as Customer (Login during Checkout)
  • Checkout as New Customer (Register during Checkout)
  • Checkout as Guest

Scenarios

The load test scenarios were built with typical user behavior and 10% conversion rate (10% users created orders). The idea behind this approach is to have a group of users where each group follows some behavior. The list below is a complete list of predefined customer behavior groups that were used during the load test:

  • 40% Abandoned Cart. Scenario focused on users who drop from the eCommerce journey during the cart phase using the following steps:
    1. Open Homepage
    2. Open Random Category Page
    3. Open Random Simple product and add it to Shopping Cart
    4. Open Random Configurable product and add it to Shopping Cart
    5. Open Shopping Cart Page
  • 25% Browse only. Visitors only – users who browse products and are not planning to make a purchase. This scenario could be used in the future to more carefully test Varnish effectiveness for a busy site.
    1. Open Homepage
    2. Open Random Category Page
    3. Open Random Simple product
    4. Open Random Configurable product
  • 25% Browse with Layer. Visitors only – users who browse products using layered navigation.
    1. Open Homepage
    2. Open Random Category Page
    3. Open Filtered Category Page by single attribute
    4. Open Random Simple product
    5. Open Random Configurable product
  • 10% Checkout. Typical buyers – users who know what product they want and complete the purchase.
    1. Open Homepage
    2. Open Random Category Page
    3. Open Random Simple product and add it to Shopping Cart
    4. Open Random Configurable product and add it to Shopping Cart
    5. Open Shopping Cart Page
    6. Perform checkout as Guest
    7. Order Success page

Test Results

The load test was created for 20 users operating on a site for 10 minutes using different behavior groups as specified above.

When comparing results from the two servers, it should be noted that Magento 2 cannot run on a small server (4 CPU cores) and requires more resources to operate properly compared to what we consider sufficient for Magento 1. On a smaller server (E1), Magento 1 performs almost 5 times better in general than Magento 2. Our test included 20 users simulated using the site for 10 minutes and found that the M1 deployment can create nearly five times more site requests with better response time. Some page’s performance, for instance configurable product page or place order action, require deeper analysis in the Magento 2 code base, as the pages/actions are very slow.

Graph 1. Difference between M1 and M2 deployment performance (number of request and response time) on E1 server (More requests in <800ms is better)

The difference in performance results between M1 and M2 is not substantial when the load test was run on a larger instance with more CPU (36 CPU cores). Still, on the more powerful server (E2 with 36 CPU cores), M1 performance is better in general than M2. The difference between them is no longer as great anymore – 20 users within 10 minutes can create two times more requests to the site using M1 deployment.

Graph 2. Difference between M1 and M2 deployment performance on E2 server (More requests in <800ms is better)

Anyone willing to use Magento 2 should keep in mind that it requires more CPU resources compared to what we could use having an operational site on Magento 1. This is also very important information for smaller merchants running their eCommerce sites on shared hosting where resources are either limited or shared between different systems.

Based on the results of the load test, Magento 1 performs twice as fast when compared to Magento 2. However, there are pages where the difference in response time is much bigger and Magento 2 is very slow. For example, on product page or place order action.

Graph 3. Response time distribution over the load test time for the simple product page – M1 and M2 comparison for the large on E2 server (More requests with lower response times is better)

Instance E1

The following data provide additional details of the load test focusing on specific pages.

The table below provides details on the load test results for Magento 1 and Magento 2 deployments when installed on E1 (CPU 4 cores, RAM 8 Gb).

Graph 4. Load test rusts: Number of requests for M1 and M2 deployments on smaller E1 instance when running base scenario with 20 simultaneous operating on site for 10 minutes.

Instance E2

The table below and links to the Gatling report provide details on the load test results for Magento 1 and Magento 2 deployments when installed on E2 (CPU 36 cores, RAM 60 Gb). This server environment will be used in future analysis for any performance comparison statistics.

Gatling load test results

Gatling load test results Magento CE 1.9

Gatling load test results Magento CE 2

Graph 5. Load test rusts: Number of requests for M1 and M2 deployments on big E2 instance when running base scenario with 20 simultaneous operating on site for 10 minutes.

Graph 6. Load test rusts: Server response time for the base scenario.

Graph 7. Load test rusts: Number of requests processed within a second.

Conclusions

When compared to Magento 1, Magento 2 preformed slower for the same load test scenario and data sets. The results clearly indicates that Magento 2 must run on a more powerful server in order to operate properly or at the same level of Magento 1. Even when a proper server is selected, Magento 2 performance is still lagging behind when compared to Magento 1. The need for additional CPU resources will increase server hosting expenses by 7X as Magento 1 performance on C4.XLARGE ($0.239/h) is almost the same as Magento 2 performance on C4.8XLARGE ($1.675/h).

To be clear and fair to Magento 2, we are comparing a fine-tuned machine (Magento 1, with all possible performance improvements created over time) with a new product that has a larger and more complex code base to support modularity and security. We are confident that Magento 2 will improve over time.

In upcoming articles, we will explore why it is that Magento 2 performs slower for specific pages (configurable product page or place order action) as well as what can be done to improve it. This will include topics such as server configuration and code analysis.

About the Author

As the former lead Architect of the Magento project, Dima was responsible for planning and executing technical delivery for all the Magento Inc. products and making sure Magento delivers high preforming scalable product to its customers. In addition, Dima’s leadership and involvement in countless complex implementation projects made him one of the most authoritative and respected figures in the Magento community.

Following his successful career at Magento, Dima joined forces with Yoav Kutner and Jary Carter to found Oro Inc. As the CTO of Oro Inc. and its service arm MageCore, Dima is leading the development and technical execution of highly scalable open-source business applications and continues to leads complex Magento implementations projects.

5 comments
  • ivanchepurnyi

    Dmitry, can you please place a link to repository with benchmark code, both Magento instance code bases and databases, so it will be possible to validate and give constructive feedback on it?

      • ivanchepurnyi

        Hi Dima,

        There is a problem with test scenarios that does not make it possible to replicate Magento 1.x behaviour that was described in your Gatling simulations.

        I cannot find anywhere /ajax/ route, either in Magento 1.9.2.4 core nor Phoenix Varnish Cache module (https://github.com/PHOENIX-MEDIA/Magento-PageCache-powered-by-Varnish), even 4.0.5 version of the extension does not have it. Can you please clarify, where are those routes coming from?

        As well, your test case does not repeat a core Magento 1.x behavior, when you are redirected back to a shopping cart page or a product page that you have visited before, after adding a product to cart.

        Would be so nice from your side to publish full Magento 1.x code base that you’ve used for load tests, as with the current subset of data, it does not make possible even to run Magento 1.x load test with same dataset and simulation.

        Thank you for your co-operation!

sso for oroinc.comsso for oroinc.desso for oroinc.fr