Google's Search, Development and Enterprise Discussions

Google on Ulitzer

Subscribe to Google on Ulitzer: eMailAlertsEmail Alerts newslettersWeekly Newsletters
Get Google on Ulitzer: homepageHomepage mobileMobile rssRSS facebookFacebook twitterTwitter linkedinLinkedIn


Google Authors: Kevin Jackson, APM Blog, Ben Bradley, Progress Blog, OnPage Blog

Related Topics: Online Shopping, Marketing and Sales, Google

Blog Feed Post

2017 Cyber Week Coverage / Black Friday and Cyber Monday Retail Digital Performance


It’s that time of year again, Cyber Week 2017.  Black Friday and Cyber Monday are a few days away and Retailers have been scrambling for months to make sure everything goes right this week.  We will be providing this live blog like we did in 2016, 2015 and 2014.  Through the course of this week we will be looking at which retailers are providing the best performance, some of the techniques and best practices they use and the challenges they face.

Table of Contents:

Fastest Visually Complete Retail Sites: Friday, Nov 24th, 5:00pm EST

This BlackFriday is almost in the books now, so let’s have a look at the past week and which retailers were providing the fastest Visually Complete results for online shoppers.

Apple, Staples and Wayfair provided the fastest Visually Complete times through CyberWeek.  In the chart above I include a sample pf other retailers just to compare.  Even though there is a level playing field with the internet, the decisions that are made by IT will absolutely impact shoppers.

Trying to understand what works and what doesn’t can be a daunting task.  After working with hundreds of retail customers, Dynatrace understands this and has built a team which can work with retailers and non-retailers alike.  If you want to know more about this practice and have them do even more in-depth comparisons like this reach out to the Dynatrace Insights team.

If you are more hands on, and want to get access to the type of data that I’ve shown through out this weeks coverage, here is a Free Trial of Dynatrace that you can use yourself.

How’s Your Client/Browser Monitoring Doing?: Friday, Nov 24th, 4:15pm EST

Over the past week we’ve definitely seen an increase in errors across the retailers we track as part of the Dynatrace Benchmarking Practice.  These errors have been on the network side, server side and on the client side.

Earlier today I showed an example of how Dynatrace captured a server side issue impacting end users for a retailer and pointed to the root cause of the issue.  Now let’s show you an example of a client side error.   By client side error we mean error within the web or mobile web browser (note this could just as easily be the errors being seen in native apps as well).

Below is a map of the US for an online retailer showing the geographic impact of JavaScript for the past 24 hours (notice the increase today).  In this example, we are seeing more JavaScript errors for shoppers in California.

Dynatrace automatically detects these errors and determines their impact on shoppers.  Below we can see that 244 user actions/minute were being impacted for around 37 minutes.

Not only does Dynatrace determine the number of shoppers being impacted but it would also show the root cause of the issue, pointing to who was responsible for delivering the JavaScript and what user actions and browser versions were being impacted.  Dynatrace takes the details further providing a stack trace of the issue as well (not shown in the example below which is not from the above retailer).

These types of regional JavaScript errors are not uncommon (sorry about the double negative). In many cases some retailers allow third parties to drop JavaScripts onto their pages.  In these cases you can see regionally focused ads cause local regional problems.

BlackFriday Afternoon Visually Complete Update: Friday, Nov 24th, 1:30pm EST

Here is an update for the past 6 hours showing that Apple has the fastest Visually Complete time today, with REI and Staples coming in second and third.  I’ve pulled out a top 10 (of 50) list of retailers with the fastest Visually Complete time below, and included a number of the slower retailers for comparison.

Solving Complex Problems in Retail Applications: Friday, Nov 24th, 11:15am EST

Here is a unique problem detected by Dynatrace Real User Monitoring.  This retailer experienced an issue with increased traffic coming from Asia.

Identifying increased traffic is useful to understand if a campaign it successful or if you are under attack by a bot network.

In this case Dynatrace identified where the traffic was coming from and indicated that it was impacting several services for the retailer.  One of these services is a PHP service and for the increased traffic around 5-6% of the requests were failing, including requests to shopping cart functions.

We’ve talked at length about how complex these retail applications can be, and in this example, we see a retail application based on heterogeneous technologies running in a cloud environment.   The SmartScape below shows just how complex these retail applications can be.

More updates coming.

The Global View: Friday, Nov 24th, 10:00am EST

Dynatrace provides a benchmarking service where we actively track the performance for leading companies in specific industry verticals.  One of these is of course retail.   We do this from across the globe, testing local retailers from locations within their local markets.

Below is a view of the local market average for Visually Complete from Australia, Benelux, China, France, Germany, Japan, United Kingdom and the United States.

Japan, on average provides the fastest results for shoppers, whereas China provides some of the slowest experiences for online shoppers.

Black Friday Horror Stories: Friday, Nov 24th, 9:15am EST

Most retailers are performing well this Cyber Week and we will do some more performance comparisons today, but I want to share some macabre horror stories of issues we’ve seen so far.   We are see an interesting traffic trend early in the evening on Thanksgiving.  In the past people would just slip into a turkey induced food coma to charge up for their in store shopping adventures, but now we are seeing full fledge tryptophan induced online shopping frenzies.  During these increased traffic periods, some retailers have fallen victim to issues that most likely were not part of their test cases leading up to Cyber Week.

In this example below a social media widget on the page is preventing it from loading properly.  When we looked at the detail we could see it impacting the Visually Complete time which would make the page appear to load slowly for shoppers.

In this example below for a different retailer, we see network issues causing delays in the initial connection from the browser to the retailers site.  This would frustrate shoppers as they likely would not see any activity on the page when they try to load it.

In this example below a third party component had an issue negotiating SSL (secure) connection. Because this third party was delivery a javascript it could impact the load time for the rest of the page.

In this example below, a CDN (Content Delivery Network) routing table was clearly mishandled.  This issue would have resulted in failed page loads for shoppers.

But even when everything seems to be performing right we are still seeing issues like this one below.   These are the issues which more often than not go unnoticed unless you were closely monitoring your conversion counts in real time.

Next update will include some more performance comparisons.

Shopping with Mobile Devices: Thursday, Nov 23rd, 4:30pm EST

Last year we saw some interesting patterns where successful conversions switched from being predominantly on desktop machines earlier during Cyber Week to mobile devices later in the week.   This year we are seeing something different related to conversions from mobile devices.

Below is a view of real user devices from Dynatrace for a US based retailer.   Dynatrace can track desktop, mobile, tablet, synthetic and bot traffic.

Below is how Dynatrace can track specific versions of browsers running on these devices.  This is important because specific versions of these browser and devices can provide different performance results and execute JavaScript using different engines.  Even the smallest difference can impact shoppers.

This data below is real user conversion count from Dynatrace UEM.  This from a different retailer than the one above.   Compared to last year, we see that most of the successful conversions are being done on Safari Mobile browsers for the entire week.   This is definitely a shift towards mobile being the primary mechanism that people are using to purchase during Cyber Week (at least for this retailer).

One thing we noticed which should be mentioned.   Be careful if you are using rates to monitor your conversion.   Using a rate metric over time can provide misleading indication as to what is happening with your conversions.  If you are monitoring conversions in real time it is best to use a count measure as opposed to a rate. When there is less traffic you sample size decreases which can impact the rate metric.  Rate metrics are useful if you are summarizing activity for a longer duration.

More updates on the way.

Understanding How Performance and Errors Impact Bounce: Thursday, Nov 23rd, 12:00am EST

Here is our first comparison for Cyberweek 2017.  Apple, Neiman Marcus and Amazon are providing some of the fastest Visually Complete times for shoppers.  When compared to some of the slower retailers these fastest retailers are offering more than 4X faster experience for shoppers.

Let’s have a look at some real user traffic data from a major retailer.  In this example, we see that performance across the US is consistent for the past 6 hours.  Unfortunately, it is currently consistently poor according to the APDEX score.

Now for this retailer this is actually good ”-ish” news (I am a glass half full guy) because if we look at their APDEXfor their entry actions (where shoppers enter the site) from the previous week (below) we can see it improved from unacceptable to poor over the past 6 hours.

We can see that the past week compared to the previous week saw other improvements.   The week before last there was a major issue with JavaScript errors.  JavaScript errors occur in the browser can be just as impactful as server side errors to shoppers.

For example, when we look at these errors we can see a direct causation that users were “Bouncing” off the site when they encountered these JavaScript errors.   We know this because Dynatrace sees the user behavior for each shoppers user session.

JavaScript errors impact bounce rate, but there are other things which can impact bounce rates.  Below is a view showing a histogram comparing the performance impact on the bounce rate for this week to the previous week.   We can see in this example that things are headed in the right direction with lower bounce rates as performance improves.

We will provide some more comparisons and real user examples over the next few days.

You Have to be Monitoring from an End User Perspective: Thursday, Nov 23rd, 10:30am EST

Happy Thanksgiving morning everyone. This is an interesting one. Here we see a retailer having ongoing issues and a different issue at the same time of day for the past week.

When we analyze this, we see that during the daily issues there are an excessive number of redirects which are happening (below).   This could be caused by a misconfigured network device or an issue with the Content Delivery Network (CDN).

This is a real world issue that is currently impacting shoppers trying to use this retail site.  This illustrates that when things go wrong there can be multiple unrelated event some of which might be impacting end users, and others which are not.

That’s where automatically analyzing your traffic using Artificial Intelligence can quickly help you determine which problem needs to be addressed first.  It can tell you how many users are being impacted.  This allows retailers to address the issues that are most problematic.

Performance Metrics, Which Ones are You Using?: Wednesday, Nov 22nd, 6:00pm EST

As everyone checks out of work and starts their holiday travels let’s have a look at some web performance metrics we will be tracking this Black Friday and Cyber Monday.  I am often asked what is the industry standard for web performance metrics?  I used to answer this that the wonderful thing about standards in the IT industry is that there are so many to choose from.  For years, we have searched for the right metrics to best monitor user experience.

User experience is driven by a combination of design, architecture and delivery.  Retailers, regardless of their size, sales or market cap all play on the web on an equal footing.  How a retailer chooses to implement their design, architecture and delivery is a choice that they make.  I highlight this as these are things retailers can control and that is why we provide these benchmark comparisons.

For example, below is a snapshot of some live data from the last five days for a retailer who has completely optimized their user experience.

The metrics I am looking at are things like:

W3C Request/First Byte time, this is a measure of how fast the servers can respond to a shoppers request.  Optimizing this measure not only provides the end user with a faster experience it means that this retailer can actually scale to support more shoppers with less cost because transaction are running faster, consuming fewer server-side resources (like CPU and memory).

First Paint time, this is a measure telling us when something is being painted to the screen.  This helps because shoppers want to perceive activity immediately…  waiting with a blank screen leads to frustration which can lead to increased rates of bounce (abandonment).

Dom Interactive time, this is a measure which approximated when a user can start interacting with a page.  Not being able to click on an item as the page loads leads to shopper frustration which can lead to undesired user behavior.

Speed Index, this is a newer measure which tells us how fast it takes it to load the majority of the viewable window.  Shoppers want to see activity when they load a page, the faster this metric is the better the user experience will be from a performance perspective.

Visually Complete, I’ve talked about this at length in this blog, this is the best metric available which gives retailers an understanding as to what shoppers are experiencing when they load pages.

Dom Complete time, this measure tells a retailer when all the dynamic logic for that page has loaded for a shopper.   If the retailer is using lots of client side logic through Java Script then they need to pay attention to this metric.

Page Load time, this is when the shopper’s browser/client considers the page to have loaded.  This is how most organizations have historically gauged web performance.

Response time, this is the complete “over the wire” end to end response time including any asynchronous activity and beacons which might called.  This measure is useful for retailers operations teams to understand if specific IT assets are performing and meeting SLAs (Service Level Agreements).

Below is an example showing the same metrics but for a retailer exhibiting slower performance.  To put things in perspective, the fast retailers entire Response time (meaning absolutely everything) is done loading before anything even paints to the screen for the slow retailer.

Don’t believe me? Don’t think that online retail is competitive when it comes to web performance?  Have a look at this chart below showing the response times for different retailers.   Given a preference which site would you want to visit, a slower or a faster site?

We will start with some more detailed comparisons tomorrow leading into Black Friday.  For now, safe travels and Happy Thanksgiving!

Conversion is the Name of the Game: Wednesday, Nov 22nd, 10:00am EST

Over the next few days what every Retailer is looking to do is “convert” shoppers into customers.  Tracking conversion is critical for Retailers to understand their business success.   Typically, this is done through online Analytics tools like Google Analytics or Omniture.  The challenge with these tools is that they are often page focused and the data which they provide is rarely real time…  unless you pay extra for it.

Modern applications, specifically what are known as single page applications can introduce challenges in defining meaningful conversion goals.  Below is an example of how Dynatrace can define conversion goals on any user action, including XHR actions from click events, like “Click on Proceed to checkout“.

This is a much more powerful and easy way to define very granular conversion goals.

Dynatrace allows retailers to capture individual user sessions.  It can uniquely identify which retail applications a shopper visits, where they are visiting from and what type of device they are using.  Tracking a shopper moving from a mobile device to a desktop to complete a purchase allows retailers to provide more targeted offerings.  Below is an view showing how Dynatrace would capture a unique shopper visiting Amazon.com.

Dynatrace enables a retailer to see the details of an individual session including things like their screen size (which, by the way, can impact end user experience which is why retailer should capture Visually Complete times), Device details, ISP and the number page and xhr actions which are observed during the shoppers visit.

Below we can see each specific user action that a shopper does during their visit to the retailers site.  You can see that the XHR action “Click on Proceed to checkout” was marked as a conversion goal within this shopper’s visit.

Since the conversion goal is a critical step for a retailer to track, below is a view from Dynatrace showing how that XHR request was executed.   Unlike web analytic tools, Dynatrace can automatically trace this transaction all the way back to and through the server side including back into containers and the cloud.

Dynatrace can provide this level of detail on every user session.  When a Retailer applies this to thousands or tens of thousands of user sessions they will need help to make the most of this data.  That’s where Artificial Intelligence comes in.  It automatically analyzes the user sessions to determine if a performance event is impacting shoppers.  Below is a view showing an interactive infographic which is determining that shopper from a specific region using Safari mobile are being impacted by an increase in Java Script errors.

Dynatrace analyzes those user sessions to see if the performance problem impacting shopper’s behavior.  If it is increasing the bounce rate, specific entry/exit actions and of course conversion goals.

Overall conversion goals are automatically tracked and baselined by Dynatrace.   This allows you to compare your current conversion results to historical results.

And of course specific conversion goals can be managed individually.

Retailers need to understand how user experience and performance impacts user behavior and conversion.  Retailers can look at undertaking time consuming ELK (Elastic, Logstash and Kibana) projects to marry disparate data sets from their performance tooling and analytics tooling, but this always proves difficult and expensive.  Dynatrace provides an all in one solution to do this automatically for retailers.

Visually Complete, What You Need to Know: Tuesday, Nov 21st, 8:00pm EST

I mentioned that this year we will be doing more comparisons based on a new metric called Visually Complete.  This is important because it is a better representation of what shopper experience when the view retail web pages.  Below is a visualization showing how Visually Complete uses an algorithm to identify when the browser finishes displaying the viewable portion of the screen.  This is the “above the fold” view, there may be content which is being loaded “below the fold” that a shopper would scroll down to see, but that is not included in the Visually Complete time.

Dynatrace uses this in both our Synthetic and Real User measurements.  Below is a view of a synthetic measurement in Dynatrace showing the visible screen which was measured.

When we look at the details of how the page loads, Dynatrace analyzes a rich set of metrics based on network observations and client side observations.  These navigation timing milestones are shown in the context of how the page loads.  Depending on how the retail site has been architected these milestones can be faster or slower from one retailer to another.

These are decisions that retailers have to make.  Do we want to focus on displaying ads first or do we want the user to be able to interact with options on the page before certain ads or images have loaded?  These milestones tell us when items are first visible, when the user can interact with page, when the page appears to be complete and of course when the page is fully loaded.

Some Retailer’s, like Apple have realized that Visually Complete is a metric that needs to be managed.  Below we can see how someone at Apple decided that they needed to optimize their site before Black Friday and Cyber Monday to make sure that all customers are getting a consistent and quick Visually Complete experience.   We can see from the data below that early in the morning of the 17th Apple pushed a build out that dramatically improved the consistency of their page as measured by our Visually Complete metric.

Visually Complete is gaining traction in the Retail Industry, if you are not managing to it now, you likely will be soon.

Pre Black Friday Carnage: Tuesday, Nov 21st, 9:45am EST

It is starting off as an interesting Cyber Week.   Again, it is better to find this out now than on Black Friday or Cyber Monday.  Below is a live view of retail transaction which is now failing.

It looks like this retailer has tried to push several builds out into production unsuccessfully in the past 24 hours.

A quick analysis (below) shows that the issue is not a third party or network problem.  The issue is not isolated to a particular geography or device type, it is pretty wide spread.  One interesting thing to note, while the response time looks like it is improving, this is actually a false indicator.  In this case as the Retailer pushes out an automated error message the measured response changes as the error page loads faster than the normal page.

Looking at the details, we see a 503 error.  This is a server side error which tell us that something on the backend is causing an issue.

Having shoppers receive a 503 error is clearly a bad thing, but having that message show up for hours is a REALLY bad thing.

I’ve talked in previous years (see last year’s blog) about how complex these retail applications have become.   It has gotten to the point that these applications cannot be managed unassisted by humans.  We need help.   That is where Artificial Intelligence comes.  Retailers need to embrace AI to automatically uncover issues like this, and automatically diagnose the root cause of a problem.

Above is a view from Dynatrace showing a problem which is impacting customers.  Dynatrace’s AI shows how many customers and which applications are being impacted by a problem and what the root cause of the problem is.   Fixing an issue like what we saw from that retailer would take minutes instead of hours.

We will continue to post findings as we see them, but this Retailer’s problem are not unique we are seeing multiple retailers slipping days before Black Friday and Cyber Monday.

Cyber Week Kickoff: Monday, Nov 20th, 4:30pm EST

It’s that time of year again, Cyber Week 2017.  Black Friday and Cyber Monday are a few days away and Retailers have been scrambling for months to make sure everything goes right this week.  We will be providing this live blog like we did in 2016, 2015 and 2014.  Through the course of this week we will be looking at which retailers are providing the best performance, some of the techniques and best practices they use and the challenges they face.

This year I am super excited to be doing some comparison based on new metrics like Visually Complete.

Visually Complete

Businesses and IT need reliable benchmarks to know what a good customer experience looks like. Until now, it’s been difficult to see and capture the visual experience of a real user. With Visually complete and Speed index, Dynatrace provides metrics that are as close to a real user experience as it can get.

Traditional website performance optimization metrics are based on measured time, number of requests and kilobytes. While these numbers are important, they do not reflect on when users are able to see the information they are looking for. Users care more about the time it takes a website to become visually loaded and ready to use.

Visually complete shows you the exact time your real users spent waiting for above-the-fold content to load completely – for each and every location, device, operating system and browser type. It can also be combined with powerful waterfall charts to understand and tune the rendering of a web page. Dynatrace is the only solution that provides these consistent metrics for real user monitoring as well as synthetic tests.

It will be interesting to see how Retailers compare using this new metric.

To set the stage for this week’s coverage, let’s start with a view of a major retail event that happened before Cyber Week.

Singles Day 2017

Singles Day (11/11/17) is now the biggest single day shopping event of the year.   According to Forbes, this year’s Singles Day took in $17.8 billion in gross sales.   That’s $740 million per hour or $12.3 million per minute, and yup that’s $205k per second.  I have to take a pause and say…  wow.  That’s a lot of money, and that is a lot of sales at risk for Retailers if they don’t get it right.

Below is a view of the digital performance from some of the main Retailer’s in China.  Note how one of the retailers had some very poor performance During singles Day (yellow line), which was likely due to an issue on the server side.

The view on the top left tile shows the Visually Complete times for each of the retailers leading into, during and after Singles Day.  The average Visually Complete time for China Retailers is 8.92 seconds.  The tile on bottom left is Server Time.  We measure this through the time to first byte (which tells us how long it takes the server to respond to a request) on the root or base page.  The industry average for China retailers is 257 milliseconds on the sever side.

What we see is that Retailers in China are focusing a lot of effort on optimizing the server side of the equation, but are they doing this at the expense of overall digital experience? With an average Visually Complete time of over 8 seconds, shoppers in China are getting some of the worse digital experience when compared to other markets.

I see this as an opportunity for retailers in China.  Future Singles Day will see increasing competition and the main way Retailers can differentiate themselves will be by improving digital experience.

Cyber Week 2017 Kickoff

When it comes to Cyber Week, it is way better to fail early than to fail later during the week.   Here is an example of a retailer that clearly was looking to push a new build out ahead of Cyber Week, only to have to pull that build back relatively quickly.  Below we can see around 4am this retailer pushed a new build for Cyber Week into production.  Immediately Dynatrace saw issues from an end user perspective that was impacting a typical retail transaction flow.

When we look at the details around this failure we see Server side response times slowing down and throwing errors.  Modern web applications are exceedingly complex and any number of things can cause an issue which can impact end users.  It is better to find out issues like this well ahead of time, however sometimes issues can occur at most inopportune times. The faster you can find and fix these issues, the less the impact will be on your bottom line.

This year we have new metrics and will of course be using more Dynatrace AI (Artificial Intelligence) to monitor key retailers.

We will be here all week, check back often to see updates on 2017 retailer web performance.  Also feel free to follow my updates on twitter @davidlewsjones or on Linkedin

The post 2017 Cyber Week Coverage / Black Friday and Cyber Monday Retail Digital Performance appeared first on Dynatrace blog – monitoring redefined.

Read the original blog entry...

More Stories By APM Blog

APM: It’s all about application performance, scalability, and architecture: best practices, lifecycle and DevOps, mobile and web, enterprise, user experience