Over the last two years, as we have gathered millions of records of Winshuttle usage data – often times people ask about the validity of the data. For example, how does one become confident in the information put forth as it relates to the time savings figures outlined in the Winshuttle Business Value Assessment (BVA) results?
So we consulted Jeff Sauro – www.measuringusability.com– who is a practitioner of software usability statistics and author of a variety of papers and books on this topic in order to seek his help in answering this important question. The results from his initial analysis of the data were very helpful – so in this blog posting we will walk through this process using his recommended analyses and an example set of time savings from actual customer usage of Winshuttle to optimize ME21N.
First of all, note the following table highlighting a set of Winshuttle transaction data based on users of Winshuttle who have optimized ME21N (Create Purchase Order) and note the variance in time savings, records processed, etc.
This variability is what causes questions to arise as to the validity of the time savings or records processed. Leveraging Sauro’s techniques, let’s see how we can obtain a confidence factor for the data represented. So the first step in the statistical analysis is to do a distribution plot and then subsequently a probability plot of the seconds saved per ME21N record processed:
In this sample set, we have 26 observations with a median value of 142 seconds or 1 minute 22 seconds for this set of ME21N records. Now let’s take a look at the probability plot to ensure our data is positively skewed and within a normal range of the trend lines.
Interesting to note when we plot this with a trend line – you can visualize the two large outliers… This is a great example of why we want to use a median calculation for the time saved as opposed to using an average (imagine how the average would be so much higher than the median). Another way to interpret this is that it shows that the median is best for describing the center of data when you have this positive skew.
Now let’s take a closer look at the confidence interval. In this case, the 95% confidence interval around the median value of 142 seconds is 120 to 187 seconds per ME21N (Create Purchase Order). This confidence interval means if you were to measure all users doing this procedure, you could expect the time savings to be between 120 and 187 seconds. Or put another way you can say we’re 95% confident you’ll save at least 120 seconds or 2 minutes by using Winshuttle to optimize each of your ME21N transactions. But how did we come to this “confidence interval”?
Sauro highlights the confidence interval calculation in his book (Practical Statistics for User Research) and using this example it works as follows:
Where, n is the sample size (in this case 26); p is the percentile expressed as a proportion (.5 for the median);
is the critical value from the normal distribution (1.96 for a 95% confidence level); and
is the Standard Error. The results of the equation are rounded up to the next integer and the boundary of the confidence interval is between the two values in the ordered dataset or in this case position 8 is the low confidence level while position 18 is the high confidence level.
Now looking at the following ordered data set of our ME21N transaction saving times we can see that we have 95% confidence that our savings will be at least 120 seconds. Note the yellow highlighted position information and the related seconds saved for that position.
With this procedure in hand, we can now leverage the millions of records of data that have been collected and provide an even higher level of trust and integrity in the information that we put forth in our business value assessments. Stay tuned to this blog where I will be providing a confidence interval summary for over 750 SAP transactions…
To learn more about practical usability statistics visit www.measuringusability.com.
Questions or comments about this article?
Tweet @Winshuttle to continue the conversation!