Author | Bas Schouten
One of the biggest challenges with any software is determining the impact of changes on the user experience. Whether it’s the processing speed of video editing software or the smoothness of the browsing experience, testing in a well-controlled experimental environment can only tell us limited information. While local experiments can provide many metrics, improvements in these metrics may not necessarily lead to a better user experience.
This is especially challenging for complex client software running third-party code, such as Firefox. This is one of the main reasons we 're working on Speedometer 3 alongside other browsers . Our goal is to build performance tests that simulate real user experiences so browsers have better tools to deliver improvements for real users on real websites. Of course, we can see improvements in various test metrics that Firefox has achieved through this work this year, but what we really care about is how much benefit these improvements can bring to our users.
In order to measure user experience, Firefox collects and anonymously records various browser performance-related parameters such as page loading, responsiveness, startup, etc. Collecting data while maintaining the highest levels of privacy standards is also a challenge. For example, because we rely on aggregated metrics, we cannot track data from any specific website. But the bigger challenge may be analyzing the data collected and drawing actionable conclusions. We’ll cover these challenges and our solutions in more detail in the future, but in this post we wanted to share how some of the fundamental metrics that reflect a user’s browser experience have improved this year.
We start with page loading. First Contentful Paint (FCP) onloadreflects performance experience better than events. We track the time from receipt of the first network byte to FCP. This tells us how quickly the browser provides feedback on a successfully loaded page, a key metric for understanding user experience. While this largely depends on the web page itself, we'd expect this number to drop if overall browser performance improves.
Figure 1: Median time (milliseconds) from response start to first content draw
We can see that this time has improved by close to 15% from about 250 milliseconds at the beginning of the year to 215 milliseconds in October. This means that users are reporting that pages load nearly 15% faster than at the beginning of the year. Importantly, this is entirely the result of optimization efforts that are not explicitly targeted at page loading.
To understand where this improvement comes from, let’s look at another piece of timing data: the time spent executing JavaScript code during page load. Here we’ll look at the 95th percentile, which represents the pages that rely the most on JS and highlights our chances of reducing user wait time.
Figure 2: 95th percentile time (milliseconds) to execute JS during page load
This shows a drop from around 1560 milliseconds at the beginning of the year to 1260 milliseconds in October. This represents a significant improvement of about 300 milliseconds or nearly 20%, and is likely the main reason for the decrease in FCP times. This makes sense because Speedometer 3 works with a lot of optimizations for the SpiderMonkey JavaScript engine (that's a story for another article).
We also want to know how responsive the page is once it loads. For example, how smoothly I type while writing this blog post. The main metric we collect here is "key feedback latency": the time between pressing the keyboard and the result being rendered on the screen. Rendering some text to the screen may seem simple, but there's actually a lot of work behind the scenes, especially when the web page responds to typing events via main-thread JavaScript. Most typing speeds are mainly limited by hardware (such as the refresh rate of the monitor), but poor performance can greatly affect the user experience. This means we need to reduce the worst case scenario, so we'll still look at the 95th percentile.
Figure 3: 95th percentile of key feedback latency
Again we see clear improvements. The 95th percentile fluctuated around 65 milliseconds most of the time, dropping to below 59 milliseconds after the release of Firefox 116 and 117 in August. The feedback of the slowest keys has been improved by 10%, which means users will get a smoother feedback experience and less interference when typing.
We are excited about the progress we are seeing from our telemetry data. I believe that this year’s work will have a real positive impact on Firefox users. We have more optimization measures under development and will share details in the future.
IntelliJ IDEA 2023.3 & JetBrains Family Bucket annual major version update new concept "defensive programming": make yourself a stable job GitHub.com runs more than 1,200 MySQL hosts, how to seamlessly upgrade to 8.0? Stephen Chow's Web3 team will launch an independent App next month. Will Firefox be eliminated? Visual Studio Code 1.85 released, floating window Yu Chengdong: Huawei will launch disruptive products next year and rewrite the history of the industry. The US CISA recommends abandoning C/C++ to eliminate memory security vulnerabilities. TIOBE December: C# is expected to become the programming language of the year. A paper written by Lei Jun 30 years ago : "Principle and Design of Computer Virus Determination Expert System"