Inside PCWorld: The Rigorous Testing Process for Laptops

Since 1983, PCWorld has been testing PCs, and while we made the jump from paper to digital years ago, our mission remains the same: We’re here to help you make better choices about your PC hardware purchases. Most importantly, this mission extends to our laptop coverage, which we handle with utmost care. If you’re wondering how we evaluate laptops and Chromebooks, buckle up for an in-depth explanation of the testing process. To determine performance, we use benchmarking software like PCMark, 3DMark, and Cinebench. These tests, which we’ll cover in more depth below, are designed to give the CPU and GPU a sweat-inducing workout. We also evaluate more subjective matters like the laptop’s build quality, its typing experience, audio quality, available ports and more. Real-world testing is central to the PCWorld method. After the testing process, we rate the laptop on a 0 to 5 scale, and determine whether the machine deserves an Editor’s Choice award. Looking to pick up a brand new laptop? Check out PCWorld’s roundup of the best laptops available right now. How we test Windows laptops IDG / Matthew Smith All of the Windows laptops we review are put under a series of stress tests. PCMark 10 measures CPU performance while the Handbrake video encoder test gauges CPU performance over time as well as indicating how well the cooling system handles throttling. Whether the Windows laptop is a budget machine or a more premium model, we run the same tests regardless of price. That said, to keep things fair, we don’t dock the final score of budget laptops for having weaker performance than more premium machines. We evaluate budget laptops based on what they’re designed for: to provide value for more simple computing tasks. Now, here are more details on our benchmarks: PCMark 10: We run PCMark 10 to determine a laptop’s general performance. It runs workloads like web browsing, video chat, and light spreadsheet use. If the score is north of 5,000, then it’s a good laptop for productivity. Cinebench R20: This benchmark is designed to run all of the cores of a laptop’s CPU. It does this by rendering a complex 3D scene in a short amount of time. The end result reflects how well the laptop handles processor-intensive multi-threaded workloads. Handbrake 0.9.9 Encode: Handbrake is a multithreaded, long-duration test. We use it to encode a 32GB 1080p MKV file with the Android Tablet option switched on. The idea is to see how much performance drops when a laptop is under a lengthy computer load. The more CPU cores you throw at this benchmark, the faster it will run. This is a timed test, so lower scores are better. 3DMark Time Spy: We use the 3DMark Time Spy benchmark to determine a PC’s general graphics performance. It’s more demanding than 3DMark’s other benchmarks, making it a suitable test for more premium laptops. Time Spy’s graphics tests are rendered at 2560×1440 resolution and we use overall score. Higher numbers are better than lower ones. Battery life: To test the battery, we loop a 4K file of the short film Tears of Steel until the laptop dies. We run this test twice with the brightness level set at 250 to 260 nits. We also make sure to unplug any USB drives, as they consume power. Battery life can be impacted by a multitude of variables such as screen size, display type, battery size, and so on. A high resolution touch display will drain the battery much faster than a non-touch display, for example. How we test gaming laptops Gaming laptops run through the same series of tests as the Windows machines above. However, in addition to those benchmarks, we also run the built-in benchmarks in Shadow of the Tomb Raider and Metro Exodus: Enhanced Edition. While these titles are older, they’re still valuable because they offer stable performance. Neither the developers nor graphics card makers are providing performance updates for these titles anymore, making it easy to compare a newer laptop’s performance against older ones. In the future, we intend on adding more gaming benchmarks to our testing regimen. Shadow of the Tomb Raider: The Shadow of the Tomb Raider benchmark is built into the game and is super easy to run. We run this benchmark at the Highest graphics setting at 1080p. When the test wraps up, we record the average frame rate. The higher the frame rate, the better the gaming performance. This test can optionally show you how well the laptop handles ray tracing, which makes lighting effects look more realistic. Metro Exodus: Enhanced Edition: The Metro Exodus: Enhanced Edition benchmark is more demanding. We run this one at the Extreme graphics detail at 1080p. Again, the higher the frame rate, the better the performance. Like the Shadow of the Tomb Raider benchmark, we also record the average frame rate. This benchmark can also test ray-tracing effects as well, if desired. How we test Chromebooks IDG / Matthew Smith Chromebooks aren’t designed to run the benchmarks covered above, which means we have to turn to web-based benchmarks to gauge performance. CrXPRT 2’s performance benchmark runs through six workloads total and is designed to see how the Chromebook handles day-to-day tasks. We also use CrXPRT 2’s battery benchmark, which loops videos and various tasks until the laptop dies. Below you’ll find the specifics of each web-based test. CrXPRT 2: The CrXPRT 2 performance benchmark measures the speed of the Chromebook using six different workloads. In other words, it determines how well the Chromebook handles day-to-day tasks like watching movies, editing images, and so on. It’s a good baseline for simpler tasks, which is what Chromebooks are specifically designed for. The higher the number, the better. Speedometer 2.0: Speedometer is a browser benchmark that gauges the responsiveness of web applications, which is pretty darn important for Chromebooks, as they’re entirely cloud-based. It evaluates the responsiveness by adding items to a to-do list over and over again. Basemark Web 3.0: Basemark Web 3.0 gauges how well the Chromebook can use web-based applications. This benchmark includes a variety of system and graphics tests to determine this. To get the best results possible, we first close all open tabs. The test itself then only takes a few minutes from start to completion. Similar to the CrXPRT 2 performance test, higher numbers are more desirable. Kraken 1.1: The Kraken 1.1 benchmark is a creation of Mozilla’s. Essentially, it’s a JavaScript performance benchmark that, according to Mozilla, measures the speed of several different test cases that have been extracted from real-world applications and libraries. The results are then recorded in milliseconds, so the lower number is the one to aim for in this specific case. Jetstream 2: The Jetstream 2 benchmark runs various advanced workloads as well as programming techniques, effectively combining both JavaScript and WebAssembly tests. Essentially, the Chromebook gets the “gold star” for booting up quickly and executing code quickly. CrXPRT 2 battery test: Not only is CrXPRT 2 a performance test, but it’s also a battery test! First, we make sure to disable the power settings, so the Chromebook doesn’t go to sleep on us in the middle of the battery test. We also lower the brightness to about 50 percent hoping to emulate realistic day-to-day use. As for the test itself, it continuously loops videos and various tasks until the laptop runs out of juice. Subjective, real-world testing Subjective evaluations are just as important as the raw data we receive from the software benchmarks. It’s important to know how typing on a specific laptop keyboard feels, or what the audio sounds like when it’s cranked up all the way. That’s the real meat and potatoes, folks. We ask all our laptop testers to spend a good amount of time with each machine, and make observations about what the laptop is actually like to live with. Design: Sometimes design boils down to aesthetics and personal preference. Maybe you love the color scheme or the texture of a laptop’s lid. Some laptops are just boring gray slabs, which is entirely valid to call out. But more importantly, a laptop’s weight, shape, and overall “feel” can be evaluated against other machines in the same category and pricing class. Build quality: Laptops need to have a rugged build, especially if they’re lightweight and portable, as they need to be able to withstand the daily abuses.