Just pushed an update to the WebDriver Herald site (webdriver-herald.herokuapp.com) with the most recent results.
Some thoughts on the current results, a thread (1/20):
Some thoughts on the current results, a thread (1/20):
The tests run against Windows were run against Windows 10 Insider Build 17758 (released 11 September 2018). This allows the tests to run against the very latest driver for Microsoft Edge. (2/20)
The Internet Explorer tests were run using Windows 7 and the latest shipping version of the IE on that OS, this time. This decision was due to a bug in the Windows 10 17758 build that destabilized the IE driver. The bug has been reported, and will hopefully be fixed soon. (3/20)
Microsoft's Edge driver in the preview build, running using the W3C spec-compliant dialect of the wire protocol, is very nearly as good as Mozilla's geckodriver implementation (for Firefox's stable channel release) on Selenium tests. (4/20)
WPT tests for Edge, on the other hand, show a number of failures blocked by a bug in the driver where alerts are not handled by the user prompt handler during navigation commands (back, forward, refresh). (5/20)
If the Edge team could be convinced to fix this issue, their test results would be far, far better than they are even today. (6/20)
However, Microsoft's development processes, resource allocation, and priority selection almost guarantee this issue won't be fixed before the RS5 release is widely available. I think that's a shame, but I also understand it. (7/20)
Nevertheless, the @MSEdgeDev team deserves some real congratulations for the great work they've done on this. (8/20)
The Chrome implementation, when run with the spec-compliant protocol, performs abysmally on the WPT test suite. This is expected, since the Chromium team has not yet finished their spec-compliance work. (9/20)
ChromeDriver performs less poorly using the same protocol dialect on the Selenium test suite, mostly because the Selenium test suite (and the library itself) is more forgiving of protocol non-compliance, focusing more on browser behavior. (10/20)
I've been pretty harsh on the @ChromiumDev team about their lack of focus on spec compliance in ChromeDriver. After talking with some members of the team, I'm hopeful to see them turn the corner on this. (11/20)
Mozilla's geckodriver implementation for Firefox remains solid, as it has been for months. Incremental improvements are happening all the time, and the tests running against Nightly builds are incredibly compliant and stable in both test suites. (12/20)
If, for some reason, you're one of the misguided people who's remained on Selenium 2.x because you believe the geckodriver implementation is substandard, the test suite numbers bear out that the Mozilla implementation is as good as any Firefox driver has ever been. (13/20)
The team at @mozilla (run by @AutomatedTester) has done a phenomenal job both in enhancing their implementation, and in all their work on the WPT test suite. (14/20)
Finally, on the subject of Apple's Safari driver, the real results to focus on are the numbers when run against Safari Tech Preview, which uses the W3C protocol dialect by default. (15/20)
The Safari implementation is the fastest of the lot, but it's also the least compliant when running the Selenium test suite (second to ChromeDriver in the WPT test suite). (16/20)
I know that that the good folks over at Apple are committed to the development and maintenance of the driver; I question whether they'll be able to significantly improve the driver before the general availability of Safari 12. (17/20)
I'm hopeful that @brrian and his team will be able to pull off a miracle and get the Safari driver stabilized soon. (18/20)
If you have questions about the methodology or how the test results are evaluated, feel free to contact me. It's really just a little project I started to track WebDriver implementations for myself. (19/20)
Once the RS5 release of Windows 10 and Safari 12 become generally available, and once spec-compliance comes to ChromeDriver, I'll likely be reducing the amount of tests to generate it, but I intend to keep the WebDriver Herald updated every few weeks. (20/20)
