Can we find the perfect test automation tool? It’s more of a rhetorical question because finding an ideal framework to solve all the problems we face in test programming is hard. Nonetheless, it is important to understand the differences that make a solution more or less effective depending on the project we are working on.
Table of contents:
- Why compare Cypress and Playwright?
- What is Cypress?
- What is Playwright?
- Playwright architecture
- Cypress architecture
- Cypress vs. Playwright head-to-head
- Cypress and Playwright compared – summary
Why compare Cypress and Playwright?
Cypress and Playwright have made significant contributions to web application test automation, each in their unique way. As you’re probably aware, Cypress is already well-known for its widespread use, established position, and sizable user base. Playwright, on the other hand, has busted into the test automation world and, thanks to what it has to offer, has been successfully elbowing its way in ever since.
Although these tools share some similarities, some differences may affect their final usability in our project. Today, we'll list all those aspects to provide you with a comprehensive picture of both testing tools' potential and limitations. Such a comparison can help you make the best decision about selecting a specific solution from the standpoint of the entire testing process.
What is Cypress?
Before we get into the basics of Cypress, let's look at its mission statement, as presented on its official website:
"Our mission is to build a thriving, open source ecosystem that enhances productivity, makes testing an enjoyable experience, and generates developer happiness. We hold ourselves accountable to champion a testing process that actually works."
– Cypress mission, via Cypress official website
As the authors write, it is “a tool for reliably testing anything that runs in a web browser.” More specifically, Cypress is the modern-day framework that allows you to automate component/unit integration and end-to-end UI tests. Additionally, it offers full support for API testing.
Cypress is intended to be a framework with built-in features to assist users in getting started with test automation. However, unlike many other frameworks, you do not need to install the components required for writing automated tests separately. So, rather than installing multiple libraries (as with Selenium or Playwright), Cypress provides a fully functional solution out of the box. To install Cypress, all you have to do is run one command – and that’s it.
What is Playwright?
Playwright is a Microsoft-founded, open-source test automation tool released in 2020. It was created by a team that has formerly developed another web automation tool – Puppeteer – at Google, and it was built with testers and developers in mind. Playwright enables reliable end-to-end testing for modern web applications based on Node.js, including running tests across multiple browsers.
Both Playwright awareness and the number of its users are growing yearly, as seen in the diagrams below.
Playwright architecture
Playwright uses the Chrome DevTools Protocol (CDP), which makes it compatible with the architecture of modern browsers and allows for out-of-process testing (when browsers run web content from different sources in different processes). It does not use the browser execution loop and thus requires an external Node.js process to drive the automation.
In essence, Playwright works similarly to Selenium, except that each HTTP request to the server is sent through the WebSocket connection. All modern browsers support this protocol, and communication remains active until the client or server terminates it. This translates into lower latency when compared to establishing a new connection for each request (long polling).
Cypress architecture
Cypress is based on the Electron framework, which enables the development of desktop applications in HTML, CSS, and JavaScript. This technology is built on a combination of Node.js and Chromium libraries. So, how does that work?
Cypress and the application under test are both injected into the browser and run in the same browser execution loop. The tool then automates the app without the need for any additional drivers. On the Cypress side, Node.js is behind the server process. In other words, Cypress and the Node.js server are constantly communicating, synchronizing, and performing tasks on their behalf.
Communication between the Node.js process and the browser happens via WebSocket, which starts after the proxy is created. This is when Cypress sends HTTP requests and responses from the Node.js server to the browser. It operates at the application's network layer, reading and changing traffic. By intercepting commands, it can modify everything that enters or exits the browser and alter code that may interfere with its ability to automate.
Cypress vs. Playwright head-to-head: pros, cons, features and limitations comparison
You already know what Cypress and Playwright tools are and how they work. It is now time to take a step further and compare their functionalities that can be used in the daily work of test automation engineers.
Supported languages & test runners
Because Cypress runs in the browser, it is limited to Javascript and transpiled languages. You get a testing framework based on Mocha (including BDD testing), an assertion library, as well as mocking and stubbing libraries. By the way, all of the libraries can be found in a bundle here. On top of that, Cypress includes a runner that displays executed actions and applications under test with real-time data.
Playwright is a cross-language tool with full Python, C#, and Java support. It can easily be used with its own JS/TS runner prepared as a VS Code extension instead of Mocha, Jest, or Jasmine. It is still possible to use the mentioned frameworks, but I highly recommend you go with the official runner (i.e., the Jest extension repository). Overall, Playwright supports a variety of testing frameworks to choose from, including Pytest, NUnit, JUnit, or TestNG.
Both tools have built-in runners, but only Playwright allows you to change them to operate with third-party runners. Summing up, in Cypress, we get a ready-made solution with numerous dependencies. In Playwright, we are responsible for adding libraries, but they are far fewer in number than in Cypress.
Cross-browser & platform support
Both Cypress and Playwright have extensive support for modern browsers. The first tool lets users launch tests in Firefox and Chromium-based browsers like Chrome, Edge, and Electron. However, at this moment, Cypress only offers experimental support for the Safari browser engine (as it is WebKit). This can quickly become a significant problem as Safari often shows compatibility issues with other browsers. As a result, Cypress needs to be adapted to each type of browser.
This is precisely why Playwright performs better with its built-in default support for Safari and other WebKit-based browsers. For picky users, the cons can be found in the experimental support of the Electron browser. However, it is worth noting that Playwright also provides experimental support for native mobile app testing, including Chrome for Android, Android WebView, and Mobile Safari. So, with WebKit builds for Windows, Linux, and macOS, you can now test how an app behaves in Apple Safari.
Testing capabilities
Interacting with elements
The interaction with elements is similar and quite enjoyable in both Cypress and Playwright. The syntax of Cypress is similar to that of jQuery, while Playwright syntax is much closer to JavaScript. It is important to distinguish between selectors and locators in Playwright, where a selector is a type of query that points to something on a page. In its API, these two are entirely separate classes – as the locator employs a selector to fetch an element from a page.
await page.getByLabel('User Name').fill('John');
await page.getByLabel('Password').fill('secret-password');
await page.getByRole('button', { name: 'Sign in' }).click();
await expect(page.getByText('Welcome, John!')).toBeVisible();
const locator = page.getByRole('button', { name: 'Sign in' })
await locator.hover();
await locator.click();Creating selectors practices in Playwright (JavaScript). Source: Playwright
Playwright provides a variety of methods for targeting elements, including CSS, XPath, and plain text. Cypress has two main selector-finding commands: cy.get() with a query selector as an input (id, className etc.) and cy.contains() based on text.
Both solutions come with good documentation, best practices, and tips on how to make applications more testable.
iframes support
Working with iframes is difficult in Cypress as it lacks native access to them. Hence, Cypress is unable to automate or communicate with an <iframe>. We can still use an additional plugin, ‘cypress-iframe’, with custom commands to simplify working with elements inside an iframe, but these solutions do not always work correctly. Another workaround that you can find is to manipulate the DOM and gain access to the iframe element this way, as Cypress is integrated with jQuery.
Playwright provides an API method .frameLocator() for switching to any given iframe, which takes away the extra leg work of installing a plugin.
Native browser alerts
Although alerts are becoming less common in modern applications, native browser events continue to be important in test automation. Playwright delivers a consistent solution for dialogs verification as you can test alert, confirm and prompt using a generic method. Cypress, on the other hand, differs in event types and requires a unique approach to each of them.
Multiple tabs & browsers
Cypress introduces an intriguing concept for testing scenarios by opening multiple browser tabs. According to the authors, users should not click on the <a> tag to see if a new browser tab will open. They argue that this could be perceived as testing the native behavior of the browser. Instead, they encourage testers to monitor things that cause the browser to perform this behavior – as opposed to testing the behavior itself. Nevertheless, the official documentation recommends solutions based on removing attributes from the link. Navigating to a new window is not supported in Cypress. It can only control one browser window at a time.
What services does Playwright offer to handle similar cases? We can make use of the Browser and Page classes. A single instance of the Browser class may contain multiple instances of the Page class. Each corresponds to a single tab or popup in the browser context. Plus, bringing the page to the front is unnecessary because it behaves like a focused, active page.
Anti-flakiness mechanisms
A “flaky test” is a test that passes and fails across multiple executions with no code changes in between. One of the primary causes of such results is a problem with rendering elements in modern applications. The handling of waiting for elements to appear in the application is undoubtedly a topic that causes minor or major issues for anyone who has ever automated UI tests. Fortunately, both Cypress and Playwright provide several solutions to solve the problem of unstable tests.
One of these solutions is the auto-wait mechanism, which is available in both Cypress and Playwright. The clue of this functionality lies in no extra work required to wait for an element. Both tools monitor the current state of the DOM and wait for commands and assertions before proceeding with the test. This method avoids asynchronous problems by executing commands sequentially.
It is possible to set global or on-command timeouts, after which checks that do not pass the test will fail. Cypress and Playwright also provide methods for indicating specific conditions when waiting for an element is more complex. Another feature that helps to avoid flakiness is working with the easily configurable Test Retry settings. We cannot forget about running tests in isolation, independently from any other test with its own session storage, cookies, etc.
Test parallelization
Parallel execution allows multiple test cases to run simultaneously, reducing execution time. It is useful and has the potential to be a critical feature in test automation. This is precisely why each framework we're discussing today supports parallel execution.
Parallelization is available in Cypress, starting with version 3.1.0. By default, Cypress executes all its commands serially, with consistent behavior for each test run. With the help of Cypress Cloud, we can enable parallel execution across multiple virtual machines in CI. It is worth mentioning that we can trigger up to 500 tests in the free basic plan. After that, we’re hitting a paywall (see Cypress pricing).
Playwright supports parallel execution across multiple machines. In fact, it runs tests in parallel by default. Tests from a single file are usually run in the order of declaration, but it is configurable to parallelize them too. So, if you want to give this functionality a shot, you should know that Playwright offers it for free.
Component testing
From version 7.0.0, Cypress offers component testing by default (by removing the experimentalComponentTesting flag), while Playwright focuses more on system / end-to-end testing and provides this functionality experimentally with React, Vue, and Svetle frameworks.
Authentication
Repetitive login operations in automated tests are time-consuming and can significantly extend the total execution time. Both frameworks we discuss here address the problem of repeated logins and provide solutions to facilitate authentication reuse.
Playwright provides clear and wide documentation on saving and reusing the authentication state. As Playwright executes tests in isolated browser contexts, they can load existing authenticated states stored as cookies or in local storage that can be used across browsers.
Cypress used to have its own limitations with multi-domain support, but at the end of April 2022, its authors released an integrated solution for caching session information between tests. Before they published the new integrated solution with new cy.origin() and cy.session() commands, they documented the approach only with the second one. You can find out more here (for the older solution) and here (for the new one).
API testing
In test automation, UI elements verification and user action are not everything. Verification of data bound to these elements on the API side is also important in today’s web application testing.
Both frameworks allow sending and intercepting requests from test code. They also permit mock queries. In Playwright, we must create a new context and then run requests from it. Microsoft’s tool uses asynchronous syntax with async/await and lets you easily assign the response to a new variable. In comparison, Cypress chooses a specific promise syntax, chain request, and testing commands. We receive a response in a call and can then extract properties such as body or status from it.
However, the implementation in both Playwright and Cypress is similar, and choosing one over the other is a personal preference.
cy.request("GET", 'https://petstore.swagger.io/v2/pet/1')
.then(({status, body}) => {
expect(status).to.eq(200)
expect(body.id).greaterThan(0)
})Cypress API test example (JavaScript)
const response = await request.get('https://petstore.swagger.io/v2/pet/1');
await expect(response).toBeOK();
const body = await response.json();
expect(body.id).toEqual(1);Playwright API test example (JavaScript)
Support for headless mode
Both frameworks run tests in headless mode by default, but they can be configured to run in headed mode with the appropriate command line flags or settings.
Using Page Object Model
The Page Object Model is a popular design pattern in which web application pages are represented as objects. The application stores a single web page as a class file, each containing corresponding web elements and related functions. This approach improves test maintenance while reducing code duplication.
Page objects are supported by both Cypress and Playwright, but they are not considered to be the best option in Cypress. This framework comes with an interesting architectural concept of Cypress App Actions. It allows interaction with applications under test by utilizing the application logic and its various states. So, instead of adding new items through the UI, setting the application’s state directly from the test is possible. By avoiding UI element interaction, testing becomes faster and more stable. This concept is described further in an official Cypress blog article.
Debugging
As test engineers, we debug on a daily basis. Failing tests need to be debugged to find out what could help identify and fix the issues causing flakiness. Therefore, the framework's methods, functionalities, and tools will greatly impact our test debugging experience.
In this regard, both Playwright and Cypress have a lot to offer. Of course, the methods they propose differ from one another, which results from their specificity, but we can indeed say that the authors of both tools have done an excellent job in providing users with intriguing solutions. Because the technical aspects are the subject of a separate article, I will present what our two frameworks offer without delving too deeply into technical details.
Test time-travel
Both tools have a time-travel feature that allows you to go through the performed tests and see what has happened in the browser at every single step.
The Cypress desktop GUI application lets us dig deeper into the anatomy of an error by providing its name, message, code frame, stack trace, and even the option to print the output to the console.
With the use of debugger, .debug(), or .pause() commands and Developer Tools, a tester can verify and interact with elements in the console and inspect the web application, the DOM, the network, and any storage.
This feature is implemented slightly differently in Playwright. Users of the official Visual Studio Code extension can set breakpoints, debug tests live in the browser, and review error messages.
But that’s not all. Apart from the above-mentioned extensions, Playwright Inspector is another recommended tool for test troubleshooting. This feature permits you to highlight elements in the browser and select locators you can copy directly to your tests and see if they pass.
Debugging in the CI environment
Debugging failures in CI environments is a challenging area of test automation. To address that issue, Cypress prepared a solution that collects run details such as results, errors, and time duration – but also recommendations, screenshots, videos, and CI logs generated by the test. Additionally, Cypress comes with the Flaky Test Management feature, which detects, flags, and tracks flaky tests automatically. These solutions to the problem seem to be of high quality, but the service isn’t free.
Microsoft brings us the Playwright Trace Viewer GUI tool to investigate test traces. It allows you to go through each action, analyze the state of the browser, and gain a broad understanding of what is happening during the tests. Traces can be downloaded directly from CI and saved and run locally or in the web application. During debugging, we can easily interact with the browser at any time during the test, but traces are more than static screenshots – they can also be rendered as a film strip with access to the screencast timeline.
Video recording and screenshots
Screenshots and videos help us understand why and where our test cases fail. It also makes sense to use test artifacts with better documentation and less flakiness in interpretations of test failures. Cypress and Playwright come with these capabilities out of the box. Cypress has an in-built feature to take screenshots and record videos, and in case of working in Playwright – it just needs to be switched on.
Performance
When the test repository is constantly growing, the total execution time of test suites can make a real difference in the overall CI/CD flow. In the framework selection stage, it is also worth considering performance capabilities. You can find at least a few independent articles comparing the performance of both Playwright and Cypress on the Internet (for example, here, here, or here). As for today, Playwright remains a much faster tool due to its architecture and use of the Chrome DevTools protocol. And while Cypress is obviously working on performance aspects, Playwright is still 4-5 times faster. It has direct control over the browser, translating into much higher performance.
Community & documentation
Cypress is unquestionably a tool with more “experience” in the automated testing market. Its community is mature and supportive. The documentation is of excellent quality, use cases are well-described, and users can follow the project’s official blog. There are many tutorials, videos, and articles available to help you get started with Cypress.
Even though Playwright is a new player with a smaller community at the moment, the team has already prepared great documentation. They are also active and responsive to GitHub user bug reports and feature requests. Plus, with increased awareness, it is becoming easier to get help with Playwright not only through official channels or forums but also through the increasing number of blog articles and video tutorials.
Cypress and Playwright compared – summary
In this blog post, I attempted to compare Cypress and Playwright on various testing-related reference points. If you took the time to skim (but I hope you paid much closer attention!) through my observations, you'd notice that each tool has its own pros, cons, and limitations.
However, it should be noted that tool selection should be made in a multi-vector manner. Appropriate testing tools and their functionalities can improve the overall testing process and contribute to more efficient quality assurance of the tested application.
If you need help selecting the right testing toolset or want to discuss some other QA-related issues, leave your question in the comments, and I'll do my best to recommend the best solution.
