Run Playwright tests against Dendrite & Pinecone periodically (#28888)

* Switch to TestContainers for manging services in Playwright Signed-off-by: Michael Telatynski <7t3chguy@gmail.com> * Iterate Signed-off-by: Michael Telatynski <7t3chguy@gmail.com> * Flip fixture dependency order Signed-off-by: Michael Telatynski <7t3chguy@gmail.com> * Remove mas dep Signed-off-by: Michael Telatynski <7t3chguy@gmail.com> * Iterate Signed-off-by: Michael Telatynski <7t3chguy@gmail.com> * Iterate Signed-off-by: Michael Telatynski <7t3chguy@gmail.com> * Iterate Signed-off-by: Michael Telatynski <7t3chguy@gmail.com> * Iterate Signed-off-by: Michael Telatynski <7t3chguy@gmail.com> * Iterate Signed-off-by: Michael Telatynski <7t3chguy@gmail.com> * Iterate Signed-off-by: Michael Telatynski <7t3chguy@gmail.com> * Update matrix-authentication-service in Playwright tests Signed-off-by: Michael Telatynski <7t3chguy@gmail.com> * delint Signed-off-by: Michael Telatynski <7t3chguy@gmail.com> * Fix SMTP port Signed-off-by: Michael Telatynski <7t3chguy@gmail.com> * Iterate Signed-off-by: Michael Telatynski <7t3chguy@gmail.com> * Comments Signed-off-by: Michael Telatynski <7t3chguy@gmail.com> * Strip ansi from playwright logs to make them more readable Signed-off-by: Michael Telatynski <7t3chguy@gmail.com> * Actually do the update Signed-off-by: Michael Telatynski <7t3chguy@gmail.com> * Iterate Signed-off-by: Michael Telatynski <7t3chguy@gmail.com> * Remove access to homeserver.config.baseUrl field in favour of homeserver.baseUrl Signed-off-by: Michael Telatynski <7t3chguy@gmail.com> * Use sane default_server_config and specify server.invalid in the specific tests which demand it Signed-off-by: Michael Telatynski <7t3chguy@gmail.com> * Fix mas run Signed-off-by: Michael Telatynski <7t3chguy@gmail.com> * break cycle Signed-off-by: Michael Telatynski <7t3chguy@gmail.com> * Iterate Signed-off-by: Michael Telatynski <7t3chguy@gmail.com> * typo Signed-off-by: Michael Telatynski <7t3chguy@gmail.com> * Iterate Signed-off-by: Michael Telatynski <7t3chguy@gmail.com> * Iterate Signed-off-by: Michael Telatynski <7t3chguy@gmail.com> * Iterate Signed-off-by: Michael Telatynski <7t3chguy@gmail.com> * prettier Signed-off-by: Michael Telatynski <7t3chguy@gmail.com> * Wire up basics of dendriteHomeserver Signed-off-by: Michael Telatynski <7t3chguy@gmail.com> * Run Playwright tests against Dendrite & Pinecone periodically Signed-off-by: Michael Telatynski <7t3chguy@gmail.com> * Iterate Signed-off-by: Michael Telatynski <7t3chguy@gmail.com> * Iterate Signed-off-by: Michael Telatynski <7t3chguy@gmail.com> * Iterate Signed-off-by: Michael Telatynski <7t3chguy@gmail.com> * Iterate Signed-off-by: Michael Telatynski <7t3chguy@gmail.com> * Iterate Signed-off-by: Michael Telatynski <7t3chguy@gmail.com> * Iterate Signed-off-by: Michael Telatynski <7t3chguy@gmail.com> * Iterate Signed-off-by: Michael Telatynski <7t3chguy@gmail.com> * Iterate Signed-off-by: Michael Telatynski <7t3chguy@gmail.com> * Iterate Signed-off-by: Michael Telatynski <7t3chguy@gmail.com> * Fix playwright-image-updates.yaml workflow Signed-off-by: Michael Telatynski <7t3chguy@gmail.com> * Add `X-Run-All-Tests` label for running all tests Signed-off-by: Michael Telatynski <7t3chguy@gmail.com> * Ignore failing tests in stale-screenshot-reporter.ts to avoid confusing errors Signed-off-by: Michael Telatynski <7t3chguy@gmail.com> * Iterate Signed-off-by: Michael Telatynski <7t3chguy@gmail.com> * Iterate Signed-off-by: Michael Telatynski <7t3chguy@gmail.com> * Specify Synapse ui_auth.session_timeout only on tests which require it As Dendrite lacks this configuration option Signed-off-by: Michael Telatynski <7t3chguy@gmail.com> * Iterate Signed-off-by: Michael Telatynski <7t3chguy@gmail.com> * Iterate Signed-off-by: Michael Telatynski <7t3chguy@gmail.com> * Iterate Signed-off-by: Michael Telatynski <7t3chguy@gmail.com> * Iterate Signed-off-by: Michael Telatynski <7t3chguy@gmail.com> * React to MatrixEvent sender/target sentinels being updated for rendering state events Signed-off-by: Michael Telatynski <7t3chguy@gmail.com> * Iterate Signed-off-by: Michael Telatynski <7t3chguy@gmail.com> * Iterate Signed-off-by: Michael Telatynski <7t3chguy@gmail.com> * Remove test code Signed-off-by: Michael Telatynski <7t3chguy@gmail.com> * Iterate Signed-off-by: Michael Telatynski <7t3chguy@gmail.com> * React to sentinel changes in EventListSummary Signed-off-by: Michael Telatynski <7t3chguy@gmail.com> * Iterate Signed-off-by: Michael Telatynski <7t3chguy@gmail.com> * Iterate Signed-off-by: Michael Telatynski <7t3chguy@gmail.com> * Docs Signed-off-by: Michael Telatynski <7t3chguy@gmail.com> * Avoid reusing user1234 Signed-off-by: Michael Telatynski <7t3chguy@gmail.com> * Fix stale-screenshot-reporter.ts Signed-off-by: Michael Telatynski <7t3chguy@gmail.com> * Clean up public rooms between tests on reused homeserver Signed-off-by: Michael Telatynski <7t3chguy@gmail.com> * Deflake spotlight when homeserver is reused Signed-off-by: Michael Telatynski <7t3chguy@gmail.com> * Deflake more tests using existing username Signed-off-by: Michael Telatynski <7t3chguy@gmail.com> * Clean mailhog between tests Signed-off-by: Michael Telatynski <7t3chguy@gmail.com> * Fix more flakes Signed-off-by: Michael Telatynski <7t3chguy@gmail.com> * Fix missing _request Signed-off-by: Michael Telatynski <7t3chguy@gmail.com> * Iterate Signed-off-by: Michael Telatynski <7t3chguy@gmail.com> * Iterate Signed-off-by: Michael Telatynski <7t3chguy@gmail.com> * Iterate Signed-off-by: Michael Telatynski <7t3chguy@gmail.com> * Iterate Signed-off-by: Michael Telatynski <7t3chguy@gmail.com> * Fix playwright flaky tests Signed-off-by: Michael Telatynski <7t3chguy@gmail.com> * Wipe mailhog between test runs Signed-off-by: Michael Telatynski <7t3chguy@gmail.com> * Delint Signed-off-by: Michael Telatynski <7t3chguy@gmail.com> * Iterate Signed-off-by: Michael Telatynski <7t3chguy@gmail.com> * delint Signed-off-by: Michael Telatynski <7t3chguy@gmail.com> * Deflake more tests Signed-off-by: Michael Telatynski <7t3chguy@gmail.com> * Iterate Signed-off-by: Michael Telatynski <7t3chguy@gmail.com> * Fix flaky tests Signed-off-by: Michael Telatynski <7t3chguy@gmail.com> * Fix flaky tests Signed-off-by: Michael Telatynski <7t3chguy@gmail.com> * Fix mas config Signed-off-by: Michael Telatynski <7t3chguy@gmail.com> * Fix another flaky test Signed-off-by: Michael Telatynski <7t3chguy@gmail.com> * Fix playwright flakes due to floating promises Signed-off-by: Michael Telatynski <7t3chguy@gmail.com> * Iterate Signed-off-by: Michael Telatynski <7t3chguy@gmail.com> * Iterate Signed-off-by: Michael Telatynski <7t3chguy@gmail.com> * Fix flaky playwright tests Signed-off-by: Michael Telatynski <7t3chguy@gmail.com> * Iterate Signed-off-by: Michael Telatynski <7t3chguy@gmail.com> * Iterate Signed-off-by: Michael Telatynski <7t3chguy@gmail.com> * Iterate Signed-off-by: Michael Telatynski <7t3chguy@gmail.com> * Iterate Signed-off-by: Michael Telatynski <7t3chguy@gmail.com> * Iterate Signed-off-by: Michael Telatynski <7t3chguy@gmail.com> * Iterate Signed-off-by: Michael Telatynski <7t3chguy@gmail.com> * Iterate Signed-off-by: Michael Telatynski <7t3chguy@gmail.com> * Update services.ts --------- Signed-off-by: Michael Telatynski <7t3chguy@gmail.com>
2025-01-14 18:00:30 +00:00
parent bb41616d5f
commit dbce48b23d
55 changed files with 259 additions and 95 deletions
@@ -9,6 +9,7 @@ Please see LICENSE files in the repository root for full details.
 import { type Page } from "@playwright/test";

 import { test, expect } from "../../element-web-test";
+import { isDendrite } from "../../plugins/homeserver/dendrite";

 async function expectBackupVersionToBe(page: Page, version: string) {
    await expect(page.locator(".mx_SecureBackupPanel_statusList tr:nth-child(5) td")).toHaveText(
@@ -19,6 +20,7 @@ async function expectBackupVersionToBe(page: Page, version: string) {
 }

 test.describe("Backups", () => {
+    test.skip(isDendrite, "Dendrite lacks support for MSC3967 so requires additional auth here");
    test.use({
        displayName: "Hanako",
    });
@@ -8,8 +8,10 @@ Please see LICENSE files in the repository root for full details.

 import { test, expect } from "../../element-web-test";
 import { logIntoElement } from "./utils";
+import { isDendrite } from "../../plugins/homeserver/dendrite";

 test.describe("Complete security", () => {
+    test.skip(isDendrite, "Dendrite lacks support for MSC3967 so requires additional auth here");
    test.use({
        displayName: "Jeff",
    });
@@ -11,6 +11,7 @@ import { expect, test } from "../../element-web-test";
 import { autoJoin, copyAndContinue, createSharedRoomWithUser, enableKeyBackup, verify } from "./utils";
 import { Bot } from "../../pages/bot";
 import { ElementAppPage } from "../../pages/ElementAppPage";
+import { isDendrite } from "../../plugins/homeserver/dendrite";

 const checkDMRoom = async (page: Page) => {
    const body = page.locator(".mx_RoomView_body");
@@ -67,6 +68,7 @@ const bobJoin = async (page: Page, bob: Bot) => {
 };

 test.describe("Cryptography", function () {
+    test.skip(isDendrite, "Dendrite lacks support for MSC3967 so requires additional auth here");
    test.use({
        displayName: "Alice",
        botCreateOpts: {
@@ -28,6 +28,8 @@ test.describe("Cryptography", function () {
    });

    test.describe("decryption failure messages", () => {
+        test.skip(isDendrite, "Dendrite lacks support for MSC3967 so requires additional auth here");
+
        test("should handle device-relative historical messages", async ({
            homeserver,
            page,
@@ -219,9 +219,10 @@ test.describe("Device verification", { tag: "@no-webkit" }, () => {
        /* And we're all done! */
        const infoDialog = page.locator(".mx_InfoDialog");
        await infoDialog.getByRole("button", { name: "They match" }).click();
-        await expect(
-            infoDialog.getByText(`You've successfully verified (${aliceBotClient.credentials.deviceId})!`),
-        ).toBeVisible();
+        // We don't assert the full string as the device name is unset on Synapse but set to the user ID on Dendrite
+        await expect(infoDialog.getByText(`You've successfully verified`)).toContainText(
+            `(${aliceBotClient.credentials.deviceId})`,
+        );
        await infoDialog.getByRole("button", { name: "Got it" }).click();
    });
 });
@@ -8,8 +8,10 @@ Please see LICENSE files in the repository root for full details.

 import { test, expect } from "../../element-web-test";
 import { createRoom, enableKeyBackup, logIntoElement, sendMessageInCurrentRoom } from "./utils";
+import { isDendrite } from "../../plugins/homeserver/dendrite";

 test.describe("Logout tests", () => {
+    test.skip(isDendrite, "Dendrite lacks support for MSC3967 so requires additional auth here");
    test.beforeEach(async ({ page, homeserver, credentials }) => {
        await logIntoElement(page, credentials);
    });