Building an Agentic QA Workflow with Hermes Agent, MCP and Playwright

A practical QA engineering experiment with real Playwright evidence, Hermes reference patterns, and guardrails for trustworthy agentic testing.

What was actually tested

This experiment separates reproducible evidence from reference agent behaviour. A local login AUT was exercised with Playwright, while Hermes Agent and Playwright MCP were installed and their connectivity verified. A live Hermes LLM session was not run because no inference provider was configured.

LayerStatusLogin AUTReproduced on :4173Playwright suiteReal run: 4 PASS, 1 FAILHermes AgentInstalled, v0.21.1Playwright MCPConnected, 24 tools discoveredLive Hermes inferenceNot run

The Playwright results, screenshots, traces, HAR and JSON are real local evidence. Hermes tool-call examples later in this article are reference patterns based on public Hermes browser documentation and the Playwright MCP tools discovered locally.

Why agentic QA is different

Most AI-assisted testing still focuses on generation: write test cases, draft automation, create data, or summarize failures. Useful, but the engineer still moves context between tools.

An agentic workflow changes the control model. Instead of giving the model browser steps, we give it a testing objective. Hermes reasons about what to investigate, MCP exposes a controlled tool interface, and Playwright executes browser actions. The goal is not to replace deterministic regression. It is to let an agent investigate ambiguous test objectives while producing evidence a QA engineer can review.

Architecture: Hermes + MCP + Playwright

Figure 1. Hermes reasons, MCP exposes tools, and Playwright executes browser actions.

This separation is important. Hermes should not implement browser automation, and Playwright should not decide what to test next. MCP becomes the boundary for tool discovery, RPC and access control. It also gives teams a place to expose only the capabilities the agent actually needs.

Connecting Hermes to Playwright MCP

hermes chat

hermes mcp test playwright
# → Connected
# → Tools discovered: 24

The local MCP configuration used a stdio Playwright server:

mcp_servers:
  playwright:
    command: "/home/box/.hermes/node/bin/npx"
    args: ["-y", "@playwright/mcp@latest", "--headless"]
    idle_timeout_seconds: 900
    max_lifetime_seconds: 86400

Hermes registers MCP tools using a stable naming pattern such as mcp_playwright_browser_navigate. Locally discovered tools included navigation, snapshots, clicks, typing, screenshots, console inspection and network inspection. In a QA environment, a smaller allowlisted tool surface is easier to reason about and safer to operate.

Figure 2. Local Hermes ↔ Playwright MCP wiring verified with 24 tools.

Experiment 1: deterministic login validation

The fixture application exposed a login form with these requirements: valid credentials must reach /dashboard; email and password are mandatory; invalid credentials must fail with an error; and malformed email must be rejected before authentication.

Scenario	Result	Observed
Valid login	PASS	Redirected to /dashboard
Invalid password	PASS	401 and visible error
Empty email	PASS	Client-side validation, no auth request
Empty password	PASS	Client-side validation, no auth request
Malformed email: qa.user@	FAIL	POST /login still sent

✓ valid login
✓ invalid password
✓ empty email
✓ empty password
✘ malformed email should be rejected before authentication

1 failed
4 passed

Figure 3. Evidence flow for the reproducible Playwright run.

LOGIN-001: malformed email reaches authentication

Input: qa.user@. Expected: reject locally and do not call authentication. Actual: the browser remained on /login, but the debug output showed Authentication request attempted: yes. Network evidence contained POST /login, which returned 401.

{
  "emailField": "qa.user@",
  "debugText": "Authentication request attempted: yes",
  "authPosted": true,
  "requests": [
    { "method": "GET",  "url": "/login" },
    { "method": "POST", "url": "/login" }
  ]
}

This evidence contract matters more than the defect itself. URL, network activity, visible UI state, screenshots and traces are the same evidence we should demand from an agent before accepting a PASS or FAIL.

From scripted tests to a testing objective

Traditional Playwright explicitly defines the path:

test('valid user can login', async ({ page }) => {
  await page.goto('/login');
  await page.getByLabel('Email').fill('[email protected]');
  await page.getByLabel('Password').fill(process.env.TEST_PASSWORD!);
  await page.getByRole('button', { name: 'Login' }).click();
  await expect(page).toHaveURL(/dashboard/);
});

An agentic run should receive intent, constraints and an evidence contract instead:

You are testing login functionality.

Validate:
- valid login
- invalid password
- empty email
- empty password
- malformed email

Use Playwright browser tools.
Do not modify application data.

For every scenario record:
scenario, actions, expected, actual, PASS/FAIL/NEEDS REVIEW.

Never infer PASS from the absence of an obvious error.
Only report PASS when observed evidence supports it.

The last two rules are critical. An agent that assumes success is less trustworthy than deterministic automation.

Figure 4. Objective → act through MCP → observe → judge.

What an agent session could look like

Hermes browser documentation uses a navigate → snapshot → interact → snapshot loop. Mapped to the Playwright MCP tools discovered locally, a reference sequence looks like this:

[tool_call] mcp_playwright_browser_navigate
  url: "/login"

[tool_call] mcp_playwright_browser_snapshot
→ textbox "Email", textbox "Password", button "Login"

[tool_call] mcp_playwright_browser_type
→ Email = "[email protected]"

[tool_call] mcp_playwright_browser_type
→ Password = "***"

[tool_call] mcp_playwright_browser_click
→ Login

[tool_call] mcp_playwright_browser_snapshot
→ heading "Dashboard"; URL /dashboard

→ valid login: PASS, supported by observed redirect

This is a reference interaction pattern, not a captured live Hermes inference session.

Exploration needs guardrails

If we remove the explicit scenario list, the workflow becomes exploratory. The agent can propose whitespace tests, boundary values, repeated failures, protected-route navigation or browser-back behaviour. That flexibility is useful only when bounded.

Safety constraints:
- Use only the designated test account.
- Never access production.
- Do not create or delete records unless requested.
- Limit repeated authentication attempts.
- Never perform security testing without authorization.
- Stop if account lockout is detected.
- Never expose passwords, tokens or secrets.

An autonomous agent without environment, permission and data boundaries is not advanced QA automation. It is an uncontrolled client.

Reusable QA behaviour as a Hermes Skill

Long prompts do not scale. Hermes Skills let teams package a repeatable QA procedure:

---
name: exploratory-web-testing
description: Structured exploratory testing of web applications
version: 1.0.0
---

1. Read requirements and acceptance criteria.
2. Inspect browser state.
3. Build and risk-rank scenarios.
4. Execute one scenario at a time.
5. Record expected versus observed.
6. Capture evidence for failures.
7. Separate observation from hypothesis.
8. Report PASS / FAIL / NEEDS REVIEW.

Never convert uncertainty into PASS.

Experiment 2: agentic failure triage

Failure triage may deliver faster value than autonomous exploration. Imagine 25 failed tests in CI. Instead of inspecting each failure independently, supply the agent with Playwright results, traces, screenshots, console logs and network failures.

Analyse these failed Playwright tests.
Cluster failures when evidence suggests a shared cause.
Classify each cluster as:
application / automation / environment / insufficient evidence.
Cite the evidence used.
Do not invent root causes.

If eleven tests contain POST /api/customer/search → 503 and seven UI tests time out waiting for customer results, the agent can flag a likely shared dependency and recommend checking that service before engineers analyse 18 failures separately. The value is reduced investigation duplication, not automatic root-cause authority.

Where deterministic Playwright still wins

await expect(order.status).toBe('CONFIRMED');

For deployment gates and known business rules, deterministic assertions remain the correct tool. Agents add value when the next action depends on what was just observed: exploration, triage and investigation. They should complement, not replace, stable regression suites.

What to measure in a real POC

• Useful scenarios discovered versus duplicates
• Defects found and false-positive rate
• Agent execution plus human review time
• Manual versus agent-assisted triage time
• Failure classification and clustering accuracy
• Unsupported root-cause hypotheses

If manual triage takes 60 minutes and agent-assisted triage takes 15 minutes including review, the workflow has measurable value. If engineers spend most of that time correcting invented conclusions, it does not.

Key takeaways

1. Agents and deterministic automation solve different problems. Playwright owns repeatable assertions; agents are better suited to ambiguity and investigation.
2. MCP creates a useful control boundary. It separates reasoning from execution and enables narrow tool exposure.
3. Prompt rules are part of the test architecture. “Do not infer PASS” directly affects result reliability.
4. Evidence is non-negotiable. A model conclusion is not test evidence.
5. Reusable Skills encode testing procedure. They turn QA methodology into repeatable agent guidance.
6. Human review remains the final control. Product risk, ambiguous requirements and defect validity still require engineering judgement.

Final thought

Traditional automation taught machines how to execute our tests. Agentic testing asks whether they can also help decide what to investigate next.

The practical path is not an autonomous QA department. Start with one bounded workflow, expose the smallest useful tool surface, require evidence for every conclusion, and measure whether the agent genuinely reduces engineering effort. That is where Hermes, MCP and Playwright become more than an AI demo and start to look like a credible QA architecture.

References

• Hermes Agent — MCP
• Use MCP with Hermes
• Hermes Agent — Browser automation
• NousResearch/hermes-agent
• Playwright MCP (@playwright/mcp)