{"id":82983,"date":"2026-09-09T23:43:55","date_gmt":"2026-09-09T18:13:55","guid":{"rendered":"https:\/\/tothenewone.online\/blog\/?p=82983"},"modified":"2026-10-27T23:48:47","modified_gmt":"2026-10-27T18:18:47","slug":"building-an-agentic-qa-workflow-with-hermes-agent-mcp-and-playwright","status":"publish","type":"post","link":"https:\/\/tothenewone.online\/blog\/building-an-agentic-qa-workflow-with-hermes-agent-mcp-and-playwright\/","title":{"rendered":"Building an Agentic QA Workflow with Hermes Agent, MCP and Playwright"},"content":{"rendered":"\r\n

 <\/p>\r\n\r\n

Building an Agentic QA Workflow with Hermes Agent, MCP and Playwright<\/strong><\/h1>\r\n\r\n\r\n\r\n

A practical QA engineering experiment with real Playwright evidence, Hermes reference patterns, and guardrails for trustworthy agentic testing.<\/em><\/p>\r\n\r\n\r\n\r\n

What was actually tested<\/h2>\r\n\r\n\r\n\r\n

This experiment separates reproducible evidence from reference agent behaviour. A local login AUT was exercised with Playwright, while Hermes Agent and Playwright MCP were installed and their connectivity verified. A live Hermes LLM session was not run because no inference provider was configured.<\/p>\r\n\r\n\r\n\r\n

LayerStatusLogin AUTReproduced on :4173Playwright suiteReal run: 4 PASS, 1 FAILHermes AgentInstalled, v0.21.1Playwright MCPConnected, 24 tools discoveredLive Hermes inferenceNot run\r\n\r\n\r\n\r\n\r\n
\u00a0<\/th>\r\n<\/tr>\r\n
\u00a0<\/td>\r\n<\/tr>\r\n<\/tbody>\r\n<\/table>\r\n<\/figure>\r\n\r\n\r\n\r\n

The Playwright results, screenshots, traces, HAR and JSON are real local evidence. Hermes tool-call examples later in this article are reference patterns based on public Hermes browser documentation and the Playwright MCP tools discovered locally.<\/p>\r\n\r\n\r\n\r\n

Why agentic QA is different<\/h2>\r\n\r\n\r\n\r\n

Most AI-assisted testing still focuses on generation: write test cases, draft automation, create data, or summarize failures. Useful, but the engineer still moves context between tools.<\/p>\r\n\r\n\r\n\r\n

An agentic workflow changes the control model. Instead of giving the model browser steps, we give it a testing objective. Hermes reasons about what to investigate, MCP exposes a controlled tool interface, and Playwright executes browser actions. The goal is not to replace deterministic regression. It is to let an agent investigate ambiguous test objectives while producing evidence a QA engineer can review.<\/p>\r\n\r\n\r\n\r\n

Architecture: Hermes + MCP + Playwright<\/h2>\r\n\r\n\r\n\r\n
\r\n
\"Agentic

Figure 1. Hermes reasons, MCP exposes tools, and Playwright executes browser actions.<\/p><\/div>\r\n<\/figure>\r\n\r\n\r\n\r\n

This separation is important. Hermes should not implement browser automation, and Playwright should not decide what to test next. MCP becomes the boundary for tool discovery, RPC and access control. It also gives teams a place to expose only the capabilities the agent actually needs.<\/p>\r\n\r\n\r\n\r\n

Connecting Hermes to Playwright MCP<\/h2>\r\n\r\n\r\n\r\n
hermes chat\r\n\r\nhermes mcp test playwright\r\n# \u2192 Connected\r\n# \u2192 Tools discovered: 24<\/code><\/pre>\r\n\r\n\r\n\r\n
The local MCP configuration used a stdio Playwright server:<\/p>\r\n\r\n\r\n\r\n
mcp_servers:\r\n  playwright:\r\n    command: \"\/home\/box\/.hermes\/node\/bin\/npx\"\r\n    args: [\"-y\", \"@playwright\/mcp@latest\", \"--headless\"]\r\n    idle_timeout_seconds: 900\r\n    max_lifetime_seconds: 86400<\/code><\/pre>\r\n\r\n\r\n\r\n
Hermes registers MCP tools using a stable naming pattern such as mcp_playwright_browser_navigate<\/code>. Locally discovered tools included navigation, snapshots, clicks, typing, screenshots, console inspection and network inspection. In a QA environment, a smaller allowlisted tool surface is easier to reason about and safer to operate.<\/p>\r\n\r\n\r\n\r\n
\r\n
\"Verified
Figure 2. Local Hermes \u2194 Playwright MCP wiring verified with 24 tools.<\/p><\/div>\r\n<\/figure>\r\n\r\n\r\n\r\n
Experiment 1: deterministic login validation<\/h2>\r\n\r\n\r\n\r\n
The fixture application exposed a login form with these requirements: valid credentials must reach \/dashboard<\/code>; email and password are mandatory; invalid credentials must fail with an error; and malformed email must be rejected before authentication.<\/p>\r\n\r\n\r\n\r\n
\r\n\r\n\r\n\r\n\r\n\r\n\r\n\r\n\r\n
Scenario<\/th>\r\nResult<\/th>\r\nObserved<\/th>\r\n<\/tr>\r\n
Valid login<\/td>\r\nPASS<\/td>\r\nRedirected to \/dashboard<\/td>\r\n<\/tr>\r\n
Invalid password<\/td>\r\nPASS<\/td>\r\n401 and visible error<\/td>\r\n<\/tr>\r\n
Empty email<\/td>\r\nPASS<\/td>\r\nClient-side validation, no auth request<\/td>\r\n<\/tr>\r\n
Empty password<\/td>\r\nPASS<\/td>\r\nClient-side validation, no auth request<\/td>\r\n<\/tr>\r\n
Malformed email: qa.user@<\/td>\r\nFAIL<\/td>\r\nPOST \/login still sent<\/td>\r\n<\/tr>\r\n<\/tbody>\r\n<\/table>\r\n<\/figure>\r\n\r\n\r\n\r\n
\u2713 valid login\r\n\u2713 invalid password\r\n\u2713 empty email\r\n\u2713 empty password\r\n\u2718 malformed email should be rejected before authentication\r\n\r\n1 failed\r\n4 passed<\/code><\/pre>\r\n\r\n\r\n\r\n
\r\n
\"Evidence
Figure 3. Evidence flow for the reproducible Playwright run.<\/p><\/div>\r\n<\/figure>\r\n\r\n\r\n\r\n
LOGIN-001: malformed email reaches authentication<\/h3>\r\n\r\n\r\n\r\n
Input: qa.user@<\/code>. Expected: reject locally and do not call authentication. Actual: the browser remained on \/login<\/code>, but the debug output showed Authentication request attempted: yes<\/code>. Network evidence contained POST \/login<\/code>, which returned 401.<\/p>\r\n\r\n\r\n\r\n
{\r\n  \"emailField\": \"qa.user@\",\r\n  \"debugText\": \"Authentication request attempted: yes\",\r\n  \"authPosted\": true,\r\n  \"requests\": [\r\n    { \"method\": \"GET\",  \"url\": \"\/login\" },\r\n    { \"method\": \"POST\", \"url\": \"\/login\" }\r\n  ]\r\n}<\/code><\/pre>\r\n\r\n\r\n\r\n
This evidence contract matters more than the defect itself. URL, network activity, visible UI state, screenshots and traces are the same evidence we should demand from an agent before accepting a PASS or FAIL.<\/p>\r\n\r\n\r\n\r\n
From scripted tests to a testing objective<\/h2>\r\n\r\n\r\n\r\n
Traditional Playwright explicitly defines the path:<\/p>\r\n\r\n\r\n\r\n
test('valid user can login', async ({ page }) => {\r\n  await page.goto('\/login');\r\n  await page.getByLabel('Email').fill('qa.user@example.com');\r\n  await page.getByLabel('Password').fill(process.env.TEST_PASSWORD!);\r\n  await page.getByRole('button', { name: 'Login' }).click();\r\n  await expect(page).toHaveURL(\/dashboard\/);\r\n});<\/code><\/pre>\r\n\r\n\r\n\r\n
An agentic run should receive intent, constraints and an evidence contract instead:<\/p>\r\n\r\n\r\n\r\n
You are testing login functionality.\r\n\r\nValidate:\r\n- valid login\r\n- invalid password\r\n- empty email\r\n- empty password\r\n- malformed email\r\n\r\nUse Playwright browser tools.\r\nDo not modify application data.\r\n\r\nFor every scenario record:\r\nscenario, actions, expected, actual, PASS\/FAIL\/NEEDS REVIEW.\r\n\r\nNever infer PASS from the absence of an obvious error.\r\nOnly report PASS when observed evidence supports it.<\/code><\/pre>\r\n\r\n\r\n\r\n
The last two rules are critical. An agent that assumes success is less trustworthy than deterministic automation.<\/p>\r\n\r\n\r\n\r\n
\r\n
\"Agent
Figure 4. Objective \u2192 act through MCP \u2192 observe \u2192 judge.<\/p><\/div>\r\n<\/figure>\r\n\r\n\r\n\r\n
What an agent session could look like<\/h2>\r\n\r\n\r\n\r\n
Hermes browser documentation uses a navigate \u2192 snapshot \u2192 interact \u2192 snapshot loop. Mapped to the Playwright MCP tools discovered locally, a reference sequence looks like this:<\/p>\r\n\r\n\r\n\r\n
[tool_call] mcp_playwright_browser_navigate\r\n  url: \"\/login\"\r\n\r\n[tool_call] mcp_playwright_browser_snapshot\r\n\u2192 textbox \"Email\", textbox \"Password\", button \"Login\"\r\n\r\n[tool_call] mcp_playwright_browser_type\r\n\u2192 Email = \"qa.user@example.com\"\r\n\r\n[tool_call] mcp_playwright_browser_type\r\n\u2192 Password = \"***\"\r\n\r\n[tool_call] mcp_playwright_browser_click\r\n\u2192 Login\r\n\r\n[tool_call] mcp_playwright_browser_snapshot\r\n\u2192 heading \"Dashboard\"; URL \/dashboard\r\n\r\n\u2192 valid login: PASS, supported by observed redirect<\/code><\/pre>\r\n\r\n\r\n\r\n
This is a reference interaction pattern, not a captured live Hermes inference session.<\/p>\r\n\r\n\r\n\r\n
Exploration needs guardrails<\/h2>\r\n\r\n\r\n\r\n
If we remove the explicit scenario list, the workflow becomes exploratory. The agent can propose whitespace tests, boundary values, repeated failures, protected-route navigation or browser-back behaviour. That flexibility is useful only when bounded.<\/p>\r\n\r\n\r\n\r\n
Safety constraints:\r\n- Use only the designated test account.\r\n- Never access production.\r\n- Do not create or delete records unless requested.\r\n- Limit repeated authentication attempts.\r\n- Never perform security testing without authorization.\r\n- Stop if account lockout is detected.\r\n- Never expose passwords, tokens or secrets.<\/code><\/pre>\r\n\r\n\r\n\r\n
An autonomous agent without environment, permission and data boundaries is not advanced QA automation. It is an uncontrolled client.<\/p>\r\n\r\n\r\n\r\n
Reusable QA behaviour as a Hermes Skill<\/h2>\r\n\r\n\r\n\r\n
Long prompts do not scale. Hermes Skills let teams package a repeatable QA procedure:<\/p>\r\n\r\n\r\n\r\n
---\r\nname: exploratory-web-testing\r\ndescription: Structured exploratory testing of web applications\r\nversion: 1.0.0\r\n---\r\n\r\n1. Read requirements and acceptance criteria.\r\n2. Inspect browser state.\r\n3. Build and risk-rank scenarios.\r\n4. Execute one scenario at a time.\r\n5. Record expected versus observed.\r\n6. Capture evidence for failures.\r\n7. Separate observation from hypothesis.\r\n8. Report PASS \/ FAIL \/ NEEDS REVIEW.\r\n\r\nNever convert uncertainty into PASS.<\/code><\/pre>\r\n\r\n\r\n\r\n
Experiment 2: agentic failure triage<\/h2>\r\n\r\n\r\n\r\n
Failure triage may deliver faster value than autonomous exploration. Imagine 25 failed tests in CI. Instead of inspecting each failure independently, supply the agent with Playwright results, traces, screenshots, console logs and network failures.<\/p>\r\n\r\n\r\n\r\n
Analyse these failed Playwright tests.\r\nCluster failures when evidence suggests a shared cause.\r\nClassify each cluster as:\r\napplication \/ automation \/ environment \/ insufficient evidence.\r\nCite the evidence used.\r\nDo not invent root causes.<\/code><\/pre>\r\n\r\n\r\n\r\n
If eleven tests contain POST \/api\/customer\/search \u2192 503<\/code> and seven UI tests time out waiting for customer results, the agent can flag a likely shared dependency and recommend checking that service before engineers analyse 18 failures separately. The value is reduced investigation duplication, not automatic root-cause authority.<\/p>\r\n\r\n\r\n\r\n
Where deterministic Playwright still wins<\/h2>\r\n\r\n\r\n\r\n
await expect(order.status).toBe('CONFIRMED');<\/code><\/pre>\r\n\r\n\r\n\r\n
For deployment gates and known business rules, deterministic assertions remain the correct tool. Agents add value when the next action depends on what was just observed: exploration, triage and investigation. They should complement, not replace, stable regression suites.<\/p>\r\n\r\n\r\n\r\n
What to measure in a real POC<\/h2>\r\n\r\n\r\n\r\n