Test, publish, observe, improve

Production is an operating practice

Green checks let you publish. Evidence tells you whether you should.

Use the simulator to test expected ownership and full responses, Flow Checks to catch structural faults, immutable revisions to protect the live configuration, and Analytics traces to understand real behavior. No single signal is enough on its own.

A release loop that protects the live experience

1. 01

   Edit the builder draft

   Changes to Workflow Settings, agents, workflow tools, handoffs, and layout remain in editable storage. If a revision is already live, visitors keep using that published snapshot.

2. 02

   Resolve Flow Checks

   Errors block publication except for the intentional unsafe-HTTP override. Warnings and notes require judgment and usually stronger configuration.

3. 03

   Run a repeatable test set

   Compare actual and Expected handler, inspect the reply, exercise exits, and keep action execution off until the route is trustworthy.

4. 04

   Test real actions deliberately

   Use sandbox recipients/endpoints and understand that enabling Run action tools can send real email, call a webhook, or fire a WordPress hook.

5. 05

   Publish with a useful note

   SmartSite stores a new integrity-checked revision and activates it atomically. The previously live flow, if different, is paused.

6. 06

   Review real traces and outcomes

   Watch incorrect routes, tool failures, action uncertainty, fallback, latency, and visitor language. Improve the draft and repeat.

Simulator: every control and what it proves

The Simulator appears in the builder inspector under Flow Test.

Simulator controls

Control	What it does	How to use it
Visitor message	Runs the editable workflow with a synthetic newest message.	Use realistic phrasing, short follow-ups, corrections, ambiguity, topic changes, typos, and adversarial or frustrated language. Never paste real personal data.
Expected handler	Lets you select Base Assistant or one specialist as the expected owner. The result reports matched or differed.	Set an expectation before running. “No expected handler” is useful for exploration, not for a repeatable regression set.
Run action tools	When off, email, webhook, and WordPress action tools return a simulated result without executing. When on, the simulator permits real execution.	Keep off for routing and conversational tests. Turn on only in a controlled environment with sandbox destinations and a specific action test.
Run Test	Streams the response and trace while the current builder configuration runs.	Review more than the final wording: owner, route reason, tools, confirmation, errors, timing, and expected comparison all matter.
Replay Trace	Replays the last normalized run on the canvas, highlighting nodes and edges in sequence.	Use it to explain an unexpected path to a non-technical reviewer. Replay visualizes the saved last test; it does not run the AI again.

Run action tools can have real side effects

With the switch on, a test may send email to the configured recipient, submit a webhook request, or trigger site code through a WordPress hook. Use Local or staging, synthetic data, and endpoints created for testing. Confirmation gates still apply, but do not rely on them as permission to test against production systems.

What a successful test does—and does not—prove

An Expected handler matched result proves that one message resolved to the chosen owner in that test context. It does not prove the answer is factually correct, the Knowledge Base is current, a tool used the right arguments, an external business process completed, or future paraphrases will route the same way.

Review each result in layers:

1. Ownership: Did Base or the correct specialist receive the turn?
2. Continuity: Did a short answer stay with the active specialist?
3. Evidence: Were changing facts searched or looked up instead of invented?
4. Capability: Was the correct tool selected only when appropriate?
5. Safety: Were exact details shown and confirmed before action?
6. Outcome: Did the reply accurately distinguish completed, requested, failed, cancelled, or uncertain?
7. Experience: Was the answer concise, in the visitor’s language, and free of internal workflow terminology?

Flow Checks: errors, warnings, and notes

Flow Checks evaluates structure every time the builder loads or a test completes.

Current Flow Check findings

Level	Finding	What to do
Blocking error	OpenAI connection is not ready; runtime package unavailable; no specialist agents; selected start agent missing; policy allows no entry targets.	Repair the global connection, package/database state, or workflow structure. Publish remains disabled.
Blocking error	Enabled tool points to a missing agent, has no connected SmartSite Tool, or is a Custom Instruction planning node.	Reassign, connect an enabled existing tool, disable the node, or remove it. Custom Instruction is not executable.
Blocking error	Handoff routes to itself or its source/target no longer exists.	Choose two valid different actors or remove the damaged handoff.
Overrideable error	An enabled connected webhook intentionally uses Allow unsafe HTTP.	Prefer safe public HTTPS. If intentional, review the publication warning and choose Publish anyway. This is the only overrideable error.
Warning	Agent has no Description or Instructions, relies on generated activation, or has no policy escape path.	Clarify scope/playbook, create an explicit Base entry, and provide Base return or explicit exits.
Warning	Active specialists have no general policy escape path; enabled tool is unassigned; handoff Condition is too short.	Review routing posture, attach capabilities to an owner, and write a useful decision rule.
Note	Agent has neither Knowledge Base nor enabled tools.	This can be intentional for a narrow conversational role. Confirm its instructions alone are enough.

Warnings do not block publication because some are legitimate design choices. For example, De-escalation may intentionally have no knowledge or tools. Document the reason and test the outcome. Do not dismiss warnings merely because the Publish button remains enabled.

Advanced Agents health: site-level readiness

The expandable Advanced Agents health panel is broader than one workflow’s Flow Checks.

Health checks

Check	Ready means	If it needs attention
OpenAI connection	An API key is configured.	Return to API Settings. A stored key still needs a real model request to prove project access and billing are usable.
Agent runtime	The PHP runtime package classes are loaded.	Reinstall a complete production plugin package; do not publish from a source tree missing dependencies.
Action protection	Approval and duplicate-protection storage exists.	Run the plugin database upgrade path before allowing consequential actions.
Published versions	Immutable revision storage exists.	Complete the database upgrade before relying on publication or rollback.
Live workflow	An active revision exists and passes its integrity check.	Publish a tested flow, or investigate a missing/corrupt revision. If none is live, Basic Assistant remains available.
Actions needing review	No unexpired action is stuck in executing or unknown state.	Check the destination system before retrying. “Unknown” means an action may have happened and automatic repetition would be unsafe.

Ready with notes can simply mean no flow has been published. Needs attention indicates a missing requirement or invalid live version and should be resolved before production use.

Publish: what becomes live

The Live Version card shows the active revision number, whether the builder has unpublished changes, an optional Version note, and Publish Workflow / Publish Changes.

When publishing succeeds, SmartSite:

• runs the server-side readiness check again;
• records the complete current workflow, agents, tools, handoffs, routing policy, and applicable unsafe-HTTP consent in a new revision;
• hashes that snapshot so the runtime can verify its integrity;
• marks the new revision active in one database transaction;
• pauses any other active workflow, so only one workflow serves visitors;
• keeps the editable builder available for the next change.

Use a Version note that helps a future administrator understand risk and intent:

Tightened Billing entry condition; Appointments now returns general price
questions to Base; verified booking email confirmation in staging.

Avoid notes such as “update,” “test,” or “fixed stuff.” Notes are capped in the interface and should summarize behavior, not contain personal data or secrets.

Save is not publish

Saving Workflow Settings, an Agent, Tool, Handoff, or canvas layout updates the builder. Visitors continue using the immutable live revision until Publish Changes succeeds. The “You have unpublished changes” indicator is the reminder that builder and live behavior differ.

Status and deletion behavior

Workflow status

Status	Visitor effect	Admin use
Draft	Not selected as the live workflow.	New or not-yet-published work. Draft configuration can be simulated and reviewed.
Active	Its selected published revision is available to the Advanced runtime.	Set by successful publication or rollback. Editing the builder does not mutate the active revision.
Paused	The workflow is stored but does not serve visitors.	Use during maintenance or before deletion. Publishing another workflow automatically pauses the previous active one.

An active workflow cannot be deleted directly. Pause it first. Deleting a paused/draft workflow removes its workflow record, agents, workflow tools, handoffs, approval records, and revision history. Treat deletion as permanent and prefer pausing when you may need the design again.

Version history and rollback

Every successful publish creates a numbered revision with its note, time, and publishing admin. Use this revision makes an earlier snapshot live after verifying its stored integrity. Other active workflows are paused.

Rollback changes the live revision; it does not overwrite current builder edits. That is valuable during an incident: visitors can return to the known-good snapshot while admins keep the unfinished draft for diagnosis. After rollback, clearly identify whether the builder differs from the restored live version before the next Publish Changes, or you may unintentionally reintroduce the problem.

Use rollback when a new revision causes incorrect ownership, missing tools, degraded responses, or integration problems and the previous revision is known good. Do not use it to conceal a compromised API key, unsafe endpoint, or corrupt plugin installation; resolve the underlying system issue.

Runtime limits and safe fallback

Advanced runs have guardrails against loops and runaway work. By default, one turn allows up to eight model turns, four handoffs, eight tool calls, three repeated calls to the same tool, and 45 seconds. Reaching a boundary stops the flow safely and records the limit in its trace. These are system protections, not targets a normal workflow should approach.

Fallback to Basic is permitted only while it is safe to start over—before response output, tool calls, or handoffs have begun. Once the Advanced run may have changed state or partially communicated, SmartSite returns an honest error instead of blindly retrying the Basic path. This reduces duplicate actions and contradictory replies.

Read Analytics traces as an incident story

Open Analytics → Chat History, select a conversation, and inspect Advanced Agent details. Depending on the run, you may see:

• routing mode and run identifier;
• route decisions with From, To, stage, confidence, and a reason;
• registered and executed handoffs;
• tools called and their result status;
• the event timeline: routing, workflow start, agent invocation, response, handoff, tool events, completion, failure, or fallback;
• model-turn, handoff, tool-call, duration, and safety-limit information;
• fallback metadata if Advanced could not safely complete.

Trace questions for non-technical reviewers

Observation	Question to ask	Likely place to improve
Wrong initial owner	Was the visitor intent clear, and did the selected specialist have a distinctive scope?	Agent Description, Base entry Condition, or routing test coverage.
Correct owner then unnecessary move	Was the new message actually a follow-up or correction?	Stickiness, Switch Confidence, overlapping descriptions, or overly broad handoff.
No handoff count despite visible owner change	Did routing select the new owner before Workflow Started?	No defect if the Route Decision explains it; route and handoff are separate event types.
Tool not called	Were required details present and was the tool enabled, connected, and assigned to this owner?	Tool Usage Guidance, parameters, source tool state, or agent ownership.
Action status unknown	Could the destination have received the request before communication failed?	Verify externally. Do not repeat automatically. Review endpoint idempotency and reliability.
Safety limit reached	Is the workflow looping between agents or repeatedly requesting the same tool?	Handoff conditions, tool result instructions, route topology, or failing integration.
Fallback occurred	Was it before any output/action and what made Advanced unavailable?	Runtime health, package, OpenAI error, configuration, or transient service issue.

Conversation logs can contain visitor messages, responses, IP-related data, URLs, tool information, and timing details. Restrict access, apply a retention policy, and use synthetic screenshots in support material. See conversation analytics and privacy guidance.

Production rollout checklist

Basic Assistant and knowledge work correctly before Advanced is enabled. Every specialist has a distinct Description, playbook, non-goals, and exit. Important entries and specialist changes have explicit Conditions. Every enabled workflow tool is assigned, connected, enabled at source, and tested. Action recipients/endpoints, parameters, confirmation, cancellation, failure, and duplicate behavior were verified. Unsafe HTTP is off—or the exception is documented and deliberately approved. The routing regression set includes positive, negative, ambiguous, follow-up, correction, topic-change, and frustrated messages. Flow Checks have no blocking errors; every warning/note has been resolved or consciously accepted. Advanced Agents health is understood and has no unresolved errors. A rollback revision and human support route are available. A named admin will review traces and business outcomes after release.

Start with a narrow workflow and monitor it. Do not add specialists because the canvas looks sparse. Add them when traces and customer journeys show a distinct job that deserves separate instructions or capabilities.

agentic-flows-test-publish-operate

Screenshot capture pending /images/screenshots/agentic-flows/test-publish-operate.webp

Flow Simulator result, readiness checks, and live-version controls

Capture

Run a synthetic routing test with action execution off, show a matched expected handler, then keep Flow Checks and the Live Version card visible.

Show

Test message, Expected handler, Run action tools off, route decision, trace replay, Flow Checks, version note, and Publish Changes

Viewport

Desktop, 1440 × 1000

Annotate

Call out the safe default for Run action tools and the difference between route decision and handoff.

Redact

API keys, private URLs, personal data, real visitor messages, and production action details

The operating principle

OpenAI’s general observability guidance recommends using traces first to understand individual runs, then turning stable behavior into repeatable evaluation. Apply that principle here: investigate real failures in the trace, add a synthetic example to your saved test set, change one design variable, and run the set again. Production readiness is not a one-time publish button; it is a feedback loop your website team can understand and repeat.