Data Orchestrator Agent
The Data Orchestrator Agent gives you a visual workspace to design, run, and monitor multi-step dataflows without writing code. It acts as the central control point for your data operations, managing the full lifecycle of your dataflows from the moment you build them to every subsequent run in production.
A dataflow in the orchestrator agent is a declarative representation of end-to-end data execution logic, where multiple data operations, pipelines, conditional branches, scripts, triggers, and dependencies are composed into a coordinated workflow to achieve a specific business outcome. It defines how data moves, transforms, and is processed across systems through a sequence of interconnected execution steps, including schema crawling, transformation pipelines, parallel task execution, conditional decision nodes, custom script execution, retries, error handling, and schedules.
The platform handles all sequencing, dependency checking, and task handoffs automatically. Before any pipeline starts, it verifies that all required inputs and connections are available. During execution, it continuously monitors workflow health and surfaces issues before they affect production. Your team stays focused on outcomes rather than on managing pipeline operations.
Key Capabilities
Visual Dataflow Builder - Design dataflows on a graphical canvas by adding and connecting task nodes.
Point-and-Click Execution - Run dataflows directly from the canvas without using a command line or separate tool. You trigger a run with a single action, and the execution engine handles the rest.
Integrity Validation - The agent verifies that all dependencies are met before a dataflow starts, preventing pipelines from failing mid-run due to missing inputs or unresolved connections.
Managed Execution - The platform executes the tasks seamlessly from start to finish. It handles sequencing, triggers, and task handoffs automatically so your team does not need to intervene at each step.
Dependency Maintenance - Throughout the entire automated workflow, the agent ensures all connections between pipeline steps remain intact.
To access this feature, go to the Home page > Data Agents > Data Orchestrator Agent.
Creating a Dataflow
This feature enables you to define and execute a dataflow that runs multiple tasks or pipelines to achieve the required data.
To create a new dataflow:
Go to the Home page > Data Agents > Data Orchestrator Agent.
A dataflow can be created using one of two methods:
Automatic - Use the platform agent to agentically generate a dataflow based on your natural language input. Refer to Creating a Dataflow with an Agent more information.
Manual - Create a dataflow manually with the platform interface.
To create a pipeline manually, click Add Dataflow.
The Add Dataflow window opens.
Enter a name for your dataflow.
Enter a brief description of what this dataflow does.
Select the target Knowledge Cluster from the dropdown.
Click Add.
The dataflow is saved, and its canvas opens automatically. Refer to Creating a Dataflow Diagram for the dataflow execution process.
Creating a Dataflow with an Agent
The platform's agent-assisted dataflow creation lets you create and manage dataflow configurations through natural language interaction. Describe your requirements in plain language, and the agent analyzes your input, requests clarification when needed, and automatically generates the dataflow with its corresponding diagram and configuration.
You can also review and manually edit the generated diagram and configuration after you save the dataflow, giving you full control over dataflow behavior and output.
To access the feature:
Select a Knowledge Cluster from the dropdown.
Click Agent.
The agent interaction panel opens on the left side of the screen.
Describe your requirements in plain language, including the data you want to use and the outcome you want to achieve, and then press Enter.
Example: Create a dataflow that pulls new support tickets every morning, removes duplicate entries, and loads the results into the Support knowledge cluster.
The agent processes your input and asks clarifying questions if it needs more information. Once the agent has sufficient context, it identifies the most relevant pipelines within the selected KC by running a similarity search across all available pipelines, then displays them with a matching score for your approval.
Click the Info
icon next to Score to view the reason for the matching score.If the identified pipelines are incorrect, click Reject to discard the agent's analysis. Continue the conversation to provide more context about your requirements.
Review the identified pipelines, select the required ones, and click Approve.
The agent generates dataflow diagrams based on data lineage to meet your requirements and displays them in the right panel.
If the generated dataflow diagram is incorrect, click Discard Design to clear it. Continue the conversation to provide more context about your requirements.
If the dataflow diagrams require refinement, simply describe the changes in plain language, and the agent will update them accordingly.
Once the dataflow diagram is finalized, click Save Dataflow in the right-side panel.
The Add Dataflow panel opens.
Note
If the generated dataflow diagram is incorrect, click Discard Design to clear it. Continue the conversation to provide more context about your requirements.
Update the dataflow Name and Description.
The Knowledge Cluster is pre-selected based on your earlier selection.
Click Add.
A success message is displayed, and the new dataflow is created with Version 1. This version is active by default.
Tip
You can manually update the dataflow diagram. Refer to Creating a Dataflow Diagram for more information.
You can execute the dataflow to generate the data. Refer to Running a Dataflow for more information.
To update, continue the conversation with the Agent to provide context about your requirements. After the agent updates the dataflow, click Update Dataflow.
The update dataflow dialog box opens:
To update the current version, click Update Active Version.
To create a new version, click Save as New Version.
Use the available agent controls to manage your session:
Clear Chat - Clears the current conversation for a fresh start.
Copy
icon - Copies the chat message to the clipboard.
Creating a Dataflow Diagram
A dataflow diagram helps visualize how data moves from a source through a series of processing steps into a Knowledge Cluster.
To create a dataflow diagram:
Open the Dataflow.
The canvas opens and displays the Version tab by default.
Drag the required task nodes from the left panel onto the canvas. Refer to Task Node Types for more information.
Arrange the nodes sequentially or in parallel based on data requirements for business use cases.
Connect adjacent nodes by drawing an arrow from the output port of one node to the input port of the next.
The arrow defines the direction of data flow.
Select each node and configure its settings as required. Double-click a node to open its configuration settings in the right panel.
After the configuration, you need to execute the dataflow. Refer to Running a Dataflow for more information.
Task Node Types
A task node represents a single step in a Dataflow and defines the operation performed at that point in the execution sequence. Nodes are connected in a sequence to form the full Dataflow logic, and each node type serves a distinct purpose with its own configuration options.
The following task node types are available in the platform:
Schema Crawl - Connects to a configured data source and crawls its schema. Use this node to detect structural changes in the source before running downstream pipelines.
Unstructured Dataflow - Executes an existing unstructured data processing flow as a step within the current Dataflow. Use this node when your orchestration sequence includes unstructured document processing.
Pipeline Build - Executes a pre-configured data pipeline. Use this node to run ingestion, transformation, or enrichment pipelines within a Dataflow sequence.
Decision - Routes execution to one of two downstream branches based on whether the preceding node succeeded or failed. Use this node to implement conditional logic, such as triggering an alert when a pipeline fails or skipping a cleanup step when all preceding steps succeed.
Run Script - Executes a custom script. Use this node for actions not covered by other node types.
Barrier - A placeholder node that performs no operation; it just serves as the convergence point that holds execution until all branches are done. Use this node when you have multiple upstream nodes running in parallel, and you want to wait for all of them to finish before proceeding. You place a Barrier node downstream and configure its Run Settings to execute only after all upstream nodes have completed.
The Barrier node has two distinct uses:
No-op terminal - the "do nothing" branch in a Decision node.
Merge/sync point - wait for all parallel upstream nodes to finish before the Dataflow continues.
Task Nodes Configuration
Each task node has a settings panel with three tabs:
Configuration tab
Dataflow Variables tab
Run Settings tab
Use these tabs to view and manage detailed information about the node.
Note
Mandatory fields are marked with 
.
 |
Task Nodes | Fields | Values |
|---|---|---|
Schema Crawl | Name | Update this field to assign a meaningful name that reflects the node's role in the Dataflow. |
Description | Enter a short description of what this node does. | |
Source | From the dropdown, select the structured KB source. The list displays KBs associated with the selected Knowledge Cluster (KC). | |
Unstructured Dataflow | Name | Update this field to assign a meaningful name that reflects the node's role in the Dataflow. |
Description | Enter a short description of what this node does. | |
Dataflow | From the dropdown, select the unstructured KB source. The list displays KBs associated with the selected KC. | |
Pipeline Build | Name | Update this field to assign a meaningful name that reflects the node's role in the Dataflow. |
Description | Enter a short description of what this node does. | |
Pipeline | From the dropdown, select the pipeline. The list displays pipelines associated with the selected KC. If the list is empty, no pipelines have been created for the KC yet. | |
Decision | Name | Update this field to assign a meaningful name that reflects the node's role in the Dataflow. |
Description | Enter a short description of what this node does. | |
Mode |
| |
On Success - Go to task | From the dropdown, select the task node to execute when the preceding node completes successfully. Available when Simple mode is selected. | |
On Failure - Go to task | From the dropdown, select the task node to execute when the preceding node fails. Available when Simple mode is selected. | |
Route | Routes define the available output paths from this node. By default, Route 1 is created automatically.
Available when Advanced mode is selected. | |
Go to Node | For each route, select an existing node from the dropdown. This is the node the Dataflow executes when the route's expression evaluates to true. Available when Advanced mode is selected. | |
Expression | For each route, enter the condition that determines when this route is taken. The Dataflow evaluates each route's expression at runtime and follows the first route whose expression returns true. Available when Advanced mode is selected. | |
Run Script | Name | Update this field to assign a meaningful name that reflects the node's role in the Dataflow. |
Name | Enter a short description of what this node does. | |
Image URL | Enter the URL of the container image to use when executing the script. | |
Script Mode | Select how the script is provided.
| |
Script | Enter the script to execute directly in the text editor. Available when Script mode is selected. | |
Script File | Click the upload area or drag and drop a file to upload the script to execute. Available when Script File mode is selected. | |
Barrier | Name | Update this field to assign a meaningful name that reflects the node's role in the Dataflow. |
Description | Enter a short description of what this node does. |
The Dataflow Variables tab displays two categories of variables associated with the node.
 |
Auto Generated Variables - Displays variables created automatically after the node executes. These variables can be referenced by downstream nodes in the Dataflow.
Reserved Variables - Displays the input variables passed into this node from other task nodes in the Dataflow. These are grouped by source node and can be referenced at runtime within the script.
 |
Field | Description |
|---|---|
Number of Retries | Enter the number of times the node should retry execution if it fails. Set to 0 to disable retries. |
Delay between Retries | Enter the wait time in seconds between each retry attempt. This field is active only when the number of retries is greater than 0. |
Execute When | Choose the required options:
|
Click Save to save all configurations.
Running a Dataflow
After you finish setting up your Dataflow diagram and its configuration, you are ready to run it. When you run a Dataflow, the platform automatically processes each pipeline in sequence, following the order you defined in the canvas.
Select the KC from the dropdown to view its associated Dataflows.
Select a Dataflow from the list to open the canvas.
Select the Active version from the dropdown.
Important
The Run button is enabled only for the active SQL version. If you select a version that is not active, Run is disabled. To run a query, first set the version as active.
Click Run.
A confirmation message appears, and the Dataflow job is successfully submitted. The platform begins executing each pipeline in the order defined by your connections.
Note
After submitting the run, switch to the Runs tab to track the progress and status of your Dataflow execution in real time.
Click the dropdown next to Run.
Tip
Use the Schedules History option to view a complete list of schedulers and their execution details.
Click Schedule.
The Schedule window opens.
Enter relevant details in the fields.
Field
Description
Name
Enter a name for this schedule.
Description
Enter additional details about the purpose or scope of the schedule.
Schedule Type
Select how frequently the schedule should run:
Hourly
Daily
Weekly
Monthly
Yearly
Repeat Every
Specify the interval for the recurring schedule.
Hourly schedule - 1 to 23
Daily schedule - 1 to 31
Notice
If the scheduler tries to start a new job while a previous one is still running, the system will cancel the new job to avoid overlapping executions.
Days of the week
Select one or more days of the week to run the schedule.
Available for a Weekly schedule only.
Day of the Month
Select the specific date of the month to run the schedule.
Available for Monthly schedules only (1 to 30).
Day and Month
Select the specific month and date to run the schedule.
Available for Yearly schedules only.
Start Time
Set the time when the schedule should begin.
Timezone
Select your timezone to ensure the schedule runs at the correct local time.
Click Create.
The scheduled run is created, and dataflow execution will trigger automatically according to your selected schedule.
Tip
Follow the same procedure to update the existing dataflow schedule.
Managing a Dataflow Version
The platform supports multi-version management for dataflows. This versioning model allows you to maintain parallel configurations of the same dataflow, enabling controlled experimentation and safe iteration without impacting production execution. At any point, one version is designated as active; only the active version executes during both manual and scheduled runs.
Versions are created by copying an existing version as a base. Each version maintains an independent configuration state, including the canvas diagram, task node configurations, and run settings.
Create a Dataflow Version
You can create additional dataflow versions based on any existing version.
Important
Make sure you have already created at least one base dataflow version.
To create a new dataflow version:
Click Add.
The Create New Version window opens.
From the dropdown, select the base version you want to copy from.
To make this new version the active dataflow, enable the Make Active option.
Click Create.
The new version is created from the base version you selected.
Edit the dataflow diagram and its configuration as needed:
Manual edits - Update the diagram directly. The platform auto-saves your changes.
Agent-assisted edits - Continue the conversation with the Agent to provide context about your requirements. After the agent updates the dataflow, click Update Dataflow.
The update dataflow dialog box opens:
To update the current version, click Update Active Version.
To create a new version, click Save as New Version.
Set a Dataflow Version as Active
Only one dataflow version can be active at a time. To change the active version:
From the version dropdown, select the dataflow version you want to activate.
Click the Ellipsis
icon next to Add.Note
The dropdown options are disabled for the active dataflow version.
From the dropdown, select Mark this Version Active.
The selected version is now set as the active dataflow version.
View Dataflow Version Audits
The platform keeps a complete history of every change made to a dataflow version, including what changed, when it changed, and which events occurred.
To view the dataflow version audit:
Click the Ellipsis
icon next to Add Click Audits.
The Dataflow Audits window opens.
Select a version from the Version dropdown.
The audit log updates to display the change history for the selected version.
Delete a Dataflow Version
Note
You cannot delete the active version. Set a different version as active before deleting.
To remove the version:
From the version dropdown, select the version you want to remove.
Click the Ellipsis
icon next to Add.From the dropdown, select Delete this Version.
The confirmation dialog box opens.
Click Delete.
The selected dataflow version is removed.
Viewing Dataflow Runs History
The platform allows you to review past and current executions of a Dataflow. You can check the status of each run, cancel an in-progress run, restart a failed run, and download logs for troubleshooting.
To view Dataflow runs:
Select a KC from the dropdown to view its associated Dataflows.
Select the Dataflow from the list.
The dataflow canvas opens in the Version tab.
Switch to the Runs tab.
All runs and their details are displayed.
Search for and select the required run from the left panel.
The details appear in the right panel.
You can do the following actions on this tab:
If a run is currently executing, click Cancel to stop it.
Note
Canceling a run stops all active tasks immediately. Tasks that have already been completed are not affected.
If a run has failed, click Restart to try again.
Tip
Review the run logs before restarting to understand why the run failed.
To export the logs for all nodes in a specific run, click Download Logs.
Double-click the node to view its Configuration details on the right panel.
Switch to the Logs tab to view logs related to the selected node run.
To view logs for that specific attempt, select the attempt number from the Try Number dropdown.
To copy the logs for that attempt, click Copy All.
To export all attempt logs for a specific node, click Download Logs.
Modifying a Dataflow Configuration
The platform allows you to modify a Dataflow and its configuration at any time.
To update a Dataflow:
Select the KC from the dropdown to view its associated Dataflows.
Select a Dataflow from the list.
The canvas opens in the Version tab.
Update the diagram as needed. Refer to Creating a Dataflow Diagram for more information.
Switch to the Config tab.
Update the Dataflow name and description as needed.
The Knowledge Cluster is pre-selected.
Click Save to save the updated changes.
Viewing Dataflow Audits
The Audits tab records all changes to the dataflow, allowing you to review and validate dataflow modifications. It supports change tracking and validation for administrators and technical users responsible for maintaining the integrity of the dataflow across the platform.
Switch to the Audits tab. It displays a list of all the dataflow changes, sorted with the most recent change at the top. Each entry shows the type of change and the date and time it occurred.
Delete a Dataflow
The platform allows you to delete a Dataflow associated with a Knowledge Cluster.
To delete a Dataflow:
Select a KC from the dropdown to view its associated Dataflows.
Select the Dataflow from the list and click the Delete
icon on the right.A confirmation dialog appears.
Click Delete.
The Dataflow is removed from the list.