Inspecting and Debugging Pipeline Steps

When data issues or unexpected outputs occur, the Pipeline Steps view helps you debug each stage of your pipeline. This feature allows you to inspect inputs, outputs, and processing logic step-by-step.

---

Viewing Pipeline Steps

1. Select a pipeline from the Pipelines Dashboard.
2. Open the Pipeline Map and click on a specific step.
3. The Step Details View shows:
   • Input Data: Data entering the step.
   • Transformation Logic: The operations applied to the data.
   • Output Data: Data produced by the step.

---

Debugging Issues in a Step

1. Use the Execution Logs tab to:
   • View logs for each execution of the step.
   • Identify errors or anomalies in the input/output data.
2. Use the Preview Transformation tool to:
   • Test the step with sample data.
   • Observe how the transformation logic is applied.

---

Manual Overrides

1. Click the Override Input button in the Step Details View.
2. Edit the input data for the step directly.
3. Rerun the pipeline from this step to test the effect of the changes.
   • If the issue is resolved, apply the fix to the pipeline configuration.

---

Common Debugging Scenarios

• Error Case: Missing or malformed input data.
  • Solution: Inspect the upstream step for issues and adjust transformation logic.
• Error Case: Latency spikes in a specific step.
  • Solution: Optimize the logic or parallelize processing.
• Error Case: Unexpected final outputs.
  • Solution: Trace the data through intermediate steps to identify where the error originates.

---

Next Steps

• Monitoring Pipeline Performance