Overview

The Examples feature in Datograde enables teams to collect, organize, and manage real-world or synthetic examples for fine-tuning, testing, and training their Generative AI pipelines. Examples act as a bridge between model behavior, prompts, and pipeline outputs, allowing teams to iteratively refine their Generative AI pipelines based on real data.

---

What Are Examples?

Examples represent specific input-output pairs that you use to:

• Train or fine-tune AI models.
• Validate how models respond to various inputs.
• Benchmark and compare prompt or pipeline performance.

Each example consists of:

• Input Data: The raw data being processed (e.g., a transcript, image, or JSON object).
• Expected Output: The ideal or correct result for the given input (e.g., a summary, label, or transformed data).
• Metadata: Tags, annotations, or context for categorization and analysis.

---

Key Features of Examples

1. Centralized Example Management:

   • Store and access all examples in one organized location.
   • Filter and search examples by tags, use case, or metrics.

2. Annotations and Feedback:

   • Add notes or labels to examples to document edge cases or problem areas.
   • Collaborate with your team to refine expectations.

3. Testing and Validation:

   • Run examples through prompts and pipelines to evaluate performance.
   • Compare actual outputs with expected outputs to identify gaps.

4. Integration with Prompts and Pipelines:

   • Use examples to validate prompt logic or pipeline configurations.
   • Collect data from pipeline failures to create new examples for debugging or training.

5. Export and Reuse:

   • Export curated examples for model fine-tuning or retraining workflows.
   • Share datasets across teams or projects.

---

Use Cases for Examples

• Fine-Tuning: Collect examples to train models on domain-specific tasks or edge cases.
• Testing and Debugging: Validate AI behavior against real-world or synthetic inputs.
• Performance Monitoring: Continuously track how well prompts or pipelines perform on benchmark examples.
• Collaborative Improvement: Build a shared library of examples that reflect evolving business needs.

---

Examples Workflow Overview

1. Collect Examples:

   • Manually upload or import examples from logs, datasets, or real-world interactions.
   • Annotate and categorize examples with tags and metadata.

2. Test Examples:

   • Run examples through prompts or pipelines to evaluate performance.
   • Compare actual outputs to expected outputs.

3. Analyze Results:

   • Use discrepancies between actual and expected outputs to debug or refine system behavior.
   • Highlight and address recurring failure patterns.

4. Refine and Export:

   • Update examples with improved annotations or expectations.
   • Export selected examples for retraining or team collaboration.