Skip to main content

7. Data Transformations

Data transformations are a crucial component of Data Steward, allowing you to clean, standardize, enrich, and optimize your data. This section will guide you through the process of understanding, configuring, and applying transformations to your data submissions.

7.1 Overview of Transformation Process​

Data transformations in Data Steward follow a structured process:

  1. Transformation Templates: Global definitions of transformation types.
  2. Transformation Types: Organization-specific instances of transformation templates.
  3. Transformations: Actual application of a transformation type to a specific data submission.

This hierarchy allows for flexibility and customization while maintaining consistency across your organization.

7.2 Available Transformation Templates​

Data Steward offers several pre-defined transformation templates to address common data processing needs:

  1. Normalize Columns

    • Description: Standardizes column names to a defined format.
    • Use Case: Ensuring consistency in data structure across different submissions.

  2. Enrich Product Type

    • Description: Classifies product type from SKU and descriptions.
    • Use Case: Automatically categorizing products based on existing data.

  3. Enrich System Information

    • Description: Adds system-related details to product descriptions.
    • Use Case: Enhancing product data with standardized system specifications.

  4. Enrich CPU Information

    • Description: Enhances descriptions with CPU details.
    • Use Case: Adding or standardizing CPU-related information in product data.

  5. Enrich GPU Data

    • Description: Adds GPU-related information to descriptions.
    • Use Case: Enhancing product data with detailed GPU specifications.

7.3 Configuring Transformations for Your Organization​

To use transformations effectively, you need to configure them for your specific needs.

Creating a Transformation Type​

  1. Navigate to "Settings" > "Transformations" in the main menu.

  2. Click "Create New Transformation Type."

  3. Select the base Transformation Template.

  4. Provide a name and description for your transformation type.

  5. Configure the parameters specific to your needs. These may include:

    • Column mappings
    • Lookup tables
    • Regex patterns
    • Threshold values

  6. Save your new transformation type.

Assigning Transformations to Submission Types​

To apply transformations to incoming data:

  1. Go to "Submission Types" and select the relevant type.
  2. Navigate to the "Transformations" tab.
  3. Click "Add Transformation."
  4. Select the transformation type you want to apply.
  5. Set the order of application if using multiple transformations.
  6. Save your changes.

7.4 Monitoring Transformation Progress​

Once transformations are configured, you can monitor their application to your data submissions.

Viewing Transformation Status​

  1. Navigate to the "Submissions" section.
  2. Select a specific submission.
  3. Go to the "Transformations" tab to see:
    • List of applied transformations
    • Status of each transformation (Pending, Processing, Completed, Failed)
    • Execution time and resource usage

Transformation Logs​

For detailed information on the transformation process:

  1. In the submission's "Transformations" tab, click on a specific transformation.
  2. View the detailed log, which includes:
    • Start and end times
    • Number of records processed
    • Any warnings or errors encountered

Handling Failed Transformations​

If a transformation fails:

  1. Check the error message in the transformation log.

  2. Common issues include:

    • Unexpected data formats
    • Missing required fields
    • Lookup failures

  3. Adjust your transformation configuration or data as needed.

  4. Rerun the transformation on the submission.

7.5 Advanced Transformation Techniques​

Chaining Transformations​

You can apply multiple transformations in sequence:

  1. In the submission type configuration, add multiple transformations.
  2. Use the drag-and-drop interface to set the order.
  3. Output from one transformation becomes input for the next.

Custom Transformations​

For unique needs, you can create custom transformations:

  1. Go to "Settings" > "Custom Transformations."
  2. Click "Create Custom Transformation."
  3. Write your transformation logic using the provided scripting interface.
  4. Test your transformation thoroughly before deploying.

AI-Assisted Transformations​

Data Steward leverages AI for complex transformations:

  1. Enable AI assistance in your transformation configuration.

  2. The AI can help with tasks like:

    • Product categorization
    • Attribute extraction from unstructured text
    • Anomaly detection

  3. Review AI-suggested transformations before applying them.

7.6 Best Practices for Data Transformations​

  • Start with a clear understanding of your desired output data structure.
  • Use the simplest transformation that meets your needs to ensure efficiency.
  • Test transformations thoroughly with a variety of input data, including edge cases.
  • Monitor transformation performance and optimize as needed.
  • Regularly review and update your transformations to adapt to changing data patterns or business needs.
  • Document your transformation logic for future reference and knowledge sharing.
  • Use version control for your transformation configurations to track changes over time.
  • Leverage Data Steward's AI capabilities for complex transformation tasks, but always validate the results.

By effectively using Data Steward's transformation capabilities, you can ensure that your data is clean, consistent, and enriched with valuable insights. This transformed data forms a solid foundation for accurate analytics and informed decision-making in your semiconductor and high-tech manufacturing processes.