Skip to main content

6. Data Submissions

Data submissions are at the core of Data Steward's functionality. This section will guide you through the process of submitting data, understanding the submission lifecycle, reviewing and approving submissions, and handling any issues that may arise during the process.

6.1 Submitting Data​

Data can be submitted to Data Steward through various methods, catering to different user needs and system integrations.

Manual Upload​

For users submitting data directly through the Data Steward interface:

  1. Navigate to the "Submissions" section in the main menu.
  2. Click "New Submission."
  3. Select the appropriate Submission Type from the dropdown menu.
  4. Choose the file you want to upload. Ensure it meets the format requirements specified in the submission type.
  5. Add any necessary metadata or comments about the submission.
  6. Click "Submit" to initiate the submission process.

API Submission​

For automated or high-volume submissions, use the Data Steward API:

  1. Authenticate using your API credentials.
  2. Prepare your data according to the API specifications.
  3. Make a POST request to the submissions endpoint with your data and metadata.
  4. Check the API response for the submission ID and status.

Refer to the API documentation for detailed instructions on using the API for submissions.

Scheduled Imports​

For regular data imports from external systems:

  1. Go to "Settings" > "Integrations."
  2. Set up a connection to your data source (e.g., SFTP server, database).
  3. Configure the import schedule and mapping to the appropriate submission type.
  4. Data Steward will automatically create submissions based on this schedule.

6.2 Submission Lifecycle​

Understanding the lifecycle of a submission helps you track its progress and handle any issues that may arise.

Submission Statuses​

  1. AWAITING_PREPROCESSING: Initial state when a submission is created but not yet ready for processing.
  2. PENDING: The submission is queued for processing.
  3. PROCESSING: Data Steward is actively validating and transforming the data.
  4. PROCESSED: Initial processing is complete; the submission is ready for review.
  5. FAILED: An error occurred during processing that requires attention.
  6. VENDOR_REVIEW: (If applicable) The processed data is awaiting vendor approval.
  7. VENDOR_APPROVED: The vendor has approved the processed data.
  8. VENDOR_REJECTED: The vendor has rejected the processed data.
  9. ORG_REVIEW: The submission is awaiting final organizational approval.
  10. ORG_APPROVED: The organization has approved the submission.
  11. ORG_REJECTED: The organization has rejected the submission.
  12. READY_FOR_DELIVERY: The submission is approved and ready to be exported or used.
  13. DELIVERED: The data has been successfully exported or integrated into target systems.

Tracking Submission Progress​

  1. Go to the "Submissions" section in the main menu.
  2. Use filters to find the submission you're interested in.
  3. Click on the submission to view its details page.
  4. The status bar at the top shows the current stage in the lifecycle.
  5. Scroll down to see a detailed log of each stage the submission has passed through.

6.3 Reviewing and Approving Submissions​

Proper review and approval of submissions ensure data quality and accuracy.

Vendor Review (if applicable)​

If your process includes a vendor review step:

  1. Vendors receive a notification when a submission is ready for their review.
  2. They log into their Data Steward portal to access the processed data.
  3. Vendors can:
    • Approve the submission if everything looks correct.
    • Reject the submission and provide comments for necessary changes.
    • Request additional information or clarification.

Organizational Review​

For the final organizational review:

  1. Navigate to the "Submissions" section and filter for submissions in "ORG_REVIEW" status.
  2. Click on a submission to review its details.
  3. Check the validation results, transformation logs, and final data output.
  4. Review any comments or notes from previous stages.
  5. Decide to:
    • Approve the submission: Click "Approve" and add any necessary comments.
    • Reject the submission: Click "Reject," provide a reason, and add any instructions for resubmission.
    • Request changes: Use the comments feature to ask for specific modifications before approval.

Bulk Approvals​

For efficiency in handling multiple similar submissions:

  1. In the "Submissions" list, use checkboxes to select multiple submissions.
  2. Click "Bulk Action" and choose "Approve" or "Reject."
  3. Confirm your action and add any general comments that apply to all selected submissions.

6.4 Handling Submission Issues​

Despite validation checks, issues may sometimes arise with submissions. Here's how to handle them:

Addressing Failed Submissions​

  1. Navigate to the "Submissions" section and filter for "FAILED" status.
  2. Click on the failed submission to view details.
  3. Check the error log to understand the cause of failure.
  4. Common actions include:
    • Correcting data: If the issue is with the data, correct it and resubmit.
    • Adjusting validation rules: If the failure is due to overly strict validation, you may need to adjust the rules in the submission type configuration.
    • Handling exceptions: For unique cases, you might need to create an exception process.

Data Quality Issues​

For submissions with data quality concerns:

  1. Review the validation warnings and errors in the submission details.
  2. Use the data preview feature to examine problematic records.
  3. Decide whether to:
    • Approve with notes: Accept the data but flag it for future reference.
    • Reject and request corrections: Send it back to the submitter for improvements.
    • Apply manual corrections: For minor issues, you may be able to edit the data directly in Data Steward.

Audit Trail​

Every action taken on a submission is logged:

  1. In the submission details page, navigate to the "Audit Log" tab.
  2. Here you can see a chronological list of all actions, including who performed them and when.
  3. Use this for troubleshooting, compliance purposes, or understanding the submission's history.

6.5 Best Practices for Data Submissions​

  • Establish clear guidelines for data preparation and submission for all data providers.
  • Regularly review and update validation rules to catch common errors early in the process.
  • Use automation where possible to reduce manual errors and increase efficiency.
  • Implement a thorough review process, especially for critical data submissions.
  • Maintain open communication channels with vendors and internal teams to quickly resolve any data issues.
  • Regularly analyze submission trends to identify recurring problems or opportunities for process improvement.
  • Use Data Steward's reporting features to monitor submission quality, timeliness, and other KPIs.

By effectively managing the data submission process, you ensure that high-quality, accurate data flows through your organization, enabling better decision-making and operational efficiency. In the next section, we'll explore how to leverage Data Steward's powerful data transformation capabilities to further enhance your data.