Invalid Data S3 Output
- John Mitchell
- Ben Clark
If data fails validation you will want to understand why, and understand how it got into the system.
This Template uses the validation field as provided by the How to validate your data through the GET CXID as a flag and simply exports out of the Rule any data rows that have failed validation. The parameters below allow the configuration of the AWS S3 bucket. A buffer size and time period can be set to ensure the data is transfered in suitable sized chunks.
Input Data
Field | Description | Example |
|---|---|---|
AlterianCX_isvalid | Added by the Rule start Tile the RIS or the PFE and updated by the GET CXID (How to validate your data through the GET CXID ) to show whether this invocation of the Rule has failed validation at any point or not. This step identifies whether this invocation has a value of N and if it does it will export the data for review. | AlterianCX_isvalid = Y - Validation was passed and will not be exported AlterianCX_isvalid = N- Validation was not passed and will be exported AlterianCX_isvalid = null - Validation has not yet been run so this row will not be exported |
Configuration
The configuration items are focused on connecting to AWS and breaking the file outputs into suitable chunks.
Parameter | Description | Example |
|---|---|---|
Access Key | Access Key |
|
Bucket | The name of the S3 Bucket the output should be written to |
|
Buffer Size | Number of rows to buffer before writing the file (Default to 5) | 100 |
Enclosure | Field Enclosure |
|
Extension | file Extension |
|
File Prefix | File name | myfilename |
Folder | Folder the output should be written to |
|
Region | The AWS region to use when connecting to the Bucket | eu-west-2 |
Secret Key | Secret Key |
|
Separator | Row Delimitor |
|
Time Period | Time Period in Minutes to buffer rows before writing to file. Works with Buffer size | 60 |
Processing
This tile template identifies whether the data in this invocation is marked as Valid or not. If it is marked as Invalid (AlterianCX_isvalid= N) it will export the row to AWS S3 following the configurations Rules set above.
You can then review the validation failures to understand how you should change the data being supplied or the Rule to allow it to work better.
Field | Description | Example |
|---|---|---|
|
|
|