S3 File Output
- 1 Overview
- 2 Plugin Configuration
- 2.1 General
- 2.2 Content
- 2.2.1 Fields
- 2.2.2 Example
- 2.2.3 Understanding how files are formatted
Overview
Used to output data from DDE to a file in an AWS S3 Bucket
Can be used as an integration point with third party systems e.g. Data warehouse etc
Plugin Configuration
General
Within the General tab:
The following settings can be configured:
Setting | Description |
---|---|
Access key | The AWS Access Key used to authenticate the requests sent to Amazon S3 |
Secret key | The AWS Secret Key used to authenticate the requests sent to Amazon S3 |
Bucket | The S3 buckets that the data should be stored in |
Bucket Region | The region the bucket is stored in e.g. eu-west-1 |
File prefix | The prefix of the filename output to S3 |
File prefix (Field/Param) | Allows you to add a File prefix via a Field or Parameter |
Include Date/Time | Allows the selection of various Date / Time values to add to the Filename* |
StepNr | Adds the start step in case more than one step could start the KTR file. This provides another element of uniqueness. On by default. |
Node Name | Adds the Node Name (IP) to the File name. This provides another element of uniqueness if files are being output quickly from different nodes. On by default. |
Folder (optional) | The sub-folder within the S3 bucket where the data should be output |
Extension | The extension of the file that should be output e.g. csv, txt |
Buffer Size | During rule development it is useful to have a fast buffer retention period so that data written out by rules appears frequently in the S3 bucket. However when the rule is in production a short buffer period can cause a large number of files to accumulate. This Buffer Size setting allows you to adjust the Buffer and the frequency the data is passed to S3 |
Time Period (Mins) | Allows a time period to be set to control when the data is written to S3 |
Write only if | Choose to write only if the data is valid or not. See data validity. |
File Uploaded to S3 step | When either buffer is met you can choose to optionally call out to another rule which can then trigger to handle the exported file. We pass some fields through the the output step that can be used to identify the new file. These include:
Loopcount is included as it’s required for the DDE Rule step, but I’m it’s configured to always output 1.
|
This was made configurable in August 2020. Prior to this the mandated Date / Time value made duplicates filenames extremely unlikely and therefore files could not be overwritten. Being able to set this and other values and choose to overwrite a file was advantageous so these changes were made. Customers should check the filename selections to ensure they are unique enough in a CX environment to not overwrite if this is not required.
Content
Within the Content tab:
The following settings can be configured:
Setting | Description |
---|---|
Separator | The delimiter that is used to separate the values within the file.
To use a TAB as a delimiter select the “Insert TAB” button |
Enclosure | The data qualifier enclosing the data
|
Header | If “Header” is selected the first row of data within the output file created will contain the field names output. |
Fields
The Fields tab is required to be populated with the fields that are required to be output to the file:
Example
The following table illustrates an example of how data will be exported to S3 with certain setting applied:
Content | Fields | Example Data Output |
---|---|---|
Field1,Field2,Field3
1,"A","A"
2,"B","B"
3,"C","C" |
Understanding how files are formatted
The rule files will be formatted as you specify in the step configuration, however the file names will be different when executed within . Example:
Description | File name |
---|---|
File prefix | MyFile_clientname_s3output_3_172.31.11.190_20180202_114856_0_6.txt |
DDE Client name | MyFile_clientname_s3output_3_172.31.11.190_20180202_114856_0_6.txt |
Rule name | MyFile_clientname_s3output_3_172.31.11.190_20180202_114856_0_6.txt |
Rule version | MyFile_clientname_s3output_3_172.31.11.190_20180202_114856_0_6.txt |
Node Name (internal) IP address | MyFile_clientname_s3output_3_172.31.11.190_20180202_114856_0_6.txt |
Inlcude Date / Time stamp | MyFile_clientname_s3output_3_172.31.11.190_20180202_114856_0_6.txt |
File count (Split every X rows) | MyFile_clientname_s3output_3_172.31.11.190_20180202_114856_0_6.txt |