/
S3 File Output

S3 File Output

Owned by Ben Clark
Last updated: Oct 04, 2023
4 min read

Overview

 

  • Used to output data from DDE to a file in an AWS S3 Bucket

  • Can be used as an integration point with third party systems e.g. Data warehouse etc

Plugin Configuration

General

Within the General tab:

 

The following settings can be configured:

Setting

Description

Setting

Description

Access key

The AWS Access Key used to authenticate the requests sent to Amazon S3

Secret key

The AWS Secret Key used to authenticate the requests sent to Amazon S3

Bucket

The S3 buckets that the data should be stored in

Bucket Region

The region the bucket is stored in e.g. eu-west-1

File prefix

The prefix of the filename output to S3

File prefix (Field/Param)

Allows you to add a File prefix via a Field or Parameter

Include Date/Time

Allows the selection of various Date / Time values to add to the Filename*

StepNr

Adds the start step in case more than one step could start the KTR file. This provides another element of uniqueness. On by default.

Node Name

Adds the Node Name (IP) to the File name. This provides another element of uniqueness if files are being output quickly from different nodes. On by default.

Folder (optional)

The sub-folder within the S3 bucket where the data should be output

Extension

The extension of the file that should be output e.g. csv, txt

Buffer Size

During rule development it is useful to have a fast buffer retention period so that data written out by rules appears frequently in the S3 bucket. However when the rule is in production a short buffer period can cause a large number of files to accumulate. This Buffer Size setting allows you to adjust the Buffer and the frequency the data is passed to S3

Time Period (Mins)

Allows a time period to be set to control when the data is written to S3

Write only if

Choose to write only if the data is valid or not. See data validity.

File Uploaded to S3 step

When either buffer is met you can choose to optionally call out to another rule which can then trigger to handle the exported file.

We pass some fields through the the output step that can be used to identify the new file. These include:

  1. Time Stamp

  2. loopcount

  3. File Name

  4. Folder Name

  5. Bucket Name

Loopcount is included as it’s required for the DDE Rule step, but I’m it’s configured to always output 1.


As a sequence Number isn’t available, the steps that can be used with the output are limited (as a lot of steps require Sequence Number), so DDE Rule step will most likely be the sensible route.

 

  • This was made configurable in August 2020. Prior to this the mandated Date / Time value made duplicates filenames extremely unlikely and therefore files could not be overwritten. Being able to set this and other values and choose to overwrite a file was advantageous so these changes were made. Customers should check the filename selections to ensure they are unique enough in a CX environment to not overwrite if this is not required.

Content

Within the Content tab:

The following settings can be configured:

Setting

Description

Setting

Description

Separator

The delimiter that is used to separate the values within the file.

  • i.e. Comma (,), Pipe (|) etc

To use a TAB as a delimiter select the “Insert TAB” button

Enclosure

The data qualifier enclosing the data

  • i.e. Double Quotes (“)

Header

If “Header” is selected the first row of data within the output file created will contain the field names output.

Fields

The Fields tab is required to be populated with the fields that are required to be output to the file:

Example

The following table illustrates an example of how data will be exported to S3 with certain setting applied:

Content

Fields

Example Data Output

Content

Fields

Example Data Output

Field1,Field2,Field3 1,"A","A" 2,"B","B" 3,"C","C"

Understanding how files are formatted

The rule files will be formatted as you specify in the step configuration, however the file names will be different when executed within . Example:

Description

File name

Description

File name

File prefix

MyFile_clientname_s3output_3_172.31.11.190_20180202_114856_0_6.txt

DDE Client name

 MyFile_clientname_s3output_3_172.31.11.190_20180202_114856_0_6.txt

Rule name

 MyFile_clientname_s3output_3_172.31.11.190_20180202_114856_0_6.txt

Rule version

 MyFile_clientname_s3output_3_172.31.11.190_20180202_114856_0_6.txt

Node Name (internal) IP address

 MyFile_clientname_s3output_3_172.31.11.190_20180202_114856_0_6.txt

Inlcude Date / Time stamp

 MyFile_clientname_s3output_3_172.31.11.190_20180202_114856_0_6.txt

File count (Split every X rows)

 MyFile_clientname_s3output_3_172.31.11.190_20180202_114856_0_6.txt

Related content