You can't use this until you register with us!



postgres database nomnomdata


Description

Used for running sql queries on a Postgres database. Uses the task execution system to keep track of ranges and batches.


Purchasing Options



Available Actions

Run

Execute SQL steps.


Variables

target_db required

Database where you want to run the script on.


sql_steps required

SQL Steps to be run. SQL Steps separated by a ;. Ranges are allowed. To learn more about ranges visit ...


Parameter Info
user_params

If in your sql you use any parameters then you need to create a parameter list here. To use ranges visit ...

user_range

Used to create a sliding window of data to be processed Json formatted. Documentation here...


Run - Sliding Date Window

Execute SQL steps. Based on configurable sliding date window.


Variables

target_db required

Database where you want to run the script on.


sql_steps required

SQL Steps to be run. SQL Steps separated by a ;. Can reference :start_date :end_date


Parameter Info
user_params

If in your sql you use any parameters then you need to create a parameter list here. To use ranges visit ...


User Range
range_data_offset

Time offset at which data becomes available. Example 1 Day

range_step_size

How large the chunk sizes should be.

range_start_date

When to start backfill from.

range_end_date

When to end data collection, leave blank for ongoing.

range_extra_collections

JSON list describing which time ranges of data to collect. Can be contain multiple ranges


range_current_pointer

The current date this task is at. Leaving blank to initalize with start_date, will be updated as task progresses.


Run - Sliding Integer Window

Execute SQL steps. Based on configurable sliding integer window.


Variables

target_db required

Database where you want to run the script on.


sql_steps required

SQL Steps to be run. SQL Steps separated by a ;. Can reference :start_i :end_i


Parameter Info
user_params

If in your sql you use any parameters then you need to create a parameter list here. To use ranges visit ...


User Range
irange_step_size required

How large the chunk sizes should be.

irange_start_i

When to start backfill from.

irange_start_i_query

Optional SQL query to use to instead of Start Index

irange_end_i

End Index

irange_end_i_query

Optional SQL query to use to instead of End Index


range_current_pointer

The current date this task is at. Leaving blank to initalize with start_date, will be updated as task progresses.


Export

Export data from Postgres database to various locations. SQL Steps stored in git


Variables

target_db required

Database where you want to run the script on.


sql_steps required

Single step to be run. Treated as one statement regardless of ;


S3 Info
aws_connection

AWS Credentials to be used for all things AWS related in this task.

aws_iam

AWS IAM Credentials to be used for all things AWS related in this task.

s3_bucket

S3 Bucket where you want to unload the data to.


SFTP Info
sftp_credentials

SFTP Credentials with access target SFTP host

sftp_host

Target SFTP host.

sftp_port

Target SFTP host Port. Format is hostname:port. Port optional if default port is Ok.


File/Formatting Options
file_name required

Name of the file.

format_type required

Format type that you want to export the data as. Can be delimited or json

delimiter

Delimiter you want to use to separate columns. Default is tab delimited

compression

Compression options, zip, gzip, bz2 are currently supported. Default is gzip.

null_as

The string you want to indicate a null value. Default is empty space AKA empty space.

escape

Escape character used to preceed special characters. Default is backslash

allow_overwrite

Overwrite files with the same name

add_quotes

For delimited only. Quote strings. Recommended espcially if you may have double quotes in your data.


Parameter Info
user_params

If in your sql you use any parameters then you need to create a parameter list here. To use ranges visit ...

user_range

Used to create a sliding window of data to be processed Json formatted. Documentation here...


Export - Sliding Date Window

Export data from Postgres database to various locations. SQL Steps stored in git. Based on configurable sliding date window.


Variables

target_db required

Database where you want to run the script on.


sql_steps required

Single step to be run. Treated as one statement regardless of ;


Parameter Info
user_params

If in your sql you use any parameters then you need to create a parameter list here. To use ranges visit ...


User Range
range_data_offset

Time offset at which data becomes available. Example 1 Day

range_step_size

How large the chunk sizes should be.

range_start_date

When to start backfill from.

range_end_date

When to end data collection, leave blank for ongoing.

range_extra_collections

JSON list describing which time ranges of data to collect. Can be contain multiple ranges


range_current_pointer

The current date this task is at. Leaving blank to initalize with start_date, will be updated as task progresses.


S3 Info
aws_connection

AWS Credentials to be used for all things AWS related in this task.

aws_iam

AWS IAM Credentials to be used for all things AWS related in this task.

s3_bucket

S3 Bucket where you want to unload the data to.


SFTP Info
sftp_credentials

SFTP Credentials with access target SFTP host

sftp_host

Target SFTP host.

sftp_port

Target SFTP host Port. Format is hostname:port. Port optional if default port is Ok.


File/Formatting Options
file_name required

Name of the file.

format_type required

Format type that you want to export the data as. Can be delimited or json

delimiter

Delimiter you want to use to separate columns. Default is tab delimited

compression

Compression options, zip, gzip, bz2 are currently supported. Default is gzip.

null_as

The string you want to indicate a null value. Default is empty space AKA empty space.

escape

Escape character used to preceed special characters. Default is backslash

allow_overwrite

Overwrite files with the same name

add_quotes

For delimited only. Quote strings. Recommended espcially if you may have double quotes in your data.


Export - Sliding Integer Window

Export data from Postgres database to various locations. SQL Steps stored in git. Based on configurable sliding integer window.


Variables

target_db required

Database where you want to run the script on.


sql_steps required

Single step to be run. Treated as one statement regardless of ;


Parameter Info
user_params

If in your sql you use any parameters then you need to create a parameter list here. To use ranges visit ...


User Range
irange_step_size required

How large the chunk sizes should be.

irange_start_i

When to start backfill from.

irange_start_i_query

Optional SQL query to use to instead of Start Index

irange_end_i

End Index

irange_end_i_query

Optional SQL query to use to instead of End Index


range_current_pointer

The current date this task is at. Leaving blank to initalize with start_date, will be updated as task progresses.


S3 Info
aws_connection

AWS Credentials to be used for all things AWS related in this task.

aws_iam

AWS IAM Credentials to be used for all things AWS related in this task.

s3_bucket

S3 Bucket where you want to unload the data to.


SFTP Info
sftp_credentials

SFTP Credentials with access target SFTP host

sftp_host

Target SFTP host.

sftp_port

Target SFTP host Port. Format is hostname:port. Port optional if default port is Ok.


File/Formatting Options
file_name required

Name of the file.

format_type required

Format type that you want to export the data as. Can be delimited or json

delimiter

Delimiter you want to use to separate columns. Default is tab delimited

compression

Compression options, zip, gzip, bz2 are currently supported. Default is gzip.

null_as

The string you want to indicate a null value. Default is empty space AKA empty space.

escape

Escape character used to preceed special characters. Default is backslash

allow_overwrite

Overwrite files with the same name

add_quotes

For delimited only. Quote strings. Recommended espcially if you may have double quotes in your data.


Run - (Git)

Execute SQL steps. SQL steps stored in git.


Variables

target_db required

Database where you want to run the script on.


Git Configuration
git_repo required

Git repository to clone from.

git_ref required

Branch, tag to checkout the repo at. Note SHA references are not supported at this time.

git_filepath required

SQL script to execute, path is relative to the root of the git repo.

git_ssh_key

Private ssh key to use to authenticate with git repository. Leave blank if no auth is required.


Parameter Info
user_params

If in your sql you use any parameters then you need to create a parameter list here. To use ranges visit ...

user_range

Used to create a sliding window of data to be processed Json formatted. Documentation here...


Run - Sliding Date Window (Git)

Execute SQL steps. SQL steps stored in git. Based on configurable sliding date window.


Variables

target_db required

Database where you want to run the script on.


Git Configuration
git_repo required

Git repository to clone from.

git_ref required

Branch, tag to checkout the repo at. Note SHA references are not supported at this time.

git_filepath required

SQL script to execute, path is relative to the root of the git repo.

git_ssh_key

Private ssh key to use to authenticate with git repository. Leave blank if no auth is required.


Parameter Info
user_params

If in your sql you use any parameters then you need to create a parameter list here. To use ranges visit ...


User Range
range_data_offset

Time offset at which data becomes available. Example 1 Day

range_step_size

How large the chunk sizes should be.

range_start_date

When to start backfill from.

range_end_date

When to end data collection, leave blank for ongoing.

range_extra_collections

JSON list describing which time ranges of data to collect. Can be contain multiple ranges


range_current_pointer

The current date this task is at. Leaving blank to initalize with start_date, will be updated as task progresses.


Run - Sliding Integer Window (Git)

Execute SQL steps. SQL steps stored in git. Based on configurable sliding integer window.


Variables

target_db required

Database where you want to run the script on.


Git Configuration
git_repo required

Git repository to clone from.

git_ref required

Branch, tag to checkout the repo at. Note SHA references are not supported at this time.

git_filepath required

SQL script to execute, path is relative to the root of the git repo.

git_ssh_key

Private ssh key to use to authenticate with git repository. Leave blank if no auth is required.


Parameter Info
user_params

If in your sql you use any parameters then you need to create a parameter list here. To use ranges visit ...


User Range
irange_step_size required

How large the chunk sizes should be.

irange_start_i

When to start backfill from.

irange_start_i_query

Optional SQL query to use to instead of Start Index

irange_end_i

End Index

irange_end_i_query

Optional SQL query to use to instead of End Index


range_current_pointer

The current date this task is at. Leaving blank to initalize with start_date, will be updated as task progresses.


Export (Git)

Export data from Postgres database to various locations. SQL Steps stored in git


Variables

target_db required

Database where you want to run the script on.


Git Configuration
git_repo required

Git repository to clone from.

git_ref required

Branch, tag to checkout the repo at. Note SHA references are not supported at this time.

git_filepath required

SQL script to execute, path is relative to the root of the git repo.

git_ssh_key

Private ssh key to use to authenticate with git repository. Leave blank if no auth is required.


S3 Info
aws_connection

AWS Credentials to be used for all things AWS related in this task.

aws_iam

AWS IAM Credentials to be used for all things AWS related in this task.

s3_bucket

S3 Bucket where you want to unload the data to.


SFTP Info
sftp_credentials

SFTP Credentials with access target SFTP host

sftp_host

Target SFTP host.

sftp_port

Target SFTP host Port. Format is hostname:port. Port optional if default port is Ok.


File/Formatting Options
file_name required

Name of the file.

format_type required

Format type that you want to export the data as. Can be delimited or json

delimiter

Delimiter you want to use to separate columns. Default is tab delimited

compression

Compression options, zip, gzip, bz2 are currently supported. Default is gzip.

null_as

The string you want to indicate a null value. Default is empty space AKA empty space.

escape

Escape character used to preceed special characters. Default is backslash

allow_overwrite

Overwrite files with the same name

add_quotes

For delimited only. Quote strings. Recommended espcially if you may have double quotes in your data.


Parameter Info
user_params

If in your sql you use any parameters then you need to create a parameter list here. To use ranges visit ...

user_range

Used to create a sliding window of data to be processed Json formatted. Documentation here...


Export - Sliding Date Window (Git)

Export data from Postgres database to various locations. SQL Steps stored in git. Based on configurable sliding date window.


Variables

target_db required

Database where you want to run the script on.


Git Configuration
git_repo required

Git repository to clone from.

git_ref required

Branch, tag to checkout the repo at. Note SHA references are not supported at this time.

git_filepath required

SQL script to execute, path is relative to the root of the git repo.

git_ssh_key

Private ssh key to use to authenticate with git repository. Leave blank if no auth is required.


Parameter Info
user_params

If in your sql you use any parameters then you need to create a parameter list here. To use ranges visit ...


User Range
range_data_offset

Time offset at which data becomes available. Example 1 Day

range_step_size

How large the chunk sizes should be.

range_start_date

When to start backfill from.

range_end_date

When to end data collection, leave blank for ongoing.

range_extra_collections

JSON list describing which time ranges of data to collect. Can be contain multiple ranges


range_current_pointer

The current date this task is at. Leaving blank to initalize with start_date, will be updated as task progresses.


S3 Info
aws_connection

AWS Credentials to be used for all things AWS related in this task.

aws_iam

AWS IAM Credentials to be used for all things AWS related in this task.

s3_bucket

S3 Bucket where you want to unload the data to.


SFTP Info
sftp_credentials

SFTP Credentials with access target SFTP host

sftp_host

Target SFTP host.

sftp_port

Target SFTP host Port. Format is hostname:port. Port optional if default port is Ok.


File/Formatting Options
file_name required

Name of the file.

format_type required

Format type that you want to export the data as. Can be delimited or json

delimiter

Delimiter you want to use to separate columns. Default is tab delimited

compression

Compression options, zip, gzip, bz2 are currently supported. Default is gzip.

null_as

The string you want to indicate a null value. Default is empty space AKA empty space.

escape

Escape character used to preceed special characters. Default is backslash

allow_overwrite

Overwrite files with the same name

add_quotes

For delimited only. Quote strings. Recommended espcially if you may have double quotes in your data.


Export - Sliding Integer Window (Git)

Export data from Postgres database to various locations. SQL Steps stored in git. Based on configurable sliding integer window.


Variables

target_db required

Database where you want to run the script on.


Git Configuration
git_repo required

Git repository to clone from.

git_ref required

Branch, tag to checkout the repo at. Note SHA references are not supported at this time.

git_filepath required

SQL script to execute, path is relative to the root of the git repo.

git_ssh_key

Private ssh key to use to authenticate with git repository. Leave blank if no auth is required.


Parameter Info
user_params

If in your sql you use any parameters then you need to create a parameter list here. To use ranges visit ...


User Range
irange_step_size required

How large the chunk sizes should be.

irange_start_i

When to start backfill from.

irange_start_i_query

Optional SQL query to use to instead of Start Index

irange_end_i

End Index

irange_end_i_query

Optional SQL query to use to instead of End Index


range_current_pointer

The current date this task is at. Leaving blank to initalize with start_date, will be updated as task progresses.


S3 Info
aws_connection

AWS Credentials to be used for all things AWS related in this task.

aws_iam

AWS IAM Credentials to be used for all things AWS related in this task.

s3_bucket

S3 Bucket where you want to unload the data to.


SFTP Info
sftp_credentials

SFTP Credentials with access target SFTP host

sftp_host

Target SFTP host.

sftp_port

Target SFTP host Port. Format is hostname:port. Port optional if default port is Ok.


File/Formatting Options
file_name required

Name of the file.

format_type required

Format type that you want to export the data as. Can be delimited or json

delimiter

Delimiter you want to use to separate columns. Default is tab delimited

compression

Compression options, zip, gzip, bz2 are currently supported. Default is gzip.

null_as

The string you want to indicate a null value. Default is empty space AKA empty space.

escape

Escape character used to preceed special characters. Default is backslash

allow_overwrite

Overwrite files with the same name

add_quotes

For delimited only. Quote strings. Recommended espcially if you may have double quotes in your data.