Amazon Kinesis

Least Privileged User

Navigate to the AWS console and create an IAM user with programmatic access. The user will need the following permissions. You can create a policy specifically for these permissions and apply the permissions to the user.

For more information, see: High Level AWS Source Configuration

{
  "Version": "2012-10-17",
  "Statement": [
    {
      "Sid": "VisualEditor0",
      "Effect": "Allow",
      "Action": [
        "kinesisvideo:ListFragments",
        "cloudwatch:GetMetricData",
        "kinesisvideo:ListStreams",
        "kinesis:GetShardIterator",
        "kinesisvideo:GetDataEndpoint",
        "kinesis:DescribeStream",
        "kinesisvideo:GetMedia",
        "kinesisanalytics:DescribeApplication",
        "cloudwatch:GetMetricStatistics",
        "cloudwatch:ListMetrics",
        "kinesis:ListTagsForStream",
        "firehose:DescribeDeliveryStream",
        "kinesis:ListStreams",
        "kinesisvideo:GetMediaForFragmentList",
        "kinesisanalytics:DiscoverInputSchema",
        "kinesisvideo:ListTagsForStream",
        "cloudwatch:DescribeAlarmHistory",
        "cloudwatch:DescribeAlarmsForMetric",
        "cloudwatch:DescribeAlarms",
        "firehose:ListDeliveryStreams",
        "kinesis:GetRecords",
        "kinesisanalytics:ListApplications",
        "kinesisvideo:DescribeStream",
        "kinesis:DescribeLimits"
      ],
      "Resource": "*"
    }
  ]
}

Connection Parameters

Name	Required?	Description
Region
Access Key ID	Required
Secret Access Key	Required
Additional Threads		The number of additional threads allowed to be utilized during collection.
Request Timeout (seconds)		The number of seconds to allow for the API to return a response.
Collect CloudWatch Metrics
CloudWatch Historic Mode		If enabled, retrieves a history of data points from CloudWatch. Otherwise, collects only the most recent data point for each metric.

Metrics

Analytic

Name	Description
Application	The name of the analytic application.
Data Throughput (Bytes)	The amount of data flowing through the application.
Firehose ARN	The Amazon Resource Name (ARN) of the Amazon Kinesis Firehose delivery stream.
Firehose Role ARN	The ARN of the IAM role that Amazon Kinesis Analytics assumes to access the Firehose delivery stream.
Flow	The direction of data movement, either input or output.
ID	The ID of the analytic.
In Application Stream Names	A CSV list of in-application stream names that are mapped to the stream source.
Input Parallelism Count	The number of in-application streams mapped to the streaming source.
Lambda ARN	The ARN of the AWS Lambda function that is used to preprocess the records in the stream.
Lambda Role ARN	The ARN of the IAM role that is used to access the AWS Lambda function.
Records (Records)	The number of records being processed by the application.
Region	The AWS Region this object belongs to.
Starting Position	The starting position on the stream.
Stream ARN	The Amazon Resource Name (ARN) of the Amazon Kinesis stream.
Stream Role ARN	The ARN of the IAM role that Amazon Kinesis Analytics assumes to access the Kinesis stream.
Successes (Operations)	The number of successful operations performed by the application.
Time Behind Latest (Milliseconds)	Indicates how far behind an application is reading from the streaming source.

Application

Name	Description
ARN	The Amazon Resource Name (ARN) of the application.
Code	Returns the application code that was provided to perform data analysis on any of the in-application streams in the application.
Description	Description of the application.
Name	Name of the application.
Region	The AWS Region this object belongs to.
Status	Status of the application.
Version ID	The current application version.

Firehose Delivery Stream

Name	Description
ARN	The Amazon Resource Name (ARN) of the delivery stream.
Data Read From Kinesis Stream (Bytes)	The amount of data read from the stream.
Data Read From Kinesis Stream Records (Records)	The number of records read from the stream.
Delivery To Elasticsearch Data (Bytes)	The amount of data indexed to Amazon ES over the specified time period.
Delivery To Elasticsearch Records (Records)	The number of records indexed to Amazon ES over the specified time period.
Delivery To Elasticsearch Successes (%)	Sum of the successfully indexed records over the sum of records that were attempted.
Delivery To Redshift Data (Bytes)	The amount of data copied to Amazon Redshift over the specified time period.
Delivery To Redshift Records (Records)	The number of records copied to Amazon Redshift over the specified time period.
Delivery To Redshift Successes (%)	Sum of successful Amazon Redshift COPY commands over the sum of all Amazon Redshift COPY commands.
Delivery To S3 Data (Bytes)	The amount of data delivered to Amazon S3 over the specified time period.
Delivery To S3 Data Freshness (Seconds)	Age (from getting into Kinesis Firehose to now) of the oldest record in Kinesis Firehose. Any record older than this age has been delivered to the S3 bucket.
Delivery To S3 Records (Records)	The number of records delivered to Amazon S3 over the specified time period.
Delivery To S3 Successes (%)	Sum of successful Amazon S3 put commands over sum of all Amazon S3 put commands.
Delivery To Splunk Data (Bytes)	The amount of data delivered to Splunk over the specified time period.
Delivery To Splunk Data Freshness (Seconds)	Age (from getting into Kinesis Firehose to now) of the oldest record in Kinesis Firehose. Any record older than this age has been delivered to Splunk.
Delivery To Splunk Records (Records)	The number of records delivered to Splunk over the specified time period.
Delivery To Splunk Successes (%)	The sum of the successfully indexed records over the sum of records that were attempted.
DescribeDeliveryStream Latency (Milliseconds)	The time taken per DescribeDeliveryStream operation, measured over the specified time period.
DescribeDeliveryStream Requests (Requests)	Total number of DescribeDeliveryStream requests.
Incoming Data (Bytes)	The amount of data ingested into the Kinesis Firehose delivery stream over the specified time period.
Incoming Records (Records)	The number of records ingested into the Kinesis Firehose delivery stream over the specified time period.
Kinesis Time Behind Latest (Milliseconds)	Time difference between the current server time stamp and the server time stamp of the last fragment sent.
ListDeliveryStreams Latency (Milliseconds)	The time taken per ListDeliveryStream operation, measured over the specified time period.
ListDeliveryStreams Requests (Requests)	Total number of ListFirehose requests.
Name	The name of the Firehose Delivery Stream.
PutRecord Batch Data (Bytes)	The amount of data put to the Kinesis Firehose delivery stream using PutRecordBatch over the specified time period.
PutRecord Batch Latency (Milliseconds)	The time taken per PutRecordBatch operation, measured over the specified time period.
PutRecord Batch Records (Records)	Total number of records from PutRecordBatch operations.
PutRecord Batch Requests (Requests)	Total number of PutRecordBatch requests.
PutRecord Data (Bytes)	The amount of data put to the Kinesis Firehose delivery stream using PutRecord over the specified time period.
PutRecord Latency (Milliseconds)	The time taken per PutRecord operation, measured over the specified time period.
PutRecord Requests (Requests)	Total number of PutRecord requests, which is equal to total number of records from PutRecord operations.
Region	The AWS Region this object belongs to.
Source Stream ARN	The ARN of the source Kinesis stream.
Source Stream Role	The ARN of the role used by the source Kinesis stream.
Status	The status of the delivery stream.
Stream Type	The delivery stream type.
Throttled DescribeStream (Requests)	The total number of throttled DescribeStream requests.
Throttled GetRecords (Requests)	The total number of throttled GetRecords requests.
Throttled GetShardIterator (Requests)	The total number of throttled GetShardIterator requests.
UpdateDeliveryStream Latency (Milliseconds)	The time taken per UpdateDeliveryStream operation, measured over the specified time period.
UpdateDeliveryStream Requests (Requests)	Total number of UpdateDeliveryStream requests.
Version ID	Each time the destination is updated for a delivery stream, the version ID is changed. This value is the latest version.

Shard

Name	Description
Adjacent Parent Shard ID	The shard ID of the shard adjacent to the shard's parent.
ID	The shard ID within the Kinesis stream.
Incoming Data (Bytes)	The amount of data successfully put to the shard over the specified time period. This metric includes data from PutRecord and PutRecords operations.
Incoming Records (Records)	The number of records successfully put to the shard over the specified time period. This metric includes record counts from PutRecord and PutRecords operations.
Iterator Age (Milliseconds)	The age of the last record in all GetRecords calls made against a shard, measured over the specified time period. Age is the difference between the current time and when the last record of the GetRecords call was written to the stream.
Outgoing Data (Bytes)	The amount of data retrieved from the shard, measured over the specified time period.
Outgoing Records (Records)	The number of records retrieved from the shard, measured over the specified time period.
Parent Shard ID	The shard ID of the shard's parent.
Read Provisioned Throughput Exceeded (Requests)	The number of GetRecords calls throttled for the shard over the specified time period.
Region	The AWS Region this object belongs to.
Stream Name	The name of the shard's parent Kinesis stream.
Write Provisioned Throughput Exceeded (Records)	The number of records rejected due to throttling for the shard over the specified time period. This metric includes throttling from PutRecord and PutRecords operations.

Stream

Name	Description
ARN	The Amazon Resource Name (ARN) for the stream.
Creation Timestamp	The approximate time that the stream was created.
Encryption Type	The server-side encryption type used on the stream.
GetRecords Data (Bytes)	The amount of data retrieved from the Kinesis stream, measured over the specified time period.
GetRecords Iterator Age (Milliseconds)	The age of the last record in all GetRecords calls made against an Kinesis stream, measured over the specified time period. Age is the difference between the current time and when the last record of the GetRecords call was written to the stream.
GetRecords Latency (Milliseconds)	The time taken per GetRecords operation, measured over the specified time period.
GetRecords Record Count (Records)	The number of records retrieved from the shard, measured over the specified time period.
GetRecords Successes (%)	The number of successful GetRecords operations per stream, measured over the specified time period.
Incoming Data (Bytes)	The amount of data successfully put to the Kinesis stream over the specified time period. This metric includes bytes from PutRecord and PutRecords operations.
Incoming Records (Records)	The number of records successfully put to the Kinesis stream over the specified time period. This metric includes record counts from PutRecord and PutRecords operations.
KMS Key ID	The GUID for the customer-managed KMS key to use for encryption.
Name	The name of the Kinesis stream.
PutRecord Data (Bytes)	The amount of data put to the Kinesis stream using the PutRecord operation over the specified time period.
PutRecord Latency (Milliseconds)	The time taken per PutRecord operation, measured over the specified time period.
PutRecord Successes (%)	Reflects the percentage of successful writes to a stream.
PutRecords Data (Bytes)	The amount of data put to the Kinesis stream using the PutRecords operation over the specified time period.
PutRecords Latency (Milliseconds)	The time taken per PutRecords operation, measured over the specified time period.
PutRecords Record Count (Records)	The number of successful records in a PutRecords operation per Kinesis stream, measured over the specified time period.
PutRecords Successes (Requests)	The number of PutRecords operations where at least one record succeeded, per Kinesis stream, measured over the specified time period.
Read Provisioned Throughput Exceeded (Requests)	The number of GetRecords calls throttled for the stream over the specified time period.
Region	The AWS Region this object belongs to.
Retention Period (Hours)	The current retention period.
Status	The current status of the stream.
Write Provisioned Throughput Exceeded (Records)	The number of records rejected due to throttling for the stream over the specified time period. This metric includes throttling from PutRecord and PutRecords operations.

Video Stream

Name	Description
ARN	The Amazon Resource Name (ARN) of the video stream.
Data Retention Period (Hours)	How long the stream retains data.
Device Name	The name of the device associated with the stream.
GetMediaForFragmentList Outgoing Data (Bytes)	Total amount of data sent out from the service as part of the GetMediaForFragmentList API for a given stream.
GetMediaForFragmentList Outgoing Fragments	Total number of fragments sent out from the service as part of the GetMediaForFragmentList API for a given stream.
GetMediaForFragmentList Outgoing Frames (Frames)	Total number of frames sent out from the service as part of the GetMediaForFragmentList API for a given stream.
GetMediaForFragmentList Requests (Requests)	Total number of GetMediaForFragmentList API requests for a given stream.
GetMedia Outgoing Data (Bytes)	Total amount of data sent out from the service as part of the GetMedia API for a given stream.
GetMedia Outgoing Fragments	Total number of fragments sent while doing GetMedia for the stream.
GetMedia Outgoing Frames (Frames)	Total number of frames sent during GetMedia on the given stream.
GetMedia Requests (Requests)	Total number of GetMedia API requests for a given stream.
GetMedia Successes (%)	Indicates the rate of success.
GetMedia Time Behind Latest (Milliseconds)	Time difference between the current server time stamp and the server time stamp of the last fragment sent.
KMS Key ID	The ID of the AWS Key Management Service (AWS KMS) key that Kinesis Video Streams uses to encrypt data on the stream.
ListFragments Latency (Milliseconds)	Latency of the ListFragments API calls for the given stream name.
Media Type	The MediaType of the stream.
Name	The name of the Kinesis video stream.
PutMedia Buffering ACK Latency (Milliseconds)	Time difference between when the first byte of a new fragment is received by Kinesis Video Streams and when the Buffering Ack is sent for the fragment.
PutMedia Connection Errors (Errors)	Total number of errors while establishing PutMedia connection for the stream.
PutMedia Error ACK Count	Total number of Error ACKs sent while doing PutMedia for the stream.
PutMedia Fragment Ingestion Latency (Milliseconds)	Time difference between when the first and last bytes of a fragment are received by Kinesis Video Streams.
PutMedia Fragment Persist Latency (Milliseconds)	Time taken from when the complete fragment data is received and archived.
PutMedia Incoming Data (Bytes)	Amount of data received as part of PutMedia for the stream.
PutMedia Incoming Fragments	Total number of complete fragments received as part of PutMedia for the stream.
PutMedia Incoming Frames	Total number of complete frames received as part of PutMedia for the stream.
PutMedia Persisted ACK Latency (Milliseconds)	Time difference between when the last byte of a new fragment is received by and when the Persisted ACK is sent for the fragment.
PutMedia Received ACK Latency (Milliseconds)	Time difference between when the last byte of a new fragment is received by and when the Buffering ACK is sent for the fragment.
PutMedia Requests (Requests)	Total number of PutMedia API requests for a given stream.
PutMedia Successes (%)	Indicates how many complete, valid fragments are sent.
Region	The AWS Region this object belongs to.
Status	The status of the stream.
Version	The version of the stream.