---
title: Real-Time Analytics with Amazon Kinesis
description: A growing number of customers require streaming data on AWS to obtain actionable insights from their data in real time. AWS Kinesis can help to achieve this
image: https://info.bigindustries.be/hubfs/AWS%20Kinesis%20Firehose.png
---

[![Analogue Cloud Retina Logo](https://analogue.cloud/wp-content/uploads/2017/03/analoguecloud3x2.png)](http://www.analogue.cloud/)

- [![](https://www.bigindustries.be/wp-content/uploads/2015/12/home-big-data-hadoop-mesosphere-apache-cassandra-nosql-cloud.png?width=24&height=24&name=home-big-data-hadoop-mesosphere-apache-cassandra-nosql-cloud.png)](https://www.bigindustries.be/#home)
- [Home](http://analogue.cloud/)
- [Solutions](http://analogue.cloud/#solutions)
- [Services](http://analogue.cloud/#services)
- [Jobs](http://analogue.cloud/#jobs)
- [Blog](https://info.bigindustries.be/analogue-cloud)
- [Contact](http://analogue.cloud/#contact)
- [![](https://analogue.cloud/wp-content/uploads/2018/09/aws_logo_small.jpg)](https://info.bigindustries.be/analogue-cloud/real-time-analytics-with-amazon-kinesis#)

SolutionsServicesJobsBlogContact

# Real-Time Analytics with Amazon Kinesis

BIG DATA MATTERS blog

![AWS Kinesis Firehose.png](https://info.bigindustries.be/hs-fs/hubfs/AWS%20Kinesis%20Firehose.png?width=870&height=343&name=AWS%20Kinesis%20Firehose.png)

Many Amazon Web Services (AWS) customers use batch data reports to gain strategic insight into long-term business trends, and a growing number of customers also require streaming data to obtain actionable insights from their data in real time.

## Streaming Data

Streaming Data is data that is generated continuously by thousands of data sources, which typically send in the data records simultaneously, and in small sizes (order of Kilobytes). Streaming data includes a wide variety of data such as log files generated by customers using your mobile or web applications, ecommerce purchases, in-game player activity, information from social networks, financial trading floors, or geospatial services, and telemetry from connected devices or instrumentation in data centers.

This data needs to be processed sequentially and incrementally on a record-by-record basis or over sliding time windows, and used for a wide variety of analytics including correlations, aggregations, filtering, and sampling. Information derived from such analysis gives companies visibility into many aspects of their business and customer activity such as –service usage (for metering/billing), server activity, website clicks, and geo-location of devices, people, and physical goods –and enables them to respond promptly to emerging situations. For example, businesses can track changes in public sentiment on their brands and products by continuously analyzing social media streams, and respond in a timely fashion as the necessity arises.

## Challenges in working with Streaming Data

Streaming data processing requires two layers: a storage layer and a processing layer. The storage layer needs to support record ordering and strong consistency to enable fast, inexpensive, and replayable reads and writes of large streams of data. The processing layer is responsible for consuming data from the storage layer, running computations on that data, and then notifying the storage layer to delete data that is no longer needed. You also have to plan for scalability, data durability, and fault tolerance in both the storage and processing layers. As a result, many platforms have emerged that provide the infrastructure needed to build data applications including Amazon Kinesis Streams, Amazon Kinesis Firehose, Apache Kafka, Apache Flume, Apache Spark Streaming and Apache Storm.

## Working with Streaming Data on AWS

Amazon Web Services (AWS) provides a number options to work with streaming data. You can take advantage of the managed streaming data services offered by Amazon Kinesis, or deploy and manage your own streaming data solution in the cloud on Amazon EC2.

Amazon Kinesis is a platform for streaming data on AWS, offering powerful services to make it easy to load and analyze streaming data, and also enables you to build custom streaming data applications for specialized needs. It offers two services: Amazon Kinesis Firehose, and Amazon Kinesis Streams.

In addition, you can run other streaming data platforms such as –Apache Kafka, Apache Flume, Apache Spark Streaming, and Apache Storm –on Amazon EC2 and Amazon EMR. 

## Amazon Kinesis

Amazon Kinesis makes it easy to collect, process, and analyze <https://aws.amazon.com/streaming-data/> so you can get timely insights and react quickly to new information. Amazon Kinesis offers key capabilities to cost effectively process streaming data at any scale, along with the flexibility to choose the tools that best suit the requirements of your application. With Amazon Kinesis, you can ingest real-time data such as application logs, website clickstreams, IoT telemetry data, and more into your databases, data lakes and data warehouses, or build your own real-time applications using this data. Amazon Kinesis enables you to process and analyze data as it arrives and respond in real-time instead of having to wait until all your data is collected before the processing can begin.

![Schermafbeelding 2017-09-28 om 16.09.11.png](https://info.bigindustries.be/hs-fs/hubfs/Schermafbeelding%202017-09-28%20om%2016.09.11.png?width=870&height=178&name=Schermafbeelding%202017-09-28%20om%2016.09.11.png)

 

### Amazon Kinesis Firehose

Amazon Kinesis Firehose is the easiest way to load streaming data into AWS. It can capture and automatically load streaming data into Amazon S3 and Amazon Redshift, enabling near real-time analytics with existing business intelligence tools and dashboards you’re already using today. It enables you to quickly implement an ELT approach, and gain benefits from streaming data quickly. 

### Amazon Kinesis Analytics

Amazon Kinesis Analytics is the easiest way to process streaming data in real time with standard SQL without having to learn new programming languages or processing frameworks. Amazon Kinesis Analytics enables you to query streaming data or build entire streaming applications using SQL, so that you can gain actionable insights and respond to your business and customer needs promptly.

### Amazon Kinesis Streams

Amazon Kinesis Streams enables you to build your own custom applications that process or analyze streaming data for specialized needs. It can continuously capture and store terabytes of data per hour from hundreds of thousands of sources. You can then build applications that consume the data from Amazon Kinesis Streams to power real-time dashboards, generate alerts, implement dynamic pricing and advertising, and more. Amazon Kinesis Streams supports your choice of stream processing framework including Kinesis Client Library (KCL), Apache Storm, and Apache Spark Streaming.

## What Can you build with Amazon Kinesis?

You can use Amazon Kinesis for real-time applications such as application monitoring, fraud detection, and live leader-boards. You can ingest streaming data using Kinesis Streams, process it using Kinesis Analytics, and emit the results to any data store or application using Kinesis Streams with millisecond end-to-end latency. This can help you learn about what your customers, applications, and products are doing right now and react promptly.

 

![Clickstream Analysis.png](https://info.bigindustries.be/hs-fs/hubfs/Clickstream%20Analysis.png?width=870&height=317&name=Clickstream%20Analysis.png)

### ![social media analytics.png](https://info.bigindustries.be/hs-fs/hubfs/social%20media%20analytics.png?width=870&height=303&name=social%20media%20analytics.png)

 

[![Need help with your Big Data on AWS Project?](https://hubspot-no-cache-eu1-prod.s3.amazonaws.com/cta/default/2240994/239630ca-6faf-48a5-b5c1-e4de2a113a56.png)](https://hubspot-cta-redirect-eu1-prod.s3.amazonaws.com/cta/redirect/2240994/239630ca-6faf-48a5-b5c1-e4de2a113a56) 

 

 

 Posted by [Matthias Vallaey](https://info.bigindustries.be/analogue-cloud/author/matthias-vallaey) on 29-sep-2017 15:48:36

- [Tweet](https://twitter.com/share)

[Analogue.cloud winner of the APN Star contest **](https://info.bigindustries.be/analogue-cloud/analogue.cloud-winner-of-the-apn-star-contest)

[** Looking for a cool internship?](https://info.bigindustries.be/analogue-cloud/looking-for-a-cool-internship)

![Matthias Vallaey](https://info.bigindustries.be/hs-fs/hubfs/popular-bigindustries/team/Matthias-Vallaey.png?width=64&height=64&name=Matthias-Vallaey.png)

#### About the Author

 Matthias is founder of Big Industries and a Big Data Evangelist. He has a strong track record in the IT-Services and Software Industry, working across many verticals. He is highly skilled at developing account relationships by bringing innovative solutions that exceeds customer expectations. In his role as Entrepreneur he is building partnerships with Big Data Vendors and introduces their technology where they bring most value.

### Contact Us!

- Recent
- Popular
- Categories

### Lists by Topic

- [Data Lakes and Analytics on AWS (8)](https://info.bigindustries.be/analogue-cloud/topic/data-lakes-and-analytics-on-aws)
- [AWS (5)](https://info.bigindustries.be/analogue-cloud/topic/aws)
- [Amazon Web Services (5)](https://info.bigindustries.be/analogue-cloud/topic/amazon-web-services)
- [Data Engineering (5)](https://info.bigindustries.be/analogue-cloud/topic/data-engineering)
- [Amazon (4)](https://info.bigindustries.be/analogue-cloud/topic/amazon)
- [Big Data on AWS (4)](https://info.bigindustries.be/analogue-cloud/topic/big-data-on-aws)
- [AWS Cloud (3)](https://info.bigindustries.be/analogue-cloud/topic/aws-cloud)
- [AWS Lambda (3)](https://info.bigindustries.be/analogue-cloud/topic/aws-lambda)
- [Streaming Data on AWS (3)](https://info.bigindustries.be/analogue-cloud/topic/streaming-data-on-aws)
- [analogue.cloud (3)](https://info.bigindustries.be/analogue-cloud/topic/analogue-cloud)
- [devops (3)](https://info.bigindustries.be/analogue-cloud/topic/devops)
- [AWS Government Competency (2)](https://info.bigindustries.be/analogue-cloud/topic/aws-government-competency)
- [AWS function (2)](https://info.bigindustries.be/analogue-cloud/topic/aws-function)
- [Alexa (2)](https://info.bigindustries.be/analogue-cloud/topic/alexa)
- [Big Data (2)](https://info.bigindustries.be/analogue-cloud/topic/big-data)
- [Big Data Reference Architecture (2)](https://info.bigindustries.be/analogue-cloud/topic/big-data-reference-architecture)
- [Cloud Computing (2)](https://info.bigindustries.be/analogue-cloud/topic/cloud-computing)
- [Cloud Data Lake (2)](https://info.bigindustries.be/analogue-cloud/topic/cloud-data-lake)
- [Cloudformation (2)](https://info.bigindustries.be/analogue-cloud/topic/cloudformation)
- [Customer Case (2)](https://info.bigindustries.be/analogue-cloud/topic/customer-case)
- [Datalake Devops Best Practices (2)](https://info.bigindustries.be/analogue-cloud/topic/datalake-devops-best-practices)
- [Enterprise Datawarehouse on AWS (2)](https://info.bigindustries.be/analogue-cloud/topic/enterprise-datawarehouse-on-aws)
- [Serverless Computing (2)](https://info.bigindustries.be/analogue-cloud/topic/serverless-computing)
- [Your partner for Data Lakes and Analytics (2)](https://info.bigindustries.be/analogue-cloud/topic/your-partner-for-data-lakes-and-analytics)
- [kubernetes (2)](https://info.bigindustries.be/analogue-cloud/topic/kubernetes)
- [AI (1)](https://info.bigindustries.be/analogue-cloud/topic/ai)
- [API Gateway (1)](https://info.bigindustries.be/analogue-cloud/topic/api-gateway)
- [AWS Eventbridge (1)](https://info.bigindustries.be/analogue-cloud/topic/aws-eventbridge)
- [AWS Hands-On Workshop (1)](https://info.bigindustries.be/analogue-cloud/topic/aws-hands-on-workshop)
- [AWS Kinesis (1)](https://info.bigindustries.be/analogue-cloud/topic/aws-kinesis)
- [AWS Public Sector Partner (1)](https://info.bigindustries.be/analogue-cloud/topic/aws-public-sector-partner)
- [AWS S3 (1)](https://info.bigindustries.be/analogue-cloud/topic/aws-s3)
- [AWS SES (1)](https://info.bigindustries.be/analogue-cloud/topic/aws-ses)
- [Alexa Skills (1)](https://info.bigindustries.be/analogue-cloud/topic/alexa-skills)
- [Alexa Skills Development (1)](https://info.bigindustries.be/analogue-cloud/topic/alexa-skills-development)
- [Amazon Alexa (1)](https://info.bigindustries.be/analogue-cloud/topic/amazon-alexa)
- [Apache Hive (1)](https://info.bigindustries.be/analogue-cloud/topic/apache-hive)
- [Apache Spark (1)](https://info.bigindustries.be/analogue-cloud/topic/apache-spark)
- [Artificial Intelligence (1)](https://info.bigindustries.be/analogue-cloud/topic/artificial-intelligence)
- [Big data use cases (1)](https://info.bigindustries.be/analogue-cloud/topic/big-data-use-cases)
- [Business (1)](https://info.bigindustries.be/analogue-cloud/topic/business)
- [Clickstream analysis (1)](https://info.bigindustries.be/analogue-cloud/topic/clickstream-analysis)
- [Cloud Data Governance (1)](https://info.bigindustries.be/analogue-cloud/topic/cloud-data-governance)
- [Cloud Procurement (1)](https://info.bigindustries.be/analogue-cloud/topic/cloud-procurement)
- [Cloud evaluation Criteria (1)](https://info.bigindustries.be/analogue-cloud/topic/cloud-evaluation-criteria)
- [Cloudera Director (1)](https://info.bigindustries.be/analogue-cloud/topic/cloudera-director)
- [Cloudera on AWS (1)](https://info.bigindustries.be/analogue-cloud/topic/cloudera-on-aws)
- [Cloudfront (1)](https://info.bigindustries.be/analogue-cloud/topic/cloudfront)
- [Customer Testimonial (1)](https://info.bigindustries.be/analogue-cloud/topic/customer-testimonial)
- [Data Driven Applications (1)](https://info.bigindustries.be/analogue-cloud/topic/data-driven-applications)
- [Data Lake (1)](https://info.bigindustries.be/analogue-cloud/topic/data-lake)
- [Data Science (1)](https://info.bigindustries.be/analogue-cloud/topic/data-science)
- [Data Sources (1)](https://info.bigindustries.be/analogue-cloud/topic/data-sources)
- [Data Stream Processing (1)](https://info.bigindustries.be/analogue-cloud/topic/data-stream-processing)
- [Deep Learning (1)](https://info.bigindustries.be/analogue-cloud/topic/deep-learning)
- [DynamoDb (1)](https://info.bigindustries.be/analogue-cloud/topic/dynamodb)
- [EMR (1)](https://info.bigindustries.be/analogue-cloud/topic/emr)
- [ETL (1)](https://info.bigindustries.be/analogue-cloud/topic/etl)
- [Gartner Magic Quadrant (1)](https://info.bigindustries.be/analogue-cloud/topic/gartner-magic-quadrant)
- [Infrastructure as code (1)](https://info.bigindustries.be/analogue-cloud/topic/infrastructure-as-code)
- [Kafka Cluster (1)](https://info.bigindustries.be/analogue-cloud/topic/kafka-cluster)
- [Kinesis Analytics (1)](https://info.bigindustries.be/analogue-cloud/topic/kinesis-analytics)
- [Kinesis Firehose (1)](https://info.bigindustries.be/analogue-cloud/topic/kinesis-firehose)
- [Kinesis Streams (1)](https://info.bigindustries.be/analogue-cloud/topic/kinesis-streams)
- [Lambda (1)](https://info.bigindustries.be/analogue-cloud/topic/lambda)
- [Linux (1)](https://info.bigindustries.be/analogue-cloud/topic/linux)
- [Managing Servers in the Cloud (1)](https://info.bigindustries.be/analogue-cloud/topic/managing-servers-in-the-cloud)
- [Market Leader (1)](https://info.bigindustries.be/analogue-cloud/topic/market-leader)
- [On demand big data analysis (1)](https://info.bigindustries.be/analogue-cloud/topic/on-demand-big-data-analysis)
- [Operational excellence (1)](https://info.bigindustries.be/analogue-cloud/topic/operational-excellence)
- [PXL-IT Hasselt (1)](https://info.bigindustries.be/analogue-cloud/topic/pxl-it-hasselt)
- [Redshift (1)](https://info.bigindustries.be/analogue-cloud/topic/redshift)
- [Retail use case (1)](https://info.bigindustries.be/analogue-cloud/topic/retail-use-case)
- [S3 (1)](https://info.bigindustries.be/analogue-cloud/topic/s3)
- [Saas (1)](https://info.bigindustries.be/analogue-cloud/topic/saas)
- [Skills (1)](https://info.bigindustries.be/analogue-cloud/topic/skills)
- [Software Automation (1)](https://info.bigindustries.be/analogue-cloud/topic/software-automation)
- [Students IT (1)](https://info.bigindustries.be/analogue-cloud/topic/students-it)
- [Vlaams Energie Bedrijf (1)](https://info.bigindustries.be/analogue-cloud/topic/vlaams-energie-bedrijf)
- [Webinar (1)](https://info.bigindustries.be/analogue-cloud/topic/webinar)
- [apache kafka (1)](https://info.bigindustries.be/analogue-cloud/topic/apache-kafka)
- [apn star contest (1)](https://info.bigindustries.be/analogue-cloud/topic/apn-star-contest)
- [aws partnership (1)](https://info.bigindustries.be/analogue-cloud/topic/aws-partnership)
- [better operations (1)](https://info.bigindustries.be/analogue-cloud/topic/better-operations)
- [big data skills (1)](https://info.bigindustries.be/analogue-cloud/topic/big-data-skills)
- [business risks (1)](https://info.bigindustries.be/analogue-cloud/topic/business-risks)
- [cronos group (1)](https://info.bigindustries.be/analogue-cloud/topic/cronos-group)
- [customer insights (1)](https://info.bigindustries.be/analogue-cloud/topic/customer-insights)
- [erp (1)](https://info.bigindustries.be/analogue-cloud/topic/erp)
- [internships (1)](https://info.bigindustries.be/analogue-cloud/topic/internships)
- [iot (1)](https://info.bigindustries.be/analogue-cloud/topic/iot)
- [machine data (1)](https://info.bigindustries.be/analogue-cloud/topic/machine-data)
- [machine learning (1)](https://info.bigindustries.be/analogue-cloud/topic/machine-learning)
- [machine logs (1)](https://info.bigindustries.be/analogue-cloud/topic/machine-logs)
- [predictive analytics (1)](https://info.bigindustries.be/analogue-cloud/topic/predictive-analytics)
- [prescriptive analytics (1)](https://info.bigindustries.be/analogue-cloud/topic/prescriptive-analytics)
- [project terra (1)](https://info.bigindustries.be/analogue-cloud/topic/project-terra)
- [stage (1)](https://info.bigindustries.be/analogue-cloud/topic/stage)
- [stage opdracht (1)](https://info.bigindustries.be/analogue-cloud/topic/stage-opdracht)
- [winner contest (1)](https://info.bigindustries.be/analogue-cloud/topic/winner-contest)

[see all](https://info.bigindustries.be/analogue-cloud/real-time-analytics-with-amazon-kinesis#)

[![Analogue cloud](https://analogue.cloud/wp-content/uploads/2018/09/aws_logo.jpg)](http://www.analogue.cloud/)

[Googleplus](https://plus.google.com/u/0/103022922545276758997/posts)

[Linkedin](https://www.linkedin.com/company/big-industries)

[Twitter](https://twitter.com/bigindustries)

Copyright 2017 Analogue Cloud  
BE 0807729292