Unleash the power of Data Analytics in AWS Recap

Our team had the opportunity to sit down with AWS and talk about the importance of having real time Data Analytics and how to get them through an AWS Data Pipeline. Softcrylic’s Natts Sivaramakrishnan, VP Technical Architecture and Johny Varghese, Lead Software Engineer at Softcrylic, were joined by AWS’ Ghandi Raketla Sr. Solutions Architect. In this co-hosted webinar, we dig into the reasons to be a data driven business, common use cases for real time data, the solutions AWS can provide and a demonstration of how to streamline your data using AWS solutions using Kinesis Data Firehose and Kinesis Data Stream. You can find the main breakdown below.

7:36 Heart of Data Driven Analytics

• Finding modern data solutions
• Why you need real time analytics
• What business problems does real time data solve

7:52 Modern Data Solutions

• Benefits of data driven business
• Modern Data Platform Solutions for cost effectiveness
• Find value in your own data instead of market research data
• What does it mean to be a data driven business

10:48 Need for Real Time Analytics

• Be a more agile company with real time analytics
• Get rid of lost time
• Better define busines strategies

13:27 Streaming Real Time Data with AWS (AWS Presentation)

• Why stream through AWS
  1. Easy to set up
  2. High availability
  3. Fully managed
  4. Easy to scale
  5. Seamless integration with AWS Services
  6. Only pay for what you use

16:10 Enabling Real Time Analytics

• How can you get real time analytics in reality
• How to ingest from the streaming mechanism
• How to store your data
• Stream processing
• Processing the data

23:22 How does Kinesis Data Analytics work

• Connect to streaming source
• Write sql or java code to process streaming data
• Continuously deliver results

28:00 Demo Set-up

Problem statement: How do we analyze click stream data? Users are coming into my site, and I want to know in real-time what they are doing. We will track some of the user activity in the web page to decide how we want to change the content.
Setup: The solution for the above problem can be dealt with using 2 different workflows. Workflow 1: Setup and collect data using kinesis data firehose and push it into S3 bucket. This approach will process data in batches and help develop complex analytics datasets for reporting requirements. Workflow 2: Setup Kinesis data stream and process data in real time using lamda functions and push it into DynamoDB to show realtime reports to users.

35:38 Approach 1: Kinesis Data Firehose

How do you know which variation to use? When you don’t have downstream applications that need to consume messages and do their own processing, use Firehose. Firehose is easier to push data into s3, Redshift or any downstream application at scale. In this use case, when the user clicks to the webpage, all the information is posted to the web server and then is pushed into Firehose. Firehose processes the data and every minute generates a new file to push to the S3 bucket.

40:33 Approach 2: Kinesis Data Stream

From a web application data is published to web server then sent into Kinesis Data Stream then the lambda function transforms it in real time, and the data is pushed into a dynmodb and the dashboard reads info from DynamoDB and displays it in the webpage in real time.

44:00 Infrastructure Cost and Considerations

• Data Stream size (what is your optimal size of data to stream)
• Data Storage Size (need to set up appropriate folder structures to easily navigate and find files as needed)
• Processing Cost (how many messages are you streaming in your environment, how many lambda functions will be executed concurrently?)
• Network cost

To find more information review the Amazon pricing calculator.

46:15 Security Considerations

• Data Encryption in transit and at rest
• Access to data using Least Privilege Access model
• Data Compliance standards to secure PII data
• Appropriate Network security to detect attacks
• Restrict data collection by Application Key and Application Domain
• Monitor Kinesis calls

58:40 Real Time Data Driven Analytics at Scale using AWS

See our offer with AWS to support you in understanding your customer journey with real-time insights. We are currently offering a 3-week starter pack to set-up your use case, implement your real-time solution and provide basic level reporting to deliver insights to your team.