Category Archives: Big Data Hadoop

Kinesis_firehose_example

Posted on by Sumit Kumar

sudo yum install –y aws-kinesis-agent cd /etc/aws-kinesis/ sudo vi agent.json sudo service aws-kinesis-agent start sudo chkconfig aws-kinesis-agent on python3 LogGenerator.py 1000 cd /var/log/aws-kinesis-agent/ tail -f aws-kinesis-agent.log [ec2-user@ip-172-31-24-247 aws-kinesis]$ cat agent.json { “cloudwatch.emitMetrics”: true, “kinesis.endpoint”: “”, “firehose.endpoint”: “”, “flows”: [ { “filePattern”: “/home/ec2-user/*.log*”, “deliveryStream”: “kinesis_log_s3” } ] } ######### import names import random import time import […]

Install pyspark on windows

Posted on by Sumit Kumar

In this Post we will learn how to setup learning environment for pyspark in windows. To learning spark with python, we will install pyspark in windows and we will use jupyter notebook and spider IDE to test and run pyspark code. Prerequisite:- Java should be installed. If java is not installed please install java then […]

Reason to learn scala

Posted on by Sumit Kumar

Scala is an acronym for Scalable Language. Scala is a modern multi-paradigm programming language designed to express common programming patterns in a concise ,elegant and type-safe way. multi-paradigm programming means it supports Object oriented programming as well as fuctional programming. So scala is a scalable programming language for component software with the focus on pattern […]

Install Scala in windows

Posted on by Sumit Kumar

Prerequisite :–Java should be installed. Please follow below step to install scala in Window :–   1)Download scala :-Click on below link to download scala. https://www.scala-lang.org/download/   2)After download unzip it and set class path.   3)After setting class path go to command prompt and type scala. After that you can practice scala program in […]