Tuesday, April 24, 2018
Sunday, April 22, 2018
Kafka partitions
- Each topic has one or more partitions
- The no of topics in kafka is dependent on the circumstances in which Apache Kafka is intended to be used.It can be configurable
- A partition is the basis for which kafka can
- Scale
- Become fault-tolerant
- Achieve higher level of throughput
- Each partitions are maintained at at-least one or more brokers
Overview of Pig
- Apache Pig is a high-level platform for creating programs that run on Apache Hadoop.
- The language for this platform is called Pig Latin.
- Pig can execute its Hadoop jobs in MapReduce, Apache Tez, or Apache Spark
Local mode
- In local mode, Pig runs in a single JVM and access the local file system. This mode is suitable only for small data sets but not for big data sets.
- We can set this local mode execution type by using “X” or “exectype” option. To run in local mode, set the option to local
Shell script
- Interprets user command which are directly entered by the user or which are read from a file called shell script/program
- Shell script are interpreted
- Typical operations performed by shell scripts include file manipulation, program execution, and printing text
Span vs div (CSS)
div
is a block element, span
is inline.
This means that to use them semantically, divs should be used to wrap sections of a document, while spans should be used to wrap small portions of text, images, etc.
For example:
<div>This a large main division, with <span>a small bit</span> of spanned text!</div>
Example
Display block vs inline vs inline-block
Sample html
<!DOCTYPE html>
<html>
<head>
<style>
.floating-box {
display: inline-block;
width: 150px;
height: 75px;
margin: 10px;
border: 3px solid #73AD21;
}
.after-box {
border: 3px solid red;
}
</style>
</head>
<body>
<h2>The New Way - using inline-block</h2>
<div class="floating-box">Floating box</div>
<div class="floating-box">Floating box</div>
<div class="floating-box">Floating box</div>
<div class="floating-box">Floating box</div>
<div class="floating-box">Floating box</div>
<div class="floating-box">Floating box</div>
<div class="floating-box">Floating box</div>
<div class="floating-box">Floating box</div>
<div class="after-box">Another box, after the floating boxes...</div>
</body>
</html>
Saturday, April 14, 2018
Simple demo using Kafka
Start ZooKeeper
1 | bin/zookeeper-server-start.sh config/zookeeper.properties |
Start telnet
1 | telnet localhost 2181 |
Create topic
1 | bin/kafka-topics.sh --create --zookeeper localhost:2181 --replication-factor 1 --partitions 1 --topic test |
List kafka Topics
1 | bin/kafka-topics.sh --list --zookeeper localhost:2181 |
Subscribe to:
Posts (Atom)
Labels
- Algorithms (52)
- Apache Kafka (7)
- Apache Spark (21)
- Architecture (8)
- Arrays (23)
- Big Data (98)
- Cloud services (6)
- Cognitive technologies (12)
- Data Analytics (3)
- Data Science (6)
- Design (1)
- devOps (1)
- Hadoop (26)
- Hive (11)
- Java (2)
- JavaScript (65)
- JavaScript Run-time (12)
- Machine learning (11)
- Maths (6)
- mongoDb (2)
- MySQL (1)
- Networking (3)
- No SQL (2)
- Node (20)
- Python (28)
- Security (4)
- Spark Grpahx (1)
- Spark MLlib (1)
- Spark Sql (3)
- Spark Streaming (4)
- SQL (40)
- Sqoop (2)
- ssis (3)
- Strings (13)