Showing posts with label Apache Hadoop. Show all posts
Showing posts with label Apache Hadoop. Show all posts

Daftar perintah dasar Apache Hadoop


Daftar  perintah  dasar  Apache Hadoop

Tutorial  Linux  Indonesia  -  Apache  Hadoop  salah  satu  perangkat  lunak  open  source  yang  digunakan  untuk   Big Data  Storage  dengan  skala  data  besar. 
Daftar  perintah  dasar  Apache Hadoop


Sudah banyak  perusahaan  besar   menggunakan  Apache  Hadoop untuk  Data  Storage  mereka seperti, google.com, yahoo.com dan  facebook.com.

Mereka  menggunakan  apache  hadoop  untuk  cluster  data  storage untuk layanan  mereka.

Dalam  panduan  ini,  akan  membahas  perintah  dasar   apache  hadoop  yang  sering  digunakan.

HDFS  DFS –MKDIR

Perintah  di  bawah  ini  digunakan  untuk  buat  dierektory  di system HDFS.

[hadoop@master ~]$ hdfs dfs  -mkdir /www_bagol69_com

HDFS DFS –PUT

Perintah  ini   digunakan  untuk  menyalin  data  dari  system  local  ke   system HDFS.

[hadoop@master ~]$ hdfs dfs  -put  coba2 /www_bagol69_com

HDFS DFS – CAT

Digunakan  untuk   melihat  isi   file  yang  sudah  berhasil  di  upload  ke  directory HADOOP  yang  sudah  di  buat   sebelumnya.

[hadoop@master ~]$ hdfs dfs  -cat /bagol69adi/coba
coba  push  ke   HADOOP   HDFS SALAM  BAGOL69.COM
[hadoop@master ~]$

HDFS  DFS  -GET

Digunakan  untuk   menampilkan  directory  HADOOP di system  linux  anda.

[hadoop@master ~]$ hdfs dfs -get /www_bagol69_com/ ./
[hadoop@master ~]$ ls
bagol69adi   coba   hdfstest.txt  libexec      my_storage  sbin
bagol69_com  coba2  include       LICENSE.txt  NOTICE.txt  share
bin          etc    lib           logs         README.txt  www_bagol69_com


HDFS  DFS –LS

Perintah  ini  digunakan untuk  melihat   jumlah   isi  file, pemilik   file,  hak  akses  file  di  directory.

[hadoop@master ~]$ hdfs dfs -ls  /my_storage
Found 2 items
-rw-r--r--   3 hadoop supergroup     106210 2018-10-29 02:24 /my_storage/LICENSE.txt
-rw-r--r--   3 hadoop supergroup         50 2018-10-29 14:22 /my_storage/coba
[hadoop@master ~]$

HDFS DFS -RM 

Digunakan  untuk  menghapus  file  atau direktory  yang  sudah  tidak  terpakai. 

[hadoop@master my_storage]$ hdfs dfs -rm  /my_storage/LICENSE.txt
Deleted /my_storage/LICENSE.txt
 
[hadoop@master my_storage]$ hdfs  dfs  -rm  -r  /my_storage
Deleted /my_storage


 Untuk  artikel  Daftar  Perintah  Dasar Apache  Hadoop sudah  selesai. Nanti  masih  membahas   Apache  Hadoop sampai  tuntas.  Silahkan  di  share  supaya banyak  teman  yang  tau  tentang  apache  hadoop. 
Read More

Cara Menginstall dan Konfigurasi Apache Hadoop on single node di Centos 7

Cara  Menginstall dan Konfigurasi  Apache  Hadoop  on   single  node  di  Centos 7.

Tutorial  Linux  Indonesia  --  Apache  Hadoop  meruapakan  salah  satu   tools  open  source  yang  digunakan  untuk  mendistribusikan   data  big  storage  dan    memproses  data  di   seluruh   computer  cluster.

Cara  Menginstall dan Konfigurasi  Apache  Hadoop  on   single  node  di  Centos 7

Proyek  ini   di  dasarkan  pada  komponen-komponen   sebagai  berikut;

Hadoop  Common     yang  berisi  pustaka    dan  utilitas  java   yang  di butuhkan  oleh  modul  hadoop  lainya. 

HDFS (  Hadoop  Distribute  File  System )  Sistem  File  scalable   berbaisis  java  yang  di distribusikan  di  beberapa  node. 

Map Reduce  kerangka  YARN, untuk   melakuka  pemrosesan  data  besar   parallel.

Hadoop  Yarn         merupakan  kerangka  manajemen   sumber  daya  cluster.

Artikel  ini  akan  memandu anda  tentang  cara  menginstall  dan  konfigurasi   Apache  Hadoop  pada  satu  node  di  Centos  7. 

Jenis  konfigurasi  ini  juga di   referensikan  sebagai   Hadoop Pseudo-Disributed Mode.


Install  Java  di  Centos  7

Pertama  yang  kamu  lakukan  adalah  instalasi  java    yang  terbaru.  Karena  Apache   Hadoop   membutuhkan  java   untuk  mendukung  utulitas  mereka. 

Sekarang   anda  dapat    install  java  menggunakan  command  di  bawah  ini. 

[root@master ~]# yum  install java-1.8.0-openjdk-devel –y

Setelah   melakukan  instalasi  java, untuk  memastikan   java  sudah   berhasil    terinstall. Silahkan  melakukan  check  versi   dari  java tersebut.  Menggunakan  command  di  bawah ini.

[root@master ~]# java -version
openjdk version "1.8.0_191"
OpenJDK Runtime Environment (build 1.8.0_191-b12)
OpenJDK 64-Bit Server VM (build 25.191-b12, mixed mode)
[root@master ~]#

Instalasi  Apache  Hadoop   Single  Node

Selanjutnya,  buat  akun  pengguna   baru  di  system  anda  tanpa  kekuatan   dari  root.  Yang  akan  kami  gunakan    untuk  jalur  pemasangan  hadoop  dan   alur  kerja dari  hadoop.

Direcotry  home  yang  digunakan  oleh  account  baru  adalah   /opt/hadoop

[root@master ~]# useradd -d /opt/hadoop hadoop
[root@master ~]# passwd hadoop

Sekarang  anda   dapat    melakukan  download  Apache  Hadoop  menggunakan  command  di  bawah ini.

[root@master ~]# # curl -O http://apache.javapipe.com/hadoop/common/hadoop-2.9.1/hadoop-2.9.1.tar.gz

Lakukan   extract     file   hadoop  yang  sudah anda  download   sebelumnya.  Jangan  lupa  untuk   berikan   akses   user  hadoop  terhadap   folder  /opt/hadoop/ .

[root@master ~]# tar xfz hadoop-2.9.1.tar.gz
[root@master ~]# cp  -rf hadoop-2.9.1/* /opt/hadoop/
[root@master ~]# chown -R hadoop:hadoop /opt/hadoop/

Pastikan   semua    file   di  folder   /opt/hadoop  sudah   di  berikan  akses  terhadap  user   hadoop.

[root@master ~]# ls  -al /opt/hadoop/
total 152
drwx------. 12 hadoop hadoop    284 Oct 29 02:27 .
drwxr-xr-x.  4 root   root       34 Oct 29 01:49 ..
-rw-------.  1 hadoop hadoop   1077 Oct 29 02:28 .bash_history
-rw-r--r--.  1 hadoop hadoop     18 Apr 10  2018 .bash_logout
-rw-r--r--.  1 hadoop hadoop    703 Oct 29 01:36 .bash_profile
-rw-r--r--.  1 hadoop hadoop    231 Apr 10  2018 .bashrc
drwxr-xr-x.  2 hadoop hadoop    194 Oct 29 02:04 bin
drwxr-xr-x.  3 hadoop hadoop     20 Oct 29 01:17 etc
drwxr-xr-x.  2 hadoop hadoop    106 Oct 29 01:17 include
drwxr-xr-x.  3 hadoop hadoop     20 Oct 29 01:17 lib
drwxr-xr-x.  2 hadoop hadoop    239 Oct 29 01:17 libexec
-rw-r--r--.  1 hadoop hadoop 106210 Oct 29 01:17 LICENSE.txt
drwxrwxr-x.  3 hadoop hadoop   4096 Oct 29 02:17 logs
drwxrwxr-x.  2 hadoop hadoop     25 Oct 29 02:27 my_storage
-rw-r--r--.  1 hadoop hadoop  15915 Oct 29 01:17 NOTICE.txt
-rw-r--r--.  1 hadoop hadoop   1366 Oct 29 01:17 README.txt
drwxr-xr-x.  3 hadoop hadoop   4096 Oct 29 01:17 sbin
drwxr-xr-x.  4 hadoop hadoop     31 Oct 29 01:17 share
drwx------.  2 hadoop hadoop     80 Oct 29 01:43 .ssh
-rw-------.  1 hadoop hadoop    835 Oct 29 02:07 .viminfo
[root@master ~]#

Selanjutnya,  login  dengan  pengguna  hadoop yang  sudah  anda  buat  sebelumnya.  

[root@master ~]# su - hadoop
Last login: Mon Oct 29 02:20:51 EDT 2018 on tty1

Konfigurasi  Hadoop   dan  Java  Environment  Variables  pada  sistem  anda  dengan  mengedit   .bash_profile

[hadoop@master ~]$ vi .bash_profile

Tambahkan  baris  berikut  ini   di  akhir  file;

export JAVA_HOME=/usr/java/default
export PATH=$PATH:$JAVA_HOME/bin
export CLASSPATH=.:$JAVA_HOME/jre/lib:$JAVA_HOME/lib:$JAVA_HOME/lib/tools.jar

export HADOOP_HOME=/opt/hadoop
export HADOOP_COMMON_HOME=$HADOOP_HOME
export HADOOP_HDFS_HOME=$HADOOP_HOME
export HADOOP_MAPRED_HOME=$HADOOP_HOME
export HADOOP_YARN_HOME=$HADOOP_HOME
export HADOOP_OPTS="-Djava.library.path=$HADOOP_HOME/lib/native"
export HADOOP_COMMON_LIB_NATIVE_DIR=$HADOOP_HOME/lib/native
export PATH=$PATH:$HADOOP_HOME/sbin:$HADOOP_HOME/bin

Pastikan   variable  java  dan  variable   hadoop  anda  sudah  benar. Menggunakan  command  di  bawah  ini. 

[hadoop@master ~]$ source .bash_profile
[hadoop@master ~]$ echo $HADOOP_HOME
/opt/hadoop
[hadoop@master ~]$ echo $JAVA_HOME
/usr/java/default
[hadoop@master ~]$

Terakhir, konfigurasikan authentikasi  berbasis   kunci  ssh   untuk  akun  hadoop dengan menjalankan   perintah  di  bawah  ini. 

[hadoop@master ~]$ssh-keygen -t rsa
[hadoop@master ~]$ssh-copy-id master.bagol69.com

Sekarang  saat  untuk  melakukan  men-setup   cluster  hadoop  pada   satu  node  dengan  mode  pseudo  distributed    dengan  mengedit  file   konfigurasinya. 

Lokasi   file  konfigurasi     hadoop   adalah    $HADOOP_HOME  /etc/hadoop/     yang  di wakili  dalam    tutorial    ini   /opt/hadoop   (  akun   rumah   pengguna    hadoop ).

Setelah  anda  masuk  ke  pengguna  hadoop,  anda  dapat  mulai   melakukan   edit  file  tersebut. 

Yang  pertama  yang  anda  edit  adalah     core-site.xml.   File  ini  berisi    informasi   tentang  nomor  port  yang  digunakan  oleh   hadoop,  sistem    file  yang  mengolakasikan   memory, batas memory  penyimpanan  data    dan ukuran   Read  /Write  ( buffer  ).

Anda  tambahkan  property  berikut  di  antara  tag 

   fs.defaultFS
   hdfs://master.bagol69.com:9000/


 [hadoop@master ~]$ nano  etc/hadoop/core-site.xml
   fs.defaultFS
   hdfs://master.bagol69.com:9000/



Selanjutnya,  buka   dan   edit  file   hdf-site.xml  file  ini  berisi  informasi   tentang  nilai  data   replikasi, jalur   namenode dan  jalur  data node untuk  sistem  local.

Anda   tambahkan  perintah   bawah  ini   di  bagian  

  dfs.data.dir
  file:///opt/volume/datanode

 dfs.name.dir
 file:///opt/volume/namenode


Pada  panduan  ini,  saya  akan  menggunakan  /opt/volume   directory  untuk menyimpan  sistem  file   hadoop  saya. 

[hadoop@master ~]$ nano  etc/hadoop/hdfs-site.xml
  dfs.data.dir
  file:///opt/volume/datanode

 dfs.name.dir
 file:///opt/volume/namenode



Karena  saya  telah menentukan  /opt/volume  sebagai  penyimpanan sistem  berkas    hadoop  saya.    

Saya  perlu  membuat  directory    (  datanode  dan   namenode  )  dari  akun  root  dan  memberikan semua  izin  akses  ke  pengguna   hadoop.  Dengan  menjalankan  perintah  di  bawah ini. 

[root@master ~]# mkdir -p /opt/volume/namenode/
[root@master ~]# mkdir -p /opt/volume/datanode/
[root@master ~]# chown -R hadoop:hadoop /opt/volume/
[root@master ~]# ls  -al /opt/
total 0
drwxr-xr-x.  4 root   root    34 Oct 29 01:49 .
dr-xr-xr-x. 17 root   root   224 Oct 28 23:59 ..
drwx------. 12 hadoop hadoop 284 Oct 29 03:36 hadoop
drwxr-xr-x.  4 hadoop hadoop  38 Oct 29 01:49 volume
[root@master ~]#
[root@master ~]# exit
logout
[hadoop@master ~]$

Sekarang  edit  file   mapred-site.xml  file  ini  untuk menentukan  bahwa  saya  menggunakan  kerangka   MapReduce  Frame Work.

[hadoop@master ~]$ nano  etc/hadoop/mapred-site.xml

Tambahkan  perintah  ini   di   bagian   tag 

 mapreduce.framework.name
 yarn



Sekarang  edit   file   yarn-site.xml   dan   tambahkan  perintah  setelah  tag 

[hadoop@master ~]$ nano  etc/hadoop/yarn-site.xml
  yarn.nodemanager.aux-services
  mapreduce_shuffle



Terakhir,    setting   Java  Home    untuk  hadoop   environment  di  file   hadoop-env.sh

[hadoop@master ~]$ nano  etc/hadoop/hadoop-env.sh

Sebelum

export JAVA_HOME=/usr/java/default

Sesudah

# The java implementation to use.
export JAVA_HOME=$(readlink -f /usr/bin/java | sed "s:bin/java::")



Setelah  itu,  ganti  nilai  localhost  dari  file  slaves    untuk menunjuk  ke  hostname  pada  mesin  linux  anda. 

[hadoop@master ~]$ nano  etc/hadoop/slaves
master.bagol69.com

Setelah hadoop   single  node  sudah  di   konfigurasi.  Saat  untuk   menginisialisasi  sistem   file  HDFS  dengan   menformat  directory  pennyimpanan   /opt/volume/namenode   dengan  menggunakan  perintah  ini.

[hadoop@master ~]$ hdfs namenode -format
18/10/29 03:51:15 INFO namenode.NameNode: STARTUP_MSG:
/************************************************************
STARTUP_MSG: Starting NameNode
STARTUP_MSG:   host = master.bagol69.com/10.10.10.1
STARTUP_MSG:   args = [-format]
STARTUP_MSG:   version = 2.9.1
STARTUP_MSG:   classpath = /opt/hadoop/etc/hadoop:/opt/hadoop/share/hadoop/commo                                                                                n/lib/commons-collections-3.2.2.jar:/opt/hadoop/share/hadoop/common/lib/jetty-6.                                                                                1.26.jar:/opt/hadoop/share/hadoop/common/lib/jetty-util-6.1.26.jar:/opt/hadoop/s                                                                                hare/hadoop/common/lib/jetty-sslengine-6.1.26.jar:/opt/hadoop/share/hadoop/commo                                                                                n/lib/jsp-api-2.1.jar:/opt/hadoop/share/hadoop/common/lib/jersey-core-1.9.jar:/o                                                                                pt/hadoop/share/hadoop/common/lib/jersey-json-1.9.jar:/opt/hadoop/share/hadoop/c                                                                                ommon/lib/jettison-1.1.jar:/opt/hadoop/share/hadoop/common/lib/jaxb-impl-2.2.3-1                                                                                .jar:/opt/hadoop/share/hadoop/common/lib/jaxb-api-2.2.2.jar:/opt/hadoop/share/ha                                                                                doop/common/lib/stax-api-1.0-2.jar:/opt/hadoop/share/hadoop/common/lib/activatio                                                                                n-1.1.jar:/opt/hadoop/share/hadoop/common/lib/jackson-core-asl-1.9.13.jar:/opt/h                                                                                adoop/share/hadoop/common/lib/jackson-mapper-asl-1.9.13.jar:/opt/hadoop/share/ha                                                                                doop/common/lib/jackson-jaxrs-1.9.13.jar:/opt/hadoop/share/hadoop/common/lib/jac                                                                                kson-xc-1.9.13.jar:/opt/hadoop/share/hadoop/common/lib/jersey-server-1.9.jar:/op                                                                                t/hadoop/share/hadoop/common/lib/asm-3.2.jar:/opt/hadoop/share/hadoop/common/lib                                                                                /log4j-1.2.17.jar:/opt/hadoop/share/hadoop/common/lib/jets3t-0.9.0.jar:/opt/hado                                                                                op/share/hadoop/common/lib/java-xmlbuilder-0.4.jar:/opt/hadoop/share/hadoop/comm                                                                                on/lib/commons-lang-2.6.jar:/opt/hadoop/share/hadoop/common/lib/commons-configur                                                                                ation-1.6.jar:/opt/hadoop/share/hadoop/common/lib/commons-digester-1.8.jar:/opt/                                                                                hadoop/share/hadoop/common/lib/commons-beanutils-1.7.0.jar:/opt/hadoop/share/had                                                                                oop/common/lib/commons-beanutils-core-1.8.0.jar:/opt/hadoop/share/hadoop/common/                                                                                lib/commons-lang3-3.4.jar:/opt/hadoop/share/hadoop/common/lib/slf4j-api-1.7.25.j                                                                                ar:/opt/hadoop/share/hadoop/common/lib/slf4j-log4j12-1.7.25.jar:/opt/hadoop/shar                                                                                e/hadoop/common/lib/avro-1.7.7.jar:/opt/hadoop/share/hadoop/common/lib/paranamer                                                                                -2.3.jar:/opt/hadoop/share/hadoop/common/lib/snappy-java-1.0.5.jar:/opt/hadoop/s                                                                                hare/hadoop/common/lib/commons-compress-1.4.1.jar:/opt/hadoop/share/hadoop/commo                                                                                n/lib/xz-1.0.jar:/opt/hadoop/share/hadoop/common/lib/protobuf-java-2.5.0.jar:/op                                                                                t/hadoop/share/hadoop/common/lib/gson-2.2.4.jar:/opt/hadoop/share/hadoop/common/                                                                                lib/hadoop-auth-2.9.1.jar:/opt/hadoop/share/hadoop/common/lib/nimbus-jose-jwt-4.                                                                                41.1.jar:/opt/hadoop/share/hadoop/common/lib/jcip-annotations-1.0-1.jar:/opt/had                                                                                oop/share/hadoop/common/lib/json-smart-1.3.1.jar:/opt/hadoop/share/hadoop/common                                                                                /lib/apacheds-kerberos-codec-2.0.0-M15.jar:/opt/hadoop/share/hadoop/common/lib/a                                                                                pacheds-i18n-2.0.0-M15.jar:/opt/hadoop/share/hadoop/common/lib/api-asn1-api-1.0.                                                                                0-M20.jar:/opt/hadoop/share/hadoop/common/lib/api-util-1.0.0-M20.jar:/opt/hadoop                                                                                /share/hadoop/common/lib/zookeeper-3.4.6.jar:/opt/hadoop/share/hadoop/common/lib                                                                                /netty-3.6.2.Final.jar:/opt/hadoop/share/hadoop/common/lib/curator-framework-2.7                                                                                .1.jar:/opt/hadoop/share/hadoop/common/lib/curator-client-2.7.1.jar:/opt/hadoop/                                                                                share/hadoop/common/lib/jsch-0.1.54.jar:/opt/hadoop/share/hadoop/common/lib/cura                                                                                tor-recipes-2.7.1.jar:/opt/hadoop/share/hadoop/common/lib/htrace-core4-4.1.0-inc                                                                                ubating.jar:/opt/hadoop/share/hadoop/common/lib/stax2-api-3.1.4.jar:/opt/hadoop/                                                                                share/hadoop/common/lib/woodstox-core-5.0.3.jar:/opt/hadoop/share/hadoop/common/                                                                                lib/junit-4.11.jar:/opt/hadoop/share/hadoop/common/lib/hamcrest-core-1.3.jar:/op                                                                                t/hadoop/share/hadoop/common/lib/mockito-all-1.8.5.jar:/opt/hadoop/share/hadoop/                                                                                common/lib/hadoop-annotations-2.9.1.jar:/opt/hadoop/share/hadoop/common/lib/guav                                                                                a-11.0.2.jar:/opt/hadoop/share/hadoop/common/lib/jsr305-3.0.0.jar:/opt/hadoop/sh                                                                                are/hadoop/common/lib/commons-cli-1.2.jar:/opt/hadoop/share/hadoop/common/lib/co                                                                                mmons-math3-3.1.1.jar:/opt/hadoop/share/hadoop/common/lib/xmlenc-0.52.jar:/opt/h                                                                                adoop/share/hadoop/common/lib/httpclient-4.5.2.jar:/opt/hadoop/share/hadoop/comm                                                                                on/lib/httpcore-4.4.4.jar:/opt/hadoop/share/hadoop/common/lib/commons-logging-1.                                                                                1.3.jar:/opt/hadoop/share/hadoop/common/lib/commons-codec-1.4.jar:/opt/hadoop/sh                                                                                are/hadoop/common/lib/commons-io-2.4.jar:/opt/hadoop/share/hadoop/common/lib/com                                                                                mons-net-3.1.jar:/opt/hadoop/share/hadoop/common/lib/servlet-api-2.5.jar:/opt/ha                                                                                doop/share/hadoop/common/hadoop-common-2.9.1.jar:/opt/hadoop/share/hadoop/common                                                                                /hadoop-common-2.9.1-tests.jar:/opt/hadoop/share/hadoop/common/hadoop-nfs-2.9.1.                                                                                jar:/opt/hadoop/share/hadoop/hdfs:/opt/hadoop/share/hadoop/hdfs/lib/commons-code                                                                                c-1.4.jar:/opt/hadoop/share/hadoop/hdfs/lib/log4j-1.2.17.jar:/opt/hadoop/share/h                                                                                adoop/hdfs/lib/commons-logging-1.1.3.jar:/opt/hadoop/share/hadoop/hdfs/lib/netty                                                                                -3.6.2.Final.jar:/opt/hadoop/share/hadoop/hdfs/lib/guava-11.0.2.jar:/opt/hadoop/                                                                                share/hadoop/hdfs/lib/jsr305-3.0.0.jar:/opt/hadoop/share/hadoop/hdfs/lib/commons                                                                                -cli-1.2.jar:/opt/hadoop/share/hadoop/hdfs/lib/xmlenc-0.52.jar:/opt/hadoop/share                                                                                /hadoop/hdfs/lib/commons-io-2.4.jar:/opt/hadoop/share/hadoop/hdfs/lib/servlet-ap                                                                                i-2.5.jar:/opt/hadoop/share/hadoop/hdfs/lib/jetty-6.1.26.jar:/opt/hadoop/share/h                                                                                adoop/hdfs/lib/jetty-util-6.1.26.jar:/opt/hadoop/share/hadoop/hdfs/lib/jersey-co                                                                                re-1.9.jar:/opt/hadoop/share/hadoop/hdfs/lib/jackson-core-asl-1.9.13.jar:/opt/ha                                                                                doop/share/hadoop/hdfs/lib/jackson-mapper-asl-1.9.13.jar:/opt/hadoop/share/hadoo                                                                                p/hdfs/lib/jersey-server-1.9.jar:/opt/hadoop/share/hadoop/hdfs/lib/asm-3.2.jar:/                                                                                opt/hadoop/share/hadoop/hdfs/lib/commons-lang-2.6.jar:/opt/hadoop/share/hadoop/h                                                                                dfs/lib/protobuf-java-2.5.0.jar:/opt/hadoop/share/hadoop/hdfs/lib/htrace-core4-4                                                                                .1.0-incubating.jar:/opt/hadoop/share/hadoop/hdfs/lib/hadoop-hdfs-client-2.9.1.j                                                                                ar:/opt/hadoop/share/hadoop/hdfs/lib/okhttp-2.7.5.jar:/opt/hadoop/share/hadoop/h                                                                                dfs/lib/okio-1.6.0.jar:/opt/hadoop/share/hadoop/hdfs/lib/commons-daemon-1.0.13.j                                                                                ar:/opt/hadoop/share/hadoop/hdfs/lib/netty-all-4.0.23.Final.jar:/opt/hadoop/shar                                                                                e/hadoop/hdfs/lib/xercesImpl-2.9.1.jar:/opt/hadoop/share/hadoop/hdfs/lib/xml-api                                                                                s-1.3.04.jar:/opt/hadoop/share/hadoop/hdfs/lib/leveldbjni-all-1.8.jar:/opt/hadoo                                                                                p/share/hadoop/hdfs/lib/jackson-databind-2.7.8.jar:/opt/hadoop/share/hadoop/hdfs                                                                                /lib/jackson-annotations-2.7.8.jar:/opt/hadoop/share/hadoop/hdfs/lib/jackson-cor                                                                                e-2.7.8.jar:/opt/hadoop/share/hadoop/hdfs/hadoop-hdfs-2.9.1.jar:/opt/hadoop/shar                                                                                e/hadoop/hdfs/hadoop-hdfs-2.9.1-tests.jar:/opt/hadoop/share/hadoop/hdfs/hadoop-h                                                                                dfs-nfs-2.9.1.jar:/opt/hadoop/share/hadoop/hdfs/hadoop-hdfs-client-2.9.1.jar:/op                                                                                t/hadoop/share/hadoop/hdfs/hadoop-hdfs-client-2.9.1-tests.jar:/opt/hadoop/share/                                                                                hadoop/hdfs/hadoop-hdfs-native-client-2.9.1.jar:/opt/hadoop/share/hadoop/hdfs/ha                                                                                doop-hdfs-native-client-2.9.1-tests.jar:/opt/hadoop/share/hadoop/hdfs/hadoop-hdf                                                                                s-rbf-2.9.1.jar:/opt/hadoop/share/hadoop/hdfs/hadoop-hdfs-rbf-2.9.1-tests.jar:/o                                                                                pt/hadoop/share/hadoop/yarn:/opt/hadoop/share/hadoop/yarn/lib/commons-cli-1.2.ja                                                                                r:/opt/hadoop/share/hadoop/yarn/lib/log4j-1.2.17.jar:/opt/hadoop/share/hadoop/ya                                                                                rn/lib/commons-compress-1.4.1.jar:/opt/hadoop/share/hadoop/yarn/lib/xz-1.0.jar:/                                                                                opt/hadoop/share/hadoop/yarn/lib/servlet-api-2.5.jar:/opt/hadoop/share/hadoop/ya                                                                                rn/lib/commons-codec-1.4.jar:/opt/hadoop/share/hadoop/yarn/lib/jetty-util-6.1.26                                                                                .jar:/opt/hadoop/share/hadoop/yarn/lib/jersey-core-1.9.jar:/opt/hadoop/share/had                                                                                oop/yarn/lib/jersey-client-1.9.jar:/opt/hadoop/share/hadoop/yarn/lib/jackson-cor                                                                                e-asl-1.9.13.jar:/opt/hadoop/share/hadoop/yarn/lib/jackson-mapper-asl-1.9.13.jar                                                                                :/opt/hadoop/share/hadoop/yarn/lib/jackson-jaxrs-1.9.13.jar:/opt/hadoop/share/ha                                                                                doop/yarn/lib/jackson-xc-1.9.13.jar:/opt/hadoop/share/hadoop/yarn/lib/guice-serv                                                                                let-3.0.jar:/opt/hadoop/share/hadoop/yarn/lib/guice-3.0.jar:/opt/hadoop/share/ha                                                                                doop/yarn/lib/javax.inject-1.jar:/opt/hadoop/share/hadoop/yarn/lib/aopalliance-1                                                                                .0.jar:/opt/hadoop/share/hadoop/yarn/lib/commons-io-2.4.jar:/opt/hadoop/share/ha                                                                                doop/yarn/lib/jersey-server-1.9.jar:/opt/hadoop/share/hadoop/yarn/lib/asm-3.2.ja                                                                                r:/opt/hadoop/share/hadoop/yarn/lib/jersey-json-1.9.jar:/opt/hadoop/share/hadoop                                                                                /yarn/lib/jettison-1.1.jar:/opt/hadoop/share/hadoop/yarn/lib/jaxb-impl-2.2.3-1.j                                                                                ar:/opt/hadoop/share/hadoop/yarn/lib/jersey-guice-1.9.jar:/opt/hadoop/share/hado                                                                                op/yarn/lib/commons-math3-3.1.1.jar:/opt/hadoop/share/hadoop/yarn/lib/xmlenc-0.5                                                                                2.jar:/opt/hadoop/share/hadoop/yarn/lib/httpclient-4.5.2.jar:/opt/hadoop/share/h                                                                                adoop/yarn/lib/httpcore-4.4.4.jar:/opt/hadoop/share/hadoop/yarn/lib/commons-net-                                                                                3.1.jar:/opt/hadoop/share/hadoop/yarn/lib/commons-collections-3.2.2.jar:/opt/had                                                                                oop/share/hadoop/yarn/lib/jetty-6.1.26.jar:/opt/hadoop/share/hadoop/yarn/lib/jet                                                                                ty-sslengine-6.1.26.jar:/opt/hadoop/share/hadoop/yarn/lib/jsp-api-2.1.jar:/opt/h                                                                                adoop/share/hadoop/yarn/lib/jets3t-0.9.0.jar:/opt/hadoop/share/hadoop/yarn/lib/j                                                                                ava-xmlbuilder-0.4.jar:/opt/hadoop/share/hadoop/yarn/lib/commons-configuration-1                                                                                .6.jar:/opt/hadoop/share/hadoop/yarn/lib/commons-digester-1.8.jar:/opt/hadoop/sh                                                                                are/hadoop/yarn/lib/commons-beanutils-1.7.0.jar:/opt/hadoop/share/hadoop/yarn/li                                                                                b/commons-beanutils-core-1.8.0.jar:/opt/hadoop/share/hadoop/yarn/lib/commons-lan                                                                                g3-3.4.jar:/opt/hadoop/share/hadoop/yarn/lib/avro-1.7.7.jar:/opt/hadoop/share/ha                                                                                doop/yarn/lib/paranamer-2.3.jar:/opt/hadoop/share/hadoop/yarn/lib/snappy-java-1.                                                                                0.5.jar:/opt/hadoop/share/hadoop/yarn/lib/gson-2.2.4.jar:/opt/hadoop/share/hadoo                                                                                p/yarn/lib/nimbus-jose-jwt-4.41.1.jar:/opt/hadoop/share/hadoop/yarn/lib/jcip-ann                                                                                otations-1.0-1.jar:/opt/hadoop/share/hadoop/yarn/lib/json-smart-1.3.1.jar:/opt/h                                                                                adoop/share/hadoop/yarn/lib/apacheds-kerberos-codec-2.0.0-M15.jar:/opt/hadoop/sh                                                                                are/hadoop/yarn/lib/apacheds-i18n-2.0.0-M15.jar:/opt/hadoop/share/hadoop/yarn/li                                                                                b/api-asn1-api-1.0.0-M20.jar:/opt/hadoop/share/hadoop/yarn/lib/api-util-1.0.0-M2                                                                                0.jar:/opt/hadoop/share/hadoop/yarn/lib/zookeeper-3.4.6.jar:/opt/hadoop/share/ha                                                                                doop/yarn/lib/netty-3.6.2.Final.jar:/opt/hadoop/share/hadoop/yarn/lib/curator-fr                                                                                amework-2.7.1.jar:/opt/hadoop/share/hadoop/yarn/lib/curator-client-2.7.1.jar:/op                                                                                t/hadoop/share/hadoop/yarn/lib/jsch-0.1.54.jar:/opt/hadoop/share/hadoop/yarn/lib                                                                                /curator-recipes-2.7.1.jar:/opt/hadoop/share/hadoop/yarn/lib/htrace-core4-4.1.0-                                                                                incubating.jar:/opt/hadoop/share/hadoop/yarn/lib/stax2-api-3.1.4.jar:/opt/hadoop                                                                                /share/hadoop/yarn/lib/woodstox-core-5.0.3.jar:/opt/hadoop/share/hadoop/yarn/lib                                                                                /leveldbjni-all-1.8.jar:/opt/hadoop/share/hadoop/yarn/lib/geronimo-jcache_1.0_sp                                                                                ec-1.0-alpha-1.jar:/opt/hadoop/share/hadoop/yarn/lib/ehcache-3.3.1.jar:/opt/hado                                                                                op/share/hadoop/yarn/lib/HikariCP-java7-2.4.12.jar:/opt/hadoop/share/hadoop/yarn                                                                                /lib/mssql-jdbc-6.2.1.jre7.jar:/opt/hadoop/share/hadoop/yarn/lib/metrics-core-3.                                                                                0.1.jar:/opt/hadoop/share/hadoop/yarn/lib/fst-2.50.jar:/opt/hadoop/share/hadoop/                                                                                yarn/lib/java-util-1.9.0.jar:/opt/hadoop/share/hadoop/yarn/lib/json-io-2.5.1.jar                                                                                :/opt/hadoop/share/hadoop/yarn/lib/commons-lang-2.6.jar:/opt/hadoop/share/hadoop                                                                                /yarn/lib/guava-11.0.2.jar:/opt/hadoop/share/hadoop/yarn/lib/jsr305-3.0.0.jar:/o                                                                                pt/hadoop/share/hadoop/yarn/lib/commons-logging-1.1.3.jar:/opt/hadoop/share/hado                                                                                op/yarn/lib/jaxb-api-2.2.2.jar:/opt/hadoop/share/hadoop/yarn/lib/stax-api-1.0-2.                                                                                jar:/opt/hadoop/share/hadoop/yarn/lib/activation-1.1.jar:/opt/hadoop/share/hadoo                                                                                p/yarn/lib/protobuf-java-2.5.0.jar:/opt/hadoop/share/hadoop/yarn/hadoop-yarn-api                                                                                -2.9.1.jar:/opt/hadoop/share/hadoop/yarn/hadoop-yarn-common-2.9.1.jar:/opt/hadoo                                                                                p/share/hadoop/yarn/hadoop-yarn-registry-2.9.1.jar:/opt/hadoop/share/hadoop/yarn                                                                                /hadoop-yarn-server-common-2.9.1.jar:/opt/hadoop/share/hadoop/yarn/hadoop-yarn-s                                                                                erver-nodemanager-2.9.1.jar:/opt/hadoop/share/hadoop/yarn/hadoop-yarn-server-web                                                                                -proxy-2.9.1.jar:/opt/hadoop/share/hadoop/yarn/hadoop-yarn-server-applicationhis                                                                                toryservice-2.9.1.jar:/opt/hadoop/share/hadoop/yarn/hadoop-yarn-server-resourcem                                                                                anager-2.9.1.jar:/opt/hadoop/share/hadoop/yarn/hadoop-yarn-server-tests-2.9.1.ja                                                                                r:/opt/hadoop/share/hadoop/yarn/hadoop-yarn-client-2.9.1.jar:/opt/hadoop/share/h                                                                                adoop/yarn/hadoop-yarn-server-sharedcachemanager-2.9.1.jar:/opt/hadoop/share/had                                                                                oop/yarn/hadoop-yarn-server-timeline-pluginstorage-2.9.1.jar:/opt/hadoop/share/h                                                                                adoop/yarn/hadoop-yarn-server-router-2.9.1.jar:/opt/hadoop/share/hadoop/yarn/had                                                                                oop-yarn-applications-distributedshell-2.9.1.jar:/opt/hadoop/share/hadoop/yarn/h                                                                                adoop-yarn-applications-unmanaged-am-launcher-2.9.1.jar:/opt/hadoop/share/hadoop                                                                                /mapreduce/lib/protobuf-java-2.5.0.jar:/opt/hadoop/share/hadoop/mapreduce/lib/av                                                                                ro-1.7.7.jar:/opt/hadoop/share/hadoop/mapreduce/lib/jackson-core-asl-1.9.13.jar:                                                                                /opt/hadoop/share/hadoop/mapreduce/lib/jackson-mapper-asl-1.9.13.jar:/opt/hadoop                                                                                /share/hadoop/mapreduce/lib/paranamer-2.3.jar:/opt/hadoop/share/hadoop/mapreduce                                                                                /lib/snappy-java-1.0.5.jar:/opt/hadoop/share/hadoop/mapreduce/lib/commons-compre                                                                                ss-1.4.1.jar:/opt/hadoop/share/hadoop/mapreduce/lib/xz-1.0.jar:/opt/hadoop/share                                                                                /hadoop/mapreduce/lib/hadoop-annotations-2.9.1.jar:/opt/hadoop/share/hadoop/mapr                                                                                educe/lib/commons-io-2.4.jar:/opt/hadoop/share/hadoop/mapreduce/lib/jersey-core-                                                                                1.9.jar:/opt/hadoop/share/hadoop/mapreduce/lib/jersey-server-1.9.jar:/opt/hadoop                                                                                /share/hadoop/mapreduce/lib/asm-3.2.jar:/opt/hadoop/share/hadoop/mapreduce/lib/l                                                                                og4j-1.2.17.jar:/opt/hadoop/share/hadoop/mapreduce/lib/netty-3.6.2.Final.jar:/op                                                                                t/hadoop/share/hadoop/mapreduce/lib/leveldbjni-all-1.8.jar:/opt/hadoop/share/had                                                                                oop/mapreduce/lib/guice-3.0.jar:/opt/hadoop/share/hadoop/mapreduce/lib/javax.inj                                                                                ect-1.jar:/opt/hadoop/share/hadoop/mapreduce/lib/aopalliance-1.0.jar:/opt/hadoop                                                                                /share/hadoop/mapreduce/lib/jersey-guice-1.9.jar:/opt/hadoop/share/hadoop/mapred                                                                                uce/lib/guice-servlet-3.0.jar:/opt/hadoop/share/hadoop/mapreduce/lib/junit-4.11.                                                                                jar:/opt/hadoop/share/hadoop/mapreduce/lib/hamcrest-core-1.3.jar:/opt/hadoop/sha                                                                                re/hadoop/mapreduce/hadoop-mapreduce-client-core-2.9.1.jar:/opt/hadoop/share/had                                                                                oop/mapreduce/hadoop-mapreduce-client-common-2.9.1.jar:/opt/hadoop/share/hadoop/                                                                                mapreduce/hadoop-mapreduce-client-shuffle-2.9.1.jar:/opt/hadoop/share/hadoop/map                                                                                reduce/hadoop-mapreduce-client-app-2.9.1.jar:/opt/hadoop/share/hadoop/mapreduce/                                                                                hadoop-mapreduce-client-hs-2.9.1.jar:/opt/hadoop/share/hadoop/mapreduce/hadoop-m                                                                                apreduce-client-jobclient-2.9.1.jar:/opt/hadoop/share/hadoop/mapreduce/hadoop-ma                                                                                preduce-client-hs-plugins-2.9.1.jar:/opt/hadoop/share/hadoop/mapreduce/hadoop-ma                                                                                preduce-examples-2.9.1.jar:/opt/hadoop/share/hadoop/mapreduce/hadoop-mapreduce-c                                                                                lient-jobclient-2.9.1-tests.jar:/opt/hadoop/contrib/capacity-scheduler/*.jar
STARTUP_MSG:   build = https://github.com/apache/hadoop.git -r e30710aea4e6e55e6                                                                                9372929106cf119af06fd0e; compiled by 'root' on 2018-04-16T09:33Z
STARTUP_MSG:   java = 1.8.0_191
************************************************************/
18/10/29 03:51:16 INFO namenode.NameNode: registered UNIX signal handlers for [T                                                                                ERM, HUP, INT]
18/10/29 03:51:16 INFO namenode.NameNode: createNameNode [-format]
Formatting using clusterid: CID-594323a6-b47f-4121-9f5d-9cf4498d043e
18/10/29 03:51:23 INFO namenode.FSEditLog: Edit logging is async:true
18/10/29 03:51:24 INFO namenode.FSNamesystem: KeyProvider: null
18/10/29 03:51:24 INFO namenode.FSNamesystem: fsLock is fair: true
18/10/29 03:51:24 INFO namenode.FSNamesystem: Detailed lock hold time metrics en                                                                                abled: false
18/10/29 03:51:41 INFO namenode.FSImageFormatProtobuf: Saving image file /opt/vo                                                                                lume/namenode/current/fsimage.ckpt_0000000000000000000 using no compression
18/10/29 03:51:42 INFO namenode.FSImageFormatProtobuf: Image file /opt/volume/na                                                                                menode/current/fsimage.ckpt_0000000000000000000 of size 323 bytes saved in 0 sec                                                                                onds .
18/10/29 03:51:42 INFO namenode.NNStorageRetentionManager: Going to retain 1 ima                                                                                ges with txid >= 0
18/10/29 03:51:42 INFO namenode.NameNode: SHUTDOWN_MSG:
/************************************************************
SHUTDOWN_MSG: Shutting down NameNode at master.bagol69.com/10.10.10.1
************************************************************/
[hadoop@master ~]$

Anda  dapat  menjalankan  services  Hadoop  Cluster.  Menggunakan  command  di  bawah  ini.

[hadoop@master ~]$ start-dfs.sh
Starting namenodes on [master.bagol69.com]
master.bagol69.com: namenode running as process 2532. Stop it first.
master.bagol69.com: datanode running as process 1820. Stop it first.
Starting secondary namenodes [0.0.0.0]
0.0.0.0: secondarynamenode running as process 2799. Stop it first.
[hadoop@master ~]$


[hadoop@master ~]$ start-yarn.sh
starting yarn daemons
resourcemanager running as process 2961. Stop it first.
master.bagol69.com: nodemanager running as process 3071. Stop it first.
[hadoop@master ~]$

Anda  dapat  melihat  daftar  semua   socket  terbuka  untuk   Apache  Hadoop  di  sistem  anda menggunakan  command  di  bawah ini.
[hadoop@master ~]$ ss -tuln
Netid  State      Recv-Q Send-Q Local Address:Port               Peer Address:Port
udp    UNCONN     0      0      127.0.0.1:323                   *:*            
udp    UNCONN     0      0       ::1:323                  :::*
tcp    LISTEN     0      100    127.0.0.1:25                    *:*            
tcp    LISTEN     0      128       *:50010                 *:*
tcp    LISTEN     0      128       *:50075                 *:*
tcp    LISTEN     0      128       *:50020                 *:*
tcp    LISTEN     0      128    10.10.10.1:9000                  *:*           
tcp    LISTEN     0      128       *:50090                 *:*
tcp    LISTEN     0      128    127.0.0.1:45739                 *:*            
tcp    LISTEN     0      128       *:50070                 *:*
tcp    LISTEN     0      128       *:22                    *:*
tcp    LISTEN     0      128      :::8088                 :::*
tcp    LISTEN     0      100     ::1:25                   :::*
tcp    LISTEN     0      128      :::13562                :::*
tcp    LISTEN     0      128      :::8030                 :::*
tcp    LISTEN     0      128      :::8031                 :::*
tcp    LISTEN     0      128      :::8032                 :::*
tcp    LISTEN     0      128      :::8033                 :::*
tcp    LISTEN     0      128      :::8040                 :::*
tcp    LISTEN     0      128      :::8042                 :::*
tcp    LISTEN     0      128      :::40205                :::*
tcp    LISTEN     0      128      :::22                   :::*
[hadoop@master ~]$

Pengetesan  Apache  Hadoop

Untuk   memastikan    Apache  Hadoop  sudah  berjalan  dengan  normal.  Anda  dapat  melalkukan  dengan  cara   buat   directory  untuk     data  storage  client  anda.   Menggunakan  command di bawah ini. 

[hadoop@master ~]$ hdfs dfs -mkdir /bagol69_com

Selanjutnya,   anda  dapat   melakukan  copy  file  dari  mesin  linux  anda  ke    storage   HDFS.  Menggunakan  command  di  bawah  ini.

[hadoop@master ~]$ ls
bin  include  libexec      logs        NOTICE.txt  sbin
etc  lib      LICENSE.txt  my_storage  README.txt  share
[hadoop@master ~]$ hdfs dfs -put README.txt /bagol69.com

Anda  dapat   melihat    file  yang  sudah  anda   copy  ke  storage   HDFS. Menggunakan  command  di  bawah  ini.

 [hadoop@master ~]$ hdfs dfs -cat /bagol69_com/README.txt
For the latest information about Hadoop, please visit our website at:

   http://hadoop.apache.org/core/

and our wiki, at:

   http://wiki.apache.org/hadoop/

This distribution includes cryptographic software.  The country in
which you currently reside may have restrictions on the import,
possession, use, and/or re-export to another country, of
encryption software.  BEFORE using any encryption software, please
check your country's laws, regulations and policies concerning the
import, possession, or use, and re-export of encryption software, to
see if this is permitted.  See for more
information.

The U.S. Government Department of Commerce, Bureau of Industry and
Security (BIS), has classified this software as Export Commodity
Control Number (ECCN) 5D002.C.1, which includes information security
software using or performing cryptographic functions with asymmetric
algorithms.  The form and manner of this Apache Software Foundation
distribution makes it eligible for export under the License Exception
ENC Technology Software Unrestricted (TSU) exception (see the BIS
Export Administration Regulations, Section 740.13) for both object
code and source code.

The following provides more details on the included cryptographic
software:
  Hadoop Core uses the SSL libraries from the Jetty project written
by mortbay.org.
[hadoop@master ~]$

[hadoop@master ~]$ hdfs dfs -ls /bagol69_com/README.txt
-rw-r--r--   3 hadoop supergroup       1366 2018-10-29 03:57 /bagol69_com/README.txt
[hadoop@master ~]$

Untuk  mengambil  data  dari  HDFS  ke  mesin  linux  anda,  dapat  menggunakan  command  di  bawah  ini. 

[hadoop@master ~]$ hdfs dfs -get /bagol69_com/ ./
[hadoop@master ~]$ ls
bagol69_com  etc      lib      LICENSE.txt  my_storage  README.txt  share
bin          include  libexec  logs         NOTICE.txt  sbin
[hadoop@master ~]$

Layanan  Hadoop  WEB GUI

Untuk  mengakses  layanan  Hadoop  dari  Browser  anda, kunjungi  link  ini
atau 
Pastikan  port  50070  sudah  terbuka  di  system  firewall anda.
Hadoop  menunjukan   layanan  dari  node.
Cara  Menginstall dan Konfigurasi  Apache  Hadoop  on   single  node  di  Centos 7


Untuk  melihat  directory  yang sudah  di  buat  sebelumnya.  ( Browse Directory ).


Cara  Menginstall dan Konfigurasi  Apache  Hadoop  on   single  node  di  Centos 7
Untuk  melihat  Informasi   Cluster  dan  Apps  (  Resource  Manager  )

Cara  Menginstall dan Konfigurasi  Apache  Hadoop  on   single  node  di  Centos 7
Untuk  melihat  Informasi  NodeManager


Cara  Menginstall dan Konfigurasi  Apache  Hadoop  on   single  node  di  Centos 7




Yang  perlu Anda  lakukan  adalah  menyalakan aplikasi   yang  kompatibel dengan  Hadoop dan Anda  siap   untuk  menggunakan  Hadoop  !!!!!

Untuk  melihat  Informasi  tambahan, silahkan  lihat  halaman  Apache Hadoop  Dokumentasi   dan  Hadoop   Wiki. 








Read More