Posted on   |   In   |     |   Views

Construct the platform for Big Data project

Build HardDisk RAID0

This paragraph describes the HDFS configure progress on Azure Cloud Server. You need have a azure account to login in and purchase the platform you will perform on. Here is my platform construction details to share with you.

  • Unbuntu 14.0 LTS

lvm2

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
$ sudo apt-get update
$ sudo apt-get install lvm2
// find the disk, using dmesg | grep SCSI
#  grep SCSI /var/log/dmesg
$ sudo fdisk /dev/sdc  -> n,p,1, enter, enter, p, w
$ sudo fdisk /dev/sdd  -> n,p,1, enter, enter, p, w
$ sudo fdisk /dev/sde  -> n,p,1, enter, enter, p, w
$ sudo fdisk /dev/sdf  -> n,p,1, enter, enter, p, w
$ sudo pvcreate /dev/sdc1
$ sudo pvcreate /dev/sdd1
$ sudo pvcreate /dev/sde1
$ sudo pvcreate /dev/sdf1
//  sudo pvscan
// merge all the drives together
$ sudo vgcreate datadrive /dev/sdc1 /dev/sdd1 /dev/sde1 /dev/sdf1
// sudo vgdisplay
$ sudo lvcreate -l 100%FREE datadrive -n example-log -I 64 -i 4
// sudo lvdisplay
//  sudo lvcreate -i <number of physical volumes to stripe> -I (大写i)<size of stripe in killobytes> -L <size in megabytes>M <name of virtual group>
$ sudo mkfs -t ext4 /dev/datadrive/example-log
$ sudo mkdir /datasource
$ sudo mount /dev/datadrive/example-log /datasource
  • Automatic Mount configure (fstab)
1
2
3
4
5
6
7
8
sudo blkid
sudo vim /etc/fstab
/dev/mapper/datadrive-example--datalog: UUID="b2e83db3-41b9-4f78-8b1f-f196bb9b9b6f" TYPE="ext4"

UUID=b2e83db3-41b9-4f78-8b1f-f196bb9b9b6f /datadrive ext4 defaults 0 0
// An example from mine (bolded GUID is from my blkid, be sure to change it!): UUID=63ab0827-4698-427a-818a-279b18886757 /mnt/datadrive ext3 defaults 0 0
// $ sudo chroot /target
// Test disk write speed
1
$ time sh -c "sudo dd if=/dev/zero of=/datasource/temp bs=4k count=2000000 && sync"

HDFS

  • Login in http://manage.windowsazure.com, build the server on Azure
    • Set the host: example-hadoop.cloudapp.net
    • Create VM (Ubuntu 14 LTS), with 8 cores and 14GB RAM
      • example-data01, example-data02, example-data03, example-data04
    • Hadoop FAQ
  • ssh-keygen
1
2
3
4
5
6
7
admin@example-data01 $ cd .ssh
$ ssh-keygen -t rsa  -> enter, enter, enter
$ cat id_rsa.pub >> authorized_keys
// test the key: $ ssh localhost  -> successfully
$ scp authorized_keys admin@example-data02:~/.ssh/
$ scp authorized_keys admin@example-data03:~/.ssh/
// test $ ssh example-data02

Download the Hadoop Binary on each node

1
2
3
4
$ Install java on ubuntu
$ wget http://mirrors.cnnic.cn/apache/hadoop/common/stable/hadoop-2.7.1.tar.gz
$ tar xvf hadoop-2.7.1.tar.gz
$ mv hadoop-2.7.1 hadoop

Posted on   |   In   |     |   Views

Summary the tools used in project, essential in big data direction

Tools

  • Database: MySQL, NOSQL(redis), Hive, HBase
  • Platform: Windows8, Ubuntu, Debian
  • VM: VMware, Cywin, VirtualBox
  • Language: JAVA, R, Python, go, Ruby, Matlab
  • Shell: bash, perl
  • Bigdata Sys: SPARK, Hadoop, XAMPP
  • Algorithm: ML & NL & DM (DL)
  • Transfer: winSCP
  • Notes : WizNote , evernote
  • Office : OpenOffice, WPS
  • PHP, html5, Node.js
  • Build evn: Maven, sbt
  • MLlib: liblinear-java, libsvm
  • SSH manager : putty manager, putty
  • Blog tools: WordPress, hexo
  • Redis java

Posted on   |   In   |     |   Views

The spyder tool of Python GUI works on Max OS

Install Python

Install Python 3.4.1 on Ubuntu

1
2
3
4
5
6
7
8
sudo apt-get install libssl-dev openssl
wget https://www.python.org/ftp/python/3.4.1/Python-3.4.1.tgz
tar -xvf Python-3.4.1.tgz
cd Python-3.4.1/
./configure
make
sudo make install
> then run ./python in terminal

Install Python 3.4.1 on Mac

Mac has installed the python 2.7, if you want to use the latest version, go to the Python website and download.

Install the Python GUI on Mac

Step 1, Intall the Python

Step 2, Install the Anaconda

  • Command-Line Install
1
bash Anaconda2-2.4.1-MacOSX-x86_64.sh

Step 3, Install MacPorts

Make sure that you have installed the XCode on mac, which is the encession to install “spyder”
Also for the MacPorts install reference

1
2
curl -O https://distfiles.macports.org/MacPorts/MacPorts-2.3.3.tar.bz2
tar xf MacPorts-2.3.3.tar.bz2

Then, edit the /etc/profile file, add the following comand

1
export PATH=/opt/local/bin:/opt/local/sbin:$PATH

For the first time to run:

1
sudo port -v selfupdate

And if you update failed, you can re-run it with debug output, likes:

1
sudo port -d selfupdate

Step 4, install spyder

1
2
3
4
5
$ sudo port install python27 // if python27 dose not installed, then install it before activate
$ sudo port select --set python python27  // set the default python
$ sudo port select --list python // available versions
$ sudo port -f activate python27
$ sudo port install py-spyder

After long time waiting. Then, begin our python journey, run the spyder in the terminal

1
$ spyder

Posted on   |   In   |     |   Views

Beginning the blog journey of Hexo & Gituhub

Welcome to Hexo! This is your very first post. Check documentation for more info. If you get any problems when using Hexo, you can find the answer in troubleshooting or you can ask me on GitHub.

Quick Start

Create a new post

1
$ hexo new "My New Post"

More info: Writing

Run server

1
$ hexo server

More info: Server

Generate static files

1
$ hexo generate

More info: Generating

Deploy to remote sites

1
$ hexo deploy

More info: Deployment

Create a domain of yourself bolg website

Step 1

Register on Godaddy, If possible, also should to use CDN on DNSPod

Step 2

Create Personal Rep. on github, and name the Rep. is , add the README.md file to describe the Rep.

Step 3

Using ping to get github pages ip < 103.245.222.133>

Step 4

On Mac OS, the necessary thing is to download Xcode CommandLine Tools in AppStore, and then

1
$ sudo xcode-select -switch /Applications/Xcode.app/Contents/Developer

Step 5

Install jekyll (optional)

1
2
$ sudo gem update --system
$ sudo gem install jekyll

Install Hexo (use)

  • Install Xcode and Node.js
  • Install Hexo on mac
1
2
3
4
$ sudo npm install hexo-cli -g
$ mkdir blog
$ cd blog
$ hexo init & npm install

Configure the homepage by

Update

Update hexo

1
$ npm update -g hexo

Update themes

1
2
$ cd themes/next
$ git pull

Update plugins

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
$ npm update
```  

### Add plugins

``` bash
npm install hexo-generator-index --save
npm install hexo-generator-archive --save
npm install hexo-generator-category --save
npm install hexo-generator-tag --save
npm install hexo-server --save
npm install hexo-deployer-git --save
npm install hexo-deployer-heroku --save
npm install hexo-deployer-rsync --save
npm install hexo-deployer-openshift --save
npm install hexo-renderer-marked@0.2.7 --save
npm install hexo-renderer-stylus@0.3.0 --save
npm install hexo-generator-feed@1.0.3 --save
npm install hexo-generator-sitemap@1.0.1 --save