Installing Ceph

From Ambedded
Jump to: navigation, search



Ceph need al servers to have time synced, so timezone should be set and NTP daemon started

dpkg-reconfigure tzdata
apt-get install ntp

then reboot

shutdown -r now

login again and check for correct time/date status


This has to be done for all the Cy nodes that will be running Ceph

Administrative node

The administrative node will be used to deploy Ceph on the whole cluster

wget -O - |tar xzvf - -C /
apt-get update
  • The above 3 steps has to be done for all the Cy nodes that will be running Ceph

1)Install ceph-deploy on Administrative Node (The Administrative node for the deploy can be either one of the nodes of the cluster)

apt-get install ceph-deploy

2) After repeating the procedure for every node you want to deploy you can now reefer to Ceph documentation on how to create a test cluster

3) Recommend file system is xfs. To use xfs you will need to do the following

apt-get install -y xfsprogs
mkdir /data/
mkfs.xfs -f /dev/sda2
mount /dev/sda2 /data
echo "/dev/sda2 /data xfs rw 0 0" >> /etc/fstab

PS.Still if you wish to use ext4 you need to enable user extended attributes in fstab and remount

echo "/dev/sda2 /data ext4 acl,user_xttr 0 0" >> /etc/fstab

Some notes about Ceph architecture

Type of daemons and their placement on the CY7

OSDs -> Object store server 
Those are the "blocks" of the distributed storage.
They should reside on the second partion of the HDD CY7 nodes, following the partitioning example of this wiki 
MDSs -> Metadata store server
These servers handle all of the metadata of saved files. Best if they are run from the mSATA nodes of the CY7.
While it's possible to have more than one active MDS while deploying Ceph documentation warns that:
"There is experimental support for running multiple metadata servers. Do not run multiple metadata servers in production."
Monitors -> The Ceph monitor software
Those servers "orchestrate" all of the Ceph services.
They can be clustered for redundancy and can be put either OSD nodes or MDS nodes.


Since striping/replication of saved files is handled by Ceph in background, it's advisable to configure two different networks in the CY7, taking advantage of his dual switch architecture.

For instance a network of 10.0.0.X can be set up for the eth0 of the computing modules, while a network of 192.168.0.X for the eth1.

In this setup network 1 (10.0.0.X) will be used for client-to-ceph communication (e.g. reading and writing files), and therefore named as EXTERNAL while the second one (192.168.0.X) will be dedicated to intra ceph-ceph communication. We can refer of this network as INTERNAL.

You will then have access to the EXTERNAL network on LAN1 ethernet output interface on CY7 and to the INTERNAL network on LAN2 output interface of same server.

While connection more than a CY7 in a storage cluster based on Ceph it's advised to use different switches (or VLAN's) to separate traffic of the external network and the internal network. This will ensure maximum speed and security for the configured multi-server cluster.

Customized Crush map and CY7

From Ceph online documentation

When you create a configuration file and deploy Ceph with ceph-deploy, Ceph generates a default CRUSH map for your configuration.
The default CRUSH map is fine for your Ceph sandbox environment. However, when you deploy a large-scale data cluster,
you should give significant consideration to developing a custom CRUSH map, because it will help you manage your Ceph cluster, improve performance and ensure data safety.

Defining a customized Crush map ensures that if, for instance, you have a replica count of 2 for your data the two copies will reside on two CY1 microservers located in a different CY7 chassis. So, if you will ever need to take a full CY7 chassis offline, data can still be accessed.

The concept can be expanded further, like storing the first copy and the second copy of the data on two different CY7, just as in the previous example, but each one located in a different rack, increasing data availability and safety.

Personal tools