Author: dan
Source: Planet OpenStack
In this post we’re going to go over the installation of swift as well as a couple of architectural considerations and key topics.
To work through this how-to you are doing to need at least 5 Machines, 2 for Swift Proxies and 3 for Swift Object servers, these machines can be AWS instances, VM’s or physical machines.
In a real world production environment these machines would be configured with 2 independent NICS but in this case we’re going to work with one network interface, it should be obvious how to adapt to the dual network configuration.
This guide assumes you have the required number of machines and that they are installed with Ubuntu 14.04 and have access to a working an up to date APT repo.
It’s also assumed that these machines have an OS partition of at least 15GB and an additional unformatted partition for swift storage.
Swift Components
There are 2 types of swift servers, Proxy servers and Storage servers; Proxy servers handle the communications to and from clients as well as maintaining ‘rings’ , these rings are little more than SQLITE databases which contain a mapping between the Swift entity and it’s physical location. The ring does this mapping using zones, partitions, devices and replicas with 3 replicas per partition. The Proxy also deals with failures during the GET or PUT process, when a failure occurs the proxy calls the ring to determine which server should be used in place of the failed on.
Zones are exactly what you would think, a collection of servers or disks in a single failure domain, because data is never replicated to another disk or server in a zone you use this model to create rack / chassis aware designs.
Storage policies are used to create groups of devices with differing features and services, for example, you could have multiple storage policies, one with a replication factor of 3 and another with a replication factor of 2, or a storage policy of disks magnetic spindles and another with disks using SSD’s.
Containers are essentially a space assigned to store a particular set of objects, containers could be segregated by file type, team or customer. Each container is mapped to individual storage policies – applications and clients are unaware of the storage policy as they are interacting with the container. The container maintains this storage policy throughout it’s life within swift, the storage policy in use by a container cannot be changed so give careful consideration to the long term logistics of this. The Container server process maintains a mapping of which objects are in each container without knowing physically where they are located.
Storage servers perform a wide variety of functions, the object server is responsible for reading and writing objects as blobs to the file system, the reaper process which cleans up files that have been deleted, the replication worker which takes care of replicating objects between servers and the auditor process validates file integrity of the objects, replacing corrupt objects with a freshly replicated copy. Replication is push based utilizing RSYNC for individual objects, a hash of each object is used to quickly compare objects in different locations.
The object server process which runs on storage servers, it writes the object as a binary BLOB on the disk using a path based on a hash of the object’s name as well as version number and timestamp, files are treated as versions in all cases and in fact a deleted file is treated as such. The metadata for these blobs is stored in the extensible attributes on the file system, this is an important point: ensure the filesystem you are placing swift objects on supports xattrs and they are enabled, xattrs are disabled by default on various file systems (including ext3/4 and xfs in certain configurations) XFS is the recommended filesystem but you there’s nothing stopping you using EXT4 if that’s what you are comfortable with.
When a file is deleted the reaper process removes the file and creates a zero byte tombstone file of the same name, this ensures that the replication process doesn’t replace deleted files as well as instructing it to delete the replicas of this file on other servers.
Installation
First up, let’s install all the required software on all 5 nodes servers and create the config directory:
apt-get -y install python-software-properties swift python-swiftclient openssh-server mkdir /etc/swift
Now on the proxy server, let’s create a barebones config with a randomly generated key:
cat >/etc/swift/swift.conf <<EOF [swift-hash] # random unique strings that can never change (DO NOT LOSE) swift_hash_path_prefix = `od -t x8 -N 8 -A n </dev/random` swift_hash_path_suffix = `od -t x8 -N 8 -A n </dev/random` EOF
Now scp that config file to all other nodes.
Next up, we need to set a couple of variables on both proxies and the storage nodes, these variables will be read by installation scripts, given that our systems only have one network adapter, set these variables to the IP address of your eth0 device, if you have more than one adapter you know what to do!
export STORAGE_LOCAL_NET=192.168.1.20 export PROXY_LOCAL_NET_IP=192.168.1.20
Next up we need to create our run time directory on all 5 nodes:
mkdir -p /var/run/swift chown swift:swift /var/run/swift
Given that this directory is cleared upon reboot it’s necessary to recreate at system startup, right about the ‘exit 0′ line in /etc/rc.local add the previous 2 commands.
Next up, we’ll create the directory where our objects will live on the storage servers:
mkdir /srv/node
Next we’ll create a directory for our device, in this case our device is VDA1
mkdir /srv/node/vda1
Using your favorite partitioning tool, create a single partition on the disk and format it using EXT4.
Now add the drive to /etc/fstab to ensure it’s mounted at boot time, being careful to ensure xattr’s are enable:
/dev/vda1 /srv/node/vda1 ext4 auto,users,user_xattr,noatime,nodiratime,nobarrier 0 2
It’s important to ensure that access time is not stamped onto the files this will create un-neccesary blocking on the disk as the access time is written each time a file is read.
Now mount /srv/node/vda1, ensure that the permissions for /srv/node are root:root – this will ensure that if the disks aren’t mounted that objects won’t be accidentally created in /srv/node.
Let’s make the actual drive we are using for swift objects writable by the swift user:
chown swift:swift /srv/node/vda1/
Let’s also create our cache directory and ensure it’s writable by the swift user:
mkdir -p /var/cache/swift && chown swift:swift /var/cache/swift
Now setup RSYNC on all 3 storage nodes:
cat >/etc/rsyncd.conf <<EOF uid = swift gid = swift log file = /var/log/rsyncd.log pid file = /var/run/rsyncd.pid address = $STORAGE_LOCAL_NET_IP
[account] max connections = 2 path = /srv/node/ read only = false lock file = /var/lock/account.lock [container] max connections = 2 path = /srv/node/ read only = false lock file = /var/lock/container.lock [object] max connections = 2 path = /srv/node/ read only = false lock file = /var/lock/object.lock EOF
Edit /etc/default/rsync to enable the rsync service and restart it – if the storage nodes reside on a routable network it’s important to ensure that the RSYNC port is firewalled from non-swift storage servers, this is due to the fact that RSYNC doesn’t require authentication.
Now let’s install the proxy server software on both proxies:
apt-get -y install swift-proxy memcached
The Swift proxy defaults to HTTPS if an SSL certificate is referenced in the proxy configuration, if you have an SSL certificate you must ensure it’s in X509 format and that there is no passphrase on the key. OF course, you can always create your own self signed cert:
root@swift-proxy01:/etc/swift# openssl req -new -x509 -nodes -out cert.crt -keyout cert.key Generating a 2048 bit RSA private key ..........................+++ ..........................................+++ writing new private key to 'cert.key' ----- You are about to be asked to enter information that will be incorporated into your certificate request. What you are about to enter is what is called a Distinguished Name or a DN. There are quite a few fields but you can leave some blank For some fields there will be a default value, If you enter '.', the field will be left blank. ----- Country Name (2 letter code) [AU]:US State or Province Name (full name) [Some-State]:WA Locality Name (eg, city) []:SEATTLE Organization Name (eg, company) [Internet Widgits Pty Ltd]:DAN RULES Organizational Unit Name (eg, section) []:CONVERGED CLOUD Common Name (e.g. server FQDN or YOUR name) []:swift.localdomain Email Address []:dan@thekingshotts.com
SCP this certificate to /etc/swift on the second proxy node
Next up we need to modify memcache.conf to listen on the appropriate interface, as we have 2 proxies, we’ll share memcache instances which is why we want them listening on the eth0 device rather than the loopback:
perl -pi -e "s/-l 127.0.0.1/-l $PROXY_LOCAL_NET_IP/" /etc/memcached.conf
Now restart memcache.
Now, we need to setup one more variable, PROXY_LOCAL_NET_IP2 – this variable should reference the IP address of the OTHER swift proxy, so in our case the first proxy will have this variable populated with the IP address of the second proxy and vice versa.
Let’s put all this together and create our swift proxy configuration:
cat >/etc/swift/proxy-server.conf <<EOF [DEFAULT] cert_file = /etc/swift/cert.crt key_file = /etc/swift/cert.key bind_port = 8080 workers = 8 user = swift [pipeline:main] pipeline = healthcheck proxy-logging cache tempauth proxy-logging proxy-server [app:proxy-server] use = egg:swift#proxy allow_account_management = true account_autocreate = true [filter:proxy-logging] use = egg:swift#proxy_logging [filter:tempauth] use = egg:swift#tempauth user_system_root = testpass .admin https://$PROXY_LOCAL_NET_IP:8080/v1/AUTH_system [filter:healthcheck] use = egg:swift#healthcheck [filter:cache] use = egg:swift#memcache memcache_servers = $PROXY_LOCAL_NET_IP:11211,$PROXY_LOCAL_NET_IP2:11211 EOF
Next up, we need to create the rings, these rings need to be sized according to the minimum and maximum number of drives that will be used by swift, an easy way to work this out is using the ring size calculator:
http://rackerlabs.github.io/swift-ppc/
In this case, we are going to pin this at a minimum of 3 disks and a maximum of 5 with 3 replicas, in which case we need to use the following commands to create the ring on the first proxy server only:
cd /etc/swift swift-ring-builder account.builder create 8 3 1 swift-ring-builder container.builder create 8 3 1 swift-ring-builder object.builder create 8 3 1
Next up we need to setup the ring information for each of the storage servers, You’ll need to repeat this step for each storage server and it’s associated drives, ensuring you change the IP to match the storage node’s IP address and disk reference.
export ZONE=1 export STORAGE_LOCAL_NET_IP=192.168.2.0 export WEIGHT=100 export DEVICE=vda1 swift-ring-builder account.builder add z$ZONE-$STORAGE_LOCAL_NET_IP:6002/$DEVICE $WEIGHT swift-ring-builder container.builder add z$ZONE-$STORAGE_LOCAL_NET_IP:6001/$DEVICE $WEIGHT swift-ring-builder object.builder add z$ZONE-$STORAGE_LOCAL_NET_IP:6000/$DEVICE $WEIGHT
Now validate the ring contents, you should see something similar to this:
root@swift-proxy01:/etc/swift# swift-ring-builder account.builder account.builder, build version 3 256 partitions, 3.000000 replicas, 1 regions, 1 zones, 3 devices, 100.00 balance The minimum number of hours before a partition can be reassigned is 1 Devices: id region zone ip address port replication ip replication port name weight partitions balance meta 0 1 1 192.168.1.252 6002 192.168.1.252 6002 vda1 100.00 0 -100.00 1 1 1 192.168.1.254 6002 192.168.1.254 6002 vda1 100.00 0 -100.00 2 1 1 192.168.2.0 6002 192.168.2.0 6002 vda1 100.00 0 -100.00 root@swift-proxy01:/etc/swift# swift-ring-builder container.builder container.builder, build version 3 256 partitions, 3.000000 replicas, 1 regions, 1 zones, 3 devices, 100.00 balance The minimum number of hours before a partition can be reassigned is 1 Devices: id region zone ip address port replication ip replication port name weight partitions balance meta 0 1 1 192.168.1.252 6001 192.168.1.252 6001 vda1 100.00 0 -100.00 1 1 1 192.168.1.254 6001 192.168.1.254 6001 vda1 100.00 0 -100.00 2 1 1 192.168.2.0 6001 192.168.2.0 6001 vda1 100.00 0 -100.00 root@swift-proxy01:/etc/swift# swift-ring-builder object.builder object.builder, build version 3 256 partitions, 3.000000 replicas, 1 regions, 1 zones, 3 devices, 100.00 balance The minimum number of hours before a partition can be reassigned is 1 Devices: id region zone ip address port replication ip replication port name weight partitions balance meta 0 1 1 192.168.1.252 6000 192.168.1.252 6000 vda1 100.00 0 -100.00 1 1 1 192.168.1.254 6000 192.168.1.254 6000 vda1 100.00 0 -100.00 2 1 1 192.168.2.0 6000 192.168.2.0 6000 vda1 100.00 0 -100.00 root@swift-proxy01:/etc/swift#
Now re-balance the rings:
swift-ring-builder account.builder rebalance swift-ring-builder container.builder rebalance swift-ring-builder object.builder rebalance
Now scp the *.ring.gz files from the first proxy server to the other proxy server and the storage nodes, placing them in /etc/swift
Now ensure /etc/swift and the files within it are owned by the swift user and group on all nodes and proxies.
Now start the swift proxy on both proxy nodes:
root@swift-proxy02:/etc/swift# swift-init proxy start Starting proxy-server...(/etc/swift/proxy-server.conf) WARNING: SSL should only be enabled for testing purposes. Use external SSL termination for a production deployment.
With regards to the message about using external SSL termination, be careful, some tier-1 PCI companies are required to encrypt their traffic all the way to the server depending on the data being transmitted.
Now let’s move onto the storage nodes, you’ll need to perform these operations on ALL storage nodes:
apt-get -y install swift-account swift-container swift-object
On each one of the storage nodes we need to setup the account config next:
cat >/etc/swift/account-server.conf <<EOF [DEFAULT] bind_ip = `ifconfig bond0 | grep inet addr | cut -d":" -f2 | cut -d" " -f1` workers = 2 [pipeline:main] pipeline = account-server [app:account-server] use = egg:swift#account [account-replicator] [account-auditor] [account-reaper] EOF
The container config:
cat >/etc/swift/container-server.conf <<EOF [DEFAULT] bind_ip = `ifconfig bond0 | grep inet addr | cut -d":" -f2 | cut -d" " -f1` workers = 2 [pipeline:main] pipeline = container-server [app:container-server] use = egg:swift#container [container-replicator] [container-updater] [container-auditor] [container-sync] EOF
The object config:
cat >/etc/swift/object-server.conf <<EOF [DEFAULT] bind_ip = `ifconfig bond0 | grep inet addr | cut -d":" -f2 | cut -d" " -f1` workers = 2 [pipeline:main] pipeline = object-server [app:object-server] use = egg:swift#object [object-replicator] [object-updater] [object-auditor] EOF
And now start the storage services:
swift-init all start
Now let’s authenticate with the server using the username root and the password testpass, this was set in the proxy-server.conf earlier on and can be changed to an arbitrary value
root@swift-proxy01:/home/dan# curl -k -v -H 'X-Storage-User: system:root' -H 'X-Storage-Pass: testpass'https://$PROXY_LOCAL_NET_IP:8080/auth/v1.0
* Hostname was NOT found in DNS cache * Trying 192.168.1.250... * Connected to 192.168.1.250 (192.168.1.250) port 8080 (#0) * successfully set certificate verify locations: * CAfile: none CApath: /etc/ssl/certs * SSLv3, TLS handshake, Client hello (1): * SSLv3, TLS handshake, Server hello (2): * SSLv3, TLS handshake, CERT (11): * SSLv3, TLS handshake, Server finished (14): * SSLv3, TLS handshake, Client key exchange (16): * SSLv3, TLS change cipher, Client hello (1): * SSLv3, TLS handshake, Finished (20): * SSLv3, TLS change cipher, Client hello (1): * SSLv3, TLS handshake, Finished (20): * SSL connection using AES256-GCM-SHA384 * Server certificate: * subject: C=US; ST=WA; L=SEATTLE; O=DAN RULES; OU=CONVERGED CLOUR; CN=swift.localdomain; emailAddress=dan@thekingshotts.com * start date: 2014-08-07 20:42:20 GMT * expire date: 2014-09-06 20:42:20 GMT * issuer: C=US; ST=WA; L=SEATTLE; O=DAN RULES; OU=CONVERGED CLOUR; CN=swift.localdomain; emailAddress=dan@thekingshotts.com * SSL certificate verify result: self signed certificate (18), continuing anyway. > GET /auth/v1.0 HTTP/1.1 > User-Agent: curl/7.35.0 > Host: 192.168.1.250:8080 > Accept: */* > X-Storage-User: system:root > X-Storage-Pass: testpass > < HTTP/1.1 200 OK < X-Storage-Url: https://192.168.1.250:8080/v1/AUTH_system < X-Auth-Token: AUTH_tk27ae6a83ae4d48c585315dd7d43dbaa0 < Content-Type: text/html; charset=UTF-8 < X-Storage-Token: AUTH_tk27ae6a83ae4d48c585315dd7d43dbaa0 < Content-Length: 0 < X-Trans-Id: tx0151940e8c3a495e9895b-0053e40096 < Date: Thu, 07 Aug 2014 22:41:26 GMT < * Connection #0 to host 192.168.1.250 left intact
Let’s check we can connect with the account:
curl -k -v -H 'X-Auth-Token: <token-from-x-auth-token-above>' <url-from-x-storage-url-above>
So in our case:
root@swift-proxy01:/etc/swift# curl -k -v -H 'X-Auth-Token: AUTH_tk27ae6a83ae4d48c585315dd7d43dbaa0'https://192.168.1.250:8080/v1/AUTH_system * Hostname was NOT found in DNS cache * Trying 192.168.1.250... * Connected to 192.168.1.250 (192.168.1.250) port 8080 (#0) * successfully set certificate verify locations: * CAfile: none CApath: /etc/ssl/certs * SSLv3, TLS handshake, Client hello (1): * SSLv3, TLS handshake, Server hello (2): * SSLv3, TLS handshake, CERT (11): * SSLv3, TLS handshake, Server finished (14): * SSLv3, TLS handshake, Client key exchange (16): * SSLv3, TLS change cipher, Client hello (1): * SSLv3, TLS handshake, Finished (20): * SSLv3, TLS change cipher, Client hello (1): * SSLv3, TLS handshake, Finished (20): * SSL connection using AES256-GCM-SHA384 * Server certificate: * subject: C=US; ST=WA; L=SEATTLE; O=DAN RULES; OU=CONVERGED CLOUD; CN=swift.localdomain; emailAddress=dan@thekingshotts.com * start date: 2014-08-07 20:42:20 GMT * expire date: 2014-09-06 20:42:20 GMT * issuer: C=US; ST=WA; L=SEATTLE; O=DAN RULES; OU=CONVERGED CLOUD; CN=swift.localdomain; emailAddress=dan@thekingshotts.com * SSL certificate verify result: self signed certificate (18), continuing anyway. > GET /v1/AUTH_system HTTP/1.1 > User-Agent: curl/7.35.0 > Host: 192.168.1.250:8080 > Accept: */* > X-Auth-Token: AUTH_tk27ae6a83ae4d48c585315dd7d43dbaa0 > < HTTP/1.1 204 No Content < Content-Type: text/plain; charset=utf-8 < X-Account-Object-Count: 0 < X-Timestamp: 1407450167.99500 < X-Account-Bytes-Used: 0 < X-Account-Container-Count: 0 < X-Put-Timestamp: 1407450167.99500 < Content-Length: 0 < X-Trans-Id: tx6eb38a1425ec4ef287083-0053e3fc37 < Date: Thu, 07 Aug 2014 22:22:47 GMT < * Connection #0 to host 192.168.1.250 left intact root@swift-proxy01:/etc/swift#
Repeat the account verification steps on the other proxy.
Now check swift is working, you’ll need to do this on both nodes:
root@swift-proxy01:/etc/swift# swift --insecure -A https://$PROXY_LOCAL_NET_IP:8080/auth/v1.0 -U system:root -K testpass stat
Account: AUTH_system Containers: 0 Objects: 0 Bytes: 0 Content-Type: text/plain; charset=utf-8 X-Timestamp: 1407450740.75702 X-Trans-Id: tx8e3f25f8a2ba4aff99701-0053e3fe74 X-Put-Timestamp: 1407450740.75702
The –insecure option should only be used if you have a self signed certificate or the CA has not been added to the system.
Don’t worry that Containers / Objects / Bytes are 0, we haven’t added anything yet 
Let’s upload and download a big-ish file to test it:
root@swift-proxy01:/home/dan# du -sh bigfile.tgz 116M bigfile.tgz root@swift-proxy01:/home/dan# swift --insecure -A https://$PROXY_LOCAL_NET_IP:8080/auth/v1.0 -U system:root -K testpass upload myfiles bigfile.tgz bigfile.tgz root@swift-proxy01:/home/dan# swift --insecure -A https://$PROXY_LOCAL_NET_IP:8080/auth/v1.0 -U system:root -K testpass download myfiles bigfile.tgz [auth 0.017s, headers 0.053s, total 5.297s, 23.035 MB/s]
Let’s check our status again on both proxies:
root@swift-proxy01:/etc/swift# swift --insecure -A https://$PROXY_LOCAL_NET_IP:8080/auth/v1.0 -U system:root -K testpass stat Account: AUTH_system Containers: 1 Objects: 1 Bytes: 121611386 Accept-Ranges: bytes X-Timestamp: 1407450996.36535 X-Trans-Id: tx876dc87c5c55492e95a89-0053e40116 Content-Type: text/plain; charset=utf-8
root@swift-proxy02:/etc/swift# swift --insecure -A https://$PROXY_LOCAL_NET_IP:8080/auth/v1.0 -U system:root -K testpass stat Account: AUTH_system Containers: 1 Objects: 1 Bytes: 121611386 Accept-Ranges: bytes X-Timestamp: 1407450996.36535 X-Trans-Id: tx55dbb00f2faa42679832b-0053e400f4 Content-Type: text/plain; charset=utf-8
Next up, let’s upload our builder files to the cluster, builder files are important because they are used to build future rings, the most resilient way to backup these files is into swift itself:
root@swift-proxy01:/etc/swift# swift --insecure -A https://$PROXY_LOCAL_NET_IP:8080/auth/v1.0 -U system:root -K testpass upload builders /etc/swift/*.builder etc/swift/account.builder etc/swift/object.builder etc/swift/container.builder
Verify the container has been created and objects uploaded:
root@swift-proxy01:/etc/swift# swift --insecure -A https://$PROXY_LOCAL_NET_IP:8080/auth/v1.0 -U system:root -K testpass list builders myfiles
root@swift-proxy01:/etc/swift# swift --insecure -A https://$PROXY_LOCAL_NET_IP:8080/auth/v1.0 -U system:root -K testpass list builders etc/swift/account.builder etc/swift/container.builder etc/swift/object.builder
Success, you have now built a swift cluster!
References:
http://docs.openstack.org/developer/swift/howto_installmultinode.html
http://docs.openstack.org/developer/swift/overview_ring.html
http://docs.openstack.org/developer/swift/overview_architecture.html
Powered by WPeMatico