Showing posts with label openshift. Show all posts
Showing posts with label openshift. Show all posts

Sunday, 8 June 2014

Postgresql Backup To Amazon S3 On OpenShift Origin

To move forward, you need to backup. Backing up your production data is critical and with Postgresql, you can backup WAL (Write Ahead Log) archives and this post gives you steps to accomplish for backing up postgresql WALs to Amazon S3 on your OpenShift Origin using WAL-E.

WAL-E is a great tool that simplifies backup of postgresql by performing continuous archiving of PostgreSQL WAL files and base backups. Enough blabbering, you can reach out technical docs on how WAL works. I'll just mention series of commands and steps necessary for sending WAL archives to AWS S3 bucket.

On the node containing application with postgresql cartridge, run the following commands:

$ yum install python-pip lzop pv
$ rpm -Uvh
$ pip install wal-e
$ umask u=rwx,g=rx,o=
$ mkdir -p /etc/wal-e.d/env
$ echo "secret-key-content" > /etc/wal-e.d/env/AWS_SECRET_ACCESS_KEY
$ echo "access-key" > /etc/wal-e.d/env/AWS_ACCESS_KEY_ID
$ echo 's3://backup/production/pgsql' > \
$ chmod -R 765 /etc/wal-e.d/

Then, edit the postgresql configuration file so as to turn on wal archiving. You need to find the right container for your postgresql in /var/lib/openshift (Its quite trivial if you know OpenShift basics).

$ vi YOUR_OO_CONTAINER/postgresql/data/postgresql.conf wal_level = archive # hot_standby in 9.0 is also acceptable
archive_mode = on
archive_command = 'envdir /etc/wal-e.d/env wal-e wal-push %p'
archive_timeout = 60

Finally, you need to ensure that you are taking base backups periodically which can be achieved by utilizing cron cartridge. Clone the repo, add the following file and push to the application.

$ vi .openshift/cron/daily/postgres-backup

        /usr/bin/envdir /etc/wal-e.d/env /bin/wal-e backup-push ${OPENSHIFT_POSTGRESQL_DIR}data

$ git add .openshift/cron/daily/postgres-backup
$ git commit -m "Added pg cron script"
$ git push origin master

Make sure you use the OPENSHIFT_POSTGRESQL_DIR env-var or some other env-var that does not have two forward slashes adjacently since WAL-E hates it.

This should help you keep your data backed up regularly and you can enjoy beers.


Thursday, 5 June 2014

Setting Up JVM Heap Size In JBoss OpenShift Origin

Openshift is an awesome technology and have fell in love with it recently. In this post, I will talk about how we can set JVM Heap Size for your application using Jboss cartridge.

If you look into the content of the standalone.conf located at $OPENSHIFT_JBOSSEAP_DIR/bin, you can see that JVM_HEAP_RATIO is set to 0.5 if it is not already set.

if [ -z "$JVM_HEAP_RATIO" ]; then

And, later this ratio is used to calculate the max_heap so as to inject the maximum heap size in jboss java process. You can see how gear memory size is used to calculate the value of heap size. This is the very reason why the default installation allocates half of total gear memory size.

max_heap=$( echo "$max_memory_mb * $JVM_HEAP_RATIO" | bc | awk '{print int($1+0.5)}')

OpenShift keeps its number of environment variables inside /var/lib/openshift/OPENSHIFT_GEAR_UUID/.env so what I did was SSH to my OO node and run the command below (you should replace your gear's UUID):

$ echo -n 0.7 > /var/lib/openshift/52e8d31bfa7c355caf000039/.env/JVM_HEAP_RATIO

Alternatively, rhc set-env JVM_HEAP_RATIO=0.7 -a appName should also work but I have not tried it.


Friday, 11 April 2014

Patching Your OpenShift Origin Against Heartbleed vulnerability

Recently the heartbleed bug was exposed which existed in all the services that used OpenSSL 1.0.1 through 1.0.1f (inclusive) for years already. This weakness allows stealing the information protected, under normal conditions, by the SSL/TLS encryption used to secure the Internet by reading the memory of the system without need of any kind of access.

I've been administering OpenShift applications recently and this post outlines the measures I took to secure the OpenShift applications from this critical vulnerability.

In order to check if you are vulnerable or not, you can either check OpenSSL version:

# openssl version -a
OpenSSL 1.0.1e-fips 11 Feb 2013
built on: Wed Jan 8 07:20:55 UTC 2014
platform: linux-x86_64

Alternatively, you can use one of the online tools or the offline python tool to check if you are vulnerable or not.

Note that in case of OpenShift origin, you will have to update the OpenSSL package in brokers and nodes such that all the OpenShift apps are secure.

# yum install -y openssl

Once completed, verify the installation of patched version:

# openssl version -a
OpenSSL 1.0.1e-fips 11 Feb 2013
built on: Tue Apr 8 00:29:11 UTC 2014
platform: linux-x86_64

# rpm -q --changelog openssl | grep CVE-2014-0160
- pull in upstream patch for CVE-2014-0160

We'll have to restart the proxy systems (node-proxy) for the nodes for the effect of the patch. In fact, we will have to restart all the services that use the vulnerable OpenSSL versions.

# systemctl restart openshift-node-web-proxy.service

# /bin/systemctl reload httpd.service

I hope this helps :)