Installing mongrel2 on rhel6 beta2

Red Hat Enterprise Linux beta has been out for a while now and one very cool feature of the new RHEL is that it brings some pretty minimal "installation profiles", meaning that you don't end up with two thousand packages when you install a "server" machine.
It appears Red Hat finally have decoupled the packages properly, so it does not pull things like gnome when you want to do a clean server-only install.

For this post I used the minimal profile that is VERY minimal. It does not even bring perl by default, only ~225 packages total.

We will start by installing the following packages that are needed later for the different parts of a mongrel deployment.

yum install libuuid-devel gcc make gcc-c++ libstdc++-devel glib2-devel sqlite-devel python-devel git wget rpm-build python-setuptools
yum groupinstall "Development tools"

It is assumed you are logged in as a non privileged user that has full sudo access.

1. Install zeromq 2.0.7 (If you have already zeromq installed skip to step 2)
# Download zeromq-2.0.7.tar.gz

wget http://www.zeromq.org/local--files/area:download/zeromq-2.0.7.tar.gz

# Untar the archive

tar xf zeromq-2.0.7.tar.gz

# Get the following patch
zeromq-spec-rhel6.patch:

--- zeromq.spec 2010-06-04 12:36:40.000000000 -0500
+++ zeromq-2.0.7/zeromq.spec    2010-08-08 17:39:05.000000000 -0500
@@ -8,7 +8,7 @@
 Source:        http://www.zeromq.org/local--files/area:download/%{name}-%{version}.tar.gz
 Prefix:        %{_prefix}
 Buildroot:     %{_tmppath}/%{name}-%{version}-%{release}-root
-BuildRequires: uuid-devel, gcc, make, gcc-c++, libstdc++-devel
+BuildRequires: libuuid-devel, gcc, make, gcc-c++, libstdc++-devel
 Requires:      uuid, libstdc++
 
 # Build pgm only on supported archs
@@ -60,11 +60,9 @@
 # Install the package to build area
 %makeinstall
 
-%post
-/sbin/ldconfig
+%post -p /sbin/ldconfig
 
-%postun
-/sbin/ldconfig
+%postun -p /sbin/ldconfig
 
 %clean
 [ "%{buildroot}" != "/" ] && %{__rm} -rf %{buildroot}
@@ -101,6 +99,8 @@
 %{_mandir}/man3/zmq_bind.3.gz
 %{_mandir}/man3/zmq_close.3.gz
 %{_mandir}/man3/zmq_connect.3.gz
+%{_mandir}/man3/zmq_errno.3.gz
+%{_mandir}/man3/zmq_getsockopt.3.gz
 %{_mandir}/man3/zmq_init.3.gz
 %{_mandir}/man3/zmq_msg_close.3.gz
 %{_mandir}/man3/zmq_msg_copy.3.gz
@@ -118,6 +118,7 @@
 %{_mandir}/man3/zmq_strerror.3.gz
 %{_mandir}/man3/zmq_term.3.gz
 %{_mandir}/man3/zmq_version.3.gz
+
 %{_mandir}/man7/zmq_cpp.7.gz
 %{_mandir}/man7/zmq_epgm.7.gz
 %{_mandir}/man7/zmq_inproc.7.gz

# Apply the patch to spec file

[otaeguis@rhel6test src]$ pwd
/home/otaeguis/src
[otaeguis@rhel6test src]$ cd zeromq-2.0.7
[otaeguis@rhel6test zeromq-2.0.7]$ patch -p0 < ../zeromq-spec-rhel6.patch
patching file zeromq.spec

# Change the owner of the directory to the unprivileged user or rpmbuild will complain and fail:

[otaeguis@rhel6test src]$ pwd
/home/otaeguis/src
[otaeguis@rhel6test src]$ chown -R otaeguis.otaeguis zeromq-2.0.7/

# rpmbuild the binary and source packages (depending on the cpu power can take quite some time)

[otaeguis@rhel6test zeromq-2.0.7]$ rpmbuild -ba zeromq.spec
Executing(%prep): /bin/sh -e /var/tmp/rpm-tmp.AcHgKn
+ umask 022
+ cd /home/otaeguis/rpmbuild/BUILD
+ LANG=C
+ export LANG
+ unset DISPLAY
+ cd /home/otaeguis/rpmbuild/BUILD
+ rm -rf zeromq-2.0.7

(snip) ... Lots of compiler logs here ... (snip)

Checking for unpackaged file(s): /usr/lib/rpm/check-files /home/otaeguis/rpmbuild/BUILDROOT/zeromq-2.0.7-1.el6.i386
Wrote: /home/otaeguis/rpmbuild/SRPMS/zeromq-2.0.7-1.el6.src.rpm
Wrote: /home/otaeguis/rpmbuild/RPMS/i686/zeromq-2.0.7-1.el6.i686.rpm
Wrote: /home/otaeguis/rpmbuild/RPMS/i686/zeromq-devel-2.0.7-1.el6.i686.rpm
Wrote: /home/otaeguis/rpmbuild/RPMS/i686/zeromq-debuginfo-2.0.7-1.el6.i686.rpm
Executing(%clean): /bin/sh -e /var/tmp/rpm-tmp.0KJJ9l
+ umask 022
+ cd /home/otaeguis/rpmbuild/BUILD
+ cd zeromq-2.0.7
+ '[' /home/otaeguis/rpmbuild/BUILDROOT/zeromq-2.0.7-1.el6.i386 '!=' / ']'
+ /bin/rm -rf /home/otaeguis/rpmbuild/BUILDROOT/zeromq-2.0.7-1.el6.i386
+ exit 0

# Now install the main and devel packages:

rpm -ivh /home/otaeguis/rpmbuild/RPMS/i686/zeromq-2.0.7-1.el6.i686.rpm
rpm -ivh /home/otaeguis/rpmbuild/RPMS/i686/zeromq-devel-2.0.7-1.el6.i686.rpm

2. Install pyzmq (if you have pyzmq already installed skip to step 3)
# We need to get pyzmq from github

git clone git://github.com/zeromq/pyzmq.git pyzmq

# Them we need to install it

[otaeguis@rhel6test src]$ pwd
/home/otaeguis/src
[otaeguis@rhel6test src]$ cd pyzmq
[otaeguis@rhel6test pyzmq]$ cp setup.cfg.template setup.cfg
[otaeguis@rhel6test pyzmq]$ sed -i 's/local\/zeromq-dev\///' setup.cfg
[otaeguis@rhel6test pyzmq]$ sudo python setup.py install
running install
running build
running build_py
creating build
creating build/lib.linux-i686-2.6
creating build/lib.linux-i686-2.6/zmq
copying zmq/__init__.py -> build/lib.linux-i686-2.6/zmq
creating build/lib.linux-i686-2.6/zmq/tests

(snip) ... Lots of compiler logs here ... (snip)

running install_egg_info
Removing /usr/lib/python2.6/site-packages/pyzmq-0.1-py2.6.egg-info
Writing /usr/lib/python2.6/site-packages/pyzmq-0.1-py2.6.egg-info

3. Install mongrel2 
# Go to the fossil download site and get the latest release (url: http://www.fossil-scm.org/download.html) fossil comes as a binary statically compiled so its a single binary file that does not require any depency
# Checkout mongrel2:

[otaeguis@rhel6test ~]$ mkdir fossils
[otaeguis@rhel6test ~]$ cd fossils/
[otaeguis@rhel6test fossils]$ fossil clone http://mongrel2.org mongrel2
Bytes Cards Artifacts Deltas
Send: 49 1 0 0
Received: 95997 2086 0 0
Send: 10025 225 0 0
Received: 545392 222 40 160
Send: 18849 415 0 0
Received: 234812 408 27 373
Send: 37649 801 0 0
Received: 764200 808 197 603
Send: 32244 686 0 0
Received: 1839432 693 237 448
Total network traffic: 53353 bytes sent, 1014325 bytes received
Rebuilding repository meta-data...
2086 (100%)...
project-id: 1bfa0026ac3ef7aeb5eae4e49018abf54bc189b3
server-id: 923970873e073c2ca7e16cfb7dfebbc748d24a05
admin-user: otaeguis (password is "d63850")
[otaeguis@rhel6test fossils]$ cd ~
[otaeguis@rhel6test ~]$ cd src/
[otaeguis@rhel6test src]$ mkdir mongrel2
[otaeguis@rhel6test src]$ cd mongrel2/
[otaeguis@rhel6test mongrel2]$ fossil open /home/otaeguis/fossils/mongrel2

(snip) ... list of files .. (snip)

tests/tst_tests.c
tests/unixy_tests.c
tests/valgrind.sup
project-name: mongrel2
repository: /home/otaeguis/fossils/mongrel2
local-root: /home/otaeguis/src/mongrel2/
project-code: 1bfa0026ac3ef7aeb5eae4e49018abf54bc189b3
server-code: 4b5b804f136d8cdca9f624322fafd5c22d110a17
checkout: e998597d3fd6acb549931702776ebb4fb3e40107 2010-08-08 03:55:31 UTC
parent: d4d13a2c74a226e77eafb32e6e48a38d911dfd86 2010-08-07 08:39:05 UTC
tags: trunk
[otaeguis@rhel6test mongrel2]$

# Build the mongrel2

[otaeguis@rhel6test mongrel2]$ make
cc -g -Wall -Isrc -c -o src/adt/dict.o src/adt/dict.c
cc -g -Wall -Isrc -c -o src/adt/hash.o src/adt/hash.c
cc -g -Wall -Isrc -c -o src/adt/heap.o src/adt/heap.c
cc -g -Wall -Isrc -c -o src/adt/list.o src/adt/list.c
cc -g -Wall -Isrc -c -o src/adt/tst.o src/adt/tst.c
cc -g -Wall -Isrc -c -o src/bstr/bsafe.o src/bstr/bsafe.c
cc -g -Wall -Isrc -c -o src/bstr/bstraux.o src/bstr/bstraux.c
cc -g -Wall -Isrc -c -o src/bstr/bstrlib.o src/bstr/bstrlib.c
cc -g -Wall -Isrc -c -o src/config/config.o src/config/config.c
cc -g -Wall -Isrc -c -o src/config/db.o src/config/db.c
cc -g -Wall -Isrc -c -o src/http11/http11_parser.o src/http11/http11_parser.c
cc -g -Wall -Isrc -c -o src/mem/halloc.o src/mem/halloc.c
cc -g -Wall -Isrc -c -o src/task/context.o src/task/context.c
cc -g -Wall -Isrc -c -o src/task/fd.o src/task/fd.c
cc -g -Wall -Isrc -c -o src/task/net.o src/task/net.c
cc -g -Wall -Isrc -c -o src/task/print.o src/task/print.c
cc -g -Wall -Isrc -c -o src/task/qlock.o src/task/qlock.c
cc -g -Wall -Isrc -c -o src/task/rendez.o src/task/rendez.c
cc -g -Wall -Isrc -c -o src/task/task.o src/task/task.c
cc -g -Wall -Isrc -c -o src/cache.o src/cache.c
cc -g -Wall -Isrc -c -o src/connection.o src/connection.c
cc -g -Wall -Isrc -c -o src/dir.o src/dir.c
cc -g -Wall -Isrc -c -o src/handler.o src/handler.c
cc -g -Wall -Isrc -c -o src/handler_parser.o src/handler_parser.c
cc -g -Wall -Isrc -c -o src/headers.o src/headers.c
cc -g -Wall -Isrc -c -o src/host.o src/host.c
cc -g -Wall -Isrc -c -o src/mac_specific.o src/mac_specific.c
cc -g -Wall -Isrc -c -o src/mime.o src/mime.c
cc -g -Wall -Isrc -c -o src/pattern.o src/pattern.c
cc -g -Wall -Isrc -c -o src/proxy.o src/proxy.c
cc -g -Wall -Isrc -c -o src/register.o src/register.c
cc -g -Wall -Isrc -c -o src/request.o src/request.c
cc -g -Wall -Isrc -c -o src/response.o src/response.c
cc -g -Wall -Isrc -c -o src/routing.o src/routing.c
cc -g -Wall -Isrc -c -o src/server.o src/server.c
cc -g -Wall -Isrc -c -o src/setting.o src/setting.c
cc -g -Wall -Isrc -c -o src/state.o src/state.c
cc -g -Wall -Isrc -c -o src/superpoll.o src/superpoll.c
cc -g -Wall -Isrc -c -o src/unixy.o src/unixy.c
cc -g -Wall -Isrc -c src/task/asm.S -o src/task/asm.o
ar rcs build/libm2.a src/adt/dict.o src/adt/hash.o src/adt/heap.o src/adt/list.o src/adt/tst.o src/bstr/bsafe.o src/bstr/bstraux.o src/bstr/bstrlib.o src/config/config.o src/config/db.o src/http11/http11_parser.o src/mem/halloc.o src/task/context.o src/task/fd.o src/task/net.o src/task/print.o src/task/qlock.o src/task/rendez.o src/task/task.o src/cache.o src/connection.o src/dir.o src/handler.o src/handler_parser.o src/headers.o src/host.o src/mac_specific.o src/mime.o src/pattern.o src/proxy.o src/register.o src/request.o src/response.o src/routing.o src/server.o src/setting.o src/state.o src/superpoll.o src/unixy.o src/state.o src/http11/http11_parser.o src/task/asm.o
ranlib build/libm2.a
cc -g -Wall -Isrc -c -o src/mongrel2.o src/mongrel2.c
cc -g -Wall -Isrc -lzmq -lsqlite3 src/mongrel2.o -o bin/mongrel2 build/libm2.a
sqlite3 tests/config.sqlite < src/config/config.sql
sqlite3 tests/config.sqlite < src/config/example.sql
sqlite3 tests/config.sqlite < src/config/mimetypes.sql
cc -g -Wall -Isrc -lzmq -lsqlite3 -o tests/bstr_tests tests/bstr_tests.c build/libm2.a
cc -g -Wall -Isrc -lzmq -lsqlite3 -o tests/cache_tests tests/cache_tests.c build/libm2.a
cc -g -Wall -Isrc -lzmq -lsqlite3 -o tests/config_tests tests/config_tests.c build/libm2.a
cc -g -Wall -Isrc -lzmq -lsqlite3 -o tests/connection_tests tests/connection_tests.c build/libm2.a
cc -g -Wall -Isrc -lzmq -lsqlite3 -o tests/db_tests tests/db_tests.c build/libm2.a
cc -g -Wall -Isrc -lzmq -lsqlite3 -o tests/dir_tests tests/dir_tests.c build/libm2.a
cc -g -Wall -Isrc -lzmq -lsqlite3 -o tests/handler_parser_tests tests/handler_parser_tests.c build/libm2.a
cc -g -Wall -Isrc -lzmq -lsqlite3 -o tests/handler_tests tests/handler_tests.c build/libm2.a
cc -g -Wall -Isrc -lzmq -lsqlite3 -o tests/heap_tests tests/heap_tests.c build/libm2.a
cc -g -Wall -Isrc -lzmq -lsqlite3 -o tests/host_tests tests/host_tests.c build/libm2.a
cc -g -Wall -Isrc -lzmq -lsqlite3 -o tests/mime_tests tests/mime_tests.c build/libm2.a
cc -g -Wall -Isrc -lzmq -lsqlite3 -o tests/pattern_tests tests/pattern_tests.c build/libm2.a
cc -g -Wall -Isrc -lzmq -lsqlite3 -o tests/proxy_tests tests/proxy_tests.c build/libm2.a
cc -g -Wall -Isrc -lzmq -lsqlite3 -o tests/register_tests tests/register_tests.c build/libm2.a
cc -g -Wall -Isrc -lzmq -lsqlite3 -o tests/request_tests tests/request_tests.c build/libm2.a
cc -g -Wall -Isrc -lzmq -lsqlite3 -o tests/routing_tests tests/routing_tests.c build/libm2.a
cc -g -Wall -Isrc -lzmq -lsqlite3 -o tests/server_tests tests/server_tests.c build/libm2.a
cc -g -Wall -Isrc -lzmq -lsqlite3 -o tests/setting_tests tests/setting_tests.c build/libm2.a
cc -g -Wall -Isrc -lzmq -lsqlite3 -o tests/state_tests tests/state_tests.c build/libm2.a
cc -g -Wall -Isrc -lzmq -lsqlite3 -o tests/tst_tests tests/tst_tests.c build/libm2.a
cc -g -Wall -Isrc -lzmq -lsqlite3 -o tests/unixy_tests tests/unixy_tests.c build/libm2.a
sh ./tests/runtests.sh
----
RUNNING: ./tests/cache_tests
ALL TESTS PASSED
Tests run: 2
----
RUNNING: ./tests/config_tests
ALL TESTS PASSED
Tests run: 1
----
RUNNING: ./tests/connection_tests
ALL TESTS PASSED
Tests run: 3
----
RUNNING: ./tests/db_tests
ALL TESTS PASSED
Tests run: 3
----
RUNNING: ./tests/dir_tests
ALL TESTS PASSED
Tests run: 1
----
RUNNING: ./tests/handler_parser_tests
ALL TESTS PASSED
Tests run: 1
----
RUNNING: ./tests/handler_tests
ALL TESTS PASSED
Tests run: 3
----
RUNNING: ./tests/heap_tests
ALL TESTS PASSED
Tests run: 4
----
RUNNING: ./tests/host_tests
ALL TESTS PASSED
Tests run: 1
----
RUNNING: ./tests/mime_tests
ALL TESTS PASSED
Tests run: 3
----
RUNNING: ./tests/pattern_tests
ALL TESTS PASSED
Tests run: 1
----
RUNNING: ./tests/proxy_tests
ALL TESTS PASSED
Tests run: 2
----
RUNNING: ./tests/register_tests
ALL TESTS PASSED
Tests run: 3
----
RUNNING: ./tests/request_tests
ALL TESTS PASSED
Tests run: 1
----
RUNNING: ./tests/routing_tests
ALL TESTS PASSED
Tests run: 1
----
RUNNING: ./tests/server_tests
ALL TESTS PASSED
Tests run: 3
----
RUNNING: ./tests/setting_tests
ALL TESTS PASSED
Tests run: 2
----
RUNNING: ./tests/state_tests
ALL TESTS PASSED
Tests run: 3
----
RUNNING: ./tests/tst_tests
ALL TESTS PASSED
Tests run: 7
----
RUNNING: ./tests/unixy_tests
ALL TESTS PASSED
Tests run: 4

# Install the mongrel2

[otaeguis@rhel6test mongrel2]$ sudo make install
cd examples/python && sudo python setup.py install
running install
running bdist_egg
running egg_info
writing requirements to m2py.egg-info/requires.txt
writing m2py.egg-info/PKG-INFO
writing top-level names to m2py.egg-info/top_level.txt
writing dependency_links to m2py.egg-info/dependency_links.txt
reading manifest file 'm2py.egg-info/SOURCES.txt'
writing manifest file 'm2py.egg-info/SOURCES.txt'
installing library code to build/bdist.linux-i686/egg
running install_lib
running build_py
creating build/bdist.linux-i686/egg
creating build/bdist.linux-i686/egg/mongrel2
creating build/bdist.linux-i686/egg/mongrel2/config
copying build/lib/mongrel2/config/commands.py -> build/bdist.linux-i686/egg/mongrel2/config
copying build/lib/mongrel2/config/rc.py -> build/bdist.linux-i686/egg/mongrel2/config
copying build/lib/mongrel2/config/model.py -> build/bdist.linux-i686/egg/mongrel2/config
copying build/lib/mongrel2/config/__init__.py -> build/bdist.linux-i686/egg/mongrel2/config
copying build/lib/mongrel2/config/args.py -> build/bdist.linux-i686/egg/mongrel2/config
copying build/lib/mongrel2/handler.py -> build/bdist.linux-i686/egg/mongrel2
copying build/lib/mongrel2/request.py -> build/bdist.linux-i686/egg/mongrel2
creating build/bdist.linux-i686/egg/mongrel2/sql
copying build/lib/mongrel2/sql/config.sql -> build/bdist.linux-i686/egg/mongrel2/sql
copying build/lib/mongrel2/__init__.py -> build/bdist.linux-i686/egg/mongrel2
byte-compiling build/bdist.linux-i686/egg/mongrel2/config/commands.py to commands.pyc
byte-compiling build/bdist.linux-i686/egg/mongrel2/config/rc.py to rc.pyc
byte-compiling build/bdist.linux-i686/egg/mongrel2/config/model.py to model.pyc
byte-compiling build/bdist.linux-i686/egg/mongrel2/config/__init__.py to __init__.pyc
byte-compiling build/bdist.linux-i686/egg/mongrel2/config/args.py to args.pyc
byte-compiling build/bdist.linux-i686/egg/mongrel2/handler.py to handler.pyc
byte-compiling build/bdist.linux-i686/egg/mongrel2/request.py to request.pyc
byte-compiling build/bdist.linux-i686/egg/mongrel2/__init__.py to __init__.pyc
creating build/bdist.linux-i686/egg/EGG-INFO
installing scripts to build/bdist.linux-i686/egg/EGG-INFO/scripts
running install_scripts
running build_scripts
creating build/bdist.linux-i686/egg/EGG-INFO/scripts
copying build/scripts-2.6/m2sh -> build/bdist.linux-i686/egg/EGG-INFO/scripts
changing mode of build/bdist.linux-i686/egg/EGG-INFO/scripts/m2sh to 755
copying m2py.egg-info/PKG-INFO -> build/bdist.linux-i686/egg/EGG-INFO
copying m2py.egg-info/SOURCES.txt -> build/bdist.linux-i686/egg/EGG-INFO
copying m2py.egg-info/dependency_links.txt -> build/bdist.linux-i686/egg/EGG-INFO
copying m2py.egg-info/requires.txt -> build/bdist.linux-i686/egg/EGG-INFO
copying m2py.egg-info/top_level.txt -> build/bdist.linux-i686/egg/EGG-INFO
zip_safe flag not set; analyzing archive contents...
creating 'dist/m2py-0.2-py2.6.egg' and adding 'build/bdist.linux-i686/egg' to it
removing 'build/bdist.linux-i686/egg' (and everything under it)
Processing m2py-0.2-py2.6.egg
creating /usr/lib/python2.6/site-packages/m2py-0.2-py2.6.egg
Extracting m2py-0.2-py2.6.egg to /usr/lib/python2.6/site-packages
Adding m2py 0.2 to easy-install.pth file
Installing m2sh script to /usr/bin

Installed /usr/lib/python2.6/site-packages/m2py-0.2-py2.6.egg
Processing dependencies for m2py==0.2
Searching for storm
Reading http://pypi.python.org/simple/storm/
Reading https://storm.canonical.com
Reading https://launchpad.net/storm/+download
Best match: storm 0.17
Downloading http://launchpad.net/storm/trunk/0.17/+download/storm-0.17.tar.bz2
Processing storm-0.17.tar.bz2
Running storm-0.17/setup.py -q bdist_egg --dist-dir /tmp/easy_install-vS7lVd/storm-0.17/egg-dist-tmp-6oq5hv
Adding storm 0.17 to easy-install.pth file

Installed /usr/lib/python2.6/site-packages/storm-0.17-py2.6-linux-i686.egg
Searching for pyrepl
Reading http://pypi.python.org/simple/pyrepl/
Reading http://codespeak.net/pyrepl/
Best match: pyrepl 0.8.1
Downloading http://codespeak.net/pyrepl/pyrepl-0.8.1.tar.gz
Processing pyrepl-0.8.1.tar.gz
Running pyrepl-0.8.1/setup.py -q bdist_egg --dist-dir /tmp/easy_install-fBNZsX/pyrepl-0.8.1/egg-dist-tmp-tceCkH
zip_safe flag not set; analyzing archive contents...
Adding pyrepl 0.8.1 to easy-install.pth file
Installing pythoni script to /usr/bin

Installed /usr/lib/python2.6/site-packages/pyrepl-0.8.1-py2.6.egg
Searching for simplejson
Reading http://pypi.python.org/simple/simplejson/
Reading http://undefined.org/python/#simplejson
Best match: simplejson 2.1.1
Downloading http://pypi.python.org/packages/source/s/simplejson/simplejson-2.1.1.tar.gz#md5=0bbe3a2e5e4cac040013733aca159d89
Processing simplejson-2.1.1.tar.gz
Running simplejson-2.1.1/setup.py -q bdist_egg --dist-dir /tmp/easy_install-edSt7Z/simplejson-2.1.1/egg-dist-tmp-YNSy42
Adding simplejson 2.1.1 to easy-install.pth file

Installed /usr/lib/python2.6/site-packages/simplejson-2.1.1-py2.6-linux-i686.egg
Searching for nose
Reading http://pypi.python.org/simple/nose/
Reading http://somethingaboutorange.com/mrl/projects/nose/
Best match: nose 0.11.4
Downloading http://somethingaboutorange.com/mrl/projects/nose/nose-0.11.4.tar.gz
Processing nose-0.11.4.tar.gz
Running nose-0.11.4/setup.py -q bdist_egg --dist-dir /tmp/easy_install-X1E5FW/nose-0.11.4/egg-dist-tmp-C_Wjxy
no previously-included directories found matching 'doc/.build'
Adding nose 0.11.4 to easy-install.pth file
Installing nosetests-2.6 script to /usr/bin
Installing nosetests script to /usr/bin

Installed /usr/lib/python2.6/site-packages/nose-0.11.4-py2.6.egg
Finished processing dependencies for m2py==0.2
sudo install bin/mongrel2 /usr/local/bin/

And that is pretty much it.

To test it and use it you can go read: Getting Started on the mongrel2.org site

A small python script to have automated audit trail of war files deployed on a jboss server that runs on windows

The requirements were:

1. Must be able to tell if the application war file was changed.
2. Must run on windows.
3. Must log to a centralized searchable repository.

For #1 an easy way to identify a file is hash it, sha1 is a safe enough hash algorithm.
For #3 we use splunk as log server with syslog-ng as network log server, we could use the splunk listeners but if splunk goes down for any reason we would lose logging so syslog-ng logs to a file and then splunk indexes that file. win-win

The following script runs daily and the results (hashes) submitted to a log server and the change management paperwork for the deployment includes the hash of the file to do a control.

You can find the source in my github

#!/usr/bin/env python
# encoding: utf-8
'''
   do-audit is a small script to create hashes from web applications in a jboss
   server for audit purposes. 
'''
import hashlib
import os
import stat
# time convertions
import time
# for hostname
import platform
# logging facilities
import logging
import logging.handlers
# email facilities
import smtplib
from email.mime.text import MIMEText

def sha1(_filename):
  '''Returns a sha1 of the file received'''
  if os.path.isfile(_filename):
    return hashlib.sha1(open(_filename).read()).hexdigest()
  else:
    return "Cannot hash file: " + _filename

def md5(_filename):
  '''Returns a sha1 of the file received'''
  if os.path.isfile(_filename):
    return hashlib.md5(open(_filename).read()).hexdigest()
  else:
    return "Cannot hash file: " + _filename

def get_fileinfo(_filename):
  '''Returns the file size in bytes and the Last modified attribute'''
  if os.path.isfile(_filename):
    file_stats = os.stat(_filename)
    file_info = {
      'fsize': file_stats [stat.ST_SIZE],
      'f_lm': time.strftime("%Y%m%d-%H:%M",time.localtime(file_stats[stat.ST_MTIME])),
      'f_ct': time.strftime("%Y%m%d-%H:%M",time.localtime(file_stats[stat.ST_CTIME]))
    }
  return 'Size=%(fsize)s LastMod=%(f_lm)s' % file_info

#This is where we setup which instances (jboss profiles) we are going to be monitoring
audited_instances = [
'APPSRV1',
'APPSRV2',
'APPSRV3',
'APPSRV4'
]

# these are the default locations but we want to be able to change this
jboss_basedir = '/srv/jboss-eap/server'
deployment_directory = 'theappdir'
# we only audit war files but we could audit more than that
audited_extensions = ['war']
host = platform.node()


my_logger = logging.getLogger(host)
my_logger.setLevel(logging.INFO)
# We setup the logging host
handler_syslog = logging.handlers.SysLogHandler(address=('syslogserver', 514))
my_logger.addHandler(handler_syslog)

period = time.strftime("%Y%m" , time.localtime())
tmp_email = []

for instance in audited_instances:
  print instance
  fullsrvdir = os.path.join(jboss_basedir, instance, deployment_directory)
  for root, dirs, files in os.walk(fullsrvdir):
    for f in files:
      if f[-3:].lower() in audited_extensions:
        filename = os.path.join(root, f)
        msg = "APPAUDIT Host=%s Instance=%s Period=%s Artifact=%s %s SHA1=%s MD5=%s" % \
          (host, instance, period, f, get_fileinfo(filename), sha1(filename), md5(filename))
        print msg
        # now we submit the log line to splunk or the syslog server
        my_logger.info(msg)
        tmp_email.append("%s\n" % (msg))

# We setup the email settings
email_from = "auditor@server.name"
email_to = "feniix@server.name" 
# we do unauthenticated smtp delivery
email_server = "smtp.server.name" 

# and we build the email
email = MIMEText(''.join(tmp_email))
email['Subject'] = "App Audit - %s %s %s" % (time.strftime("%Y %B", time.localtime()), 
                                             period, host)
email['From'] = email_from
email['To'] = email_to

# and we send the email
s = smtplib.SMTP(email_server)
s.sendmail(email_from, email_to, email.as_string())
s.close()

A better startup script for jboss

One of the approaches to run multiple instances in one box is the Service Binding Manager. The other way, and the recommended one is to use multiple IPs attached to the box and this is for many reasons.

1. When you have a port conflict, it makes it very difficult to troubleshoot, given a large amount of ports and app servers.
2. Too many ports makes firewall rules too difficult to maintain.
3. Isolating the IP addresses gives you a guarantee that no other app server will be using the ports.
4. Each upgrade requires that you go in and re set the binding manager again. Most upgrades will upgrade the conf/jboss-service.xml file, which has the Service binding manager configuration in it.
5. The configuration is much simpler. When defining new ports (either through the Service Binding manager or by going in and changing all the ports in the configuration), it's always a headache trying to figure out which ports aren't taken already. If you use a NIC per JBoss Instance, all you have to change is the Ip address binding argument when executing the run.sh or run.bat. (-b ip_address)
6. Once you get 3 or 4 applications using different ports, the chances really increase that you will step on another one of your applications ports. It just gets more difficult to keep ports from conflicting.
7. JGroups will pick random ports within a cluster to communicate. Sometimes when clustering, if you are using the same ip address, two random ports may get picked in two different app servers(using the binding manager) that conflict. You can configure around this, but it's better not to run into this situation at all.

In rhel this is relatively easy because the package comes with that ready to implement(i.e. you just need to create some symlinks and modify some text files), but in the case of jboss on windows you have to depend on jboss-native to install the application as a windows service. The problem is that as of jboss native 2.0.8 the service.bat script cannot handle multiples IPs.
For the startup there is no problem, because if we wanted we could "hard-code" the IPs in the run.bat and then the problem is the shutdown where we need to connect to the IP and port 1099 using jnp. If we do that, we end up with IPs in duplicated in the run.bat, and shutdown.bat.

Requirements:

• jboss.home.dir = D:\Jboss-eap\jbossas
• jboss.server.home.dir = D:\Jboss-eap\jbossas\server\ThisIsTheTestServer
• ip = 192.168.10.40
• jmx user configured (is necessary for the shutdown only)

Once the scripts are copied to the jboss bin directory you need to do the following:

Install the windows service:

D:\>cd \jboss-eap\jbossas\bin
D:\jboss-eap\jbossas\bin>service_ThisIsTheTestServer.bat install

Start the jboss instance from the windows services console (Start -> run... services.msc)
The services name is JB_ThisIsTheTestServer

service_ThisIsTheTestServer.bat is the service control script used to start, stop and restart.

@echo off
REM sets the version of runtime you run.
set JAVA_HOME=D:\Java\jdk1.6.0_16
REM sets the bind ip
set SERVERIP=192.168.10.40
REM sets the multicast ip, it's only used if you have configured the jboss profile to be a cluster and jgroups replicates to the other nodes.
set MCASTIP=229.11.0.4
REM sets the name of the profile used.
set SERVERCFG=ThisIsTheTestServer
REM sets the jmx user/password.
set ADMUSER=-u admin -p CHANGEME

@if not "%ECHO%" == "" echo %ECHO%
@if "%OS%" == "Windows_NT" setlocal
set DIRNAME=%CD%

set SVCNAME=JB_%SERVERCFG%
set SVCDISP=JB_%SERVERCFG%
set SVCDESC=JBoss_%SERVERCFG%
set NOPAUSE=Y

@if "%1" == "install"   goto cmdInstall
@if "%1" == "uninstall" goto cmdUninstall
@if "%1" == "start"     goto cmdStart
@if "%1" == "stop"      goto cmdStop
@if "%1" == "restart"   goto cmdRestart
@if "%1" == "signal"    goto cmdSignal
echo Usage: service install^|uninstall^|start^|stop^|restart^|signal
goto cmdEnd

:errExplain
@if errorlevel 1 echo Invalid command line parameters
@if errorlevel 2 echo Failed installing %SVCDISP%
@if errorlevel 4 echo Failed removing %SVCDISP%
@if errorlevel 6 echo Unknown service mode for %SVCDISP%
goto cmdEnd

:cmdInstall
jbosssvc.exe -iwdc %SVCNAME% "%DIRNAME%" "%SVCDISP%" "%SVCDESC%" service_%SERVERCFG%.bat
@if not errorlevel 0 goto errExplain
echo Service %SVCDISP% installed
goto cmdEnd

:cmdUninstall
jbosssvc.exe -u %SVCNAME%
@if not errorlevel 0 goto errExplain
echo Service %SVCDISP% removed
goto cmdEnd

:cmdStart
REM Executed on service start
run_%SERVERCFG%.bat  -b %SERVERIP% -c %SERVERCFG% -u %MCASTIP% -Djboss.partition.name=%SERVERCFG% >>run_%SERVERCFG%.log
goto cmdEnd

:cmdStop
call shutdown.bat -S -s jnp://%SERVERIP%:1099 %ADMUSER% >shutdown_%SERVERCFG%.log
goto cmdEnd

:cmdRestart
call shutdown.bat -S -s jnp://%SERVERIP%:1099 %ADMUSER% >>shutdown_%SERVERCFG%log
call run_%SERVERCFG%.bat  -b %SERVERIP% -c %SERVERCFG% -u %MCASTIP% -Djboss.partition.name=%SERVERCFG% >>run_%SERVERCFG%.log
goto cmdEnd

:cmdSignal
@if not ""%2"" == """" goto execSignal
echo Missing signal parameter.
echo Usage: service signal [0...9]
goto cmdEnd
:execSignal
jbosssvc.exe -k%2 %SVCNAME%
goto cmdEnd

:cmdEnd

run_ThisIsTheTestServer.bat is called by service_ThisIsTheTestServer.bat

@echo off

set TEMP=D:\temp\%SERVERCFG%
set TMP=D:\temp\%SERVERCFG%

if not exist %TEMP% mkdir %TEMP%
if not exist %TMP% mkdir %TMP%

@if not "%ECHO%" == ""  echo %ECHO%
@if "%OS%" == "Windows_NT"  setlocal

set DIRNAME=.\
if "%OS%" == "Windows_NT" set DIRNAME=%~dp0%
set PROGNAME=run_%SERVERCFG%.bat
if "%OS%" == "Windows_NT" set PROGNAME=%~nx0%

pushd %DIRNAME%..
set JBOSS_HOME=%CD%
popd

REM Add bin/native to the PATH if present
if exist "%JBOSS_HOME%\bin\native" set PATH=%JBOSS_HOME%\bin\native;%PATH%
if exist "%JBOSS_HOME%\bin\native" set JAVA_OPTS=%JAVA_OPTS% -Djava.library.path="%PATH%"

set RUNJAR=%JBOSS_HOME%\bin\run.jar
if exist "%RUNJAR%" goto FOUND_RUN_JAR
echo Could not locate %RUNJAR%. Please check that you are in the
echo bin directory when running this script.
goto END

:FOUND_RUN_JAR

if not "%JAVA_HOME%" == "" goto ADD_TOOLS

set JAVA=java

echo JAVA_HOME is not set.  Unexpected results may occur.
echo Set JAVA_HOME to the directory of your local JDK to avoid this message.
goto SKIP_TOOLS

:ADD_TOOLS

set JAVA=%JAVA_HOME%\bin\java

if not exist "%JAVA_HOME%\lib\tools.jar" goto SKIP_TOOLS

set JAVAC_JAR=%JAVA_HOME%\lib\tools.jar

:SKIP_TOOLS

if not "%JAVAC_JAR%" == "" set RUNJAR=%JAVAC_JAR%;%RUNJAR%
if "%JBOSS_CLASSPATH%" == "" set RUN_CLASSPATH=%RUNJAR%
if "%RUN_CLASSPATH%" == "" set RUN_CLASSPATH=%JBOSS_CLASSPATH%;%RUNJAR%

set JBOSS_CLASSPATH=%RUN_CLASSPATH%

set JAVA_OPTS=%JAVA_OPTS% -Dprogram.name=%PROGNAME%

"%JAVA%" -version 2>&1 | findstr /I hotspot > nul
if not errorlevel == 1 (set JAVA_OPTS=%JAVA_OPTS% -server)

set JAVA_OPTS=%JAVA_OPTS% -Dsun.rmi.dgc.client.gcInterval=3600000 -Dsun.rmi.dgc.server.gcInterval=3600000

set JAVA_OPTS=-Djava.awt.headless=true %JAVA_OPTS% 

REM Memory settings
set JAVA_OPTS=%JAVA_OPTS% -Xms128m -Xmx200m -XX:MaxPermSize=128m
REM Variable definitions in case you need to pass parameters for the application
set JAVA_OPTS=-Dvariable=value %JAVA_OPTS%

REM This is required as it is reducing the signals received from the OS, if it is not setup
REM when you log off, the service stops
set JAVA_OPTS=%JAVA_OPTS% -Xrs

set JBOSS_ENDORSED_DIRS=%JBOSS_HOME%\lib\endorsed

echo ===============================================================================
echo.
echo   JBoss Bootstrap Environment
echo.
echo   JBOSS_HOME: %JBOSS_HOME%
echo.
echo   JAVA: %JAVA%
echo.
echo   JAVA_OPTS: %JAVA_OPTS%
echo.
echo   CLASSPATH: %JBOSS_CLASSPATH%
echo.
echo ===============================================================================
echo.

:RESTART
"%JAVA%" %JAVA_OPTS% "-Djava.endorsed.dirs=%JBOSS_ENDORSED_DIRS%" -classpath "%JBOSS_CLASSPATH%" org.jboss.Main %*
if ERRORLEVEL 10 goto RESTART

:END
if "%NOPAUSE%" == "" pause

:END_NO_PAUSE

shutdown.bat is also called by service_ThisIsTheTestServer.bat common to all the servers

@echo off

if not "%ECHO%" == ""  echo %ECHO%
if "%OS%" == "Windows_NT"  setlocal

set MAIN_JAR_NAME=shutdown.jar
set MAIN_CLASS=org.jboss.Shutdown

set DIRNAME=.\
if "%OS%" == "Windows_NT" set DIRNAME=%~dp0%
set PROGNAME=run.bat
if "%OS%" == "Windows_NT" set PROGNAME=%~nx0%

set ARGS=
:loop
if [%1] == [] goto end
        set ARGS=%ARGS% %1
        shift
        goto loop
:end

set MAIN_JAR=%DIRNAME%\%MAIN_JAR_NAME%
if exist "%MAIN_JAR%" goto FOUND_MAIN_JAR
echo Could not locate %MAIN_JAR%. Please check that you are in the
echo bin directory when running this script.
goto END

:FOUND_MAIN_JAR

if not "%JAVA_HOME%" == "" goto HAVE_JAVA_HOME

set JAVA=java

echo JAVA_HOME is not set.  Unexpected results may occur.
echo Set JAVA_HOME to the directory of your local JDK to avoid this message.
goto SKIP_SET_JAVA_HOME

:HAVE_JAVA_HOME

set JAVA=%JAVA_HOME%\bin\java

:SKIP_SET_JAVA_HOME

set JBOSS_CLASSPATH=%JBOSS_CLASSPATH%;%MAIN_JAR%;%DIRNAME%/../client/jbossall-client.jar

set JAVA_OPTS=%JAVA_OPTS% -Djboss.boot.loader.name=%PROGNAME%

"%JAVA%" %JAVA_OPTS% -classpath "%JBOSS_CLASSPATH%" %MAIN_CLASS% %ARGS%

:END
if "%NOPAUSE%" == "" pause

:END_NO_PAUSE

So it starts like this

A couple of days ago, I needed to create a list of the third party libraries provided by jboss eap 4.3, that list was going to be used to allow the developers to know which versions of the libraries they can configure in their applications and it should be put in our wiki for everyone to see.

We have the following requirements:

• ant 1.6.5+
• subversion
• python 2.5+
• a running version of confluence

The first step is to checkout some files from the corresponding tag from the jboss public repository

mkdir ~/temp
cd ~/temp
svn co http://anonsvn.jboss.org/repos/jbossas/tags/JBPAPP_4_3_0_GA/build
svn co http://anonsvn.jboss.org/repos/jbossas/tags/JBPAPP_4_3_0_GA/tools
svn co http://anonsvn.jboss.org/repos/jbossas/tags/JBPAPP_4_3_0_GA/thirdparty

The second step is to run the corresponding ant build to generate the components info files

cd ~/temp/build
ant -f build-thirdparty.xml

Download this script and replace CHANGEME with the path to the correct location.

import os
import urllib2
import re
from BeautifulSoup import BeautifulSoup


basedir = '/CHANGEME/temp/thirdparty' # this is the directory that needs to be changed
infofile = 'component-info.xml'
ts = '|'
header = '||Component||Library||Version||Description||Comment||'

print header

for dir in sorted(os.listdir(basedir)):
    if dir == '':
        print 'The directory structure is not adequate'
        break
    filepath = ''.join([basedir, '/', dir, '/', infofile])
    if os.path.exists(filepath):
        filepath = ''.join(['file://',filepath])
        file = urllib2.urlopen(filepath)
    else:
        continue

    xml = file.read()
    soup = BeautifulSoup(xml)

    for attr, value in soup.find('component').attrs:
        if attr == 'id':
            c_attr = value
        elif attr == 'description':
            d_attr = str(value).replace('\n', '').replace('  ', '')
        elif attr == 'version':
            v_attr = value
        else:
            continue

    artifacts = soup.findAll('artifact')
    for line in artifacts:
        ids = re.split('"', str(line))
        print ts + c_attr + ts + ids[1] + ts + v_attr + ts + d_attr + ts + " " + ts

And finally run the script

python /path/to/the/script/jbjarversion.py > list

In 'list' you have the list formatted to be a confluence table.