User talk:Darek Kedra/sandbox 21

Content moved to: http://openwetware.org/wiki/Wikiomics:Ensembl_local_install

=ENSEMBL Install=

At this moment (2010-05-08) this is not a validated procedure guaranteed to give you a fully functional local ENSEMBL mirror / instance. These are observations observations made during an ongoing attempt to get such working installation.

There is another page dealing with errors during ENSEMBL installation (in Japanese) here: 

Use translate.google.com / find a speaker.

Hardware

 * 4 cores Intel(R) Xeon(R) CPU E5405 @ 2.00GHz
 * 16GB RAM

This is not a prerequisite, but compilation times on slower machine will take longer time.

Software

 * Fedora 8 64-bit Linux
 * gcc 4.1.2

Requirements

 * without any databases sources directories < 1 Gb,
 * destination directory (apache2, mysql, perl, perl modules, ensembl + bioperl code, etc. ca 600Mb

General remarks
This is an attempt to get a working local ENSEMBL in a step-by-step fashion. Most likely one can get away with a bit smaller number of perl modules, no brand new copy of Perl, using distribution-specific apache2, etc. On the other hand for an ENSEMBL newbie, the safest path is to stay at the beginning as close to the provided configuration as possible. Instead of performing 100 steps, from software components installation, MySQL database creation, data loading and with final editing of ENSEMBL configuration files it may be safer to treat each of the steps separately and use error messages as checkpoints.

=Web server installation= ENSEMBL relies on mod_perl.

Perl
http://www.perl.org/get.html got perl-5.12.0 cd perl-5.12.0 cd perl-5.12.0/ CFLAGS='-m64 -mtune=nocona' ./Configure -des -A ccflags=-fPIC -Dprefix=/home/ensembl_usr/local/ make make test make install

The "CFLAGS" line is required on 64-bit Linux system to compile mod_perl. See relevant section below

Apache httpd
http://httpd.apache.org/download.cgi got httpd-2.2.15.tar.gz

installation: ./configure --enable--deflate --enable--headers --enable--expires --prefix=/home/ensembl_usr/local/apache2 make make test make install

checking what is build in: /home/ensembl_usr/local/apache2/bin/apachectl -t -D DUMP_MODULES

my output: core_module (static) authn_file_module (static) authn_default_module (static) authz_host_module (static) authz_groupfile_module (static) authz_user_module (static) authz_default_module (static) auth_basic_module (static) include_module (static) filter_module (static) log_config_module (static) env_module (static) setenvif_module (static) version_module (static) mpm_prefork_module (static) http_module (static) mime_module (static) status_module (static) autoindex_module (static) asis_module (static) cgi_module (static) negotiation_module (static) dir_module (static) actions_module (static) userdir_module (static) alias_module (static) so_module (static) perl_module (shared)

I do not get here modules: deflate, headers, expires

What works (in httpd-2.2.15 directory): cd ./modules/filters/ /home/ensembl_usr/local/apache2/bin/apxs -c mod_deflate.c cp -i .libs/mod_deflate.so /home/ensembl_usr/local/apache2/modules/

cd ../metadata/ /home/ensembl_usr/local/apache2/bin/apxs -c mod_headers.c cp -i .libs/mod_headers.so /home/ensembl_usr/local/apache2/modules/

/home/ensembl_usr/local/apache2/bin/apxs -c mod_expires.c cp -i .libs/mod_expires.so /home/ensembl_usr/local/apache2/modules/

cd /home/ensembl_usr/local/ensembl/modules ln -s /home/ensembl_usr/local/apache2/modules/*.so.

mod_per 4 apache2.x
http://perl.apache.org/download/index.html got mod_perl-2.0.4

cd mod_perl-2.0.4 perl Makefile.PL MP_APXS=/home/ensembl_usr/local/apache2/bin/apxs make make test


 * 1) t/modules/apache_status.t ............... 1/15 # Failed test 14 in t/modules/apache_status.t at line 47
 * 2) Failed test 15 in t/modules/apache_status.t at line 47 fail #2
 * 3) t/modules/apache_status.t ............... Failed 2/15 subtests

make install

Perl modules required by ENSEMBL
Assumes that you installed Perl in /home/ensembl_usr/local/ and got perl binary in /home/ensembl_usr/local/bin/

Check the list of modules here i.e: http://browser.1000genomes.org/info/docs/webcode/install/non-ensembl-code.html

There are several versions of this list of modules, but ultimately you may be missing several not listed modules, and you will get their names (one by one) after trying to start your ENSEMBL site.

Despite the advise of installing always the newest module versions there is one important exception: LWP. LWP version 5.812 is required by latest (2.57) ParallelUserAgent. This will be covered in a separate section of this page.

Also some modules do not install (at least on my machine) from perls CPAN shell. These may require installation by hand from sources (described later).

CPAN Shell
Easy things first:

export PATH=/home/ensembl_usr/local/bin/:$PATH which perl # ~/local/bin/perl perl -MCPAN -e shell install Cache::Memcached # Cache-Memcached-1.28.tar.gz install CGI # CGI.pm-3.49.tar.gz install CGI::Ajax # CGI-Ajax-0.707.tar.gz install CGI::Session # CGI-Session-4.42.tar.gz install Class::Accessor # Class::Accessor is up to date (0.34). // checked after installation of all modules install Class::Data::Inheritable # Class::Data::Inheritable is up to date (0.08). // checked after inst. of all modules install Class::Std # Class-Std-0.011.tar.gz install	Class::DBI::Sweet # Class-DBI-Sweet-0.10.tar.gz install Class::Std::Utils # Class-Std-Utils-v0.0.3.tar.gz install Compress::Zlib # Compress::Zlib is up to date (2.027). install Compress::Raw::Zlib # Compress-Raw-Zlib-2.027.tar.gz install SHLOMIF/Config-IniFiles-2.57.tar.gz       #//  i  /Config::Inifiles/ lists 3 modules, this one is the only one which version is close to the one listed on 1000genomes.org web page (see above) install DBI #DBI-1.611.tar.gz install DB_File # DB_File is up to date (1.820). // day 2, after installation of other modules install Devel::StackTrace # Devel-StackTrace-1.22.tar.gz install GD	#GD-2.45.tar.gz install Data::UUID #Data-UUID-1.203.tar.gz install Digest::MD5 #Digest::MD5 is up to date (2.39). install Exception::Class # Exception-Class-1.30.tar.gz install install Hash::Merge # Hash-Merge-0.12.tar.gz install Storable #Storable is up to date (2.22). install SOAP::Lite # SOAP-Lite-0.711.tar.gz install XML::Parser # XML::Parser is up to date (2.36). install XML::Simple # XML-Simple-2.18.tar.gz install Parse::RecDescent # Parse-RecDescent-1.965001.tar.gz install	PDF::API2 # PDF-API2-0.73.tar.gz install Spreadsheet::WriteExcel # Spreadsheet-WriteExcel-2.37.tar.gz install OLE::Storage_Lite # OLE::Storage_Lite is up to date (0.19) install Time::HiRes # Time-HiRes-1.9721.tar.gz install HTML::Template # HTML-Template-2.9.tar.gz install File::Temp # File::Temp is up to date (0.22) install	Mail::Mailer # MailTools-2.06.tar.gz install Math::Bezier # Math-Bezier-0.01.tar.gz install IO::String # IO-String-1.08.tar.gz install Image::Size # Image-Size-3.221.tar.gz install K/KM/KMACLEOD/libxml-perl-0.08.tar.gz        # // OK install List::MoreUtils #List-MoreUtils-0.22.tar.gz install Log::Log4perl # Log-Log4perl-1.28.tar.gz install Number::Format # Number-Format-1.73.tar.gz install DCONWAY/Parse-RecDescent-1.965001.tar.gz       # // closest with ver. number from 1000genomes.org list install S/SM/SMUELLER/PathTools-3.31.tar.gz        # // closest with ver. number from 1000genomes.org list install Template::Plugin::Number::Format # Template-Plugin-Number-Format-1.02.tar.gz install Time::HiRes # Time-HiRes-1.9721.tar.gz install XML::DOM # XML-DOM-1.44.tar.gz install XML::RSS # XML-RSS-1.48.tar.gz install Compress::Bzip2 # Compress-Bzip2-2.09.tar.gz
 * 1) commit: wrote '/home/ensembl_usr/local/lib/perl5/5.12.0/CPAN/Config.pm'
 * 1) commit: wrote '/home/ensembl_usr/local/lib/perl5/5.12.0/CPAN/Config.pm'

These modules were missing when starting httpd with ENSEMBL's httpd.conf:

install BSD::Resource # BSD-Resource-1.2904.tar.gz install Sys::Hostname::Long # Sys-Hostname-Long-1.4.tar.gz install JSON # JSON-2.21.tar.gz

From sources
Problem: requires MySQL (at least mysql client) on the same machine mysql-5.1.46 ./configure --prefix=/home/ensembl_usr/local/ make make check make test this is a long compile even on a fast machine.
 * DBD::mysql
 * install MySQL

cd DBD-mysql-4.014 perl Makefile.PL --with-mysql=/home/ensembl_usr/local/mysql/ make make install
 * DBD::mysql itself

Not tested: for proper testing you have to have an access to a running MySQL instance

===> problem in CPAN shell: install Bio::Das::Lite
 * Bio::Das::Lite

depends on WWW::Curl , reports SZBALINT/WWW-Curl-4.11.tar.gz Running make install make test had returned bad status, won't install without force going for source install got: WWW-Curl-4.11
 * WWW::Curl

cd WWW-Curl-4.11/ perl Makefile.PL PREFIX=/home/ensembl_usr/local/ make make test
 * 1) Failed 1/20 test programs. 3/177 subtests failed.
 * 2) make: *** [test_dynamic] Error 255

make install

cd Bio-Das-Lite-2.03/ perl Makefile.PL PREFIX=/home/ensembl_usr/local/ make make test
 * Bio::Das::Lite
 * 1) Failed 18/22 test programs. 0/1 subtests failed.

make install

This one does not look good. A number of errors like "Bareword "CURLOPT_PROXYUSERNAME" not allowed while "strict subs" in use at lib/Bio/Das/Lite.pm line 861.". No solution at this point.

Used by BioMart got Template-Toolkit-2.22 cd Template-Toolkit-2.22 perl Makefile.PL PREFIX=/home/ensembl_usr/local/ make make test make install
 * Template::Toolkit
 * 1) All tests successful.

ParallelUserAgent
The most current version (2.57) has not been updated in years, and requires old (5.812) wersion of LWN.

http://git-scm.com/ got git-1.7.1
 * git (in case you do not have it)

./configure --prefix=/home/ensembl_usr/local/ make make test make install

Two steps below are kind contribution of Dr K.B from England. (I will give her full credits once I get her permission to do so).


 * libwww ver. 5.812

mkdir libwww_old cd libwww_old git clone git://gitorious.org/libwww-perl/mainline.git cd mainline git checkout R5.812 perl Makefile.PL PREFIX=/home/ensembl_usr/local/ make make test
 * 1) local/httpsub.t     (Wstat: 512 Tests: 0 Failed: 0)
 * 2)  Non-zero exit status: 2
 * 3)  Parse errors: No plan found in TAP output
 * 4) Failed 1/33 test programs. 0/833 subtests failed.
 * 1) Failed 1/33 test programs. 0/833 subtests failed.


 * ParallelUserAgent (itself)

export PERL5LIB=/home/ensembl_usr/local/lib/perl5/

wget http://search.cpan.org/CPAN/authors/id/M/MA/MARCLANG/ParallelUserAgent-2.57.tar.gz tar xfvz ParallelUserAgent-2.57.tar.gz cd ParallelUserAgent-2.57 perl Makefile.PL PREFIX=/home/ensembl_usr/local/ make make test make install
 * 1) All tests successful.

Summary of this stage
Most of the components install OK to this point. Broken stuff:
 * Bio::Das::Lite (possibly older version of Perl may have been more tolerant?)

=ENSEMBL code=

prepare the directory
cd /home/ensembl_usr/local/ mkdir ensembl_2010.05.08 ln -s ensembl_2010.05.08 ensembl cd ensembl

API + web-code
in: /home/ensembl_usr/local/ensembl

cvs passwd: CVSUSER cvs -d :pserver:cvsuser@cvs.sanger.ac.uk:/cvsroot/ensembl login cvs -d :pserver:cvsuser@cvs.sanger.ac.uk:/cvsroot/ensembl co -r branch-ensembl-57 ensembl-api ensembl-website

BioMart
in: /home/ensembl_usr/local/ensembl cvs passwd: CVSUSER cvs -d :pserver:cvsuser@cvs.sanger.ac.uk:/cvsroot/biomart login cvs -d :pserver:cvsuser@cvs.sanger.ac.uk:/cvsroot/biomart co -r release-0_6 biomart-perl

BioPerl
The required version for parsing BLAST results in ENSEMBL is BioPerl 1.2.3. According to some users posting at dev-ensembl mailing list, you may try to go with newest BioPerl. BLAST searches will not work, but ensembl-api should be OK. Since I am not sure what will give us better overall functionality, we can fork here and get both branches for future testing.

UNRESOLVED 2010-05-08: "normal" installation of BioPerl requires building it (perl Makefile.PL etc.). So far none of the pages describing ENSEMBL install mentions this step. For time being I stop at unpacking BioPerl.

CVS repository does not exist anymore, we have to get the whole thing from: http://search.cpan.org/~birney/bioperl-1.2.3/
 * BioPerl 1.2.3

wget http://search.cpan.org/CPAN/authors/id/B/BI/BIRNEY/bioperl-1.2.3.tar.gz tar xfvz bioperl-1.2.3.tar.gz mv bioperl-1.2.3/ /home/ensembl_usr/local/ensembl_2010.05.08 cd /home/ensembl_usr/local/ensembl_2010.05.08 ln -s bioperl-1.2.3 bioperl-live/

Requires that you have subvertion installed (not covered here in case you do not) in: some place you unpack sources svn co svn://code.open-bio.org/bioperl/bioperl-live/trunk bioperl-live mv bioperl-live bioperl-live.2010.05.08 mv bioperl-live.2010.05.08 /home/ensembl_usr/local/ensembl_2010.05.08/ cd /home/ensembl_usr/local/ensembl_2010.05.08 ln -s ./bioperl-live.2010.05.08 ./bioperl-live
 * Latest bioperl-live

This is where we should have (almost, see below ) all what is needed for running the site.

=ENSEMBL configuration=

I change the configuration in an incremental fashion, starting with the small changes.

Step 1
cd /home/ensembl_usr/local/ensembl/conf/ cp Plugins.pm-dist Plugins.pm
 * create default config files:

cd /home/ensembl_usr/local/ensembl/public-plugins/mirror/conf/ cp SiteDefs.pm SiteDefs.pm-dist


 * Edit: /home/ensembl_usr/local/ensembl/conf/httpd.conf

change: 
 * 1) !/usr/local/bin/perl -w

to:


 * 1) !/home/ensembl_usr/local/bin/perl -w

in /home/ensembl_usr/local/ensembl/conf/httpd.conf there are few modules required. To check this, simply run:
 * loadable apache modules

/home/ensembl_usr/local/apache2/bin/httpd -d /home/ensembl_usr/local/ensembl/

If you got any complains about mod_perl, mod_deflate, mod_headers, mod_expires, one possibility is to link these from our apache2/modules to ensembl/modules directory: cd /home/ensembl_usr/local/ensembl_2010.05.08 ln -s /home/ensembl_usr/local/apache2/modules/*.so.

when starting "/home/ensembl_usr/local/apache2/bin/httpd -d /home/ensembl_usr/local/ensembl/" you should have:

Starting up Ensembl server on host: myhost.mydomain.org

defined(%hash) is deprecated at /home/ensembl_usr /local/lib/perl5/site_perl/5.12.0/LWP/Parallel/UserAgent.pm line 1327. (Maybe you should just omit the defined?) defined(%hash) is deprecated at /home/ensembl_usr/local/lib/perl5/site_perl/5.12.0/SOAP/Lite.pm line 465. (Maybe you should just omit the defined?) defined(%hash) is deprecated at /home/ensembl_usr/local/lib/perl5/site_perl/5.12.0/SOAP/Lite.pm line 2203. (Maybe you should just omit the defined?) -- - INFO :    0.000 :    0.000 : Parser     >> Starting to parse tree -- - INFO :    0.010 :    0.010 : Parser     >> Child objects attached - INFO :    0.011 :    0.001 : Filesystem >> Trawled web tree -- - INFO :    0.011 :    0.000 : Parser     >> Parsing ini files and munging dbs -- - INFO :    0.018 :    0.006 : Parsing    >> DEFAULTS ini file - INFO :    0.019 :    0.001 : Parsing    >> Anolis_carolinensis ini file [WARN] Can't connect to DATABASE_WEBSITE [WARN] DBI connect('database=ensembl_website_57;host=mysql.mydomain.org; DB server hostname/IP address;port=3306','mysqluser',...) failed: Unknown MySQL server host 'mysql.mydomain.org' (2) at /home/dkedra/local/ensembl/modules/EnsEMBL/Web/ConfigPacker_base.pm line 91

It does not complain about missing modules here, so lets hope everything to the point of getting MySQL database connection looks OK.

Step 2
There is info about mysql access to ENSEMBL databased @ensembl.org: http://www.ensembl.org/info/data/mysql.html The "mysql.mydomain.org" is defined in just one place: find. -type f | xargs grep -s mysql.mydomain.org
 * go to the ensembl directory
 * 1) ./conf/ini-files/DEFAULTS.ini:


 * edit ./conf/ini-files/DEFAULTS.ini:

comment out original lines:
 * DATABASE_HOST                 = mysql.mydomain.org; DB server hostname/IP address
 * DATABASE_HOST_PORT            = 3306               ;DB server TCP/IP port
 * DATABASE_DBUSER               = mysqluser    ;DB read-only user
 * DATABASE_DBPASS               =              ;DB read-only password

paste this & save DATABASE_HOST                 = ensembldb.ensembl.org ; DB server hostname/IP address DATABASE_HOST_PORT            = 5306                  ;DB server TCP/IP port DATABASE_DBUSER               = anonymous    ;DB read-only user DATABASE_DBPASS               =              ;DB read-only password
 * 1) modified on 2010-05-08 by some-user

Nope: "Can't locate MIME/Types.pm in @INC"
 * check if it works: "/home/ensembl_usr/local/apache2/bin/httpd -d /home/ensembl_usr/local/ensembl/":

perl -MCPAN -e shell install MIME::Types #MIME-Types-1.29.tar.gz
 * Install missing module:


 * check it again: "/home/ensembl_usr/local/apache2/bin/httpd -d /home/ensembl_usr/local/ensembl/":

ENSEMBL_BLAST_METHODS config unavailable at /home/ensembl_usr/local/ensembl/modules/EnsEMBL/Web/BlastView/BlastDefs.pm line 66. (13)Permission denied: make_sock: could not bind to address [::]:80 (13)Permission denied: make_sock: could not bind to address 0.0.0.0:80 no listening sockets available, shutting down Unable to open logs

This is a good result: stuff to be tweaked are default port for apache2 and permissions in ./log directory. ENSEMBL_BLAST_METHODS is just icing on the cake, so we will leave it for later.

Step 3
still in /home/ensembl_usr/local/ensembl
 * log permissions

This is a temporary change. In the end if things go right we will not be running apache2 under our user name on port 8080. So these are temporary fixes in a testing phase. mkdir logs

edit ./conf/SiteDefs.pm
 * changing port (from 80 to 8080)

change line: ENSEMBL_PORT          = 80; to: ENSEMBL_PORT          = 8080;

/home/ensembl_usr/local/apache2/bin/httpd -d /home/ensembl_usr/local/ensembl/
 * check if it works:

gives:

=
================================================================= Server information:

=
=================================================================

Ensembl version:   57

External address:  http://www.mydomain.org:8080/ Real address:      http://your.workstation.org:8080

Cache namespace:   http://www.mydomain.org:8080

Server root:       /home/ensembl_usr/local/ensembl

PID file:          /home/ensembl_usr/local/ensembl/logs/bee.crg.es.httpd.pid Access logs:       /home/ensembl_usr/local/ensembl/logs/bee.crg.es.access_log ensembl_extended Error logs:        /home/ensembl_usr/local/ensembl/logs/bee.crg.es.error_log

CSS file:          http://www.mydomain.org:8080/minified/b6ca3ca4ce26fe395240a66f0e285ee4.css JavaScript file:   http://www.mydomain.org:8080/minified/ba1ee50aa5b6d1e173d240a25b2f560c.js

Plugins installed: EnsEMBL::Mirror EnsEMBL::Ensembl

=
=================================================================

ENSEMBL_BLAST_METHODS config unavailable at /home/ensembl_usr/local/ensembl/modules/EnsEMBL/Web/BlastView/BlastDefs.pm line 66.

Not perfect (www.mydomain.org etc.) but we can check how it looks in the browser. http://localhost:8080

OK, we got something ENSEMBL-like, but there are no species to look at. When clicking at human icon (or go to: http://localhost:8080/Homo_sapiens/Info/Index ) We get nice error page with: "Can't locate GD/Text.pm" Back to Perl modules

perl -MCPAN -e shell
 * install missing GD::Text

install GD::Text exit

killall -9 /home/ensembl_usr/local/apache2/bin/httpd

"/home/ensembl_usr/local/apache2/bin/httpd -d /home/ensembl_usr/local/ensembl/" browser to: http://localhost:8080 then http://localhost:8080/Homo_sapiens/Info/Index
 * Check the progress again:

Novel error: DBI connect('ensembl_web_user_db:ensembldb.ensembl.org:5306','',...) failed: Access denied for user ... 'your_user_name'@'your_IP_number' (using password: NO) at /home/ensembl_usr/local/lib/perl5/site_perl/5.12.0/Ima/DBI.pm

So while "anonymous" in ./conf/ini-files/DEFAULTS.ini did the job for accessing database=ensembl_website_57 @ensembldb.ensembl.org:5306 during startup this will not do here.

Step 4
Configuring DB access again. Tomorrow.

fonts
ttf-mscorefonts-installer ttf-liberation