diff --git a/CkanUbuntuTomcatInstall.md b/CkanUbuntuTomcatInstall.md new file mode 100644 index 0000000000000000000000000000000000000000..7543258018de14f68892c48cefa22cb5f8d6132a --- /dev/null +++ b/CkanUbuntuTomcatInstall.md @@ -0,0 +1,239 @@ +# CKAN with Solr-Tomcat installation notes on Ubuntu 14 + +### Reference + * The [CKAN installation from sources guide](http://docs.ckan.org/en/latest/maintaining/installing/install-from-source.html). + * Installing [solr-tomcat on Ubuntu](https://drupal.stackexchange.com/questions/95897/apache-solr-4-6-0-installation-and-compatibility-with-drupal7/124038#124038) + +### Prepare local server context +Configure ssh for login: consists in setting up ad-hoc entry in your +`~/.ssh/config`. The `ssh ckan` command should now be effective. + +Edit `/etc/group` to allow ssh_user to sudo. The sudo entry of `/etc/group` should look like `sudo:x:27:liris,ssh_user`. + +This is indeed Ubuntu version 14.04 as shown by `lsb_release -a` that yields +``` +No LSB modules are available. +Distributor ID: Ubuntu +Description: Ubuntu 14.04.3 LTS +Release: 14.04 +Codename: trusty +``` + +When encountering problems with messages of the form `locale: Cannot set LC_ALL to default locale: No such file or directory` consider looking at this [local problem on cloud images](http://askubuntu.com/questions/454260/how-to-solve-locale-problem) + +### Starting with solr-tomcat +Following [installing solr-tomcat on Ubuntu](https://drupal.stackexchange.com/questions/95897/apache-solr-4-6-0-installation-and-compatibility-with-drupal7/124038#124038) +``` +sudo apt-get install solr-tomcat +``` +that pulls `openjdk-7-jre-headless:amd64` (as displayed by `dpkg -l | grep openjdk`). Now assert that `solr` is properly running +``` +sudo apt-get install lynx +lynx http://localhost:8080/solr +``` +Note: in case of despair with installing `solr-tomcat` this [link might provide useful](https://drupal.stackexchange.com/questions/95897/apache-solr-4-6-0-installation-and-compatibility-with-drupal7/124038#124038). + +### Deploy CKAN +We now follow [CKAN installation from sources guide](http://docs.ckan.org/en/latest/maintaining/installing/install-from-source.html). The numbering of the following list follows the numbering of that install doc... + +#### 1. Install the required packages +``` +sudo apt-get install python-dev postgresql libpq-dev python-pip python-virtualenv git-core +``` +Note: `solr-jetty` is not thrown in (since we chose to install `solr-tomcat`) nor is `openjdk-6-jdk` (since version 7 was already pulled by `solr-tomcat`). +The following is not really mandatory. Simply it locates `ckan` in `ssh_user` home dir: +``` +cd +mkdir -p ~/ckan/lib +sudo ln -s ~/ckan/lib /usr/lib/ckan +mkdir -p ~/ckan/etc +sudo ln -s ~/ckan/etc /etc/ckan +``` + +#### 2. Create the ad-hoc python virtual environment +``` +sudo mkdir -p /usr/lib/ckan/default +sudo chown `whoami` /usr/lib/ckan/default +virtualenv --no-site-packages /usr/lib/ckan/default +. /usr/lib/ckan/default/bin/activate +``` +WARNING: from now on always **make sure this python virtual environment is always active.** One can make sure by checking the bash prompt that should be prefixed with `(default)`. +Install CKAN's current stable version and the required modules +``` +pip install -e 'git+https://github.com/ckan/ckan.git@ckan-2.4.1#egg=ckan' +pip install -r /usr/lib/ckan/default/src/ckan/requirements.txt +``` +which terminates with a complaint about [some invalid syntax on line 320 of file python2.7/site-packages/pyutilib/component/core/core3.py](https://gist.github.com/gregelin/6662042#file-chef-ckan-install-error-2-info-sh-L495) + +Just making sure the `virtualenv` is ok: +``` +deactivate +. /usr/lib/ckan/default/bin/activate +``` + +#### 3. Setup a PostgreSQL database +Create the PostgreSQL database **`user`** named `ckan_default` +``` +sudo -u postgres createuser -S -D -R -P ckan_default +``` +Create the PostgreSQL **`database`** named `ckan_default` +``` +sudo -u postgres createdb -O ckan_default ckan_default -E utf-8 +``` + +#### 4. Create a CKAN config file +``` +sudo mkdir -p /etc/ckan/default +sudo chown -R `whoami` /etc/ckan/ +cd /usr/lib/ckan/default/src/ckan +paster make-config ckan /etc/ckan/default/development.ini +``` +Customize ckan by editing `development.ini` file +``` +sudo vim /etc/ckan/default/development.ini + ---> sqlalchemy.url : change the "pass" string for the password of ckan_default user + ---> ckan.site_id = default (i.e. unchanged) + ---> ckan.site_url = http://liris-vm-32.insa-lyon.fr +``` + +#### 5. Setup solr +Refer above to the `solr-tomcat` section. Then configure the schema.xml to point to `ckan`'s one: +``` +cd /etc/solr/conf +sudo mv schema.xml schema.xml.bak +sudo ln -s /usr/lib/ckan/default/src/ckan/ckan/config/solr/schema.xml . +``` +and restart `solr` (through `tomcat`) and assert all is well +``` +sudo service tomcat6 restart +lynx http://localhost:8080/solr +``` +Note: the [installing `solr-tomcat` on Ubuntu](https://drupal.stackexchange.com/questions/95897/apache-solr-4-6-0-installation-and-compatibility-with-drupal7/124038#124038) documentation recommends to remove `/usr/share/solr/data/` (which is actually a symbolic link and not a directory). This was not taken into account for this install. +Change the `solr_url` setting in the CKAN config file to point to your Solr server (note that the port is 8080 and not 8983): +``` +sudo vim /etc/ckan/default/development.ini + ---> solr_url=http://127.0.0.1:8080/solr +``` + +#### 6. Create database tables +``` +cd /usr/lib/ckan/default/src/ckan +paster db init -c /etc/ckan/default/development.ini +``` + +#### 7. Set up the DataStore +Skip this step and first finish the install of CKAN. Then resume with this step (see step 10. below) afterwards otherwise you can't test that the datastore is ok (starting CKAN will complain with `No such file or directory: /etc/ckan/default/who.ini` ) + +#### 8. Link to who.ini +``` +ln -s /usr/lib/ckan/default/src/ckan/who.ini /etc/ckan/default/who.ini +``` + +#### 9. You're done (launch CKAN) +Manual launching is done with +``` +. /usr/lib/ckan/default/bin/activate +cd /usr/lib/ckan/default/src/ckan +paster serve /etc/ckan/default/development.ini +``` +In another terminal (same server host) assert that CKAN is running with +`lynx http://127.0.0.1:5000/``. +Assert there are no firewall hindrances by poking from another client host with e.g. +``` +firefox http://liris-vm-32.insa-lyon.fr:5000/ +``` + +#### 10. (Resuming with step 7:) Set up the DataStore +Follow the instructions given [here](http://docs.ckan.org/en/latest/maintaining/installing/install-from-source.html) which can be summarized as follows +``` +sudo sed -i "s/ckan.plugins =/ckan.plugins = datastore/" /etc/ckan/default/development.ini +sudo -u postgres createuser -S -D -R -P -l datastore_default +sudo -u postgres createdb -O ckan_default datastore_default -E utf-8 +``` +``` +sudo vim /etc/ckan/default/development.ini + ---> ckan.datastore.write_url = postgresql://ckan_default:pass@localhost/datastore_default (change the "pass" string for the password of ckan_default user) + ---> ckan.datastore.read_url = postgresql://datastore_default:pass@localhost/datastore_default (change the "pass" string for the password of datastore_default user) +``` +``` +paster --plugin=ckan datastore set-permissions -c /etc/ckan/default/development.ini | sudo -u postgres psql --set ON_ERROR_STOP=1 +``` +Relaunch CKAN: refer to the above step 9. + +#### 11. Test CKAN +Instructions are [here](http://docs.ckan.org/en/latest/contributing/test.html). +Just in case you left python's `virtualenv` +``` +. /usr/lib/ckan/default/bin/activate +``` +Install developer dependencies: +``` +pip install -r /usr/lib/ckan/default/src/ckan/dev-requirements.txt +``` +Create test databases: +``` +sudo -u postgres createdb -O ckan_default ckan_test -E utf-8 +sudo -u postgres createdb -O ckan_default datastore_test -E utf-8 +``` +Before running the `test-core.ini` you need to edit it in order setup all the access passwords according to what is done in `development.ini` i.e. mainly the lines shown by the +``` +grep .url /etc/ckan/default/development.ini | grep @ +``` +command: +``` +vim $HOME/ckan/lib/default/src/ckan/test-core.ini + ---> ckan.site_url = http://liris-vm-32.insa-lyon.fr + ---> solr_url=http://127.0.0.1:8080/solr + ---> sqlalchemy.url = ... + ---> ckan.datastore.write_url = ... + ---> ckan.datastore.read_url = ... +``` +``` +cd $HOME/ckan/lib/default/src/ckan/ +paster --plugin=ckan datastore set-permissions -c test-core.ini | grep -v "Genshi templates is deprecated" | sudo -u postgres psql +``` +Run the tests per se: +``` +nosetests --ckan --with-pylons=test-core.ini ckan ckanext +``` + +#### 12. Install ckan service +Create a `/etc/init.d/ckan` with the following (quite brutal) content +``` +#! /bin/sh +### Adapter from grub-common + +# Load the VERBOSE setting and other rcS variables +. /lib/init/vars.sh + +# Define LSB log_* functions. +# Depend on lsb-base (>= 3.0-6) to ensure that this file is present. +. /lib/lsb/init-functions + +case $1 in + start|restart|force-reload) + [ "$VERBOSE" != no ] && log_action_msg "Launching ckan" + cd /usr/lib/ckan/default/src/ckan + . /usr/lib/ckan/default/bin/activate + paster serve /etc/ckan/default/development.ini & + ;; + stop) + killall -r paster + ;; + status) + exit 0 + ;; + *) + echo "Usage: $0 {start|stop|status|restart|force-reload}" >&2 + exit 3 + ;; +esac + +exit 0 +``` + +``` +chmod a+x /etc/init.d/ckan +ln -s /etc/init.d/ckan /etc/rc3.d/S100ckan +ln -s /etc/init.d/ckan /etc/rc0.d/K100ckan +``` diff --git a/Readme.md b/Readme.md index 1fa0423d296823fa6d8e6f73e8f7d861e5ef65b9..ebe0242cecc4552344fdf3a371b357085aed4606 100644 --- a/Readme.md +++ b/Readme.md @@ -16,7 +16,9 @@ * [City oriented data Tools](/CityDataTool.md) ### Technologies: - * [[CKAN|CKAN]] + * CKAN: + * [Solr-Tomcat successful install from sources on Ubuntu](/CkanUbuntuTomcatInstall.md) + * [[CkanUbuntuJettyFailure|Solr-Jetty fails to install on Ubuntu]] * [[https://tech.knime.org/installation-0|Knime]] * Open source (GPL), German made * [[https://tech.knime.org/community/developers|Installation out of sources]]