=============
Upgrading AEN
=============

.. contents::
   :local:
   :depth: 1

CAUTION: These instructions are for upgrading AEN to the
current version 4.2.2 from 4.2.1 ONLY. Each version must be
upgraded iteratively from the previous version. Do not skip versions.

Upgrade instructions for previous versions:

* :doc:`AEN 4.2.1 upgrade instructions </ae-notebooks/4.2.1/admin-guide/install/upgrade>`

* :doc:`AEN 4.2.0 upgrade instructions </ae-notebooks/4.2.0/admin-guide/install/upgrade>`

* :doc:`AEN 4.1.3 upgrade instructions </ae-notebooks/4.1.3/admin-guide/install/upgrade>`

* :doc:`AEN 4.1.2 upgrade instructions </ae-notebooks/4.1.2/install/installation_update>`

* :doc:`AEN 4.1.1 upgrade instructions </ae-notebooks/4.1.1/install/installation_update>`.

* :doc:`AEN 4.1.0 upgrade instructions </ae-notebooks/4.1.0/install/installation_update>`.

* :doc:`AEN 4.0.0 upgrade instructions </ae-notebooks/4.0/install/installation_update>`.

For upgrades from versions before those listed above, please
contact your enterprise support representative.

NOTE: Named Service Account functionality is available with AEN 4.0.0+ for new
installations only. It is not available for upgraded installations. Contact
your enterprise support representative for more information.

An AEN platform update requires that each instance of the 3 node
types be upgraded individually:

* AEN Server
* AEN Gateway
* AEN Compute

The upgrade process requires that all AEN service instances be
stopped, upgraded, and then restarted.

NOTE: Any commands that call for the root user can also be done
using sudo.

If you encounter any difficulty during the upgrade process, see
:doc:`../troubleshooting` which provides guidance on:

* processes
* configuration files
* log files
* ports

If you are unable to resolve an installation or upgrade
problem, please contact your enterprise support representative.

.. _aen-installation-update-preflight:

Before you upgrade
==================

CAUTION: Make a tested backup of your installation before starting the upgrade.
Upgrading to a higher version of AEN is not reversible. Any errors during the
upgrade procedure may result in partial or complete data loss and require
restoring data from backups.

CAUTION: Terminate all AEN applications and stop all projects
before starting the upgrade process.

Before upgrading each service on each host:

#. Suspend the services on each of the nodes:

   .. code-block:: bash

      sudo service wakari-server stop
      sudo service wakari-gateway stop
      sudo service wakari-compute stop

#. Set the AEN Functional ID ("NFI") and AEN Functional Group ("NFG")
   to the NFI and NFG of the current installation:

   .. code-block:: bash

      export AEN_SRVC_ACCT="wakari"
      export AEN_SRVC_GRP="wakari"

   NOTE: The default NFI is ``wakari``, but ``aen_admin`` or any other
   name may be used instead.

   For more information on NFI and NFG, see the
   :ref:`installation instructions <aen-functional-id>`.

#. Install ``wget``::

       yum install wget


.. _aen-server:

Upgrading the AEN server node
=============================

NOTE: If you are using LDAP-based authentication, back up the
``/opt/wakari/wakari-server/etc/wakari/wk-server-config.json``
configuration file. After the server has been upgraded, copy that
file back into the same location as before the upgrade.

Complete the following steps on the server host:

#. Stop the Elasticsearch service:

   .. code-block:: bash

      sudo service elasticsearch stop

#. Remove any previous index:

   .. code-block:: bash

      sudo rm -rf /var/lib/elasticsearch/*

   NOTE: You can choose to keep the old index, but if you detect
   any issues with the search capabilities after the upgrade, you
   will need to run the following to start with a clean index:

   .. code-block:: bash

      sudo service wakari-server stop
      sudo service elasticsearch stop
      sudo rm -rf /var/lib/elasticsearch/*
      sudo service elasticsearch start
      sudo service wakari-server start

#. Upgrade the server:

   .. code-block:: bash

      pushd /tmp
      wget http://j.mp/aen-server-update-4_2_2

      sudo -E -u $AEN_SRVC_ACCT /opt/wakari/miniconda/bin/conda install \
                      -p /opt/wakari/wakari-server          \
                      --file aen-server-update-4_2_2

      sudo -E -u $AEN_SRVC_ACCT /opt/wakari/miniconda/bin/conda install \
                       -p /opt/wakari/wakari-server          \
                       --no-deps                             \
                       wakari-enterprise-server-conf-update=2.0.9
      popd

#. Start Elasticsearch:

   .. code-block:: bash

      sudo service elasticsearch start

   Or, if you do not want to use the search features, edit your
   server's ``/opt/wakari/wakari-server/etc/wakari/config.json``
   file by adding the line ``"SEARCH_ENABLED": false``.

#. Restart the *NGINX* server:

   AEN server version >= 4.1.3 uses Unix sockets for communication with NGINX.
   Restart NGINX to load this new configuration:

   .. code-block:: bash

      sudo service nginx restart

   Alternatively, you can restart NGINX with:

   .. code-block:: bash

       sudo nginx -s stop
       sudo nginx

#. Start the server:

   .. code-block:: bash

      sudo service wakari-server start

#. Check that the server is running properly:

   .. code-block:: bash

      sudo service wakari-server status

#. If you see NGINX errors, please check the configuration
   at ``/opt/wakari/wakari-server/etc/nginx/conf.d/www.enterprise.conf:18``.

#. Connect to AEN server using your web browser with the
   correct protocol (http or https), hostname and port number.

.. _aen-gw:

Upgrading the AEN gateway node
==============================

Complete the following steps on each gateway host:

#. Upgrade the gateway:

   .. code-block:: bash

      pushd /tmp
      wget http://j.mp/aen-gateway-update-4_2_2

      sudo -E -u $AEN_SRVC_ACCT /opt/wakari/miniconda/bin/conda install \
                      -p /opt/wakari/wakari-gateway         \
                      --file aen-gateway-update-4_2_2

      sudo -E -u $AEN_SRVC_ACCT /opt/wakari/miniconda/bin/conda install \
                      -p /opt/wakari/wakari-gateway         \
                      --no-deps                             \
                      wakari-enterprise-gateway-conf-update=2.0.9
      popd

#. Start the gateway:

   .. code-block:: bash

      sudo service wakari-gateway start

#. Check that the gateway is running properly:

   .. code-block:: bash

      sudo service wakari-gateway status

#. Connect to the gateway using your web browser with the correct
   http/https, hostname and port number.

.. _aen-compute:

Upgrading AEN compute nodes
===========================

Complete the following steps on each host where an AEN compute
service is running:

#. Check for any ``wakari-indexer`` processes running:

   .. code-block:: bash

      ps aux | grep wakari-indexer

   NOTE: If you stopped all the projects, you will not see any
   wakari-indexer processes running.

   Terminate any remaining ``wakari-indexer`` processes:

   .. code-block:: bash

      sudo killall wakari-indexer

   NOTE: The processes killed with ``killall`` are run by the
   $AEN_SRVC_ACCT user, so they can be killed as root with
   ``sudo killall`` or killed as the $AEN_SRVC_ACCT user with
   ``sudo -u $AEN_SRVC_ACCT killall``. Example commands show
   the ``sudo killall`` option.

#. Check for any AEN applications processes running---Workbench,
   Viewer, Terminal or Notebook:

   .. code-block:: bash

      ps aux | grep wk-app-gateone
      ps aux | grep wk-app-workbench
      ps aux | grep wk-app-viewer
      ps aux | grep wk-app-terminal
      ps aux | grep jupyter-notebook

   NOTE: If you stopped all the projects, you will not see any
   AEN app processes running.

   Terminate any remaining AEN application processes by
   running one or more of the following:

   .. code-block:: bash

      sudo killall wk-app-gateone
      sudo killall wk-app-workbench
      sudo killall wk-app-viewer
      sudo killall wk-app-terminal
      sudo killall jupyter-notebook

#. Verify the contents of ``/opt/wakari/anaconda/.condarc``.
   Modify it to contain the following entries, and possibly
   others if you customized the ``.condarc`` file.

   NOTE: Modify the file as the AEN_SRVC_ACCT user (or be sure to
   keep the same ownership).

   .. code-block:: yaml

      channels:
        - https://conda.anaconda.org/t/<TOKEN>/anaconda-nb-extensions
        - r
        - https://conda.anaconda.org/wakari
        - defaults

      create_default_packages:
        - anaconda-client
        - ipykernel

   NOTE: Contact your enterprise support representative to get
   your token for the Anaconda channel referenced above. Replace
   ``<TOKEN>`` with the actual token from your enterprise support
   representative.

#. Upgrade *Anaconda* in the root environment:

   .. code-block:: bash

      pushd /tmp
      wget http://j.mp/aen-anaconda-update-4_2_2

      sudo -E -u $AEN_SRVC_ACCT /opt/wakari/anaconda/bin/conda install \
                    -p /opt/wakari/anaconda              \
                    --file aen-anaconda-update-4_2_2
      popd

#. Upgrade each compute service:

   .. code-block:: bash

      pushd /tmp
      wget http://j.mp/aen-compute-update-4_2_2

      sudo -E -u $AEN_SRVC_ACCT /opt/wakari/anaconda/bin/conda install \
                      -p /opt/wakari/wakari-compute        \
                      --file aen-compute-update-4_2_2

      sudo -E -u $AEN_SRVC_ACCT /opt/wakari/anaconda/bin/conda install \
                      --no-deps                            \
                      -p /opt/wakari/wakari-compute        \
                      wakari-enterprise-compute-conf-update=2.0.13
      popd

   NOTE: When upgrading the wakari-compute environment, you may
   see ImportError warnings with some nbextensions. As long as
   the Validating message is OK, the ImportError warnings are
   harmless---a consequence of the post-link presence on those
   packages.

#. Initialize the root environment to prime the package cache:

   .. code-block:: bash

      sudo -E -u $AEN_SRVC_ACCT /opt/wakari/anaconda/bin/conda create \
                      -p /opt/wakari/testenv \
                      --clone root

#. Test the offline cloning step:

   .. code-block:: bash

      sudo -E -u $AEN_SRVC_ACCT /opt/wakari/anaconda/bin/conda create \
                      -p /opt/wakari/testenvoffline \
                      --clone root --offline

#. Remove the test environments:

   .. code-block:: bash

      sudo rm -rf /opt/wakari/testenv
      sudo rm -rf /opt/wakari/testenvoffline

#. Install necessary dependencies:

   NOTE: Skip this step if you already have these dependencies
   installed from previous installations.

   .. code-block:: bash

      sudo yum groupinstall "X Window System" -y
      sudo yum install git -y

   NOTE: If you don't want to install the whole X Window System,
   you must install the following packages to have R plotting
   support:

   .. code-block:: bash

      sudo yum install -y libXrender libXext libXdmcp libSM libICE libXt \
      dejavu-sans-fonts dejavu-serif-fonts dejavu-fonts-common \
      fontpackages-filesystem

#. Start the compute service:

   .. code-block:: bash

      sudo service wakari-compute start

#. Verify the compute service is running properly:

   .. code-block:: bash

      sudo service wakari-compute status

#. Restart the AEN Server with:

   .. code-block:: bash

      sudo service wakari-server restart

#. Repeat this upgrade procedure for all compute nodes in your Data Center.

After upgrading
===============

#. Restart the projects and start using AEN applications.

#. If you have a :doc:`customized default environment
   <config/set-up-default-project-env>`, you may choose to
   upgrade it depending on the needs of your users.

   Upgrade the customized default environment at
   ``/opt/wakari/anaconda/envs/default`` with the ``$AEN_SRVC_ACCT`` user:

   .. code-block:: bash

      pushd /tmp
      wget http://j.mp/aen-anaconda-update-4_2_2

      sudo -E -u $AEN_SRVC_ACCT /opt/wakari/anaconda/bin/conda install \
              -p /opt/wakari/anaconda/envs/default \
              --file aen-anaconda-update-4_2_2
      popd

   To upgrade the customized default environments for every user and
   every project at ``/projects/<USER>/<PROJECT>/envs/default``, run
   these commands for **every** user as that user:

   .. code-block:: bash

      pushd /tmp
      wget http://j.mp/aen-anaconda-update-4_2_2

      sudo -E -u <USER> /opt/wakari/anaconda/bin/conda install \
              -p /projects/<USER>/<PROJECT>/envs/default \
              --file aen-anaconda-update-4_2_2
      popd

   NOTE: Replace ``<USER>`` with the user's name. Replace ``<PROJECT>`` with
   the project name.

   NOTE: Upgrading the default environment at
   ``/opt/wakari/anaconda/envs/default`` does NOT automatically
   upgrade the default environment in the users pre-existing
   projects. For pre-existing projects, the upgrade, if
   requested, should be done on a per-user basis.

   NOTE: These commands update packages listed in ``aen-anaconda-update-4_2_2``
   and do not update any other package.

#. If you did not stop all your projects before upgrading, then
   the first time you start an application you will see an error
   page requesting that you restart the application.

#. Restart the application to complete the upgrade.

#. If you still see old applications or icons after restart, reload
   the page to reset the browser cache.
