Logiciels sur les serveurs

Un article de Informaticiens département des sciences de la Terre et l'atmosphère
Version depuis le 29 d'avril 2020 à 14:29 par Michel (Discuter | changes)
Aller à: navigation, charcher

en construction(cette page s'adresse principalement aux développeurs)

under construction(this page is mainly for developers)

serveurs

  • CentOS 6 / RHEL 6
    • st2
    • beluga
  • CentOS 7
    • robert
    • zwack
    • snow
    • glacier
  • Centos 8
    • asterix.scta.uqam.ca
  • Ubuntu 18.04
    • st1

usage mémoire

# usager, process id, resident set size, trié du plus encombrant au plus léger
ps -eo user,pid,rss --sort=-rss
# usager, process id, resident set size, trié par usager, puis du plus encombrant au plus léger
ps -eo user,pid,rss --sort=-user,-rss

iptables

Ubuntu 18.04

- un fichier avec les regles 

/etc/default/iptables

- un script pour systemd 

# cat /etc/network/iptables
#!/bin/bash
if [[ "$1" == stop ]] ; then
  echo "Stopping Firewall Rules..."
  echo "Stopping Firewall Rules..." >/tmp/iptables.log
  cat /etc/default/iptables-stop | iptables-restore
else
  echo "Loading Firewall Rules..."
  echo "Loading Firewall Rules..." >/tmp/iptables.log
  cat /etc/default/iptables | iptables-restore
fi

- un fichier pour definir le service 

# cat /etc/systemd/system/iptables.service
[Unit]
Description = Apply base firewall rules for router functionality  

[Service]
Type=oneshot
RemainAfterExit=yes
ExecStart=/etc/network/iptables
ExecStop=/etc/network/iptables stop

[Install]
WantedBy=network-pre.target

- activer le service 

systemctl enable iptables

- partir/repartir/arreter le service 

systemctl start iptables
systemctl restart iptables
systemctl stop iptables

Centos

- un fichier avec les regles 

/etc/sysconfig/iptables

- un script pour systemd 

# cat /etc/network/iptables
#!/bin/bash
if "$1" == stop  ; then
  echo "Stopping Firewall Rules..."
  echo "Stopping Firewall Rules..." >/tmp/iptables.log
  cat /etc/sysconfig/iptables-stop | iptables-restore
else
  echo "Loading Firewall Rules..."
  echo "Loading Firewall Rules..." >/tmp/iptables.log
  cat /etc/sysconfig/iptables | iptables-restore
fi

- le fichier de regles nulles 

# cat /etc/sysconfig/iptables-stop
*filter
:INPUT ACCEPT [0:0]
:FORWARD ACCEPT [0:0]
:OUTPUT ACCEPT [0:0]
COMMIT

- un fichier pour definir le service 

# cat /etc/systemd/system/local_firewall.service
[Unit]
Description = Apply base firewall rules for router functionality  

[Service]
Type=oneshot
RemainAfterExit=yes
ExecStart=/etc/network/iptables
ExecStop=/etc/network/iptables stop

[Install]
WantedBy=network-pre.target

- activer le service 

systemctl enable local_firewall

- partir/repartir/arreter le service 

systemctl start local_firewall
systemctl restart local_firewall
systemctl stop local_firewall

NOTE: pour une raison qui m'echappe, Centos ne semble pas aimer iptables comme nom de service, ce qui a provoque le changement de nom de iptables a local_firewall.

PBS

recette d'installation (comme root) (Centos 8 et 7)

logiciel dans /opt/PBS_6.1.3

fichiers de configuration/operation dans /opt/torque

repertoire de compilation /opt/PBS/torque 

mkdir /opt/torque /opt/PBS /opt/PBS_6.1.3    # make directories
cd /opt/PBS
git clone /sca/compilers_and_tools/sources/PBS/torque
cd torque
git checkout 6.1.3
./autogen.sh
export CFLAGS=-fpermissive
./configure --prefix=/opt/PBS_6.1.3 --with-server-home=/opt/torque --disable-gcc-warnings --without-tcl
make
make install
make packages
./torque.setup  username host.sca.uqam.ca
vi /opt/torque/server_priv/nodes  # host.sca.uqam.ca np=xx
. /etc/profile.d/torque.sh     # make commands available
pbsnodes -a     # check node availability
qmgr -c 'p s'   # print server configuration
qmgr -c 'set server keep_completed = 30'        # retain completed status for 30 seconds only
qterm           # terminate server in setup mode
killall trqauthd

# copy systemd service description files into the proper directory
cp contrib/systemd/pbs_sched.service  /usr/lib/systemd/system/
cp contrib/systemd/pbs_server.service /usr/lib/systemd/system/
cp contrib/systemd/pbs_mom.service    /usr/lib/systemd/system/
cp contrib/systemd/trqauthd.service   /usr/lib/systemd/system/

for i in trqauthd pbs_server pbs_mom pbs_sched ; do systemctl enable $i ; done   # enable services
for i in pbs_sched pbs_mom pbs_server trqauthd ; do systemctl stop   $i ; done   # stop them if running
for i in trqauthd pbs_server pbs_mom pbs_sched ; do systemctl start  $i ; done   # start them

current configuration (zwack) 

$ qmgr -c 'p s'
#
# Create queues and set their attributes.
#
#
# Create and define queue batch
#
create queue batch
set queue batch queue_type = Execution
set queue batch resources_default.walltime = 01:00:00
set queue batch resources_default.nodes = 1
set queue batch enabled = True
set queue batch started = True
#
# Set server attributes.
#
set server scheduling = True
set server acl_hosts = zwack.sca.uqam.ca
set server managers = toupin@zwack.sca.uqam.ca
set server managers += valin@zwack.sca.uqam.ca
set server managers += winger@zwack.sca.uqam.ca
set server operators = toupin@zwack.sca.uqam.ca
set server operators += valin@zwack.sca.uqam.ca
set server operators += winger@zwack.sca.uqam.ca
set server default_queue = batch
set server log_events = 2047
set server mail_from = adm
set server node_check_rate = 150
set server tcp_timeout = 300
set server job_stat_rate = 300
set server poll_jobs = True
set server down_on_error = True
set server mom_job_sync = True
set server keep_completed = 30
set server next_job_number = 4
set server moab_array_compatible = True
set server nppcu = 1
set server timeout_for_job_delete = 120
set server timeout_for_job_requeue = 120
set server note_append_on_error = True

/sca

  • /sca/modulefiles (modules pour GNU modules)
  • /sca/compilers_and_tools (compilateurs et logiciels utilitaires installés avec GNU modules)
    • ifort/icc (Intel compilers)
      • 16.0.1 + openmpi-2.1.5
      • 18.0.5 + openmpi-2.1.5
      • 19.0.1 + openmpi-2.1.5
    • gfortran/gcc (GNU compilers)
      • 7.3.0 + openmpi-2.1.5
    • pgf90/pgcc (PGI compilers)
    • flang/clang
      • 6.0.1
    • netcdf
    • cdo, ncview, nco
export PREFIX=/sca/compilers_and_tools/netcdf  # utilise pour netcdf-dev
export LD_LIBRARY_PATH=$PREFIX/lib             # utilise pour netcdf-dev
export PREFIX=/sca/compilers_and_tools/cdo     # utilise pour netcdf-utils
export CPPFLAGS=-I$PREFIX/include
export LDFLAGS=-L$PREFIX/lib
export PREFIX=/sca/compilers_and_tools/netcdf-f/$COMP_ARCH  # utilise pour netcdf-dev-fortran

libraries for cdo as found on cdo site
/sca/compilers_and_tools/sources/libs4cdo-0.0.11.tar.xz
(using ./configure ... ; make ; make check ; make install )

szip functionality
aec 1.0.2+patch /sca/compilers_and_tools/sources/libaec-1.0.2.tar.gz  (alternative 1)
./configure --prefix=$PREFIX
szip 2.1 libs4cdo-0.0.11/szip-2.1  (alternative 2, utilise pour netcdf-utils)
./configure --prefix=$PREFIX --enable-production --enable-encoding

udunits2 2.2.26 /sca/compilers_and_tools/sources/udunits-2.2.26.tar.gz
# must add #define XML_STATUS_OK    1 after #include "expat.h" in lib/xml.c
./configure --prefix=$PREFIX --with-pic=yes

export CPPFLAGS=$PREFIX/include
export LDFLAGS=$PREFIX/lib

hdf5 1.8.21 /sca/compilers_and_tools/sources/hdf5-1.8.21.tar.bz2 (utilise pour netcdf-dev)
hdf5 1.8.15 libs4cdo-0.0.11/hdf5-1.8.15 (utilise pour netcdf-utils)
#./configure --prefix=$PREFIX --enable-shared --with-szlib --enable-hl --disable-cxx --disable-fortran --enable-threadsafe --with-pthread --with-zlib
#./configure --prefix=$PREFIX --enable-shared --with-szlib --enable-hl --disable-cxx --disable-fortran --with-zlib #  utilise pour netcdf-dev
./configure --prefix=$PREFIX --enable-shared --with-szlib=$PREFIX --disable-cxx --disable-fortran --enable-threadsafe --with-pthread --with-zlib # utilise pour netcdf-utils

netcdf 4.3.3.1 libs4cdo-0.0.11/netcdf-4.3.3.1 (utilise pour netcdf-utils)
netcdf 4.6.1 /sca/compilers_and_tools/sources/netcdf-c-v4.6.1.tar.gz (utilise pour netcdf-dev)
./configure --prefix=$PREFIX --enable-shared --enable-netcdf-4

netcdf-fortran 4.4.4 /sca/compilers_and_tools/sources/netcdf-fortran-v4.4.4.tar.gz (utilise pour netcdf-dev)
export FC=ifort/gfortran/flang/pgf90
export FCFLAGS=-O2
export CFLAGS=-O2
export CPPFLAGS="$(nc-config --cflags)"
export LDFLAGS="$(nc-config --libs)"
./configure --prefix=$PREFIX   # needs :  module load development/netcdf-c and module load compiler/xxxxx

jasper 1.900.29 /sca/compilers_and_tools/sources/jasper-1.900.29.tar.gz
jasper 1.900.1 libs4cdo-0.0.11/jasper-1.900.1 (utilise pour netcdf-utils)
./configure --prefix=$PREFIX --enable-shared

grib_api 1.13.1 /sca/compilers_and_tools/sources/grib_api-1.13.1.tar.gz libs4cdo-0.0.11/grib_api-1.13.1
./configure --prefix=$PREFIX --with-jasper=$PREFIX --disable-fortran

proj 4.8.0 libs4cdo-0.0.11/proj-4.8.0 (utilise pour netcdf-utils)
./configure --prefix=$PREFIX --without-mutex --without-jni

cdo 1.9.5 /sca/compilers_and_tools/sources/cdo-1.9.5.tar.gz (module load compiler/gcc-7.3)
./configure --prefix=$PREFIX --with-szlib --with-netcdf --with-proj --with-grib_api --with-udunits2 --with-threads --disable-fortran

libpng 1.15.30 /sca/compilers_and_tools/sources/libpng-1.5.30.tar.xz (utilise pour netcdf-utils)
./configure --prefix=$PREFIX --with-pic

ncview 2.1.7 /sca/compilers_and_tools/sources/ncview-2.1.7.tar.gz (will need to get liggsl source for all platforms version)
./configure --prefix=$PREFIX 

libgsl 1.16 /sca/compilers_and_tools/sources/gsl-1.16.tar.gz (utilise pour netcdf-utils)
./configure --prefix=$PREFIX --with-pic

antlr 2.7.7 /sca/compilers_and_tools/sources/antlr-2.7.7.tar.gz (utilise pour netcdf-utils)
./configure --prefix=$PREFIX

nco 4.7.8 /sca/compilers_and_tools/sources/nco-4.7.8.tar.gz (utilise pour netcdf-utils)
./configure --prefix=$PREFIX --with-pic

eccodes-2.13.0 /sca/compilers_and_tools/sources/eccodes-2.13.0-Source.tar.gz
module load utils/misc development/netcdf-c  # cmake recent + netcdf developpement
cmake -DCMAKE_INSTALL_PREFIX=/sca/compilers_and_tools/eccodes -DCMAKE_BUILD_TYPE=Release \
      -DNETCDF_PATH=/sca/compilers_and_tools/netcdf -DJASPER_PATH=/sca/compilers_and_tools/cdo \
      -DENABLE_AEC=ON -DAEC_PATH=/sca/compilers_and_tools/netcdf \
      -DENABLE_PYTHON=OFF \
      ../eccodes-2.13.0-Source
  • /sca/armnssm (environnement logiciel pour les modèles, accès avec l'utilitaire ssm)

GNU modules

[| accès aux logiciels]

[| environnement python]

netcdf et al.

accès aux programmathèques netcdf (gnu gcc/gfortran) 

module load development/netcdf-c
module load development/netcdf-f/gfortran

accès a netcdf (intel version 16) 

module load development/netcdf-c
module load development/netcdf-f/intel1600

accès aux outils netcdf 

module load utils/cdo
ou
module load utils/ncview
ou
module load utils/netcdf

conda / anaconda

commandes utiles:

module load python2/miniconda2            # conda + python 2.7 /sca/compilers_and_tools/python/miniconda2
module load python3/miniconda3            # conda + python 3.7 /sca/compilers_and_tools/python/miniconda3

(pour les commandes conda, il est recommandé de faire : cd /where/conda/is/installed)
(CondaFileIOError .... contains unsafe path:  .... )
conda info --envs                         # lister les environnements
conda list -n nom_d_environnement         # contenu d'un environnement
conda create --name nom_d_environnement   # creer un environnement
conda remove --name nom_d_environnement   # effacer un environnement
conda search 'pattern'                    # trouver les packages selon 'pattern'
conda install package[=version[=tag]]     # installer un package

source activate nom_d_environnement       # activer (utiliser) un environnement
source deactivate                         # sortir d'un environnement

environnements disponibles avec python2

source activate jupyter        # jupyter notebooks
source activate magics         # magics (ECMWF)
source activate mysql2
source activate pynngl2        # NCAR Graphics (PyNGL et NCL)
source activate spyder         # environnement spyder

environnements disponibles avec python3

source activate jupyter        # jupyter notebooks
source activate spyder         # environnement spyder
source activate pynngl3        # NCAR Graphics (PyNGL et NCL)

PERCCLI

reference guide for perccli

example commands 

perccli64 /c0 show all          # show all info about controller 0
perccli64 /c0/e32/s3 set jbod   # controller 0 enclosure 32 slot 3 set as jbod

VOLUMES

pvcreate /dev/xxx
vgcreate pv_name /dev/xxx
lvcreate -l 100%FREE -n lv_name vg_name
Pris de "http://wikiscta.scta.uqam.ca/wikiscta/index.php?title=Logiciels_sur_les_serveurs&oldid=3557"