Essun: Difference between revisions

From Wiki-Fou
 
(16 intermediate revisions by 2 users not shown)
Line 77: Line 77:


<code>mpv http://essun.calafou.org:8000</code>
<code>mpv http://essun.calafou.org:8000</code>
== Libros ==
Los libros en la antigua mediateca se pueden visualizar a traves de [https://libros.calafou.org/ calibre-web].
Se pueden subir libros desde el enlace anterior o a traves del [https://nuvol.calafou.org/index.php/apps/files?dir=/MEDIATECA/libros directorio mediateca en nuvol].
El indice y los metadatos de los libros se actualizan cada noche.
== Peliculas, Musica, Fotos... ==
Las peliculas, musica, fotos, etc.. se pueden visualizar a traves de [https://calaflix.calafou.org/ jellyfin] (calaflix).


== File server ==
== File server ==
Se puede acceder a tanto a los ficheros de la antigua mediateca a traves de [https://nuvol.calafou.org/index.php/apps/files/?dir=/MEDIATECA nuvol]
<!--
'''Update''': the FTP sever has been disabled.
=== Mediateca FTP anonymous access ===


<code>ftp://essun.calafou.org</code>
<code>ftp://essun.calafou.org</code>
Line 84: Line 105:
<code>systemctl status proftpd</code>
<code>systemctl status proftpd</code>


<code>/etc/proftpd/conf.d/anonftp</code>
The file server is for uploading and downloading files to / from the Mediateca.  You can access it with [https://filezillapro.com FileZilla] from popular operating systems.  Or you can use a command line client such as <code>ncftp</code>.
 
The file server is for uploading and downloading files.  You can access it with <code>FileZilla</code> from popular operating systems.  Or you can use a command line client such as <code>ncftp</code>.


<code>ncftp ftp://essun.calafou.org</code>
<code>ncftp ftp://essun.calafou.org</code>
Line 93: Line 112:


[[File:Ftpessun.png]]
[[File:Ftpessun.png]]
=== Essun SFTP access with a ssh key ===
To access Essun's Calafou files with a FTP client like Filezilla, you need a ssh pair of keys, to generate this pair of key under linux :
==== Create RSA Key Pair ====
You create your rsa key pair in the computer you want to log to the server
Be careful to pay attention from which user you create the rsa key pair in your local machine (computer). Since the rsa key pairs are saved in a hidden directory in my user home directory (~/.ssh), if trying to login from a different user, it will not work
So, from the user in my localmachine that I wish to use to access the server, I will create my rsa key pair. When generating the key, you will need to introduce a **"passphrase".**
$ ssh-keygen -t rsa -b 4096
By default it will generate two keys in ~/.ssh, a public key and a private key. The public key we will send to an admin of Anarcha server.
==== Send public key to the server ====
After generating an SSH key pair, you will want to copy your public key to the server.
Assuming you generated an SSH key pair using the previous step, use the following command at the terminal of your local machine to print your public key (id_rsa.pub):
cat ~/.ssh/id_rsa.pub
This should print your public SSH key, which should look something like the following:
ssh-rsa AAAB3NzaC1yc2EAAAADAQABAAABAQDBGTO0tsVejssuaYR5R3Y/i73SppJAhme1dH7W2c47d4gOqB4izP0+fRLfvbz/tnXFz4iOP/H6eCV05hqUhF+KYRxt9Y8tVMrpDZR2l75o6+xSbUOMu6xN+uVF0T9XzKcxmzTmnV7Na5up3QM3DoSRYX/EP3utr2+zAqpJIfKPLdA74w7g56oYWI9blpnpzxkEd3edVJOivUkpZ4JoenWManvIaSdMTJXMy3MtlQhva+j9CgguyVbUkdzK9KKEuah+pFZvaugtebsU+bllPTB0nlXGIJk98Ie9ZtxuY3nCKneB+KjKiXrAvXUPCI9mWkYS/1rggpFmu3HbXBnWSUdf localuser@machine.local
Select the public key, and copy it to your clipboard and send it by email to someone who administrate essun
==== Configure Filezilla to access Essun with SFTP and this key ====
* See this tutorial https://filezillapro.com/docs/v3/basic-usage-instructions/ssh-private-keys-for-sftp/
# In FileZilla Pro->Settings.. select Connection->SFTP
# Press the Add key file… button
# Press Command-Shift-G to bring up a path selection window and type “~/.ssh”
# Select the “id_rsa” key file and click Open (this imports the key)
# Click OK to close the Settings dialog
# Open File->Site Manager…
# Select the site with which you want to use the key
# Choose Protocol “SFTP” and select Logon Type “Normal”. Don’t worry about a password if you key file doesn’t have a password, if it has one set it here.
# Click Connect and you’ll see your files
-->


== NextCloud ==
== NextCloud ==
Line 107: Line 171:
* To integrate external storage as part of Nextcloud files
* To integrate external storage as part of Nextcloud files
** Possible to use the "Local Storage" application : activate it > then see as an admin in the settings the local storage
** Possible to use the "Local Storage" application : activate it > then see as an admin in the settings the local storage
** To allow people to put file with anonymous FTP + via Nextcloud : https://serverfault.com/questions/184548/ftp-and-apache-permission-issues
* To allow people to put file with anonymous FTP + via Nextcloud : https://serverfault.com/questions/184548/ftp-and-apache-permission-issues or more recently https://serverfault.com/questions/938143/proftpd-and-www-data-group
So :  
So it was :  
  groupadd www-ftp
  groupadd www-ftp
  usermod -a -G www-ftp www-data
  usermod -a -G www-ftp www-data
Line 114: Line 178:
  chown -R :www-ftp /mnt/3t/ftp/MEDIATECA
  chown -R :www-ftp /mnt/3t/ftp/MEDIATECA
  chmod -R g+rw /mnt/3t/ftp/MEDIATECA
  chmod -R g+rw /mnt/3t/ftp/MEDIATECA
Now it is config on proftpd side :
DefaultRoot /mnt/3t/ftp/MEDIATECA/
<Anonymous /mnt/3t/ftp/MEDIATECA/>
<Limit LOGIN>
    AllowAll
</Limit>
Group www-data
User www-data
UserAlias anonymous anonftp www-data
GroupOwner www-data
UserOwner www-data
Umask 007
<Directory *>
    <Limit ALL>
        AllowAll
    </Limit>
</Directory>
</Anonymous>
and on MEDIATECA side
chmod -R 0775 MEDIATECA/
chmod g+w+s MEDIATECA/
chown -R www-data: MEDIATECA/


=== Even more config to optimize things ===
=== Even more config to optimize things ===
Line 139: Line 228:
or more wild
or more wild
  TRUNCATE TABLE oc_file_locks;
  TRUNCATE TABLE oc_file_locks;
As of 07082021, if "Memcache \OC\Memcache\APCu not available for local cache (Is the matching PHP module installed and enabled?)" then :
  nano /etc/php/7.3/mods-available/apcu.ini
add this line :
  apc.enable_cli=1
restart apache2
  systemctl restart apache2
then this command shoud work
  root@essun:/var/www/html/nextcloud# sudo -u www-data php occ db:add-missing-indices
cf https://github.com/nextcloud/vm/issues/2039#issuecomment-876833425


== Set https certificate on each subdomain.calafou.org with a certbot wildcard ==
== Set https certificate on each subdomain.calafou.org with a certbot wildcard ==
Line 176: Line 275:
  crontab -e
  crontab -e
  0 0 * * 0 certbot renew -q -a certbot-plugin-gandi:dns --certbot-plugin-gandi:dns-credentials /etc/letsencrypt/gandi/gandi.ini --server https://acme-v02.api.letsencrypt.org/directory
  0 0 * * 0 certbot renew -q -a certbot-plugin-gandi:dns --certbot-plugin-gandi:dns-credentials /etc/letsencrypt/gandi/gandi.ini --server https://acme-v02.api.letsencrypt.org/directory
New method 2022 to create a certificate for a subdomain (as the previous one seems to cause trouble while renewing)
certbot certonly -d subdomain.calafou.org -a certbot-plugin-gandi:dns --certbot-plugin-gandi:dns-credentials /etc/letsencrypt/gandi/gandi.ini --server https://acme-v02.api.letsencrypt.org/directory


==LXC container por Libretime y umap ==
==LXC container por Libretime y umap ==
Line 181: Line 283:
=== Installation of the containers ===
=== Installation of the containers ===
* https://wiki.debian.org/LXC
* https://wiki.debian.org/LXC
* https://archives.flockport.com/lxc-networking-guide/
=== Use of lxcbr0 bridge to allow autostart ===
=== Use of lxcbr0 bridge to allow autostart ===


Line 228: Line 332:
* http://2tazasdelinux.blogspot.com/2015/11/moniitorizando-nuestro-sai-con-nut.html
* http://2tazasdelinux.blogspot.com/2015/11/moniitorizando-nuestro-sai-con-nut.html
* https://blog.shadypixel.com/monitoring-a-ups-with-nut-on-debian-or-ubuntu-linux/
* https://blog.shadypixel.com/monitoring-a-ups-with-nut-on-debian-or-ubuntu-linux/
= Essun backup =
Essun backup is being stored gpg-encrypted in nusse.
The software used is https://github.com/sukria/Backup-Manager/
We created the user `backup` both in essun and nusse to create/store the backups.
The user backup in essun has a gpg keyring with a public key.
The private key it is not in the servers and it's needed only to decrypt the backups.
There's only 1 configuration file, `/etc/backup-manager.conf`.
These are the changed lines:
    export BM_REPOSITORY_ROOT="/mnt/mirror/backup"
    export BM_REPOSITORY_USER="backup"
    export BM_REPOSITORY_GROUP="backup"
    export BM_ARCHIVE_METHOD="tarball-incremental"
    export BM_ENCRYPTION_METHOD="gpg"
    export BM_ENCRYPTION_RECIPIENT="264128C3E9B4F451C0105757350C60B958D846EC"
    export BM_TARBALL_DIRECTORIES="/mnt/mirror/calafou_actual"
    export BM_TARBALL_BLACKLIST="/mnt/mirror/backup"
    export BM_UPLOAD_METHOD="ssh-gpg"
    export BM_UPLOAD_HOSTS="172.31.17.202"
    export BM_UPLOAD_DESTINATION="/mnt/backup1T/backup"
    export BM_UPLOAD_SSH_USER="backup"
    export BM_UPLOAD_SSH_KEY="/mnt/mirror/backup/.ssh/id_ed25519"
    export BM_UPLOAD_SSH_HOSTS="172.31.17.202"
    export BM_UPLOAD_SSH_PORT="22"
    export BM_UPLOAD_SSH_DESTINATION="/mnt/backup1T/backup"
    export BM_UPLOAD_SSH_TTL="23"
    export BM_UPLOAD_SSHGPG_RECIPIENT="264128C3E9B4F451C0105757350C60B958D846EC"
It runs as often as configured in `/etc/cron.d/backup`
Also a pre-script is run to mount /var/www on the piweb server as a directory /mnt/ynh via sshfs to backup also the var/www folder on this small machine. For this a backupmirror user was created on the pi/yunohost machine to allow sshfs from essun via an authorized_key indicated in /etc/fstab in Essun

Latest revision as of 01:24, 21 October 2022

Essun

Essun is the mediaserver of Calafou local network. It is running Debian, the Universal Operating System. It is providing services for the people who are on the territory of Calafou. It is 8Gb ram and has 4 discs in it: 160Gb for system, 1Tb mirror discs, 3T for mediateca. You can connect here http://essun.calafou.org/

What does "Essun" mean?

Essun is one of the names of the protagonist of the Broken Earth trilogy by N. K. Jemisin. Essun belongs to a small group of people, called origenes, who have a special power to shape and redirect energy. Origenes are feared and hated by their society because of their powers, so if they are not murdered as children, they are carefully controlled, enslaved and exploited.

As the world ends, Essun is forced to join together with others. As she learns the strength of community and solidarity, she learns to use her powers without shame. And she learns that resistance is possible, even if it means causing the destruction of the world that had previously been her home.

Boot sequence: What happens when you turn on the machine?

  1. BIOS
    1. Here you can press F12 to choose boot device. The default is to boot the system on the hard drive. This is useful if you want to boot a rescue or install image from a pendrive, for example.
    2. Here you can press F2 to enter the BIOS configuration, but we have to find out the BIOS password. At the moment, we have no idea what is the BIOS password! We have to search how to reset this password...
  2. GRUB
    1. Here you can choose to start the system as usual, or to enter rescue mode. It is also possible to modify the kernel boot parameters. This is useful if you forgot the root password and want to reset it, for example.
  3. Initramfs
    1. Here cryptsetup asks for the LUKS encryption password. The password can be typed in using a keyboard connected to essun, or through SSH if your SSH keys are installed in initramfs (in /etc/dropbear-initramfs/authorized_keys cf this. For the second option, use the unlock-essun.sh script. The actual password can be found in the calafou keyringer repository, and you can see it from your computer like this: keyringer calafou decrypt lan.essun.luks.system
    2. To connect with ssh to ssh dropbear server at boot :
ssh root@172.31.17.23

once in busybox, type : 

cryptroot-unlock

and type the luks passphrase

  1. Systemd
    1. The hard drives ("block devices") specified in /etc/crypttab are unlocked, usually using key files in /etc/luks/.
    2. The hard drives ("block devices") specified in /etc/fstab are mounted, such as /mnt/3t (a big hard drive for mediateca) and /mnt/mirror (a mirrored drive for services).
    3. Systemd manages the services, so when the computer starts, it tries to turn on the web server (apache2), etc.
  2. System
    1. You can log in if your SSH key is installed in the system: ssh $USERNAME@essun.calafou.org (where $USERNAME is your username).
  3. Services to re-launch : (don't know why they doesn't start well at boot time)
systemctl restart etherpad-lite.service
systemctl restart apache2.service
systemctl status omnioussonification.service
  1. Container to start (as root)
lxc-start umap
lxc-start bibretime
lxc-ls -f

Services

Most of these services are only available from the local network!

Web Server

http://essun.calafou.org/

systemctl status apache2

/etc/apache2/sites-available/essun.calafou.org

The web server shows a page that explains Essum and links to the other services.

Pads

http://etherpad.calafou.org

systemctl status etherpad

/opt/etherpad-lite/settings.json

The etherpad is providing real-time multi-user editable shared persistent documents.

A few plug-ins have been installed from this framasoft list (all did not instal ...) https://github.com/YunoHost-Apps/etherpad_mypads_ynh

Streaming

http://essun.calafou.org:8000

systemctl status icecast2

/etc/icecast2/icecast.xml

Icecast2 is providing a stream ("radio") that you can play on your computer, for example with mplayer or mpv:

mplayer http://essun.calafou.org:8000

mpv http://essun.calafou.org:8000

Libros

Los libros en la antigua mediateca se pueden visualizar a traves de calibre-web.

Se pueden subir libros desde el enlace anterior o a traves del directorio mediateca en nuvol.

El indice y los metadatos de los libros se actualizan cada noche.

Peliculas, Musica, Fotos...

Las peliculas, musica, fotos, etc.. se pueden visualizar a traves de jellyfin (calaflix).

File server

Se puede acceder a tanto a los ficheros de la antigua mediateca a traves de nuvol


NextCloud

We could try to install it like this:

https://docs.nextcloud.com/server/17/admin_manual/installation/source_installation.html#example-installation-on-ubuntu-18-04-lts-server

We tried with this : https://www.howtoforge.com/tutorial/how-to-install-nextcloud-on-debian-10/ it seems to work (change the version of nextcloud to download)

More Nextcloud config

So it was : 

groupadd www-ftp
usermod -a -G www-ftp www-data
usermod -a -G www-ftp anonftp
chown -R :www-ftp /mnt/3t/ftp/MEDIATECA
chmod -R g+rw /mnt/3t/ftp/MEDIATECA

Now it is config on proftpd side : 

DefaultRoot /mnt/3t/ftp/MEDIATECA/
<Anonymous /mnt/3t/ftp/MEDIATECA/>
<Limit LOGIN>
    AllowAll
</Limit>
Group www-data
User www-data
UserAlias anonymous anonftp www-data
GroupOwner www-data
UserOwner www-data
Umask 007
<Directory *>
    <Limit ALL>
        AllowAll
    </Limit>
</Directory>
</Anonymous>

and on MEDIATECA side 

chmod -R 0775 MEDIATECA/ 
chmod g+w+s MEDIATECA/
chown -R www-data: MEDIATECA/

Even more config to optimize things

As seen in https://nuvol.calafou.org/index.php/settings/admin/overview there were things that needed to be optimized : 

cd /var/www/html/nextcloud
sudo -u www-data php occ db:add-missing-indices
apt install php-intl php-imagick
sudo -u www-data php occ maintenance:mode --on
sudo -u www-data php occ db:convert-filecache-bigint
sudo -u www-data php occ maintenance:mode --off

Activate cache 

apt install php-apcu
phpenmod intl
nano config/config.php

Add this in the file before ); 

'memcache.local' => '\OC\Memcache\APCu',

Restart Apache after all this 

systemctl restart apache2

Rescan the files 

sudo -u www-data php occ files:scan --all -v

If problem while scanning with unlocked files : 

mysql -u root -p
connect nextclouddb
DELETE FROM oc_file_locks WHERE 1;

or more wild 

TRUNCATE TABLE oc_file_locks;

As of 07082021, if "Memcache \OC\Memcache\APCu not available for local cache (Is the matching PHP module installed and enabled?)" then : 

 nano /etc/php/7.3/mods-available/apcu.ini

add this line : 

 apc.enable_cli=1

restart apache2 

 systemctl restart apache2

then this command shoud work 

 root@essun:/var/www/html/nextcloud# sudo -u www-data php occ db:add-missing-indices

cf https://github.com/nextcloud/vm/issues/2039#issuecomment-876833425

Set https certificate on each subdomain.calafou.org with a certbot wildcard

One certificate for all the subdomains

Install Gandi plug-in 

apt remove python-cryptography
pip install cryptography
pip install certbot-plugin-gandi
certbot certonly -a certbot-plugin-gandi:dns --certbot-plugin-gandi:dns-credentials gandi.ini -d \*.calafou.org --server https://acme-v02.api.letsencrypt.org/directory

Activate mod ssl for Apache 

a2enmod ssl
systemctl restart apache2

Create the virtual hosts : 

<IfModule mod_ssl.c>
<VirtualHost *:443>
       ServerName essun.calafou.org
       ServerAdmin webmaster@localhost
       DocumentRoot /var/www/essun.calafou
       ErrorLog ${APACHE_LOG_DIR}/essun-error.log
       CustomLog ${APACHE_LOG_DIR}/essun-access.log combined
      SSLEngine on
      SSLProxyEngine On
      SSLProxyVerify none
      SSLProxyCheckPeerCN off
      SSLProxyCheckPeerName off
      SSLProxyCheckPeerExpire off
SSLCertificateFile /etc/letsencrypt/live/calafou.org/fullchain.pem
SSLCertificateKeyFile /etc/letsencrypt/live/calafou.org/privkey.pem
Include /etc/letsencrypt/options-ssl-apache.conf
</VirtualHost>
</<IfModule mod_ssl.c>

Renew auto 

crontab -e
0 0 * * 0 certbot renew -q -a certbot-plugin-gandi:dns --certbot-plugin-gandi:dns-credentials /etc/letsencrypt/gandi/gandi.ini --server https://acme-v02.api.letsencrypt.org/directory

New method 2022 to create a certificate for a subdomain (as the previous one seems to cause trouble while renewing) 

certbot certonly -d subdomain.calafou.org -a certbot-plugin-gandi:dns --certbot-plugin-gandi:dns-credentials /etc/letsencrypt/gandi/gandi.ini --server https://acme-v02.api.letsencrypt.org/directory

LXC container por Libretime y umap

About Libretime et Umap

Installation of the containers

Use of lxcbr0 bridge to allow autostart

Create a lxc-net default config file : 

sudo nano /etc/default/lxc-net

USE_LXC_BRIDGE="true"
LXC_BRIDGE="lxcbr0"
LXC_ADDR="10.0.3.1"
LXC_NETMASK="255.255.255.0"
LXC_NETWORK="10.0.3.0/24"
LXC_DHCP_RANGE="10.0.3.2,10.0.3.254"
LXC_DHCP_MAX="253"

Restart lxc network 

 systemctl restart lxc-net.service

Adapt the config of each container 

nano /mnt/mirror/lxc/bibretime/config

# Network configuration
lxc.net.0.type = veth
lxc.net.0.hwaddr = 00:16:3e:cf:c1:dd
lxc.net.0.link = lxcbr0
lxc.net.0.flags = up

# Autostart
lxc.start.delay = 10
lxc.start.auto = 1

Use the container

list containers 

lxc-ls -f

start 

lxc-start bibretime

stop 

lxc-stop bibretime

attach to a container to have a root prompt 

lxc-attach bibretime

Onduleur Salicru SPS ONE

The software provided is ugly (tomcat java), so let's use NUT to replace it, see :

Essun backup

Essun backup is being stored gpg-encrypted in nusse.

The software used is https://github.com/sukria/Backup-Manager/

We created the user `backup` both in essun and nusse to create/store the backups.

The user backup in essun has a gpg keyring with a public key.

The private key it is not in the servers and it's needed only to decrypt the backups.

There's only 1 configuration file, `/etc/backup-manager.conf`.

These are the changed lines: 

   export BM_REPOSITORY_ROOT="/mnt/mirror/backup"
   export BM_REPOSITORY_USER="backup"
   export BM_REPOSITORY_GROUP="backup"
   export BM_ARCHIVE_METHOD="tarball-incremental"
   export BM_ENCRYPTION_METHOD="gpg"
   export BM_ENCRYPTION_RECIPIENT="264128C3E9B4F451C0105757350C60B958D846EC"
   export BM_TARBALL_DIRECTORIES="/mnt/mirror/calafou_actual"
   export BM_TARBALL_BLACKLIST="/mnt/mirror/backup"
   export BM_UPLOAD_METHOD="ssh-gpg"
   export BM_UPLOAD_HOSTS="172.31.17.202"
   export BM_UPLOAD_DESTINATION="/mnt/backup1T/backup"
   export BM_UPLOAD_SSH_USER="backup"
   export BM_UPLOAD_SSH_KEY="/mnt/mirror/backup/.ssh/id_ed25519"
   export BM_UPLOAD_SSH_HOSTS="172.31.17.202"
   export BM_UPLOAD_SSH_PORT="22"
   export BM_UPLOAD_SSH_DESTINATION="/mnt/backup1T/backup"
   export BM_UPLOAD_SSH_TTL="23"
   export BM_UPLOAD_SSHGPG_RECIPIENT="264128C3E9B4F451C0105757350C60B958D846EC"

It runs as often as configured in `/etc/cron.d/backup`

Also a pre-script is run to mount /var/www on the piweb server as a directory /mnt/ynh via sshfs to backup also the var/www folder on this small machine. For this a backupmirror user was created on the pi/yunohost machine to allow sshfs from essun via an authorized_key indicated in /etc/fstab in Essun

Retrieved from "http://wiki.calafou.org//index.php?title=Essun&oldid=4448"