Libervia v0.8 « La Cecília »

goffi 30/11/2021, 23:02 planet-libre seenthis SàT Libervia project libre release

Je suis fier d'annoncer la sortie de Libervia 0.8 « La Cecília » (anciennement nommé « Salut à Toi »), après plus de 2 ans de développement.

Cette version est une avancée majeure pour préparer le futur du projet. Voyons les principales nouveautés.

Changement de nom

Par soucis de simplicité, le projet a été renommé en « Libervia » (qui était auparavant le nom du frontal web), et tous les frontaux officiels ont maintenant un nom évident comme Libervia Web , Libervia Desktop / Mobile (c'est le même frontal dans les 2 cas), Libervia CLI pour Command-Line Interface (interface en ligne de commande), et Libervia TUI pour Terminal User Interface (interface utilisateur pour terminaux). Le backend (service d'arrière-plan) devient, comme vous pouvez le deviner, Libervia Backend. Les anciens noms sont toujours utilisés comme alias.

En plus de la volonté de simplification, le changement de nom a été fait pour s'adapter à un public international : certaines personnes pensaient que « Salut à Toi » était réservé à des personnes francophones uniquement. Espérons que ce sera plus simple pour tout le monde, et que les gens ne seront plus perdus avec tous les noms qui étaient utilisés auparavant.

Notez que le changement de nom influence aussi votre fichier de configuration qui s'appelle désormais libervia.conf (sat.conf fonctionne toujours pour le moment). Les sections y ont été mises à jour avec de nouveau noms (pour configurer l'interface en ligne de commande, vous devez maintenant utiliser la section [cli] au lieu de [jp], pour l'interface web, c'est à présent [web] au lieu de [libervia]). Veuillez vérifier la documentation si nécessaire.

Changements techniques

Libervia a été porté en Python 3, ce qui a ouvert la porte à d'autres changements.

Brython a été intégré à Libervia Web, pour remplacer "Pyjamas" (un port en Python de GWT), qui n'était plus maintenu.

Nunjucks est aussi désormais intégré à Libervia Web, ce qui permet l'utilisation commune de certains modèles avec Jinja. C'est particulièrement utile pour faire des pages qui fonctionnent avec ou sans JavaScript.

L'implémentation d'OMEMO a été complétée avec le chiffrement des salons de discussion (MUC) et des fichiers (via XEP-0454: OMEMO Media Sharing).

De nombreuses autres modifications ne sont pas expliquées dans cette note pour éviter de la rendre indigeste. Vous pouvez vous référer à la liste des changements (en anglais) pour plus d'informations.

Nouveau thème par défaut pour Libervia Web

Un nouveau thème basé sur le très bon cadriciel CSS Bulma est devenu le thème par défaut. Grâce à lui, l'interface est plus claire et agréable à utiliser.

La liste des événements avec le nouveau thème

Invitations

Un système d'invitation simple a été intégré au frontal web, et il peut être utilisé soit pour donner accès à quelque chose à quelqu'un ayant déjà un compte XMPP, soit pour inviter quelqu'un par courriel. Le but est d'être capable de partager des choses (comme un album photo ou un événement) avec famille et amis sans s'attendre à ce qu'ils ou elles installent un logiciel ou créent un compte quelconque.

Inviter des personnes à voir un album photo

Listes

Un gestionnaire de tickets décentralisé est implémenté depuis la verison 0.7, et il est notamment utilisé pour gérer les tickets de Libervia lui-même. Jusqu'ici, il fonctionnait grâce à une fonctionnalité non standard disponible uniquement sur Libervia Pubsub (anciennement « SàT Pubsub », un composant Pubsub/PEP indépendant du serveur développé en parallèle de Libervia).

Cette fonctionnalité a été renommée avec le terme plus générique « listes » et utilise désormais XEP-0346: Form Discovery and Publishing, ce qui la rend utilisable avec un service pubsub générique.

Toutes sortes de listes peuvent être créées, que ce soit des tickets pour suivre les rapports de bogues ou demandes de fonctionnalités pour un projet, des listes de tâches à faire, de course, etc. Grâce à l'utilisation de XMPP pubsub, ces listes peuvent être fédérées, et les permissions peuvent être gérées facilement (pour, par exemple, permettre aux membres de la famille de modifier une liste de courses).

Liste de courses sur Libervia Web

Actuellement, 3 types de listes sont disponibles (tickets génériques, tâches, courses), mais on peut s'attendre à en voir plus dans les futures versions.

Albums photo

Des nombreuses améliorations ont été faites pour les albums photo du frontal web. Ils peuvent désormais être créés ou supprimés depuis Libervia Web, des photos ou des vidéos peuvent être téléversées, un diaporama utilisable avec un écran tactile et/ou sur petits écrans est disponible, ogv.js a été intégré pour rendre possible le visionnage de vidéos en Ogg Vorbis/Opus/Theora ans WebM VP8/VP9/AV1 sur les plateformes qui ne le supportent pas nativement et le système d'invitations mentionné plus haut a été intégré.

Vous pouvez désormais utiliser un diaporama pour voir vos photos et vidéos

Bureau

L'interface utilisateur de Libervia Desktop a également été mise à jour : la barre de menu supérieure a été supprimée, le glisser/déposer de fichiers est maintenant possible suivant la plateforme, la messagerie instantanée a un défilement infini, un nouvel écran de « sélection de conversation » rend le choix d'une entité avec qui discuter ou d'un salon à joindre plus faciles, les pièces jointes des messages sont affichées de manière plus agréable, entre autres améliorations.

Le sélecteur de tchat sur Libervia Desktop

Du travail a aussi été effectué sur Libervia Mobile (qui ne fonctionne que sur Android pour le moment), mais ce frontend n'est pas encore assez stable pour les utilisateurs finaux.

Joindre un fichier sur Libervia Mobile

Ligne de commande

L'interface en ligne de commande est à présent entièrement documentée (en anglais) et suite au changement de nom peut être utilisé soit avec la commande libervia-cli, soit la version courte li (l'ancien jp fonctionne encore pour le moment). Parmi les nouvelles commandes on peut mettre en avant li file get qui peut télécharger un fichier avec la gestion du schéma aesgcm (c.-à-d. « OMEMO Media Sharing », le partage de média via OMEMO), qui en fait une sorte de wget avec OMEMO. li file upload gère aussi le chiffrement de bout en bout, il est ainsi facile de partager un fichier chiffré depuis la ligne de commande ou un script.

La couleur d'arrière-plan est maintenant automatiquement détectée sur les émulateurs de terminaux compatibles et le thème est choisi en conséquence.

Mais aussi…

Composant de partage de fichiers

Libervia peut s'utiliser comme un composant (ce qui peut être vu comme un plugin générique pour serveur XMPP), et il inclut un composant de partage de fichiers.

Ce composant stocke les fichiers qui peuvent être retrouvés soit selon les permissions accordées, soit publiquement.

Les fichiers peuvent être téléversés ou téléchargés via XEP-0234: Jingle File Transfer et XEP-0363: HTTP File Upload est maintenant aussi implémenté, ce qui rend possible de partager des fichiers avec des liens HTTPS.

Ce composant peut maintenant être utilisé pour remplacer les implémentations internes aux serveurs XMPP de « HTTP File Upload ». En plus de la gestion des permissions fines, il n'y a pas de taille limite pour les fichiers et des quotas par utilisateurs peuvent être mis en place. Veuillez vous référer à la documentation pour voir comment faire. Les fichiers téléversés peuvent être retrouvés en utilisant XEP-0329: File Information Sharing et supprimés grâce aux commandes Ad-Hoc (XEP-0050: Ad-Hoc Commands).

Ce composant est nécessaire pour utiliser la fonctionnalité d'album photos.

Libervia Pubsub

Un composant Pubsub/PEP (anciennement appelé « SàT Pubsub ») est développé en parallèle de Libervia. Il vise à fournir une implémentation complète et indépendante du serveur.

Libervia Pubsub est publié en même temps que le client XMPP Libervia, et a, lui aussi, été porté sur Python 3.

Parmi les nouveautés, la recherche en texte plein a été implémentée (XEP-0431: Full Text Search in MAM), ainsi que la XEP-0346: Form Discovery and Publishing qui remplace les schémas de nœud (qui étaient non standard), et PEP fonctionne à présent pour le serveur lui-même, ce qui le rend utilisable pour la XEP-0455: Service Outage Status.

Images Docker

Les images Docker on été mises à jour et déplacées directement dans le dépôt libervia-backend (dans le sous-dossier docker).

Site officiel

Le site officiel a été mis à jour avec un nouveau thème (basé sur le nouveau thème de Libervia Web).

Installation

Libervia est disponible sur plusieurs distributions NU/Linux (au moins Debian et dérivées et Arch Linux). Malheureusement, la version actuellement dans Debian n'est pas à jour (à cause d'une publication à des dates différentes de Debian et Libervia), espérons que la nouvelle version sera disponible dans un dépôt « backport » rapidement.

Vous pouvez facilement installer Libervia sur n'importe quelle distribution en utilisant pipx:

$ pipx install libervia-backend
$ pipx install libervia-desktop

Ensuite lancez libervia-backend et un frontal (comme libervia-cli ou libervia-desktop). Reportez-vous à la documentation pour plus de détails.

Des images Docker sont disponibles, et en particulier un fichier web-demo.yml peut être utilisé avec docker-compose pour rapidement mettre en place une démo locale :

$ hg clone https://repos.goffi.org/libervia-backend 
$ cd libervia-backend/docker
$ docker-compose -f web-demo.yml up

Ensuite ouvrez votre butineur sur http://localhost:8880 et utilisez l'identifiant demo avec le mot de passe demo.

Et après ?

Un projet basé sur Libervia a été sélectionné par NLnet pour une subvention. Ce projet est en 2 parties : travailler sur une passerelle XMPP ⟺ ActivityPub, puis sur le chiffrement de bout en bout pour pubsub et les fichiers. Vous trouverez plus d'informations (en anglais) sur ce billet de blog et sur la page du projet NLnet. Ce projet est déjà bien entamé, et vous pouvez suivre la progression sur mon blog (qui est propulsé par Libervia/XMPP) ou sur le système de gestion de tickets (lui aussi propulsé par Libervia/XMPP). Un énorme merci à NLnet/NGI0 Discovery Fund !

D'autre part, il est prévu de travailler pour améliorer l'expérience utilisateur et la fonctionnalité de messagerie instantanée (en particulier sur le frontal web). Libervia vise à être particulièrement adapté pour les réseaux privés destinés à la famille et aux amis.

Enfin, j'ai eu la bonne surprise de voir que Libervia Web est utilisé pour le blog de jmp.chat. JMP est une entreprise qui vous fourni un numéro de téléphone qui peut être utilisé avec XMPP et SIP (vous pouvez appeler ce numéro depuis un téléphone traditionnel et recevoir l'appel vocal depuis un client XMPP)

Voilà qui conclu cette note de version. À bientôt !

E

edhelas 01/12/2021, 07:54

M

matlag 01/12/2021, 17:09

Libervia v0.8 « La Cecília »

goffi 30/11/2021, 22:51 jabber-xmpp-en SàT Libervia project libre release

I'm proud to announce the release of Libervia 0.8 « La Cecília » (formerly known as « Salut à Toi »), after more than 2 years of development.

This version is a big milestone preparing the future of the project. Let's have an overview of some major changes.

Project Renaming

In the interest of simplicity, the project has been renamed to " Libervia " (with was formerly the name of the web frontend), and all official frontends have now a straightforward name such as Libervia Web , Libervia Desktop / Mobile (same frontend for both), Libervia CLI for Command-Line Interface , and Libervia TUI for Terminal User Interface. The backend is, as you can guess, Libervia Backend. The former names are for now still used as aliases.

Beside simplicity, the name change was also due to concerns with international audience: some people were thinking that "Salut à Toi" was dedicated to French-speaking people only. Hopefully, it will be easier for everybody, and people won't get confused any more by all the names which were previously used.

Note that the renaming has implications on your configuration file which is now named libervia.conf (sat.conf is still working for now). The sections have been updated with new names (to configure the CLI frontend you now use [cli] section instead of [jp], for the web frontend it's now [web] instead of [libervia]). Please check documentation in case of doubt.

Technical Changes

Libervia has been ported to Python 3, which has opened the door to other changes.

Brython has been integrated to Libervia Web, to replace the unmaintained "Pyjamas" (which was a Python port of GWT).

Nunjucks is also now integrated in Libervia Web, making it possible to share some templates with Jinja. This is notably useful to make some pages working with or without JavaScript.

OMEMO implementation has been completed with support for MUCs (group chats) and files (via XEP-0454: OMEMO Media Sharing).

Many other changes are not explained in this note to avoid it being indigestible, you can check the CHANGELOG for more information.

Libervia Web New Default Theme

A new theme for Libervia Web based on the nice Bulma CSS framework has been made and is now the default one. Thanks to it the interface is clearer and more pleasant to use.

Events list with the new theme

Invitations

An easy to use invitation system has been implemented in Web frontend, and can be used either to give access to something to somebody with an existing XMPP account, or to invite somebody by email. The goal is to be able to share things (e.g. photo album, event) with family or friends without having to expect them to install a software or create an account.

Inviting people to see a photo album

Lists

A decentralised issue tracking system was implemented since version 0.7, which was notably used to manage Libervia's own tickets. It was using a non-standard feature available only in Libervia Pubsub (formerly "SàT Pubsub", server independent Pubsub/PEP component, a side project).

This feature has been renamed to "Lists" and now uses XEP-0346: Form Discovery and Publishing which makes it usable with a generic Pubsub service.

Any kind of list can be created, from project tickets to keep track of bug report of feature requests, to To-do list, grocery list, etc. Being based on XMPP pubsub, lists can be federated, and permissions can be managed easily (for instance to allow various family members to modify a shopping list).

Grocery List on Libervia Web

For the moment 3 kinds of lists are available (generic tickets, To-Do, grocery), but more are expected to come in future versions.

Photo albums

Lots of improvements have been made on the photo albums in the web frontend. They can now be created or deleted from Libervia Web, photos or videos can be uploaded, a touch/mobile-friendly slideshow is available, ogv.js has been integrated to make possible the viewing of videos in Ogg Vorbis/Opus/Theora ans WebM VP8/VP9/AV1 on platforms not supporting them natively, and the invitation system mentioned above has been integrated.

you can now use a slideshow to see your photos and videos

Desktop

Libervia Desktop UI has also been updated, the top menu has been removed, file dropping is now possible on suitable platforms, chat has infinite scrolling, a new "chat selector" screen makes it easier to select entity to chat with or room to join, message attachments are show in a more user-friendly way, and several other improvements has been done.

Chat Selector on Libervia Desktop

Work has also been done on Libervia Mobile (which is Android only for now), but this frontend is not user-friendly enough yet for end-user.

Attachment on Libervia Mobile

CLI

The CLI frontend is now fully documented and following the renaming can now be accessed either by libervia-cli or the shorter li (legacy jp is still working for now). Among new commands we can highlight li file get which retrieve a file with support of aesgcm scheme (i.e. OMEMO Media Sharing), which makes it a kind of OMEMO enabled wget like. li file upload also handle end-to-end encryption, it's thus easy to share an encrypted file from command-line or a script.

Background colour is now automatically detected on compatible terminal emulator, and theme is adapted consequently.

But Also…

File Sharing Component

Libervia can act as a component (which can be seen as generic XMPP server plugins), and it includes a File Sharing Component.

This component store files which can be retrieved either according to given permissions or publicly.

Files can be uploaded or downloaded via XEP-0234: Jingle File Transfer and XEP-0363: HTTP File Upload is now also implemented, making it possible to share files via HTTPS link.

This component can now be used to replace internal XMPP servers HTTP File Upload implementations. In addition to the fine permission management, it does not have a size limit and user quotas can be set, check the documentation to see how to set them. Files uploaded can be retrieved using XEP-0329: File Information Sharing and deleted with XEP-0050: Ad-Hoc Commands.

This component is necessary to use the Photo Album feature.

Libervia Pubsub

A Pubsub/PEP component (formerly named "SàT Pubsub") is developed next to Libervia. It aims to provide a server independent feature-full implementation.

Libervia Pubsub is released at the same time as the Libervia XMPP client, and has also been ported to Python 3.

Among novelties, Full-Text Search has been implemented (XEP-0431: Full Text Search in MAM), as well as XEP-0346: Form Discovery and Publishing which replaces the former non-standard node schema, and PEP is now working for the server itself, making it usable for XEP-0455: Service Outage Status.

Docker Images

Docker images have been updated and moved directly to libervia-backend repository (in docker subdirectory).

Official Website

The Official Website has been updated with a new theme (based on Libervia Web new theme).

Installation

Libervia is available on several GNU/Linux distributions (at least Debian and derivative and Arch Linux). Unfortunately, the current Debian version is outdated (due to incompatible Debian and Libervia release dates), hopefully the new version will be available as a backport soon.

You can easily install Libervia on any distribution by using pipx:

$ pipx install libervia-backend
$ pipx install libervia-desktop

Then launch libervia-backend, and a frontend (e.g. libervia-cli or libervia-desktop). Check the documentation for details.

Docker images are available, and notably a web-demo.yml file can be used with docker-compose to quickly try a local demo:

$ hg clone https://repos.goffi.org/libervia-backend 
$ cd libervia-backend/docker
$ docker-compose -f web-demo.yml up

Then open your browser on http://localhost:8880 and use the login demo with password demo.

What's Next

A Libervia based project has been selected by NLnet for a grant. This project is in 2 parts: working on an XMPP ⟺ ActivityPub gateway, then on pubsub and files end-to-end encryption. You'll find more information on this blog post and on NLnet project page. The project has already well started, and you can follow the progress on my blog (which is Libervia/XMPP powered) or on the ticket tracker (which is also Libervia/XMPP powered). A huge thanks to NLnet/NGI0 Discovery Fund!

Besides, work is planned to improve user experience and instant messaging feature (notably on Web frontend). Libervia aims to be a good fit for private networks for family and friends.

Last but not least, I've been pleased to see that Libervia Web is used to power jmp.chat blog. JMP is a company which give you a real phone number which can be used with XMPP and SIP (you can call this number from a traditional phone and get the voice call from a XMPP client).

This concludes this release post. Stay tuned!

debacle 01/12/2021, 01:07

S

sthaan 02/12/2021, 16:52

Libervia progress note 2021-W38

goffi 27/09/2021, 06:54 jabber-xmpp-en SàT Libervia project libre Libervia progress SàT progress

Hello,

it's time for a new progress note. The work is currently focused on ActivityPub Gateway, and progress has been done on pubsub cache search and the base component.

Pubsub Cache Full-Text Search

Next to the pubsub cache implementation, it was necessary to have a good way to search among items.

So far, Libervia was doing pubsub search using pubsub service's capabilities, and notably the XEP-0431(Full Text Search in MAM) implementation. This is working well (it's what is currently used on this very blog when you do use the search box), but has some pitfalls: the pubsub service must implement this XEP (and as far as I know, Libervia Pubsub is the only one which does it at the moment), the search can be done in a single node at a time only, each search request imply a new XMPP request to the pubsub service, and pubsub items must be in plain text (which is currently always the case, but pubsub end-to-end encryption is planned as second part of the granted NLNet project on which I'm working).

In regard to that, a local search is necessary. SQLAlchemy doesn't really have Full-Text Search (or FTS) support for SQLite out of the box, but it allows to use any SQL directly, thus I could use the really nice FTS engine available within it (FTS5). This is an extension, but in practice it is already installed most of the time (it is part of the SQLite amalgamation).

Thanks to the JSON support in SQLite, it is also possible to filter search requests on parsed data. That's really useful for features like blogs where you often want to do that (e.g. filtering on tags).

The cache search can be operated on all data in cache, that means that you can do search on items coming from multiple nodes and even multiple services. That opens the door to features like hashtags or blog suggestions.

Last but not least, search requests can be ordered by any parsed field. In other terms it will be possible to order a blog by declared publication date — which may be important if you want to import a blog —, or events by location.

To have an idea of the possibilities, you can check the documentation of the CLI search command.

Base ActivityPub Component

Once the preparatory steps have been done, the ActivityPub component itself could be started. In short, for people not used to XMPP, a "component" is a kind of generic plugin to server. You declare it in your server configuration, choose a JID and a "shared secret" (a password), run it with those parameters, and voilà.

For the AP gateway, Libervia runs the component. There is documentation to explain how to launch it, don't worry it's simple.

As I've got questions about this, here is a small schema giving an overview on how the whole thing is working:

global overview of Libervia ActivityPub Gateway

I hope that it makes the whole thing more clear, otherwise don't hesitate to ask me for clarification.

As you can see, the gateway includes an HTTP server to communicate with AP software, but in many cases there will already be an HTTP server (website, XMPP web client, etc.). In this case, you'll have to redirect /.well-known/webfinger and /_ap requests to the gateway server.

For the development, I'm using Prosody as reference XMPP server implementation, and Mastodon as reference ActivityPub server implementation. I've set a local Mastodon installation, and I've chosen to use Docker for that, as it makes things easy to have a reproducible environment and to save and restore a specific state. It was not as trivial as I would expect to find the right configuration to use, I've found outdated tutorials, but I could manage to run the thing relatively easily.

Because we work with HTTPS, I've made a custom docker image with locale certification authority, so Mastodon could validate my gateway HTTP server certificate. I'm already doing that for docker image used for end-to-end tests of Libervia, nothing difficult. Surprisingly though, Mastodon could not resolve my instance, when HTTPie running from the same container could do it flawlessly. I've quickly realised that Mastodon was not respecting hosts declared in /etc/hosts (and added via extra_hosts in Compose file) and found a relevant bug report on Mastodon tracker. That was annoying, and I had to find a way to work around that. I've done it by running a local DNS Server, and Twisted offers a nice built-in one. Twisted DNS can easily use /etc/hosts to direct my local domains to my local IP, it's just a one liner such as twistd3 -n dns --hosts-file=/etc/hosts -r.

After that the domain was resolving, but to my surprise, Mastodon was still not able to communicate with my gateway, and even more bizarre my server was receiving no request at all. After a quick round of tcpdump/wireshark, I saw that indeed nothing was sent to my server.

Thanks to the Libre nature of Mastodon, I could resolve this by reading the source code, the Mastodon::HostValidationError
led me to a section that made the whole picture clear: my server is on a local IP and Mastodon by default refuses to reach it (to avoid the confused deputy attack). With the ALLOWED_PRIVATE_ADDRESSES setting I could finally make Mastodon communicate with my server.

The How to implement a basic ActivityPub server tutorial made by Eugen Rochko (Mastodon original developer) is a nice article to start an ActivityPub implementation, it has been useful to build the base component (despite being a bit outdated, notably regarding signature).

I have to rant a bit, though, as the ActivityPub specification are not available in EPUB or PDF, making it difficult to read on an e-book reader. I could overcome that thanks to pandoc (git clone https://github.com/w3c/activitypub.git then pandoc index.html --pdf-engine=xelatex -o activitypub.pdf), it's really more comfortable to keep the reference like this.

So the base component is now available but only usable by developers (and only capable of sending message to ActivityPub for now). Things will be really exiting with the next 2 steps, as bidirectional communications will be available, and the gateway will be usable for early adopters. I don't expect those steps to be really long.

test message sent with Libervia AP Gateway

Oh, and to answer another question that I've had, yes you can use the same ActivityPub actor identifier as your XMPP JID. I'll explain next time how everything is accessed.

That's all for today.

debacle 27/09/2021, 08:02

M

matlag 28/09/2021, 00:30

S

sthaan 14/11/2021, 19:48

Libervia progress note 2021-W31

goffi 03/08/2021, 11:03 jabber-xmpp-en SàT Libervia project libre Libervia progress SàT progress

Hello,

last weeks have been exhausting with lot of things at the same time. I've been preparing the release of the 0.8 version, and I wanted to have a couple of thing ready for that, notably a proper way to do translation.

Preparation of 0.8

As you may know, I've implemented a docker integration into Libervia to be able to run relatively easily third party software. This is working, but when testing in the production website I had to put the finishing touches to make it work (notably I've improved HTTP proxy and HTTPS management). I have then created projects and updated a couple of translations files.

As you can now see on https://salut-a-toi.org/, there is a translate menu. Unfortunately I've closed the account creation for the moment, as I have to deal with licensing first. Indeed, nearly all Libervia ecosystem is for now in AGPL v3+, as there are only a few contributors (2 mains one, then only a small patches). The intent was and still is to be sure that the ecosystem stays in an Libre license, but this license may cause trouble in some edge cases, notably if we want to make an iOS frontend (the fruit store is notoriously causing trouble with AGPL licences).

Thus, I'll bring the subject at next general assemble of the "Salut à Toi" association, and see what we should do. One option could be to use FSF's Fiducial Licence Agreement to let the association the possibility to modify the licence as long as it stays a libre one. It would then be possible to add an exception for an iOS frontend. An other would be to avoid totally iOS. Anyway, this need some time and discussions, and if I open translations and get several contributions under AGPL v3+, it may be harder to set this up.

Weblate integrated in the official website

An other time consuming task was to continue with renaming and adapt package names (notably in Pypi). I've used a little trick to redirect legacy packages to the new ones: a new version of each legacy package is a simple setup.py depending on the new package (you can see it e.g. for sat package). I've also put in place a redirection on the Mercurial repositories, using the old repos will redirect to the new ones.

Finally, I've published the 0.8.0 beta 1. You can install it easily with pipx:

  • First install pipx as explained in its documentation
  • Then install the backend with pipx install libervia-backend. You can follow the documentation to see how to configure it and launch it. This will include the CLI and TUI frontends.
  • If you want to test graphical frontends, you'll have to install Libervia Media with hg clone https://repos.goffi.org/libervia-media (assuming that you have Mercurial already installed), then add it into your libervia.conf
  • To install the Desktop frontend, use pipx install libervia-desktop
  • To install the Web frontend, use pipx install libervia-web

Note that the Desktop frontend is still for early adopters, I need to refactor message handling and do some optimisation and stabilisation to make it pleasant to use.

Please send feedbacks either as bug reports/feature requests on the official bug tracker, or on the XMPP chat room at sat@chat.jabberfr.org. I plan to only fix major issues though, as I'm now fully working on 0.9 and I'm focusing mainly on the ActivityPub gateway. However, bug reports/feature requests will be taken into account for 0.9 if not fixed directly in 0.8.

ActivityPub Gateway

After the hard work to move 0.8 close to the finish line has been done, I've started to work on 0.9 and thus the ActivityPub gateway. The first major thing to do was a refactoring of offline storage. Indeed Libervia (or SàT at the time) was started a long time ago with an Async framework (Twisted) long before asyncio even existed. SQLite has been chosen as backend to store data, and a hand made module based on Twisted's adbapi has been created. Despite the rough edges is has been working quite well all this time, and there was even a semi automatic way to update schemas between version. But the whole thing was starting to be difficult to maintain, and the schema update instructions were all kept in the same file.

Fortunately, SQLAlchemy, the most famous SQL databases abstraction with an optional Object Relational Mapper has recently added support for asyncio.

SQLAlchemy is a very powerful and widely used tool, so it has been a quite obvious decision to use it to replace the old system. But to use it, Twisted needs to use an asyncio loop, and Libervia was using GLib loop (or reactor in Twisted terms), due to the dependency to dbus-python.

Dbus-Python is, by its own authors words, not be the best D-Bus binding to use due to unfortunate design decision, so it was the occasion to replace it, and I've moved the backend to TxDBus, a Twisted native D-Bus implementation, which can run with any Twisted reactor. For technical reason, dbus-python is still used for frontends at the moment, but I plan to completely replace it before the end of 0.9 development.

This has required some work, but it was worth it, and after that I could switch to asyncio reactor and implement SQLAlchemy. I've decided to go with the ORM and not the core only as it is opening neat possibilities. I've first made a mapping corresponding to the last version of the database used for Libervia 0.8.

Once SQLAlchemy has been implemented and working, the next obvious decision was to use Alembic, the recommended SQLAlchemy based database migration tools (by the same authors). Thanks to this, migration files are now in separated files, and are really easy to create (Alembic can autogenerate a good part of a script when a migration is needed).

Thanks to all this, I can now easily make changes in database (while in old system I was hesitating due to the work implied). SQLAlchemy also paves the way to support other databases than SQLite. Even if I'm currently sticking with SQLite only, to keep focus, I'll probably add support for PostgreSQL and MariaDB/MySQL at some point.

Once all this work on storage backend has been finalised, the pubsub cache has been implemented.

Pubsub cache is operated transparently for end-user, and stores locally pubsub items (according to internal criteria). This is useful for many reasons: performances of course, but also it allows to do data analyse/indexing, for instance to retrieve all items with some terms (e.g.: to search by categories or hashtags). Pubsub cache is also useful to store data in a component (what is of interest for the ActivityPub gateway), or to store decrypted data (which will be of interest when we'll work on the e2e encryption with pubsub).

I'll pass the implementation details, you'll find the code on the 0.9 bookmark, notably in the pubsub cache plugin, and I've written documentation for developers for some explanations.

New commands has been added to libervia-cli to manage the cache, in particular there is a purge command to delete items according to given criteria, which will save resources and disk space. With it, it's possible to delete only certain types of items (e.g. only blog posts), for all or only some profiles (for instance, only for the AP gateway). You can say a limit (e.g. delete all items which have not been updated for 6 months). Here again, documentation has been written to explain the commands.

While doing all that, I've noticed problem to cache correctly items (because of the flexibility of XMPP Pubsub, it's hard to impossible to say if we can share cache between users), thus I've written a protoXEP (i.e. a proposition for an XMPP Extension Protocol, or XEP) to fix the problem: https://xmpp.org/extensions/inbox/pubsub-caching-hints.html.

I've also submitted a pull request to fix a problem in XEP-0060 (Publish-Subscribe).

While I was a working with standards, I've updated a XEP I've authored a couple of years ago to specify order of items: XEP-0413: Order-By.

Last but not least, while doing the tests for the pubsub cache I've created some helping methods (or fixtures in pytest terms) to help doing unit test.

This concludes the first step of the XMPP-ActivityPub gateway which was, as anticipated, a big one. The following steps should be done more quickly, and work on 0.8 should not be on the way anymore (I plan to publish 0.8 in early September).

That's all for this note, see you next time.

E

edhelas 03/08/2021, 11:08

M

matlag 03/08/2021, 12:50

debacle 03/08/2021, 19:56

Libervia progress note 2021-W18

goffi 06/05/2021, 11:39 jabber-xmpp-en SàT Libervia project libre Libervia progress SàT progress

Hi,

again, lot of things have happened since last progress note, so I'll only talk about major changes here.

"Salut à Toi" is now "Libervia"

The project has been renamed to "Libervia". Even if I personally loved the former name (which was a reference to a French punk band song, an which could be translated to "hi to you", a nice fit for a communication tool), it proved to be hard to pronounce and remember for non French speakers, and the many names of frontends and project components were confusing. The name change has been discussed for long in the association, but the new ActivityPub/Pubsub end-to-end encryption project accelerated things: after a talk with NLnet, we decided to move forward on this so project name would not change in the middle of its development.

After doing a quick poll, we confirmed that "Libervia" (which was formerly the name of the web frontend only) would be the new name.

All parts are now named in straightforward way: "Libervia Backend", "Libervia Web", "Libervia Desktop/Mobile" (currently the same Kivy frontend for both), "Libervia TUI" and "Libervia CLI", with matching executable names (libervia-backend, libervia-web, libervia-desktop, libervia-tui, libervia-cli also aliased as li). The former names are kept internally and as aliases.

The non-profit (French "loi 1901") association behind it stays with the name "Salut à Toi".

This renaming has involved a lot for work, it took weeks to update code, web sites, doc, etc. and according to our statuses, we had to make a general assembly to discuss this decision. It's still not fully finished (notably the official web site URL is still https://salut-a-toi.org, while https://www.libervia.org is currently used for the demo server), and source code repositories have not been modified for the moment, but most of the renaming is done, and you can now reference the whole project as "Libervia"

Official Website Update

Following the changes in Libervia Web themes, the official website one has also been updated and is now based on Libervia's Web Bulma theme. The news now links to my personal blog as it is where you'll have most up-to-date informations about Libervia development (and the former page was broken). Tickets/Bug tracker is now directly accessible from the official site, as it makes more sense to have it there. It's still accessible from goffi.org, and thanks to its decentralised nature, it's usable transparently on both locations.

I've also temporarily disabled account registration on the bug-tracker due a wave of spammy accounts. I will have to put in place a protection for that, but I'm reluctant to use popular non-libre options.

Flatpak and Docker

While working on the renaming and website, I've updated the Flatpaks (they were really outdated), and Docker images. Flatpaks is for now using a specific dev repos, but I hope to see Libervia on Flathub after the release.

I've created Docker images and Docker Compose file to run quickly a local demo of Libervia Web, you can see the instruction on the Official Website.

Ideally, I would like to also create Snaps, Appimages, Nix packages, etc. But I'm lacking time (Flatpak and Docker are already too much time consuming) and prefer to focus on the code rather than on the packaging, help is more than welcome though.

User Friendly URLs

As you may have noticed on the last blog posts, URLs are now more user friendly:

A blog post is referenced using its item ID, and previously a unique ID was used for that, which is relatively long and doesn't give any information about the content, but is necessary to avoid conflict (writing a blog post with an existing ID will overwrite the previous one).

To make it more pleasant, a URL friendly extension was then added, and not used to retrieve the item, so in the example above, www.goffi.org/b/LFMqr7xC2aNf4MDgkbamBY links to the same blog post as www.goffi.org/b/LFMqr7xC2aNf4MDgkbamBY/sat-progress-note-2020-w53. The resulting URL is long and not easy to read, but the item is unique.

The new behaviour directly use URL friendly item IDs, and to avoid conflict, a short random suffix is appended (on the example above, QGqK is the suffix). After some tests, the collision risk for a short suffix like that is not that high (I've tested millions of IDs without collision), and it may anyway happen only if 2 blog posts have the exact same title, so the risk is very low. The resulting URL is more pleasant.

This URL friendly ID is used by default when a blog post is created, but it can be deactivated if user_friendly_id is set to false in blog post metadata, or by specifying manually an item id.

To accompany this change, a new Libervia CLI rename subcommand has been added to li blog and li pubsub, which will change the ID of an item. As there is no standard rename operation in XMPP Pubsub, this is done by copying the item to the new ID, then delete the former one in case of success.

Navigation Helpers in Libervia Web

It was not really easy so far to know where we were in Libervia Web. To help with this, the selected menu is now shown activated, and a breadcrumb has been added.

The breadcrumb is only shown when there are at least 2 elements to show (i.e. not on root pages). It is generated automatically by default, but can be customised with specific label, sub-elements, or even icons, like in the file sharing screenshot below:

Libervia Web 0.8 Breadcrumbs Screenshot

Blog Editor

As it was not possible anymore to write a new blog item with Libervia Web, I've made a blog item editor, which is relatively basic for now, but working. If JavaScript is activated, you'll get a tags editor, preview, and autosaving:

Libervia Web 0.8 Blog Editor Screenshot

File Sharing Quotas

One last missing piece I was needing before release was to put in place quotas on the file sharing component, this is now done.

Indeed, this component doesn't work with a per-file limit like most others do, but with a per-user quota, and you can upload any file size you want at long as you're not over quota.

Release to come

It's more than time to think about the release. I wanted to improve the chat notably in Libervia Web where it's still really basic since we moved out from the former frontend, but finally I've decided to report this to next release, as I plan to refactor messages handling, and for now I need to concentrate on the ActivityPub gateway.

So I'll soon prepare a beta version, and plan to do the release in a couple of weeks. I'll do bugfix on the 0.8 version during this time, but will avoid any important new development.

ActivityPub gateway project

With all the work done above (and other things, I've not mentioned everything), I've been late to start working on the ActivityPub project, but now I can focus on it. The first task is about developing a Pubsub cache as Libervia is currently getting its data for Pubsub related feature directly from the services.

Beside the obvious speed improvement, having a local cache will give the possibility to do data search/manipulation (such as doing Full-Text Search when the Pubsub service doesn't implement it, or doing feature-specific data analysis), handle message received unordered, allow to keep decrypted data when received from e2ee items, etc.

So far, SQLite was used for data storage in Libervia, by using Twisted's adbapi and custom semi-automatic schema update/data migration. It has been working relatively well so far, but it's no pleasant to maintain.

Fortunately, SQLAlchemy has recently added support for AsyncIO, thus it can now be used in Libervia. This is great, as SQLAlchemy is popular and rock solid, so I've decided to go with it. This will open the possibility to use other backends (like PostgreSQL), and refactor Libervia to use SQLAlchemy's ORM. Logically, Alembic will be used for data migration, which should make database modifications easier.

Such a cache will make possible to implements things like items discovery based on categories (or search by "hashtags" as it named in other software).

That's all for this note, see you soon.

E

errormovim 06/05/2021, 11:40

E

edhelas 06/05/2021, 11:59

M

matlag 06/05/2021, 13:40

debacle 06/05/2021, 14:27

Passerelle ActivityPub et chiffrement de bout en bout Pubsub

goffi 23/04/2021, 11:48 planet-libre seenthis jabber-xmpp SàT Libervia project libre ActivityPub e2ee nlnet

Bonjour,

c'est avec grand plaisir que je vous annonce qu'un projet de passerelle ActivityPub <=> XMPP doublé d'une adaptation du chiffrement de bout en bout à PubSub a été sélectionné pour une subvention par NLnet/NGI0 Discovery Fund (avec un soutien financier de la commission européenne "Next Generation Internet") : https://nlnet.nl/project/Libervia/

Ce gros projet est divisé en 27 étapes, et va occuper la plupart de mon temps dédié au projet Libervia (anciennement "Salut à Toi", le projet a été renommé, je vais expliquer cela, en anglais, dans une "progress note" à venir).

La passerelle XMPP <=> ActivityPub va joindre 2 protocoles ouverts et décentralisés majeurs. En pratique, ce sera un composant pour serveur XMPP (utilisable avec n'importe quel serveur), et il va implémenter le protocole "serveur à serveur" de ActivityPub (aussi connu comme « protocole de fédération » ou « Federation Protocol »). Côté XMPP, ce sera principalement un service PubSub (avec quelques extras, comme les messages privés convertis en messages XMPP).

Le blogage XMPP (XEP-0277: Microblogging over XMPP) va être utilisé, et donc n'importe quel client qui l'implémente aura accès aux publications ActivityPub (Libervia et Movim par exemple).

Pour les fonctionnalités présentes dans ActivityPub et pas encore dans XMPP, il est prévu de proposer des « protoXEPs » (c.-à-d. des propositions d'extensions de XMPP) pour les implémenter. Les évènements font aussi parti du projet, et une compatibilité entre Mobilizon et Libervia est prévue, ainsi qu'une protoXEP pour standardiser cela côté XMPP.

Tout ceci est très excitant, parce que ça va étendre les deux réseaux, et donner un coup d'accélérateur aux projets intégrant le blogage et le chat XMPP.

La seconde partie du projet concerne le chiffrement de bout en bout. XMPP a connu des améliorations majeures sur le chiffrement de bout en bout suite au travail fait sur OMEMO, notamment initié avec Conversations, et sur OX, une intégration moderne de OpenPGP. C'est excellent, mais cela se concentrait jusqu'ici sur la messagerie instantanée. Le but va être d'ajouter le chiffrement de bout en bout au PubSub XMPP, avec les protoXEPs et l'implémentation dans Libervia. En d'autres termes, à la fin de ce projet, il sera possible d'utiliser le chiffrement de bout en bout avec toutes les fonctionnalités basées sur Pubsub (comme les blogs, forums, listes ou autre évènements que vous pouvez utiliser avec Libervia), c'est énorme ! Les signatures feront partie du projet aussi, signifiant qu'il sera possible d'authentifier un élément comme un billet de blog d'une façon standardisée.

En plus des standards qui vont bénéficier à la communauté XMPP entière, tout ceci sera implémenté dans Libervia, ce qui inclus mettre à jour l'implémentation actuelle pour arriver à l'état de l'art (c.-à-d. mettre à jour l'implémentation actuelle d'OMEMO et implémenter OX).

La dernière partie du projet sera l'implémentation du chiffrement de bout en bout dans le frontal web. En effet, l'architecture spécifique de Libervia fait qu'OMEMO n'est actuellement pas utilisable depuis le navigateur (l'implémentation est faite dans le « backend »). Pour rendre ceci possible, la bibliothèque Python OMEMO qui est actuellement utilisée va être portée sur WebAssembly et Brython, ce qui va permettre de faire du chiffrement et du déchiffrement directement depuis le navigateur.

Comme vous pouvez vous en rendre compte, c'est beaucoup de travail. Je vais faire la majeure partie de tout ceci, mais je ne serai pas seul (en particulier l'auteur de Python OMEMO fera le port sur wasm/Brython à travers ce projet). Le développement devrait prendre environ un an, la passerelle ActivityPub <=> XMPP étant la première partie.

Je profite de ce billet pour remercier encore NLnet et le programme NGI de l'UE, ainsi que mon employeur (Sourcefabric, qui développe en particulier Superdesk) pour m'avoir permis d'adapter mon rythme de travail.

J'ai adapté les tâches au traqueur de bogues de Libervia (qui fonctionne avec XMPP), aussi vous pouvez voir leur détails (en anglais) et suivre leur progression à cette adresse : https://salut-a-toi.org/bugs?search=nlnet

Enfin, je vais continuer à publier des notes de progression (en anglais), qui devraient aussi être disponible sur ActivityPub plus tard dans l'année ;). Le site web a été mis à jour, avec de nouvelles installations via Flatpak et Docker, RDV sur https://www.salut-a-toi.org.

Si vous avez des questions ou commentaires, n'hésitez pas à vous rentre sur le salon officiel XMPP de Libervia à sat@chat.jabberfr.org, ou à me joindre par exemple via ActivityPub (@Goffi@mastodon.social).

À bientôt.

debacle 23/04/2021, 18:56

E

errormovim 06/05/2021, 11:13

ActivityPub Gateway and Pubsub e2ee

goffi 23/04/2021, 10:21 jabber-xmpp-en SàT Libervia project libre ActivityPub e2ee

Hello,

it's my pleasure to announce that an ActivityPub <=> XMPP gateway doubled with Pubsub end-to-end encryption project has been selected for a grant by NLNet/NGI0 Discovery Fund (with financial support from European Commission's Next Generation Internet programme): https://nlnet.nl/project/Libervia/

This big project is split in 27 steps, and will take most of my time dedicated to the Libervia project (formerly "Salut à Toi", the project has been renamed, I'll explain that in a upcoming progress note).

The XMPP <=> ActivityPub gateway will join two major open and decentralised protocols. In practice it will be a XMPP server component (usable with any server), and implement the ActivityPub server to server protocol (or "Federation Protocol"). On XMPP side, it will be mostly a Pubsub service (with some extra, like private messages converted to XMPP message stanza).

XMPP blogging (XEP-0277: Microblogging over XMPP) will be used, and thus any client supporting it will have access to ActivityPub publications (Libervia and Movim for instance).

For features present in ActivityPub and not yet in XMPP, it is planned to propose protoXEPs (i.e. proposition of XMPP extensions), to implement them. Events will also be part of the project, with a compatibility between Mobilizon and Libervia expected, and a protoXEP to have this standardised on XMPP side.

This is quite exiting, as it will extend both networks, and boost projects integrating blogging and XMPP chat.

The second part of the project is about end-to-end encryption. XMPP has enjoyed major improvements on end-to-end encryption following the work done on OMEMO, notably initiated with Conversations, and on OX, modern OpenPGP integration. This is great, but has been so far mainly focusing on instant messaging. The goal will be here to add end-to-end encryption to XMPP Pubsub, which includes protoXEPs and implementation in Libervia. In other terms, at the end of this project, it will be possible to use e2ee with all Pubsub based features (like blogs, forums, lists, or events that you can do on Libervia), this is huge! Signing will be part of the project too, meaning that it will be possible to authenticate something like a blog post in a standardised way.

Beside the standards which will benefit to the whole XMPP community, all of this will be implemented in Libervia, this include updating current implementation to the state of the art (i.e. updating current OMEMO implementation and implementing OX).

Last part of the project will be the implementation of e2ee in the web frontend. Due to Libervia specific architecture, OMEMO is not currently usable from the browser (the implementation is done on the backend). To make this possible, the Python OMEMO library which is currently used will be ported to WebAssembly and Brython, allowing to do encryption and decryption directly within the browser.

As you can see this is massive. I'll do most of this but I won't be alone (notably the author of Python OMEMO will do the wasm/Brython port as part of the project). The project should last circa one year, with the ActivityPub <=> XMPP gateway being the first part worked on.

I would like to thanks again NLNet and EU's NGI for allowing this, and my employer (Sourcefabric, which notably develops Superdesk) for letting me adapt my working schedule.

I've adapted the tasks to Libervia's (XMPP powered) bug tracker, so you can see step details and follow progress there: https://salut-a-toi.org/bugs?search=nlnet

Also I'll continue to publish progress notes so stay tuned to this blog (that should be available also on ActivityPub later this year ;) ). The website has been updated, with new Flatpak and Docker installations, check https://www.salut-a-toi.org.

If you have questions or comments, feel free to join Libervia's XMPP room at sat@chat.jabberfr.org or to contact me for instance via ActivityPub (@Goffi@mastodon.social).

See you soon.

debacle 23/04/2021, 18:57

E

errormovim 06/05/2021, 11:13

S

sthaan 14/11/2021, 19:20