danigm.net - gnome

Take it easy. A guide to avoid burnown during the Vulnpocalypse

danigm — Fri, 05 Jun 2026 12:00:00 +0200

Do not let the AI to remove the fun part from software development. We shouldn't allow gen AI to write software just because it "can". First, we must ask if it "should" do it, and even then, we should ask if we want to delegate the fun part, the thinking, the writing, the learning.

Remember what's important, journey before destination, we are the Code:

Do not let AI to destroy the community, do not let it destroy the technological knowledge commons.

tl;dr

Open Source maintainers are dealing with a lot of new reports and pressure to "fix" the project due to generative AI.

We need to find a way of stopping this and get back to something maintainable before all maintainers get burned out and look for a job in a farm:

100% secure software doesn't exists, so there will be always a possible CVE there. As Spaf said in 1989:

The only truly secure system is one that is powered off, cast in a block of concrete and sealed in a lead-lined room with armed guards - and even then I have my doubts.

Fixing bugs, adds new bugs, and if you need to fix something quick, the probability of new bugs will be higher. Do not forget about the First Law of Programming:

If it works, don't touch it

The amount of CVE reports is lowering the CVE credibility and quality, so if everything is a "high" security issue, we can't prioritize now and these reports are not different from random issues in github. Do not listen to The Boy Who Cried Wolf
Stable software is sable because it doesn't change too much. It's something that we are willing to loose trying to reach the impossible of 100% secure software?

The actual problem

There's a lot of money in AI tech right now, and everyone is trying to make the best gen AI tool or just pretend that their tool is the best.

In relation with the software analysis and writing, targeting the open source is the obvious strategy.

It's interesting to scrap every line of code, patch, pull request, issue and discussion around software to train your model, so AI scrappers are DDoSing open source projects infrastructure.
To promote their tools or themselves, Security Researches are using AI to target any project, reporting High security vulnerabilities, with the only goal of getting a CVE number to say how good they are.

This second point is affecting maintainers, because now you are receiving a lot of poor quality security reports, that are generated with AI and that looks plausible and are hard to read. You need to spend a lot of time to check if there's an actual wolf there or if it's again this boy that's tricking me.

This is burning the energy of maintainers, that instead of doing something productive are wasting their limited time talking with a Stocatic Parrot.

Do not let the AI Bros to use classic manipulation techniques on you!

A lot of open source projects are maintained by volunteers that do the work with passion and love. And even if it's the job that paid your bills, the maintainer can feel the pressure. When someone put a lot of love in something and work on it during years, it's part of his identity, so attacking the software is like attacking the person behind it.

This is nothing new, and a lot of people take advantage of this emotional link to manipulate the maintainer to do something that he do not want to do.

AI bros are using these techniques, do not let them to manipulate you and define your project agenda.

Here's a (not complete) list of known manipulation techniques that you can detect (and disarm!) in your daily community work:

Flooding the queue. Just create so many new issues that the actual maintainers can't deal with it. You feel responsible for the project and feel bad because your TO-DO list is growing.
This software is not secure (doesn't do what I want), I will use this other one instead that's better. The classic, "GNOME doesn't allow me to change this specific preference, I'll use KDE from now on".
This software is low quality, it doesn't follow the (my random) quality standards. Direct attack to the maintainer self-esteem.
Gaslighting software development. LLM are expert at this and people that uses it just copy the tactic. When the maintainer detects something weird and just tries to blame the other person for reporting nonsense and wasting all people time, it starts to invent new arguments and ignore the previous interaction.

So, take it easy, and remember the best clause in almost any software project, THE ENTIRE RISK AS TO THE QUALITY AND PERFORMANCE OF THE PROGRAM IS WITH YOU:

Disclaimer of Warranty.

THERE IS NO WARRANTY FOR THE PROGRAM, TO THE EXTENT PERMITTED BY
APPLICABLE LAW. EXCEPT WHEN OTHERWISE STATED IN WRITING THE COPYRIGHT
HOLDERS AND/OR OTHER PARTIES PROVIDE THE PROGRAM “AS IS” WITHOUT
WARRANTY OF ANY KIND, EITHER EXPRESSED OR IMPLIED, INCLUDING, BUT NOT
LIMITED TO, THE IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR
A PARTICULAR PURPOSE. THE ENTIRE RISK AS TO THE QUALITY AND
PERFORMANCE OF THE PROGRAM IS WITH YOU. SHOULD THE PROGRAM PROVE
DEFECTIVE, YOU ASSUME THE COST OF ALL NECESSARY SERVICING, REPAIR OR
CORRECTION.

Is the software more insecure in 2026?

No. Anyone old enough could remember how insecure old software was. Do you remember windows 98? Do you remember the internet when everything was http (without that little s at the end), when people use ftp to logging into their server and modify the php code directly on production?

It's true that today we have more dependency on technology, but it's also true that everything is more secure, we have more and better cryptography, we have different levels of isolation, virtual environments, containers, virtual machines...

But we have the feeling that since AI can analyse all the software and look for vulnerabilities, we are doomed, because any stupid kid can hack my over engineered GNU/Linux machine!

First, that's not true, you need to know about security to get something useful from any AI tool. But even if it was true, what can you do about it? We need to be practical and find a balance between risk and usefulness, so do not overestimate the risk just because everyone is talking about it right now.

But even then, the security paranoia is not good for anyone. Software is inherently buggy, people write software and makes mistakes, so a possible vulnerability appears. In theory, these bugs are fixed when discovered, so it's always recommended to update to the latest version, because almost all known bugs will be fixed.

But it's also known that new versions comes with new functionality and code, and that means new "unknown" bugs or different behavior. That's a headache, so that's why the stable and Long Term Support are popular distributions, because "if it works, don't touch it".

Stable packages just get the fixes, not new features, but fixes are also code changes, so there's always a possibility to break something, even with a patch update.

The stable software has a lot of value, do not let the AI security paranoia destroy that, and convert everything in a rolling release with the latest and greatest (and possibly broken) software. Sometimes it's better to keep using something old, with known vulnerabilities that you can mitigate, than use the latest with unknown new vulnerabilities that you can't do anything about.

I will fight AI with AI

Please, do not do that. What I was trying to argue during this long post is not a technical problem. The current burnout problem in open source is a social problem, you can't fix it with a new layer of probabilistic tokens.

Community reaction against AI. The current industry push for the usage of AI everywhere is affecting a lot of people, and as a reaction a lot of people are directly fighting back. Using gen AI just sends the message that you do not care enough to do it yourself, and destroy the trust on the project.
It doesn't worth it. Even if the AI works (that it doesn't) it doesn't worth it. Writing code is easier than reviewing, you learn and grow with every new line of code that you write, delegating the fun part and personal growth part to an AI will make you work more miserable and you will be a junior forever.
It doesn't create community. Think about it, it's hard to get someone involved in a software project, but who will want to read or improve the code produced by a gen AI? The only future collaborator will be another AI.

Take it easy

Just remember, you can always say no, there's no hurry, and there's no need to work on something that you don't want just because other people consider that important.

Free Source is something done by people, for people. The software is important, but the community around it is sometimes more important. We use Free source not because it's technically better (that it is), but because we trust who, how and why are writing it.

Remember why are you doing this, do not remove the Fun part, continue with the Just for Fun mood.

GNOME Tour in openSUSE and welcome app

danigm — Tue, 21 Oct 2025 12:00:00 +0200

As a follow up of the Hackweek 24 project, I've continued working on the gnome-tour fork for openSUSE with custom pages to replace the welcome application for openSUSE distributions.

GNOME Tour modifications

All the modifications are on top of upstream gnome-tour and stored in the openSUSE/gnome-tour repo

Custom initial page

A new donations page. In openSUSE we remove the popup from GNOME shell for donations, so it's fair to add it in this place.

Last page with custom openSUSE links, this one is the used for opensuse-welcome app.

opensuse-welcome package

The original opensuse-welcome is a qt application, and this one is used for all desktop environments, but it's more or less unmaintained and looking for a replacement, we can use the gnome-tour fork as the default welcome app for all desktop without a custom app.

To do a minimal desktop agnostic opensuse-welcome application, I've modified the gnome-tour to also generate a second binary but just with the last page.

The new opensuse-welcome rpm package is built as a subpackage of gnome-tour. This new application is minimal and it doesn't have lots of requirements, but as it's a gtk4 application, it requires gtk and libadwaita, and also depends on gnome-tour-data to get the resoures of the app.

To improve this welcome app we need to review the translations, because I added three new pages to the gnome-tour and that specific pages are not translated, so I should regenerate the .po files for all languages and upload to openSUSE Weblate for translations.

Log Detective: GSoC 2025 (part 2)

danigm — Wed, 27 Aug 2025 07:00:00 +0200

This week is the last week of Google Summer of Code for this year edition, and I'll do a summary of what we have been doing and future plans for the Log Detective integration in openSUSE.

I wrote a blog post about this project when it starts, so if you didn't read, you can take a look.

All the work and code was done in the logdetective-obs github repository.

Aazam, aka the intern, also wrote some blog post about his work.

Initial Research

The idea of the project was to explore ways of integration of LogDetective with the openSUSE dev workflow. So we started collecting some build failures in the openSUSE build service and testing with log detective to check if the output is smart or even relevant.

The intern creates a script to collect log from build failures and we store the LLM answer.

Doing this we detected that the log-detective.com explain is very slow and the local tool is less accurate.

The results that we got weren't too good, but at this point of the project is something expected. The model is being trained and will improve with every new log that users send.

Local vs Remote, model comparison

The logdetective command line tool is nice, but LLM requires a lot of resources to run locally. This tool also uses the models published by fedora in huggingface.co, so it's not as accurate as the remote instance that has a better trained model and is up to date.

In any case, the local logdetective tool is interesting and, at this moment, it's faster than the deployed log-detective.com.

Using this tool, Aazam did some research, comparing the output with different models.

logdetective apache-arrow_standard_x86_64.log\
  --model unsloth/Qwen2.5-Coder-7B-Instruct-128K-GGUF\
  -F Qwen2.5-Coder-7B-Instruct-Q4_K_M.gguf\
  --prompt ~/prompt.yaml

So using other specific models, the logdetective tool can return better results.

The plugin

openSUSE packagers uses the osc tool, to build packages in build.opensuse.org. This tool can be extended with plugins and we created a new plugin to use log-detective.

So a packager can get some AI explanation about a build failure with a simple command:

$ osc ld -r -p devel:languages:python --package python-pygame --repo openSUSE_Tumbleweed
🔍 Running logdetective for python-pygame (openSUSE_Tumbleweed/x86_64)...
Log url: https://api.opensuse.org/public/build/devel:languages:python/openSUSE_Tumbleweed/x86_64/python-pygame/_log
🌐 Sending log to LogDetective API...
 The RPM build process for the `python-pygame` package failed during the
`%check` phase, as indicated by the "Bad exit status from /var/tmp/rpm-
tmp.hiddzT (%check)" log snippet. This failure occurred due to seven test
failures, one error, and six skipped steps in the test suite of the package.



The primary issue causing the build failure seems to be the traceback error in
the `surface_test.py` file, which can be seen in the following log snippets:



[[  278s] Traceback (most recent call last):
[  278s]   File "/home/abuild/rpmbuild/BUILD/python-
pygame-2.6.1-build/BUILDROOT/usr/lib64/python3.11/site-
packages/pygame/tests/surface_test.py", line 284, in test_copy_rle
]
[[  278s] Traceback (most recent call last):
[  278s]   File "/home/abuild/rpmbuild/BUILD/python-
pygame-2.6.1-build/BUILDROOT/usr/lib64/python3.11/site-
packages/pygame/tests/surface_test.py", line 274, in
test_mustlock_surf_alpha_rle
]
[[  278s] Traceback (most recent call last):
[  278s]   File "/home/abuild/rpmbuild/BUILD/python-
pygame-2.6.1-build/BUILDROOT/usr/lib64/python3.11/site-
packages/pygame/tests/surface_test.py", line 342, in test_solarwolf_rle_usage
]



These traceback errors suggest that there are issues with the test functions
`test_copy_rle`, `test_mustlock_surf_alpha_rle`, and `test_solarwolf_rle_usage`
in the `surface_test.py` file. To resolve this issue, you should:



1. Identify the root cause of the traceback errors in each of the functions.
2. Fix the issues found, either by modifying the test functions or correcting
the underlying problem they are testing for.
3. Re-build the RPM package with the updated `surface_test.py` file.



It's also recommended to review the other test failures and skipped steps, as
they might indicate other issues with the package or its dependencies that
should be addressed.

Or if we are building locally, it's possible to run using the installed logdetective command line. But this method is less accurate, because the model used is not smart enough, yet:

$ osc build
[...]
$ osc ld --local-log --repo openSUSE_Tumbleweed
Found local build log: /var/tmp/build-root/openSUSE_Tumbleweed-x86_64/.build.log
🚀 Analyzing local build log: /tmp/tmpuu1sg1q0
INFO:logdetective:Loading model from fedora-copr/Mistral-7B-Instruct-v0.3-GGUF
...

Future plans

Adding submit-log functionality to osc-ld-plugin. In-progress. This will make it easier to collaborate with log-detective.com, allowing openSUSE packagers to send new data for model training.
Gitea bot. openSUSE development workflow is moving to git, so we can create a bot that comment on Pull Request with packages that fails to build. Something similar to what fedora people have for centos gitlab
Add a new tab to log-detective.com website to submit logs directly from OBS

This was an interesting Summer of Code project. I learned a bit about LLM. I think that this project has a lot of potential, this can be integrated in different workflows and an expert LLM can be a great tool to help packagers, summarizing big logs, tagging similar failures, etc.

We have a lot of packages and a lot of data in OBS, so we should start to feed the LLM with this data, and in combination with the fedora project, the log-detective could be a real expert in open source RPM packaging.

Log Detective: Google Summer of Code 2025

danigm — Mon, 09 Jun 2025 12:00:00 +0200

I'm glad to say that I'll participate again in the GSoC, as mentor. This year we will try to improve the RPM packaging workflow using AI, as part of the openSUSE project.

So this summer I'll be mentoring an intern that will research how to integrate Log Detective with openSUSE tooling to improve the packager workflow to maintain rpm packages.

Log Detective

Log Detective is an initiative created by the Fedora project, with the goal of

"Train an AI model to understand RPM build logs and explain the failure in simple words, with recommendations how to fix it. You won't need to open the logs at all."

As a project that was promoted by Fedora, it's highly integrated with the build tools around this distribution and RPM packages. But RPM packages are used in a lot of different distributions, so this "expert" LLM will be helpful for everyone doing RPM, and everyone doing RPM, should contribute to it.

This is open source, so if, at openSUSE, we want to have something similar to improve the OBS, we don't need to reimplement it, we can collaborate. And that's the idea of this GSoC project.

We want to use Log Detective, but also collaborate with failures from openSUSE to improve the training and the AI, and this should benefit openSUSE but also will benefit Fedora and all other RPM based distributions.

The intern

The selected intern is Aazam Thakur. He studies at University of Mumbai, India. He has experience in using SUSE as he has previously worked on SLES 15.6 during his previous summer mentorship at OpenMainFrame Project for RPM packaging.

I'm sure that he will be able to achieve great things during these three months. The project looks very promising and it's one of the things where AI and LLM will shine, because digging into logs is always something difficult and if we train a LLM with a lot of data it can be really useful to categorize failures and give a short description of what's happening.

Python 2

danigm — Mon, 27 Jan 2025 12:00:00 +0100

In 2020, the Python foundation declared Python 2 as not maintained anymore.

Python 2 is really old, not maintained and should not be used by anyone in any modern environment, but software is complex and python2 still exists in some modern Linux distributions like Tumbleweed.

The past week the request to delete Python 2 from Tumbleweed was created and is going through the staging process.

The main package keeping Python 2 around for Tumbleweed was Gimp 2, that doesn't depends directly on Python 2, but some of the plugins depends on it. Now that we've Gimp 3 in Tumbleweed, we are able to finally remove it.

Python 2

The first version of Python 2 was released around 2000, so it's now 25 years old. That's not true, because software is a living creature, so as you may know, Python 2 grew during the following years with patch and minor releases until 2020 that was the final release 2.7.18.

But even when it was maintained until 2020, it was deprecated for a long time so everyone "should" have time to migrate to python 3.

Py3K

I started to write python code around the year 2006. I was bored during a summer internship at my third year of computer science, and I decided to learn something new. In the following months / years I heard a lot about the futurist Python 3000, but I didn't worry too much until it was officially released and the migration started to be a thing.

If you have ever write python2 code you will know about some of the main differences with python3:

print vs print()
raw_input() vs input()
unicode() vs str
...

Some tools appeared to make it easier to migrate from python2 to python3, and even it was possible to have code compatible with both versions at the same time using the __future__ module.

You should have heard about the six package, 2 * 3 = 6. Maybe the name should be five instead of six, because it was a Python "2 and 3" compatibility library.

Python in Linux command line

When python3 started to be the main python, there were some discussion about how to handle that in different Linux distributions. The /usr/bin/python binary was present and everyone expect that to be python2, so almost everyone decided to keep that relation forever and distribute python3 as /usr/bin/python3, so you can have both installed without conflicts and there's no confusion.

But python is an interpreted language, and if you have python code, you can't tell if it's python2 or python3. The shebang line in the executable python scripts should point to the correct interpreter and that should be enough like #!/usr/bin/python3 will use the python3 interpreter and #!/usr/bin/python will use python2.

But this is not always true, some distributions uses python3 in /usr/bin/python like Archlinux or if you create a virtualenv with python3, the python binary points to the python3 interpreter, so a shebang like #!/usr/bin/python could be something valid for a python3 script.

In any case, the recommended and safest way is to always use python3 binary because that way it'll work correctly "everywhere".

Goodbye

It's time to say goodbye to python2, at least we can remove it now from Tumbleweed. It'll be around for some more time in Leap, but it's the time to let it go.

Hackweek 24

danigm — Fri, 22 Nov 2024 12:00:00 +0100

It's the time for a new Hack Week. The Hack Week 24 was from November 18th to November 22th, and I've decided to join the New openSUSE-welcome project this time.

The idea of this project is to revisit the existing openSUSE welcome app, and I've been trying to help here, specifically for the GNOME desktop installation.

openSUSE-welcome

Right now after installing any openSUSE distribution with a graphical desktop, the user is welcomed on first login with a custom welcome app.

This custom application is a Qt/QML with some basic information and useful links.

The same generic application is used for all desktops, and for popular desktops right now exists upstream applications for this purpose, so we were talking on Monday morning about it and decided to use specific apps for desktops.

So for GNOME, we can use the GNOME Tour application.

gnome-tour

GNOME Tour is a simple rust/gtk4 application with some fancy images in a slideshow.

This application is generic and just shows information about GNOME desktop, so I created a fork for openSUSE to do some openSUSE specific customization and use this application as openSUSE welcome in GNOME desktop for Tumbleweed and Leap.

Desktop patterns, the welcome workflow

After some testing and investigation about the current workflow for the welcome app:

x11_enhanced pattern recommends opensuse-welcome app.
We can add a Recommends: gnome-tour to the gnome pattern
The application run using xdg autostart, so gnome-tour package should put the file in /etc/xdg/autostart and set to hidden on close.
In the case of having a system with multiple desktops, we can choose the specific welcome app using the OnlyShowIn/NotShowIn config in desktop file

So I've created a draft PR to do not show the openSUSE-welcome app in GNOME, and I've also the gnome-tour fork in my home OBS project.

I've been testing this configuration in Tumbleweed with GNOME, KDE and XFCE installed and it works as expected. The openSUSE-welcome is shown in KDE and XFCE and the gnome-tour app is only shown in GNOME.

Next steps

The next steps to have the GNOME Tour app as default welcome for openSUSE GNOME installation are:

Send forked gnome-tour package to GNOME:Next project in OBS.
Add the Recommends: gnome-tour to patterns-gnome to GNOME:Next project in OBS.
Make sure that any other welcome application is not shown in GNOME.
Review openQA tests that expect opensuse-welcome and adapt for the new application.

rpmlint: Google Summer of Code 2024

danigm — Thu, 30 May 2024 00:00:00 +0200

I'm glad to say that I'll participate again in the GSoC, as mentor. This year we will continue the work done during the past year, as part of the openSUSE project.

So this summer I'll be mentoring an intern and we'll continue working on improving the testing framework of the rpmlint project.

This year we've a better testing framework, thanks to the work done during the past Summer of Code, by Afrid. So the goal for this year is to try to modernize existing tests and remove as much files as possible from test/binary directory, replacing those with mock packages defined with python code.

The selected intern is Luz Marina Montilla Marín. She has done some initial work in the rpmlint project, creating the mock packages for some tests and we've just started with the work to do during the GSoC program, evaluating the tests that we've right now and planning were to start.

She studies at Córdoba, Spain, my hometown. Every year I try to reach young people at different local universities, here in Andalucía, and sometimes I'm able to convince some students to participate, like the GSoC 2020, when Alejandro Dominguez, from Seville, were working on Fractal. So I'm happy that I'm increasing the number of free software developers in my local community :D

I'm sure that she will be able to achieve great things during these three months, so I'm looking forward to start to code and see how far can we go.

Python 3.13 Beta 1

danigm — Wed, 22 May 2024 00:00:00 +0200

Python 3.13 beta 1 is out, and I've been working on the openSUSE Tumbleweed package to get it ready for the release.

Installing python 3.13 beta 1 in Tumbleweed

If you are adventurous enough to want to test the python 3.13 and you are using openSUSE Tumbleweed, you can give it a try and install the current devel package:

# zypper addrepo -p 1000 https://download.opensuse.org/repositories/devel:languages:python:Factory/openSUSE_Tumbleweed/devel:languages:python:Factory.repo
# zypper refresh
# zypper install python313

What's new in Python 3.13

Python interpreter is pretty stable nowadays and it doesn't change too much to keep code compatible between versions, so if you are writing modern Python, your code should continue working whit this new version. But it's actively developed and new versions have cool new functionalities.

New and improved interactive interpreter, colorized prompts, multiline editing with history preservation, interactive help with F1, history browsing with F2, paste mode with F3.
A set of performance improvements.
Removal of many deprecated modules: aifc, audioop, chunk, cgi, cgitb, crypt, imghdr, mailcap, msilib, nis, nntplib, ossaudiodev, pipes, sndhdr, spwd, sunau, telnetlib, uu, xdrlib, lib2to3.

Enabling Experimental JIT Compiler

The python 3.13 version will arrive with an experimental functionality to improve performance. We're building with the --enable-experimental-jit=yes-off so it's disabled by default but it can be enabled with a virtualenv before launching:

$ PYTHON_JIT=1 python3.13

Free-threaded CPython

The python 3.13 has another build option to disable the Global Interpreter Lock (--disable-gil), but we're not enabling it because in this case it's not possible to keep the same behavior. Building with disabled-gil will break compatibility.

In any case, maybe it's interesting to be able to provide another version of the interpreter with the GIL disabled, for specific cases where the performance is something critical, but that's something to evaluate.

We can think about having a python313-nogil package, but it's not something trivial to be able to have python313 and python313-nogil at the same time in the same system installation, so I'm not planning to work on that for now.

Where's my python code?

danigm — Sat, 03 Feb 2024 00:00:00 +0100

Python is a interpreted language, so the python code are just text files with the .py extension. For simple scripts it's really easy to have your files located, but when you starts to use dependencies and different projects with different requirements the thing starts to get more complex.

PYTHONPATH

The Python interpreter uses a list of paths to try to locate python modules, for example this is what you can get in a modern GNU/Linux distribution by default:

Python 3.11.7 (main, Dec 15 2023, 10:49:17) [GCC] on linux
Type "help", "copyright", "credits" or "license" for more information.
>>> import sys
>>> sys.path
['',
 '/usr/lib64/python311.zip',
 '/usr/lib64/python3.11',
 '/usr/lib64/python3.11/lib-dynload',
 '/usr/lib64/python3.11/site-packages',
 '/usr/lib64/python3.11/_import_failed',
 '/usr/lib/python3.11/site-packages']

These are the default paths where the python modules are installed. If you install any python module using your linux packaging tool, the python code will be placed inside the site-packages folder.

So system installed python modules can be located in:

/usr/lib/python3.11/site-packages for modules that are architecture independent (pure python, all .py files)
/usr/lib64/python3.11/site-packages for modules that depends on the arquitecture, that's something that uses low level libraries and needs to build so there are some .so files.

pip

When you need a new python dependency you can try to install from your GNU/Linux distribution using the default package manager like zypper, dnf or apt, and those python files will be placed in the system paths that you can see above.

But distributions doesn't pack all the python modules and even if they do, you can require an specific version that's different from the one packaged in your favourite distribution, so in python it's common to install dependencies from the Python Package Index (PyPI).

Python has a tool to install and manage Python packages that looks for desired python modules in PyPI.

You can install new dependencies with pip just like:

$ pip install django

And that command looks for the django python module in the PyPI, downloads and install it, in your user $HOME/.local/lib/python3.11/site-packages folder if you use --user, or in a global system path like /usr/local/lib or /usr/lib if you run pip as root.

But the usage of pip directly in the system is something not recommended today, and even it's disabled in some distributions, like openSUSE Tumbleweed.

[danigm@localhost ~] $ pip install django
error: externally-managed-environment

× This environment is externally managed
╰─> To install Python packages system-wide, try
    zypper install python311-xyz, where xyz is the package
    you are trying to install.

    If you wish to install a non-rpm packaged Python package,
    create a virtual environment using python3.11 -m venv path/to/venv.
    Then use path/to/venv/bin/python and path/to/venv/bin/pip.

    If you wish to install a non-rpm packaged Python application,
    it may be easiest to use `pipx install xyz`, which will manage a
    virtual environment for you. Install pipx via `zypper install python311-pipx` .

note: If you believe this is a mistake, please contact your Python installation or OS distribution provider. You can override this, at the risk of breaking your Python installation or OS, by passing --break-system-packages.
hint: See PEP 668 for the detailed specification.

virtualenvs

Following the current recommendation, the correct way of installing third party python modules is to use virtualenvs.

The virtualenvs are just specific folders where you install your python modules and some scripts that make's easy to use it in combination with your system libraries so you don't need to modify the PYTHONPATH manually.

So if you've a custom project and want to install python modules you can create your own virtualenv and use pip to install dependencies there:

[danigm@localhost tmp] $ python3 -m venv myenv
[danigm@localhost tmp] $ . ./myenv/bin/activate
(myenv) [danigm@localhost tmp] $ pip install django
Collecting django
...
Successfully installed asgiref-3.7.2 django-5.0.1 sqlparse-0.4.4

So all dependencies are installed in my new virtualenv folder and if I use the python from the virtualenv it's using those paths, so all the modules installed there are usable inside that virtualenv:

(myenv) [danigm@localhost tmp] $ ls myenv/lib/python3.11/site-packages/django/
apps  contrib  db        forms  __init__.py  middleware   shortcuts.py  templatetags  urls   views
conf  core     dispatch  http   __main__.py  __pycache__  template      test          utils
(myenv) [danigm@localhost tmp] $ python3 -c "import django; print(django.__version__)"
5.0.1
(myenv) [danigm@localhost tmp] $ deactivate

With virtualenvs you can have multiple python projects, with different dependencies, isolated, so you use different dependencies when you activate your desired virtualenv:

activate $ . ./myenv/bin/activate
deactivate $ deactivate

High level tools to handle virtualenvs

The venv module is a default Python module and as you can see above, it's really simple to use, but there are some tools that provides some tooling around it, to make it easy for you, so usually you don't need to use venv directly.

pipx

For final python tools, that you are not going to use as dependencies in your python code, the recommended tool to use is pipx.

The tool creates virtualenv automatically and links the binaries so you don't need to worry about anything, just use as a way to install third party python applications and update/uninstall using it. The pipx won't mess your system libraries and each installation will use a different virtualenv, so even tools with incompatible dependencies will work nicely together in the same system.

Libraries, for Python developers

In the case of Python developers, when you need to manage dependencies for your project, there are a lot of nice high level tools for managing dependencies.

These tools provides different ways of managing dependencies, but all of them relies in the use of venv, creating the virtualenv in different locations and providing tools to enable/disable and manage dependencies inside those virtualenvs.

For example, poetry creates virtualenvs by default inside the .cache folder, in my case I can find all poetry created virtualenvs in:

/home/danigm/.cache/pypoetry/virtualenvs/

Most of these tools add other utilities on top of the dependency management. Just for installing python modules easily you can always use default venv and pip modules, but for more complex projects it's worth to investigate high level tools, because it'll make easy to manage your project dependencies and virtualenvs.

Conclusion

There are a lot of python code inside any modern Linux distribution and if you're a python developer it's possible to have a lot of python code. Make sure to know the source of your modules and do not mix different environments to avoid future headaches.

As a final trick, if you don't know where's the actual code of some python module in your running python script, you can always ask:

>>> import django
>>> django.__file__
'/tmp/myenv/lib64/python3.11/site-packages/django/__init__.py'

This could be even more complicated if you start to use containers and different python versions, so keep you dependencies clean and up to date and make sue that you know where is your Python code.

Hackweek 23

danigm — Fri, 10 Nov 2023 00:00:00 +0100

Hack Week is the time SUSE employees experiment, innovate & learn interruption-free for a whole week! Across teams or alone, but always without limits.

The Hack Week 23 was from November 6th to November 10th, and my project was to gvie some love to the GNOME Project.

Before the start of the Hack week I asked in the GNOME devs Matrix channel, what project needs some help and they gave me some ideas. At the end I decided to work on the GNOME Calendar, more specifically, improving the test suite and fixing issues related to timezones, DST, etc.

GNOME Calendar

GNOME Calendar is a Gtk4 application, written in C, that heavily uses the evolution-data-server library. It's a desktop calendar application with a modern user interface that can connect handle local and remote calendars. It's integrated in the GNOME desktop.

The current gnome-calendar project has some unit tests, using the GLib testing framework. But right now there are just a few tests, so the main goal right now is to increase the number of tests as much as possible, to detect new problems and regressions.

Testing a desktop application is not something easy to do. The unit tests can check basic operations, structures and methods, but the user interaction and how it's represented is something hard to test. So the best approach is to try replicate user interactions and check the outputs.

A more sophisticated approach could be to start to use the accessibility stack in tests, so it's possible to verify the UI widgets output without the need of rendering the app.

With gnome-calendar there's another point of complexity for tests because it relies on the evolution-data-server to be running, the app communicates with it using dbus, so to be able to do more complex tests we should mock the evolution-data-server and we should create fake data for testing.

My contribution

By the end of the week I've created four Merge requests, three of them have been merged now, and I'll continue working on this project in the following weeks/months.

I'm happy with the work that I was able to do during this Hack Week. I've learned a bit about testing with GLib in C, and a lot about the evolution-data-server, timezones and calendar problems.

It's just a desktop calendar application, how hard it could be? Have you ever deal with dates, times and different regions and time zones? It's a nightmare. There are a lot of edge cases working with dates that can cause problems, operations with dates in different time zones, changes in dates for daylight saving, if I've an event created for October 29th 2023 at 2:30 it will happens two times?

A lot of problems could happen and there are a lot of bugs reported for gnome-calendar related to this kind of issues, so working on this is not something simple, it requires a lot of edge case testing and that's the plan, to cover most of them with automated tests, because any small change could lead to a bug related to time zones that won't be noticed until someone has an appointment at a very specific time.

And this week was very productive thanks to the people working on gnome-calendar. Georges Stavracas reviews my MR very quickly and it was possible to merge during the week, and Jeff Fortin does a great work with the issues in gitlab and leading me to most relevant bugs.

So after a week of going deep into the gnome-calendar source code it could be a pity to just forget about it, so I'll try to keep the momentum and continue working on this project, of course, I'll have just a few hours per week, but any contribution is better than nothing. And maybe for the next summer I can propose a Google Summer of Code project to get an intern working on this full time.

Updating GNOME shell extensions to GNOME 45

danigm — Fri, 03 Nov 2023 00:00:00 +0100

The new version of the GNOME desktop was released more than one month ago. It takes some time to arrive to the final user, because distributions should integrate, tests and release the new desktop, and that's not something simple, and it should integrate in the distribution release planning.

So right now, it's possible that there's just a few people with the latest version of the desktop right now, just people with rolling release distros or people using testing versions of mayor distributions.

This is one of the reasons because a lot of gnome-shell extensions aren't updated to work with the latest version of GNOME, even after a few months, because even developers doesn't have the latest version of the desktop and it's not something "easy" to install, without a virtual machine or something like that. Even if the update is just a change in the metadata.json, there should be someone to test the extension, and even someone to request this update, and that will happen once the mayor distributions release a new version.

I'm using Tumbleweed, that's a rolling release and GNOME 45 is here just after the official release, but of course, a lot of extensions are not working, and in the infamous list of non working extensions there where the three that I'm maintaining right now:

mute-spotify-ads, extension to mute the spotify app when it's playing ads.
calc, a simple calculator in the alt+F2 input.
hide-minimized, hide minimized windows from alt+tab and overview.

The correct way

The correct way to maintain and update a gnome-shell extension should be to test and update "before" the official release of GNOME. It should be something easy for me, using Tumbleweed I can just add the GNOME Next repository, install the new desktop when it's in beta stage, and update the extension there.

But that's something that require an active maintainership... And right now what I do is to update the extensions when they are broken for me, so just when I need them, and that's after I get the new desktop and I find some time to update.

I know that's not the correct way and this produces a bad experience for other people using the extensions, but this is the easier thing to do for me. Maybe in the future I can do it correctly, and provide a tested update before the official release, maybe using snapper to be able to go back to stable, without the need of using virtual machines.

Update your extension to GNOME 45

The update to GNOME 45 was a bit more complex than previous ones. This version of the shell and gjs change a lot of things, so the migration it's not just to add a new number to the metadata.json, but requires incompatible changes, so extensions that works for GNOME 45 won't work for previous versions and vice versa.

But the people working in gnome-shell does a great work documenting and there's a really nice guide about how to upgrade your extension.

The most important part is the import section, that now it has a different syntax.

// Before GNOME 45
const Main = imports.ui.main;

// GNOME 45
import * as Main from 'resource:///org/gnome/shell/ui/main.js';

And some changes in the extension class, that now can inherit from existing ones to provide common usage, like preferences window:

import Adw from 'gi://Adw';

import {ExtensionPreferences, gettext as _} from 'resource:///org/gnome/Shell/Extensions/js/extensions/prefs.js';

export default class MyExtensionPreferences extends ExtensionPreferences {
    fillPreferencesWindow(window) {
        window._settings = this.getSettings();

        const page = new Adw.PreferencesPage();

        const group = new Adw.PreferencesGroup({
            title: _('Group Title'),
        });
        page.add(group);

        window.add(page);
    }
}

One year of Tumbleweed

danigm — Fri, 06 Oct 2023 00:00:00 +0200

More than a year has passed since I switched to openSUSE Tumbleweed Linux distribution, in both, my work computer (for obvious reasons) and in my personal computer and I can say that I'm really happy with the change.

Tumbleweed is a rolling release distribution, and in this kind of distributions there are a lot of changes every week, if you want the latest software, this kind of distribution is the way to go. But with high update frequency you are exposed to some kind of instability, it's impossible to have the latest changes without some broken program here and there, because not everyone is able to follow upstream changes without some weeks or months to update.

My distro history

I've been always a Linux user, since I get my first computer at 2003. In those days I was using a debian base distribution called knoppix. Then I switched to ubuntu when it appeared around 2004. But at that time I was a computer science student and I was exploring the whole free software and Linux ecosystem, so I was changing my distribution every time that I found a new one.

Like a lot of distro-hoppers, at some point I landed at ArchLinux and there I discovered the rolling release concept. And that was my home for some time, it was nice to have the latest available software just after the release.

At some point I bough a new computer and it was too new to work correctly with the kernel distributed in ArchLinux, so I tried different distributions and at that moment Fedora was the distro that works without too much complications with that computer, so I picked that one.

In 2019 I started to work at Endless and at that time I should try the EndlessOS, so I played a bit with the dual boot, having Fedora and EndlessOS at the same time. That was the first time that I get in contact with immutable distributions, something that's getting more popular everyday, but this distributions rely a lot on containers (flatpak, podman) and, even being something that could work, as a software engineer, I don't feel comfortable enough needing a container with another distribution to do something that could be in my system.

In 2022 I started to work at SUSE and for the first time I tried the openSUSE distribution until today.

The Tumbleweed

Today I've three different computers with Tumbleweed running. One for work, Thinkpad T14s, one for personal usage, Dell inspiron 5490, and another one as a personal media server, Libre computer La-frite.

The best thing of having Tumbleweed for me is that I get the latest GNOME as soon as it's released. And another big thing for this distribution is how easy it's to fix something upstream thanks to the Open Build Service, but I work everyday with that, so I'm biased. For sure, any other community distribution has different ways to contribute, but I find this one easy enough.

Even being a rolling release distro, Tumbleweed doesn't break a lot. I can't say that it's stable, because the API of everything is broken everyday, but the distribution is tested for every release and at least some level of package compatibility check is done. That makes Tumbleweed a good distribution and I can update without fearing some weird package breakage.

I usually update my work and personal laptops once a week, and la-frite not so often, maybe every 6 months.

With the default installation, Tumbleweed uses btrfs with snapshots, and it's really easy to go back and forward using the snapper tool. So it's really easy to go back to a good state if the distribution is broken for some reason, and wait for a fix.

The problems that I found during this year

Some problems with the NVidia graphic card in my Dell laptop, some times the kernel and the driver were not working correctly. I had to use snapper to get the NVidia working again, but fixed a few days later.
Currently I'm having some random crashes because some bug with amdgpu and wayland and mutter, but it's not too annoying for me to go back, so I didn't use snapper this time and I'm facing this random crashes waiting for the fix.

Long live the Tumbleweed

So far so good. Tumbleweed is a nice distribution that I'm enjoying. It's not getting in the way and I can find almost anything that I need for work, programming, gaming, media, etc. I'm really happy with this distribution and it's the perfect distribution for people like me, that want to have the latest things.

I know that there are other openSUSE flavors that are interesting, like the immutable ones, Leap or the latest one Slowroll, but Tumbleweed is the one for me.

rpmlint updates (August 2023)

danigm — Wed, 30 Aug 2023 00:00:00 +0200

We are at the end of the summer and this means that this year Google Summer of code is ending.

The recent changes applied now in the main branch include:

Remove usage of pkg_resource because it's deprecated.
Fix elf binary check with ELF files with a prefix.
New check for python packages with multiple .pyc files for different python versions.
Improve the testing framework (merged the work done during the GSoC 2023)

Summer of Code 2023 updates

The summer of code is ending and the work done by Afrid was good enough to be merged, so I merged it the past week.

I'm really happy with the work done during the GSoC program, now we've a more simple way to define tests for rpmlint checks mocking the rpm, so it's not always needed to build a fake rpm binary for each new test. This will make a lot easier to create simple tests, so I hope that we can increase the code coverage using this new framework.

During this time Afrid has extended the FakePkg class, so it's possible now to define fake metadata and files with fake tags and attributes. It's not complete and it's not a simple task to replace all the rpm binaries used for tests, because the Pkg class and RPM tags is a complex thing, but the current state allow us to replace a lot of them. Afrid has replaced some of the tests that uses binaries, but in the following months we can continue working on this and replace more.

After this work, we can now start to use more the FakePkg class in tests, so another task that we can do is to provide some common fake pkgs to use in different tests and new checks, so now it's possible to create fake packages with dynamic random data, so we can extend tests with fuzz testing and maybe this will help to improve the tool reliability.

Conclusion

I've participated as mentor several times now in the summer of code, and outreachy, and almost always was a good experience. With the gnome foundation in previous programs and this year with opensuse. These two communities are very open to collaboration and makes the whole process really simple, for me as mentor, and also for the intern.

I want to congratulate Afrid, because it was nice to work with him during this summer, he has done a great work, not just technically, but communicating, asking and finding his own solutions without requiring a continuous guidance.

He is very passionate and looks like a nice person, so I hope that he will continue around the open source, it could be opensuse, rpmlint or any other community, but this kind of people is what you want to find in any community.

After many years collaborating with different free software communities, it's amazing that there are so many great people in every project, of course you can find toxic communities and people, but in my experience, that's usually just noise, there are a lot of nice people out there, doing a great work, and I'm happy that young people like Afrid can be part of the free software movement, because this is what makes the free software great, the people that is working on it.

So Thanks a lot to Google for another summer of code, thanks to SUSE for letting me, and encourage me, to mentor, and thanks to all the free software developers that are out there.

I encourage everyone to participate in this kind of programs, for interns, it's a good opportunity to learn and to make some money working on free software, for mentors it's an opportunity to get some help in your project and help newcomers to be part of the community.

Have a lot of fun!

Python in openSUSE Tumbleweed

danigm — Sat, 05 Aug 2023 00:00:00 +0200

openSUSE Tumbleweed is a rolling release distribution, so it's ideal for developers and users that like to have the bleeding edge software.

It's also really "stable" to be a rolling release, from time to time you can find a broken package because of one package is updated and another one is not compatible yet, but it's something that doesn't happen too often thanks to openqa tests, so you don't need to worry about a breaking system.

You can find the Python interpreter and a lot of python modules in every Linux distribution, but Tumbleweed does an interesting thing for Python.

Default Python version (python3 -> python-3.11)

If you don't worry about the python version, you can just rely on python3. In Tumbleweed python3 is not a real package, but the default python version provides python3, so depending on when you install python3 you will get a different package, if you install it today (August 2023) you'll get python311.

In your system you'll have the /usr/bin/python3 binary that points to the default python, so you don't need to worry about the current version, you'll have there the default Python version for the operating system.

In addition to the Python interpreter, you can find in the distribution a lot of python modules, but again you can use the default version, so, for example, if you want to install poetry, you just use zypper install python3-poetry and that will install the real package python311-poetry.

Multiple python versions (3.8, 3.9, 3.10, 3.11)

Besides the default Python, in Tumbleweed you can also find other supported Python interpreters. Right now you can find all the Python versions currently supported by the Python Foundation, from 3.8 to 3.11.

For Python 3.8 you'll only find the interpreter, because the python modules are not built anymore, but for all the other versions you can find almost the same modules.

All python modules that provide binaries uses the update-alternatives, so you can configure in your system the version that you want to use as default. For example, if you want to use the 3.9 version of poetry, having installed different versions you can decide what /usr/bin/poetry points to:

$ sudo zypper in python311-poetry python39-poetry
$ sudo update-alternatives --config poetry

/usr/bin/python3 is a link provided by the default python package, so you can't modify with update-alternatives, so if you want to use a different python version, make sure to do the correct call with the full name python3.9, and use the correct shebang in your python scripts, for example #!/usr/bin/env python3.9.

What happens when default changes?

This way of distributing Python interpreter and modules is useful, because you don't need to update your software to work with the latest Python version, if your software requires another version, you can just install and continue using it, even on a bleeding-edge distribution like Tumbleweed.

But this method has some problems. When the default Python interpreter is changed in the distribution, all packages that depends on python3 will be updated, and that works correctly. But if you have installed some python module using the python3 prefix, that package, and all dependencies, is not updated automatically.

For example, if you installed python3-poetry when python3.10 was the default system, and then the distribution updates the system Python to python3.11, you don't get the python311-poetry package by default, you'll need to install it again. This could break you software, because if you use python3 and the dependencies are not updated, you'll find that some dependencies are not installed after updating.

For that reason, when the default python is changed in Tumbleweed, if you've software that rely on python3 you should make sure to install dependencies again by hand. And also you can do a cleanup and remove all old version packages that you don't need anymore.

If you just want the latest version you can just do all at once with a simple script like this:

#!/bin/bash
# up-py-tw.sh

FROM=$1
TO=$2

if [ $# -lt 2 ]
then
    echo "Usage:   up-py-tw.sh FROM TO"
    echo "example: up-py-tw.sh 310 311"
    exit 0
fi

echo "Updating python packages from $FROM to $TO"

OLDP=$(zypper -q search -i python${FROM}-* | tail --lines +4 | cut --delimiter \| --fields 2)
NEWP=$(echo $OLDP | sed "s/python${FROM}/python${TO}/g")

sudo zypper in $NEWP
sudo zypper rm $OLDP

You should verify what will be installed and what will be removed to make sure that just the python related packages are removed. It's possible that other packages depend on a python version that's older than the system one and that's okay.

Beta version (3.12.0b4)

In Tumbleweed, right now you can find all the supported Python versions, but it's not just that. At this moment you can also find the beta version of the next Python interpreter.

$ sudo zypper in python312

So if you are adventurous and want to test some new feature in the next Python release, you can do it with the version provided there. This version could also be used by openSUSE packagers to test the packages before the Python version is released so it allow us to prepare everything for the release and it could be in the distribution earlier and more tested.

Development

All this multiple python versions is done at distribution level and usually there's only one source package that produces the python39-foo, python310-foo and python311-foo. The source package is usually called python-foo and can be found in the Python devel project.

These packages spec uses the python-rpm-macros to generate all the versions from one source. If you're a packager you can find more information on the openSUSE wiki about Python Packaging.

rpmlint updates (July 2023)

danigm — Thu, 06 Jul 2023 00:00:00 +0200

I'm spending some time every week working in the rpmlint project. The tool is very stable and the functionality is well defined, implemented and tested, so there's no crazy development or a lot of new functionalities, but as in all the software, there are always bugs to solve and things to improve.

The recent changes applied now in the main branch include:

Update the usage of rpm to not use old API.
Fixes for rpmdiff -v, check for NULL char, special macros in comments and spell checking of description in different languages.
Move all the metadata from setup.py to pyproject.toml.
Releasing rpmlint as pre-commit hook
Improvements to the PythonCheck in the dependency checking.

Summer of Code 2023 updates

The first month of the Summer of Code has passed and Afrid is doing a great job there. We've now a draft Pull Request with some initial changes that allow us to mock rpm packages in tests so it's easier to create new tests without the need of creating a binary package.

The first step done was to extend the existing FakePkg class to allow us to define package files and some package metadata.

Now he's working in replacing all of the test_python.py tests that uses binaries rpm to something that doesn't needed.

The idea is to replace as much tests as possible to reduce the number of rpm binaries and after that, provide helper functions, decorators and classes to make it easy to write tests, writing less code.

Roadmap

In any software project there's always room for improvements, fixes and enhancements. If the project is there for enough time, it's even more critical to modernize the code to reduce the technical debt.

My plan for 2023 is to improve the tests around rpmlint as much as possible. First with the GSoC project, making it easier to write more tests, improving the testing tools that we've. And after the summer, improving the test coverage.

There's also a tool that shares some of the ideas with rpmlint, spec-cleaner, it's also written in Python, so the next step, after the tests improvements will be to take a deep look into the code of these two tools and try to integrate in some way. Maybe it's possible to refactor the common code into an external module, maybe we can bring some ideas from spec-cleaner to rpmlint. Not sure yet, but that'll be my next step.

Don't forget that this is free software, so you can participate too! If you find any issue in rpmlint or have an idea to improve it, don't hesitate and create a new issue.

rpmlint: Google Summer of Code 2023

danigm — Tue, 16 May 2023 00:00:00 +0200

I'm glad to say that I'll participate again in the GSoC, as mentor. This year will be a bit different from the previous ones, because I'm not mentoring a GNOME project but a openSUSE project.

I started to work at SUSE the past year and with this new job I get involved in the openSUSE community and I started to contribute to rpmlint.

So this summer I'll be mentoring an intern and we'll work on improving the testing framework of the rpmlint project.

The rpmlint project is a command line tool to check rpm packages, the correctness of these kind of packages and warn the packagers about usual problems or good practices. It's widely used in all Linux distributions based on rpm, mainly SUSE and RedHat.

It is written in Python and uses pytest for testing the code. Right now there are a lot of .rpm binary packages, to check different functionality, but that way of testing makes a bit hard to write new tests and to maintain with changes. The idea of this GSoC project is to extend the testing framework of rpmlint to support an easy way of writting tests that doesn't require a real rpm, something that can mock what it's in the .rpm binary and try to replace some of the current binary tests with this new mock.

The selected intern is Afrid Hussain. He has done some initial work in the rpmlint project, solving some minor issues and we're now preparing the work to be done during the GSoC program. I'm sure that he will be able to achieve great things during these three months, so I'm looking forward to start to code and see how far can we go.

Hackweek 2023

danigm — Tue, 03 Jan 2023 00:00:00 +0100

Hack Week is the time SUSE employees experiment, innovate & learn interruption-free for a whole week! Across teams or alone, but always without limits.

This year the Hack Week was this week, the last week of January and for my first SUSE hack week I decided to work in something funny, LILS.

Linux Immersive Learning System (LILS)

I don't think that this is a good name, but don't focus on it. The main idea of this project is to create some basic machinery to be able to write "interactive" tutorials or games using the INK language.

This is not an original idea, indeed all I've done is something that's currently working on EndlessOS, and was the main idea behind the dead project Hack Computer, you can even take a look to the Hack app in flathub. But I wanted to work around this, and create something simpler, from scratch.

I wanted to build something simple, with just Python, and make it simple enough to be able to build other tools on top. The design is simple, an INK parser, with a simple game runner. In the INK script you can define commands, to do something special, and wait for events with listeners, to wait for an event in the OS to continue.

With this basic functionality it's possible to build different user interfaces for different environments. And the original idea was to make the commands and listeners something extensible with a simple API, but that's something that I have not done yet, it's all Python functions without extension point.

The code can be found in github.

The INK parser

The most complex part of this project is the INK language parser. The Ink parser is free software and there's a Linux version that you can use to parse and compile to json, but I wanted to create my own parser with Python.

I've spent most of the Hack Week time fighting with the parser and indeed was the most challenging and fun part, because I've not worked a lot with parsers and it's not something easy as pie 😛️.

I remember creating a java compiler long time ago, when I was in the Seville University, for the Language Processors course. We did that with ANTLR, so starting from that, and looking for a Python lib, I found the Lark project. So if you like regular expressions, writing a grammar is a lot more FUN.

At the end I was able to support some basic INK language with support for:

Text
Tag support
Options, with suppress text support
Knots, Stitches and Diverts
Include other .ink files
Variable definition and basic operations
Knots and Stitches automatic visiting count variables
Conditional options using variables

It still fails in some cases, the comments and TODO placed in between text is not detected correctly and there's a lot of complex stuff that's not supported yet, but with what's supported right now it's possible to create complex scripts with loops and complex game graphs, so it's good enough to build games just with it.

GNOME shell extension

To integrate with the system I've done a simple GNOME shell extension. The extension just shows the text as bubbles and options as buttons, it's really simple and I've no time to make it something ready to be used, but I was able to make something usable.

To be able to run the LILS python library from gjs I've created a simple dbus service that exposes the basic InkScript class functionality as a dbus API.

I was thinking about being able to change the desktop background, depending of the value of a background variable in the script and do something similar to play music and sounds, so it could be a cool game engine with some additions.

SUSE Hack Week

So this Hack week was really fun and I learned a lot. It's really great that SUSE does things like this, letting us work in different projects for a week, to learn, to grow or to just explore different paths.

The end of 2022

danigm — Fri, 30 Dec 2022 00:00:00 +0100

Is the end of 2022 and the beginning of 2023, so it's the right time to look back and see what great things happened during the year, and also it's the time to plan some new year resolutions.

My contribution to GNOME in 2022

I've been focused this year on the GNOME Translation Editor, migrating it to Gtk4. It's not ready yet, but I hope I will be able to have a working version soon™.

It started this summer with one intern from Outreachy, that did some initial work, building with the new Gtk4, and after the summer I started to clean the code and fixing issues, and lately, I'm trying to replace some deprecations for the next Gtk release.

so I'm working on the code, modernizing it, and replacing every usage of GtkDialog and GtkTreeView, and trying to follow the GNOME HIG as much as possible.

Live coding Streaming

I started the year spending a lot of time working on Gtk on live stream on Twitch, but after the summer I was spending less time... I didn't find the time or the energy to do it regularly.

Changes in the business

This year was also a year of changes in my work life. After the summer I left Endless, after three years of working there, to join SUSE.

I really enjoy my time at EndlessOS, the first years working on the Hack Computer and the last year working on the Endless Key. I really like the EndlessOS mission and what I was doing there, with a lot of great people.

But I was looking for something more in my career path, and I had the opportunity to join SUSE. Since September, I'm part of the Python packaging team at SUSE and I've to say that I really love this job. It's a bit different from what I have been doing before, but the maintainer life is something that I enjoy, having the opportunity to contribute to a lot of different projects and debugging and fixing random bugs is one of the most rewarding tasks to do (when you are able to find the problem).

Tumbleweed is now my favourite GNU/Linux distribution and the future of SUSE ALP looks really promising.

Changes in life

This year comes also with a lot of changes in my personal life too! At the beginning of the year I was able to bought a house, so I've now a permanent residency, after a lot of years of changing from one rent house to another one.

And this year was also the year that I was able to get the kickboxing black belt, after almost 8 years of practicing. This is the end of a learning path, and also the beginning of a different one, trying to master this fighting sport.

New year resolutions

2022 was great, but we can make 2023 even better, I'm looking always to learn and improve, so it's good to do some new year resolutions to try to complete during the next year.

Complete the Gtranslator migration to gtk4 and GNOME HIG improvements!
Be more regular with the live coding streams
Give some love to the PyGObject project
Do more serious kickboxing training and loose 10Kg
Play more chess games

That's all! Happy new year and Have a lot of fun...

SUSE is my new distribution (new job)

danigm — Sun, 18 Sep 2022 00:00:00 +0200

This week I've started to work at SUSE. I'll be working as Python Specialist, in the packaging team, so I will go back to work on packaging and distribution after more than ten years. My first job in 2008 was working on a Ubuntu based local distribution, Guadalinex, so packaging and distribution work is not something new for me.

Python

Python was the first language that I fell in love. I learned to write code with C and C++, but when I discovered Python, in 2006, I found a really nice language to be able to create amazing things really fast and with a great community behind.

I'm very happy for this new opportunity to be able to collaborate to the Python distribution in all the SUSE flavours, and also to be able to collaborate in the creation of one of the most famous and used Linux distributions.

Tumbleweed

As part of this job change I've also installed SUSE Tumbleweed for the first time. Tumbleweed is a rolling release distribution with the latests packages. In the past I was using other rolling releases distributions like Arch, but this one looks more user friendly.

I've not spent a lot of time here, but from the point of view of a GNOME developer, I can say that it's a great distribution for development with updated packages, and it looks "stable". You can choose the desktop to use on installation and the GNOME desktop is there without any customization that I've detected, so it looks like it's a good vanilla GNOME desktop distribution.

Endless, it's not the end

I'm not working for EndlessOS now, but it's not the end. I've been working here for almost 4 years. At first I worked on the Hack Computer and after that project didn't work, I was working on the Endless Key.

During this time I've also collaborated a bit with the EndlessOS distribution, and I can say that's a really nice distribution to use, the ostree usage for the whole filesystem is a great idea, and the amount of content that comes with the installation is really good.

The EndlessOS Foundation Goal is to reduce the digital divide, providing content and tools for offline people, centered on kids. This is a great mission and in the future, if I find the opportunity to help my local community, I'll try to use the EndlessOS tools and content to provide good learning content for kids without online access.

I was very happy these years at Endless, and I've learned a lot from different great people. It's incredible the number of talented software engineers that are related to Endless, and for me it was a real privilege to be able to share this space and mission for a few years.

The future!

So there we go, I'm exited for this change, and also sad about leaving a great project, but life is change and we should go ahead and think about the future! And my future is green now.

And if you don't know how to pronounce it, here you've a music video:

Berlin Mini GUADEC 2022

danigm — Fri, 22 Jul 2022 00:00:00 +0200

The GUADEC is back!

This year, the famous GNOME developers meeting event, the GUADEC, gets back to "normal" after the last year global COVID-19 situation. And when I say normal, I'm talking about people meeting in one place to share knowledge and to build a great community around this great software project.

This year the even was in Guadalajara, Mexico, and that's great, but a bit far to go from Spain... But thankfully, some of GNOME folks who can't go to Mexico organized a parallel event to attend remotely in Berlin, Europe, that's the Berlin Mini GUADEC.

I decided to join this people, because there's a direct flight from Málaga, so it was not a very long travel for me. This was my first big trip after COVID, and I was a bit afraid, but besides the heat wave, everything was great.

My talk

The GUADEC event allows a remote attendance and also remote participation, so I proposed a talk that I finally did from Berlin thanks to the c-base people that build a great setup to give the talk to the local people at the same time that it was streamed to the main event in Mexico.

As you may know, I started to do live coding streaming this year, and there are also other GNOME Developers doing that, so I wanted to talk about this new way of sharing and build community, and that was my talk about.

You can watch my full talk in Youtube:

Endless

It's not all party and fun in this event, there's also time to work. We take advantage of this event and some of the Endless people take the opportunity to meet together.

We're fully remote and I started to work at Endless in 2019. We had the opportunity to meet one time in London, but after that, this is the second time we had the opportunity to share some time together and it was great. Besides the day to day work, that can be done in remote completely, it's always nice to have the opportunity to share some time with coworkers to meet the real person.

Thanks

My travel was sponsored by the GNOME Foundation, and thanks to the EndlessOS Foundation, to give me the opportunity to be part of this great community.

Gambito, a chess App in Rust

danigm — Sat, 09 Jul 2022 00:00:00 +0200

Chess

I've been always interested in Chess, but never learned to play it correctly or even spent some time playing more than a couple of matches with friends. But during the COVID lockdown I started to watch people playing chess, and then I realized that the game is even more fun than I expected.

After that I started to play a bit online and discovered the great online platform Lichess, that's even free software. I do not play a lot, but I discovered that I like a lot the game and I watch chess streamers everyday and I even followed the professional competition.

So if you don't play chess, give it a try, it's a really nice board game, and a beauty "video game" that you can play online in a complete free software platform with a lot of community resources to learn, and even if you are a bad player like me, you can always enjoy the beauty of the chess watching other people playing.

Gambito

With this new interesting hobby, I installed the chess application that I found in gnome. GNOME Chess is a good application, but then I ask myself, why not rewrite it in Rust? Okay, it's not a rewrite, it's just a new application, my idea is not to replace GNOME Chess, but to experiment with Gtk4 and Rust, and that's the reason I started to work on Gambito, that's just a new Chess game for GNOME, written in Rust with Gtk4.

The first idea was to create a simple app, without widget inheritance or anything, just draw a board with existing Gtk Widgets, and do the drawing with CSS and the interaction with drag & drop. And I did that for almost all the application, but at some point I needed something more specific that I was unable to do without a new widget. So right now almost everything is done that way but the marks and arrows, that are custom widgets.

I'm doing almost all the development in live streaming (in Spanish), in my twitch account, so you can find some of the videos here in my youtube channel.

Right now you can use the application to play against Stockfish or to analyse a match. At the beginning the idea was to create just a chess game, but right now I'm thinking more about a chess learning application, so I think I'll work more in analysis tools, tactic training games, and maybe some good content for beginners, inside the application, like a tutorial and some kind of assistant with theory.

And there's a lot of fun things to do, like linking with the Lichess platform, to be able to watch games in live, play online using the app instead the web interface, analyze Lichess matches and look for cheaters doing some kind of IA (yes, there are cheaters using chess engines to win matches in online chess platforms, I don't know why, but there are people like that).

So I'm getting a lot of fun from the old game Chess, fun playing, fun watching an fun writing a Chess game.

Have fun!

GNOME Outreachy 2022

danigm — Mon, 30 May 2022 00:00:00 +0200

GNOME Translation Editor, Road to Gtk4

It's time to move to Gtk4. That could be an easy task for new project or for small projects without a lot of custom widgets, but gtranslator is old and the migration will require some time.

Some time ago I did the Gtk2 to Gtk3 migration. It was fun and during the journey we redesigned a bit the interface, but the internals didn't change a lot. Now we can do the same, migrate to Gtk4 and also update the User Interface.

Thankfully, I'm not alone this time, the GNOME community is there to help. A couple of months ago, Maximiliano started a series of commits to prepare the project to the Gtk4 migration, and today starts the Outreachy program and we've a great intern to work in this. Afshan Ahmed Khan will be working during this summer in the GNOME Translation Editor migration to Gtk4.

Outreachy

The Outreachy program provides internship to work in Free and Open Source Software. This year I've proposed the "Migrate GNOME Translation Editor to Gtk4" project and we had a lot of applicants. We had some great contributions during the application phase, and at the end Afshan was selected.

We've now an initial intern blog post and he is working now in the first step, trying to build the project with Gtk4. It's not a simple task, because gtranslator uses a lot of inheritance and there's a lot of widgets in the project.

User Interface redesign?

Once we've the project working with Gtk4 and libadwaita we can start to think about user interface improvements, and all the collaboration here is welcome, so if some designer or translator want to help, don't hesitate to take a look to the current interface and propose some ideas in the corresponding task

Twitch: GNOME live coding streaming

danigm — Sat, 22 Jan 2022 00:00:00 +0100

This year I've started with something that I wanted to do since the first COVID lockdown, but never did. So as a first year resolution I decided to start streaming and I created my Twitch channel Abentogil.

I've been thinking about streaming since lockdown, when a group of Spanish people created a nice initiative to teach kids how to code and other tech stuff. I never participated on that initiative, but it was the seed of this.

This year I've seen other GNOME streamers doing live coding, like ebassi and Georges and, at the end, I pluck up the courage to start.

The plan

I've started with one our at the end of my day, so I'm trying to stream from Monday to Thursday from 20:00 to 21:00 (CET).

This routing also will help me to spend more time in free software development regularly, so not just on weekends or holidays, but a bit of work everyday.

Right now I've started working on my own projects, like Timetrack and Loop. I've fixed some issues that I had with Gtk4 in Timetrack and I've started to work on the Loop app to add a MIDI keyboard.

The other day I discovered the Music Grid app idea in the gitlab and it's related with the MIDI implementation I've been working on, so the next thing that I'll do on my streams is to create this music grid widget and add it to the Loop app, to have a nice Music creation App.

The language: Live coding in Spanish

I've decided to do the streaming mainly in Spanish. The main reason to do that is because now you can find documentation and a lot of videos in English, but for people that doesn't master the English language, it's harder to follow this content and even harder to participate, to ask or say something.

Spanish is also my main language, and the idea is not just create tutorials or something like that, this is just for fun, and if I'm able to create a small community or influence someone, I want to show that the language is not a unbreakable barrier.

I want to make this for the Spanish community, but that doesn't mean that for some streams I cannot talk in English, all depends on the audience, of course, if a non Spanish speaker comes and ask something, I'll answer in English and try to do multilingual stream.

The International English is the language to use in Free Software, but it's really important to do not convert that in a barrier, because there are a lot of different communities and people that don't know the language and the first contact with something new is always better in your main language. Everything is easier with help from people that's near to your cultural background.

The future

Right now I'm working on Loop and every stream is similar, live coding on this Gtk4 music APP, but in the future I'll do different streams, doing different things on each week day. These are the current ideas that could be a day on my stream:

Maintenance day: Work on my apps, code review, fixes, releases, and development.
GNOME Love day: Give some love to the GNOME project, look for simple bugs to solve in different projects, help with translations, initiatives, and other possible one hour task to give some love to the project in general.
Learning day: Pick some technology and learn by doing, for example, I can explore new programming languages writing a simple app, learn about Linux writing a driver, learn how to animate with blender, anything new to play and discover.
Newcomers day: Create simple tutorial or introduction to some technology, How to write a GNOME Shell extension, how to create a Gtk4 app in python, teach to code from zero, etc.

This is just a list of ideas, if I discover any other interesting use of this time and stream, I can do that. The final goal of this is to have fun, and if someone learns something in the journey, that's even better.

I'll also upload any interesting stream to youtube, so the good content could be viewed on demand.

So if you like this, don't hesitate to view some of the streams and say "Hola":

Loop: A simple music application

danigm — Sat, 11 Dec 2021 00:00:00 +0100

In the last year I've seen some really good musician that performs all the instruments in a song with just a loop machine, recording each instrument one by one in tracks and looping.

I was thinking that it should be easy to have a desktop application that does exactly the same, just some tracks to record some sounds and the playback with a loop option, and that's what I created during this week.

The Loop application is just a simple Gtk4 application that uses gstreamer to record tracks and then you can play each one at the same with or without loop option. With that, a good musician could create the base melody of the song and then sign on top of that. Unfortunately, I'm not a good musician, but I can use this to play around.

I've just created the request on flathub to add the application, so if everything is okay it will be available there soon, so more people can play with this awesome toy.

Right now it has the basic record, play and loop functionality with just four tracks, so don't expect to have a professional music app (yet). The recording and play times are not perfect and there's a delay, that's a known issue, but I'm planning to fix this issue and add more functionality in the future, like:

Make number of tracks configurable
Import track from files
Add a trim slider per track, to be able to adjust the recorder track to loop
Metronome tick and clock to have a visual reference for recording tracks
A record button, but export the combination of all tracks and mic to a mp3 file

And that's all. The logo and design is an initial version done by myself, so if any designer wants to take a look, all contributions are welcome. And of course any code contribution is also welcome.

If you use this application and do some good or fun performance, please, ping me on social networks and let me know.

Endless Orange Week: Hack content creators platform (2)

danigm — Mon, 15 Nov 2021 00:00:00 +0100

The past Friday was the last day of the Endless Orange Week. It was a nice and fun experience, and even if I was not able to do as much as I wanted, we were able to make something that "works" in the Hack project.

The Hack Quest editor

The first step to have custom quests on the Hack app was to complete the Ink language support. We started to work on this some time ago, but never completed the functionality.

I worked on that the first three days, updating the ink library and building the missing pieces to be able to load quests from random paths. I've implemented that in a way that the Hack application is able to receive a path to a .ink file, and it's able to build and run the quets.

The Quests are not just the script, but they have some metadata, like title, subtitle, description, difficulty and the card image to show on the interface. To solve that I defined a "custom quest bundle format", that's bassically a folder with:

questId
quest.jpg
metadata.json
quest.ink

So I also added the functionality to import a bundle zip file and export with the quest information.

I created some command line options to use this new functionality:

  --play-ink=FULL_INK_FILE_PATH                 Start a custom ink quest.
  --import-quest=PATH_TO_BUNDLE_OR_INK_FILE     Import a custom ink quest.
  --export-quest=CUSTOM_QUEST_ID                Export a custom quest bundle.

Quest creation interface and the Inky Editor

The first idea was to try to provide a full quest creation experience in the app, but that was too much, so we decided to simplify the way to create quests and depend on the Inky editor external tool. Manuel Quiñones took some time to update the flatpak application with the latests ink version, so we can use to create custom quests.

The Inky editor provides help about the language, syntax highlighting and a simple way to test the script, so it's a nice tool. The main problem with this tool is that it doesn't provide a way to launch it with a file path so it's not possible to integrate with the Hack app.

So at the end, the Quest creation dialog is just a way to define the metadata and to select the Quest ink files from your filesystem. How the ink script is created is a decision to make for the content creator.

The future

We've no time to complete all we wanted to do, and I didn't create a new release, so this new functionality is still not there. But we'll try to do a release soon.

Simon is working on some interface improvements and also on a new tutorial Quest, so we can introduce the Custom Quest creation tool in the same app.

The Character editor

The other part of this week planning was the character editor. Joana did a really nice work designing the application, the initial assets and the user experience, but I had not too much time to work on the implementation. So I spent just one day working on this.

The main idea was to create a new independent app, and then provide a way to integrate with the Hack application and the custom Quest creation dialog. And it'll be a simple application so maybe it could be useful or interesting for other people, it's a fun way to play around and create random faces.

We just created the application Avatar Creator. I created a simple python Gtk4 application and worked a bit on the basic functionality. So right now it loads a list of svg assets and provide the 3x3 grid. You can click on a grid cell and then choose what basic image should go there.

I added the initial set of basic images, created by Joana, to create this funny robot faces, but the format is simple enough to extend with different "avatar libraries" in the future.

Right now it's also possible to export to png, so the app is functional, but it needs a bit more work.

My idea is to work a bit more in the following weeks, when I have some time, on weekends or holidays and at some point, publish it in flathub. And lets see if there are more developers interested on this app so it can grow.

The application is simple enough to be a good place for GNOME newcomers and it's also a fun project to work on. A simple toy app to create faces that could have some potential, some future ideas:

"Smart" random faces generator
Configurable grid: Maybe is interesting to make it bigger or smaller to play around
Programmed simple base image manipulations, like rotation, mirror, color
Animation creation, maybe be able to export to gif

The Endless Orange Week experience

This week was a really nice experience, because we were working in a "personal" chosen project, that we liked and without the day to day meetings, times schedules and other related work stuff.

But that was not all. In Endless we've different teams that work mostly isolated, because we're working on different fields, we've some overlapping, but we work day to day as small teams, and this week we were all using the same slack channel to show our progress, and it was nice.

Maybe now that we're not a big organization with a lot of workers, we can do something like this more often, it's always good to know more about other coworkers and to learn something that maybe it's not related with your main project, but it could be interesting.

I'm really happy that we did this Endless Orange Week, it's sad that it ended too soon, I'm waiting to learn from my coworkers what amazing things they do during this week and I'm looking forward the next year Orange Week!

Endless Orange Week: Hack content creators platform

danigm — Mon, 08 Nov 2021 00:00:00 +0100

This week (Nov 8 – 12) I am participating in Endless Orange Week, a program where the entire Endless team engages in projects designed to grow our collective learning related to our skills, work and mission.

We propose a project, that could be anything, and then work during the whole week, without distraction. I've choosed to work on the Hack project, that's a really nice project that needs some love, because since the past year, we have other priorities, so there's no time to improve the Hack app.

The project: Hack content creators platform

The Hack application is a "Quest" launcher, and each Quest is an interactive learning experience, a guided introduction to some technology or topic.

Quests are just python scripts, with a simple library to expose messages to the user, ask questions, or wait for desktop events, like launch an application, focus an application, etc. And all these Quests are inside the application, and are created by the Hack team and released with a new Hack app flatpak.

The main idea of the project is to provide a simple Quest editor to allow any Hack user to create and share their our Quests.

To have this Hack content creators platform we'll need:

To simplify the way we create Quests, instead of a python script, we'll uses a Domain Specific Language, called Ink. We started to work on this, but we never ended the support.
To create the interface to be able to import and export custom Quests, that could be zip bundles, with the Ink script and some images.
To create the interface to write the actual Quests and save or bundle.
Create some introductory Quest to explain "how to create your own Quests!".
Create an character editor, to be able to "design" new characters for Quests.

The Quest editor

The Quests will be written using the Ink language, and there's something done before. The first idea is to just provide a text editor and some helpful information about the format, and maybe a button to validate. But if there's time we can use something advanced or even integrate the Inky editor.

The Character editor

Each Quest has a main character, and we've five in the Hack app right now, but it could be great to be able to define new ones for custom Quests. That's the idea of this part of the project.

The initial idea is to have a library of character parts to combine, and the editor will allow the user to combine this parts and maybe change colors, to be able to create unique characters for your Quests.

The Team

I'm not working alone in this "side" project during the Endless Orange Week, Simon Schampijer and Joana Filizola will be working on this too, so this is a big task but we've a great team. Let's see how far we are able to go during just one week.

The impact

This is just a project to try to keep alive more time the Hack application, without a lot of effort or a whole team behind it. We are not able to put more content there periodically, so if there's a way to create new content easily and (maybe in the future) a way to publish, it'll be possible to create a community around the project.

And we have also new possibilities, in the near future, we can add some Hack content to the Endless Key, and using the Ink language, so this editor could help to bring more content there easily.

And the final piece, the Character editor, could be an independent application, a nice simple application that could be used to create your character for your profile photo, or to generate random character pics.

Fractal: Refactoring and the review process

danigm — Tue, 30 Jun 2020 00:00:00 +0200

In this year GSoC, Alejandro is working on Fractal, moving code from the backend to the client, to try to simplify the code used to communicate with the matrix.org server and maybe in the future we can replace fractal-matrix-api with the matrix-rust-sdk. And then we'll have less code in our project to maintain.

This is a great work, something needed in a project with a technological debt of several years. I created this project to learn Rust, and also I was learning about the matrix protocol during the project build. And other contributors do the same. So we've been building one thing on top another for a lot of years.

In this kind of community driven projects it's the way to go. For some time we've people interested and developers think about the design and start change some parts or to write new functionality following a new design pattern. But voluntary developers motivation change in time and they left the project and the next one continues the work with a different vision.

It's not something bad, it's the greatness of the open source. Different people has different motivations to participate in a free software project, and every contribution is welcome. I'm the maintainer of the project and I've spent a lot of time building Fractal, but I don't have the same motivation now to work on the project, to it's good to have other people working on it so it can continue alive.

Alejandro is doing a great work and he's not a 4 months contributor. He's working on the backend refactoring for 2 years now, step by step and he has plans for the future.

Refactoring a big project is always hard, because there's a lot of code movement and always there's the fear to regressions.

Rust is a great language and it shines when big code refactoring comes to the scene. If it compiles, you know that there's no memory errors, dangling pointers and that kind of problems. If it builds it will work.

But maybe it will work different, so the review process is needed to ensure that the application continues working.

Automated tests are really useful for big code changes and project refactoring, because you have a quick picture and some certainty that the project is working. But we don't have tests in fractal :D, so someone should do that.

So here I am. Reviewing large MR, with a lot of lines. At least gitlab makes this process a bit easier.

What I'm trying to do in the review process is to just read the whole diff and check if there's some problem in the code. And after every change, I run the app and I do some tests, trying to use the functionality that could be broken by those new changes.

This takes a lot of time and it's not something fun to do... But someone has to do that. And during that process, sometimes I learn something new. Reading code is an interesting task and try to find bugs in code while reading it, is something useful. To think about that code, what it does and why.

Fractal: Google Summer of Code 2020

danigm — Fri, 08 May 2020 00:00:00 +0200

I'm glad to say that I'll participate again in the GSoC, as mentor. This summer we'll try to implement multi-account support in Fractal.

The selected student is Alejandro Dominguez (aledomu), that is collaborating with Fractal since the Seville Hackfest in December 2018, doing a great work in the backend. Alejandro is young developer with a lot of energy and ideas, so it's great to have this kind of people working on GNOME.

I've not spend much time lately developing Fractal, the time and energy is limited, but I'll try to use this GSoC mentorship to go back to the active Fractal development. My objective during this summer will be to stabilize, fix bugs and improve the performance and create a new release at the end of the GSoC, because we've a lot of new functionality in master, but I didn't spend the time to do the release.

The google summer of code is a great opportunity for free software, it give us a full time working student, during three months, working on free software, so it's really appreciated in a community where a lot of work is volunteer work.

There are a total of 14 projects selected for this GSoC round for gnome, you can take a look to the full list in the GSoC page.

End of GNOME Outreachy 2019

danigm — Fri, 13 Mar 2020 00:00:00 +0100

The Outreachy Program

The outreachy program ended the past week and we've done great improvements during this four months of work. I'm very happy with the result and with the work of the two interns and also the GNOME co-mentors that make this possible.

If we're lucky the interns will continue contributing in the future and we can see the GNOME comunity growing in developers and diversity 🎉.

GNOME translation editor (Gtranslator)

Video showing the new gtranslator search bar

Priyanka Saggu has been working on the new gtranslator search bar. It's a replacement for the old search dialog with a new and modern search bar, inspired in the gnome-builder search.

This is how it looks in the current gtranslator version and the video is from gtranslator master, that I'll try to release as 3.36.0 this weekend.

I want to thank to the other co-mentor of this project, Daniel Mustieles, who has been testing and reviewing this new functionality.

Fractal

Sonja Heinze has been working on the video player for Fractal, so now we can see videos inside the fractal message history instead of open it with an external video player.

Video showing the new fractal video player

This is now in master and will appear in the next release, that I'll try to publish soon, maybe during this month. I want to fix some performance issues first.

Jordan Petridis (alatiera) has done a great work as co-mentor, guiding the project and helping with the gstreamer.

I've not presented any project proposal for the next outreachy, I want to take a break and rest a bit before the Google Summer of Code, when I'll try to get the multi-account support implemented in Fractal.

This programs give me a bit of work, reviewing and guiding the intern, but it's really great to have paid people working on free software, so I'm very happy to be able to be a mentor in GNOME to help to boost some free software projects using these resources.

GNOME Outreachy 2019

danigm — Mon, 16 Dec 2019 00:00:00 +0100

The Outreachy Program

The Outreachy program provides internship to work in Free and Open Source Software. This year I've proposed two projects as part of the GNOME project and we've two interns working for three months, so we'll have a lot of improvements in the following months!

I'll be mentoring these interns, so I will need to spend some time helping them to work on the existing codebase, but it worth it, if this makes more people to collaborate in free software development and if this help us to improve some useful apps.

These two projects are Fractal and the GNOME translation editor. You can take a look to the list of outreachy interns.

Fractal

Fractal is a Matrix.org gtk client, and I've proposed for this year program to implement a video player in the message list. We've a preview for images, for audio files but nothing for video files.

Sonja Heinze is the one that will be working on this during the next three months. She has been working during the past month in some small issues in Fractal so I'm really sure that she will be able to do great contributions to the project.

Jordan Petridis (alatiera) will be helping in this project as a co-mentor, I don't know a lot about gstreamer, so he'll be really helpful here with the gstreamer and rust.

GNOME translation editor (Gtranslator)

GNOME translation editor (gtranslator) is a simple .po editor. I've proposed to Rework the search and replace dialog. We've right now a simple find/replace modal dialog and I want to modernize the interface to integrate better in the window as a popover.

Priyanka Saggu is the one that will be working on this during the next three months. She has been working on gtranslator during the past month and she has done great contributions and improvements during this time.

Daniel Mustieles is the other co-mentor for this project. He's an experienced GNOME translator so he will help us a lot with the app user experience design and testing.

LAS 2019, Barcelona

danigm — Thu, 14 Nov 2019 00:00:00 +0100

The event

The Linux App Summit (LAS) is a great event that bring together a lot of linux application developers, from the bigger communities, it's organized by GNOME and KDE in collaboration and it's a good place to talk about the Linux desktop, application distribution and development.

This year the event was organized in Barcelona, this is not too far from my home town, Málaga, so I want to be there.

I sent a talk proposal and was accepted, so I was talking about distributing services with flatpak and problems related to service deployment in a flatpaked world.

Clicking in this image you can find my talk in the event streaming. The sound is not too good and my accent doesn't help, but there it is :D

The event was a really great event, with really good talks, about different topics, we've some technical talks, some talks about design, talks about language, about distribution, about the market and economics, and at least two about "removing" the system tray 😋

It was really interesting the talk about the "future" inclusion of payments in flathub because I think that this will give a new incentive to people to write and publish apps in flathub and could be a great step to get donations for developers.

Another talk that I liked was the one about the maintenance of flatpak repositories, it's always interesting to know how the things works and this talk give an easy introduction to ostree, flatpak, repositories and application distribution.

Besides the talks, this event is really interesting for the people that bring together. I've been talking with a lot of people, not too much, because I'm a shy person, but I've the opportunity to talk a bit with some Fractal developers, and during a coffee talk with Jordan Petridis, we've time to share some ideas about a cool new functionality that maybe we can implement in the near future, thanks to the outreachy program and maybe some help from the gstreamer people.

I'm also very happy to be able to spend some time talking with Martín Abente, about sugar labs, the hack computer and the different ways to teach kids with free software. Martín is a really nice person and I liked a lot to meet him and share some thoughts.

The city

This is not my first time in Barcelona, I was here at the beginning of this year, but this is a great city and I've no time to visit all the places the first time.

So I've spent the Thursday afternoon doing some tourism, visiting the "Sagrada Familia" and the "Montjuïc" fountain.

If you have not been in Barcelona and you have the opportunity to come here, don't hesitate, it's a really good city, with a great architecture to admire and really nice culture and people, and here you can find good food to enjoy.

Thank you all

I was sponsored by the GNOME Foundation, I'm really thankful for this opportunity, to come here, give a talk and share some time with great people that makes the awesome Linux and open source community possible.

I want to thank to my employer Endless because it's really a privilege to have a job that allows this kind of interactions with the community, and my team Hack, because they I've missed some meetings this week and I was not very responsive during the week.

And I want to thank to the LAS organization, because this was a really good event, good job, you can be very proud.

Andaluh-rs, a lib to transcript Spanish to Andaluh

danigm — Sat, 06 Jul 2019 00:00:00 +0200

Spain is a big region with a lot of languages, we've the official one for the whole region, Spanish or Castilian (es-ES), and other officials languages spoken in other regions, like the Galician (gl-ES) spoken in Galicia, the Basque (eu-ES), spoken in the Basque Country, the Catalan (ca-ES) spoken in Catalonia.

But we've more languages that doesn't have the same official support, the Valencian, the Aragonese, the Asturian, and more. And also we've some dialects like the Andalusian.

All of this languages was discredit during the dictatorship period when the only official language was the Spanish and the others was treated like vulgar languages or non educated one, and in some cases the language was prohibited.

Whe the democracy arrives, some regions spend a lot of resources trying to recover the language and are supporting it until now with official institutions and lessons in the official education system. But in other regions the stigma continued until today.

That's the case of the Andalusian, that right now is not considered a language but a dialect. In any case, this dialect is treated as a non cultivated language, spoken by illiterates. We've a lot of Spanish movies and Series where the character that spoke Andalusian was the illiterate, from the countryside or the servant of the family.

Be proud of it

In Andalusia we've a lot of culture, literature and music done in Andalusian, but we don't have a way to write that and many people try to avoid the acent and the local words to look more cultured and because there's no a writing system, we write in Spanish, it's hard to write lyrics, poetry and other kind of literature. The Andalusian is not only a way to talk, some words are short and we've contractions and other vowel sounds so if you write in Spanish, it's not the same as spoken, some information is missed and for example in music or poetry the metric doesn't match.

There's a movement trying to define a writing Andalusian and promoting the language, trying to make people proud of it and talk and write without complexes.

Translator, here comes the code

And there's a group of developers that are working in some tools to provide direct translation from Spanish and other tools to ease the Andalusian writing.

I like to write code and I'm always happy to find new problems to solve, to learn new languages, tools and to spend some time trying to code something that I've not done before. So I decided to write a translator from Spanish to Andaluh using rust, and I've created the andaluh-rs lib.

The translator is more or less easy, there're some rules that should be applied from top to bottom that basically replaces some group of letters. There's a implementation in python that uses regular expressions for that. There're a lot of regular expressions, so I thougth that it could be easy to use a parser, so I used the pest parser.

// supress muted /h/

H = { ("h" | "H") }
initial_h = { H ~ letter }
CH = { C ~ H }
inner_ch = { CH ~ letter }
inner_h = { !inner_ch ~ H ~ letter }
hua = { H ~ ("ua" | "UA" | "Ua" | "uA") }
hue = { H ~ ("ue" | "UE" | "Ue" | "uE") }
noh = { !CH ~ !H ~ letter }
h = _{ ((sp|SOI)? ~ initial_h* ~ ((hua | hue | inner_ch | inner_h) | noh+)+)+ }

I've defined each rule in the pest format, so I've a parser for each rule and then I can replace the word with the correct replacement.

pub fn h_rule(input: &str) -> Result<String, Error> {
    rule!(Rule::h, &input, Some(&defs::H_RULES_EXCEPT),
        Rule::initial_h | Rule::inner_h => |pair: Pair<Rule>| {
            let s = pair.as_str();
            let h = slice!(s, 0, 1);
            let next = slice!(s, 1);
            keep_case(&next, &h)
        },
        Rule::hue => |pair: Pair<Rule>| {
            keep_case("güe", &pair.as_str())
        },
        Rule::hua => |pair: Pair<Rule>| {
            keep_case("gua", &pair.as_str())
        })
}

To simplify the code, I've defined the rule macro, with the code used in all rules:

macro_rules! rule {
    ($rule: expr, $input: expr, $( $($t: pat)|* => $r: expr ),* ) => {{
        let map: Option<HashMap<&str, &str>> = None;
        rule!($rule, $input, map, $( $($t)|* => $r ),*)
    }};
    ($rule: expr, $input: expr, $map: expr, $( $($t: pat)|* => $r: expr ),* ) => {{
        let (repl, input) = match $map {
            Some(ref m) => replace_exceptions($input, m),
            None => (vec![], $input.to_string())
        };

        let pairs = AndaluhParser::parse($rule, &input)?;
        let mut output: Vec<String> = vec![];

        for pair in pairs {
            let chunk = match pair.as_rule() {
                $( $($t)|* => {
                    $r(pair)
                } ),*
                _ => {
                    String::from(pair.as_str())
                },
            };
            output.push(chunk);
        }

        let mut outstr = output.join("");

        if $map.is_some() {
            outstr = replace_exceptions_back(&outstr, repl);
        }

        Ok(outstr)
    }}
}

And because the Spanish and the Andaluh languages uses unicode and rust Strings can not be iterated by unicode, I've used unicode_segmentation crate, and defined some utility macros to get the real String len and to get slices of that String.

macro_rules! chars {
    ($input: expr) => {
        UnicodeSegmentation::graphemes($input, true)
    }
}

macro_rules! slice {
    ($input: expr, $start: expr, $end: expr) => {
        chars!($input)
            .skip($start)
            .take($end - $start)
            .collect::<String>()
    };
    ($input: expr, $start: expr) => {
        chars!($input)
            .skip($start)
            .collect::<String>()
    }
}

macro_rules! len {
    ($input: expr) => {
        chars!($input).count()
    }
}

With all this done, we only have to apply all rules, in the correct order, to the input string so we can get the translated String as output.

pub fn epa(input: &str) -> Result<String, Error> {
    // TODO: escape links
    let rules = [
        h_rule,
        x_rule,
        ch_rule,
        gj_rule,
        v_rule,
        ll_rule,
        l_rule,
        psico_rule,
        vaf_rule,
        word_ending_rule,
        digraph_rule,
        exception_rule,
        word_interaction_rule,
    ];

    let mut output = input.to_string();
    for r in rules.iter() {
        let out = r(&output)?;
        output = out.to_string();
    }

    Ok(output)
}

Performance

This code is not the best one, I'm doing a lot of string operations with copies and clones, I'm sure that anyone with more experience with rust can view a lot of points where we can optimize this code. At first I thought that the translation could be done during the parsing, keeping a length to be able to view backward and forward.

Maybe it's possible to read char by char, keeping a buffer, and detecting if we can apply any of the rules with the content in the buffer, but I've based this lib in the python one, so for me it was easier to translate each regex to pest regex and then do the same translations in the lib.

But I still think that there's a better solution for this problem, but some times it's better to have something that just works instead of a never done best solution.

During this process I've learned to use pest and I've been playing a lot with regular expressions, so it was a fun project.

EndlessOS dual boot with Fedora

danigm — Sat, 25 May 2019 00:00:00 +0200

I've a ThinkPad X1 Yoga, that's basically a ThinkPad X1 Carbon 4th but with a touch screen with a pencil that works like a wacom tablet.

I've this laptop since 2016. The first thing that I did when I received it was install a GNU/Linux operating system. I'm a GNU/Linux user since the year 2000 going through a lot of distributions, Debian, Ubuntu, Archlinux, etc.

When I received this computer I've a customized Archlinux there, and I wanted to dd my harddisk and boot, but I was unable to do that. I didn't know nothing about UEFI and I was unable to boot the Archlinux installer.

So I decided to go ahead and change my main distribution. I installed Fedora and almost all the hardware worked so I keep that one and was happy, until today :P

OStree, the new way of distribute GNU/Linux

With OStree and flatpak, there's a new way to distribute GNU/Linux, instead of use directly a package manager and update each package, we can use OStree and mount the root filesystem as read only and do full OS upgrades without broken packages and dependencies and so. The operating system go as is and the user should try hard to break it.

The other great thing about OStree is that it's like a git repository, so you can have different branches and a history, so you can easily go back and forward, it's really easy to test the beta and go back without breaking your system.

The main problem is that you can't install anything on your OS, you should use contained apps like flatpak or install by hand, you can't use apt-get. But that's okay for a day to day user, a power user always can unlock the OS and use it as a normal GNU/Linux distribution.

I wanted to try one of these distributions. The logical choice was Silverblue because it's a Fedora and I'm using it for three years now, but there's another option, EndlessOS is also OStree based, and I'm working with this OS, so I should give a try and use EndlessOS.

The EndlessOS install process

Like all the new modern GNU/Linux distros, EOS comes with a easy to use installer, you only need to boot from the USB and click next until it's done...

But here we've the first problem. I've more than one partition in my disk:

/dev/sda1: UEFI
/dev/sda3: Fedora /
/dev/sda8: Swap
/dev/sda4: Fedora /boot
/dev/sda6: /home

I want to keep my Fedora (who knows if something bad happens) and try to use the same home partition for my new EOS. But the installer only give me the option to erase all and have a clean system. But that won't stop me.

Let's go back and instead of reformat I will click on Try Endless OS:

What we need to do, Robert McQueen gave me some directions:

The constraints for booting Endless are 1) you use our grub, and 2) the root partition is labelled "ostree"

So if you have an EFI system, you can copy our EFI binaries into the ESP, and create a new partition for Endless, then dd the endless ostree filesystem into it

Then you should be able to boot, if you add a boot entry for the endless grub to your firmware, or make it the default (by providing the fallback boot64.efi or whatever it's called), or chain load it from another Linux loader

Install EOS with other linux (EFI system)

Boot from USB, select try
Launch the gnome-disk-utility and prepare a partition. I've not free space, but I was able resize my Fedora partition and I split in two of the same size, now I've a new ext4 partition /dev/sda7 with 25GB.
Copy the endless ostree:

$ sudo su
# dd if=/dev/mapper/endless/image3 /dev/sda7

Copy endless grub to EFI. I mounted all partitions in /tmp, the first partition in /tmp/sda1 and the EOS efi in the /tmp/EOS:

# mkdir /tmp/sda1 /tmp/EOS
# mount /dev/sda1 /tmp/sda1
# mount /dev/mapper/endless-image1 /tmp/EOS
# cp -rf /tmp/EOS/EFI/endless/ /tmp/sda1/EFI

Add the new boot entry:

# efibootmgr -c -d /dev/sda -p 1 -L EOS -l \\EFI\\endless\\grubx64.efi

Set as default boot:

# cp /tmp/EOS/EFI/endless/grubx64.efi /tmp/sda1/EFI/Boot/bootx64.efi

Reboot and create my default user. Then I add my home partition to the /etc/fstab file:

UUID=c885e171-1a03-4afb-8519-f9fe26fe92b7 /sysroot/home ext4 defaults 1 2

And because the first user in EOS is the shared account, with UID 1000, I've to change the UID of my user editing the file in /etc/passwd. Then I rebooted again and all works. I've all my flatpak apps installed in the user space working.

So here we're, with a shiny new OS working like a charm:

The EFI and efibootmgr (who needs grub to select the OS?)

I didn't know much about UEFI and I was very impressed about how easy is to update this from GNU/Linux. There's a tool called efibootmgr that does all the work, and you can mount the partition, that's a FAT32, and put files there.

In my ThinkPad, I can boot directly to the UEFI boot menu pressing F12 during the boot, and that menu can be changed using the efibootmgr so it's not needed anymore to use the grub2 OS selection interface, I can use the UEFI menu for that!

This has some disadvantages, if you remove files from the UEFI partition, you can break the whole boot, so review all before any change.

Kung-fu Master, Blindfolded debugging

danigm — Sun, 05 May 2019 00:00:00 +0200

Recently I've been working in the EOS update. The change is really big because EOS has some modifications over the upstream code so there are a lot of commits applied after the last stable upstream commit.

EOS 3.5 was based on gnome 3.26 and we are updating to use the gnome 3.32 so all the downstream changes done since the last release should be rebased on top of the new code and during this process we refactor the code and commits using new tools and remove what's now in gnome upstream.

I've been working in the Hack computer custom functionality that's on top of the EOS desktop and basically I've been rebasing the code in the shell to do the Flip to Hack, the Clubhouse, a side component and notification override to propse hack quests and the wobbly windows effects.

I've been working mainly with gnome shell and updating javascript code to the new gjs version, but this was a big change and some rare bugs.

Here comes the blindfolded debugging. I've a development system with a lot of changes and a functionality in the gnome shell that depends on different projects, applications and technologies.

I don't know the code a lot, because I'm not the one that wrote all of this, I'm working here since February, so I know a bit how things works, but I'm not the one who knows everything, there are a lot of rare cases that I don't know about.

I've found and fixes several bugs in different projects, that I don't know about a lot, during this process. How can I do that? If you are a developer that write code since a few years maybe you've experienced something similar, I'm calling this the blindfolded debugging technique:

Start to change code without knowing exactly what you're doing, but with a small feeling that maybe that line is the problem.

This technique is only for experienced programmers, as a kungfu master that put a blindfold in their eyes to fight against an opponent, the developer that will be brave enough to try this should have a lot of experience or he will fail.

You're an experienced developer, but you don't know the software that you're debugging. It doesn't matter. The same way that in a kungfu fight you don't know your opponent, but almost every fighter has two arms and two legs, so more or less you'll know that he'll try to punch you or kick you, you've a clue. As a programmer, every software has an structure, functions, loops...

No matters who wrote that or how old that code is, if you're an experienced programmer you'll feel the code and without knowing exactly why or how, you will be able to look at one line and says: Here you're little buggy friend.

Maybe I'm only trying to justify my lucky with a story to feel better and say that I'm not a dumb changing random lines to try to find a bug, but I'm sure that other developers has this feeling too, that feeling that guides you to the exact problem but that you're not able to rationalize.

I think that this is the experience, the expertise is your body and your inner brain doing the work meanwhile you don't need to think about it or know exactly what you are doing.

I'm a hacker

danigm — Sun, 17 Feb 2019 00:00:00 +0100

I've the strength of the one that fails, trains and returns

Some time ago I wrote about my interview process trying to get a job at GNOME. After this blog post was published in the GNOME planet, I received a lot of comments from the Great GNOME community.

This guided me to take a look to the companies that was looking for GNOME developers and after some interviews I finally get an offer to work on Endless.

All the process was online, I've been working from home since 2011 and I want to continue like this. I'm living in Málaga, the south of Spain and I was looking for a remote job.

I've a very nice interview process, talking with developers that I knew about from their contributions to free software and these interviews weren't technical, that's a good thing about the free software contribution and the community, if someone wants to know my technical skills, he only need to look to my gitlab or github and he will find a lot of code. These interviews were personal and with some management questions, to know if I'll fit in the team and the company.

I did the interview to work directly on the Endless OS, in the desktop team, but after some interviews I ended in other team, working with the Endless OS and all GNOME technology, but in the Hack Computer.

The hack computer

The hack computer is an educational project. The main idea is to provide a fully functional computer for kids to be their first personal computer and also provide a way to teach kids to hack.

The computer is a usual laptop, with an Endless OS running, and with some extra applications that try to guide kids to explore, modify and finally hack the computer learning in the process about GNU/Linux, the code and that any part of your software can be hacked to make it better, an introduction to the free software and the great world of open source code that you can modify to learn, play or simply because you can.

The way to guide kids to learn is like a video game, with characters that will appears in your desktop and will guide you with tips and challenges and after each quest completed the kid will learn something, playing with his computer.

If you like this and want to collaborate, we're hiring.

Endless OS

The hack computer is built on top of the Endless OS. Endless OS is based on debian and the desktop is a modified gnome shell, but it's not the usual debian derivative, it's based on OSTree. The main difference is that the root filesystem is read only and updates are managed with ostree, that's like a git repository.

This kind of Operating System is easier to maintain, because the user can't modify the base system, so this means that he was unable to break it. All user applications are installed via flatpak, so are independent of the OS version and because of flathub you can install latests version of apps without the need to update the full operating system.

This is the way that Fedora SilverBlue is trying to follow and is a new way to build and distribute GNU/Linux.

The end of an Era

I've been working in Wadobo since 2011. I created that company with two college friends to try to continue hacking like we were hacking in the Seville LUG SUGUS and earn money from that, and we did it for a long time.

We've been working in free software projects and with free software technology, contributing to the community as much as we can and always trying to free our work.

We created some projects that grow up and follow its own path, like the agora voting system, that Edulix (Eduardo Robles) converts in nvotes.

The self employment was a really good experience. I've been doing was I want for a long time, taking time to contribute to GNOME and other projects when I need it, because I was the one deciding what I want to do. But in the other hand, I've been stuck in the local consultancy market for a long time.

I'm not a business man, I'm a developer and I'm based in Spain, where there is no money inverted in innovation or we was unable to find that money, so we were doing django web apps for a living and spending some time in our interests in the extra time we've. So when I found the possibility to work full time in an innovative project, I've no choice.

We'll try to continue with the Wadobo idea, like a group of people interested in free software and new technologies, maybe this will derive into a Linux User Group or maybe in the future someone takes the initiative and build a business around this again.

Hacking

I'm really happy with this change. Endless OS and the hack computer are really great projects with a lot of bleeding edge technology and a really great group of people, and I'm really exited to be able to work with people all around the world in a project with a global vision.

I don't want to leave the roots I've here with my Wadobo friends and the local community and local technology ecosystem. This is also related with the Endless vision. Spain, and here in the south, in Andalucía, we've a technological breach that we try to fill with free software.

I'll continue working with the Seville University, because they give me a lot and I think it's a talent pool, we only need to guide those students to the GNOME community or to other free software communities out there, to save their souls from the privative software and the dark side.

I'll continue supporting initiatives like the Free Software Contest

By the way, I'll be talking about rust in the Seville University the next Tuesday 5th of March 2019, if you're in Seville, come to talk with me :D

Timetrack app for GNOME

danigm — Sat, 26 Jan 2019 00:00:00 +0100

This week I started a new small project called Timetrack, you can find the code in the gnome gitlab.

This is a simple app to store an activity log in a sqlite database and to provide useful reports in the future. The main idea is to use this for may day to day work, so I'll be able to keep track of my working hours and other activities.

The need

There's a lot of timetrack applications. There are good web apps and also some desktop applications. At first, I tried to find a gnome-shell plugin, but I didn't find any extension that is good for me.

I've had a custom simple web app to track my time.

I've been using this app for a long time, but I don't like to use web apps for simple tasks so I want something local.

So I tried with some gtk desktop applications, but none of the applications that I've found looks good to me.

I wanted a simple timetrack app, without any complex stuff, so I thought that it won't be hard to implement it. So I started with a simple python app based on the code of Password Safe.

So I started to code and in two days I had a simple application that works. The applications is now on flathub so it's available for use.

The functionality

Right now, Timetrack has a limited set of features, but I think I'll improve the app with new features as soon as I need something. So, the current version can:

Track activity time with a simple button, showing the activity time spent
List last activities
Edit / Delete activities
Simple report by day, week or month
Report navigation using a calendar

I've plans to add more functionalities in the near future, like:

Report export to plain text and csv
Comments for activities
Tags for activities
Detailed reports
Activity Graphs

All of this is stored in a sqlite database, if you install the app using flatpak you can find the database in this path: ~/.var/app/net.danigm.timetrack/data/timetrack.sqlite3.

I've done the icon for this app using the gnome-clocks icon and adding some colors to it.

If you find this app useful, don't hesitate and use it. Any feature request or bug report is welcome. And of course, all is free software, so if you want to collaborate, go ahead and send me some Merge Request.

GNOME Outreachy mentorship

danigm — Sun, 13 Jan 2019 00:00:00 +0100

The Outreachy program is a three month internship to work in FOSS. There are two periods for the outreachy, the first one from December to March and the other one from May to August. It's similar to the Google Summer Of Code, but in this case the interns doesn't need to be students.

I proposed some ideas for interts to work on GNOME, with me as a mentor. I wrote three proposals this time:

Extend Fractal media viewer with video support and explore video conference
Create Gtranslator initial integration with Damned Lies
Books : Improve the epub support in gnome-books

There was people interested in all of three projects, but for the Books app we don't have any real contribution so there was no real applicants.

I've two good proposals for Fractal and for Gtranslator, so I approve both and the Outreachy people approve these two interts. So we get two new devs working in GNOME for three months as interns.

This is something great, paid developers working in my proposals is a good thing, but this implies that I need to do the mentor work for these two interns during the three months period, so it's more work for me :/

But I think this is a really important work to do to bring more people to the free software, so I've less time for hacking, but I think it's good, because the fresh blood can do the hacking and if, after the Outreachy, one of the interns continues collaborating with GNOME, that will be more important for the GNOME project that some new features in one app.

GNOME Translation Editor

Teja is the intern working in gtranslator. She is working in the Damned Lies integration.

Damned Lies is a web application for GNOME translators. This app provides updated .po file for each GNOME module and language and translators can download and update that file using the web interface. That web is able to do the commit to the original repository with the upload version from translators.

The idea is to provide a simple integration with this platform in the GNOME Translation Editor app, using the web json API to be able to open .po files from the web directly without the need to download the file and then open it.

The current API of DL is really simple so we can't implement a real integration without adding more functionality to this API. So this project requires some work in the DL app too.

In the future we can improve the integration adding the posibility to upload the new .po after saving to DL so translators doesn't need to go to the web interface and can do all the translation flow using only the Translation Editor.

Fractal

Maira is the intern working in Fractal. They are working in the initial video preview widget.

Fractal is a instant messaging app that works over matrix.org. Currently we support different types of messages, like text, images, audio and files. But for video we're using the same widget as we're using for files, so you can download or open, but we've not a preview or inline player.

The main idea of this project is to provide a simple video player using gstreamer to play the video inside the Fractal app.

This is not an easy task, because we're using Rust in Fractal and we need to deal with bindings and language stuff, but I think it's doable.

During the internship, Maira is also working fixing some bugs in the audio player, becuase it uses gstreamer too, so during the code review, Maira detected some problems and they are fixing it.

Fractal December'18 Hackfest (part 2)

danigm — Sun, 16 Dec 2018 00:00:00 +0100

The Friday 14th was the last day of the second Fractal Hackfest. I've not spend much time writing real code, the Thursday was mainly another hacking day and I've been able to continue with the fractal-backend creation, but there's a lot of work to do there.

But the hackfest was really productive, we've talked about big issues, project management, some design ideas, new functionalities, the application refactor, etc.

GNOME newcomers experience

We talked about how to improve the GNOME newcomers experience and how to improve the main view of Fractal. I think that Tobias will talk more about this, he was working in some cool design for this and I think we can start to implement this new views soon.

Best practices

We've been developing Fractal in a fast way, without spend a lot of time thinking about the code quality, maintainability and that stuff. Recently we've set the rustfmt linter in the CI pipeline and we've some tests, but the Merge Requests process wasn't defined and for example I was pushing directly to master.

To improve the quality of Fractal we've started a new wiki page to have a list of best practices to follow. There we'll add some guidances on how we should write code and the processes to follow to improve Fractal.

We've decided that we should be more strict with the Merge Request code review and now, direct push to master is not allowed, all changes will go through the review process. We should wait at least two days to merge something and have at least two people that approve the change.

This will slow down the MR process, but will improve the code quality and will reduce regressions. Any help is welcome, if you're able to test the MR, you can leave a comment and other reviewers will have more confident in the change.

We're also working in the code quality using the cargo clippy tool. There's a Merge Request waiting for review, so we'll have a better rust source code soon.

Fractal is now in the GNOME group

The Fractal project was on the World group inside the GNOME gitlab. Fractal is a GNOME application, the most active developers are GNOME developers and we try to follow the GNOME Human Interface Guidelines.

Fractal is one of the first new applications that born just during the gitlab migration so we go through a new process. At first Fractal was in my personal gitlab under /danigm/fractal then we move to the World group under /World/fractal and finally we're in the main GNOME group /GNOME/fractal.

New release 4.0.0

The last release was the 3.30.0, more than three months has passed since this release and we've a lot of changes so we want to provide a new stable release.

We discuss a bit about the version number that we should follow and we decided that we should do use our own version number system, because we want to release as much as possible, when we've important changes.

We're working in the 4.0.0 release, we're stabilizing and fixing important bugs before the release and maybe we can have the new release during the next week.

So I've spend the last day looking for bugs and preparing the new release.

Matrix Live

We've a meeting with the people from Matrix.org to talk about Fractal and the hackfest. You can view the full interview in youtube:

Friday sponsored lunch

We've a sponsored lunch the Friday 14th, the local group Plan4D invite us to a great lunch in the city center.

After that, I goes back to my home and leave the other people there in Seville doing some tourism, I've to go back to Málaga.

The hackfest was great and we have done a lot of things. This was the second Fractal hackfest in 2018, and we meet at the GUADEC too. We've had two GSoC students and now we've another intern thanks to the outreachy program. There's a lot of people contributing to Fractal and that's great.

GNOME is a great community and I think that Matrix.org and Rust are helping to this project a lot. The Matrix.org people are supporting us, indeed, Matthew comes to the first hackfest and I think that the success of Fractal and the community behind has a lot to thank to the Rust language and to the people working in the Rust + GNOME integration, Gtk-rs is a great project.

I want to apologize about the network problem during the hackfest. We've been working all days thank to Julian network sharing, with eduroam, because we aren't able to have guest access in the university.

The university has a strict internet connection filtering, so only a professor can ask for a guest connection for events and we do the request too late.

Fractal December'18 Hackfest (part 1)

danigm — Wed, 12 Dec 2018 00:00:00 +0100

First two days of work

The Tuesday 11th started the second Fractal Hackfest. I've organized this hackfest in Seville, the city where I studied computer science and here I've a lot of friends in the University so is a good place to do it here.

The weather was important too for the hackfest selection, in December Seville is a good choice because the weather is not too cold, we're having sunny days.

The first day was a good day, thinking about some relevant issues and planning what we want to do. We talked about the work needed for the interface split, about the E2EE support, new features and the need for a new release.

We're having some problems with the internet connection, because the University has a restricted network policy and we ask for the guess internet connection the Monday, but we're still waiting.

Meantime we have to thanks to Julian Sparber the hackfest wifi, because he's using his laptop to stream the eduroam connection.

Newcomers

The first day we try to promote as a newcomers day, and some people comes. We've some contributions and I spend some time trying to help people to introduce to the GNOME community.

GNOME Foundation sponsored dinner

The GNOME Foundation payed for a Fractal dev day dinner, so we should thanks this great dinner to GNOME:

We've a good time eating and drinking in the Coco Verde. We talk about GNOME, the desktop, matrix.org and other communication tools.

Hacking day

Today was a hacking day, so we started to work in the stuff that we talk about yesterday. We've resolved some minor issues and I started with the fractal-backend crate.

We talked about move all the app data model and logic to the fractal-backend and leave fractal-gtk only as UI management.

We talked about the LMDB use to store rooms and messages and we decided that the best solution should be to use a relational database, because we've relations and with the key-value thing we'll end creating those relations and maintaining by hand. In any case, I've this decision in mind and I'm implementing all this with a trait to hide the storage detail so we can change easily in the future.

So today I've spend a lot of time implementing this trait and a first implementation for the Room struct. I've decided to use rusqlite instead the diesel orm because I want to keep it simple.

I want to finish all the database storage and move the AppOp main loop to the fractal-backend and try to update the fractal-gtk to use the backend instead the AppOp struct to get the rooms and messages. But maybe is a lot of work, I don't know if I'll be able to finish this before the end of the hackfest.

Thanks

This hackfest is possible because there's a lot of volunteer work and people helping us. First of all, the Fractal core team that comes Tobias Bernard, Julian Sparber and Alexandre Franke and of course the GNOME Foundation. And also we've to thank Alejandro Domínguez, a newcomer that is doing a really good job fixing bugs and cleaning some old code.

Then I want to thank the Linux local group, SUGUS for the help and also I want to thank to other free software related group Plan4D.

Plan4D is giving us some help with the place and cookies, fruits, juices and some tea.

And finally I want to thank again to my coworkers in wadobo because they support me to spend a full work week working in gnome and they also spend some time helping us to organize all the hackfest stuff.

GNOME Gtk Developer

danigm — Wed, 05 Dec 2018 00:00:00 +0100

I tried and I failed.

This summer, during the GUADEC 2018, the GNOME foundation announces some job positions. One of that job was for a Gtk+ core developer. I tried to get that job and after a long period I was rejected.

The final developer selected for this possition is Emmanuel Bassi, a very active developer in the core of GNOME and the one that's behind the great The History of GNOME Podcast.

To be honest, I really wanted that job. It's a dream job for me, working fulltime in a free software project, by a foundation, with the great GNOME technology and community.

But I've to say that just when I saw the foundation announcement about this position I thought about ebassi, because as far as I know, he's the best one to do that job. In any case, I wanted to give a try and go for that job, I didn't know about other people going for it, so maybe I can do it.

Some background

I'm working in my own little company since 2010. We're three coworkers with the same percentage of the company, like a cooperative and we work mainly as external consultant for bigger companies. We do a lot of work related with GNU/Linux and free software, but lately we mainly do web apps with python and react, because that's what our current clients ask for.

But the first paid project for wadobo was a GNOME project, I started the company with a small project to improve accessibility in PDF documents, working directly in evince. And then we worked with Qt, QML and other technologies, but I wasn't able to get a paid work on GNOME or Gtk+ since that initial work.

In any case, I continued working in GNOME in my spare time, trying to contribute, first with libgepub, trying to bring epub support to evince, and then to GNOME Books. And the last year I became more active because of Fractal and Gnome Translation Editor (gtranslator). And I've been mentor for the GSoC 2018 and now I've two interns in the Outreachy program.

I've some time to spend in GNOME, but I'm not making money from that, so now I'm starting to look for that actively, because I really want to have more time to spend working with free software.

So, I talked with my coworkers, to be honest with them, and I told them that I was going to try to get the Gtk+ core developer position.

The process

This was a long process, I think that it was long because there was a lot of people for that position.

The Email: I sent the initial email in mid July, with a long description about me and my work in gnome and free software, with a little hope in my heart.
The Survey: Then I received an email with a link to a survey to complete with more information. That was great, because it sounds like I've passed the first cut.
The Interview: At the end of August I received a conglatulations email. My resume was positively reviewed and I'll have a personal interview.

That was really nice and I started to think that maybe it was possible after all.

At the mid of September, I've a video conference meeting with Carlos Soriano and Matthias Clasen. For me was a great interview, talking about my experience, GNOME and Gtk+. Maybe the interview was a disaster and they are just a very nice people being nice with a poor developer, but at the end I thought that it was well and I was very exited about this.

The failure

After a long time without knowing anything, I received the email that I was waiting for. I wasn't selected for the position. The next week, the foundation announces the new staff member and in that moment I noticed that I've had no chance to get the job. He's the better candidate and I'm really happy because this is really good for GNOME.

And now what?

At the end of the process, I think that this was a great experience for me, because this forces me to think about my career, what I want to do in the future and I went through a full selection process, something that I've not done since 2009.

I will continue trying to find the way to work in GNOME or something realated with the GNU/Linux desktop or core technologies.

I did a lot of web development, I've a lot of experience and I'm really good doing web with django, but I really want to go back and work with other technologies, programming in C, the new Rust language and with some low level technologies.

If someone need help, I'm here and I can work as a contractor. For now I'll continue working in Wadobo, trying to find clients that requires desktop or low level work.

GNOME Translation Editor 3.30.0

danigm — Sat, 03 Nov 2018 00:00:00 +0100

I'm pleased to announce the new GNOME Translation Editor Release. This is the new release of the well known Gtranslator. I talked about the Gtranslator Ressurection some time ago and this is the result:

This new release isn't yet in flathub, but I'm working on it so we'll have a flatpak version really soon. Meantime you can test using the gnome nightly flatpak repo.

New release 3.30.0

This release doesn't add new functionality. The main change is in the code, and in the interface.

We've removed the toolbar and move the main useful buttons to the headerbar. We've also removed the statusbar and replaced it with a new widget that shows the document translation progress.

The plugin system and the dockable window system has been removed to simplify the code and make the project more maintenable. The only plugin that is maintained for now is the translation memory, that's now integrated. I'm planning to migrate other useful plugins, but that's for the future.

Other minor changes that we've made are in the message table, we've removed some columns and now we only show two, the original message and the translated one and we use colors and text styles to show fuzzy status and untranslated.

The main work is a full code modernization, now we use meson to build, we've flatpak integration and this simplify the development because gtranslator know works by default in Gnome Builder without the need to install development dependencies.

There's others minor changes like the new look when you open the app without any file:

Or the new language selector that autofill all the profile fields using the language:

And for sure we've tried to fix most important bugs:

New name and new Icon

Following the modern GNOME app names, we've renamed the app from Gtranslator to GNOME Translation Editor. Internally we'll continue with the gtranslator name, so the app is gtranslator, but for the final user the name will be Translation Editor.

And following the App Icon Redesign Initiative we've a new icon that follows the new HIG.

Thanks

I'm not doing this alone. I became the gtranslator maintainer because Daniel Mustieles push to have a modern tool for GNOME translators, done with gnome technology and fully functional.

The GNOME Translation Editor is a project done by the GNOME community, there are other people helping with code, documentation, testing, design ideas and much more and any help is always welcome. If you're insterested, don't hesitate and come to the gnome gitlab and collaborate with this great project.

And maybe it's a bit late, but I've publish a project to the outreachy.org, so maybe someone can work on this as an intern for three months. I'll try to get more people involved here using following outreachy and maybe GSoC, so if you're a student, now is the right time to start contributing to be able to be selected for the next year internship programs.

Gtranslator Resurrection

danigm — Mon, 10 Sep 2018 00:00:00 +0200

The last week I received a telegram message about Gtranslator, that was unmaintained for a long time. GNOME translators uses different tools to translate .po files, Gtranslator is a tool for translator that is integrated with the GNOME desktop, but with the time, Gtranslator is getting old and there are several known bugs that never get fixed.

So I decided to go ahead and become the maintainer of Gtranslator with the main idea of update the interface and fix mayor bugs.

Why we should care about Gtranslator

PO files are plain text files that can be edited with any text editor, but there're a lot of tools for translator to simplify the edition of .po files, these way the program will avoid formatting problems and can also provide some tools to help in the translation.

There's a lot of tools to edit .po files, so, why we need another application? why translators can't pick another useful tool if Gtranslator is unmaintained?

The main reason is the same as we've a text editor or the gnome-builder or other developer tools, because all users want to use applications integrated in the desktop, so it always will be better for a GNOME user a native GNOME app to edit .po files than a Qt one or a webpage.

The other reason is that we need to simplify the translators life to have better and more translations and that can only be done if we're developping the tool that translator will use, because in the future we can integrate Gtranslator with the gnome gitlab, or with Damned Lies, to download or upload translation files.

What's the plan

Gtranslator has a GNOME 2.0 interface, with toolbar, menus, statusbar and dockable widgets:

But in the git repo, the main branch was broken. Someone starts a big refactor, in 2015, to try to update the interface to look more like a modern GNOME 3 application but that refactor never ended so the master branch compiles, but doesn't do anything. There was a lot of widgets that was half ported to a modern code.

There's also some designs to modernize the interface, but the work wasn't finished.

My plan is to continue that work, modernize the UI, remove all deprecated code and warnings and then continue the development from there.

I don't want to spend a lot of time reworking the whole app to have a new one in two months, I want to have a functional app as soon as possible, and continue with a continuous development, improving the UI and the functionality needed by the translator team.

So the roadmap that I've in mind is:

Fix the current master state
Fix the known bugs
Redesign the interface
Enhance linking with Damned Lies

What I've done in one week

I started updating the build system, from autotools to meson build, and then I've added a flatpak manifest file, so now gnome-builder is able to build Gtranslator using meson and flatpak and it's easier to develop this way.

This build tools modernization is great because with the new gitlab we've now continuous integration, and that means that the gitlab compiles and generates a flatpak bundle for each commit, making it easier for testers to test new functionality or Merge Requests, you only need to download the .flatpak and install locally, no need to build at all.

Then I started to fix the master branch. There was a project selector view and then you go to the translation view. There was only the .ui files and not working yet, so I reworked that a bit and convert the project selector to a file selector for now and then when you select a file, you've almost the same old interface, but without the menu, toolbar and statusbar.

I've been moving the toolbar icons to the headerbar so we can have the main functionality working. We've now a functional application.

Plugins

Gtranslator has a plugin system and has eight plugins, but that plugins was disabled some time ago. So I've started to recover that functionality but instead of as plugins, I've started to move the plugins to the core app code.

The only plugin ported currently is the translation memory. I'll take a look to the other plugins and view if it's desirable to reimplement that or remove that code.

What's next

All the statusbar functionality is missing right now, I need to find some place in the new UI to show that.

I wan't to start as soon as possible to work in a new widget to show messages, instead of the current table.

The project selection part is a bit hard because I need to think about what's a project in this context. Currently Gtranslator is a .po file editor, but if we change that to manage projects like a group of .po files, I need to think about how to store that information and the easier way for translator to use the interface.

So if you've time and want to contribute in this project, go ahead, you can ask in the #gtranslator IRC channel or talk to me directly. Currently there's no a lot of issues in the gitlab, but as soon as I get a functional version in gnome-nightly flatpak repo and some translators starts to use it, we'll have a lot of things to do.

GIR support in gnome-class

danigm — Sat, 01 Sep 2018 00:00:00 +0200

Recently I've been working again in the rust port of libgepub, libgepub is C code, but in the rust-migration branch almost all the real functionality is done with rust and the GepubDoc class is a GObject wrapper around that code.

For this reason I was thinking about to use gnome-class to implement GepubDoc.

Gnome-class is a rust lib to write GObject code in rust that's compatible with the C binary API so then you can call this new GObject code written with gnome-class from C. I've worked a little in gnome-class, implementing a basic properties support.

GObject Introspection

A great advantage of write GObject code in C is that you can have automatic bindings using GObject Introspection. This is great because you only need to write the C code, write correctly the documentation of the public API and then you'll get GIR for free, and that means that you can use your lib from gjs, python and other automatic bindings.

GIR defines a simple XML document to declare the public API and there is the meta information needed for bindings, to know how to call the lib functions and how to convert params and also provides memory management information, for languages with garbage collector and so. These files are places under /usr/share/gir-1.0/ and it looks like this:

      <method name="get_cover" c:identifier="gepub_doc_get_cover">
        <return-value transfer-ownership="full">
          <doc xml:space="preserve">cover file path to retrieve with
gepub_doc_get_resource</doc>
          <type name="utf8" c:type="gchar*"/>
        </return-value>
        <parameters>
          <instance-parameter name="doc" transfer-ownership="none">
            <doc xml:space="preserve">a #GepubDoc</doc>
            <type name="Doc" c:type="GepubDoc*"/>
          </instance-parameter>
        </parameters>
      </method>

This xml is compiled to a binary format to be used from bindings and those compiled gir files are places under /usr/lib64/girepository-1.0/ with the .typelib extension.

This is great, and the GepubDoc class is exported as GIR and indeed it's used from javascript code in the gnome-books application using this. If we want to migrate libgepub from C to rust, we need to provide this GIR to don't break the compatibility in other apps. That's done currently in the rust-migration branch by hand, creating the GepubDoc in C and calling the rust code from there.

With gnome-class it should be possible to write that GepubDoc code in rust and have the same ABI, but currently gnome-class doesn't generate GIR, that's a little problem that I can try to solve using my dev skills.

Adding GIR to gnome-class

So, libgepub is the excuse to start to implement GIR in gnome-class. There's an issue in the gitlab with an initial syntax proposal and I started from that:

#[generate_gir("foo.gir")]
#[generate_c_header("foo.h")]
gobject_gen! {
    "foo.gir",
    "foo.h",

    class Foo {
        ...
    }

    impl Foo {
        ...
    }
}

The first thing to do is to add a way to tell that we want to generate the GIR file in the class definition. In this proposal, the GIR tag is before the gobject_gen proc-macro, that can't be done, or I don't know how to do that easily, so I changed to a declaration inside the proc-macro, that's what we're parsing in gnome-class:

gobject_gen! {
    #[generate_gir("Counter.gir")]
    class Counter {
      f: Cell<u32>,
    }

    impl Counter {
        pub fn add(&self, x: u32) -> u32 {
            self.get_priv().f.set(self.get() + x);
            self.get()
        }

        pub fn get(&self) -> u32 {
            self.get_priv().f.get()
        }
    }
}

With the code insithe de gobject_gen I only need to update the parser to allow this new syntax:

named!{parse_gir -> Option<LitStr>,
    option!(do_parse!(
        punct!(#)                               >>
        name: brackets!(do_parse!(
            call!(keyword("generate_gir"))      >>
            name: parens!(syn!(LitStr))         >>
            (name.1)
        )) >>
        (name.1)
    ))
}

impl Synom for ast::Class {
    named!(parse -> Self, do_parse!(
        gir: call!(parse_gir)                                    >>
        call!(keyword("class"))                                  >>
        name: syn!(Ident)                                        >>
        extends: option!(do_parse!(
            punct!(:)                                            >>
            superclass: syn!(Path)                               >>
            // FIXME: interfaces
            (superclass)))                                       >>
        fields: syn!(FieldsNamed)                                >>
        (ast::Class {
            gir,
            name,
            extends,
            fields,
        })
    ));

Then, I'm storing the GIR file name in the ast class so I can check if that field is None or Some and generate the GIR in that case.

Generating the XML

The parser was the easier part, we've now the hability to define if we want to generate the GIR file and we're storing that information in the High-level Internal Representation (hir).

To generate the XML I've to add a new call before generating the code:

     let result: Result<proc_macro2::TokenStream> =
-        hir::Program::from_ast_program(&ast_program).and_then(|program| Ok(gen::codegen(&program)));
+        hir::Program::from_ast_program(&ast_program)
+            .and_then(|program| {
+                gen::gir::generate(&program)?;
+                Ok(gen::codegen(&program))
+            });

     match result {
         Ok(tokens) => {

The gir module generates the XML, iterating over all classes defined in the program.

The XML isn't really hard to generate, we've almost all the information so we can iterate over all the class methods and add to the XML. There's some information that's not really easy to get so I've let some TODO in this initial version, like:

Find the parent class name (GObject.Object by default)
Get the library version and shared-lib path (lib#CLASSNAME#1.0.0 by default)
Get dependencies

So this is not a full featured GIR support, but it's an initial step to get it, and it works, at least with simple examples :D

Example project

To test that this is working correctly I've created a simple example project, that uses gnome-class, generates the gir and has an small script to call the generated code from python.

This is the rust code:

gobject_gen! {
    #[generate_gir("Counter.gir")]
    class Counter {
      f: Cell<u32>,
    }

    impl Counter {
        pub fn add(&self, x: u32) -> u32 {
            self.get_priv().f.set(self.get() + x);
            self.get()
        }

        pub fn get(&self) -> u32 {
            self.get_priv().f.get()
        }
    }
}

I've a bash script to compile and generate the typelib file from the xml, generated by gnome-class:

cargo +nightly build
g-ir-compiler Counter.gir > Counter-1.0.typelib

And I've written a test in python using gi.repository:

import unittest
from gi.repository import Counter

class TestCounter(unittest.TestCase):

    def test_new(self):
        c = Counter.Counter()
        self.assertEqual(c.get(), 0)

    def test_add(self):
        c = Counter.Counter()
        self.assertEqual(c.add(1), 1)
        self.assertEqual(c.add(3), 4)

    def test_get(self):
        c = Counter.Counter()
        self.assertEqual(c.get(), 0)
        self.assertEqual(c.add(1), 1)
        self.assertEqual(c.get(), 1)

if __name__ == '__main__':
    unittest.main()

This is working right now, so we're getting closer to the final goal of having libgepub written with rust code and have GIR generation.

libgepub + rust

danigm — Thu, 09 Aug 2018 00:00:00 +0200

In 2010 I was working with evince, the gnome PDF document viewer, trying to add some accessibility to PDF files. That was really hard, not because GTK+ or ATK technology but because the PDF format itself. The PDF format is really cool for printing because you know that the piece of paper will look the same as the PDF doc, and because it's vector it scales and don't loose quality and files are smaller than image files, but almost all PDF files have not any metadata for sections, headings, tablets or so, this depends on the creation tool, but it's really hard to deal with PDF content text, because you don't know event if the text that you're reading is really in the same order that you read from the PDF.

After my fight against the PDF format hell and poppler, I discovered the epub format that's a really simple format for electronic books. An epub is a zip with some XML files describing the book index and every chapter is a xhtml and xhtml is a good format compared to PDF because you can parse easily with any XML lib and the content is tagged and well structured so you know what's a heading, what's a paragraph, etc.

So I started to write a simple C library to read epub files, thinking about add epub support to evince. That's how libgepub was born. I tried to integrate libgepub in evince, I've something working, rendering with webkit, but nothing really useful, because evince needs pages and it's not easy to split an xhtml file in pages with the same height, because xhtml is continuous text and it adapts to the page width, so I give up and leave this branch.

gnome-books

After some time, I discovered gnome-books, and the idea of epub support comes to me again. Books has an initial epub support, that consists only in showing in the collection, but when you click on the book, an error message was shown because there's no epub preview support.

I've libgepub with GIR support and books was written with gjs so it was easy for me to add an initial support for epub documents using libgepub and rendering with webkit.

So libgepub is used in a gnome app to render epub documents and for that reason gnome depends on libgepub now.

Rustifying everything

In 2017 I discover rust and fell in love with this language, so I wanted to write some code with rust and I started porting this epub library and I ended creating the epub crate, that's basically the same implementation but with rust instead of C, the API is really similar.

After that, I read somewhere that Federico was porting the librsvg code to rust, so I thought that I can do the same with libgepub and replace almost all the C code with this new rust crate.

I copied all the code for autotools to build using cargo and I've copied to the glue code from librsvg and adapt to my own lib. That worked really well and I was able to remove the core libgepub C code and replace with the new rust code.

But rust is really new and it was not full supported in all distributions, so release depending on rust will make the release people's live harder. That's the reason because I decided to leave the C version for now and wait a little to make the rust migration.

And here we're, the libgepub has changed a little, some fixes and small changes, but the main change is that now we're using meson to build instead of autotools, and it's not easy to integrate a rust+cargo lib in the meson desc. There's a PR in meson to support this but it seems that meson devs doesn't like the idea.

So yesterday I was playing around with meson to get back the rust code working in libgepub, and was not easy, but now I've a working build configuration that works. It's a hack with a custom script to build with cargo, but it's working so I'm really happy.

Show me the code

To integrate the rust epub crate in the libgepub C lib I needed to write a simple cargo project with the glue code that converts the rust types to glib/c types and expose that in a C lib, and that's the libgepub_internals, that's full of glue rust+glib code like:

#[no_mangle]
pub extern "C" fn epub_new(path: *const libc::c_char) -> *mut EpubDoc {
    let my_path = unsafe { &String::from_glib_none(path) };
    let doc = EpubDoc::new(my_path);
    let doc = doc.unwrap();
    Box::into_raw(Box::new(doc))
}

#[no_mangle]
pub unsafe extern "C" fn epub_destroy(raw_doc: *mut EpubDoc) {
    assert!(!raw_doc.is_null());
    let _ = Box::from_raw(raw_doc);
}

Then, in the gepub-doc.c, that's the glib object that expose the GIR API, I added calls to this functions, defining each function heading that'll be used from the rust static lib at link time:

// Rust
void      *epub_new(char *path);
void       epub_destroy(void *doc);
void      *epub_get_resource(void *doc, const char *path, int *size);
void      *epub_get_resource_by_id(void *doc, const char *id, int *size);
void      *epub_get_metadata(void *doc, const char *mdata);
void      *epub_get_resource_mime(void *doc, const char *path);
void      *epub_get_resource_mime_by_id(void *doc, const char *id);
void      *epub_get_current_mime(void *doc);
void      *epub_get_current(void *doc, int *size);
void      *epub_get_current_with_epub_uris(void *doc, int *size);
void       epub_set_page(void *doc, guint page);
guint      epub_get_num_pages(void *doc);
guint      epub_get_page(void *doc);
gboolean   epub_next_page(void *doc);
gboolean   epub_prev_page(void *doc);
void      *epub_get_cover(void *doc);
void      *epub_resource_path(void *doc, const char *id);
void      *epub_current_path(void *doc);
void      *epub_current_id(void *doc);
void      *epub_get_resources(void *doc);
guint      epub_resources_get_length(void *er);

gchar     *epub_resources_get_id(void *er, gint i);
gchar     *epub_resources_get_mime(void *er, gint i);
gchar     *epub_resources_get_path(void *er, gint i);

And then calling to this functions:

static gboolean
gepub_doc_initable_init (GInitable     *initable,
                         GCancellable  *cancellable,
                         GError       **error)
{
    GepubDoc *doc = GEPUB_DOC (initable);

    g_assert (doc->path != NULL);
    doc->rust_epub_doc = epub_new (doc->path);
    if (!doc->rust_epub_doc) {
        if (error != NULL) {
            g_set_error (error, gepub_error_quark (), GEPUB_ERROR_INVALID,
                         "Invalid epub file: %s", doc->path);
        }
        return FALSE;
    }

    return TRUE;
}

To make this work, I needed to add the libgepub_internals dependency to the libgepub meson.build like this:

gepub_deps = [
  dependency('gepub_internals', fallback: ['libgepub_internals', 'libgepub_internals_dep']),
  dependency('webkit2gtk-4.0'),
  dependency('libsoup-2.4'),
  dependency('glib-2.0'),
  dependency('gobject-2.0'),
  dependency('gio-2.0'),
  dependency('libxml-2.0'),
  dependency('libarchive')
]

This definition looks for the gepub_internals lib in the libgepub_internals subproject with this meson.build:

project(
  'libgepub_internals', 'rust',
  version: '3.29.6',
  license: 'GPLv3',
)

libgepub_internals_version = meson.project_version()
version_array = libgepub_internals_version.split('.')
libgepub_internals_major_version = version_array[0].to_int()
libgepub_internals_minor_version = version_array[1].to_int()
libgepub_internals_version_micro = version_array[2].to_int()

libgepub_internals_prefix = get_option('prefix')

cargo = find_program('cargo', required: true)
cargo_vendor = find_program('cargo-vendor', required: false)
cargo_script = find_program('scripts/cargo.sh')
grabber = find_program('scripts/grabber.sh')
cargo_release = find_program('scripts/release.sh')

c = run_command(grabber)
sources = c.stdout().strip().split('\n')

cargo_build = custom_target('cargo-build',
                        build_by_default: true,
                        input: sources,
                        output: ['libgepub_internals'],
                        install: false,
                        command: [cargo_script, '@CURRENT_SOURCE_DIR@', '@OUTPUT@'])

libgepub_internals_lib = static_library('gepub_internals', cargo_build)

cc = meson.get_compiler('c')
th = dependency('threads')
libdl = cc.find_library('dl')

libgepub_internals_dep = declare_dependency(
  link_with: libgepub_internals_lib,
  dependencies: [th, libdl],
  sources: cargo_build,
)

I'm using here a custom_target to build the lib using a custom script that simply calls to cargo and then copies the result lib to the correct place:

if [[ $DEBUG = true ]]
then
    echo "DEBUG MODE"
    cargo build --manifest-path $1/Cargo.toml && cp $1/target/debug/libgepub_internals.a $2.a
else
    echo "RELEASE MODE"
    cargo build --manifest-path $1/Cargo.toml --release && cp $1/target/release/libgepub_internals.a $2.a
fi

Then I declared the static_library and the dependency with declare_dependency. I need to add threads and dl because the epub crate depends on it and this works!

I'll need to vendor all dep crates with cargo-vendor for releasing, but I think that this is working and it's the way to go with libgepub.

The future of libgepub + rust

Currently, with the libgepub_internals lib, the gepub-doc.c code is basically to provide a gobject and GIR information to be able to work with epub docs, but the real work is done in Rust. The gnome-class provides a simple way to build this gobject with rust code, but currently it's not completed and there's no way to generate GIR, but in the future, it could be cool to remove the gepub-doc.c code and generate all with gnome-class. I can wait until Federico writes the piece of code that I need for this or maybe I should contribute to gnome-class to be able to do this.

Libgepub also provides a widget that inherits from WebkitWebView to render the book. That widget is written in C to provide GIR data also and we can try to do the same and use gnome-class to write this widget.

But for now, we're really far from this, we need to spend some time improving gnome-class to be able to write all the code in rust and expose the gobject GIR. Meantime we can start to use rust with this glue code, and that's great, because if you've a gobject library and you want to migrate to rust, you don't need to migrate all the code at once, you can do the same that librsvg is doing and migrate function by function and that's really cool.

mdl

danigm — Fri, 03 Aug 2018 00:00:00 +0200

The last month I wrote a blog post about the LMDB Cache database and my wish to use that in Fractal. To summarize, LMDB is a memory-mapped key-value database that persist the data to the filesystem. I want to use this in the Fractal desktop application to replace the current state storage system (we're using simple json files) and as a side effect we can use this storage system to share data between threads because currently we're using a big struct AppOp shared with Arc<Mutex<AppOp>> and this cause some problems because we need to share and lock and update the state there.

The main goal is to define an app data model with smaller structs and store this using LMDB, then we can access to the same data querying the LMDB and we can update the app state storing to the LMDB.

With this change we don't need to share these structs, we only need to query to the LMDB to get the data and the work with that, and this should simplify our code. The other main benefit will be that we'll have this state in the filesystem by default so when we open the app after close, we'll stay in the same state.

Take a look to the gtk TODO example app to view how to use mdl with signals in a real gtk app.

What is mdl

mdl is Data model library to share app state between threads and process and persist the data in the filesystem. Implements a simple way to store structs instances in a LMDB database and other methods like BTreeMap.

I started to play with the LMDB rust binding and writing some simple tests. After some simple tests, I decided to write a simple abstraction to hide the LMDB internals and to provide a simple data storage and to do that I created the mdl crate.

The idea is to be able to define your app model as simple rust structs. LMDB is a key-value database so every struct instance will have an unique key to store in the cache.

The keys are stored in the cache ordered, so we can use some techniques to store related objects and to retrieve all objects of a kind, we only need to build keys correctly, following an scheme. For example, for fractal we can store rooms, members and messages like this:

rooms with key "room:roomid", to store all the room information, title, topic, icon, unread msgs, etc.
members with key "member:roomid:userid", to store all member information.
messages with key "msg:roomid:msgid" to store room messages.

Following this key assignment we can iterate over all rooms by querying all objects that starts with "room", we can get all members and all messages from a room.

This have some inconveniences, because we can't query directly an message by id if we don't know the roomid. If we need that kind of queries, we need to think about another key assignment or maybe we should duplicate data. key-value are simple databases so we don't have the power of relational databases.

Internals

LMDB is fast and efficient, because it's in memory so using this cache won't add a lot of overhead, but to make it simple to use I've to add some overhead, so mdl is easy by default and can be tuned to be really fast.

This crate has three main modules with traits to implement:

model: This contains the Model trait that should implement every struct that we want to make cacheable.
store: This contains the Store trait that's implemented by all the cache systems.
signal: This contains the Signaler trait and two structs that allow us to emit/subscribe to "key" signals.

And two more modules that implements the current two cache systems:

cache: LMDB cache that implements the Store trait.
bcache: BTreeMap cache that implements the Store trait. This is a good example of other cache system that can be used, this doesn't persist to the filesystem.

So we've two main concepts here, the Store and the Model. The model is the plain data and the store is the container of data. We'll be able to add models to the store or to query the store to get stored models. We store our models as key-value where the key is a String and the value is a Vec<u8>, so every model should be serializable.

This serialization is the bigger overhead added. We need to do this because we need to be able to store this in the LMDB database. Every request will create a copy of the object in the database, so we're not using the same data. This can be tuned to use pointers to the real data, but to do that we'll need to use unsafe code and I think that the performance that we'll get with this doesn't deserve the complexity that this will add.

By default, the Model trait has two methods fromb and tob to serialize and deserialize using bincode, so any struct that implements the Model trait and doesn't reimplement these two methods should implement Serialize and Deserialize from serde.

The signal system is an addition to be able to register callbacks to keys modifications in the store, so we can do something when a new objects is added, modified or deleted from the store. The signaler is optional and we should use it in a explicit way.

How to use it

First of all, you should define your data model, the struct that you want to be able to store in the database:

#[derive(Serialize, Deserialize, Debug)]
struct A {
    pub p1: String,
    pub p2: u32,
}

In this example we'll define a struct called A with two attributes, p1, a String, and p2, an u32. We derive Serialize and Deserialize because we're using the default fromb and tob from the Model trait.

Then we need to implement the Model trait:

impl Model for A {
    fn key(&self) -> String {
        format!("{}:{}", self.p1, self.p2)
    }
}

We only reimplement the key method to build a key for every instance of A. In this case our key will be the String followed by the number, so for example if we've something like let a = A { p1: "myk", p2: 42 }; the key will be "myk:42".

Then, to use this we need to have a Store, in this example, we'll use the LMDB store that's the struct Cache:

    // initializing the cache. This str will be the fs persistence path
    let db = "/tmp/mydb.lmdb";
    let cache = Cache::new(db).unwrap();

We pass the path to the filesystem where we want to persist the cache as the first argument, in this example we'll persist to "/tmp/mydb.lmdb". When we ran the program for the first time a directory will be created there. The next time, that cache will be used with the information from the previous execution.

Then, with this cache object we can instantiate an A object and store in the cache:

    // create a new *object* and storing in the cache
    let a = A{ p1: "hello".to_string(), p2: 42 };
    let r = a.store(&cache);
    assert!(r.is_ok());

The store method will serialize the object and store a copy of that in the cache.

After the store, we can query for this object from other process, using the same lmdb path, or from the same process using the cache:

    // querying the cache by key and getting a new *instance*
    let a1: A = A::get(&cache, "hello:42").unwrap();
    assert_eq!(a1.p1, a.p1);
    assert_eq!(a1.p2, a.p2);

We'll get a copy of the original one.

This is the full example:

extern crate mdl;
#[macro_use]
extern crate serde_derive;

use mdl::Cache;
use mdl::Model;
use mdl::Continue;

#[derive(Serialize, Deserialize, Debug)]
struct A {
    pub p1: String,
    pub p2: u32,
}
impl Model for A {
    fn key(&self) -> String {
        format!("{}:{}", self.p1, self.p2)
    }
}

fn main() {
    // initializing the cache. This str will be the fs persistence path
    let db = "/tmp/mydb.lmdb";
    let cache = Cache::new(db).unwrap();

    // create a new *object* and storing in the cache
    let a = A{ p1: "hello".to_string(), p2: 42 };
    let r = a.store(&cache);
    assert!(r.is_ok());

    // querying the cache by key and getting a new *instance*
    let a1: A = A::get(&cache, "hello:42").unwrap();
    assert_eq!(a1.p1, a.p1);
    assert_eq!(a1.p2, a.p2);
}

Iterations

When we store objects with the same key prefix we can iterate over all of the objects, because we don't know the full key of all objects.

Currently there's two ways to iterate over all objects with the same prefix in a Store:

all

This is the simpler way, calling the all method we'll receive a Vec<T> so we've all the objects in a vector.

    let hellows: Vec<A> = A::all(&cache, "hello").unwrap();
    for h in hellows {
        println!("hellow: {}", h.p2);
    }

This has a little problem, because if we've a lot of objects, this will use a lot of memory for the vector and we'll be iterating over all objects twice. To solve this problems, the iter method was created.

iter

The iter method provides a way to call a closure for every object with this prefix in the key. This closure should return a Continue(bool) that will indicates if we should continue iterating of if we should stop the iteration here.

    A::iter(&cache, "hello", |h| {
        println!("hellow: {}", h.p2);
        Continue(true)
    }).unwrap();

Using the Continue we can avoid to iterate over all the objects, for example if we're searching for one concrete object.

We're copying every object, but the iter method is better than the all, because if we don't copy or move the object from the closure, this copy only live in the closure scope, so we'll use less memory and also, we only iterate one. If we use all, we'll iterate over all objects with that prefix to build the vector so if we iterate over that vector another time this will cost more than the iter version.

Signal system

As I said before, the signal system provide us a way to register callbacks to key modifications. The signal system is independent of the Model and Store and can be used independently:

extern crate mdl;

use mdl::Signaler;
use mdl::SignalerAsync;
use mdl::SigType;
use std::sync::{Arc, Mutex};
use std::{thread, time}; 

fn main() {
    let sig = SignalerAsync::new();
    sig.signal_loop();
    let counter = Arc::new(Mutex::new(0));

    // one thread for receive signals
    let sig1 = sig.clone();
    let c1 = counter.clone();
    let t1: thread::JoinHandle<_> =
    thread::spawn(move || {
        let _ = sig1.subscribe("signal", Box::new(move |_sig| {
            *c1.lock().unwrap() += 1;
        }));
    });

    // waiting for threads to finish
    t1.join().unwrap();

    // one thread for emit signals
    let sig2 = sig.clone();
    let t2: thread::JoinHandle<_> =
    thread::spawn(move || {
        sig2.emit(SigType::Update, "signal").unwrap();
        sig2.emit(SigType::Update, "signal:2").unwrap();
        sig2.emit(SigType::Update, "signal:2:3").unwrap();
    });

    // waiting for threads to finish
    t2.join().unwrap();

    let ten_millis = time::Duration::from_millis(10);
    thread::sleep(ten_millis);

    assert_eq!(*counter.lock().unwrap(), 3);
}

In this example we're creating a SignalerAsync that can emit signal and we can subscribe a callback to any signal. The sig.signal_loop(); init the signal loop thread, that wait for signals and call any subscribed callback when a signal comes.

        let _ = sig1.subscribe("signal", Box::new(move |_sig| {
            *c1.lock().unwrap() += 1;
        }));

We subscribe a callback to the signaler. The signaler can be cloned and the list of callbacks will be the same, if you emit a signal in a clone and subscribe in other clone, that signal will trigger the callback.

Then we're emiting some signals:

        sig2.emit(SigType::Update, "signal").unwrap();
        sig2.emit(SigType::Update, "signal:2").unwrap();
        sig2.emit(SigType::Update, "signal:2:3").unwrap();

All of this three signals will trigger the previous callback because the subscription works as a signal starts with. This allow us to subscribe to all new room messages insertion if we follow the previous described keys, subscribing to "msg:roomid" and if we only want to register a callback to be called only when one message is updated we can subscribe to "msg:roomid:msgid" and this callback won't be triggered for other messages.

The callback should be a Box<Fn(signal)> where signal is the following struct:

#[derive(Clone, Debug)]
pub enum SigType {
    Update,
    Delete,
}

#[derive(Clone, Debug)]
pub struct Signal {
    pub type_: SigType,
    pub name: String,
}

Currently only Update and Delete signal types are supported.

Signaler in gtk main loop

All the UI operations in a gtk app should be executed in the gtk main loop so we can't use the SignalerAsync in a gtk app, because this signaler creates one thread for the callbacks so all callbacks should implement the Send trait and if we want to modify, for example, a gtk::Label in a callback, that callback won't implement Send because gtk::Label can't be send between threads safely.

To solve this problem, I've added the SignalerSync. That doesn't launch any threads and where all operations runs in the same thread, even the callback. This is a problem if one of your callbacks locks the thread, because this will lock your interface in a gtk app, so any callback in the sync signaler should be non blocking.

This signaler should be used in a different way, so we should call from time to time to the signal_loop_sync method, that will check for new signals and will trigger any subscribed callback. This signaler doesn't have a signal_loop because we should do the loop in our thread.

This is an example of how to run the signaler loop inside a gtk app:

    let sig = SignalerSync::new();

    let sig1 = sig.clone();
    gtk::timeout_add(50, move || {
        gtk::Continue(sig1.signal_loop_sync())
    });

    // We can subscribe callbacks using the sig here

In this example code we're registering a timeout callback, every 50ms this closure will be called, from the gtk main thread, and the signal_loop_sync will check for signals and call the needed callbacks.

This method returns a bool that's false when the signaler stops. You can stop the signaler calling the stop method.

Point of extension

I've tried to make this crate generic to be able to extend in the future and provide other kind of cache that can be used changing little code in the apps that uses mdl.

This is the main reason to use traits to implement the store, so the first point of extension is to add more cache systems, we're currently two, the LMDB and the BTreeMap, but it would be easy to add more key-value storages, like memcached, unqlie, mongodb, redis, couchdb, etc.

The signaler is really simple, so maybe we can start to think about new signalers that uses Futures and other kind of callbacks registration.

As I said before, mdl does a copy of the data on every write and on every read, so it could be cool to explore the implication of these copies in the performance and try to find methods to reduce this overhead.

GUADEC 2018

danigm — Fri, 13 Jul 2018 00:00:00 +0200

GUADEC 2018

GUADEC is the GNOME Users And Developers European Conference, is an annual conference that take place in Europe, and this year was in Spain, so I should go. I've became a foundation member this year and I've two Google Summer of Code students from GNOME organization working on Fractal, so this year GUADEC was an important one for me.

This year GUADEC was in Almería and I was there for the main days. I can't stay there for the full GUADEC because I'm using my holidays, so I stay in Almería from Thursday 5th to Sunday 8th and then I continue to Murcia to enjoy my free time in the beach.

Not my first GUADEC

Until the end of 2017 I've not been GNOME foundation member, but I've been doing small contributions to GNOME since 2010.

My first GUADEC was the Gran Canaria Desktop Summit, a big event that joined in one city the GUADEC and Akademy so there was a lot of desktop developers, from GNOME, KDE and other open source projects. I was there with some friends, I was working with some GNOME developers, in the guadalinex project, the year before so I started to know the GNOME community.

The next year I was in the Hague GUADEC. This year I was there as a developer, I was working in the Evince accessibility support so my employer paid the travel and all costs.

After that, I don't have more paid work related to GNOME, and started to work with Qt, so I started to go to Akademy, but I don't leave the GNOME community, I was always a GNOME user and was there doing small contributions.

My last GUADEC was the 2012, in A Coruña, Spain again. Indeed I'm in the wikipedia GUADEC article photo, from that year, try to find me there.

Almería

This year GNOME was a great event, with good social events. We've a beach party with a big paella and the famous Ice Cream Death Match.

We've a good time in the Alcazaba of Almería, with a great performance done by the GNOME crew.

Then we've a flamenco show in a great place with drinks and some food.

GNOME

As I said before, this is not my first GUADEC, but this one is an important one for me, because Fractal is becoming important and this year I've been working with a lot of great people and it was good to meet them in the GUADEC.

This year I've been in the Madrid Rust hackfest and in the Strasbourg Fractal hackfest, unfortunately I can't stay there for the working days because there was interesing BoFs about Rust and other technologies.

It was great to meet again with Julian and Tobias, and to meet in person other people that I've been working with, like Carlos Soriano and Alberto Fanjul.

The most important thing from this GUADEC, for me, is the growth plan for GNOME, the new board of directors want to grow the GNOME projects, attracting more money and more developers.

The GNOME project starts to loose interest and developers when the mobile app ecosystem starts to grow, but it seems that now the interest in this great project is increasing again, thanks to a lot of innovative projects like flatpak, gnome-builder, librem5, etc and a lot of great changes like the gitlab migration. The gnome Board of Directors is doing a great work too, increasing the money spend in hackfest, to make it easy for developers to work in the gnome projects.

I've no doubt that GNOME will grow during the next year in developers and importance. I've been around a lot of time, doing a little in my free time, but I'll try to spend more time in GNOME and I'll try to find a way to get payed for it :D

See you in the 2019 GUADEC.

LMDB: Cache database in memory

danigm — Sat, 30 Jun 2018 00:00:00 +0200

Some days ago I was in a meeting talking about the E2E implementation for Fractal, to know the current implementation state and to talk with other developers. This meeting was promoted by Puri.sm people, because they want to use Fractal for the Librem5 phone and they want to have E2E. There's people working in the E2E and I think we can have this on Fractal at the end of the year, but this is not what I want to talk about today.

In this meeting there was Fractal developers but we've also other people, like the nheko developer, mujx (nheko is a Qt matrix client). During the meeting, mujx ask us about the cache storage that we're using, because for the E2E is too important to don't lose any key, because that'll be catastrophic, you won't be able to read room messages. So it's important to have a transactional database storage for this information. They are using LMDB, I didn't know nothing about LMDB so after this meeting I start to about it.

LMDB

Lightning Memory-Mapped Database Manager (LMDB) is a key-value database, it's memory mapped so it's fast, and also uses filesystem storage so we've persistence. This database has transactions so it's safe to read/write from different threads or process.

In Fractal we're using a simple json file for cache, but this doesn't support transactions and if the app crash or if something bad happens, we can lose data. This method is simple, but is slow and insecure, so using LMDB will improve Fractal in several ways.

But LMDB is in memory and in Fractal we've a lot of interface code sharing the app state, so we're passing the state between threads with copies and complex data sharing. This can simplify the interface code because using LMDB for the application global state will make this state accesible from different threads.

Testing LMDB

Fractal is written in Rust so I want to write some tests before start to use this on Fractal. There's a simple LMDB rust crate, and I've been writting an example lib to test it.

Basically what I've done is to write a simple trait that you can implement for simple Rust structs so that struct can be stored and recover from the cache with a simple method:

#[derive(Serialize, Deserialize, Debug)]
struct B {
    pub id: u32,
    pub complex: Vec<String>,
}
impl Cacheable for B {
    fn db() -> &'static str { "TESTDB" }
    fn key(&self) -> String {
        format!("b:{}", self.id)
    }
}

The struct should be serializable/deserializable with serde because in this example I'm using bincode to convert structs to [u8].

The Cacheable trait only have two required methods, the db and the key. The db is the db name to use to store this struct instances and the key is the key to use when storing a concrete instance.

With this, we can store in the cache and query from the cache, using the key:

let db = &format!("{}-basic", DB);
let mut cache = Cache::new(db).unwrap();

let b = B{ id: 1, complex: vec![] };
let r = b.store(&mut cache);
assert!(r.is_ok());

let b1: B = B::get(&mut cache, "b:1").unwrap();
assert_eq!(b1.id, b.id);
assert_eq!(b1.complex.len(), 0);
b1.complex.push("One string".to_string());
b1.complex.push("Second string".to_string());
b1.store(&mut cache);

let b2: B = B::get(&mut cache, "b:1").unwrap();
assert_eq!(b2.id, b.id);
assert_eq!(b2.complex.len(), 2);
assert_eq!(&b2.complex[0][..], "One string");

This is thread safe, so we can read from the cache from different threads and we'll always get the last version in the database.

LMDB is a key-value database, so we don't have relations. This is not a real problem for us because we can model the relations in the keys, for example, we can store Room messages with keys like this "message:ROOMID:messageID" and then we can iterate over all objects with this prefix: "message:ROOMID" that will give us all room messages. Something like this should work:

let prefix = format!("message:{}", room.id);
Message::iter(&mut cache, &prefix, |m| {
    // m is a Message struct fetched from the database, we can do
    // what we want here
    // ...
    Continue(true)
});

LMDB in Fractal data model

We're thinking about moving the Fractal data model from the AppOp struct to a new crate, independent of the UI, to simplify the UI code and to be able to use the same data model from different UIs (we wan't to split fractal in two different apps), Julian wrote about this.

I think that we can use the LMDB cache to store this new app state that we want to have and this will simplify a lot our code, because we will share the same state and this state will be persisted in filesystem.

I'll start to write a new crate for Fractal to start to move all the app state to this new crate. I think I can write a generic tool to simplify the LMDB use, maybe I'll publish another crate in crates.io and use that in Fractal, but I need to think a little more about the pattern to follow.

I write a lot of web code in my day to day work, and I've been working with react+redux. This LMDB cache thing in Fractal reminds me a lot to the redux store and I want to follow a similar pattern so we can have only an app state and only a way to update this state.

Stickers in Riot

danigm — Sat, 19 May 2018 00:00:00 +0200

Yesterday I read a blog post about the new Riot.im Stickers. This is not a matrix.org feature, it's implemented as a widget, when you send an sticker you're sending a new event with the type "m.sticker", which is similar to the "m.image" event.

The matrix.org protocol is flexible so this is a good example of how to add new features to the clients that uses matrix without the need to change the protocol.

This is not a core feature because you can send images, but I think this is great and add a simple way to show reactions for the users, so as I was reading I thought that we can add this to Fractal, so I started to read how we can add support for this.

Reading the doc

The first thing to implement a feature is to read the specifications or the technical documentation so we can know what is needed... But there's no documentation yet about Stickers or widgets yet.

This is a problem because we can't implement a feature if we don't know what we should do. But free software give us a great opportunity when this happens, we've the Riot source code so we can look at the code and learn what they are doing.

Reading the code

Riot web is a javascript application that uses AJAX to communicate with different server APIs so the first thing that I did to start to understand the stickers thing was to open the firefox debugger and view how riot is communicating with the server.

From here I've learned that for stickers riot is asking to the scalar.vector.im server. But I don't understand the whole thing with the requests because riot does a lot of request to different APIs and I can't isolate the stickers thing easily.

To fill my understanding gap I go to the matrix-js-sdk and matrix-react-sdk and I did a quick grep to the source code looking for the API calls that I've view in firefox. With this I can understand the full stickers process.

Writing an example

To say that I know how this is working, it's not enough the code reading. To make sure that I've understood the whole process I need to write a simple program that does all the process and then I can say that I understand this.

So I started to write a simple python script using requests. This simple script does the request to the server and list all the stickers json so I can say that I'm able to communicate with the API.

Stickers in Fractal

After this small research I'm able to implement an initial sticker support for Fractal. I'll try to add a simple way to show and use stickers and a way to render stickers in the messages history.

If there's no secret problems we'll have a basic stickers support in Fractal soon.

Fractal Hackfest, Strasbourg (day 2)

danigm — Sat, 12 May 2018 00:00:00 +0200

The second day of Fractal hackfest was really productive, we talk about a lot of topics and takes some important decisions.

E2E Encryption

The encryption is a needed feature but encryption is hard to do in rooms. Matrix uses public-key cryptography, for rooms they are using Megolm, that's a protocol to exchange encrypted messages with more than one and share that message keys in a one-to-one secure communication.

I don't know a lot about this E2E because for me it's more important to have the client working with a basic functionality before the encryption. So you should read the official doc because maybe this that I'm writing here is completely wrong.

To do all this E2E key sharing, client side encryption and communication, Riot has three different implementations of the same lib, so they have this code in the JavaScript SDK, the same ported to iOS version in ObjectiveC and the same ported to Android in Java. Below this lib there's the libolm that does the real encryption.

For the future it should be a good solution to have only one lib that does the encryption thing, so we need to push in that direction and maybe matrix.org and Purism can pay for that work to be done.

From Fractal, we've two options or wait until we've the official E2E C/C++-lib or something, or try to port the iOS lib to Rust or C to use in our codebase. I think that the best solution is the first one, in any case, we'll need to wait some time to view E2E in Fractal.

Calls

Calls are easy compared to the Encryption thing, the protocol is well documented and it's really simple. The hard part here is that the call is implemented using the webRTC and that is a web protocol... But it seems that there's something done in this case with GStreamer working with webRTC.

Multiple accounts

We was talking yesterday about the multiple accounts problem, if we want to support and in that case what solution we should take.

We've two possible solutions, one is to support multiple accounts at the same time and a way to change between accounts, and the other one is to open a window for each account.

At the end we think that the best solution could be to open a separated window for each account so if you've multiple accounts, at first you should choose what account you want to open and then from the menu you can open a new window and go to the account selection menu again.

We need to work in the design of this feature and it's not confirmed but I think that this was the best solution, because it's simple and powerful.

Communities

Communities are something really new in the Matrix.org protocol and it's not well documented yet and there's some functionality that is not implemented so we decide to wait until this functionality is more mature to view how we can integrate this in Fractal.

Communities today are a group of rooms and a group of people and a full html description to show in the community page. There's other features like show a label for each member of the community or group, so I think these features are cool. We'll need to support some parts of the communities protocol soon.

Google Summer of Code

As I said this year we've two students working on Fractal thanks to Google, as part of the GNOME organization, and both students comes to the Hackfest so thank you Julian and Eisha for coming and for the work done before the GSoC and for sure for the work you'll be doing during the summer.

We're talking about how we'll be organizing, I'm the mentor of both but we'll have the help from Tobias Bernard and from Alexandre Franke so we'll have a summer hacking group to improve Fractal a lot.

We'll have a weekly meeting to talk about what we're doing and what to do in the next week.

For this first week we'll start with two needed features. Julian will be working in the user preference dialog and Eisha will start to work in the internationalization (i18n) because it's not easy to use gettext in rust and we want to translate the app to make it more accessible for everyone in the world.

Hacking a bit

In the meantime, between functionality discussion I was working in a way to use Fractal without SecretService, so if you're using Fractal outside GNOME or KDE you can configure it to store password and token in a plain text. We shouldn't store passwords in a clear text file, but I think that this will simplify the day for many people that wants to use Fractal but don't want to write the user and password every time.

This is not finished yet, but we'll have this very soon, and with this it's possible to implement other password & token storage services and make it configurable, so maybe in the future someone implements the storage for MacOS or something.

Holidays

This was my last day in the hackfest, I've the travel back to Spain the Sunday but I'll do some tourism here so I need to leave the people, they will continue working the Saturday and the Sunday.

This have been the first Fractal hackfest and it was a great experience, to meet other people working on it and to talk about the future of the application. It's really cool that with less than a year project we've now a big community, so thank you all form coming and see you on GUADEC.

There's a lot of work to do, but we're lucky because we're two students working in Fractal this summer and the people from Matrix.org and Purism are working with us, so thank you all.

I want to thank again the GNOME foundation and GNOME people for make this possible and to my company Wadobo, for let me spend some time working on this great project.

Fractal Hackfest, Strasbourg (day 1)

danigm — Fri, 11 May 2018 00:00:00 +0200

Yesterday was the first day in the first Fractal Hackfest. I'll try to write an small blog post every day to share the development with the world.

My travel to Strasbourg was not an easy travel because I've to take two flights to get here from Málaga so a long day travelling.

I met with Mathew from Matrix.org at the London airport because we took the same flight to here and it was really cool to meet him in person and we talk a little about the current Matrix situation.

I've met the other Fractal people and collaborators at the event, and it's great that people from Purism, Matrix, Gnome and the two GSoC students come here to work together in this great application.

Barbecue and Banquet

We've spend almost all the day talking about the different uses cases for the messaging problem and Tobias has come with a proposal for split the current application in two because the use case is totally different.

We've the first case when you use the app to talk to family, friends and small groups of people where all conversations are private and you want to read almost every message there, we call that situation the "Barbecue".

The second case is the one where you use the messaging app to talk in a public or private room with a lot of people, where you are receiving a lot of messages but you don't want to follow the full conversation you'll talk with someone or read a topic conversation, we call this one the "Banquet".

Currently Matrix clients are more oriented to the Banquet use case so there's a lot of people and communities using it for this. The idea behind this split is to provide two apps to the user to make both cases possible.

We need to work on this idea but it seems that is the correct way to go.

Initial Syn speed up

At the afternoon. We've reviewing the current initial sync filter and detect some points to speedup.

I changed the filter and the initial sync goes a lot faster. I've removed the member events from the initial sync so there's a lot of less information to retrieve from the server.

This little change breaks the one to one room name and default avatars because that name is calculated using the members in the room so I need to fix that making the name calculation a backend process. So now if a room has no name, We'll ask for the room member list and when we've that we'll calculate the room name with the same algorithm, if there're only two people, we'll use the other person username, if there's three people we'll use "one and another" and if there're more we'll use "one and others".

Gnome dinner

We've a good dinner sponsored by the gnome foundation. Thank you so much.

GNOME 💙 Rust Hackfest in Madrid

danigm — Thu, 26 Apr 2018 00:00:00 +0200

The last week was the GNOME 💙 Rust hackfest in Madrid. I was there, only for the first two days, but was a great experience to meet the people working with Rust in GNOME a great community with a lot of talented people.

The event was in the OpenShine office, that was a great place and the OpenShine people was very friendly too so thank you very much to OpenShine for support this kind of events.

What I did during the Hackfest

I arrive at 10:30, more or less, and there are some people there, but two or three comes after the lunch, so the first morning we started to work, each one in their project.

I've some patches for Fractal to review and I was working on the glade file splitting. After that I downloaded the gnome-class code and started to read and try to understand the code because I want to help in the gnome-class development during the Hackfest.

After the lunch, the missing people from the morning was there so we did an official "Hackfest presentation" everyone did a little presentation about his work and what want to do during the event.

Gnome-class was the main project to join GNOME and Rust, and it's still in a early alpha stage so Federico give us a little talk about what's gnome-class and its parts.

Gnome-class

Gnome-class is basically a compiler that translate custom syntax to Gobject binary compatible Rust code. Using the proc-macro, gnome-class parses a Gobject like declaration and creates all the Rust code needed to use the Glib so we can have binary code that can be called from C or from Rust. With this we can do Object Oriented programming in Rust using the Gobject library and types.

So we've three parts in gnome-class, the parser, the High-level Internal Representation (HIR) and the code generation.

gobject_gen! {
    class Counter {
      f: Cell<u32>,
    }

    impl Counter {
        pub fn add(&self, x: u32) -> u32 {
            self.get_priv().f.set(self.get() + x);
            self.get()
        }

        pub fn get(&self) -> u32 {
            self.get_priv().f.get()
        }
    }
}

#[test]
fn test() {
    let c: Counter = Counter::new();

    println!("Counter has value: {}", c.get());

    c.add(2);
    c.add(20);
    assert_eq!(c.get(), 22);

    println!("Counter has value: {}", c.get());
}

This is Rust code with gnome-class, here we're creating a new Class called Counter with two public methods, then we've a test that use this. Behind this code, gnome-class generates a lot of ugly code with pointers and so to make all of this C-Compatible and generate all the needed binary using the GObject data scheme.

My work in gnome-class

Properties declaration was not supported in gnome-class so took that task. There was a proposed syntax and I started to work in the parser to try to support that syntax and convert all the relevant information into a Rust struct in the HIR tree.

To parse that I needed to learn syn that's a parser based in nom. That was not easy, but there's a lot of code in gnome-class so I can read and learn from that and at the end of the day I had a working properties parser.

The second day I continue with my work and started to generate code. Here I have more problems because the code generated is too low level and I didn't now much about the GObject internals so I was playing around types conversions.

I've to go just after the lunch, but during the back to home train trip I was able to create a Merge Request with all the work I've done.

I've work to do during the week so I can't continue working on properties support in gnome-class. But this week I've more time and I've been working to complete the code generation, so now we've this test working:

#![feature(proc_macro)]

extern crate gobject_gen;

#[macro_use]
extern crate glib;
use gobject_gen::gobject_gen;

use std::cell::Cell;

gobject_gen! {
    class ClassWithProps {
        p: Cell<u32>,
        p2: Cell<u32>,
    }

    impl ClassWithProps {
        pub fn get(&self) -> u32 {
            self.get_priv().p.get() +
            self.get_priv().p2.get()
        }

        property MyProp: T where T: u32 {
            get(&self) -> T {
                let private = self.get_priv();
                return private.p.get();
            }

            set(&self, value: T) {
                let mut private = self.get_priv();
                private.p.set(value);
            }
        }

        property Prop2: T where T: u32 {
            get(&self) -> T {
                let private = self.get_priv();
                return private.p2.get();
            }

            set(&self, value: T) {
                let mut private = self.get_priv();
                private.p2.set(value);
            }
        }
    }
}

#[test]
fn test_props() {
    let obj: ClassWithProps = ClassWithProps::new();
    assert_eq!(obj.get(), 0);

    assert_eq!(obj.get_property_prop2(), 0);
    obj.set_property_prop2(42);
    assert_eq!(obj.get(), 42);
    assert_eq!(obj.get_property_prop2(), 42);

    obj.set_property_myprop(58);
    assert_eq!(obj.get_property_myprop(), 58);
}

My first GNOME hackfest

This was my first GNOME hackfest. I've been in different GUADECs and other events with GNOME devs, but this was the first time that I travel to work on a specific project and not just to meet the people and view talks from the distance.

It was a great experience. Rust is really new in the desktop development, but there's a lot of people working so it's great.

The next milestone is the Fractal Hackfest, we'll be in Strasbourg four days working.

I want to thank all the great GNOME community that makes collaboration so easy, it's a great community.

Currently all of my work on this is voluntary so I need to thank my coworkers at Wadobo, because I'm spending some work time in this. I can do this because my company is great.