Rafael Kassner

Delta Chat: encrypted chat with boring technology

2025-05-08T00:00:00Z

Hi,

A couple of days ago I’ve heard about Delta Chat, and for once I got thrilled with some piece of technology again. At the surface it’s just a chat app to the likes of Telegram, Signal and WhatsApp. The exciting part, however, is that is uses plenty of open standards technology under the hood. It’s a chat app built on top the regular e-mail protocols, SMTP and IMAP, with a sprinkle of PGP to make it end-to-end encrypted.

It has the same secure initial key exchange issue that PGP e-mail has, but at least now we have an interface that is straight up easy to use, specially for people outside of the infosec world. It support rich media (images, videos, voice messages) and even groups (untested, but I assume it won’t allow newjoiners to read old messages).

What makes me so excited about this is that each message is sent as a plain e-mail message. One MTA sending an e-mail to another MTA. Simple, old, functional. I know e-mail, I have my own server, I can just go to my maildir and read everything. Can I?

Reading Delta Chat messages from the mail server

I’ve used the iOS app for my initial tests, and you can absolutely export your entire chat history decrypted in the app, but I want to prove this point to myself. Let’s try with a message found in my maildir:

Return-Path: <client2@example.com>
Delivered-To: client1@example.com
Content-Type: multipart/encrypted; protocol="application/pgp-encrypted";
boundary="183d0f93c6372f58_1fb940b49edbf5c5_64e1f6b034240ef1"
From: <client2@example.com>
To: <client1@example.com>
Subject: [...]
Date: Tue, 6 May 2025 21:07:17 +0000
Message-ID: <0b5a2c48-472d-45b9-8096-219313f5dea1@localhost>
In-Reply-To: <3325e202-0957-4fd3-ae69-1a8fa94c16ba@localhost>
References: <51784fab-1af1-4ad3-a0b6-a868bd2ad685@localhost>
<51784fab-1af1-4ad3-a0b6-a868bd2ad685@localhost>
<3325e202-0957-4fd3-ae69-1a8fa94c16ba@localhost>
Chat-Version: 1.0
Autocrypt: addr=client2@example.com; prefer-encrypt=mutual; keydata=[...]
--183d0f93c6372f58_1fb940b49edbf5c5_64e1f6b034240ef1
Content-Type: application/octet-stream; name="encrypted.asc";
charset="utf-8"
Content-Description: OpenPGP encrypted message
Content-Disposition: inline; filename="encrypted.asc";
Content-Transfer-Encoding: 7bit
-----BEGIN PGP MESSAGE-----
hF4D6OmtN[...]s9SkewYsB
-----END PGP MESSAGE-----
--183d0f93c6372f58_1fb940b49edbf5c5_64e1f6b034240ef1--

I recognize these PGP MESSAGE headers. Where is the private key? In the backup I’ve exported before. It can be extracted with this command:

sqlite3 dc_database_backup.sqlite 'select hex(private_key) from keypairs limit 1' | xxd -r -p > deltachat.key

Decrypting the PGP message with the exported key, sure enough, reveals our content:

Content-Type: text/plain; charset="utf-8"; protected-headers="v1"
From: "Test 2" <client2@example.com>
To: "Test 1" <client1@example.com>
Subject: Re: Message from Test 1
Date: Tue, 6 May 2025 21:07:17 +0000
In-Reply-To: <3325e202-0957-4fd3-ae69-1a8fa94c16ba@localhost>
References: <51784fab-1af1-4ad3-a0b6-a868bd2ad685@localhost>
<51784fab-1af1-4ad3-a0b6-a868bd2ad685@localhost>
<3325e202-0957-4fd3-ae69-1a8fa94c16ba@localhost>
Chat-Version: 1.0
Chat-Disposition-Notification-To: client2@example.com
Chat-Verified: 1
Message-ID: <0b5a2c48-472d-45b9-8096-219313f5dea1@localhost>
Content-Transfer-Encoding: 7bit
Hello world!

Outro

I’ve tested Delta Chat with my own mail server, which uses Postfix and has everything configured for public e-mail, like DKIM signing, spamd, IP blocklist checks and so on, and each message took about 2 seconds from one device to another. Using a public server it sure feels below 300ms, so there is room for improvement when self-hosting a dedicated chatserver.

I really love this, and I apologize in advance to my friends whom will be pestered to move to Delta Chat.

Thank you!

Resizing a LUKS-backed BTRFS RAID1 filesystem

2025-04-12T00:00:00Z

Hi.

After a series of hardware upgrades, I finally have 2x 2TB SDD for data on my NAS! However, they are still configured with ~900GiB data partitions, and today it is time to expand it!

Initial state

The current disk layout is:

# fdisk -l /dev/sda
Disk /dev/sda: 1.82 TiB, 2000398934016 bytes, 3907029168 sectors
Disk model: PNY CS900 2TB SS
Sector size (logical/physical): 512 bytes / 512 bytes
Device Boot Start End Sectors Size Id Type
/dev/sda1 * 2048 499711 497664 243M 83 Linux
/dev/sda2 499712 43468799 42969088 20.5G 83 Linux
/dev/sda3 43468800 1918322687 1874853888 894G 83 Linux
# fdisk -l /dev/sdb
Disk /dev/sdb: 1.82 TiB, 2000398934016 bytes, 3907029168 sectors
Disk model: Samsung SSD 870
Sector size (logical/physical): 512 bytes / 512 bytes
Device Start End Sectors Size Type
/dev/sdb1 2048 1001471 999424 488M Linux filesystem
/dev/sdb2 1001472 98656255 97654784 46.6G Linux filesystem
/dev/sdb3 98656256 1973510143 1874853888 894G Linux filesystem

Reading the output above, the data partitions can be found at /dev/sd*3, both sized at 894GiB with the exact same sector size (512 bytes) and sector count (1874853888).

Both partitions are LUKS encrypted, and I’m using crypt keys to avoid typing passwords all the time. They are declared in crypttab to be auto-mounted on boot:

# cat /etc/crypttab | grep data
data1 /dev/sda3 /etc/cryptkey-cs900 luks,discard,timeout=10
data3 /dev/sdb3 /etc/cryptkey-evo870 luks,discard,timeout=10

And finally, both encrypted partitions are used to create a single BTRFS filesystem in RAID1 mode:

# btrfs filesystem show /data
Label: none uuid: 079c6185-4a18-4f8f-8a62-bb741aabb758
Total devices 2 FS bytes used 661GiB
devid 1 size 894GiB used 661GiB path /dev/mapper/data1
devid 2 size 894GiB used 661GiB path /dev/mapper/data3

The challenge

The disks have between 920GiB and 946GiB of free space that I want to reclaim. For consistency, I’ll resize them both to reclaim 920GiB, keeping the number of sectors identical.

Note: BTRFS’ RAID1 does not need identical sizes for each partition, it will keep allocating blocks into its devices as needed, and you could use uneven disk sizes (i.e.: 1x2TB + 2x2TB) in RAID1 mode. As long as it can write each block to two devices, it doesn’t matter which ones they are. Read more.

So I am dealing with three layers: disk partitions, LUKS and BTRFS. I’ll have to resize them all, starting from the outer one.

Step 0: backup

Make sure you have backups, and you’re fairly confident you can restore your data from them.

Step 1: resize the partitions

I didn’t spend much time investigating if I would be able to do an online resize, I went ahead and booted in the rescue mode I had in GRUB.

I’ve used cfdisk, but any partition manager should be able to do the job. The only important thing (for me), was to keep the same number of sectors for both partitions. I first resized the disk with less free space and rebooted into the system to make sure it was still working. Had I screwed it up, I could recover by dd-ing the second disk back into the first. Thankfully it wasn’t needed.

Once both partitions were resized, I’ve booted back into the regular mode.

Step 2: resize the LUKS container

This step was fairly straight-forward, I ran cryptsetup resize once for each LUKS container. By default, it will expand to use the entire partition.

# cryptsetup --key-file=/etc/cryptkey-cs900 resize data1
# cryptsetup --key-file=/etc/cryptkey-evo870 resize data3

Step 3: resize the BTRFS devices

Also fairly simple, I ran btrfs filesystem resize. The caveat is that you have to run it for each device in your filesystem (devid column in the btrfs filesystem show output).

# btrfs filesystem resize 1:max /data
# btrfs filesystem resize 2:max /data

Conclusion

My system can now store 1.77TiB of data, and I’ve already put it in use.

# btrfs filesystem show /data
Label: none uuid: 079c6185-4a18-4f8f-8a62-bb741aabb758
Total devices 2 FS bytes used 1.16TiB
devid 1 size 1.77TiB used 1.17TiB path /dev/mapper/data1
devid 2 size 1.77TiB used 1.17TiB path /dev/mapper/data3

Thank you.

Home A-CI-stant

2025-03-23T00:00:00Z

Hi!

For years I lurked in the Home Assistant channel of a community Discord, but never had much interest in it. Earlier this month I had some discount coupons in hand and decided that it was time to give it a go. I got Home Assistant, the ZBT-1 and one IKEA Inspelning and managed to make it control my balcony lights. In my opinion, the sun “device” in Home Assistant might be the most underrated out-of-the-box integration!

In any case, I live in an apartment, so after that I was mostly done. I don’t have much else to automate, and I do really value physical buttons, so it all sat there for a while. This weekend, while doing some maintenance on my NAS, it got me thinking: what if I use Home Assistant to manage a Continuous Integration server?

I run my own Forgejo instance, and I’ve been meaning to use the Actions feature for a while, but never got around it in my NAS because I don’t want to deal with VMs nor Docker-in-Docker. I also have a fairly decent Optiplex laying around, and while I already had it running a Forgejo Actions runner before, I simply have way to little usage of it to justify have it running 24x7.

So now it finally clicked. The Optiplex has a feature to ATX-power-on the motherboard once it is energized, and combining that with the smart plug, I can power it on just when there is demand. With that, I made a daemon that will wait for Forgejo’s webhooks, and turn on the CI if needed:

m.HandleFunc("POST /webhook", func(w http.ResponseWriter, r *http.Request) {
 // ...

 if payload.Repository.FullName != "canastra.online/monorepo" {
 w.WriteHeader(http.StatusNoContent)
 return
 }

 version, ok := strings.CutPrefix(payload.Ref, "refs/tags/")
 if !ok {
 w.WriteHeader(http.StatusNoContent)
 return
 }

 isOn, err := hassIsOn(hassEntityId)
 // ..

 if isOn {
 w.WriteHeader(http.StatusNoContent)
 return
 }

 err = hassTurnOn(hassEntityId)
 // ...

 log.Printf("Waking up CI server for %s:%s\n", payload.Repository.FullName, version)
 w.WriteHeader(http.StatusNoContent)
})

However, that only addresses half of the issue. Once the server is on, it must be powered off in case it’s not needed anymore. A systemd timer can take care of this, powering off the server 30 minutes after boot. We can abuse the runner’s shutdown_timeout to make systemd wait for the running jobs to complete.

# forgejo-runner.service
[Service]
TimeoutStopSec=30m # same as your runner's shutdown_timeout
# hass-ci.service
[Service]
ExecStart=systemctl stop forgejo-runner.service && systemctl poweroff
TimeoutStopSec=30m
# hass-ci.timer
[Timer]
OnBootSec=30m

Check out the complete code, MIT licensed.

The last piece of the puzzle is a Home Assistant automation that will turn off the plug once the server is completely off.

And that’s about it. Granted, it’s totally unnecessary, I could’ve just used Wake-Up on LAN, but it was the perfect project for me to dig into Home Assistant and tell myself I need just one more smart plug.

Thank you!

Internet via IPv6

2024-10-12T00:00:00Z

Hi!

I know I’m late to the party, but with the IPv4 exhaustion now being expressed in monetary costs by VPS/Cloud providers, it is time to seek IPv6. My main use-case is to be able to reach servers exclusively over IPv6, so I don’t need to pay for IPv4 assignments if those servers will not serve direct public traffic (i.e.: database, storage and job servers).

My home provider did have IPv6 support, but they pulled the plug in 2020, and now 3 years later, I still don’t have IPv6 access. Let’s fix that today.

I had a brief experience experience with SixXS back in 2013, and I’ve learned the importance of latency for such tunnels (150ms+ is NOT ok), so my only option is tunnelbroker.net, operated by HE.NET, which has a PoP 10ms away from me.

Configuration

I run an EdgeRouter X as my gateway. The configuration is fairly straight forward:

root@edgerouter# show interfaces tunnel tun0
address <Client IPv6 Address>/64
description "HE.NET Tunnel"
encapsulation sit
firewall {
in {
ipv6-name WANv6_IN
}
local {
ipv6-name WANv6_LOCAL
}
}
local-ip <Client IPv4 Address>
multicast disable
remote-ip <Server IPv4 Address>
ttl 255
root@edgerouter# show interfaces switch switch0
address 192.168.1.1/24
address <Routed /64 prefix>::1/64
[...]
root@edgerouter# show protocols static
interface-route6 ::/0 {
next-hop-interface tun0 {
}
}

And the firewall configuration:

root@edgerouter# show firewall ipv6-name
ipv6-name WANv6_IN {
default-action drop
description "WAN to LAN"
enable-default-log
rule 10 {
action accept
description "Allow established/related"
state {
established enable
related enable
}
}
rule 20 {
action drop
description "Drop invalid state"
state {
invalid enable
}
}
rule 30 {
action accept
description "Allow IPv6 ICMP"
protocol ipv6-icmp
}
}
ipv6-name WANv6_LOCAL {
default-action drop
description "WAN to Router"
enable-default-log
rule 10 {
action accept
description "Allow established/related"
state {
established enable
related enable
}
}
rule 20 {
action drop
description "Drop invalid state"
state {
invalid enable
}
}
rule 30 {
action accept
description "Allow IPv6 ICMP"
protocol ipv6-icmp
}
}

The values for the items between brackets are to be obtained from the tunnel details page on tunnelbroker.net.

Notably missing from the configuration is DHCPv6, Prefix Delegation and Route Advertisement. I decided to not allow for auto-configuration in my network, but rather manually configure the machines I want with IPv6. Here’s why:

Caveats

The HE.NET tunnel works. I’m able to reach servers via SSH, HTTP and other protocols just fine. Speed isn’t great, but given it’s a free service, I can’t complain.

A tangential observation: fast.com and speedtest.net both couldn’t test the speed correctly. fast.com used GeoIP to infer my location (sending all the requests from Stockholm to Los Angeles, because HE.NET GeoIP locations). ipv6.speedtest.net sent requests to dual-stack hostnames, so some of the packets did not go through the tunnel.

However, free things are prone to abuse, and the HE.NET tunnel is no different, having fell into despair on the wide internet. This can be translated as CAPTCHAs everywhere and Google simply refusing to serve any request coming from their IP range:

I’ve looked into getting a IPv6 PI Assignment from RIPE NCC, but HE.NET charges a $500 setup fee for new BGP sessions, not to mention the LIR sponsorship costs, so it quickly becomes too expensive compared to renting IPv4s from VPS/Cloud providers.

Lastly, HE.NET only supports SIT/GRE tunnels, which don’t work behind CGNAT. If you are behind CGNAT, look for WireGuard tunnels as an alternative.

Outro

I’m happy with the HE.NET service. It fixes my problem and the limitations are manageable for now. If you are in a similar situation and is looking for a IPv6 tunnel, tunnelbroker.services has a small list of IPv6 tunnel services. Alternatively, if you just need IPv6 to test an HTTP endpoint, a SOCKS proxy with ssh -D 1337 your-ipv6-enabled-vps is often enough.

Thank you.

Auto-deployments with systemd

2023-11-05T00:00:00Z

Hi!

I like Kubernetes, and having written my own operators in the past, you quickly grow fond of the high level of automation that it allows you to achieve. However, now that I’m spending the days building my own projects, my requirements (and budget) can’t fit k8s anymore. A couple of production VPSs and my GitHub Actions free quota is all I’m allowing myself to afford on vanity projects, so let’s work with what we have.

GitHub Actions already builds the Canastra binary, so I’m just missing is a way to auto-deploy it to the production VPS. This is composed of two steps:

Copying the binary to the VPS

This can be as simple as creating a new Linux user, generating a SSH keypair and using scp/rsync. I’m going the extra mile to have an user that can’t do anything else, which means chroot. My /etc/ssh/sshd_config now has:

Match Group sftpusers
AuthorizedKeysFile /etc/ssh/sftp-authorized-keys/%u
ChrootDirectory /data/sftp/%u
ForceCommand internal-sftp

It worked when scping from my machine, but when inside the GitHub Action, I got a This service allows sftp connections only error. Maybe something related to TTY? The sftp command, however, worked, so I used this:

- name: push to production
if: github.ref == 'refs/heads/main'
run: |
mkdir -p ~/.ssh
echo "$SSH_KNOWN_HOSTS" >> ~/.ssh/known_hosts
eval "$(ssh-agent)"
ssh-add - <<< "$SSH_AUTH_KEY"
cat <<EOF | sftp user@server
cd bin/
put canastra-server
put canastra-server.sha256
bye
EOF
env:
SSH_AUTH_KEY: ${{ secrets.PROD_UPLOAD_SSH_KEY }}
SSH_AUTH_SOCK: /tmp/ssh_agent.sock
SSH_KNOWN_HOSTS: ${{ secrets.PROD_SSH_KNOWN_HOSTS }}

Restarting the service

Once the files were uploaded, all I needed was to run a single shell script in the server to restart the server. A common way to achieve this is using SSH’s ForceCommand feature, which unfortunately doesn’t work if you are already limiting the user to SFTP-only. You could have a second user that would be able to only run the script, but that’s way too much user management for my taste.

Thankfully, systemd offers a neat solution: systemd.path. My canastra-deploy.path unit looks like this:

[Unit]
Description=Monitor /data/sftp/canastra/bin for new uploads
[Path]
PathChanged=/data/sftp/canastra/bin/canastra-server.sha256
Unit=canastra-deploy.service
[Install]
WantedBy=multi-user.target

And the service unit:

[Unit]
Description=Canastra Deployer
[Service]
ExecStart=/usr/local/bin/canastra-deploy.sh

With this approach, systemd monitors for changes in a particular file, and triggers the configured Unit= once the target is modified. It’s important to use PathChanged= instead of PathModified= to avoid triggering the unit too early. From the manual:

PathChanged= may be used to watch a file or directory and activate the configured unit whenever it changes. It is not activated on every write to the watched file but it is activated if the file which was open for writing gets closed. PathModified= is similar, but additionally it is activated also on simple writes to the watched file

That’s all! Once I push a commit to the main branch, GitHub Actions will build it, and copy the new binary to the server. systemd, once sees a change in the binary file, triggers the deploy script that eventually restarts the service, loading the new version.

Thank you.

Launching: canastra.online

2023-10-15T00:00:00Z

Hi!

Today I’m launching canastra.online! It’s a website I’ve built where you can play Canastra for free, without ads, both single-player and multi-player.

Canastra is a card game, quite popular in Southern Brazil. There are several variations of the game, but for now canastra.online only supports one set. Read the rules on our website.

Building

I’ve built the game in 2018 to play with my significant other, as we couldn’t find anything that was both multiplatform and that supported the subset of rules we wanted to play. Since then, our families and us play it on a weekly basis.

This year, after becoming unemployed, I put my efforts into polishing and finishing the game for public release. Right now I’m bearing all the costs alone, but if you want to contribute, check our donations page.

Future

There aren’t any plans for the future of the game just now. The bot needs more work, as just taking random actions isn’t smart. I’ll also likely introduce new game modes, or the ability to customize the rules for each game.

However, one things is clear: the game will always be free and without ads. If there is demand I might implement a few paid resources, but the game as it is today will always be free without ads.

Thank you!

Project: Canastra on the go

2023-10-06T00:00:00Z

Hi!

I’ve spent a considerable amount of time this year rewriting my Canastra project in Go. While I wanted to write a specific post about this conversion, it has been over 6 months, so it’s unlikely it will still happen. For the purposes of this post though, all it matters is that the project is now built in Go, but the frontend and database architectures are still the same as described in the project page, with Go now handling all the HTTP and Websocket communication.

So a few weeks ago I got a newsletter from my local Raspberry Pi reseller stating the Pi Zero 2 was back in stock and something clicked for me: what if I run Canastra on the Pi, so wife and I can play it on the airplane when we’re travelling? The project is already in Go, so cross-compiling for ARM should be trivial. So I bought the Pi Zero, together with the official case, for a bit less than 300 SEK.

Challenges

Booting

In my first attempt I dd-ed the Raspberry Pi OS Lite to a SD card and booted in the Pi, just to realize that it uses mini-HDMI (as opposed o micro-HDMI from the Raspberry Pi 4B) and I did’t want spend any more money in this project. The latest versions of Pi OS will prompt you to create a user during the first boot, so just touch ssh in the boot partition isn’t enough, as there isn’t a user to login yet. Thankfully the Pi Imager is able to produce a SSH-able image, although it is quite annoying that you have to use a special software instead of just dd if=pi-os.img && touch /mnt/boot/ssh.

Wi-Fi AP

There are several great tutorials for how to turn a Raspberry Pi into a Wireless access point, and the steps are roughly 1) run hostapd; 2) run a DHCP server; and 3) IPTables for routing. While there is nothing out of the ordinary here, the fact that I only had SSH access to the Pi Zero meant I couldn’t get hostapd running because I was relying on the Wi-Fi connection to SSH into it, and bringing hostapd up meant this connection would drop. I ended up plugging the SD card into a Pi 4B, chroot into the SD card, run all the commands, unmount, plug it back into the Pi Zero. Some screw-ups meant that the Pi Zero would boot but not connect to my home network neither create the AP, so I had to do this steps maybe 3 or 4 times until I got it right.

The DHCP server I ran with dnsmasq, mostly because I also wanted to run a DNS server to be able to configure a Captive Portal. Given the only use of the Pi Zero will be for the game, I just redirected everything to the Go application using address=/#/192.168.1.1 and added extra checks in the Go side for the redirect:

type CaptivePortalHandler struct {
 handler http.ServeMux
}

func (c *CaptivePortalHandler) ServeHTTP(w http.ResponseWriter, r *http.Request) {
 if r.Host != os.Getenv("HOST") {
 url := fmt.Sprintf("https://%s/", os.Getenv("HOST"))
 w.Header().Add("Location", url)
 w.WriteHeader(302)
 w.Write([]byte{})
 return
 }

 c.handler.ServeHTTP(w, r)
}

// on func main()
 http.ListenAndServe(listenAddr, &CaptivePortalHandler{handler: *handler})

Data storage

Canastra in production uses a PostgreSQL database to store all the game events, a single row for each event/player action. It’s not that the Raspberry Pi wouldn’t be able to run Postgres, but I wanted to store the events in a file in disk for this case, mostly so I could experiment supporting multiple persistence layers in Go. For that, I refactored my persistence layer in a way that I could support both Postgres and JSON files, just replacing the DATABASE_URL environment variable, so the changes between production and the Pi Zero build would be minimal.

Given this project doesn’t use CGO, compiling was a simple matter of using GOARCH=arm64 and uploading the binary to the Pi.

Results

I did a test run last night and it was successful!

The game works as intended, I can see no performance differences, even saving the events to the SD card.

Given the power requirements for the Pi Zero are quite low, I can run it from a powerbank for several hours, or even from the USB port on the in-flight entertainment. I’m already looking forward for our next trip in December, I’ll report back here then with a quick update.

Thank you!

New project: What to cook?

2023-09-21T00:00:00Z

Hi!

For a few years now, my partner and I had a dilemma almost daily: what would we make for lunch/dinner that day? It’s not a matter of lack of skill or ideas for recipes, or even having the ingredients available, but more of a nothing sounds appealing for the amount of effort I want to put in cooking this meal type of situation.

On top of that, I am particularly terrible to remember things I could cook. We’d have something that we both truly enjoyed, yet would not make it again for a long time simple because it was forgotten as an option. I need to see what options I have, and then maybe I can choose.

So for a few months I was joking about this idea, that I’d create some software that would just tell us what to cook. The internet is mostly converging to this approach too, with personalized feeds and “for you” pages, so we might as well take inspiration on the worse parts of what the internet has to offer.

You don’t know what to make for dinner? Click this button, and you’ll have a recipe. You don’t have a particular ingredient at home? No worries, exclude it, and no recipes with that ingredient will be offered again tonight. Don’t feel like cooking this recipe? Well, too bad, you won’t get another. You can’t just refresh the page or start over, you’ll only be able to get a different recipe in a few hours.

Jokes aside, I don’t think this is a problem that only we have, although I believe most people have the opposite problem: too many ideas for a single meal. I tried to search for projects that tried to tackle this problem, but most would just hover around voting systems or recipe management. I want something less democratic, something that takes the burden off of deciding what to eat.

So I built What to cook?. Run with Docker, input your recipes with the ingredients and click the button. The project is open to external contributions, but I plan to keep the project around the scope I presented here. I don’t want to turn it into another recipe management software, as there are tons of them out there. Maybe send a PR to import recipes from your favorite database?

I just started using it with my partner, and we’re still loading the database with our recipes as we cook them. My expectation is that it can go both ways: either we rely on it more than we currently want to admit, or it forces us to think better about what we want. In either case, I see it as a win.

Thank you.

fail2ban + Caddy with JSON logs

2023-09-10T00:00:00Z

Hi!

I’m running Caddy and saving access logs to disk in the JSON format. I want to integrate fail2ban to block bots trying /wp-login.php and other known URLs, and I couldn’t find much about how to make fail2ban read Caddy’s logs.

This is a hack that I quickly came up with, barely tested, but I managed to make it work:

/etc/fail2ban/filter.d/caddy-forbidden.local:

[Definition]
failregex = "client_ip":"<HOST>"(.*)"status":403
datepattern = "ts":<DATE>\.
ignoreregex =

Append to /etc/fail2ban/jail.local:

[caddy-forbidden]
port = http,https
logpath = /var/log/caddy/*.log
enabled = true

So an entry like this will successfully match and ban the IP 192.0.2.6:

{"level":"info","ts":1694328885.166141,"logger":"http.log.access.log2","msg":"handled request","request":{"remote_ip":"192.0.2.6","remote_port":"64305","client_ip":"192.0.2.6","proto":"HTTP/1.1","method":"GET","host":"www.kassner.com.br","uri":"/wp-login.php","headers":{...},"tls":{...}},...,"status":403,...}

Lastly, I’ve added to my Caddyfile:

(common) {
error /wp-login.php "Forbidden" 403
}

–

It’s worth noting this isn’t well tested, and given there is user input in the logs, you might ran into false-positives with a well-crafted URL or headers. Use at your own risk.

Thank you.

Using a Java class for DB migration

2023-08-29T00:00:00Z

Hi!

I got some good feedback from @phxql@fosstodon.org related to my previous post:

with flyway it’s possible to have a JavaMigration, which let’s you write custom code.

So let’s test this out today!

JavaMigration

It took me a while to make this work. I couldn’t figure it out what it was meant by adding the class to the db.migration package. Some places mentioned src/db/migration, some places mentioned src/main/java/db/migration and I even tried src/main/resources/db/migration (shouldn’t work, but here are my SQL files), but it was fruitless. In the end, @ComponentScan helped me again, as I created a new sub-package migration in my Spring project and annotated my class with @Component, that made it work, with the added benefit of allowing me to control the location of the files.

package com.ytemail.migration;

import org.flywaydb.core.api.migration.BaseJavaMigration;
import org.flywaydb.core.api.migration.Context;
import org.springframework.security.crypto.password.PasswordEncoder;
import org.springframework.stereotype.Component;

@Component
public class V3__RecalculateHashes extends BaseJavaMigration
{
 final private PasswordEncoder passwordEncoder;

 public V3__RecalculateHashes(PasswordEncoder passwordEncoder)
 {
 this.passwordEncoder = passwordEncoder;
 }

 @Override
 public void migrate(Context context) throws Exception
 {
 // code goes here
 }
}

Given Spring also initializes this object, I can use dependency injection to get any service or Bean I need to handle the migration. I couldn’t, however, use a Repository, as Spring would consider it a circular dependency:

***************************
APPLICATION FAILED TO START
***************************
Description:
The dependencies of some of the beans in the application context form a cycle:
┌─────┐
| flyway defined in class path resource [org/springframework/boot/autoconfigure/flyway/FlywayAutoConfiguration$FlywayConfiguration.class]
↑ ↓
| v3__RecalculateHashes defined in file [/home/kassner/workspace/ytemail/build/classes/java/main/com/ytemail/migration/V3__RecalculateHashes.class]
↑ ↓
| userRepository defined in com.ytemail.repository.UserRepository defined in @EnableJpaRepositories declared on JpaRepositoriesRegistrar.EnableJpaRepositoriesConfiguration
↑ ↓
| jpaSharedEM_entityManagerFactory
└─────┘
Action:
Relying upon circular references is discouraged and they are prohibited by default. Update your application to remove the dependency cycle between beans. As a last resort, it may be possible to break the cycle automatically by setting spring.main.allow-circular-references to true.

So we must do things the old way today:

@Override
public void migrate(Context context) throws Exception
{
 context.getConnection().beginRequest();

 PreparedStatement updateStmt = context.getConnection().prepareStatement("UPDATE public.user SET password = ? WHERE id = ?");
 PreparedStatement selectStmt = context.getConnection().prepareStatement("SELECT id, email FROM public.user");
 ResultSet resultSet = selectStmt.executeQuery();

 while (resultSet.next()) {
 Long id = resultSet.getLong("id");
 String email = resultSet.getString("email");
 String password = resultSet.getString("password");

 updateStmt.setString(1, passwordEncoder.encode(convertOldHash(password)));
 updateStmt.setLong(2, id);
 updateStmt.executeUpdate();
 }

 context.getConnection().commit();
}

Callback

Flyway has the concept of Callbacks, which allows you to plug either a SQL file or a Java class to run after each Event. You can also leverage that to run some Java code.

package com.ytemail.migration;

import org.flywaydb.core.api.callback.Callback;
import org.flywaydb.core.api.callback.Context;
import org.flywaydb.core.api.callback.Event;
import org.springframework.stereotype.Component;

@Component
public class Test1 implements Callback
{
 @Override
 public boolean supports(Event event, Context context)
 {
 if (event != Event.AFTER_EACH_MIGRATE) {
 return false;
 }

 if (!context.getMigrationInfo().getVersion().getVersion().equals("3.0.1")) {
 return false;
 }

 return true;
 }

 @Override
 public boolean canHandleInTransaction(Event event, Context context)
 {
 return true;
 }

 @Override
 public void handle(Event event, Context context)
 {
 // migration code goes here
 }

 @Override
 public String getCallbackName()
 {
 return "test1";
 }
}

In this example, supports is telling Flyway that this callback should only be executed for the event AFTER_EACH_MIGRATE and if the version that was just applied is 3.0.1. You could also tie back to your SQL file using context.getMigrationInfo().getScript() if you prefer. This is a quite similar approach to Magento’s deprecated UpgradeData classes, although I rather use JavaMigration for this purpose. I can see it being useful though, like running external commands like flushing a cache layer or reporting about the migration if you have observability tooling built into code.

Thank you.

First project with Java Spring

2023-08-25T00:00:00Z

Hi!

It has been a long time since I last wrote any meaningful Java code. Aside from some Jenkins plugin debugging here and there, the bulk of my Java experience is from the 2000s, mostly desktop apps (Swing/AWT), so after a nudge from a friend I’ve decided to dust off my long forgotten Java skills and I set to rewrite YT Email using Spring as a learning exercise.

First impressions

A big chunk of my software engineering experience is using PHP, that’s where it’s easier to compare things to, and right from the start I felt right at home, as most of the concepts are also present in Symfony/Doctrine and there is a high grade of translatability between both frameworks. There were a few things that Magento also does similarly to Spring but with different names.

start.spring.io makes it a breeze easy to bootstrap a new project, similar to the Symfony CLI, although in a web interface. Define your dependencies, extract the zip on your machine and git init is all you need to be up and running. Ah, and also IntelliJ IDEA, which allowed me to keep my PhpStorm keybindings.

Hello world

We start with a simple hello world.

package com.example;

import org.springframework.stereotype.Controller;
import org.springframework.web.bind.annotation.RequestMapping;

@Controller
public class HomeController
{
 @RequestMapping(path = "/")
 @ResponseBody
 public String home()
 {
 return "Hello World!";
 }
}

Comparing that to Symfony:

<?php

namespace App\Controller;

use Symfony\Bundle\FrameworkBundle\Controller\AbstractController;
use Symfony\Component\HttpFoundation\Response;

class HomeController extends AbstractController
{
 #[Route('/', name: 'home')]
 public function index()
 {
 return new Response('Hello World!');
 }
}

There are a few differences here that I’d like to unpack here. First, you need the @ResponseBody annotation in order to return the contents directly in the controller, otherwise the default will be to render a template with that name available in src/main/resources/templates. With Symfony, you’d return an object that implements the ResponseInterface, be that a JSON, a redirect or a supertype that handles the template parsing (i.e.: return $this->render('my/template.html.twig');). Annoyingly, if you want to redirect in Spring, you have to either return "redirect:/my/new/url"; or change the method signature to return a org.springframework.web.servlet.view.RedirectView object, which sounds like it would reduce the flexibility, although I haven’t had troubles with it yet.

The second thing that caught my eye was the @Controller annotation. In Symfony, the controllers are often placed in the src/controllers folder, although you can configure a different folder to be scanned (if you’re using the #[Route] annotation) or directly point each route to a method in any PHP class. In Spring, you just give your class the @Controller annotation and the routes (@RequestMapping or a subtype) will be parsed from it. This also means you can put your controller anywhere.

This sounds a bit too magic at first, but the answer lies in the @SpringBootApplication annotation from your main Application class (which came in the ZIP file from start.spring.io). @SpringBootApplication inherits @ComponentScan, which tells Spring to scan everything in the current package (and sub-packages), and that is true for other types of objects (like your @Configuration classes can be anywhere). I’m assuming this is done at compile-time, so in theory the resulting JAR could have a Map/List/Array with all the controllers named (similar to what composer dump-autoloader would do, but for controllers/routes/configuration/etc). Even if it doesn’t work as I imagine, I still appreciate the flexibility of just using the @Controller annotation, as it gives me more freedom to organize the code.

Dependency Injection

It’s not quite clear to me yet if it’s Java or Spring that provides each feature that I use here, but I do really enjoyed working with dependencies in this project. Both Symfony and Magento also have advanced dependency injection capabilities, so I started with the standard that I was used to, declaring the dependencies in the constructor:

@Controller
@RequestMapping(path = "/settings")
public class Settings
{
 private final MailService mailService;
 private final PasswordEncoder passwordEncoder;
 private final UserRepository userRepository;

 public Settings(MailService mailService, PasswordEncoder passwordEncoder, UserRepository userRepository)
 {
 this.mailService = mailService;
 this.passwordEncoder = passwordEncoder;
 this.userRepository = userRepository;
 }
}

I could also just use the @Autowired annotation without having them in the constructor, but you will get a NullPointerException if you ever do instantiate the class without using the DI container, so I rather avoid it.

Similarly to Symfony, you can also declare dependencies in your controller’s methods:

 @GetMapping
 public String index(@AuthenticationPrincipal User user)
 {
 //
 }

That injects the authenticated User into the controller, ready to be used.

Beans

Another concept that I see here and there are Beans. My first encounter was with Spring Security, in a situation similar to:

@EnableWebSecurity
@Configuration
public class SecurityConfig
{
 @Bean
 PasswordEncoder getPasswordEncoder()
 {
 return PasswordEncoderFactories.createDelegatingPasswordEncoder();
 }
}

The best comparision I can make is to the di.xml’s <preference> in Magento. With @Bean I’m telling Spring’s DI container whenever the PasswordEncoder is to be injected, pass the result of this method. The method getPasswordEncoder in the example above will only be called once (unclear to me if once per-request or once during the lifetime of the application), so Spring will just send the same instance of the object returned by the method whenever another class needs a PasswordEncoder object. Alternatively, this is basically the same as auto-wiring the objects (or interfaces!), but they require some setup to be instantiated. I’m also allowed to have a Bean for a ConcreteClass and it return a subtype of it, but unlike Magento, I can’t use a @Bean to override a class that has already been registered as a Bean somewhere else (i.e.: the @Service annotation). Maybe there is a package hierarchy involved, but I haven’t tested that deeper yet.

Relational databases

Having worked extensively with Doctrine 2, Hibernate felt like home. Entities, repositories, custom queries and life cycle hooks behave the same, using them in services is straight-forward, the schema can be auto-generated from the entities, etc. However, I do miss something like Doctrine Migrations in the Java world, as both Flyway and Liquibase don’t allow me to write any Java code to handle migrations. With Doctrine Migrations, I can write PHP to handle some edge cases in migrations, like this snippet I’ve extracted from the Money project:

<?php

declare(strict_types=1);

namespace DoctrineMigrations;

use Doctrine\DBAL\Schema\Schema;
use Doctrine\Migrations\AbstractMigration;

final class Version20230825001 extends AbstractMigration
{
 public function up(Schema $schema): void
 {
 $this->addSql('ALTER TABLE `transaction` ADD COLUMN amount_in_cents BIGINT;');
 $updateStmt = $this->connection->prepare('UPDATE `transaction` SET amount_in_cents = :amount_in_cents WHERE id = :id');

 $result = $this->connection->fetchAllAssociative('SELECT id, amount FROM `transaction`');
 foreach ($result as $item) {
 $amount = $this->convertFromStringToInteger($item['amount']);
 $updateStmt->bindValue('id', $item['id']);
 $updateStmt->bindValue('amount_in_cents', $amount);
 $updateStmt->executeQuery();
 }
 }

 public function down(Schema $schema): void
 {
 $this->addSql('ALTER TABLE `transaction` DROP COLUMN;');
 }
}

This would have been a nice feature, specially given YT Email already had a database and I wanted to just point the Spring project to the same DB in production. Thankfully I had only a few cases that would need a more elaborated migration, so I just fixed them manually.

Templating

I haven’t done much complex templating with Thymeleaf yet, but looking at the documentation it seems as feature-complete as Twig. Its syntax is much more tied to the HTML syntax, so to some extent you could also render the view directly in the browser for a quicker turnaround. I rather have Twig’s style though, but this isn’t unpleasant to use. To transfer data between the Controller and a Thymeleaf template, you can add attributes to either a org.springframework.ui.Model or org.springframework.web.servlet.ModelAndView that you declare in your controller method’s signature, i.e.:

 @GetMapping
 public String index(Model model, @AuthenticationPrincipal User user)
 {
 model.addAttribute("username", user.getUsername());
 return "settings/index";
 }

And to read them in Thymeleaf, <input th:value="${username}" /> will do the trick.

Deployment

Go allows me to deploy to deploy a single file, and that simplifies a lot the deployment process. With Spring Boot, I can do the same (provided I have java installed in the server), or the OCI image is also quite straight-forward:

FROM docker.io/library/openjdk:20
COPY build/libs/*.jar /app/app.jar
CMD ["java", "-jar", "/app/app.jar"]

Although I love PHP and have used the mainstream options to deploy projects with it, it is unfortunately a lot more involved, especially so that PHARs never took off and it’s cumbersome to make them work for web projects. If you are using OCI containers in and out, it won’t make much difference. Spring handles signals for you, so I don’t need supervisord in my containers anymore.

Outro

In general, I quite enjoyed working with Spring, and will likely build a few more projects with it in the future. I do miss a few tools, however, like Symfony’s Profiler and the Web Debug Toolbar, as they give me so much information during development, and the SymfonyMakerBundle, that with a single CLI command auto-generate a bunch of different classes to speed up the process. I haven’t really searched for them though, and there might also be IntelliJ extensions that do some parts of their job. All in all, I’m looking forward for my next feature/project using Java and Spring!

Thank you.

SSH server behind NAT

2023-05-21T00:00:00Z

Hi!

I have an always-on Raspberry Pi at home, and once in a while I need to connect to something on my home network, or even exit to the internet as if I were at home (quite handy to access services that block datacenter/country IP ranges). This post documents all the steps needed to make it work.

Architecture

My home connection is behind a few of layers of (CG)NAT, so I can’t connect to it directly from outside my home network. Instead, I’ll be tunnelling through a VPS that I own. This approach consists of two parts: a persistent SSH tunnel between the Raspberry Pi and a VPS, and a connection from my laptop to the Raspberry Pi, through the SSH tunnel.

This approach isn’t quite performant, specially with Raspberry Pi 3B scoring low on SSH performance. I often see the latency increasing four fold through the tunnel (with everything being ~100ms apart of each other), and bandwidth is also expected to be limited (my tests yielded around 20Mbps, with the weakest link capped at 50Mbps). It works alright for my use case, but I wouldn’t expect to stream video through the tunnel.

Configuration

Raspberry Pi: generate SSH key

Generate a SSH keypair for the root user of the Raspberry Pi. Copy the public key to configure in the VPS.

VPS: dedidated user for the tunnel

Assuming you already can SSH into your VPS from anywhere, we now need to allow the Raspberry Pi to create a tunnel to the VPS. I’m setting up a dedicated Linux user for this:

root@vps# adduser -m -s /usr/sbin/nologin ssh-a1b2c3d4

Then configure the .ssh/authorized_keys of this user:

command="/usr/sbin/nologin",port-forwarding,permitlisten="127.0.0.1:50001",no-pty ssh-public-key-goes-here

The extra parameters at the beginning make sure the user can only setup the tunnel and forward ports, but has no access to a shell.

Raspberry Pi: setup the persistent tunnel

I’m using autossh, as it monitor the connection and restarts if it becomes stale. Quite handy if your home connection isn’t stable. To make it persistent across reboots, I’m setting it up as a systemd unit:

# cat /etc/systemd/system/sshtunnel.service
[Unit]
Description=SSH Tunnel
After=network.target
[Service]
Restart=always
RestartSec=20
User=root
ExecStart=/usr/bin/autossh -M 0 -nNT -o ServerAliveInterval=120 -R 127.0.0.1:50001:localhost:22 ssh-a1b2c3d4@vps-ip-address
[Install]
WantedBy=multi-user.target

This will bind the port 50001 in the VPS to the port 22 of the Raspberry Pi.

Don’t forget to run systemctl enable sshtunnel and systemctl start sshtunnel.

Usage

To make use of the tunnel, run:

$ ssh -J vps-user@vps.ip.goes.here -p 50001 pi-user@127.0.0.1

This will connect to the pi-user in the Raspberry Pi, via the SSH tunnel we set up earlier. To use the same tunnel as a SOCKS5 proxy, so I can exit to the internet as if I was home, use:

$ ssh -J vps-user@vps.ip.goes.here -p 50001 pi-user@127.0.0.1 -D 1337 -q -C -N

This opens a SOCKS5 proxy on port 1337 on my computer, which then I can configure in the browser/system.

.ssh/config

To simplify usage it’s possible to run ssh pi-home and connect to it from anywhere. Add those lines to the .ssh/config of the computer that will access the Raspberry Pi:

Host vps
HostName vps.ip.goes.here
User vps-user
Host pi-home
HostName 127.0.0.1
Port 50001
User pi-user
ProxyJump vps

Thank you.

Reusing old hardware for self-hosting

2023-05-16T00:00:00Z

Hi!

I used to self-host a NAS in my home network out of a Raspberry Pi 4B and a couple of HDDs, but after a few years I’ve learned that the Pi is somewhat underpowered for my needs, getting in the way of my backup strategy. Once in a while I would search for NUCs and other small form factor computers, but I never found something in the range that I was comfortable to pay. Eventually, however, something clicked: I had the right solution under my nose the entire time: a 2011 Dell Vostro 3450.

My partner had purchased this device for her college years and it had reasonably beaten up when she graduated, so it was retired and sat in a closet ever since. Apart from the bend in the body and a few keys not working (notably the Ctrl and Alt keys), the display is also somewhat loose and the right side ports (HDMI, USB, eSATA!?) are shorted, so the system would sometimes reboot or display garbage when messing with the USB port. I also got a few a small discharges that I initially attributed to low humidity ESD (somewhat frequent for me with any hardware), but thankfully I didn’t lose any USB devices until I realized what was going on. Covering the ports with electrical tape was a quick and cheap solution.

The 2.3GHz i3-2350M with 4GB DDR3 isn’t particularly better than the Raspberry Pi 4B in the benchmarks, but given my use case could see better single-core performance (Nextcloud image resizing kept me waiting) and it had cryptographic acceleration, I figured it was worth the try. I installed Debian, configured BTRFS (I’m using btrfs snapshots for incremental backups) and it quickly became part of my local infrastructure.

The way I use my NAS isn’t quite standard, however. I initially went with Nextcloud as a Google Photos replacement, and for me it meant mostly archival, so the initial task was just backing up my phone’s photos. I’d turn on the NAS, run the sync, run the incremental backup scripts and shut it down. I eventually started using the file sync with a couple of computers at home, but was still using a free Dropbox account for day-to-day small things. If I edited something from my Nextcloud folder in a computer, I’d likely wait a few days until it was “worth the hassle” to turn the NAS on again, and at this point I was making the same requires willpower mistakes once again.

Eventually I wanted to bring home more services, like LinkAce, FreshRSS and Gitea, so I decided to transform it into an always-on solution. First I had to address the spinning rust noise, as I caught myself avoiding turning the NAS on just because of the low hum it made in the background. To fix this I purchased an 1TB SSD and went on to replace it, just to realize that the disk was the last thing I could remove:

Given I had it open and fully disassembled at this point, and also because of the shorted ports and other rusty parts in the motherboard, I second guessed myself and decided to not try with a brand new SSD before making sure I didn’t screw it up. So I bought some cleaning supplies (a can of compressed air, isopropanol, thermal paste), scavenged a 240GB SSD and a pair of 4GB DDR3 DIMMs out of a 2009 MacBook Pro and did a thorough cleaning and reassembly.

It booted! Debian was then installed, I ran my Ansible playbook and synced back my Nextcloud BTRFS snapshots into the SSD. The baseload uses around 300MB RAM and is silent, at 47°C it can be mistaken for a fanless system, as I only ever hear the fan when I’m browsing through Nextcloud’s gallery and it needs to generate thumbnails. It idles at around 12.6W with the screen on, and I have seen peaks of 38W with a couple of USB HDDs connected and a lot of load. 12W is quite low, but still 2.5 times more than a Pi, and I thought there was room for improvement.

The first win came out of the consoleblank Linux kernel parameter. I configured it to 60 seconds, and after the timeout it completely shuts off the display, bringing the idle consumption to 8.1W. While 8W amounts to about 30 SEK/month with my current electricity contract, I’m not particularly fond of wasting energy, but fortunately I got lucky as this machine had one last surprise hidden for me: full Wake on LAN support! I did some initial tests with WOL using Magic packets and it worked flawlessly. The system in the suspended state consumes just 1.1W and a magic packet is enough to wake it back to fully functional in less than 5 seconds. One problem I’m still looking for a fix is that the consoleblank that does not turn off the display once the system resumed from suspend.

So my next challenges in this project are all in software. I want to automatically suspend the system based on my usage patterns. I also want to avoid the requires willpower pitfall and have something awake the system when there is usage, but I suspect using the WOL unicast mode might conflict with the Nextcloud desktop app trying to stay connected. Alternatively, I can have my home DNS server (an always-on Raspberry Pi 3B) to send the magic packet when it detects usage.

Thank you.

Understanding Go URL handling

2023-03-24T00:00:00Z

Hi.

A couple of years ago I “took inspiration” for a HTTP reverse proxy in Go from Stack Overflow without putting too much thought into it, and this week it bit me back. A co-worker found out that it was normalising some URLs (/something//else will 301-redirect to /something/else) against their will. So I decided to take the opportunity and understand better how net/http handles URLs, and here are my findings.

Minimal Go HTTP Server

package main

import "net/http"

func main() {
 http.ListenAndServe(":8080", nil)
}

This simplest HTTP server you can write in Go. Given we’re providing the handler argument with nil, it won’t do much other than open the TCP port and return 404 for any URL you request. While that’s not particularly useful, is enough to make sure we’re able to compile, run and make requests.

However, it’s more likely that you’ll want the server to show some content, so for that you could write:

package main

import "net/http"

func main() {
 http.HandleFunc("/", func(writer http.ResponseWriter, request *http.Request) {
 writer.Write([]byte("Hello, world"))
 })

 http.ListenAndServe(":8080", nil)
}

And sure enough, it works! Requesting / returns Hello, world. But so does requesting /something. Hum. Why is that?

The first argument in HandleFunc is pattern, which will start matching at the beginning of the URL. So http.HandleFunc("/test/", ...) will return 200 for /test/ and /test/another/, but 404 for /another/test/. However, http.HandleFunc("/test", ...) (without the trailing slash) will only match /test, and will return 404 for both /test/ and /test/another. So ending the pattern with / has some special handling.

http.HandleFunc(pattern)	Requested URL	HTTP status
`"/"`	`/`	200
`"/"`	`/test`	200
`"/"`	`/test/`	200
`"/"`	`/test/another`	200
`"/"`	`/test/another/`	200
–	–	–
`"/test"`	`/`	404
`"/test"`	`/test`	200
`"/test"`	`/test/`	404
`"/test"`	`/test/another`	404
`"/test"`	`/another/test`	404
`"/test"`	`/another/test/`	404
–	–	–
`"/test/"`	`/`	404
`"/test/"`	`/test`	301 to `/test/`
`"/test/"`	`/test/`	200
`"/test/"`	`/test/another`	200
`"/test/"`	`/another/test`	404
`"/test/"`	`/another/test/`	404
–	–	–

So how do I match / and only /? Let’s try with an empty string for http.HandleFunc:

$ go run main.go
panic: http: invalid pattern
goroutine 1 [running]:
net/http.(*ServeMux).Handle(0x14622e0, {0x0, 0x0}, {0x12e7c80?, 0x12a8380})
/usr/local/Cellar/go/1.20.2/libexec/src/net/http/server.go:2510 +0x25f
net/http.(*ServeMux).HandleFunc(...)
/usr/local/Cellar/go/1.20.2/libexec/src/net/http/server.go:2553
net/http.HandleFunc(...)
/usr/local/Cellar/go/1.20.2/libexec/src/net/http/server.go:2565
main.main()
/Users/kassner/workspace/test/main.go:6 +0x33
exit status 2

It doesn’t compile. Digging into it I found out that it’s actually not possible using the default behaviour. But as it was pointed out in the thread:

The ServeMux isn’t fundamental to the net/http package. If you don’t like its behavior, you can write your own mux.

Terminology is hard

So what is a ServeMux? Quoting from Go’s source:

ServeMux is an HTTP request multiplexer. It matches the URL of each incoming request against a list of registered patterns and calls the handler for the pattern that most closely matches the URL.

I have known that as a router from some Web Frameworks (Express, Spring, Symfony). All it does is to match a particular URL to an arbitrary piece of code that I declared. Sounds what I need, right?

Turns out the term ServeMux only confused me. What I needed is a Handler, and ServeMux is only one implementation of a handler that comes baked into Go. Of course, in some situations ServeMux might be enough, I just wish it was clearer to me it was a DefaultHandler.

It also didn’t help me that Go has both http.HandlerFunc and http.HandleFunc. The former is used to wrap a func into Handler, which is what we need to create our own Handler, and the latter does something similar, but registering the function with a pattern matching against the DefaultServeMux. A simple typo can cause you a good amount of head-scratching.

I have a preference for verbose and explicit intent, so I find this more comprehensive to read:

package main

import "net/http"

func main() {
 http.DefaultServeMux.HandleFunc("/", func(writer http.ResponseWriter, request *http.Request) {
 writer.Write([]byte("Hello, world"))
 })

 http.ListenAndServe(":8080", nil)
}

That way it’s explicit that I’ll be using the ~~DefaultHandler~~ServeMux provided by Go, with all the behaviours that come out of the box.

Crafting a Handler

So, if a Handler is all we need, how do we rewrite our server using it?

package main

import "net/http"

func main() {
 handler := http.HandlerFunc(func(writer http.ResponseWriter, request *http.Request) {
 writer.Write([]byte("Hello, world"))
 })

 http.ListenAndServe(":8080", handler)
}

As we didn’t add any logic to it, any URL will return Hello, world. More importantly though, a request with double-slashes, which was my original problem, will not be normalized, which confirms we’re overriding the default behaviour.

$ curl -v http://localhost:8080/something//else
* Trying 127.0.0.1:8080...
* Connected to localhost (127.0.0.1) port 8080 (#0)
> GET /something//else HTTP/1.1
> Host: localhost:8080
> User-Agent: curl/7.79.1
> Accept: */*
>
* Mark bundle as not supporting multiuse
< HTTP/1.1 200 OK
< Date: Fri, 24 Mar 2023 06:01:38 GMT
< Content-Length: 12
< Content-Type: text/plain; charset=utf-8
<
* Connection #0 to host localhost left intact
Hello, world

And if I want to match just /:

package main

import "net/http"

func main() {
 handler := http.HandlerFunc(func(writer http.ResponseWriter, request *http.Request) {
 if request.URL.Path == "/" {
 writer.Write([]byte("Hello, world"))
 }

 http.NotFound(writer, request)
 })

 http.ListenAndServe(":8080", handler)
}

Conclusions

My main takeaway is that I need to be more careful with assumptions and read the source/manual more often, instead of just assume things with the same name work similarly across languages. Going through the issue also led me to read Go’s source code, which I honestly expected to be a lot harder than my previous experiences, mainly because it’s written in the same language. Lastly, it also showed me how a developer could take some shortcuts that would not please my explicit intent preferences, so I can be aware of those gotchas in the future. I’m glad I’ve done this.

Thank you.

Update to my ZFS backup strategy

2022-03-16T00:00:00Z

I briefly outlined in my ZFS backup strategy blogpost about my NAS setup, but here it’s a quick recap: I have a Raspberry Pi 4 4GB with a 1TB SATA HDD over USB running under the TV in my living room, and a second USB HDD for mirroring. I’ve been running this setup for around 18 months now, and unfortunately it doesn’t quite fit my needs.

In the previous post, I focused too much in the remote/cloud backups for ZFS, so I just took it for granted that mirroring the disks would be trivial using ZFS. While ZFS does mirroring by default, now I understand that it’s intended as a solution for always-online disks, so I couldn’t rely on that feature without ZFS constantly nagging that the zpool is unhealthy and resilvering the disk every time I plugged it in. To get around that, I’ve decided to keep the zpool with a single disk, and zfs send the data to the second disk once every when I felt like, mostly because once the disks were fully synced, the delta between the second disk and what’s stored on the cloud would be quite small (<100MB).

That was a rookie mistake. Not only it meant I wouldn’t have 3 copies of the data anymore (or at least for the data changes from the last full sync), the data sync between those two disks was a complete disaster. I didn’t do a proper research and did my initial sync only after I had already commited to this strategy and had moved in all my data. This meant that I failed to test a pretty trivial thing: can the Raspberry Pi power two USB HDDs at once? It can’t. It will boot fine with one disk, but it won’t boot with two, and plugging the second disk after it’s fully booted and idling still wasn’t enough for the second disk to boot.

With that limitation, I told myself that it would be fine if I plugged the second disk in my Lenovo notebook and did a sync over the network. My home network is gigabit capable, and the syncs wouldn’t be that big after the initial full sync, plus it meant that I’d have ZFS on Linux already setup in my Fedora installation, in case the main disk went bust and I’d need quick access to the files. I wrote a script for ssh nas zfs send | zfs recv and, oh my, how that was slow. The Raspberry Pi does not have any cryptographic hardware acceleration, so the transfer speed was capped to ~16MB/s, which meant several hours for the initial sync.

I’ve tried to help the Raspberry Pi a bit by doing a non-encrypted transfer with zfs send | nc and nc -l over a direct cable connection between the Pi and the notebook, and I got close to 85MB/s in that scenario, which was a lot more palatable. However, it also meant a tripping hazard (a.k.a.: cable) spanning from my living room to my office for a few hours, so I began to regret my decisions. But at least now the disks were synced and subsequential syncs were manageable over SSH, as the deltas were small.

Enter kernel updates.

Fast forward 3 months, I’m back from a holiday abroad in which I took my NAS server with me (but not the second disk). I kept the NAS server up to date, and ZFS on Linux mostly worked across updates (although it was a bit of a pain between ZOL 0.8.4 and 2.0.0). Given my Lenovo notebook is my main personal machine, I use Fedora because of the speed packages get updated. This also meant, after I was back from vacation, that I had the 5.16 Kernel running in it, but ZFS on Linux’s support for it wouldn’t come for another 2 months.

Sidenote: this post is in no way a critique to the ZFS on Linux project. Using ZFS at all is only possible because of their work, and I understand well enough the problems with unpaid Open Source contributors and “AS IS” and “no guarantees” open source licenses. I’m thankful for the multiple authors of OpenZFS and ZOL that made this possible, and all the faults on my strategy are solely my own.

This mean my strategy fell into pieces. My initial expectation was that I’d just plug the second disk to the NAS server every second week or so and it would automagically keep both disks mirrored. In practice, I had to constantly make effort into keeping both disks in sync, plugging into a second machine, fiddling around with some commands and a bit of lack of preparation made me resync the disk once again because I screwed up some parameters in the zfs recv.

ZOL 2.1.3 was released last week with 5.16 Kernel support, but I haven’t got around to sync the disks again, and now they are unsynced for about 4 months. I kept the snapshots in the primary disk, so I should be able to do an incremental sync whenever I get around it, but the situation I’ve put myself doesn’t get me excited to jump into that problem. Also, this doesn’t mean it won’t happen again, so I have to start from scratch and develop a new strategy that considers both cloud incremental backups as much as mirroring to a secondary disk that will not stay connected all the time. Maybe ZFS is not the answer for me, maybe Raspberry Pi is not the answer for me, maybe ZOL is not the answer for me. I don’t know yet.

The good part of this experiment is that I’ve got to learn what not to do. So for my future self, a small summary of learnings about this experiment:

Consider the impacts of bleeding edge package updates for infrastructure services;
The server should be capable of executing the entire backup strategy on its own;
Consider the power requirements of your system;
Servers should have cryptographic hardware acceleration;
Don’t take for granted trivial parts of the backup processes and validate the entire model before commiting to it;
Manual steps in a repetitive process must take into consideration your willpower to deal with it;
Under no circumstance assume your partner will be happy with a UTP cable crossing the living room;

If you do have any suggestions for any of my downfalls in this project, I’ll appreciate the feedback.

Thank you.

Lenovo notebook updates

2022-03-01T00:00:00Z

It has now been close to half a decade since acquisition of my Lenovo Y720-15IBK, so I thought about writing an update this machine, given it is still my main personal machine, seeing almost daily usage for general browsing and a bit of programming. I also game on it semi-regularly, having played a good deal of Forza Horizon 4 (Ultra @ 1080p) and ETS2 over this holidays season.

Hardware

Upgrading to a Kingston A2000 was the only hardware change I had done in this machine, mostly because I needed 1TB of storage. I haven’t seen the need for an upgrade to a better machine, as I can still run the games I like to play (disclaimer: I only have 60Hz displays and I don’t care much about graphics quality anyway) and all my programming happens on Linux (no VMs), so the i7-7700HQ should be plenty for a little while.

Unfortunately, opening the back aluminium case to replace the M.2 disk seems to have affected the touchpad negatively. It was never an outstanding touchpad, I often had to click more than once to produce a click with it, but I feel it became a little worse and the percentage of misclicks went up a bit. The keyboard also had its quirks since purchase (I missed the return window and was lazy to send it to Lenovo to investigate), but given I use it mostly attached to a external screen/keyboard/mouse, it’s not a big deal.

The battery always felt it wasn’t enough given the machine has a NVIDIA GTX 1060, but it didn’t degrade much so far. I can still squeeze a good couple of hours out of it when running Linux, which I consider a huge feat for the size of the machine.

Fedora

I haven’t switched away from Fedora since my attempt back in 2018. I’m now on Fedora 34 and most of the quirks from my Fedora blogpost are long gone. Eventually the NVIDIA drivers from RPM Fusion became a “it just works” approach, and given I have no need for a dual-GPU setup, I’ve been using them ever since. Secure Boot was neglected though, since the new M.2 disk I’ve just ignored it completely, as the reinstall was done in a rush before travels.

Keyboard backlight

For years I accepted that the keyboard backlight wouldn’t work with Linux. Last year I’ve found Izurii/Lenovo-Y720-KB-Led-Controller, which is an Electron-based app to emulate what the Lenovo Nerve Sense for Windows did for the keyboard backlight. And it works, but given Electron is such a resource hog compared to a single CLI tool, it didn’t made sense to keep it.

I had a day off work this week and decided to play around a bit. I reviewed what the Electron app did and it’s core was just a bunch of ioctl system calls with the right payload. I eventually made it work with static colors and decided that was enough, only to realize that someone already rewrote the app in a more old-school Linux friendly way. Alas, at least I dusted off my C skills and still got a tool with all the configs!

–

In general, I’m still happy with the machine, I feel like I got out of it what I paid for and thankfully it is still a decent machine that will see several more years of service.

Create write-only keys for Backblaze B2

2021-09-05T00:00:00Z

As far as I remember, you can’t create a write-only key via Backblaze’s dashboard without also giving read access to the key. I want to use this specifically for uploaders in servers, so, if compromized, an attacker can’t read data out of the bucket.

$ curl https://api.backblazeb2.com/b2api/v2/b2_authorize_account -u "MASTER_KEY_ID:MASTER_KEY_SECRET"
{
"apiUrl": "https://api003.backblazeb2.com",
"authorizationToken": ".....",
}

Replace apiUrl and authorizationToken in the next command:

$ curl https://$apiUrl/b2api/v2/b2_create_key -d '{"capabilities": ["listBuckets","writeFiles"],"keyName":"key-name","accountId":"MASTER_KEY_ID"}' -H 'Authorization: $authorizationToken'
{
"accountId": "0f0f0f0f0f0f",
"applicationKey": "K....",
"applicationKeyId": "00....",
"bucketId": null,
"capabilities": [
"listBuckets",
"writeFiles"
],
"expirationTimestamp": null,
"keyName": "key-name",
"namePrefix": null,
"options": [
"s3"
]
}

That’s all.

My ZFS backup strategy

2020-12-19T00:00:00Z

I am now doing some experiments running my own NAS at home (mostly out of boredom), and I went with a small solution that goes inside my IKEA PS with a Raspberry Pi 4, a couple of 1TB USB SATA disks and ZFS on Linux mirroring them. I have less than 200GB in data and a very stable 50Mbps uplink at home, so this post explains my strategy to backup my data in a remote location.

Before I even started, I had to decide what would be my backup strategy. Given I’m running this from a home connection, it wasn’t an option to do full backups every day, specially when my data rarely changes more than 50MB in a given day, so I had to think about something that would play nice with incremental backups.

I first researched tools for such tools, and was amazed by Borg, but given I planned to backup this directly to an S3/GCS/B2 bucket, I found too many limitations on it. I can’t have the borg binary running in the remote, so I either needed a full copy of the data locally or it had to download a lot of data from the bucket to be able to compute the incremental diff.

My second test was with ZFS. Seeing demos of zfs send | ssh remote zfs recv was very beautiful, but again, I won’t have a way to run ZFS on the remote. But I did a few tests with it that caught my attention. First, snapshot management in ZFS is awesome to manage changes across time. Second, you can pipe zfs send to a file. Third, while zfs send is commonly used to send a snapshot to another machine also running ZFS (zfs send | ssh remote zfs recv), when you pipe zfs send to a non-ZFS command (or file), it will dump the entire filesystem, essentially creating a full backup. Fourth, you can zfs send just the diff between two snapshots, even to a file. So I put all of them together to create this backup strategy.

Backup strategy

Note: the command backup-upload below is an unpublished tool that will compress, encrypt and upload the file to a remote location.

Creating the full backup

I’ve imported some data and then created a snapshot with zfs snapshot data@<date> (i.e.: zfs snapshot data@2020-08-24). This snapshot was then uploaded to a remote location using zfs send data@2020-08-24 | backup-upload full/2020-08-24.

Creating daily backups

Every day, a cronjob creates a new snapshot and uploads to remote using zfs send -I <yesterdaySnapshot> <todaySnapshot> | backup-upload daily/<date>).

Creating monthly backups

Once a month, after the daily snapshot was created, a cronjob uploads it using zfs send -I <fullbackup> <todaySnapshot> | backup-upload monthly/<date>).

This cronjob will also mark for deletion older monthly and daily backups, except the full backup.

Restoration strategy

Backups are worthless if they can’t be restored. This is the strategy to restore the data from the remote location. This assumes the data in both my local disks can’t be trusted anymore, so I’ll restore the backups from remote into a brand new disk with an empty zpool.

While creating the monthly backups, I run zfs send against the <fullbackup>. This means that the restoration process needs 1) the full backup; 2) the latest monthly backup and 3) all daily backups since the last monthly backup. This way most of my data will be restored by obtaining just two files, while I still retain the daily granularity if my future self wishes to use it.

The backup-download command below is an unpublished tool that will download, decrypt and decompress the file from a remote location.

Commands:

backup-download full/2020-08-24 | zfs recv data
backup-download monthly/2020-12-01 | zfs recv data
backup-download daily/2020-12-02 | zfs recv data
backup-download daily/2020-12-03 | zfs recv data
...
backup-download daily/2020-12-19 | zfs recv data

Outro

This is just an experiment and this blog post is a live document that will be updated as I fine tune my strategy. I am yet to buy a new disk and attempt full restoration from remote (I did small scale tests only). I am not too worried about losing data at this point because I have another external disk, without ZFS, that I manually copy all the data into regularly. If you want to try this yourself be careful, as you can lose data.

Fedora 28 on Lenovo Y720-15IBK

2018-06-27T00:00:00Z

It’s almost a year since I purchased this gaming computer and just now I had the need to do anything else than gaming in it. I have a friend that recently bought a notebook with a Nvidia GTX 1060 and he installed Fedora, making it a very snappy workstation, so I decided to give Fedora a go again. My computer came with Windows 10 by default and I never changed anything there, so here I am writing down all knowledge I got from installing Fedora in this machine.

Partitioning

Partitioning the disk using the Windows disk management tool worked just fine.

Live USB

Fedora Media Writer for Windows worked as intended.

Rebooting into the Live USB

Two tricky things. First one is disable Fastboot. The second one is that even disabling it on Windows, I still couldn’t get into my BIOS. I had to click on restart while holding shift and then use the Windows repair menu to get into the BIOS. Once inside the BIOS, I disabled the Fastboot and also allowed the boot over USB that was needed. Figuring out how to make into the BIOS took me some time also, but somehow I learned that I should use Fn + F2 for the BIOS and Fn + F12 for the boot menu.

Disk not recognized

Booting into the Live USB was fine, but the installer could not recognize my NVMe disk at all. After a lot of research I landed into this tip that was to set the Sata Controller Mode to AHCI. I had Intel RST Premium configured there and doing the change, Fedora was able to find my disk.

Sadly, I could not get back to Windows with that configuration. Selecting Windows on GRUB will reboot the computer and start attempt to load Windows, but eventually I get an Windows-like error message saying that was not possible to boot. The error code was INACCESSIBLE_BOOT_DEVICE.

It took me a couple of hours investigating just this issue until I managed to make it work. I had to Switch Windows 10 from RAID/IDE to AHCI, but I struggled a bit because I was doing that from the Repair tool instead of loading Windows normally. After I gave it a go from scratch, it worked as expected. Now I have dual boot on GRUB for Fedora and Windows, and my Sata Controller Mode is set to AHCI.

NVidia driver

Installation

This tutorial covers majority of the installation process, but I’ll detail here what went wrong.

Issues

ERROR: could not insert 'nvidia_drm': Operation not permitted.

Disabling Secure Boot and SELinux while installing the driver fixed this problem. Providing the x509 cert and key created before to the NVidia installer did not help. I had to disable Secure Boot and sign it manually later.

Wrong resolution

xrandr -q was showing eDP-1-1 primary connected 960x540+0+0. My monitor is 1080p. To fix that, I tried several different things, like setting the Mode manually inside the xorg.conf, setting the DPI, extracting the EDID using NVidia tools, disable EDID usage, xrandr with different combinations, and several variations inside the Xorg.conf, but nothing worked. In the end, it was just a matter of enabling DPMS and my xorg.conf now looks like this.

I also followed the scripting part of this tutoria, but I am not sure if this is doing anything for me right now. I also have the xrandr lines on my ~/.profile.

Secure Boot

I disabled Secure boot while installing the Nvidia driver, but later I enabled it back. It was less difficult than I expected. I followed this tutorial from Laurent, but I am writing here it again for backup AND because I didn’t find the .ko files in the first place (because they were compressed).

First off, I had to generate some keys and add them to the Secure Boot mechanism, otherwise UEFI wouldn’t be able to know if the modules it is loading are trusted or not. For that, I needed a configuration file, which I created at ~/x509.ini with the content:

[req]
default_bits = 4096
distinguished_name = req_distinguised_name
prompt = no
string_mask = utf8only
x509_extensions = myexts
[req_distinguised_name]
O = kassner
CN = kassner
emailAddress = email@example.com
[myexts]
basicConstraints=critical,CA:FALSE
keyUsage=digitalSignature
subjectKeyIdentifier=hash
authorityKeyIdentifier=keyid

Then:

openssl req -x509 -new -nodes -utf8 -sha256 -days 36500 -batch -config x509.ini -outform DER -out public_key.der -keyout private_key.priv;
mokutil --import public_key.der;
Reboot and follow the instructions to import the certificate;

And to sign the NVidia drivers:

find / -name 'nvidia.ko.xz'. The other files were in the same folder;
Backup them somewhere else;
xz --decompress *.ko.xz;
/usr/src/kernels/$(uname -r)/scripts/sign-file sha256 ~/private_key.priv ~/public_key.der nvidia.ko and for the other .ko files;
xz --compress nvidia.ko and the other .ko files as well;
modinfo nvidia to be sure they are working;

Language, keyboard, locale, etc

I have a weird keyboard and language combination that I had to address for this computer. My computer has a Swedish keyboard, with the ä ö å keys. I have an USB keyboard that shares the same layout. But, I was raised with a US keyboard and I do write Portuguese every day as well, so writing letters like é ç ã is essential for me. At the same time, I rather use the US keyboard layout instead, since my muscle memory is already built around it. To orchestrate all this mess, I spent several days trying out different things until I finally managed.

Sadly, since it was so scattered around and I couldn’t remember everything I did, I am most likely missing some things here. I am writing what I remember and I’ll update in the future if I ever find out what is wrong.

Dates

date and some other parts of some apps were returning localized information for me in Swedish. This was mainly due the fact that only $LANG was set to en_US.UTF-8 and all other variables were set to sv_SE.UTF-8, because I selected English as language and Sweden (English) as format.

To fix this, I tried several different things, and what I believe it worked was:

sudo localedef -v -c -i en_US -f UTF-8 en_SE.UTF-8
sudo localectl set-locale LANG=en_SE.UTF-8

And keep my /etc/locale.conf like this:

LANG=en_SE.UTF-8
LC_CTYPE="en_SE.UTF-8"
LC_NUMERIC="en_SE.UTF-8"
LC_TIME="en_SE.UTF-8"
LC_COLLATE="en_SE.UTF-8"
LC_MONETARY="en_SE.UTF-8"
LC_MESSAGES="en_SE.UTF-8"
LC_PAPER="en_SE.UTF-8"
LC_NAME="en_SE.UTF-8"
LC_ADDRESS="en_SE.UTF-8"
LC_TELEPHONE="en_SE.UTF-8"
LC_MEASUREMENT="en_SE.UTF-8"
LC_IDENTIFICATION="en_SE.UTF-8"
LC_ALL=

Compose

I copied from /usr/share/X11/locale/en_US.UTF-8/Compose to ~/.XCompose and applied the following diff.

--- Compose 2018-06-23 21:37:15.647432197 +0200
+++ .XCompose 2018-06-23 22:19:59.832729155 +0200
@@ -611,7 +611,7 @@
<Multi_key> <asterisk> <A> : "Å" Aring # LATIN CAPITAL LETTER A WITH RING ABOVE
<Multi_key> <A> <asterisk> : "Å" Aring # LATIN CAPITAL LETTER A WITH RING ABOVE
<Multi_key> <A> <A> : "Å" Aring # LATIN CAPITAL LETTER A WITH RING ABOVE
-<dead_cedilla> <C> : "Ç" Ccedilla # LATIN CAPITAL LETTER C WITH CEDILLA
+<dead_acute> <C> : "Ç" Ccedilla # LATIN CAPITAL LETTER C WITH CEDILLA
<Multi_key> <comma> <C> : "Ç" Ccedilla # LATIN CAPITAL LETTER C WITH CEDILLA
<Multi_key> <C> <comma> : "Ç" Ccedilla # LATIN CAPITAL LETTER C WITH CEDILLA
<Multi_key> <cedilla> <C> : "Ç" Ccedilla # LATIN CAPITAL LETTER C WITH CEDILLA
@@ -736,7 +736,7 @@
<Multi_key> <asterisk> <a> : "å" aring # LATIN SMALL LETTER A WITH RING ABOVE
<Multi_key> <a> <asterisk> : "å" aring # LATIN SMALL LETTER A WITH RING ABOVE
<Multi_key> <a> <a> : "å" aring # LATIN SMALL LETTER A WITH RING ABOVE
-<dead_cedilla> <c> : "ç" ccedilla # LATIN SMALL LETTER C WITH CEDILLA
+<dead_acute> <c> : "ç" ccedilla # LATIN SMALL LETTER C WITH CEDILLA
<Multi_key> <comma> <c> : "ç" ccedilla # LATIN SMALL LETTER C WITH CEDILLA
<Multi_key> <c> <comma> : "ç" ccedilla # LATIN SMALL LETTER C WITH CEDILLA
<Multi_key> <cedilla> <c> : "ç" ccedilla # LATIN SMALL LETTER C WITH CEDILLA
@@ -873,11 +873,9 @@
<Multi_key> <a> <semicolon> : "ą" U0105 # LATIN SMALL LETTER A WITH OGONEK
<Multi_key> <comma> <a> : "ą" U0105 # LATIN SMALL LETTER A WITH OGONEK
<Multi_key> <a> <comma> : "ą" U0105 # LATIN SMALL LETTER A WITH OGONEK
-<dead_acute> <C> : "Ć" U0106 # LATIN CAPITAL LETTER C WITH ACUTE
<Multi_key> <acute> <C> : "Ć" U0106 # LATIN CAPITAL LETTER C WITH ACUTE
<Multi_key> <apostrophe> <C> : "Ć" U0106 # LATIN CAPITAL LETTER C WITH ACUTE
<Multi_key> <C> <apostrophe> : "Ć" U0106 # LATIN CAPITAL LETTER C WITH ACUTE
-<dead_acute> <c> : "ć" U0107 # LATIN SMALL LETTER C WITH ACUTE
<Multi_key> <acute> <c> : "ć" U0107 # LATIN SMALL LETTER C WITH ACUTE
<Multi_key> <apostrophe> <c> : "ć" U0107 # LATIN SMALL LETTER C WITH ACUTE
<Multi_key> <c> <apostrophe> : "ć" U0107 # LATIN SMALL LETTER C WITH ACUTE

Weirdly, my current /usr/share/X11/locale/en_US.UTF-8/Compose has no difference to my .XCompose, so I am not sure which one was the modification that made the trick.

Keyboard

The keyboard layout English (intl., with dead keys) works as expected in (almost) all apps. ~ + C gives me ç as ' + e gives me é. Except PhpStorm. I can’t successfully type " ' ~ or the backtick within the editor. Just nothing gets typed. The PhpStorm Keymap settings can recognize when I type one of those keys, but the editor itself or other menus wouldn’t. I tried several different things, related with IBus, changing XMODIFIERS, /etc/environment, ~/.Xmodmap and ~/.profile, saw some bug reports on JetBrains tracker, but nothing really worked.

My current workaround is to have English (intl, with AltGr dead keys) as a second keyboard layout that I use while I am on PhpStorm. Since I only write code (and comments) in English, it won’t be much of a problem, but I am having occasional mindfucks because I’ll keep forgetting to switch between keyboard layouts when I switch applications.

SOCKS proxy with SSH

2018-06-25T00:00:00Z

That’s not a new thing, but I happened to use it during the weekend to be able to access some services back in Brazil that were IP-limited and HideMyAss couldn’t help, so I asked a friend for a small proxy help.

What I did on my side:

Opened a port on my modem to forward the connection to the port 51000 on my computer;
Started a container: docker run -p 51000:51000 -p 51001:51001 --rm -it ubuntu:xenial bash;
Installed supervisord, openssh-server and added GatewayPorts yes to /etc/ssh/sshd_config;

My friend had to run those commands in parallel:

ssh -D 51001 localhost;
ssh -R :51001:localhost:51001 use@myIP -p 51000;

And voilá, I had a SOCKS proxy over localhost:51000 that was going out to the internet using a Brazilian IP address.

Profiling browser requests with Blackfire

2017-06-16T00:00:00Z

This week’s task was optimising Magento for large carts (100+ different products) and a profiler is a good tool to find small pieces of code that could be optimised. Blackfire was my choice, mostly because previous experience, but also because it is not a resource hog like Xdebug, which was taking around 6 minutes while Blackfire took only 20 seconds in the same type of request.

Companion

Blackfire works great to profile GET requests on the browser with the help of Companion, but it doesn’t allow you to profile POST requests. Also, Companion sends multiple requests to the same endpoint, but Magento’s place order is not stateless, so the first and all other request would differ substantially.

CLI tool

Second attempt was the Blackfire CLI tool, which allows you to profile anything you can reach with curl, including POST requests and custom headers. But, you can only profile one request at the given time, so that means I need to assemble one HTTP call that does everything: login customer, add products to cart, collect totals, get shipment information, get payment information, save shipping, save payment, and finally place order. That way I am profiling a lot more than I need, not to mention that everything will be in the same requests and not have the same performance as the end customer will experience, since a lot of objects will have been cached in memory when the place order part is executed. So I ruled that out too.

Xdebug

Xdebug does exactly what I want, but takes too much time, being almost impractical. You can enable the profiling for any HTTP request, so that wouldn’t be a problem, I just use my browser and to navigate through all the needed steps (I can even do that with Xdebug disabled and enable only for the last step) and then analyse the cachegrind files. But Xdebug uses too much resources, I was hitting a 5-minute timeout on FPM too easily and the results were contaminated by the slowness (everything was slower in a factor of ~5 compared to the latest Blackfire result).

One nice tip here: you can use blackfire upload to send the Xdebug output files to Blackfire and see the analysis there, I am pretty amazed by that feature. Way to go SensioLabs!

Custom headers

With a bit of back and forth with SensioLabs support I was able to achieve what I wanted, even if requires a bit more work.

Requirements

Extension for your browser to modify the request headers (I used ModHeader for Google Chrome);
Blackfire CLI tool;
Blackfire configured on the webserver;

Steps

Do all the preparation needed. I added all products to the cart, navigated to checkout page and filled all the information, only leaving out the ‘click on place order button’ step;
Run blackfire run env and copy the value of the BLACKFIRE_QUERY outputted variable;

Using the browser extension, add the following headers to your request (example):

X-Blackfire-User-Agent: Blackfire Companion - Chrome/58.0.3029.110 Extension/1.10.0
X-Blackfire-Query: [the value copied on #2]

Execute your request (click the place order button).
Profit.

With those steps, the first AJAX/POST request (that reaches PHP) with those headers will be profiled. It only works once, so it will not profile the success page (which is the one the request I profiled redirects to). Every new request needs a new X-Blackfire-Query header, so with this approach is not (yet) possible to profile the redirected page as well or other subsequent calls inside the same page.

If the request you want to profile is not the first one that you are able to fire, instead of using the extension on the browser, change your Javascript code to include those headers only for the request you need (or keep the extension but change the Javascript to remove them).

What I have not tried

There are two other ways that I could profile what I wanted with Blackfire, but I haven’t tested:

Use the PHP SDK and start/stop the profiling on the areas that you need. It sounds like is something that will work, but that means you need to do some changes on code. I would only go that route if I hadn’t found any alternative.
Using the scenario I described on the CLI tool, but using the technic presented on Profiling HTTP Sub-Requests using Blackfire. To achieve I would had to assemble all the HTTP calls as the same way as Magento would have done, and send the X-Blackfire-Query header with them. This will profile all the calls individually (thus not having the in-memory cache problem), but it also requires a bit of coding, or preparing all the HTTP requests manually to some degree.

Conclusion

Blackfire is a nice tool and I really like the help their support provided here. I wish they cover this edge case in their roadmap, but after this discovery I am definitely not turning it down, I am very happy with the results. Please consider giving Blackfire a try if you need it.

netcat replacement Ncat time unit

2014-07-21T00:00:00Z

Hi,

It’s common to use netcat utility to work with SSH ProxyCommand, which allows to use a bridge server, very useful when you need to connect directly to a host behind a firewall. Example:

# File: ~/.ssh/config
Host workbox
HostName 192.168.1.92 # The ip address that the bridge server can see
User anotherusername
IdentityFile ~/.ssh/id_rsa
ProxyCommand ssh myusername@firewall.company.com nc -w 120 %h %p

I was using netcat until a couple of days, and works very smooth. Then I’ve setup a new server on another private network, behind a CentOS 7 firewall, and when I’ve tried the configuration above, I got this:

Ncat: Since April 2010, the default unit for -w is seconds, so your time of "120" is 2.0 minutes. Use "120ms" for 120 milliseconds. QUITTING.

I didn’t know, but I’m using Ncat instead of the old Netcat. I got not time to learn the differences, but I already found one. Every parameter that receives a time needs the unit. From ncat --help:

$ nc --help
Ncat 6.40 ( http://nmap.org/ncat )
Usage: ncat [options] [hostname] [port]

Options taking a time assume seconds. Append 'ms' for milliseconds,
's' for seconds, 'm' for minutes, or 'h' for hours (e.g. 500ms).

Then, in my SSH configuration I had to change the -w param to add the time unit (ms, s, m, h, etc):

 ProxyCommand ssh myusername@firewall.company.com nc -w 120m %h %p

Bye.

Magento: frontend and admin routes conflict

2014-06-06T00:00:00Z

Today I was working in a 3rd party module. Something very simple, an AJAX Add to Cart button. But, when I requested http://.../ajaxcart/cart/add, I’ve got a 302 HTTP status, or, a redirect, to the same requested URL, but in secure mode (HTTPS).

After debugging, I could understand that the controller wasn’t being called, so I try to find a configuration problem. The Administration Panel was configured to be served on HTTPS (Configuration > General > Web > Secure > ???), and when I’ve removed this option, everything was working as expected.

Long story short, here is the problem:

<frontend>
 <routers>
 <ajaxcart>
 <use>standard</use>
 <args>
 <module>Company_AjaxCart</module>
 <frontname>ajaxcart</frontname>
 </args>
 </routename>
 </routers>
</frontend>
<admin>
 <routers>
 <ajaxcart>
 <use>admin</use>
 <args>
 <module>Company_AjaxCart</module>
 <frontname>ajaxcart</frontname>
 </args>
 </routename>
 </routers>
</admin>

Did you see that frontname is the same both for frontend and admin? In this way, every time we call a URL with the pattern http://.../ajaxcart/*, Magento replies you to redirect to secure mode. When using AJAX, we have to main problems with this:

Developers don’t often worry about HTTP Status codes that aren’t 200;
You have to configure the Same-Origin Policy, or you’ll get CORS errors like this:

Cross-Origin Request Blocked: The Same Origin Policy disallows reading the remote resource at https://.../ajaxcart/cart/add/. This can be fixed by moving the resource to the same domain or enabling CORS.

TL;DR: On every module you develop for Magento which needs controllers in admin and frontend, pay attention and don’t use the same frontname.

Bye.

Nginx: regex on server_name

2014-05-23T00:00:00Z

Quick tip to use regex on nginx’s server_name directive.

Using a specific root directory for every subdomain:

server {
 listen 80;
 server_name ~^(.*)\.project\.com$;
 root /home/www/project/$1;
}

Using an environment variable with a single project:

server {
 listen 80;
 server_name ~^(.*)\.project\.com$;
 fastcgi_param CUSTOMER $1;
 root /home/www/project;
}

You can also use the entire domain name with this:

server {
 listen 80;
 server_name ~^(.*)\.(.*\..*)$;
 root /home/www/$2/subdomains/$1/public_html;
}

Tested with Nginx 1.4.x.

Debian: cannot connect to X server

2014-02-07T00:00:00Z

Quick tip on KDE based Debian:

When you receive the following error when trying to run a X application with sudo:

kassner@brian:~$ sudo unetbootin
No protocol specified
unetbootin: cannot connect to X server :0

Run the following command and try again:

kassner@brian:~$ xhost SI:localuser:root
localuser:root being added to access control list

Twig_Error_Syntax: A message inside a trans tag must be a simple text

2014-01-15T00:00:00Z

Quick tip:

When have a code like this:

{% raw %}{% trans %}prefix.{{ varname }}{% endtrans %}{% endraw %}

And receive this error:

Twig_Error_Syntax: A message inside a trans tag must be a simple text

You can use the following piece of code as workaround.

{% raw %}{{ ("prefix." ~ varname)|trans }}{% endraw %}

PS: maybe this is not the best way to go.

Symfony 2: Inheritance and UniqueEntity workaround

2013-11-27T00:00:00Z

Hi folks,

Today I struggled over an already known bug on Symfony2 when using UniqueEntity and Entity Inheritance. In the bug discussion, @gentisaliu recommended using a custom repository, and how do this I’m documenting here.

Entities:

/**
 * @ORM\Table(name="parent")
 * @ORM\Entity(repositoryClass="Repository\Parent")
 * @UniqueEntity(fields={"name"}, repositoryMethod="findByName", message="Name already used.")
 * @ORM\InheritanceType("JOINED")
 * @ORM\DiscriminatorColumn(name="type", type="string")
 * @ORM\DiscriminatorMap({
 * "a" = "ChildA",
 * "b" = "ChildB"
 * })
 */
class Parent { }

/**
 * @ORM\Entity(repositoryClass="Repository\Parent")
 * @ORM\Table(name="child_a")
 */
class ChildA extends Parent { }

/**
 * @ORM\Entity(repositoryClass="Repository\Parent")
 * @ORM\Table(name="child_b")
 */
class ChildB extends Parent { }

repositoryMethod:

public function findByName($criteria)
{
 return $this
 ->getEntityManager()
 ->createQuery("SELECT e FROM Parent e WHERE e.name = :name")
 ->setParameters($criteria)
 ->getResult()
 ;
}

The findByName method must explicitly query the main entity, otherwise you will check a the uniqueness only for that type (name = ? AND type = ?)
The repositoryMethod defines which method in the repository will be used for determine if an entity is unique or not. You have to return an Countable object, array allowed, but I rather use a query result.
The repositoryClass must be the same of all entities, or if you can’t use the same repository, implement your “repositoryMethod” in all repositories. In such case you have to be cautious to do the query the base entity, not the child one.

That’s all.

Debian: ping unknown host but DNS works

2013-11-14T00:00:00Z

Hello.

Today I bumped into a problem with ping:

kassner@brian$ ping git.company.local
ping: unknown host git.company.local

Of course, it’s a local address, so maybe I forgot to add the local DNS server. Let’s check:

kassner@brian$ dig A git.company.local

; < <>> DiG 9.8.4-rpz2+rl005.12-P1 < <>> A git.company.local
;; global options: +cmd
;; Got answer:
;; ->>HEADER< <- opcode: QUERY, status: NOERROR, id: 15746
;; flags: qr aa rd ra; QUERY: 1, ANSWER: 1, AUTHORITY: 4, ADDITIONAL: 4

;; QUESTION SECTION:
;git.company.local. IN A

;; ANSWER SECTION:
git.company.local. 86400 IN A 192.168.0.150

;; Query time: 0 msec
;; SERVER: 10.0.0.1#53(10.0.0.1)
;; WHEN: Thu Nov 14 12:05:45 2013
;; MSG SIZE rcvd: 249

WTF? Oh, of course. I’m using a .local suffix, so Avahi will take action. Needless for my local network, I just disabled it on Debian Wheezy:

kassner@brian$ sudo update-rc.d -f avahi-daemon remove

This should disable Avahi temporarely (until reboot). But, if you need to get rid of Avahi, just uninstall:

kassner@brian$ sudo apt-get remove avahi-daemon

Bye.

Magento Product object and loadByAttribute

2012-05-28T00:00:00Z

Hi Folks,

A little note to work with product object in Magento:

/**
 * Using setStoreId two times, otherwise it will save the data to default store
 */
$product = Mage::getModel('catalog/product')
 ->setStoreId($storeId)
 ->loadByAttribute('ean', $ean)
 ->setStoreId($storeId);

Apache Rewrite on VirtualHost = 400 Bad Request Error

2012-03-07T00:00:00Z

Mental note:

If you are moving the Rewrite from .htaccess to VirtualHost configuration and get a 400 Bad Request error, one or both tips below can be useful:

RewriteEngine On
# Use the %{DOCUMENT_ROOT}
RewriteCond %{DOCUMENT_ROOT}%{REQUEST_FILENAME} -s [OR]
RewriteCond %{DOCUMENT_ROOT}%{REQUEST_FILENAME} -l [OR]
RewriteCond %{DOCUMENT_ROOT}%{REQUEST_FILENAME} -d
RewriteRule ^.*$ - [NC,L]
# Use the absolute path
RewriteRule ^.*$ /home/www/html/index.php [NC,L]

That’s all.

Converting movies with tovid

2012-02-09T00:00:00Z

Hello

A little tip to convert AVI files to DVD with embed subtitles.

Installing tovid:

sudo apt-get install tovid

Converting AVI file to a DVD-ISO file.

tovid -dvd -in Video.avi -subtitles Legenda.srt -out Video
/usr/share/tovid/makexml Video.mpg -out Video
export VIDEO_FORMAT=NTSC
/usr/share/tovid/makedvd Video.xml
mkisofs -dvd-video -udf -R -o Video.iso Video/

So, you just need to burn your ISO file. You can use Brasero.

Magento slow

2011-01-07T00:00:00Z

Hello.

After an intense debug into Magento, we (Filipe Ibaldo and me) have found two problems in Magento code, related to slow Place Order with many products in cart (not many qty of a product).

One of the problems is the order item save, which is executed precisely in app/code/core/Mage/Sales/Model/Entity/Order/Attribute/Backend/Parent.php, on afterSave method, who spent many time on my machine (Core 2 Duo 2.26/4GB RAM), about 0.3 second for each product.

Considering the number of items average of customer’s cart is about 50 distinct products, this time, adding to another code executions and a high server load, is easily more than one minute, and this is unacceptable.

The found workaround is disable some observers from Mage_Downloadable and Mage_Rss, who execute in afterSave of each item in order.

In app/code/core/Mage/Downloadable/etc/config.xml, remove/comment the following lines:

<sales_order_item_save_commit_after>
 <observers>
 <downloadable_observer>
 <class>downloadable/observer</class>
 <method>saveDownloadableOrderItem</method>
 </downloadable_observer>
 </observers>
</sales_order_item_save_commit_after>

The code above appears twice.

In app/code/core/Mage/Rss/etc/config.xml, remove/comment the following lines:

<sales_order_save_after>
 <observers>
 <notifystock>
 <class>rss/observer</class>
 <method>salesOrderItemSaveAfterNotifyStock</method>
 </notifystock>
 </observers>
</sales_order_save_after>
<sales_order_save_after>
 <observers>
 <ordernew>
 <class>rss/observer</class>
 <method>salesOrderItemSaveAfterOrderNew</method>
 </ordernew>
 </observers>
</sales_order_save_after>

With this workaround, the save time of each product went from 0.3 second to 0.001 second, an incredible speed improvement.

However, if you are using downloadable products or Magento Inventory Management, this modifications can broke related features, so, be careful.

Another problem found is related with module disable feature into System > Configuration > Advanced > Advanced > Disable Module Output. If you disable some module, their observers did not disabled, and you need to remove their configuration or delete module folder.

Bye.

Include error on Zend_Loader

2010-11-01T00:00:00Z

Hello.

Today I found a problem in Zend Framework and his class autoloader, where an class_exists function call fires the Zend Framework autoloader. The function has an option to ignore the autoloader, but any class that I need loaded by Zend_Loader will return false.

Then, if I call class_exists without disable autoloader, I got an file not exists PHP warning, that’s why Zend_Loader doesn’t check if the file exists, just include.

The trick is change Zend/Loader.php to check if file exists before include, and this must be wrote in loadFile method.

public static function loadFile($filename, $dirs = null, $once = false)
{
 self::_securityCheck($filename);

 /**
 * Search in provided directories, as well as include_path
 */
 $incPath = false;
 if (!empty($dirs) && (is_array($dirs) || is_string($dirs))) {
 if (is_array($dirs)) {
 $dirs = implode(PATH_SEPARATOR, $dirs);
 }
 $incPath = get_include_path();
 set_include_path($dirs . PATH_SEPARATOR . $incPath);
 }

 /**
 * Try finding for the plain filename in the include_path.
 */
 if (!self::fileExists($filename)) {
 return false;
 }

 if ($once) {
 include_once $filename;
 } else {
 include $filename;
 }

 /**
 * If searching in directories, reset include_path
 */
 if ($incPath) {
 set_include_path($incPath);
 }

 return true;
}

public static function fileExists ($filename) {
 $paths = explode(PATH_SEPARATOR, get_include_path());
 foreach ($paths as $path) {
 if (file_exists($path . DIRECTORY_SEPARATOR . $filename)) {
 return true;
 }
 }
 return false;
}

Bye.