Screenshot of htop with 96 CPU cores

For 0.5 dollars per hour (or currently 0.1/hr if you reserve for 24h?) on packet.net you can get access to a dedicated dual-socket Cavium ThunderX server with 128GB RAM and a 250GB SSD. I took it for a few hours and now lang/crystal, lang/mono and some other ports work on aarch64.

Ironically, these two builds have involved long single-threaded compile processes. In the mono case, parallelism had to be disabled for C# compilation to work around a concurrency bug.

At least building things like WebKitGTK+ (to test a one line patch) and Krita felt awesome :D

BREAKING FreeBSD NEWS: AMDGPU DC works!

The 4.15 drm update apparently did not have amdgpu working yet... but I tried it anyway, since that's just an assertion failure.

Well... it does work. Even with DC (compat.linuxkpi.dc="1")! Weston didn't work at first, but commenting out its "Disable all the planes" section that was causing a kernel warning solved the problem.

So quite possibly we have Vega support?

Booted into FreeBSD, launched Epiphany (GNOME Web), noticed slowness. Opened htop: 800-1200% load in kernel! Ran dtrace to find hot kernel stacks: it's all ZFS write threads, trying to compress. WTF? I mean, my fault for choosing gzip (I have a tiny SSD for this system, want max compression), but I expect zero I/O right now?! Found a DTrace script to show file I/O.

Turns out Epiphany is constantly writing to and reading from ~/.config/epiphany/gsb-threats.db. Yeah, it was downloading the whole Google Safe Browsing database.

I guess that's the most privacy-friendly way to do Safe Browsing, but wow, that's a very surprising behavior. What if I had, like, a dial-up connection or something? :D

It is well known that Enlightenment/EFL is pretty awful. But I had to build it on my FreeBSD box since I'm testing a LuaJIT update and EFL does include LuaJIT support. So while I'm at it, I decided to add Wayland support to the port.

So at least it works, but its behavior is indeed kinda funny. Running their terminal with GL acceleration (ELM_DISPLAY=wl ELM_ACCEL=opengl terminology) actually does work. Running it without acceleration (ELM_DISPLAY=wl ELM_ACCEL=no terminology) results in the surface not showing up and… this message:

DRM_IOCTL_I915_GEM_APERTURE failed: Invalid argument
Assuming 131072kB available aperture size.
May lead to reduced performance or incorrect rendering.
get chip id failed: -1 [2]
param: 4, val: 0

It's no "SPANK! SPANK! SPANK!" but how exactly did it decide to use an Intel GPU related system call? On my machine that has a Radeon card?!?! When I asked for software rendering??

haha, Phoronix wrote about the stuff I posted on wayland-devel@. So yeah, I'm working on Rust bindings for libweston that would eventually allow me to write the best Wayland compositor ever :)

And fractional HiDPI scaling was pretty easy to add. Wayland apps look awesome. However, X11 apps are blurry now, and bypassing the scaling for Xwayland is not as easy… So I made some changes to my Ports fork to enable Wayland support in more apps. Turns out a lot of complex applications run fine — LibreOffice (!), Inkscape, MyPaint, RawTherapee, Darktable.

The Firefox Wayland support though… is not usable yet :( It looks awesome but EGL isn't working and, even worse, the screen doesn't refresh when it needs to — so you're typing and letters don't appear until you scroll or some time passes. Hopefully this will be fixed soon.

So there's no support in pretty much all Wayland compositors for fancy keyboard mapping utilities like xcape because no one wants a keylogging protocol extension. (Even an access-controlled one!! Why.)

Turns out it's better to just solve this on the evdev level. And I've done it in the coolest way possible: with a tiny sandboxed scripting environment. Meet evscript! It runs Dyon scripts in an environment with evdev, uinput, stdout and nothing else. xcape functionality is already provided in the "standard library" :)

FreeBSDDesktop

FreeBSD 12-CURRENT brings the Year of FreeBSD on the Desktop! Currently with some unofficial modifications and building some stuff from source, but you can get evdev, Wayland, Vulkan, all that stuff, with support for Intel Kaby Lake and AMD Polaris (Radeon RX 4xx/5xx) GPUs (and older).

base/ports

Build kernel with options EVDEV_SUPPORT and Wayland panic workaround (TODO: Fork)

Awesome cutting edge ports: https://github.com/myfreeweb/freebsd-ports-dank — build Xorg with UDEV to autodetect/hotplug evdev devices, Mesa with WAYLAND, Vulkan SDK, Firefox Beta (don't forget to tweak FirefoxSettings), etc.

Fresh GPU drivers: doas make -C/usr/ports/graphics/drm-next-kmod install clean (DO NOT go back to CSM boot for Radeons, just disable efifb with hw.syscons.disable=1)

Common configuration

sysctl

kern.evdev.rcpt_mask="12" — send evdev events from individual mice/keyboards instead of sysmouse/kbdmux

kern.vt.kbd_reboot="0" — do not reboot on Ctrl-Alt-Del

vfs.usermount="1" — let users mount filesystems

/boot/loader.conf

vfs.zfs.arc_max="2G" — limit ZFS ARC size — generally it yields memory to userspace programs, but doesn't seem to yield to other kernel things (network stack etc.), nice to always have a limit

hw.usb.no_boot_wait="1" — don't waste time on probing USB at early boot time (do not enable when booting from a USB disk, obviously)

Device permissions

doas sysrc devfs_system_ruleset="desktop", edit /etc/devfs.rules:

[desktop=10]
add path 'input/*' mode 0660 group video
add path 'tty*' mode 0660 group video

input and tty are reuqired to run Wayland compositors as user.

Device events (devd)

Restart devd to activate new rules!!

USB phone connection (vendor 0x18d1 is Google Nexus/Pixel) for ADB, MTP fusefs-simple-mtpfs:

notify 100 {
        match "system"          "USB";
        match "subsystem"       "DEVICE";
        match "type"            "ATTACH";
        match "vendor"          "0x18d1";
        action  "chgrp wheel /dev/$cdev; chmod g+rw /dev/$cdev";
};

U2F token: /usr/local/etc/devd/u2f.conf.sample was installed by package libu2f-host-1.1.4

Asetek liquid cooler for control scripts (see below):

notify 100 {
        match "system"          "USB";
        match "subsystem"       "DEVICE";
        match "type"            "ATTACH";
        match "vendor"          "0x2433";
        action  "chgrp wheel /dev/$cdev; chmod g+rw /dev/$cdev";
};

Various links, notes etc.

X11 and Wayland configs: here.

All laptops: powerd++ for better CPU power management.

Thinkpad: load acpi_ibm kernel module, DO NOT load acpi_video.

Asetek AIO liquid coolers: leviathan, krakenx etc. (all the Python scripts "for Linux" that use Python USB libraries work on FreeBSD)

AMD Ryzen CPUs: load amdtemp to read temperature (sysctl dev.cpu.0.temperature), BIOS overclocking works fine even though sysctl will show freq 2700 #218262, ignore that

USB keyboard/mouse multimedia keys: #222646

Bluetooth audio, fake microphone from audio files, etc.: audio/virtual_oss.

Adventures in printing:

The proprietary blob plugin for HPLIP is ported to FreeBSD by literally saying "this Linux library is actually for FreeBSD" and adding a tiny implementation of a couple glibc functions, amazing! But HPLIP is not necessary for my printer, foo2zjs is an open source driver that supports it.

PostScript is not PostScript, apparently. (Actually my printer wants PDF, it seems — setting generic PDF on the client side when network printing over CUPS worked.)

And if CUPS doesn't see the printer when using the open source drivers, it IS a permissions issue, make sure to restart devd to activate the rule that's included with the cups package. (The fact that HPLIP sees the printer is… odd. Was it running HPLIP stuff as root?)

Moving VMs from VirtualBox to Client Hyper-V

I've decided to move the VMs on my desktop from VirtualBox to Microsoft Hyper-V. Because reasons.

Actually because I've upgraded my desktop to an AMD Ryzen CPU: first, AMD-V/SVM is not supported by the Intel HAXM thing from the Android SDK, so I wanted to try out Microsoft's Hyper-V based Android "emulator" (VM configurator/runner thingy) instead. Second, giving 16 virtual CPUs on an SMT 8-core to a FreeBSD guest in VirtualBox results in a weird performance issue. (Though giving 4 vCPUs to multiple VMs on a 4-core CPU worked fine.) Third, it's Oracle VM VirtualBox and no one likes Oracle.

So, here's how you can do it as well.

How to get Hyper-V

You need Windows 10 Pro, Enterprise or Education. (Or Windows Server, obviously.) Just enable it as a feature and restart.

Alternatively, installing the MS Android "emulator" automatically enables it.

How to migrate a VM (FreeBSD, Linux or Windows with EFI)

(NOTE: older versions of FreeBSD apparently had some loader issue that prevented EFI boot in Hyper-V. Everything works for me on a recent build of 11-STABLE.)

In VirtualBox, go to the Virtual Media Manager (Ctrl+D) and copy your disk as VHD. In the Hyper-V Manager, use the Edit Disk dialog to convert the VHD to VHDX.

If you haven't done that yet, go to the Virtual Switch Manager and make a virtual switch ("External" is like bridge mode in VBox).

Now make a virtual machine. Generation 2, no dynamic memory (FreeBSD doesn't support that), select the virtual switch and the VHDX disk.

Click Connect and it should just work.

By the way, it's nice that you can always close the console window without powering off the VM, unlike in VirtualBox where you need a special "Detachable start".

Interestingly, if you create the VM without a disk and attach the disk later, you won't see "boot from hard drive" in the firmware / boot order settings. And there's no add button! (WTF?) The fix is to use PowerShell:

$vm = Get-VM "YOUR VM NAME"
Set-VMFirmware $vm -FirstBootDevice (Get-VMHardDiskDrive $vm)

Speaking of which, it's nice to have a directly integrated PowerShell interface to all the things. My little xvmmgr script was initially written for VirtualBox, and that required COM.

How to migrate a VM (other OS)

Well, a similar process, but use Generation 1.

My experience so far

Client Hyper-V has pleasantly surprised me. It's a very smooth experience: it looks like a Type 2 hypervisor even though it's actually Type 1, it runs VMs without any performance issues… what else could you ask for?

Well, the downside is its lack of flexibility in terms of paravirtualized (MS calls them "synthetic" or something) vs emulated devices.

All you get is the choice between two generations. Generation 1 means legacy BIOS boot from an emulated IDE drive with emulated all the things plus optionally some paravirtualized devices like the NIC. Generation 2 means EFI boot from a SCSI drive with paravirtualized everything. Oh and the SCSI controller is also on the vmbus. So there's no way to use EFI and SCSI with e.g. OpenBSD, you need full Hyper-V support for at least the disk and network to do that. Thankfully Microsoft contributed that support to FreeBSD! :)

Quake Champions is awesome (as in the gameplay — performance is meh).

Amazon Web Services is not awesome: it wasn't really obvious that promotional credits aren't spent on reserved EC2 instances :( Also HardenedBSD was behaving weird on it (secadm kernel panic, Python libssl segfaults).

But with regular FreeBSD I've set up a Matrix homeserver (Synapse) on EC2! I am now @greg:unrelenting.technology :) It's working as my new IRC bouncer, so with that I've been able to say goodbye to the previous VPS that served this website (which was still running my ZNC).