unrelenting.technology

So I was reading the OpenBSD Chromebook Pixel post: it did not wake up from suspend without the TPM driver. Hmm.

Disabled TPM on my Thinkpad X240 and FreeBSD RESUMES FROM SUSPEND!!! Wow. I guess I’ll stop using the TPM then (I was using it for SSH keys. And I did unload the tpm kernel module before suspend!)

Managed to get [Servo to run with Capsicum sandboxing on FreeBSD](https://github.com/servo/servo/issues/11625#issuecomment-342243384)!

FreeBSDDesktop

This infodump tries to be an up-to-date place for information regarding FreeBSD Wayland desktop setup. It’s not a step-by-step guide but if you know the basics you should be able to figure everything out.

Base

My Base Fork: https://github.com/myfreeweb/freebsd

You probably don’t need all the things from there, stock -CURRENT is good for most people.

Ports

My Ports: https://github.com/myfreeweb/freebsd-ports-dank

Some port maintainers (hello x11@ gnome@) are kinda slow sometimes :) so I maintain a patchset for the ports repo that includes new shiny things. It might eventually become a more proper “distro”/fork maybe someday.

The most important things to get from my ports are:

GPU Kernel Drivers

The git repo’s drm-v5.0 branch is current. I don’t install from ports because I modify the code sometimes.

  • YOU HAVE TO REBUILD AND REINSTALL EXTERNAL KERNEL MODULES LIKE THESE AFTER INSTALLING A NEW KERNEL! the behavior is UNDEFINED if you don’t, it might not break so hard that it doesn’t load, but it is VERY likely to break in SOME way!
  • The first two Radeon GCN generations are supported by both amdgpu and radeonkms. Use amdgpu! It’s the modern driver, it supports everything needed for Vulkan and it doesn’t have weird crashes radeonkms sometimes can have.
  • AMD Radeon (amdgpu) + UEFI very often has a framebuffer conflict problem, for a workaround disable efifb with hw.syscons.disable=1 in /boot/loader.conf and enjoy no display output until driver loads.
  • Intel Kaby Lake doesn’t refresh the vt sometimes, starting and quitting a GUI temporarily fixes it, or use compat.linuxkpi.i915_fastboot which might break brightness adjustment though.
  • amdgpu works on aarch64 machines btw

GPU tunables/sysctls

Make sure the tunables you set actually exist! Sometimes you’ll find their old names in various blogs etc. — translate to the current format:

  • Back in the day, driver tunables were drm.i915.TUNABLENAME or something.
  • For a brief period, they were compat.linuxkpi.TUNABLENAME (e.g. compat.linuxkpi.ppfeaturemask).
  • Currently, they are compat.linuxkpi.DRIVERNAME_TUNABLENAME (e.g. compat.linuxkpi.amdgpu_ppfeaturemask).

Other than driver tunables, there are tunables corresponding to Linux sysfs’s “classes”. These are under sys.class.

Anyway: DO NOT cargo-cult tunables from the internet :) Most of them are good by default. I pretty much only use these:

compat.linuxkpi.i915_enable_dc="7" # deepest power states
compat.linuxkpi.i915_enable_dpcd_backlight="1" # better brightness control on some modern laptops
# (the Pixelbook requires this and also some patching for backlight to work)
compat.linuxkpi.amdgpu_ppfeaturemask="4294803455" # default | 0x4000 for Overdrive overclocking

GPU Userspace Drivers (Mesa)

The mesa-libs/mesa-dri/clover/osmesa stuff in upstream ports is not a complete disaster (at least Wayland and Vulkan stuff is enabled out of the box now, though not VA-API it seems) but it’s still a bit outdated.

mesa-dev in my ports builds a recent(ish) git version with Meson and with all components together.

To pacify pkg, there’s dummies for mesa-libs etc.

Input Device Support

The modern way of doing input is evdev. Forget moused, xf86-input-<ANYTHING that's not libinput>, etc. (tbh forget xorg also :D)

EVDEV_SUPPORT kernel option (to expose most things like psm as evdev) was enabled by default in rS340387 (Nov 2018).

It’s REALLY IMPORTANT to set sysctl kern.evdev.rcpt_mask="12" — send evdev events from individual mice/keyboards instead of sysmouse/kbdmux.

For PS/2 Synaptics touchpads, also set loader tunable hw.psm.synaptics_support="1" and hw.psm.trackpoint_support="1" for ThinkPad TrackPoints. (The TrackPoint will appear as an extra device in evdev!)

For HID over I²C install the wulf7/iichid driver. (well, it’s a next gen input stack really, that abstracts HID from USB. I contributed pen tablet support there!) (upd: It’s in ports now)

For Wacom tablets, gamepads, and other USB devices that Linux supports, you can use webcamd, which just runs the Linux drivers in userspace (and uses cuse to provide them to the kernel) which is pretty cool.

Cool evdev tools

  • evemu provides lower-level (than libinput’s CLI) debug view, record and replay, etc
  • evhz measures event rate
  • netevent forwards devices over the network (like Synergy but without fancy seamless mouse handover, you have to press a hotkey to switch to remote)
  • evscript I made this a while ago: a scripting thing to make xcape style key tricks happen. I don’t really use it anymore because I have a Wayfire plugin for that

Session Management

A session manager is a daemon that tracks user sessions (huh). This was originally invented for GUI user switching, but now it has a more fundamental task: letting you run your GUI session as your user account without needing root (this works by passing file descriptors to GPU and input devices over D-Bus). It also configures your $XDG_RUNTIME_DIR directory.

On Linux, systemd-logind or elogind is used for this. We only have the good old ConsoleKit2 for now. (I’d like to work on a new thing in Rust but I have so many projects already)

  • included in my ports (also at D18754) is a patch for CK2 that enables FreeBSD+Wayland support, you need that;
  • for wlroots-based compositors (display servers) like sway and wayfire, you need a CK2 support patch, it’s actually available in the wlroots port as the CONSOLEKIT option;
  • some shenanigans related to our udev implementation not really being udev:

    • make sure the kernel exposes kern.evdev.input sysctls, if it doesn’t then it’s too old;
    • make sure libudev-devd is at least v0.4.1, i.e. it must include 8efdba3c;
  • to enable automatic CK2 session creation on login, add session optional pam_ck_connector.so to the # session section of /etc/pam.d/login and log out and back in;
  • you can always use ck-launch-session to start the compositor if the automatic PAM thing doesn’t work out.

Compositors / Display Servers

wlroots based compositors are the best supported (and these ones are in ports):

  • Wayfire is a general extensible compositor with various Compiz features (wobbly windows, desktop cube) and stuff, I use it daily, it’s awesome;
  • sway is an i3 (tiling WM) clone;
  • hikari is inspired by cwm;
  • cage is a kiosk (runs a single, maximized application) — like all wlr compositors it can be used in a window, so there’s some other usages for that;

If you have done the aforementioned CK2 setup, they should just work as your regular user.

My abandoned Weston fork is a thing also if you’re interested for some reason.

KDE Plasma 5 should be possible (already supported CK2!), here’s a patch attempt from some time ago, it got to the lock screen at least for me.

Power Management

Frequency Management

Most CPUs: run powerd++ (powerdxx in ports).

Current Intel CPUs (>=Skylake): the Intel Speed Shift patch which lets the CPU handle it has landed! If you have speed shift mentioned in your dmesg, your CPU is already adjusting everything dynamically.

C-states

This is the most important thing for not eating power when idle.

Set sysctl hw.acpi.cpu.cx_lowest as low as it goes e.g. "C8". (Or economy_cx_lowest="Cmax" and performance_cx_lowest="Cmax" in /etc/rc.conf as a shortcut for that if not using runit or any other alternative init.)

On Intel CPUs, use pcm.x 1 (from intel-pcm) to check C-state residency and energy consumption. You want to be in C7 (both core and package) (or deeper if some chip does that) as much as possible when not doing anything.

Use vmstat -i to check interrupt rates. Some misbehaving devices (e.g. inactive USB Bluetooth on Intel Wi-Fi cards) can spew a lot of interrupts and waste power, disable them by running e.g. usbconfig -d 0.3 power_off.

More

There’s more power tips here (e.g. for SATA disks) but don’t copy all the GPU tunables :)

Various stuff

Extra /boot/loader.conf tweaks

vfs.zfs.arc_max="4G" # limit ZFS ARC size — generally it yields memory to userspace programs, but ehh
hw.nvme.use_nvd="0" # use CAM-based NVMe disk driver (default on powerpc64 and aarch64 already)
hw.usb.no_boot_wait="1" # don't waste time on probing USB at early boot time (don't enable when booting from USB)
kern.vt.splash_cpu="1" # remember penguins on console? that's orbs on console
kern.random.use_chacha20_cipher="1" # cryptography hipster stuff (gonna be default soon)
kern.msgbufsize="512000" # more space for dmesg
kern.hwpmc.nsamples="8192" # more space for pmcstat
kern.hwpmc.nbuffers_pcpu="128" # more space for pmcstat

devd event rules

Restart devd to activate new rules!!

USB phone connection (vendor 0x18d1 is Google Nexus/Pixel) for running ADB, MTP fusefs-simple-mtpfs etc. as regular user:

notify 100 {
        match "system"          "USB";
        match "subsystem"       "DEVICE";
        match "type"            "ATTACH";
        match "vendor"          "0x18d1";
        action  "chgrp wheel /dev/$cdev; chmod g+rw /dev/$cdev";
};

U2F token: /usr/local/etc/devd/u2f.conf.sample was installed by package libu2f-host-1.1.4

Asetek liquid cooler for control scripts like leviathan, krakenx etc. (all the Python scripts “for Linux” that use Python USB libraries work on FreeBSD):

notify 100 {
        match "system"          "USB";
        match "subsystem"       "DEVICE";
        match "type"            "ATTACH";
        match "vendor"          "0x2433";
        action  "chgrp wheel /dev/$cdev; chmod g+rw /dev/$cdev";
};

Extra links/notes

AMD Ryzen CPUs: load amdtemp to read temperature (sysctl dev.cpu.0.temperature), BIOS overclocking works fine even though sysctl might show freq 2700 #218262, ignore that

USB keyboard/mouse multimedia keys: #222646

MSI RGB lighting: msi-rgb

Bluetooth audio, fake microphone from audio files, etc.: audio/virtual_oss.

Adding FreeBSD 12 support to Crystal was super easy! Unfortunately, can’t say the same about Rust.

Adventures in printing:

The proprietary blob plugin for HPLIP is ported to FreeBSD by literally saying “this Linux library is actually for FreeBSD” and adding a tiny implementation of a couple glibc functions, amazing! But HPLIP is not necessary for my printer, foo2zjs is an open source driver that supports it.

PostScript is not PostScript, apparently. (Actually my printer wants PDF, it seems — setting generic PDF on the client side when network printing over CUPS worked.)

And if CUPS doesn’t see the printer when using the open source drivers, it IS a permissions issue, make sure to restart devd to activate the rule that’s included with the cups package. (The fact that HPLIP sees the printer is… odd. Was it running HPLIP stuff as root?)

Moving VMs from VirtualBox to Client Hyper-V

I’ve decided to move the VMs on my desktop from VirtualBox to Microsoft Hyper-V. Because reasons.

Actually because I’ve upgraded my desktop to an AMD Ryzen CPU: first, AMD-V/SVM is not supported by the Intel HAXM thing from the Android SDK, so I wanted to try out Microsoft’s Hyper-V based Android “emulator” (VM configurator/runner thingy) instead. Second, giving 16 virtual CPUs on an SMT 8-core to a FreeBSD guest in VirtualBox results in a weird performance issue. (Though giving 4 vCPUs to multiple VMs on a 4-core CPU worked fine.) Third, it’s Oracle VM VirtualBox and no one likes Oracle.

So, here’s how you can do it as well.

How to get Hyper-V

You need Windows 10 Pro, Enterprise or Education. (Or Windows Server, obviously.) Just enable it as a feature and restart.

Alternatively, installing the MS Android “emulator” automatically enables it.

How to migrate a VM (FreeBSD, Linux or Windows with EFI)

(NOTE: older versions of FreeBSD apparently had some loader issue that prevented EFI boot in Hyper-V. Everything works for me on a recent build of 11-STABLE.)

In VirtualBox, go to the Virtual Media Manager (Ctrl+D) and copy your disk as VHD. In the Hyper-V Manager, use the Edit Disk dialog to convert the VHD to VHDX.

If you haven’t done that yet, go to the Virtual Switch Manager and make a virtual switch (“External” is like bridge mode in VBox).

Now make a virtual machine. Generation 2, no dynamic memory (FreeBSD doesn’t support that), select the virtual switch and the VHDX disk.

Click Connect and it should just work.

By the way, it’s nice that you can always close the console window without powering off the VM, unlike in VirtualBox where you need a special “Detachable start”.

Interestingly, if you create the VM without a disk and attach the disk later, you won’t see “boot from hard drive” in the firmware / boot order settings. And there’s no add button! (WTF?) The fix is to use PowerShell:

$vm = Get-VM "YOUR VM NAME"
Set-VMFirmware $vm -FirstBootDevice (Get-VMHardDiskDrive $vm)

Speaking of which, it’s nice to have a directly integrated PowerShell interface to all the things. My little xvmmgr script was initially written for VirtualBox, and that required COM.

How to migrate a VM (other OS)

Well, a similar process, but use Generation 1.

My experience so far

Client Hyper-V has pleasantly surprised me. It’s a very smooth experience: it looks like a Type 2 hypervisor even though it’s actually Type 1, it runs VMs without any performance issues… what else could you ask for?

Well, the downside is its lack of flexibility in terms of paravirtualized (MS calls them “synthetic” or something) vs emulated devices.

All you get is the choice between two generations. Generation 1 means legacy BIOS boot from an emulated IDE drive with emulated all the things plus optionally some paravirtualized devices like the NIC. Generation 2 means EFI boot from a SCSI drive with paravirtualized everything. Oh and the SCSI controller is also on the vmbus. So there’s no way to use EFI and SCSI with e.g. OpenBSD, you need full Hyper-V support for at least the disk and network to do that. Thankfully Microsoft contributed that support to FreeBSD! :)

evdev support in the psm driver (with proper support for Synaptics touchpads and Thinkpad Trackpoints) landed in FreeBSD -CURRENT! Just set up libinput in Xorg for all devices, very nice. (Without device autodetection — with devd, Xorg wants to use the old keyboard and mouse drivers. Using the udev backend would work, but freebsd-ports-graphics is currently meh.)

Also people are working on suspend-resume issues!

Posted a patch to FreeBSD that makes it recognize Alpine Linux's binaries as, well, Linux binaries. Because running brandelf every time you install something inside a Linux container is bad.

Quake Champions is awesome (as in the gameplay — performance is meh).

Amazon Web Services is not awesome: it wasn't really obvious that promotional credits aren't spent on reserved EC2 instances :( Also HardenedBSD was behaving weird on it (secadm kernel panic, Python libssl segfaults).

But with regular FreeBSD I've set up a Matrix homeserver (Synapse) on EC2! I am now @greg:unrelenting.technology :) It's working as my new IRC bouncer, so with that I've been able to say goodbye to the previous VPS that served this website (which was still running my ZNC).

TIL: Intel chipsets have a hardware watchdog. Just like a Raspberry Pi.

Speaking of the Raspberry Pi watchdog (bcmwd): it supports 15 seconds max interval. But if you try to run FreeBSD’s watchdogd with 15 seconds, it’ll still complain. Look in the output — for some reason watchdogd rounds it up to 17 or something. So the max interval is 9.