ChenHan's Tech Blog (Deprecated)

Deprecated. Logdown.

It's became harder to host a blog on logdown since several features have been broken. (Facebook login not working, url auto-generation not working)

This blog will be deprecated. Please go to https://swem.github.io for latest article.

How cloud-init systemd service is activated on Ubuntu 16.04 (xenial)

Recently I've trace how cloud-init is being run. First, I try to find service file in /etc/system/system or /lib/systemd/system. Although I can locate the unit files, I can't see any systemd target that will bring up these cloud-init services. I did't see how cloud-init.target is activated by systemd.

Finally, I found that cloud-init project utilize the systemd generator to generate the unit files dynamically. The dynamic unit files is located at /run/system/generator.early/multi-user.target.wants/cloud-init.target. With this unit file, when the multi-user.target is activated by systemd, the cloud-init.target will also be activated. Then the systemd will further activate related cloud-init service in /etc/systemd/system/cloud-init.target.wants/.

Below is the dynamic unit file.

$ ll /run/systemd/generator.early/multi-user.target.wants/cloud-init.target
lrwxrwxrwx 1 root root 37 Jul 14 07:52 /run/systemd/generator.early/multi-user.target.wants/cloud-init.target -> /lib/systemd/system/cloud-init.target

If I want to disable cloud-init, there are several ways to do it.

Mask the default cloud-init systemd generator.

ln -sf /dev/null /etc/systemd/system-generator/cloud-init-generator

Add kernel parameter cloud-init=disabled or touch /etc/cloud/cloud-init.disabled. This kernel parameter will be processed by cloud-init systemd generator. See http://bazaar.launchpad.net/~cloud-init-dev/cloud-init/trunk/view/head:/systemd/cloud-init.target

Ref: https://www.freedesktop.org/software/systemd/man/systemd.generator.html

use snapper to Travel back in time and compare (The ultimate snapshot tool for Linux)

Snapper is a great tool for making snapshot on linux. You can download it on ArchLinux/Debian/OpenSUSE/Ubuntu

Personally I use btrfs as underlying file system. Here are some basic command to snapshot my data.

# install snapper
sudo apt install -y snapper

# Suppose /dev/sda3 is formatted as btrfs file system.
# mount the file system
sudo mount /dev/sda3 /mnt

# create a btrfs subvolme "data"
sudo btrfs subvolume create /mnt/data
# unmount /dev/sda3
sudo umount /mnt

# mount only the subvolume "data", instead of whole file system
sudo mount -o subvol=data /dev/sda3 /data

# create the snapper config for "/data" folder. The config name could be anything, here I choose "data"
sudo snapper -c data create-config /data

# By default, the snapper will make a snapshot per hour, just like Mac OS's Time machine.
# Check the snapshot list
# sudo snapper -c data list 
Type   | #    | Pre # | Date                            | User | Cleanup  | Description | Userdata
-------+------+-------+---------------------------------+------+----------+-------------+---------
single | 1 |       | Tue 16 Aug 2016 09:17:02 AM CST | root | timeline | timeline    |         
single | 2 |       | Tue 16 Aug 2016 10:17:02 AM CST | root | timeline | timeline    |         
single | 3 |       | Tue 16 Aug 2016 11:17:03 AM CST | root | timeline | timeline    |         
single | 4 |       | Tue 16 Aug 2016 12:17:03 PM CST | root | timeline | timeline    |      

# In fact, you can find the underlying snapshot in "data" subvolume
sudo mount /dev/sda3 /mnt
ls -al /mnt/data/.snapshots/
drwxr-x--- 1 root root 230 Aug 16 12:17 ./
drwxr-xr-x 1 root root 340 Aug 15 10:47 ../
drwxr-xr-x 1 root root  32 Aug 16 09:17 1/
drwxr-xr-x 1 root root  32 Aug 16 10:17 2/
drwxr-xr-x 1 root root  32 Aug 16 11:17 3/
drwxr-xr-x 1 root root  32 Aug 16 12:17 4/

If your root file system is also a btrfs subvolume. That would be even better. You can snapshot your whole system!

sudo snapper -c root create-config /

LXD container Getting Started

Below I give some simple command to run a ubuntu:16.04 lxd container. LXD container is fast, very efficient, very low-footprint virtual machine. I recommend people who didn't need low-level system control (such as disk, loopback device) give it a try. I think this kind of lightweight virtual machine would benefit people who study machine learning or similar scientific computing.

# check current status of lxd. If you install lxd the first time, it should containing no virtual machines.
lxc list

# if my-ubuntu virtual machine not exist yet.
lxc launch ubuntu:16.04 my-ubuntu
# if my-ubuntu virtual machine already exist.
lxc start my-ubuntu

# check status of my-ubuntu virtual machine
lxc list

# login to root account
lxc exec first -- /bin/bash
# or login to ubuntu account
lxc exec first -- /bin/su - ubuntu

# stop the virtual machine
lxc stop my-ubuntu

# you can even copy virtual machine
lxc copy my-ubuntu my-ubuntu2
# or rename it
lxc mv my-ubuntu stanley-ubuntu
lxc mv stanley-ubuntu my-ubuntu

# delete my-ubuntu virtual machine
lxc delete my-ubuntu

# check my-ubuntu is deleted
lxc list

Varies images is provided by Canonical in http://images.linuxcontainers.org/

Note that lxd has some limitaions. Such as you can't

setup loopback device
mount fuse file system
have bridged network.
load/remove kernel module ... (But if you choose to run a priviledged lxd virtual machine, some limitations might not exist anymore. But I haven't dig into it currently.)

Build Debian/Ubuntu LiveCD

Note that this article introduces how to build Debian/Ubuntu LiveCD iso image from scratch. With this method, you have great flexibility to customize the LiveCD.

However, if you only want to customize the LiveCD a little bit. Such as adding some packages for installation, or changing the preseed configuration, you can refer to DebianCustomCD, Simple-CDD and LiveCDCustomization

Debian Jessie

# within Debian jessie
git clone git://anonscm.debian.org/git/debian-live/live-build.git
cd live-build
git checkout debian/4.0.5-1
dpkg-buildpackage -b -uc -us
cd ..
sudo dpkg -i live-build_4.0.5-1_all.deb

sudo apt-get install -y live-images
cp -r /usr/share/live/images/standard .
cd standard/
lb config --source false # optional, would be faster if not build source
sudo lb build
# here we got live-image-amd64.hybrid.iso, about 417MB
# refer to http://cdimage.debian.org/mirror/cdimage/archive/8.6.0-live/amd64/iso-hybrid/debian-live-8.6.0-amd64-standard.iso.log for more information

Ubuntu Xenial 16.04

Automatically build with ubuntu-defaults-builder

sudo apt install -y ubuntu-defaults-builder
sudo apt install -y syslinux-utils
ubuntu-defaults-template myconfig
cd myconfig/
dpkg-buildpackage
cd ../
ubuntu-defaults-image --package myconfig_0.1_all.deb --components main,restricted,universe,multiverse

# here we got binary.hybrid.iso and livecd.ubuntu.iso

build with ubuntu-cdimage

TODO:

Manually build with live-build (Not working right now, might need hooks to allow booting)

# First, set up the build tools and workspace.
# The scripts require that you work in /build

sudo apt-get install -y genisoimage syslinux-utils # tools for generate ISO image
sudo apt-get install -y memtest86+ syslinux syslinux-themes-ubuntu-xenial gfxboot-theme-ubuntu

sudo -i
apt-get install -y livecd-rootfs
mkdir -p /build/chroot
cd /build
cp -a /usr/share/livecd-rootfs/live-build/auto .

# All the hard work is done with live-build (lb command)
# and we have to configure it with environment variables

export SUITE=xenial
export ARCH=amd64
export PROJECT=ubuntu
export MIRROR=http://archive.ubuntu.com/ubuntu/
export BINARYFORMAT=iso-hybrid
export LB_SYSLINUX_THEME=ubuntu-xenial

# Now we can have live-build set up the workspace

lb config --initramfs-compression=gzip 

# And finally, start the build

lb build

Ref:
[1] https://code.launchpad.net/~semiosis/livecd-rootfs/fix-for-1565985/+merge/298305/comments/768010
[2] Ubuntu live filesystem builds moved to live-build
[3] LocalizedCDImageTools - Ubuntu Wiki
[4] ISO with l10 preloaded for LoCo + UEFI
[5] UbuntuStudio/Seeds - Ubuntu Wiki
[6] Building large-scale Ubuntu derivatives using seeds - Google Docs

[7] ReleaseTeam/CDImageSetup - Ubuntu Wiki

PuDB Python Debugging Tool. A full-screen, console-based Python debugger

I highly recommened pudb, which is a full-screen, console-based visual debugger for Python. The user-interface is nicely designed for python developers.

Here is an example debugging a simple python script:
Introduction to the PuDB Python Debugging Tool

Also, another great tutorial is from Jordi Gutiérrez Hermoso.
Montreal, QC, September 14, 2015 - Jordi Gutiérrez Hermoso presents PuDB, a full-screen, console-based visual debugger for Python.
An interesting comment from the speaker:

Everyone should use a debugger
...

But If pdb is the only debugger you've every seen, I won't blame you not using a debugger

...
because the debugger is ugly, not because the source code is ugly

To install pudb on Debian/Ubuntu, you can use apt-get to install the package

# for python 2

sudo apt-get -y python-pudb
# for python 3

sudo apt-get -y python3-pudb

# pudb is even better when debugging with ipython

# so that's install ipython

sudo apt-get install ipython

How to prevent Linux from waking up due to USB devices

最近換了一台工作機，但我發現在 Ubuntu 14.04.3 上，有時候會休眠失敗。
在失敗的時候，機器有 suspend 成功，但是過2秒後機器就會自動醒來。查看了 /var/log/kern.log, /var/log/syslog, /var/log/pm-suspend.log 沒有太異常的訊息。後來上網查了一下，這可能是有 BIOS 有問題或 device 不正常運作。

在 /proc/acpi/wakeup 中，列出了數種 wakeup event, 以下面的表來說，代表了機器開放了 EHC1, EHC2, PWRB, LID0 這幾種 wakeup event

RP06      S4    *disabled
PXSX      S4    *disabled
RP07      S4    *disabled
PXSX      S4    *disabled
EHC1      S0    *enabled   pci:0000:00:1d.0
EHC2      S4    *enabled  pci:0000:00:1a.0
PWRB      S4    *enabled   platform:PNP0C0C:00
LID0      S4    *enabled   platform:PNP0C0D:00

試著開關 wakeup trigger 來找出問題

# ignore XHC device wakeup event

echo "XHC" > /proc/acpi/wakeup
# try suspend, still failed


# ignore EHC1 device wakeup event

echo "EHC1" > /proc/acpi/wakeup
# try suspend, still failed


# ignore EHC1 device wakeup event

echo "EHC2" > /proc/acpi/wakeup
# try suspend. Bingo!! It suspend successfully!

發現只要關閉 EHC2 wakeup trigger, 機器就能正常 suspend。可以放罝一個 upstart script，讓每次開機時，都自動關閉 EHC2 wakeup trigger

# /etc/init/disable-EHC2.conf
start on started dbus
stop on stopping dbus

script
   sudo -u root sh -c "echo 'EHC2' > /proc/acpi/wakeup"
end script

Ref:

AlphaGo is not the solution to AI

Addressing global exploration effectively is only one of the significant challenges between what is well known now and what needs to be addressed for what I would consider a real AI.

Ref: AlphaGo is not the solution to AI

在 NAS 上架設 ubuntu chroot 環境

今天發現可以透過 Canonical Ltd. 的 linuxcontainer image 快速的在 NAS 上建立一個 chroot 環境。我的機器 cpu 是 armhf ，下面示範建立 Ubuntu 15.04 Vivid 的 chroot 環境。

步驟1：下載 Image，解開後進入 rootfs，並修改 nameserver

wget http://images.linuxcontainers.org/images/ubuntu/vivid/armhf/default/20151008_03:49/rootfs.tar.xz
mkdir rootfs
tar xvf rootfs.tar.xz -C rootfs
cd rootfs
echo "nameserver 8.8.8.8" > etc/resolv.conf

步驟2：mount 需要的 sysfs

mount -t proc proc proc/
mount -t sysfs sys sys/
mount -o bind /dev dev/
mount -t devpts pts dev/pts/

步驟3：chroot

chroot . su -l