Digging in to the Linux install on a NAS

In this post I'm going to describe my adventures in trying to bend an ASUSTOR AS3102T to my will. It all started with the built in webserver for personal pages blocking my ability to use the usual HTTP ports for docker images.

What is a NAS?

A NAS is a Network Attached Storage Device. These come in many forms from a variety of manufacturers from small single drive units with limited functionality, slightly larger 2-4 disk units which can run third party apps and/or container images, to large U1 industrial rack hardware. Most of them run Linux, as such they are fairly pliable if you have root or escalated privileges.

ASUSTOR Linux

The unit I'm using has a very very minimal Linux installed on it (had, I boot openSUSE on it now), and the base is backed up to a second partition. The disk used is a USB2 DOM (disk on module):

Device      Start     End Sectors  Size Type
/dev/sdd1    2048    6143    4096    2M EFI System
/dev/sdd2    6144  505855  499712  244M Microsoft basic data
/dev/sdd3  505856 1005567  499712  244M Microsoft basic data

sdd1 is the EFI boot, it contains only a grub loader, config, and a few modules required.

sdd2 and sdd3 are clones of each other, so if one fails then grub will boot the second one.

Booting

On first boot the minimal Linux is loaded and a lighthttp server is started which allows the user to configure disks and install the rest of the system proper.

The contents of sdd2 are:

dom2/:
total 152M
-rw-r--r-- 1 root root 134M Oct 19 00:35 builtin.tgz
-rw-r--r-- 1 root root   84 Oct 19 00:35 builtin.tgz.cksum
-rw-r--r-- 1 root root   97 Oct 19 00:35 builtin.tgz.md5sum
-rw-r--r-- 1 root root 3.9M Oct 19 00:35 bzImage
-rw-r--r-- 1 root root   78 Oct 19 00:35 bzImage.cksum
-rw-r--r-- 1 root root   93 Oct 19 00:35 bzImage.md5sum
-rw-r--r-- 1 root root  14M Oct 19 00:35 initramfs
-rw-r--r-- 1 root root   81 Oct 19 00:35 initramfs.cksum
-rw-r--r-- 1 root root   95 Oct 19 00:35 initramfs.md5sum

bzImage and initramfs are your usual suspects - Linux kernel and initial ramdisk. The one of interest is builtin.tgz which we will cover soon. First let's peek in initramfs, this can easily be extracted using Gnome Nautilus, there's no need to use a terminal for absolutely everything. It's pretty usual with a run of the mill directory layout using lib and lib64.

bin  dev  etc  init  lib  lib64  mnt  opt  proc  root  run  sbin  sys  tmp  usr  var

But what is init? It's a script run on boot with the following contents:

#!/bin/sh

# set global environment (command, apkg path)
source /etc/script/lib/command.sh
source /etc/script/lib/apkg_path.sh

# replaces the current process image with a new process image.
exec /sbin/init

and /sbin/init is in-fact a symlink to /bin/busybox... So what about the scripts?

/etc/script/lib/apkg_path.sh exports a series of globals that are used for the ASUSTOR apkg package management. command.sh does a lot of exporting of CLI tools under AS_ prefixes such as export AS_UPTIME=/usr/bin/uptime.

What else is in /etc/script?

-rwxr-xr-x 1 luke users 1.2K Apr 22  2014 apkg_script_ldr.sh
-rwxr-xr-x 1 luke users  157 Nov 20  2015 dhcpd-script.sh
-rwxr-xr-x 1 luke users  214 Aug 11  2016 FU41lighttpd.sh
-rwxr-xr-x 1 luke users  874 Jan  3  2017 FU50ssh.sh
-rwxr-xr-x 1 luke users   89 Apr 17  2017 FU99login_template.sh
-rwxr-xr-x 1 luke users  517 Aug 23  2013 kernel_log_backup.sh
drwxr-xr-x 2 luke users 4.0K Oct 27 14:20 lib
-rwxr-xr-x 1 luke users 1012 Oct  9 20:37 rcHotPlug
-rwxr-xr-x 1 luke users  927 Apr  1  2014 rcK.builtinfs
-rwxr-xr-x 1 luke users 1.6K Apr 14  2015 rcK.pluginsfs
-rwxr-xr-x 1 luke users 1.6K Apr 23  2014 rcK.raminitfs
-rwxr-xr-x 1 luke users 1.1K Aug 21  2014 rcNetwork
-rwxr-xr-x 1 luke users  492 Apr  3  2013 rcOnce
-rwxr-xr-x 1 luke users  532 Oct 29  2012 rcReset
-rwxr-xr-x 1 luke users  881 Feb  6  2014 rcS.builtinfs
-rwxr-xr-x 1 luke users 1.3K Apr 14  2015 rcS.pluginsfs
-rwxr-xr-x 1 luke users 3.6K Oct  3 22:51 rcS.raminitfs
-rwxr-xr-x 1 luke users  533 May 13  2013 rcSysEvent
-rwxr-xr-x 1 luke users  997 Jul 25  2012 rcTime
-rwxr-xr-x 1 luke users 1.1K May  6  2014 rcUserChange
-rwxr-xr-x 1 luke users  370 Apr  1  2014 script_trap.sh
-rwxr-xr-x 1 luke users  173 Nov  7  2012 sysinit.sh
-rwxr-xr-x 1 luke users  210 Oct  9 20:37 T11nasmand
-rwxr-xr-x 1 luke users 1.3K Sep 11  2015 vpn.sh
-rwxr-xr-x 1 luke users  176 Oct  9 20:37 Y11nasmand
-rwxr-xr-x 1 luke users  159 Oct  9 20:37 Y12netmand
-rwxr-xr-x 1 luke users  180 Oct  9 20:37 Y13emboardmand

The ones of note here are the *mand scripts as this correlate to the proprietary daemons. The rest of the init process is through old-school init.d, and again the ones of note are *mand. As you can see there are a basic set of services started each and every boot

-rwxr-xr-x 1 luke users 3.0K May 11  2017 NASsuspend
-rwxr-xr-x 1 luke users  223 Oct  9 20:37 P01emboardmand
-rwxr-xr-x 1 luke users  212 Oct  9 20:37 P02lcmd
-rwxr-xr-x 1 luke users  260 Oct  9 20:37 P99hostmand
lrwxrwxrwx 1 luke users   25 Oct 27 14:20 rcK -> /etc/script/rcK.raminitfs
lrwxrwxrwx 1 luke users   19 Oct 27 14:20 rcR -> /etc/script/rcReset
lrwxrwxrwx 1 luke users   25 Oct 27 14:20 rcS -> /etc/script/rcS.raminitfs
-rwxr-xr-x 1 luke users  510 Oct  9 20:37 S11nasmand
-rwxr-xr-x 1 luke users  856 Oct  9 20:37 S12netmand
-rwxr-xr-x 1 luke users  339 Oct  9 20:37 S14lcmd
-rwxr-xr-x 1 luke users  139 Oct  9 20:36 S15fan
-rwxr-xr-x 1 luke users 1.4K Jan 10  2012 S20urandom
-rwxr-xr-x 1 luke users  647 Oct  9 20:37 S21stormand
-rwxr-xr-x 1 luke users  525 May 25  2012 S41crond
-rwxr-xr-x 1 luke users  646 Aug 17  2017 S41lighttpd
-rwxr-xr-x 1 luke users 1.7K Oct  5 16:26 S41ssh
-rwxr-xr-x 1 luke users  350 Oct  9 20:37 S42hostmand
-rwxr-xr-x 1 luke users  277 Oct  9 20:37 S44httpredir
-rwxr-xr-x 1 luke users  972 Oct  9 20:37 S93emboardmand
-rwxr-xr-x 1 luke users  480 Nov  7  2012 S99chk_config
-rwxr-xr-x 1 luke users  324 Oct  9 20:37 S99crontab_check
-rwxr-xr-x 1 luke users  305 Oct  9 20:37 S99watchmand

A little further detail here:

  • S15fan starts a proprietary fan controller
  • S41lighttpd is the initial boot HTTP server which serves compiled CGI (and is actually very fast)
  • S44httpredir redirects ports 80 and 443 to 8000 and 8001 respectively
  • S99chk_config checks if two volumes exist and and symlinks init script directories on them to base init.

Setup

Once booted for the first time the user is led through a GUI to set up disks in raid. Part of this process is also the creation of some extra volumes: a second stage boot, swap partition, and the actual storage partition.

Device       Start        End    Sectors  Size Type
/dev/sda1     2048     524287     522240  255M Linux filesystem
/dev/sda2   524288    4718591    4194304    2G Linux RAID
/dev/sda3  4718592    8912895    4194304    2G Linux RAID
/dev/sda4  8912896 3907028991 3898116096  1.8T Linux RAID
  • sda1 contains raid info
  • sda2 is mount at /volume0 - builtin.tgz is unpacked here
  • sda3 is swap
  • sda4 is mounted at /volume1

The boot process is actually initramfs -> /volume0/usr/builtin/etc/init.d/ -> /volume1/.@plugins/etc/init.d/. The last directory .@plugins is where user-installed packages put their init scripts eg, Docker uses this to start its daemon. With three levels it's a bit of a pain to navigate through everything, and even with just the two base ones this looks like a maintenance nightmare.

The binaries in each stage

One

Lets look at stage one, the initramfs. The /bin contents are minimal:

bsdiff   busybox     iostat  mpstat     pidstat  xz
bspatch  cifsiostat  lz4     nfsiostat  sh

as is /sbin, only what's required to get a very minimal Linux up and running with a shell:

badblocks        e2fsck        fsck.ntfs     mdev.sh       mkfs.msdos
blkid            ethtool       fsck.vfat     mkdosfs       mkfs.ntfs
dhclient         fsck.ext2     gdisk         mke2fs        mkfs.vfat
dhclient-script  fsck.ext3     init          mkfs.ext2     parted
dmsetup          fsck.ext4     ip            mkfs.ext3     resize2fs
dosfsck          fsck.ext4dev  ldconfig      mkfs.ext4     sgdisk
dosfslabel       fsck.hfsplus  mdadm         mkfs.ext4dev  tune2fs
dumpe2fs         fsck.msdos    mdev_init.sh  mkfs.hfsplus  wpa_action.sh

/usr/bin

aggregate_js.sh     gpasswd       rsync        ssh-add            tripplite_usb
bcmxcp_usb          groups        scp          ssh-agent          upsc
blazer_usb          grub-editenv  sftp         ssh-keygen         upsdrvctl
confutil            iconv         sftp-server  ssh-keyscan        usbhid-ups
dbus-daemon         ldd           slogin       ssh-keysign        wget
FU99login_template  lighttpdutil  snmp-ups     ssh-pkcs11-helper
getent              richcomm_usb  ssh          toolbox

Of note is confutil which is a compiled binary whose sole task is to manage a few base configurations.

/usr/sbin

apkg              ezrouterd   iwlist          ntpdate        stormand
applog            fanctrl     iwpriv          pppd           syslog
avahi-daemon      ftpbackupd  iwspy           pppoe          upsctrl
badblockctrl      fwupdate    lcmctrl         pppoe-connect  upsd
bios_update_util  groupadd    lcmd            pppoe-relay    upsmon
buzzctrl          groupdel    ledctrl         pppoe-setup    upssched
chknasmode        groupmod    lighttpd        pppoe-start    useradd
chpasswd          hdparm      lighttpd-angel  pppoe-status   userdel
crontab_check     hostmand    logmand         pppoe-stop     usermanutil
dhcp6c            httpredir   logrotate       raidmand       usermod
dhcp6ctl          ifrename    mcu-update      recybind       volmand
dhcpserverctrl    iodumpman   myhttpd         sftpmand       watchmand
dmidecode         iwconfig    nasmand         shutdownctrl   wpa_cli
ecctrl            iwevent     netman          sm-notify      wpa_supplicant
emboardmand       iwgetid     netmand         sshd           zic

I want to call out myhttpd here, this little fucker was hard to kill, there (was, I think they changed it after complaints) an init script in the initramfs which would start this without fail, and its sole purpose is to redirect ports 80 and 443 to 8000 and 8001 unless the personal apache server is being used. I hated it! I ended up making a single script to run before Docker with the sole purpose of finding its PID and KILLING IT WITH FIRE (-9)! You little shitbag. DIE!

cough

So yeah, see those *mand files? Those are the proprietary ones, there's a few others also: fanctrl, bios_update_util (supplied by Intel?), and ezrouterd.

Two

/volume0/usr/builtin/bin also contains a few proprietary tools, mostly to do with system registration, updates, and maybe some non-obvious ones.

2-step-verify         l2ping                     php
7z                    ldapmodify                 php-cgi
adm-online-update     ldapsearch                 php-fpm
certificate           locate                     pptp
convert               lprm                       pushbullet
cryptsetup            lpstat                     quota
curl                  memcached                  resolveip
dbd                   msmtp                      rfcomm
dbus-send             mutt                       rsync
dcraw                 my_print_defaults          rsyncd
effectiveperm         mysql                      sdptool
exiv2                 mysqladmin                 set-used-server
ffmpeg-mini           mysqlbackup                simple-mtpfs
file                  mysqlcheck                 smbclient
ftpscrub              mysqld_safe                smbpasswd
ftpwho                mysqldump                  smbstatus
FU50-22888            mysql_install_db           sudo
gearadmin             mysql_secure_installation  sudoedit
gearman               mysql_upgrade              sudoreplay
google-authenticator  net                        thumbnail
hcitool               nmblookup                  tree
healthrecord          notification_add_column    udevadm
htop                  ntlm_auth                  ufraw-batch
htpasswd              openssl                    updatedb
identify              openvpn                    wbinfo
internalbackup        pdbedit                    webservconf
ip                    pdftops                    zip

I won't list /volume0/usr/builtin/sbin, it mimics the above with yet more userland tools, and a few more proprietary ones.

Three

Wow this is tiring. Let's not list things, there's nothing of importance except the it is yet another layer of:

bin  etc  lib  lib64  sbin  tmp  usr

nestled in the /volume1/.@plugins directory.

Those proprietary binaries

I'm keen to see what's in those pesky files, aren't you? This entire exercise stemmed from me trying to work out how to kill the port hogger above. All the *mand are:

emboardmand  logmand  netmand   sftpmand  volmand
hostmand     nasmand  raidmand  stormand  watchmand

strings stormand, a snippet of strings:

_Z20Lock_System_Usb_Diski
_Z20Lookup_Nas_Disk_InfoiiiR17_T_NAS_DISK_STAT_
_Z14Probe_Nas_DiskPKcR17_T_NAS_DISK_STAT_
_Z20Lookup_Nas_Bank_InfoiiiR17_T_NAS_BANK_STAT_
_Z18Get_Nas_Disk_AliasRK17_T_NAS_DISK_STAT_RiPc
_Z34Lookup_Nas_Archive_Id_By_Device_IdiRi
_Z29Reactivate_Nas_Storage_SystemPFvvE
_Z14Unload_Nas_Boxii
_Z17Get_Nas_Box_AliasiiiRiPc
...
[Volume]
SYSTEM
/sys/bus/usb/devices/%d-%d
/sys/bus/usb/devices/%d-%d.%d
%s/bDeviceClass
stormand: Reboot now!
/dev/sd
/mnt/booting
MyArchive
>/dev/null 2>&1
/usr/sbin/fwupdate
%s recovery %s &
%s %s &
/proc/mounts
/volume
/archive
/usr/sbin/volmand
%s %s%d %s &
/sys/bus/usb/devices/%d-%d.
[Expansion Box]
%s %s ejected.
Ejected
[USB]
[eSATA]
[Disk]
DISK %d
Detected
satadisk
%s %s detected.
...
GCC: (crosstool-NG crosstool-ng-1.22.0 - x86_64 64-bit toolchain - ASUSTOR Inc.) 4.6.4

I've only grabbed some of the more interesting strings here. Error codes and lots of hard-coded paths. Browsing through the strings it looks like stormand is indeed responsible for all storage, and is passed args via the base webserver. The rest of the Mandy's follow the same routine. Browsing the error codes it looks like ASUSTOR catches everything possible too (but don't quote me).

The initial webserver (lighthttp)

lighthttp serves up content from /usr/webman once it's booted. The server side of this contains compiled cgi, and a client side app which is a whole lot of minified but not obscured js, and all the assets required. At his point I was just getting curious about things and wondering how the overall NAS structure was done.

A quick strings of /usr/webman/index.cgi

[luke@linux-cq7k]$ strings index.cgi 
/lib64/ld-linux-x86-64.so.2
_fini
__gmon_start__
_Jv_RegisterClasses
_init
libnasman.so
_Z21Is_System_Initializedv
libdl.so.2
libc.so.6
puts
time
<snipped>

libnasman.so eh? Let's look at that later. The only other insteresting thing in webman is jslibs/ext-5.0.1/ - this is an interesting library from Sencha, I think it's used for a variety of things in the user facing UI such as fancy graphs of the system stats, the windowing system, themes etc. Overall it's quite nice but overkill - the library size is 1.9MB, gedit shits the bed on files that size, vim doesn't though (yay vim!) but I didn't fancy scrolling a very long line. This is also a very expensive library at $1895USD for the enterpri.... I'm in the wrong line of work, I should be writing JS libraries to sell to NAS makers.

At this point I've entirely forgotten the point of this post and I'm still waiting for gedit to close after trying to load that library. Maybe I should crack open that wine. I bought a bottle of wine once but didn't have a corkscrew, I thought I'd be clever and push the cork in. I mean it worked... But there was wine fucking everywhere - I guess that pressure had to go somewhere eventually. And whenever I tried to pour it in to the glass the cork would exhibit this uncanny ability to go straight up the neck and block the flow.

I ended up drinking wine from the bottle with a straw.

Right, so, libnasman.so

-rw-r--r-- 1 luke users 534K Oct  9 20:36 libnasman.so.0.0

Do the strings thing, and, well this library seems to be a jack of all trades and hands duties off to everyone. It looks to handle core functionality such as image loading and conversions, video playing, video conversions, thumbnailing, networks, firmware, drives and partitioning, encryption and more. So something of a orchestrator of many of the base tasks a user would like to do. It's linked in to the webman cgi stuff, and in to many of the Mandy daemons.

Putting it all together

Heck of a ride. I actually did a lot more than what I've covered in this post - I tracked library links, mapped init scripts to run-levels to daemons to libraries etc - but hopefully the main gist of it comes through. The ASUSTOR NAS is really quite good, they created a very small and minimal software package that is extremely robust, with a front and back-end that is fast and attractive (oh... now I'm a product reviewer?).

The overall boot flow is something like this:

  1. Boot initramfs
  2. Does /volume0 exist? No -> Go to step 4, else
  3. Does /volume1 exist? Yes -> Go to step 6, else
  4. Load initial http server and direct user through setup
  5. Unpack builtin.tgz to /volume0
  6. Run init scripts in /volume0/usr/builtin/etc/init.d/
  7. Run init scripts in /volume1/.@plugins/etc/init.d/
  8. If any error, push cork in to bottle and attempt futile task of deleting myhttpd

And the strong points of the way ASUSTOR have created this are:

  • Uses lighthttp to serve the user UI, also uses a flash JS library for windows you can cover other windows with.
  • Minimal Linux is fairly fast to boot, but then so is openSUSE
  • The dual boot partitions are a good strategy to have ones behind
  • Does not use any sort of interpreted environment anywhere. Nothing. Nadda. Everything is compiled to be fast.
  • And so uses bugger all resources - less than 100MB

What's bad though, is you as a user of this product don't have full control over it.

  • It uses apkg, which is a pain to create and is limited
  • Lots of proprietary stuff, including control of the disk activity LED
  • Tracking

This is just one way to build a NAS. It works, and works well. But in the end it wasn't what I wanted and the available OSS options weren't to my taste either so...

I'm going to be writing my own NAS front and back-end, using Rust, and based on openSUSE (targetting Kubic as the base). Keep an eye out for hardbytes.co.nz in the coming weeks.

Oh and I'll probably move this blog to rustacean.co.nz some-time soon.

Edit #1

I dug up the old posts I made on the ASUSTOR forums where I tried to find info on how to disable the myhttpd daemon - the majority of those posts are duplicated below.

I started with running netstat, which I had to scp over as it isn't available in the base install:

admin@nas:/etc/init.d $ sudo netstat -ntulp | grep 443
tcp        0      0 0.0.0.0:443             0.0.0.0:*               LISTEN      4100/myhttpd

admin@nas:/etc/init.d $ cat /proc/4100/cmdline
/usr/sbin/myhttpd-t1-p8001

And running myhttpd on its own results in:

admin@nas:/etc/init.d $ myhttpd
Usage:
  httpd -t type -p port
  type : 0(http), 1(https)

So the process that called "myhttpd -t1 -p8001" is what we need to find and kill...

If I start apache with ports 80/443 enabled then that process is killed, but now apache is using those ports. Even if apache is running, but on different ports, then something starts that process as soon as those ports are freed from apache.

I thought it was /etc/init.d/S44httpredir controlling it, but because this is in the initramfs it is restored on reboot each time, so any modification is practically useless (unless you edit the initramfs, which can be done).

The library libservice calls myhttpd in a few ways, and that lib is linked to by various things in the webman.

./lib/libservice.so.0:135995:/usr/sbin/myhttpd -t %d -p %d
./lib/libservice.so.0:136002:/usr/bin/killall -TERM myhttpd>/dev/null 2>&1
./lib/libservice.so.0:136005:/usr/bin/killall -9 myhttpd>/dev/null 2>&1
./lib/libservice.so:135995:/usr/sbin/myhttpd -t %d -p %d
./lib/libservice.so:136002:/usr/bin/killall -TERM myhttpd>/dev/null 2>&1
./lib/libservice.so:136005:/usr/bin/killall -9 myhttpd>/dev/null 2>&1
./lib/libservice.so.0.0:135995:/usr/sbin/myhttpd -t %d -p %d
./lib/libservice.so.0.0:136002:/usr/bin/killall -TERM myhttpd>/dev/null 2>&1
./lib/libservice.so.0.0:136005:/usr/bin/killall -9 myhttpd>/dev/null 2>&1
./bin/lighttpdutil:1078:libservice.so
./webman/initial/sysreset.cgi:1402:libservice.so
./webman/initial/initial.cgi:3658:libservice.so
./lib/libservice.so.0:33733:libservice.so
./lib/libplugin.so:2384:libservice.so
./lib/libbuiltin.so.0.0:11025:libservice.so
./lib/libnasman.so.0:32123:libservice.so
./lib/libnasman.so.0.0:32123:libservice.so
./lib/libplugin.so.0:2384:libservice.so
./lib/libservice.so:33733:libservice.so
./lib/libbuiltin.so:11025:libservice.so
./lib/libplugin.so.0.0:2384:libservice.so
./lib/libbuiltin.so.0:11025:libservice.so
./lib/libnasman.so:32123:libservice.so
./lib/security/pam_google_authenticator.so:1759:libservice.so
./lib/libservice.so.0.0:33733:libservice.so
./sbin/httpredir:840:libservice.so
./sbin/recybind:1755:libservice.so
./sbin/nasmand:3057:libservice.so
./sbin/sftpmand:3425:libservice.so
./sbin/stormand:4836:libservice.so
./sbin/hostmand:3991:libservice.so
./sbin/logmand:2375:libservice.so
./sbin/netmand:2925:libservice.so
./sbin/dhcpserverctrl:1142:libservice.so

Because the whole point of this exercise in frustration was to claim the 80 and 443 ports for use with docker images, so I could then use nginx for routing to various containers, it turned out the easiest way to deal with this was to create either a startup script in the /volume1/.@plugins/etc/init.d/ or append to the docker init script the following:

PID1="$(fuser 80/tcp)"
PID2="$(fuser 443/tcp)"
echo "Killing useless port 80 hog, PID=${PID1}"
kill -9 ${PID1}
echo "Killing useless port 443 hog, PID=${PID2}"
kill -9 ${PID2}

and hope that docker claimed those ports before myhttpd could be started again.