In this post I'm going to describe my adventures in trying to bend an ASUSTOR AS3102T to my will. It all started with the built in webserver for personal pages blocking my ability to use the usual HTTP ports for docker images.
What is a NAS?
A NAS is a Network Attached Storage Device. These come in many forms from a variety of manufacturers from small single drive units with limited functionality, slightly larger 2-4 disk units which can run third party apps and/or container images, to large U1 industrial rack hardware. Most of them run Linux, as such they are fairly pliable if you have root or escalated privileges.
ASUSTOR Linux
The unit I'm using has a very very minimal Linux installed on it (had, I boot openSUSE on it now), and the base is backed up to a second partition. The disk used is a USB2 DOM (disk on module):
Device Start End Sectors Size Type /dev/sdd1 2048 6143 4096 2M EFI System /dev/sdd2 6144 505855 499712 244M Microsoft basic data /dev/sdd3 505856 1005567 499712 244M Microsoft basic data
sdd1 is the EFI boot, it contains only a grub loader, config, and a few modules required.
sdd2 and sdd3 are clones of each other, so if one fails then grub will boot the second one.
Booting
On first boot the minimal Linux is loaded and a lighthttp server is started which allows the user to configure disks and install the rest of the system proper.
The contents of sdd2 are:
dom2/: total 152M -rw-r--r-- 1 root root 134M Oct 19 00:35 builtin.tgz -rw-r--r-- 1 root root 84 Oct 19 00:35 builtin.tgz.cksum -rw-r--r-- 1 root root 97 Oct 19 00:35 builtin.tgz.md5sum -rw-r--r-- 1 root root 3.9M Oct 19 00:35 bzImage -rw-r--r-- 1 root root 78 Oct 19 00:35 bzImage.cksum -rw-r--r-- 1 root root 93 Oct 19 00:35 bzImage.md5sum -rw-r--r-- 1 root root 14M Oct 19 00:35 initramfs -rw-r--r-- 1 root root 81 Oct 19 00:35 initramfs.cksum -rw-r--r-- 1 root root 95 Oct 19 00:35 initramfs.md5sum
bzImage and initramfs are your usual suspects - Linux kernel and initial ramdisk. The one of interest is builtin.tgz which we will cover soon. First let's peek in initramfs, this can easily be extracted using Gnome Nautilus, there's no need to use a terminal for absolutely everything. It's pretty usual with a run of the mill directory layout using lib and lib64.
bin dev etc init lib lib64 mnt opt proc root run sbin sys tmp usr var
But what is init? It's a script run on boot with the following contents:
#!/bin/sh # set global environment (command, apkg path) source /etc/script/lib/command.sh source /etc/script/lib/apkg_path.sh # replaces the current process image with a new process image. exec /sbin/init
and /sbin/init is in-fact a symlink to /bin/busybox... So what about the scripts?
/etc/script/lib/apkg_path.sh exports a series of globals that are used for the ASUSTOR apkg package management. command.sh does a lot of exporting of CLI tools under AS_ prefixes such as export AS_UPTIME=/usr/bin/uptime.
What else is in /etc/script?
-rwxr-xr-x 1 luke users 1.2K Apr 22 2014 apkg_script_ldr.sh -rwxr-xr-x 1 luke users 157 Nov 20 2015 dhcpd-script.sh -rwxr-xr-x 1 luke users 214 Aug 11 2016 FU41lighttpd.sh -rwxr-xr-x 1 luke users 874 Jan 3 2017 FU50ssh.sh -rwxr-xr-x 1 luke users 89 Apr 17 2017 FU99login_template.sh -rwxr-xr-x 1 luke users 517 Aug 23 2013 kernel_log_backup.sh drwxr-xr-x 2 luke users 4.0K Oct 27 14:20 lib -rwxr-xr-x 1 luke users 1012 Oct 9 20:37 rcHotPlug -rwxr-xr-x 1 luke users 927 Apr 1 2014 rcK.builtinfs -rwxr-xr-x 1 luke users 1.6K Apr 14 2015 rcK.pluginsfs -rwxr-xr-x 1 luke users 1.6K Apr 23 2014 rcK.raminitfs -rwxr-xr-x 1 luke users 1.1K Aug 21 2014 rcNetwork -rwxr-xr-x 1 luke users 492 Apr 3 2013 rcOnce -rwxr-xr-x 1 luke users 532 Oct 29 2012 rcReset -rwxr-xr-x 1 luke users 881 Feb 6 2014 rcS.builtinfs -rwxr-xr-x 1 luke users 1.3K Apr 14 2015 rcS.pluginsfs -rwxr-xr-x 1 luke users 3.6K Oct 3 22:51 rcS.raminitfs -rwxr-xr-x 1 luke users 533 May 13 2013 rcSysEvent -rwxr-xr-x 1 luke users 997 Jul 25 2012 rcTime -rwxr-xr-x 1 luke users 1.1K May 6 2014 rcUserChange -rwxr-xr-x 1 luke users 370 Apr 1 2014 script_trap.sh -rwxr-xr-x 1 luke users 173 Nov 7 2012 sysinit.sh -rwxr-xr-x 1 luke users 210 Oct 9 20:37 T11nasmand -rwxr-xr-x 1 luke users 1.3K Sep 11 2015 vpn.sh -rwxr-xr-x 1 luke users 176 Oct 9 20:37 Y11nasmand -rwxr-xr-x 1 luke users 159 Oct 9 20:37 Y12netmand -rwxr-xr-x 1 luke users 180 Oct 9 20:37 Y13emboardmand
The ones of note here are the *mand scripts as this correlate to the proprietary daemons. The rest of the init process is through old-school init.d, and again the ones of note are *mand. As you can see there are a basic set of services started each and every boot
-rwxr-xr-x 1 luke users 3.0K May 11 2017 NASsuspend -rwxr-xr-x 1 luke users 223 Oct 9 20:37 P01emboardmand -rwxr-xr-x 1 luke users 212 Oct 9 20:37 P02lcmd -rwxr-xr-x 1 luke users 260 Oct 9 20:37 P99hostmand lrwxrwxrwx 1 luke users 25 Oct 27 14:20 rcK -> /etc/script/rcK.raminitfs lrwxrwxrwx 1 luke users 19 Oct 27 14:20 rcR -> /etc/script/rcReset lrwxrwxrwx 1 luke users 25 Oct 27 14:20 rcS -> /etc/script/rcS.raminitfs -rwxr-xr-x 1 luke users 510 Oct 9 20:37 S11nasmand -rwxr-xr-x 1 luke users 856 Oct 9 20:37 S12netmand -rwxr-xr-x 1 luke users 339 Oct 9 20:37 S14lcmd -rwxr-xr-x 1 luke users 139 Oct 9 20:36 S15fan -rwxr-xr-x 1 luke users 1.4K Jan 10 2012 S20urandom -rwxr-xr-x 1 luke users 647 Oct 9 20:37 S21stormand -rwxr-xr-x 1 luke users 525 May 25 2012 S41crond -rwxr-xr-x 1 luke users 646 Aug 17 2017 S41lighttpd -rwxr-xr-x 1 luke users 1.7K Oct 5 16:26 S41ssh -rwxr-xr-x 1 luke users 350 Oct 9 20:37 S42hostmand -rwxr-xr-x 1 luke users 277 Oct 9 20:37 S44httpredir -rwxr-xr-x 1 luke users 972 Oct 9 20:37 S93emboardmand -rwxr-xr-x 1 luke users 480 Nov 7 2012 S99chk_config -rwxr-xr-x 1 luke users 324 Oct 9 20:37 S99crontab_check -rwxr-xr-x 1 luke users 305 Oct 9 20:37 S99watchmand
A little further detail here:
S15fanstarts a proprietary fan controllerS41lighttpdis the initial boot HTTP server which serves compiled CGI (and is actually very fast)S44httpredirredirects ports 80 and 443 to 8000 and 8001 respectivelyS99chk_configchecks if two volumes exist and and symlinks init script directories on them to base init.
Setup
Once booted for the first time the user is led through a GUI to set up disks in raid. Part of this process is also the creation of some extra volumes: a second stage boot, swap partition, and the actual storage partition.
Device Start End Sectors Size Type /dev/sda1 2048 524287 522240 255M Linux filesystem /dev/sda2 524288 4718591 4194304 2G Linux RAID /dev/sda3 4718592 8912895 4194304 2G Linux RAID /dev/sda4 8912896 3907028991 3898116096 1.8T Linux RAID
sda1contains raid infosda2is mount at/volume0-builtin.tgzis unpacked heresda3is swapsda4is mounted at/volume1
The boot process is actually initramfs -> /volume0/usr/builtin/etc/init.d/ -> /volume1/.@plugins/etc/init.d/. The last directory .@plugins is where user-installed packages put their init scripts eg, Docker uses this to start its daemon. With three levels it's a bit of a pain to navigate through everything, and even with just the two base ones this looks like a maintenance nightmare.
The binaries in each stage
One
Lets look at stage one, the initramfs. The /bin contents are minimal:
bsdiff busybox iostat mpstat pidstat xz bspatch cifsiostat lz4 nfsiostat sh
as is /sbin, only what's required to get a very minimal Linux up and running with a shell:
badblocks e2fsck fsck.ntfs mdev.sh mkfs.msdos blkid ethtool fsck.vfat mkdosfs mkfs.ntfs dhclient fsck.ext2 gdisk mke2fs mkfs.vfat dhclient-script fsck.ext3 init mkfs.ext2 parted dmsetup fsck.ext4 ip mkfs.ext3 resize2fs dosfsck fsck.ext4dev ldconfig mkfs.ext4 sgdisk dosfslabel fsck.hfsplus mdadm mkfs.ext4dev tune2fs dumpe2fs fsck.msdos mdev_init.sh mkfs.hfsplus wpa_action.sh
/usr/bin
aggregate_js.sh gpasswd rsync ssh-add tripplite_usb bcmxcp_usb groups scp ssh-agent upsc blazer_usb grub-editenv sftp ssh-keygen upsdrvctl confutil iconv sftp-server ssh-keyscan usbhid-ups dbus-daemon ldd slogin ssh-keysign wget FU99login_template lighttpdutil snmp-ups ssh-pkcs11-helper getent richcomm_usb ssh toolbox
Of note is confutil which is a compiled binary whose sole task is to manage a few base configurations.
/usr/sbin
apkg ezrouterd iwlist ntpdate stormand applog fanctrl iwpriv pppd syslog avahi-daemon ftpbackupd iwspy pppoe upsctrl badblockctrl fwupdate lcmctrl pppoe-connect upsd bios_update_util groupadd lcmd pppoe-relay upsmon buzzctrl groupdel ledctrl pppoe-setup upssched chknasmode groupmod lighttpd pppoe-start useradd chpasswd hdparm lighttpd-angel pppoe-status userdel crontab_check hostmand logmand pppoe-stop usermanutil dhcp6c httpredir logrotate raidmand usermod dhcp6ctl ifrename mcu-update recybind volmand dhcpserverctrl iodumpman myhttpd sftpmand watchmand dmidecode iwconfig nasmand shutdownctrl wpa_cli ecctrl iwevent netman sm-notify wpa_supplicant emboardmand iwgetid netmand sshd zic
I want to call out myhttpd here, this little fucker was hard to kill, there (was, I think they changed it after complaints) an init script in the initramfs which would start this without fail, and its sole purpose is to redirect ports 80 and 443 to 8000 and 8001 unless the personal apache server is being used. I hated it! I ended up making a single script to run before Docker with the sole purpose of finding its PID and KILLING IT WITH FIRE (-9)! You little shitbag. DIE!
cough
So yeah, see those *mand files? Those are the proprietary ones, there's a few others also: fanctrl, bios_update_util (supplied by Intel?), and ezrouterd.
Two
/volume0/usr/builtin/bin also contains a few proprietary tools, mostly to do with system registration, updates, and maybe some non-obvious ones.
2-step-verify l2ping php 7z ldapmodify php-cgi adm-online-update ldapsearch php-fpm certificate locate pptp convert lprm pushbullet cryptsetup lpstat quota curl memcached resolveip dbd msmtp rfcomm dbus-send mutt rsync dcraw my_print_defaults rsyncd effectiveperm mysql sdptool exiv2 mysqladmin set-used-server ffmpeg-mini mysqlbackup simple-mtpfs file mysqlcheck smbclient ftpscrub mysqld_safe smbpasswd ftpwho mysqldump smbstatus FU50-22888 mysql_install_db sudo gearadmin mysql_secure_installation sudoedit gearman mysql_upgrade sudoreplay google-authenticator net thumbnail hcitool nmblookup tree healthrecord notification_add_column udevadm htop ntlm_auth ufraw-batch htpasswd openssl updatedb identify openvpn wbinfo internalbackup pdbedit webservconf ip pdftops zip
I won't list /volume0/usr/builtin/sbin, it mimics the above with yet more userland tools, and a few more proprietary ones.
Three
Wow this is tiring. Let's not list things, there's nothing of importance except the it is yet another layer of:
bin etc lib lib64 sbin tmp usr
nestled in the /volume1/.@plugins directory.
Those proprietary binaries
I'm keen to see what's in those pesky files, aren't you? This entire exercise stemmed from me trying to work out how to kill the port hogger above. All the *mand are:
emboardmand logmand netmand sftpmand volmand hostmand nasmand raidmand stormand watchmand
strings stormand, a snippet of strings:
_Z20Lock_System_Usb_Diski _Z20Lookup_Nas_Disk_InfoiiiR17_T_NAS_DISK_STAT_ _Z14Probe_Nas_DiskPKcR17_T_NAS_DISK_STAT_ _Z20Lookup_Nas_Bank_InfoiiiR17_T_NAS_BANK_STAT_ _Z18Get_Nas_Disk_AliasRK17_T_NAS_DISK_STAT_RiPc _Z34Lookup_Nas_Archive_Id_By_Device_IdiRi _Z29Reactivate_Nas_Storage_SystemPFvvE _Z14Unload_Nas_Boxii _Z17Get_Nas_Box_AliasiiiRiPc ... [Volume] SYSTEM /sys/bus/usb/devices/%d-%d /sys/bus/usb/devices/%d-%d.%d %s/bDeviceClass stormand: Reboot now! /dev/sd /mnt/booting MyArchive >/dev/null 2>&1 /usr/sbin/fwupdate %s recovery %s & %s %s & /proc/mounts /volume /archive /usr/sbin/volmand %s %s%d %s & /sys/bus/usb/devices/%d-%d. [Expansion Box] %s %s ejected. Ejected [USB] [eSATA] [Disk] DISK %d Detected satadisk %s %s detected. ... GCC: (crosstool-NG crosstool-ng-1.22.0 - x86_64 64-bit toolchain - ASUSTOR Inc.) 4.6.4
I've only grabbed some of the more interesting strings here. Error codes and lots of hard-coded paths. Browsing through the strings it looks like stormand is indeed responsible for all storage, and is passed args via the base webserver. The rest of the Mandy's follow the same routine. Browsing the error codes it looks like ASUSTOR catches everything possible too (but don't quote me).
The initial webserver (lighthttp)
lighthttp serves up content from /usr/webman once it's booted. The server side of this contains compiled cgi, and a client side app which is a whole lot of minified but not obscured js, and all the assets required. At his point I was just getting curious about things and wondering how the overall NAS structure was done.
A quick strings of /usr/webman/index.cgi
[luke@linux-cq7k]$ strings index.cgi /lib64/ld-linux-x86-64.so.2 _fini __gmon_start__ _Jv_RegisterClasses _init libnasman.so _Z21Is_System_Initializedv libdl.so.2 libc.so.6 puts time <snipped>
libnasman.so eh? Let's look at that later. The only other insteresting thing in webman is jslibs/ext-5.0.1/ - this is an interesting library from Sencha, I think it's used for a variety of things in the user facing UI such as fancy graphs of the system stats, the windowing system, themes etc. Overall it's quite nice but overkill - the library size is 1.9MB, gedit shits the bed on files that size, vim doesn't though (yay vim!) but I didn't fancy scrolling a very long line. This is also a very expensive library at $1895USD for the enterpri.... I'm in the wrong line of work, I should be writing JS libraries to sell to NAS makers.
At this point I've entirely forgotten the point of this post and I'm still waiting for gedit to close after trying to load that library. Maybe I should crack open that wine. I bought a bottle of wine once but didn't have a corkscrew, I thought I'd be clever and push the cork in. I mean it worked... But there was wine fucking everywhere - I guess that pressure had to go somewhere eventually. And whenever I tried to pour it in to the glass the cork would exhibit this uncanny ability to go straight up the neck and block the flow.
I ended up drinking wine from the bottle with a straw.
Right, so, libnasman.so
-rw-r--r-- 1 luke users 534K Oct 9 20:36 libnasman.so.0.0
Do the strings thing, and, well this library seems to be a jack of all trades and hands duties off to everyone. It looks to handle core functionality such as image loading and conversions, video playing, video conversions, thumbnailing, networks, firmware, drives and partitioning, encryption and more. So something of a orchestrator of many of the base tasks a user would like to do. It's linked in to the webman cgi stuff, and in to many of the Mandy daemons.
Putting it all together
Heck of a ride. I actually did a lot more than what I've covered in this post - I tracked library links, mapped init scripts to run-levels to daemons to libraries etc - but hopefully the main gist of it comes through. The ASUSTOR NAS is really quite good, they created a very small and minimal software package that is extremely robust, with a front and back-end that is fast and attractive (oh... now I'm a product reviewer?).
The overall boot flow is something like this:
- Boot initramfs
- Does
/volume0exist? No -> Go to step 4, else - Does
/volume1exist? Yes -> Go to step 6, else - Load initial http server and direct user through setup
- Unpack
builtin.tgzto/volume0 - Run init scripts in
/volume0/usr/builtin/etc/init.d/ - Run init scripts in
/volume1/.@plugins/etc/init.d/ - If any error, push cork in to bottle and attempt futile task of deleting
myhttpd
And the strong points of the way ASUSTOR have created this are:
- Uses lighthttp to serve the user UI, also uses a flash JS library for windows you can cover other windows with.
- Minimal Linux is fairly fast to boot, but then so is openSUSE
- The dual boot partitions are a good strategy to have ones behind
- Does not use any sort of interpreted environment anywhere. Nothing. Nadda. Everything is compiled to be fast.
- And so uses bugger all resources - less than 100MB
What's bad though, is you as a user of this product don't have full control over it.
- It uses apkg, which is a pain to create and is limited
- Lots of proprietary stuff, including control of the disk activity LED
- Tracking
This is just one way to build a NAS. It works, and works well. But in the end it wasn't what I wanted and the available OSS options weren't to my taste either so...
I'm going to be writing my own NAS front and back-end, using Rust, and based on openSUSE (targetting Kubic as the base). Keep an eye out for hardbytes.co.nz in the coming weeks.
Oh and I'll probably move this blog to rustacean.co.nz some-time soon.
Edit #1
I dug up the old posts I made on the ASUSTOR forums where I tried to find info on how to disable the myhttpd daemon - the majority of those posts are duplicated below.
I started with running netstat, which I had to scp over as it isn't available in the base install:
admin@nas:/etc/init.d $ sudo netstat -ntulp | grep 443 tcp 0 0 0.0.0.0:443 0.0.0.0:* LISTEN 4100/myhttpd admin@nas:/etc/init.d $ cat /proc/4100/cmdline /usr/sbin/myhttpd-t1-p8001
And running myhttpd on its own results in:
admin@nas:/etc/init.d $ myhttpd Usage: httpd -t type -p port type : 0(http), 1(https)
So the process that called "myhttpd -t1 -p8001" is what we need to find and kill...
If I start apache with ports 80/443 enabled then that process is killed, but now apache is using those ports. Even if apache is running, but on different ports, then something starts that process as soon as those ports are freed from apache.
I thought it was /etc/init.d/S44httpredir controlling it, but because this is in the initramfs it is restored on reboot each time, so any modification is practically useless (unless you edit the initramfs, which can be done).
The library libservice calls myhttpd in a few ways, and that lib is linked to by various things in the webman.
./lib/libservice.so.0:135995:/usr/sbin/myhttpd -t %d -p %d ./lib/libservice.so.0:136002:/usr/bin/killall -TERM myhttpd>/dev/null 2>&1 ./lib/libservice.so.0:136005:/usr/bin/killall -9 myhttpd>/dev/null 2>&1 ./lib/libservice.so:135995:/usr/sbin/myhttpd -t %d -p %d ./lib/libservice.so:136002:/usr/bin/killall -TERM myhttpd>/dev/null 2>&1 ./lib/libservice.so:136005:/usr/bin/killall -9 myhttpd>/dev/null 2>&1 ./lib/libservice.so.0.0:135995:/usr/sbin/myhttpd -t %d -p %d ./lib/libservice.so.0.0:136002:/usr/bin/killall -TERM myhttpd>/dev/null 2>&1 ./lib/libservice.so.0.0:136005:/usr/bin/killall -9 myhttpd>/dev/null 2>&1
./bin/lighttpdutil:1078:libservice.so ./webman/initial/sysreset.cgi:1402:libservice.so ./webman/initial/initial.cgi:3658:libservice.so ./lib/libservice.so.0:33733:libservice.so ./lib/libplugin.so:2384:libservice.so ./lib/libbuiltin.so.0.0:11025:libservice.so ./lib/libnasman.so.0:32123:libservice.so ./lib/libnasman.so.0.0:32123:libservice.so ./lib/libplugin.so.0:2384:libservice.so ./lib/libservice.so:33733:libservice.so ./lib/libbuiltin.so:11025:libservice.so ./lib/libplugin.so.0.0:2384:libservice.so ./lib/libbuiltin.so.0:11025:libservice.so ./lib/libnasman.so:32123:libservice.so ./lib/security/pam_google_authenticator.so:1759:libservice.so ./lib/libservice.so.0.0:33733:libservice.so ./sbin/httpredir:840:libservice.so ./sbin/recybind:1755:libservice.so ./sbin/nasmand:3057:libservice.so ./sbin/sftpmand:3425:libservice.so ./sbin/stormand:4836:libservice.so ./sbin/hostmand:3991:libservice.so ./sbin/logmand:2375:libservice.so ./sbin/netmand:2925:libservice.so ./sbin/dhcpserverctrl:1142:libservice.so
Because the whole point of this exercise in frustration was to claim the 80 and 443 ports for use with docker images, so I could then use nginx for routing to various containers, it turned out the easiest way to deal with this was to create either a startup script in the /volume1/.@plugins/etc/init.d/ or append to the docker init script the following:
PID1="$(fuser 80/tcp)" PID2="$(fuser 443/tcp)" echo "Killing useless port 80 hog, PID=${PID1}" kill -9 ${PID1} echo "Killing useless port 443 hog, PID=${PID2}" kill -9 ${PID2}
and hope that docker claimed those ports before myhttpd could be started again.