Linux, in all its diversity, sometimes lacks very basic mechanisms that you would be prepared to take for granted. For instance, picture this scenario.. you are on some system you don’t know well. Maybe it’s your friend who called you over to his house for help with something. Maybe it’s the computer lab where you don’t know the setup. Maybe it’s a remote session to a box you’ve never seen. Quick question: what’s it running?

Well, bash --version tells you about bash, ls --version tells you about coreutils and so on, you can keep going with that. uname will tell you about the platform and the kernel, useful stuff. What about the distro? Distros are diverse, sometimes it helps a lot to know what kind of environment you’re in. Well, strangely enough, there’s no standard answer to that.

They all do their own thing, if they do at all. Some distros use lsb_release to dispense this information. Others have files in /etc that you can check for, if you know what they are supposed to be called. So I decided to try and detect this. I’ve checked a bunch of livecds and it works on those distros that identify themselves somehow.

# Author: Martin Matusiak <numerodix@gmail.com>
# Licensed under the GNU Public License, version 3
#
# <desc> Detect OS (platform and version) of local machine </desc>
#
# <usage>
# source this file in bash, then run `osdetect`
# </usage>
 
 
_concat() {
	local s="$1";shift;
	while [ "$1" ]; do
		s="$s $1"; shift
	done
	echo "$s" | sed "s|^[ \\t]*||g" | sed "s|[ \\t]*$||g"
}
 
_glob() {
	local file=
	local glob=
	local lst=
	while [ -z "$file" ] && [ "$1" ]; do
		glob="$1";shift;
		lst=$(ls $glob 2>/dev/null | grep -v /etc/lsb-release)
		if [ "$lst" ]; then
			file=$(echo "$lst" | head -n1)
		fi
	done
	echo "$file"
}
 
osdetect() {
	# ref: http://linuxmafia.com/faq/Admin/release-files.html
 
	local os=
	local release=
	local machine=
	if ! which uname &>/dev/null; then
		echo -e "${cred}No uname on system${creset}" >&2
		os="N/A"
	else
		os=$(uname -s)
		release=$(uname -r)
		machine=$(uname -m)
	fi
	if [ "$os" = "SunOS" ]; then
		os="Solaris"
		machine=$(uname -p)
	fi
	local platform="$(_concat "$os" "$release" "$machine")"
 
	# prefer lsb_release
	if which lsb_release &>/dev/null; then
		local id="$(_concat "$(lsb_release -i | sed "s|.*:||g")")"
		local rel="$(_concat "$(lsb_release -r | sed "s|.*:||g")")"
		local code="$(_concat "$(lsb_release -c | sed "s|.*:||g")")"
	elif [ -f /etc/lsb-release ]; then
		local id="$(_concat "$(grep DISTRIB_ID /etc/lsb-release | sed "s|.*=||g")")"
		local rel="$(_concat "$(grep DISTRIB_RELEASE /etc/lsb-release | sed "s|.*=||g")")"
		local code="$(_concat "$(grep DISTRIB_CODENAME /etc/lsb-release | sed "s|.*=||g")")"
 
	# find a file or another
	else
		local vfile=$(_glob "/etc/*-rel*" "/etc/*_ver*" "/etc/*-version")
		[ "$vfile" ] && local id=$(cat "$vfile")
 
		# distro specific
		[ "$vfile" = /etc/debian_version ] && [ "$id" ] && id="Debian $id"
	fi
 
	[ "$id" = "n/a" ] && id=
	[ "$rel" = "n/a" ] && rel=
	[ "$code" = "n/a" ] && code=
 
	local version="$(_concat "$id" "$rel" "$code")"
	[ ! -z "$version" ] && version=" ~ ${cyellow}$version${creset}"
 
	echo -e "Platform: ${ccyan}${platform}${creset}${version}"
}

Download this code: osdetect.sh

Even though bash is not my favorite programming language, I end up writing a bit of code in it. It’s just super practical to have something in bash if you can. I mentioned in the past how it’s a good idea to avoid unnecessary cpu/io while initializing the shell by doing deferred aliasing. That solves the alias problem, but I also have a bunch of bash code in terms of functions. So I was thinking the same principle could be applied again.

Let’s use findpkgs as an example here. The function is defined in a separate file and the file is source’d. But this means that every time I start a new shell session the whole function has to be read and loaded into memory. That might not be convenient if there are a number of those. networktest, for instance, defines four function and is considerably longer.

So let’s “compile to bytecode” again:

findpkgs ()
{
    [ -f ~/.myshell/findpkgs.sh ] && . ~/.myshell/findpkgs.sh;
    findpkgs $@
}

Download this code: lazyfuncionload_loadfunc

When the function is defined this way, the script actually hasn’t been sourced yet (and that’s precisely the idea), but it will be the minute we call the function. This, naturally, will rebind the name findpkgs, and then we call the function again, with the given arguments, but this time giving them to the actual function.

Okay, so that was easy. But what if you have a bunch of functions loaded like that? It’s gonna be kinda messy copy-pasting that boilerplate code over and over. So let’s not write that code, let’s generate it:

lazyimport() {
# generate code for lazy import of function
	function="$1";shift;
	script="$1";shift;
 
	dec="$function() {
		[ -f ~/.myshell/$script.sh ] && . ~/.myshell/$script.sh
		$function \\$@
	}"
	eval "$dec"
}

Download this code: lazyfuncionload_lazyimport

Don’t worry, it’s the same thing as before, just wrapped in quotes. And now we may import all the functions we want in the namespace by calling this import function with the name of the function we want to “byte compile” and the script where it is found:

## findpkgs.sh
 
lazyimport findpkgs findpkgs
 
## networktest.sh
 
lazyimport havenet networktest
lazyimport haveinet networktest
lazyimport wifi networktest
lazyimport wifiscan networktest
 
## servalive.sh
 
lazyimport servalive servalive

Download this code: lazyfuncionload_functions

So let’s imagine a hypothetical execution. It goes like this:

1. Start a new bash shell.
2. Source import.sh where all this code is.
3. On each call to lazyimport a function declaration is generated, and eval’d. The function we want is now bound to its name in the shell.
4. On the first call to the function, the generated code for the function is executed, which actually sources the script, which rebinds the name of the function to the actual code that belongs to it.
5. The function is executed with arguments.
6. On subsequent executions the function is already “compiled”, ie. bound to its proper code.

So what happens, you may wonder, in cases like the above with networktest, where several mock functions are generated, all of which will source the same script? Well nothing, whichever of them is called first will source the script and overwrite all the bindings, remember? It only takes one call to whichever function and all of them become rebound to the real thing. So all is well.

It seems to me that we have pretty good tools for managing files that aren’t changing. We have file managers that display all the pertinent details, they’ll detect the file type, they’ll even show a preview if the content is an image or a video.

But what about files that are changing? Files get transfered all the time, but network connections are not always reliable. Have you ever been in the middle of a transfer wondering if it just stopped dead, wondering if it’s crawling along very slowly, too slow, almost, to notice? Or how about downloading something where the host doesn’t transmit the size of the file, so you’re just downloading not knowing how much there is left?

These things happen, not everyday, but from time to time they do. And it’s annoying. A file manager doesn’t really do a great job of showing you what’s happening. Of course you can stare at the directory and reload the display to see if the file size is changing. (Or the time stamp? But that’s not very convenient to check against the current time to measure how long it was since the last change.) Or maybe the file manager displays those updates dynamically? But it’s still somewhat lacking.

Basically, what you want to know is:

1. Is the file being written to right now?
2. How long since the last modification?

And you want those on a second-by-second basis, ideally. Something like this perhaps?

Here you have the files in this directory sorted by modification time (mtime). One file is actively being written to, you can see the last mtime was 0 seconds ago at the time of the last sampling. Sampling happens every second, so in that interval 133kb were written to the file and the mtime was updated. The other file has not been changed for the last 7 minutes.

The nice thing about this display is that whether you run the monitor while the file is being transfered or you start it after it’s already finished, you see what is happening, and if nothing is, you see when the last action took place.

#!/usr/bin/env python
#
# Author: Martin Matusiak <numerodix@gmail.com>
# Licensed under the GNU Public License, version 3.
#
# <desc> Watch directory for changes to files being written </desc>
 
import os
import sys
import time
 
 
class Formatter(object):
    size_units = [' b', 'kb', 'mb', 'gb', 'tb', 'pb', 'eb', 'zb', 'yb']
    time_units = ['sec', 'min', 'hou', 'day', 'mon', 'yea']
 
    @classmethod
    def simplify_time(cls, tm):
        unit = 0
        if tm > 59:
            unit += 1
            tm = float(tm) / 60
            if tm > 59:
                unit += 1
                tm = float(tm) / 60
                if tm > 23:
                    unit += 1
                    tm = float(tm) / 24
                    if tm > 29:
                        unit += 1
                        tm = float(tm) / 30
                        if tm > 11:
                            unit += 1
                            tm = float(tm) / 12
        return int(round(tm)), cls.time_units[unit]
 
    @classmethod
    def simplify_filesize(cls, size):
        unit = 0
        while size > 1023:
            unit += 1
            size = float(size) / 1024
        return int(round(size)), cls.size_units[unit]
 
    @classmethod
    def mtime(cls, reftime, mtime):
        delta = int(reftime - mtime)
        tm, unit = cls.simplify_time(delta)
        delta_s = "%s%s" % (tm, unit)
        return delta_s
 
    @classmethod
    def filesize(cls, size):
        size, unit = cls.simplify_filesize(size)
        size_s = "%s%s" % (size, unit)
        return size_s
 
    @classmethod
    def filesizedelta(cls, size):
        size, unit = cls.simplify_filesize(size)
        sign = size > 0 and "+" or ""
        size_s = "%s%s%s" % (sign, size, unit)
        return size_s
 
    @classmethod
    def bold(cls, s):
        """Display in bold"""
        term = os.environ.get("TERM")
        if term and term != "dumb":
            return "\\033[1m%s\\033[0m" % s
        return s
 
class File(object):
    sample_limit = 60  # don't hold more than x samples
 
    def __init__(self, file):
        self.file = file
        self.mtimes = []
 
    def get_name(self):
        return self.file
 
    def get_last_mtime(self):
        tm, sz = self.mtimes[-1]
        return tm
 
    def get_last_size(self):
        tm, sz = self.mtimes[-1]
        return sz
 
    def get_last_delta(self):
        size_last = self.get_last_size()
        try:
            mtime_beforelast, size_beforelast = self.mtimes[-2]
            return size_last - size_beforelast
        except IndexError:
            return 0
 
    def prune_samples(self):
        """Remove samples older than x samples back"""
        if len(self.mtimes) % self.sample_limit == 0:
            self.mtimes = self.mtimes[-self.sample_limit:]
 
    def sample(self, mtime, size):
        """Sample file status"""
        # Don't keep too many samples
        self.prune_samples()
        # Update time and size
        self.mtimes.append((mtime, size))
 
class Directory(object):
    def __init__(self, path):
        self.path = path
        self.files = {}
 
    def prune_files(self):
        """Remove indexed files no longer on disk (by deletion/rename)"""
        for f in self.files.values():
            name = f.get_name()
            file = os.path.join(self.path, name)
            if not os.path.exists(file):
                del(self.files[name])
 
    def scan_files(self):
        # remove duds first
        self.prune_files()
        # find items, grab only files
        items = os.listdir(self.path)
        items = filter(lambda f: os.path.isfile(os.path.join(self.path, f)),
                       items)
        # stat files, building/updating index
        for f in items:
            st = os.stat(os.path.join(self.path, f))
            if not self.files.get(f):
                self.files[f] = File(f)
            self.files[f].sample(st.st_mtime, st.st_size)
 
    def display_line(self, name, time_now, tm, size, sizedelta):
        time_fmt = Formatter.mtime(time_now, tm)
        size_fmt = Formatter.filesize(size)
        sizedelta_fmt = Formatter.filesizedelta(sizedelta)
        line = "%6.6s   %5.5s   %6.6s   %s" % (time_fmt, size_fmt,
                                               sizedelta_fmt, name)
        if time_now - tm < 6:
            line = Formatter.bold(line)
        return line
 
    def sort_by_name(self, files):
        return sorted(self.files.values(), key=lambda x: x.get_name())
 
    def sort_by_mtime(self, files):
        return sorted(self.files.values(),
                      key=lambda x: (x.get_last_mtime(),x.get_name()))
 
    def display(self):
        time_now = time.time()
        files = self.sort_by_mtime(self.files.values())
        print("\\nmtime>   size>   delta>   name>")
        for f in files:
            line = self.display_line(f.get_name(),
                                     time_now, f.get_last_mtime(),
                                     f.get_last_size(), f.get_last_delta())
            print(line)
 
 
def main(path):
    directory = Directory(path)
    while True:
        try:
            directory.scan_files()
            directory.display()
            time.sleep(1)
        except KeyboardInterrupt:
            print("\\rUser terminated")
            return
 
 
if __name__ == '__main__':
    try:
        path = sys.argv[1]
    except IndexError:
        print("Usage:  %s /path" % sys.argv[0])
        sys.exit(1)
    main(path)
 

Download this code: peek.py