Meta: New Minisleep, site rebased

Tales of converting bash to POSIX shell scripts and a picture of my new crazy software sideproject.

Minisleep 1.1 is out:

http://halestrom.net/darksleep/software/minisleep/

There’s now a 60 second video demonstrating some of the wiki’s features. See the changelog for more interesting notes. No easter eggs are in this documentation.

Porting from Bash to POSIX sh

I thought this would be easy :D

Testing isn’t as trivial as you might think, as often bugs caused by a shell changeover are not syntactic but behavioural. Example: some features of minisleep used to read the contents of files into variables like so:

avariable="$(<afilename)"

This works fine on bash, and it even passes checkbashims fine, but it results in an empty variable when used on dash.

Instead you have to:

avariable="$(cat afilename)"

‘cat’ tends to be an external binary, not a shell builtin. Sometimes it’s worth putting the effort into avoiding external binaries to make your code go faster, but Minisleep still takes less time to render pages and do its work than most site backends I have tried, so I don’t think it’s too much of an issue. Runtime seems to be dominated by disk IO, especially on my shared host.

Lack of substring operations

This drove me mad! A basic feature of a language should be selecting substrings; but POSIX sh does not support it. Instead you have to print data out and pipe it into external tools like cut, head or tail. Grrr.

Printing untrusted data is more complex than you think

Many people believe that you should not use echo to print untrusted data:

echo "$untrusted_data_from_a_user"

The data may contain options for echo (like ‘-n’ or ‘-e’). I’m not aware of any way that this can be exploited generically to execute commands or crash your scripts, but it introduces controllability for attackers that you don’t want to have to even think about.

Instead it’s generally recommended that people use printf like so:

printf '%s' "$untrusted_data_from_a_user"

It’s important not to provide the untrusted data in the first (format) string for similar reasons: %20s might seem harmless, but there are so many ways it could stuff things up (URLencoded strings anyone?).

What I didn’t realise is that GNU coreutils printf (the version you probably have if you are on Linux) supports “—longoptions” like many other GNU tools. Take this command for example:

$ printf '-- check logs --'
bash: printf: --: invalid option
printf: usage: printf [-v var] format [arguments]

Brr. Sometimes I wish I was writing in C, with its strict function arguments and strict datatypes. Alas there are some very solid reasons Minisleep is in shell: it’s the lowest common denominator on cheap webhosts, so software written in it can work out of the box. I have not had such luck with:

• C-based backends. My current host has an odd setup where external binaries will not run (odd glibc?), you have to use their toolchain, something that you can only get access to on request and temporarily.
• Perl. Path config typically messes with me.
• Python. Multiple versions and they’re not necessarily installed on your shared host.

PHP is a good contender, but I don’t have any experience with it. From what I have read it’s as evil as shell >:D. It also has multiple common versions out there across various shared webhosts.

Sidenote: bash isn’t completely immune to being different across hosts. I discovered that some have versions that don’t support multi-digit file descriptors. I don’t think many people will care about this feature, but it bit me unexpectantly once when I was using fd 666 for file locking.

Bad line reporting for syntax errors

Messages along the lines of “EOF hit early” or “missing a closing quote’ are absolutely evil. 9/10 times I have hit these errors it has been because of mismatched brackets some 100 or so lines of code before the line the error is reported for:

 culprit_evil="${printf '%s' "$hibiscus" | tr -d '\n\r' | sed "s|moo|cow|g")" 

Squint if you can’t see the error. The only practical way to finding the culprit line is by deleting/stubbing chunks of code (binary searching by dissection) until the error message disappears.

Bash sometimes gives better errors than dash, but not always.

Was it worth it?

Performance: no. The difference has been minimal. Some tests I have run show that starting bash is a bit slower than starting dash, but the disk IO penalties of work seem to outweigh everything else.

For an interesting tangent to performance: see my performance comparison of sed versus bash’s inbuilt find-and-replace string feature on minisleep’s webpage. Bash must be doing something very unexpected here.

Interoperability: perhaps. I’m not sure whether or not BSD, haiku and other OS users will ever want to use my code, but it’s comforting to know they have a better chance now. Any step to make a project more portable is a good step, especially in terms of code longevity.

Rebasing this website on the latest minisleep

This has closed a few holes and generally made me feel a lot better. Adding comment and throttling support back in (just for this website, not publicly released) took much longer than expected.

Still todo: Email subscription support for commentors, so they can be informed when new comments are added.

What to expect soon

• Lots of lab projects with pretty pictures.
• Boring and drawn out code musings.
• Perhaps some more code/project releases.

Over in the land of feed readers

My new sidequest: a non-crappy feed reader webapp:

I’ve previously found that QuiteRSS is a nice desktop feed app, but it presents me with a few issues:

• It uses webkit. To view untrusted content from the web. Hmm.
• I can’t access it from multiple computers.

RSSguard fixes the first issue (it has a very basic understanding of HTML) and is super-fast to launch, but it is ridiculously slow to update feeds by comparison. During the feed update process many parts of the application are disabled too.

To try and fix all of my issues I started looking at the self-hosted webapp options. This would allow me to use any computer to view my feeds without having to guess at which ones are new/unread.

Unfortunately all of the webapps I have tried suffer from various problems:

• Complex to setup and run. Databases, lists of dependencies, multiple setup attempts before everything is OK.
• Unable to provide both a global list of articles or
• Being designed around ‘background updating’ of feeds. I use my feed reader in “sessions” and I don’t necessarily have a computer always running at home 24/7 for this to be daemonised on.
• Horribly over-padded interface by default. This is often fixable, but it gives me a bad feeling about what the app is targeted toward.

Bastardfeed is my attempt to work around this. Some haphazard notes about it:

• Goals:
  • to be the fastest feed reader out there. Both in terms of fetching and processing.
  • to be simple and lightweight
  • to use a tree filesystem (files and folders) rather than a relational database (SQL)
  • to be self-contained (minimal external deps)
• Uses an absolute bastard of a feed processor, written in C.
  • Full XML parsing? Pah, who needs that?
  • Understands tags about 1-2 layers deep, depending on the situation
  • Tuned to work on my (long) list of feeds.
  • Surprisingly this angle of attack seems to be working!
• Spends too long creating page indexes (lots of shell calls to external binaries like sed and date). This component is being rewritten into C, soon maybe the whole project will be like this.
• Has troubles with some favicons (notice the blanks in the picture above). Mimetype issues, I’m hoping to avoid needing to use ‘file’ to determine filetypes.
• Haphazard built-in HTTP server, with half-arsed CGI support, in the form of a shell script + socat.
• Doesn’t track article read status, relies on your web-browser history (Firefox sync’d in my case)
• Aggressively fetches feeds in parallel
• Streams the feed-fetch log over HTTP to your browser, so you can sit back with popcorn and watch the bits fly.
• Highlights new articles in feeds (red numbers on sidebar)
• Doesn’t “interleave” articles on the front (mixed feed) page. When a feed gets multiple new entries then they get clumped together, even if this is not the correct method of time-sorting them. This was initially done to simplify things, but I have now grown to like this behaviour — when someone releases a few articles then I tend to want to see them in one place, not intermingled amongst other things.

If I’m insane enough I might release this. But for now I get some satisfaction (and chuckles) every day when I use it.