Archive for the 'Sysadmin' Category

Argument list too long

Published
by
David
on April 4, 2008
in Linux, Seenly and Sysadmin
.

While moving Seenly to a new server with more capacity, I had to upload a huge amount of tiny files. Trying to do ncftpput * in a directory that contains 27000 files each with a filename of 35 characters makes Bash cry, and spit out “Argument list too long”.

Xargs to the rescue:

echo * | xargs ncftpput

Amazingly, this works: Instead of trying to ncftpput everything at once, xargs executes ncftpput repeatedly with a group of files, until all files have been sent. Echo doesn’t throw an Argument list too long error because it’s a shell built-in. This basically works for every command you need to execute on a large number of files.

Watch out though, because the command isn’t interactive anymore. So if you get this error:

Error: gl_getline(): not interactive, use stdio.

It means it’s trying to get interactive, e.g. asking you for a password, while it’s not allowed. So make sure you set all required options using switches:

ncftpput -u user -p password hostname remote/path

Seenly’s Move

Published
by
David
on April 1, 2008
in Seenly and Sysadmin
.

A Seenly PortraitWith 31000 snapshots uploaded, Seenly reached the limits of the 1.5GB of webspace provided by the previous webhost. A move to a new host was inevitable. After some looking around, I decided I’d try 1-eurohost. For readers from PlanetUgent, that’s fellow blogger Stein van Stichel’s company (Dutch link).

Since Seenly inherently has some quirky requirements, I first had to make sure they were running the correct version of PHP. Every PHP before 5.2 contains a bug that truncates uploaded binary data at the first null byte (apparantly PHP was parsing it using a C string function). Seenly sends JPEG data as binary data, so this would mean images would never be complete. Stein informed me with a quick mail that they were running 5.2.1, and I was happy.

Great, go ahead, order the damn thing. Turns out they’re running a lot more security “upgrades” than my previous host, such as base_opendir and mod_security. Base_opendir made my subdomains useless: I couldn’t create a file in files.seenly.com from a php-script running at www.seenly.com. I had to redirect all my subdomains, and modify a lot of files but it was doable. Still, maybe this is needlessly strict?

Mod_security dismissed the multipart/form-data upload I generate in Flash. This was a problem, since doing a multipart/form-data in Actionscript is still a bit of black magic, and fixing it might not even be possible. Luckily a friend gave me the idea that I might be able to disable mod_security in a .htaccess. A quick SetSecFilter Off did the job, and everything is now purring happily along.

Moving to 1-eurohost I first had some doubts, since my previous host has been very good to me. Imagine my surprise when I noticed Seenly’s speed has doubled now, even though I’m paying less. The only disadvantage is that they don’t have shell access, which would have been handy when I wanted to upload 124000 files. Uploading just a tar file, and then untarring it on the server, would have been so much easier. More on how I managed that feat in a next entry.

Poor Man’s Cron Trigger

Published
by
David
on March 31, 2008
in How To, Linux and Sysadmin
.

This is a handy trick I use when I want to execute something as root when some event triggers, but without any interaction between the event and the start of the action. It’s quite unsafe and dirty, but it gets the job done.

When the event happens, make it create a file:

touch /tmp/this-event-happened

Now add the following cronjob as root (or whatever user you want):

* * * * * /bin/rm /tmp/this-event-happened 2>/dev/null && /bin/bash\
/execute/this.sh

First come the stars, which tell cronjob to execute this command every minute. Then we try to remove the /tmp/this-event-happened file. rm will exit with an error-code when the file doesn’t exist, and in that case the script won’t be executed, because we’re chaining it using “&&”, the AND operator. In a shell, and most modern programming languages, if the first parameter of an AND operator is false, it won’t even evaluate the second parameter, because the total result will definetly be false anyway.

If the file does exist, the shell will evaluate the second parameter as well, and thus execute the script. Meanwhile the file will be removed, so everything is set up for the next trigger.

As you can see, very simple. It has some problems though. For example, any user can create files in /tmp, so it’s not very safe. You could create a directory somewhere that’s only accessible to the user that is allowed to generate the event, and put the trigger-file there. Another disadvantage: if the script takes longer than a minute to run, and the event is triggered again, the script will run twice at the same time. So take care when using this!