Extracting a Bunch of .gz Files in One Go using Bash’s For Loop While Keeping Originals

While compressing a set of single files is fairly easy (one only needs to add ‘.gz’ or similar to the end of “file name strings”) the oposite is not so easy. Nevertheless it can be done doing:

for i in *.gz; do echo extracting $i ...; sudo su -c "gzip -dc $i > ${i:0:${#i}-3}"; done && ls *xpm

sudo su -c "" is for Ubuntu systems, ${#i}-3 sais “length of i minus 3” which makes ${i:0:${#i}-3} to become each file name without trailing .gz.

Ressources

  • Bash by Example
  • man 8 bash section “Parameter Expansion” -> “Substring Expansion”

About the fuzz on rootkits and whether or not to detect one

In the last couple of days I have been reading and hearing about Rootkits and the panic that comes with it. Mainly on German forums and sites, although also e.g. on Joanna Rutkowska’s blog (author of the “famous” Bluepill hijacker technique). And it kept me thinking. But first let me summerize what I understand the fuzz is all about.

A Rootkit is some sort of malware. Depending on whom you ask or enlist it is a piece of software running on someone’s computer — preferably with an Internet connection — without the user or even administrator knowing. I understand the definition itself so that this program does not have to be hiding itself in the memory and/or on the hard drive from detection software but it may (regardless of the, to my knowledge and despite Joanna’s work, unanswered question if potentially it can do so at all). Rootkits like any other malware have to be transferred to the target computer in some way or another and are — the hiding once like any other — detectable in this non-executed state (via digital signature for example). Ones primed, i.e. executed, the code becomes a process in the computer’s memory and tries to hide itself with various methods in memory and hard drives (potentially also MBR or even BIOS, but as far as I read/heard non have been reported so far).

Another factor of Rootkits is that they most often start with a small subset of code/features/routines and, ones residing in memory, recruit more and more features via the computer’s net link through a so-called back door. The back door part is why the differentiation from Trojan Horses is blurry. I’d say the Trojan Horse technique is only one of many features of such a Rootkit but that doesn’t make it a Trojan Horse since it’s not all it can do.

One other of the many possible features, and first shown by the before named Bluepill, is to become a hypervisor (think of it as a sandbox for OSs) like Xen (virtual machines like VMware, Qemu/VirtualBox work differently). The fancy bit about bluepill’s method is that, while active, the OS’ kernel is virtualized, i.e. becomes a guest OS from being host OS before; Microsoft Vista kernel here. It’s done by forcing to swap kernel parts to pagefile.sys which than are modified on disc — no Vista kernel protection — and loaded back to memory. Let me point out: On-the-fly, no reboot or BIOS or MBR modification necessary! That means that the malware runs below the OS or, rephrased the other way around, the real OS runs on top of the malware.

From Darkreading:

The new Blue Pill comes with support for so-called “nested” hypervisors (think Blue Pill within a Blue Pill), and uses an architecture similar to that of the open-source Xen 3 virtual machine technology. It comes with “on the fly” loading and unloading features, as well as more features for avoiding detection, such as hibernating and temporarily uninstalling the hypervisor when Blue Pill detects that a tool is about to detect it.

Let me add: This utilizes Pacifica specification of AMD’s newer processors which have virtualisation technology (VT) build-in. It just has been started on AMD processors but there are also implementations for Intel processors with similar techniques.

Having said all that I came to think of how could it still be possible to detect and what are the remarkable bits here. Let me also point out that I am by no means an expert on anti virus, Rootkits, hypervisors or any of that. I just know a some basic, though advanced, computer issues, how they basically work, about TCP/IP stuff and Linux OS basics. And I claim to have common sense 🙂

Ideas mentioned elsewhere to encounter the issue and comments on them:

Ok, now there is one point that is not technical at all: How do I detect something that can hide (let’s presume so) from a running system if I don’t see it and wouldn’t get alerted by any detection software? Imagine working on your computer and thinking: “Am I infected? Let’s check and boot to this detection LiveCD [see below]… checking… Good, not infected, so reboot to work system… keep on working… Oh, an now? Infected now?… LiveCD check… reboot to work, since still not infected… work a little… Hah, now is the time, I could now be infected… reboot….”.

The rumours are that it would be easy to detect the malware hiding on active systems when the system is dormant, i.e. not booted, e.g from some LiveCD. That’s one point I could believe to be true to some extant since I guess the malware has to have saved itself on system shut down to some place on the hard drive, BIOS (graphic card’s one, too), or some kind of non-volatile memory and, more importantly, cannot defend, i.e. hide, itself actively. But as with any malware detection by signature the signature of such “saved state hiding malware” has to be known which might be hard since it’s easy for malware to change it’s “saved state form” and thereby it’s signature. And also, is it handy and operably in real life to shut down, eg., servers “only” to detect potentially infected systems (again, assuming all the while it’s not possible to detect while the system is active)?

If it’s possible to only have one hypervisor (what I don’t know right now) then wouldn’t it be easy to just check if a hypervisor is present or can be enabled. If not because one is present already but not known about by the system -> suspicious. Matasano‘s virtualized rootkit detector most likely is about even more than that (from Hacker Smackdown, June 28th, 2007):

Ptacek, Lawson, and Ferrie contend that virtualization-based malware is actually easier to detect than a normal (non-virtualized) rootkit because basically by definition it leaves a trail, introducing changes in the system’s CPU clock, for instance. And the malware would have to be bug-free to truly emulate a system, anyway, Ptacek argues. “The problem with virtualized rootkits is… They have to present the illusion they are talking to real hardware and that’s not an easy task,” he says. “In order to do that, you have to write a bug-free program whose job it is to emulate bugs. And we don’t know how to write bug-free programs.”

One very simply (that’s why I liked it!) detection method described in a German forum was to simultaneously do an outside port scan and ask the system “from inside” for open ports. Most likely the malware will show an open port to the outside (it wants to receive data here) but will hide this port to the system running the malware.

Ideas I haven’t read about so far or are not related directly but rather with malware in general but still fairly new:

  • As a basic approach (operating) systems have to be transparent (best I know of open source) for experts to know what’s going on inside and users to trust “their” system. This is no new argument I assume.
  • Digital signature (public/private keys) handling in kernel for processes similar to what I believe Vista does but holistically and, again, transparent. The idea is similar to that Debian (and other distributions since) have been using with their repositories and dpkg/apt system for years now but now within the computer itself. SecureApt as it’s called uses MD5 checksums (switch to SHA-1 when MD5 is broken) to uniquely and securely identify software packages retrieved from Internet repositories and to verify data (read byte stream) is unchanged on the way from the maintainer to the user’s computer. On top of that SecureApt uses OpenPGP (with GPG) private keys to sign repositories release summaries and public keys to verify the signature, i.e. deciding whether a repo is trusted or not. Why not taking this one step further to the kernel itself and have a module in the kernel implementing the idea of SecureApt but for processes (instances of programs from those repositories)? Though, I guess with quantum computers approaching this prevention method most likely will not hold long anyway.
  • Security systems (German) like AppArmor or even better SELinux should be used more widely to protect more systems better from so-called 0-day attacks and the like. And thereby limit distribution of malware. These two methods, of course, do nothing to increase detection on harmed systems. It only prevents from becoming infected.
  • Don’t by VT supported processors if you don’t need to. This, as with all security issues, will not work on a wide range since it’s more convenient to benefit from supposingly up to 95% performance enhancement for so-called paravirtualized guest systems (more precisely domUs). At least if you need to run unmodified OSs like MS Windows. If you can however modify the domUs, eg. Linux, you can have the same performance with eg. Xen. Let me point out that unlike virtualized guest operating system with paravirtualization the domU does know about it being virtualized and can, among others, access hardware directly.
  • Another idea on how to become suspicious of possible infections includes a second system with net link to computers at risk. I’d call it a watch server or pass-through server. Maybe it could just be your firewall of choice. The idea is to watch the traffic from an to computers in your network just like a firewall does but watch for and learn some sort of network traffic signatures or patterns. This way you get a (statistical) profile of typical traffic regarding individual systems independent of applications running, user behaviour or or the like. Just plain network traffic. This, of course, has to be done while one is certain of no infections in the network. If one can guaranty this it could be possible after this learning phase to detect suspicious traffic.

Maybe everything said here is not new at all to others. But one thing I reckon will be true: After all it will always be a game of cat-and-mouse, since the bad guys will try to detect methods like those mentioned here to hide themselves and the good guys will always try to be smarter. The most interesting part I find about self hiding malwares is that malware is turning the tables now (well, not entirely): With conservative viruses it was evolving new techniques unknown to the anti-virus guys. Now it’s (partly) malware becoming virus-detection-detectors.

And one other thing once again became clear to me: The need for researchers to “do bad things”, i.e. to develop, test, execute, issue and whatever else necessary malware of whatever kind to be able to come up with antidote! Unfortunately there are movements on the way in Germany (German, heise, 06.07.2007 14:23) and as I understand in other parts of the would, too, to prohibit this.

Happy hacking 😉

Update 2007/10/11:

In slashdot there has been a note on VM-Based Rootkits Proved Easily Detectable pointing out an article from researchers from Stanford, CMU, VMware, and XenSource “Compatibility Is Not Transparency: VMM Detection Myths and Realities” (pdf). Unfortunatelly, untill now I haven’t had the time to read it.

Resources:

How to find out what occupies space on your Linux hard drive

The other day I noticed that my settings directory (/etc) uses over 13 MB of my hard drive. So I wandered which package (I’m using a Debian based package managed system) makes the settings directory grow so large. After a couple of trails and errors I came up with the following sequence of commands:

$ du -h --max-depth=1 /etc 2> /dev/null | egrep '(^[5-9][0-9]{2}K)|M'
692K    /etc/X11
672K    /etc/acpi
712K    /etc/xdg
2.1M    /etc/brltty
500K    /etc/ssl
528K    /etc/mono
20K     /etc/NetworkManager
13M     /etc
$ dpkg -S '/etc/brltty'
brltty-x11, brltty: /etc/brltty
$ apt-cache show brltty | grep -A5 'Description'
Description: Access software for a blind person using a soft braille terminal
BRLTTY is a daemon which provides access to the Linux console (text mode)
for a blind person using a soft braille display.  It drives the braille
terminal and provides complete screen review functionality.
The following display models are supported:
* Alva (ABT3xx/Delphi)

Fortunatelly, I’m not blind so I could remove brltty with aptitude which then suggested to remove dependencies, too.

References:

  • Regex reference
  • Resources for advanced Ubuntu topics, eg. how to remove …-desktop meta packages with apt-get (instead of aptitude), secure networking setup, etc.

Ubuntu: Mounting remote filesystem using davfs2 (FUSE)

If you have access to some webdav server you might want to give your system access to those files as if they were local ones so you don’t have to use some interactive application every time you need access. FUSE is very useful for that very task, also because it for user space (you don’t have to be root to mount it). After this set up it’s meant to work for any application that works on that webdav directory files just the same as they would on the local (read: hard drive) file system. What needs to be done:

  1. Install davfs2 package (you might use Synaptic instead):
    $ apt-cache search davfs2
    davfs2 - mount a WebDAV resource as a regular file system
    $sudo aptitude install davfs2
  2. reconfigure the package since it needs to run suid if normal users should be able to use it:
    sudo dpkg-reconfigure davfs2

    davfs2 SUID dpgk-reconfigure

  3. After confirming to SUID davfs2 select a user group, e.g. “davfs2”:
    davfs2 group dpgk-reconfigure
    davfs2 infoscreen dpgk-reconfigure
  4. make a mount point, i.e. a directory where the “file system” is hung into (directory webdove in a subdir of your home):
    mkdir ~/mnt/webdove
  5. to testmount use something like (use quotes to tell bash to keep it’s hands off it):
    sudo mount.davfs 'http://domain.tld/path' /path/to/webdove

    You will be prompted for user and password

  6. To allow regular users access I could only find a way where one needs to touch /etc/fstab to add a line like this one:
    http://domain.tld/davath /path/to/webdove   davfs   user,rw,noauto   0   0

    Now any user can do mount mount /path/to/webdove and umount /path/to/webdove

From the man page:

If a proxy must be used this should be configured in /home/filomena/.davfs2/davfs2.conf
proxy proxy.mycompany.com:8080

Credentials are stored in /home/filomena/.davfs2/secrets
proxy.mycompany.com filomena “my secret”
http://webdav.org/dav webdav-username password

Note: If your webdav server supports https, i.e. encrypted transfer you might use that as well. Just replace http with https above.

Even though this works and does enable the user to mount a webdave server by himself it doesn’t integrate very well into Ubuntu (as I understand it). For example the user can’t choose where to mount it. Also, there is a lot that needs to be set up correctly by the admin. I really would like to hear comments to point me to other, easier solutions (see below). A good example for user friendliness would be sshfs.

Update 2008/05/08: A nice and working description about mounting the (Germany-based) GMX-Mediacenter via secure webdav2 I have listed below. Hopefully some day I will find the time to summerize it here as it is written in German.

Resources:

Ubuntu: Using closed-source application securely with AppArmor

If you have closed-source applications installed like Opera (I do), Skype, or whatever than AppArmor should be engaged. Especially for Skype Linux it’s irresponsible without it, since Skype Linux reads /etc/passwd, Firefox profile and other files. For Ubuntu Feisty it’s meant to be in Universe and in Gutsy it will be installed on default (without profiles, though). On the community help there is an instruction on how to install and use it for Feisty and Gutsy.

References:

Ubuntu: Tweak Logitech’s DiNovo Bluetooth Keyboard and Mouse

Eventhough with Ubuntu’s Feisty Fawn release I plugged in the BT Dongle and the mouse and keyboard was working right away; even the main multimedia keys. But still there are some problems like a “sleeping” mouse pointer every now and than, some mouse buttons, e.g. the thumb ones, don’t work as intended.

Update 2008/07/14: There seam to be possibilities to even get the LCD of DiNovo’s keypad working. So far I haven’t found a way to auto synchronize it as it does with Logitechs Windows driver.

Ressources:

Ubuntu: Mounting remote filesystem using sshfs (FUSE)

Wouldn’t it be nice and handy to go to your local home directory and from there just cd into a remote one (say university stuff or via WLAN or other sometimes unsecured lines) as if it were local data? Of course there is NFS or GNOME’s network folders (that use ssh; Places -> Connect to Server…) and I guess there are heaps of other ways to do it. I chose the sshfs way because it’s

  • easy to set up
  • only needs client side (local) side preparations
  • can be set up and mounted entirely by a “normal user”
  • data line is encripted just as ssh is (because data does go via ssh)

So, what needs to be done? I’d just list the steps with only the necessary explanation. For further introduction see below.

  1. sudo aptitude install sshfs
  2. via lsmod | grep fuse see if the fuse module is there. Otherwise modprobe it (sudo modprobe fuse).
  3. see if you user name is listed in the fuse user group: grep fuse /etc/group. If not do sudo adduser yourusername fuse. You might need to logout and log back in in order for this change to take effekt.
  4. ls -la /dev/fuse should give you crw-rw---- 1 root fuse.... The ownership root:fuse is important. If not, do sudo chown root:fuse /dev/fuse
  5. Now create the mountpoint: mkdir ~/unihome
  6. Actually mount the remote fs (syntax is like the one from ssh or scp): sshfs remoteuser@remotehost:remotepath ~/unihome . If no ssh-key stuff is configured you’ll be asked your remote password

You now can cd ~/unihome or otherwise use the data there as if it was local. To unmount the remote data do fusermount -u mountpoint. Here it would be fusermount -u ~/unihome.

More comfort

To make your daily life easier you can add a file called config to your (local) home’s .ssh directory with the following lines (insert you personal data):

Host wsl01         Hostname remotemachine's-name-or-ipUser remoteuser

After that you can shorten the mount command to sshfs -oreconnect wsl01: ~/unihome to mount the entire home directory (see bottom for why -oreconnect). Of course this only works for ssh’s default to go straight into your home directory after login. From sshfsfaq:

Automatic mounting, if desired, can be added to a shell script such as .bashrc (provided authentication is done using RSA/DSA keys).

See Kevin van Zonneveld’s Blog for how to setup everything to automatically login using ssh (and thus sshfs) without beeing promted for a password. But beware not to give anyone access to your private key file (see Kevin’s note under “Pitfalls” at the bottom)! Even though the key is user and machine specific anyone that gathers access to your machine and your user can hop to the remote machine with your remote login as well. After done generating and installing the keys you need the mount command from above in your .bashrc file in your home directory. It will be unmounted on system shut down or logout.

Now, you’re done.

Update: Tweak timeout

I’ve experienced several disconnects when the connection has been idle for to long. So I digged into it. From man 5 ssh_config:

BatchMode
If set to “yes”, passphrase/password querying will be disabled. In addition, the ServerAliveInterval and SetupTimeOut options will both be set to 300 seconds by default. This option is useful in scripts and other batch jobs where no user is present to supply the password, and where it is desirable to detect a broken network swiftly. The argument must be “yes” or “no”. The default is “no”.

(…)

ServerAliveCountMax
Sets the number of server alive messages (see below) which may be sent without ssh receiving any messages back from the server. If this threshold is reached while server alive messages are being sent, ssh will disconnect from the server, terminating the session. It is important to note that the use of server alive messages is very different from TCPKeepAlive (below). The server alive messages are sent through the encrypted channel and therefore will not be spoofable. The TCP keepalive option enabled by TCPKeepAlive is spoofable. The server alive mechanism is valuable when the client or server depend on knowing when a connection has become inactive.
The default value is 3. If, for example, ServerAliveInterval (see below) is set to 15, and ServerAliveCountMax is left at the default, if the server becomes unresponsive ssh will disconnect after approximately 45 seconds. This option works when using protocol version 2 only; in protocol version 1 there is no mechanism to request a response from the server to the server alive messages, so disconnection is the responsibility of the TCP stack.

ServerAliveInterval
Sets a timeout interval in seconds after which if no data has been received from the server, ssh will send a message through the encrypted channel to request a response from the server. The default is 0, indicating that these messages will not be sent to the server, or 300 if the BatchMode option is set. This option applies to protocol version 2 only. ProtocolKeepAlives is a Debian-specific compatibility alias for this option.

So, I added a line in my .ssh/config file saying BatchMode “yes”. This, per default, gives the line $((300 / 60)) = 5 minutes (bash simple math, use with echo on the command line) until the ssh connection is dropped.

Update 2: Automounting

Add a line like the following to your /etc/fstab file (open in graphical mode with gksudo gvim /etc/fstab):

# <file system>       <mount point>         <type>  <options>
sshfs#wsl01:         /mountpointpath            fuse    optionsset 0 0

Remember to adopt the bits written itelic, i.e. wsl01, the path to your mount point and the options. A typical option set could be comment=sshfs,users,noauto,uid=1000,gid=1000,allow_other,reconnect,transform_symlinks. It’s a mixture of basic mount options and fuse and sshfs, respectively, specific options. The main ones are:

  • users: anyone can mount this filesystem
  • noauto: don’t mount automatically on system start up since network is not up, yet
  • uid=1000,gid=1000: since mount is not run with your uid/gid this is needed (find out the numbers with id command)

Now configure fuse by using /etc/fuse.conf (infos locally in less /usr/share/doc/fuse-utils/README.gz). Add user_allow_other to be able to use the fstab option allow_other.

I was writing this section in parallel while testing it myself. And I suddenly noticed it’s not what I was looking for (which was auto reconnect). More so this seams less secure than the original since with this any local user could mount it. The only advantage was to have icons on the gnome desktop (because it’s in the fstab) or if you wanted to auto mount on network up/down. See the original forum post for how to do that.

Automatic reconnect is easily done by using the -o reconnect option with sshfs: sshfs -oreconnect wsl01: ~/mountpoint.

References:

Is just my WordPress.com blog broken or is there a general server problem? — update

Wordpress broken. Red header in admin sectionSince I was editing a post yesterday night (around midnight CET) there is a red banner at the top of my wp-admin area and it shows “WordPress.com Blog” instead of my own. I can edit post, create new ones (like this one) about fine; with the exception that the tinyMCE hangs when after hitting the ankor icon ( see picture )… I was just about to insert the screen shots as I noticed I have 10GB of upload space, I can upload more file formates, among others mp4, avi, ogg, but I cannot use them! I cannot upload anything anymore. After the upload process in the area where normally one gets the option to insert the picture into the editor it gets redirected to the wordpress.com front page. Could there be a bug in wordpress? I did try to upload a larger (but <50MB) mp3 fileSuddenly I got 10GB upload space but cannot use it earlier on today. But that could definitely not be the reason for things not to work properly since the problems started to occur already yesterday night as said before. What should I do? I hope this can be viewed at all (e.g. via direct links from search engines), since calling sysblogd.wordpress.com puts me to the wordpress.com front page again. No way to show my blog’s content! I guess I’ll just have to wait a little until the support is open again (it shows “Support Temporarily Closed” now).Update: I seam to have found the bug: As I changed the theme from Freshy (I chose Ambiru for starters), which I have used over a few month now, everything seams ok. Also, the red header is gone. So let’s try to upload the pics I promised above… works again, too. When I think about it, as I was collection of Story of the year videos using copy/past with firefox I had something like the following in my post’s code:

[dailymotion id=<code>5zYRy1JLhuGlP3BGw]</code>

This must somehow have messed up the theme Freshy. The conclusion for me is: Don’t rush or even push solutions you don’t at least feel if not know they are just right for the problem. If you feel an obstacle in the way, what-so-ever, let it rest. The solution will come flying to you when it’s write. And also, the well stressed but never-the-less true saying “There lies a chance in every problem”: Now I found an even better theme for my blog 🙂

Securing WordPress: Quick links I found

Just a couple of links for securing wordpress:

Ubuntu: Wine and PortableApps

Since I have been using PortableApps (especially Thunderbird) for a while I figured I’d use wine to avoid configuring email accounts and all that hassle. All it was, after collecting some information (I should have found that page before even installing Ubuntu!), is this:

  1. Get the PortableApps somewhere writable (I made a new ntfs partition using fuse-utils and ntfsprogs) — I have enabled compression on my Windows XP NTFS partitions which no Linux NTFS driver is capable of at the moment.
  2. Mount the partition eg. via sudo ntfsmount /dev/hda2 /media/portablestuff -o uid=1000
  3. Install wine: sudo aptitude install wine
  4. Run any program via command line something like this: $wine /media/portablestuff/path/to/ThunderbirdPortable/ThunderbirdPortable.exe
  5. To unmount use sudo fusermount -u /media/portablestuff

« Older entries Newer entries »