Batch Rename Multiple Files with Linux

Here is a list of ways how to rename multiple files with Linux. The easies and quickest way I found was to use ‘rename’.

Convert multiple Flash (flv) Videos to, for example, MPEG encoding

With ffmpeg and bash this is really easy. Also .flv and .mpeg are easily interchangeable by all other video codecs supported by ffmpeg:

#!/bin/bash
##convert all videos of $srctype type to $desttype type encoding
SRCCODEC="flv"
DESTCODEC="mpeg"
for i in *.$SRCCODEC; do
    echo
    echo -n "###################  "
    echo -n "will convert \"$i\""
    echo "  ###################"
    echo
    ffmpeg -i "${i}" -y "${i%%.${SRCCODEC}}.${DESTCODEC}" && rm -vf "$i"
done

Note that this includes removing the original video file only after a successful recode run overwriting existing destination files (-y option). The interesting part is this: ${i%%.${SRCCODEC}}. It removes the source’s postfix, i.e. file extension. You could save this as a text file, say convert_all_videos_in_pwd.sh and chmod +x filename. I have my own scripts go to ~/bin which I include in $PATH via ~/.bashrc. If you do so you would typically call this within the videos’ directory including logging all output and errors to convert.log, i.e. redirecting STDOUT (= file descriptor 1) and STDERR (= file descriptor 2) via &>:

convert_all_videos_in_pwd.sh &> convert_all.log &

If you want to see the output you could use tail -f logfile or read more on redirection and duplication. You could also do all that in a single command line:

for i in *.flv; do echo converting $i; ffmpeg -i "${i}" -y "${i%%.flv}.mpeg" && rm -vf "$i"; done

How to Move VirtualBox’s Guest Hard Drives to Another (Physical) Location

If you ever wanted to move your (very large) file that contains a Virtualbox’s guest hard drive (file extension is .vdi) to another location and did so just like that Virtualbox wouldn’t find it the next time you wanted to boot into your guest system. The only thing Virtualbox needs to be pointed to is the new location of that file.

The standart location of VDI files is (for Linux) ~/.VirtualBox/VDI/ (see global settings, where you could also change this if that’s all you want). Remember the old file name and close Virtualbox, just in case it would overwrite settings when closed after the following changes. Now let’s move one file to, say, /media/USBDrive/virtualbox/. Now do:

  1. Use your favorite editor to open ~/.VirtualBox/Virtualbox.xml
  2. Find the line that says something like:
    <VirtualDiskImage filePath="VDI/old file name.vdi"/>
    
  3. Change it to point to your new location. So it should read something like:
    <VirtualDiskImage filePath="/media/USBDrive/new file name.vdi"/>
    
  4. Done.

Making Apache server use SSL only

Here is a forum post on how to make all incoming connections to your apache web server redirect to https, i.e. take encripted connections via ssl by using apache’s rewrite module.

Recover Data and (deleted) Partition with Linux from Hard Drives, CD-ROMs or DVDs

One important thing to know before doing anything else:

Every byte written to partitions in question will decrease chances to get data back since deletion of files and even partitions does not automatically mean actually deleting that data on disk. In most cases it only means deleting administrative entries in file or partition tables. But writing new files or partitions on the very same disk might actually overwrite data blocks containing pieces associated with files still intact! The second thing you might want to consider: If you mistakenly deleted single files or even a bunch of them look somewhere else but better not reboot until you really know why you have to do so as rebooting in those cases will sometimes also mess up chances to get back your data. Here I will only handle partition or hard drive errors.

So the very first step will be to get hold of extra storage (like USB drives), boot up some Live CD, e.g. your Ubuntu installation cd. If you have liability to paranoia you might even want to physically detach your hard drive and use some USB case to access it instead. First of all do something like

ddrescue -vr-1 /dev/sda recovered_data_blocks.img gddrescue.log

using gddrescue. Adjust parameters to your needs, especially /dev/sda (complete drive) or /dev/sda1 (partition in question). For Ubuntu you need to install gddrescue (= GNU ddrescue) package first. gddrescue will create an image trying hard to copy every data block still intact so you can, at any point, restore the drive stage where you first discovered you lost data. This, or course, can be used with any block-oriented medium such as CD-ROMs or DVDs. You might want to specify “-b 2048″ for CDs. Also use -n to get the most data on first run. For subsequent attempts you might want -rX where X is the number of retries for erroneous areas. I recommend to use -vr-1 for verbose infinite retries. gddrescue first reads as many as possible intact blocks and then splits error areas with each subsequent run. Note though for partitions or even complete hard drives it will take hours so be patient and make sure to use gddrescue’s log file option! With other tools I tried it took even more time which makes sense since gddrescue reads only blocks that really need to be read.

Second you should — just in case — save your partition geometries with fdisk (output similar):

$ sudo fdisk -ls
Password:

Disk /dev/hda: 160.0 GB, 160041885696 bytes
255 heads, 63 sectors/track, 19457 cylinders
Units = cylinders of 16065 * 512 = 8225280 bytes

   Device Boot      Start         End      Blocks   Id  System
/dev/hda1   *           1        9145    73457181    b  W95 FAT32
/dev/hda2            9146       19106    80011732+  83  Linux
/dev/hda3           19107       19457     2819407+   5  Extended
/dev/hda5           19107       19457     2819376   82  Linux swap / Solaris

Disk /dev/hdb: 160.0 GB, 160041885696 bytes
255 heads, 63 sectors/track, 19457 cylinders
Units = cylinders of 16065 * 512 = 8225280 bytes

   Device Boot      Start         End      Blocks   Id  System

Disk /dev/hdc: 20.8 GB, 20847697920 bytes
255 heads, 63 sectors/track, 2534 cylinders
Units = cylinders of 16065 * 512 = 8225280 bytes

   Device Boot      Start         End      Blocks   Id  System
/dev/hdc1   *           1        2427    19494846   83  Linux
/dev/hdc2            2428        2534      859477+   5  Extended
/dev/hdc5            2428        2534      859446   82  Linux swap / Solaris

After ddrescue has finished try TestDisk for lost partitions which, among others, looks for back up partition tables, inode tables, and the like. To restore plain data there are heaps of tools. Most notably foremost and PhotoRec. Note though that for lost partitions there is no way to really know what file type found blocks belong to or even the name of associated files’ paths, at least as far as I can imagine. File names really are — in all file systems I know — only attributes of a data segment called file (or inode) such as access date, ownership and such. Sometimes file names are even stored in special files (directories or folders) which means there data blocks need to be intact and it has to be known where they are! The scenario I ran into was I mistakenly installed Ubuntu on top of Windows XP (NTFS partition) instead of resizing the partition as intended. TestDisk even though listing a NTFS partition (from it’s back-up MFT located at the end of NTFS partitions) couldn’t however restore the partition as such. Running Microsoft’s chkdsk stored a bunch of files with their filenames several folders into one called found.000 (I ran it inMicrosoft’s recovery command line from CD after manually through fdisk changed partition type to NTFS/HPFS, e.g. x07). Using the above mentioned tools I could restore some more but without there original path or file names. All together I recovered ~40 GB out of probably 60 GB data (which included system files).

Resources

Further reading:

Ubuntu System Panel (aka USP2 or USP3)

Screenshot of USP Like what you see?

Now?

Well, anyway, hit the Ubuntu forums or google code base of this project to just get it and try it. For a couple more snapshots there is another thread.

How to find out what occupies space on your Linux hard drive

The other day I noticed that my settings directory (/etc) uses over 13 MB of my hard drive. So I wandered which package (I’m using a Debian based package managed system) makes the settings directory grow so large. After a couple of trails and errors I came up with the following sequence of commands:

$ du -h --max-depth=1 /etc 2> /dev/null | egrep '(^[5-9][0-9]{2}K)|M'
692K    /etc/X11
672K    /etc/acpi
712K    /etc/xdg
2.1M    /etc/brltty
500K    /etc/ssl
528K    /etc/mono
20K     /etc/NetworkManager
13M     /etc
$ dpkg -S '/etc/brltty'
brltty-x11, brltty: /etc/brltty
$ apt-cache show brltty | grep -A5 'Description'
Description: Access software for a blind person using a soft braille terminal
BRLTTY is a daemon which provides access to the Linux console (text mode)
for a blind person using a soft braille display.  It drives the braille
terminal and provides complete screen review functionality.
The following display models are supported:
* Alva (ABT3xx/Delphi)

Fortunatelly, I’m not blind so I could remove brltty with aptitude which then suggested to remove dependencies, too.

References:

  • Regex reference
  • Resources for advanced Ubuntu topics, eg. how to remove …-desktop meta packages with apt-get (instead of aptitude), secure networking setup, etc.

How to convert pixel (raster) files (png) to vector graphic (svg) on Linux

In another post I wrote about the small tool convert as part of imagemagick. With convert it is possible to, surprisingly enough, convert picture files. ASAIK it would not convert to or from vector graphics, though.

From the man-pages of inkscape:

Inkscape can import (File > Import) most bitmap formats (PNG, BMP, JPG, XPM, GIF etc.), plain text (requires Perl), and AI format (Adobe Illustrator documents, versions up to 7 only;
requires Perl).

Unfortunatelly it only works via the GUI. You need to either open, import or drag’n’drop the png file, mark it and go to the menu “Path” and than trace it via “Trace Bitmap…”. You might also want to set the output dimensions vie the File menu and “Document Properties” to 48×48 or similar if you are working on an icon. In my tests though, the output was rather crappy. Fortunately I found the original postscript of the program icon (foobar2000 it was).

If you need the other way around, i.e. vector  -> raster, there is a comand line argument to be used like this:

inkscape filename.svg --export-png=filename.png

See more details at examples in man pages of inkscape.

Update: And there is also Potrace and Delineate.

Resources:

Let Bash do the math: Doing calculations using that bash

Have you ever wanted to “just do some math” without much fuss? I noticed a while ago (but haven’t really used it since) that bash (born again shell) has, amongst many other evaluations, an arithmetic one. Using the syntax $((expression)) you can evaluate the expression to arithmetically. Operations allowed are quite a few. From man bash:

ARITHMETIC EVALUATION
The shell allows arithmetic expressions to be evaluated, under certain circumstances (see the let and declare builtin commands and Arithmetic Expansion). Evaluation is done in fixed-width integers with no check for overflow, though division by 0 is trapped and flagged as an error. The operators and their precedence, asso ciativity, and values are the same as in the C language. The following list of operators is grouped into levels of equal-precedence operators. The levels are listed in order of decreasing precedence.

       id++ id--   variable post-increment and post-decrement
       ++id --id   variable pre-increment and pre-decrement
       - +             unary minus and plus
       ! ~             logical and bitwise negation
       **              exponentiation
       * / %         multiplication, division, remainder
       + -             addition, subtraction
       <>       left and right bitwise shifts
       =
                        comparison
       == !=       equality and inequality
       &               bitwise AND
       ^               bitwise exclusive OR
       |                bitwise OR
       &&             logical AND
       ||              logical OR
       expr?expr:expr
                        conditional operator
       = *= /= %= += -= <>= &= ^= |=
                        assignment
       expr1 , expr2
                        comma

Shell variables are allowed as operands; parameter expansion is performed before the expression is evaluated. Within an expression, shell variables may also be referenced by name without using the parameter expansion syntax. A shell variable that is null or unset evaluates to 0 when referenced by name without using the parameter expansion syntax. The value of a variable is evaluated as an arithmetic expression when it is ref erenced, or when a variable which has been given the integer attribute using declare -i is assigned a value. A null value evaluates to 0. A shell variable need not have its integer attribute turned on to be used in an expression.

Constants with a leading 0 are interpreted as octal numbers. A leading 0x or 0X denotes hexadecimal. Other wise, numbers take the form [base#]n, where base is a decimal number between 2 and 64 representing the arith metic base, and n is a number in that base. If base# is omitted, then base 10 is used. The digits greater than 9 are represented by the lowercase letters, the uppercase letters, @, and _, in that order. If base is less than or equal to 36, lowercase and uppercase letters may be used interchangeably to represent numbers between 10 and 35.

« Older entries

Follow

Get every new post delivered to your Inbox.