The Ultimate Lossless Compression Format with Hybrid Mode and Open Source: WavPack

Way the hack didn’t I trip over WavPack (wv) earlier — it’s been around for some time now and astonishingly ultimate. To make it short and obvious (see hydrogenaudio.org for complete list):

Pros

  • Open sourc
  • Good efficiency (fwd is even better than mp3’s on foobar2000)
  • Hybrid/lossy mode (see below)
  • Tagging support (ID3v1, APEv2 tags)
  • Replay Gain compatible (which is no deal with fb2k anyway, but still)

Cons

  • Limited hardware player support
  • does take long to encode with optimal settings (not really because one time procedure)

Other features

  • Supports embedded CUE sheets
  • Includes MD5 hashes for quick integrity checking
  • Can encode in both symmetrical and assymmetrical modes
  • Supports multichannel audio and high resolutions
  • Fits the Matroska container
  • streaming support
  • Error robustness

So it’s open source! Most distros even come along with WavPack preinstalled. The only other lossless formats that I know are open sourced are two: FLAC which has bad tagging support and Shorten (to put it short: out dated). MAC (Monkey Audio Codec, ape) has an open sourced version but it’s not developed any longer.

The next great feature is hybrid mode which means you decode to lossy small file and an additional file containing “the rest” of the information. Only one other format is capable of this: OptimFROG (ofr) in DualMode. That means, putting both files together you get your 100% original back. The lossy file can be used entirely on it’s own. While encoding there is a second file, called correction file, that stores the difference between lossy and original — compressed that is. So what that means is you don’t have to convert your files each time you’re to shove them onto your portable. The only bad thing is you need to ensure the device can decode (read: play) them.

WavPack Properties after encodeTo give an example: If you convert a 27.5 MB ofr file to wv, hybrid enabled with lossy bitrate set to 192, you’ll get one 6.31 MB sized .wv file and a second 22.8 MB sized .wvc file. It took 12 min. 15 sec (playtime 4:25). Insane settings where: Compression Mode “high”, Processing Mode “6” (best encoding quality), Hybrid Lossy Mode “192kbps”. When you open the .wv file only in fb2k it’ll handle correction files automatically (see picture). However moving the file with foobar2000’s dialog “” will only move .wv files. I cannot speak for other software players but this way it’s just easy to handle two qualities of one file — one hifi one and on “to go”! After I now have converted an entire album here are the approximate file sizes comparing MAC, OptimFROG and WavPack and MP3/OggVorbis as lossy counterpart to “portable wv” (to be added to .ape/.ofr):

.ape 311 MB
.ofr 306 MB
.wv 70 MB
.wvc 255 MB
.wv+.wvc 325 MB
.mp3 (V2, ~190kbps) 74 MB
.ogg (q5, ~160kbps) 61 MB

Tagging: Unlike FLAC it uses APEv2 (or ID3v1) so tags can be used with most players, software and portable devices’ ones, without intervention.

While I ran encoding test’s with foobar2000 (which has decoding WavPack “build-in” by the way) I noticed when converting from, say, OptimFROG to WavPack fb2k went right at it. No temporary wav files as with OptimFROG to MAC, for example! But mind you it does take a long time if you use optimization for file size and quality. It seams to be somewhere around 0.7x (slightly slower than plain play time). I don’t see why this really is an issue because in most cases you’ll only encode once as it’s true for all lossless formats anyway.

Resources:

Archive Audio CDs to APE (MAC) or FLAC Image Files and Toss Away Your Dust Catchers

If someone is interested in archiving your audio CD collection in a really save manner to get rid of them after the encoding, or I should say transcoding process you might want to have a look at Neil Popham’s guide. Neil Popham, by the way, wrote some features for the APEv2 tagging tool wapet. His guide utilizes Windows batch scripts, adds tags and employs PAR2 for parity information. The later is needed to overcome seldom yet possible bit failures which would compromise an ape file. The ripping is done with EAC, the meanwhile well-known audio CD accurate ripping tool by Andre Wiethoff.

Using one single image file per CD has the benefit over multiple files to include, among others, audio information from before the first track, lead-in silence and TOC information. On the other hand, for audio library handling and daily playback where tagging plays a more important role, one file per track is much more comfy. That way all tags can be written directly to the audio file. This, however is resolutely limited with cue sheets. For example, with foobar2000 it’s transparent whether you have your playlist made up from an ape image with cue sheet or from multiple audio files with tags. It’s all the same with randomizing the list, seeking forward and backwards and the like. But once it comes to playback statistics or ReplayGain this will not be written to any file besides fb2k’s library database.

For my daily usage I copied the file MAC.exe (the actual encoder) from the Monkey’s Audio directory to foobar2000’s one and set up an encoding preset in it’s converter preferences using one of the following parameter taken from hydrogenaudio:

-c1000 Fast / large file
-c2000 Normal
-c3000 High / medium file
-c4000 Extra High
-c5000 Insane / small file

Prefixed by the source and destination file variables %s %d. I mainly chose ape over flac because of ape’s flexible tagging feature and neglected the in terms of encoding far better codec OptimFROG because this one needs heaps of CPU usage for decoding (while listening). When CPU power is not a constraint any longer I will most likely switch to ofg (if at that time it still has the best compression around). Don’t forget to enable the secure mode for your CD drive in fb2k under file -> open audio cd… -> select your drive and than drive settings. You can also access the rip wizard from there. Or go via “add to playlist” and convert the tracks manually.

Update: I have found out about WavPack which IMHO is a much better choice to MAC (APE) and FLAC because of better support, tagging and hybrid mode.

Using fb2k’s Scripting Language In It’s Masstagger?!?

Well, what I’m trying to do I thought would be very simple: Add a new tag to each file that’s been added to fb2k’s media library holding the current system date and hence add a “added to library” tag. The tagz script to achieve this is not even the problem.

$if($meta(ADDED_TO_FOOBAR),,%cwb_systemdatetime%)

Or, what seams to be semantically equivalent but more readable (refer to tagz reference to understand the commands):

$if($not($meta(ADDED_TO_FOOBAR)),%cwb_systemdatetime%)

This even checks if the file has a tag already. Using the tagz parser in the preference dialog (Ctrl+P -> Display -> Title Formatting) confirms it’s working correctly when playing a song without a tag and one with the time stamp set.

The set-up is this:

  • with masstagger (right-click on song -> tagging -> manage scripts, if you haven’t changed the default context menu structure) add “Format value from other fields…“, select destination field name (ADDED_TO_FOOBAR) and use the stated script as formatting pattern. Hit Return and name your masstagger script and (important:) click the save button. Note: Using “Set value…” or the like will not work since it, despite intuitive guesses, does not evaluate tagz scripts but outputs it as a string.
  • in fb2k’s preferences dialog select Tools -> New File Tagger and from the drop-down list select Tagging/Scripts/your name (do this after extensive testing on single files with the file’s preferences box open!)

Now each time a file is added to fb2k’s lib this script is run on it. BUT: It doesn’t do what it’s supposed to! What ends up in the files tags is

  1. a ‘?’ for those with no time stamp
  2. deleting the existing field when value = ‘?’

That’s when I noticed the two scripts are not equivalent: The first add an empty string to the requested field if present where the later does simply nothing in that case because there is no else branch. But still, even the later does not do the desired job.

Then I came up with this script:

$if($not($meta_test(ADDED_TO_FOOBAR)),%cwb_systemdatetime%,$meta(ADDED_TO_FOOBAR))

But once again, the only effort is frustration but not the desired time stamp. After all it leaves existing fields untouched.

I could work around this issue with the “Stamp current Time and Date…” bit but since after reinstalling my OS and using fb2k before my music files partly are stamped already. Sidenote: Probably because of this the field name should rather be something like “ADDED_TO_LIBRARY”. Though moving on…

A working workaround I figured out is to

  1. add a “Stamp current Time and Date…”,
  2. use the following script
  3. $if($not($meta_test(ADDED_TO_FOOBAR)),$meta(TIMESTAMP),$meta(ADDED_TO_FOOBAR))
  4. add “Remove Field…” with selected “TIMESTAMP” field.

So that leaves me with speculating about a bug in either the masstagger or in foo_cwb_hooks. By the way, the time stamping option might be in foo_masstag_addons. You might want to include this masstagger script from a file.

ReplayGain using Foobar2000

In case you wonder — like I did — to employ foobar2000 (fb2k) to handle the lack of ReplayGain (RG) info in a file’s tags nicely so it doesn’t blast away your eardrum I have a set of very usefull links. At hydrogenaudio.org I found the Intermediate User Guide for fb2k explaining, among others, the options one has setting up fb2k for ReplayGain. Most importantly one should slide the bar in the playback preferences pane for “without RG info” to a value that reflects the average sound level of all tracks. This, of course, would mean to scan all your files. For my couple’o weeks worth of playback time fb2k estimates just over 24h for that job. So it was an easy choise to just stick to the suggested value -8db.

Last but not least, I will mention the two preamps in the Playback preferences. Except if you know exactly what you are doing, it is not recommended to raise the output of the preamps above the default 0.0dB in any way. However you can use these to slightly compensate for the difference between replaygained and unreplaigained Tracks. Simply estimate your average Replaygain level and lower the preamp for files without Replaygain info by that value. I found -8dB to work quite well for me. This obviously should not be used to compensate for not properly Replaygaining your tracks, but definetaly will protect your ears and your equipment when coming across tracks that miss Replaygain info.

Secondly there is a more detailed description on the Playback settings in the same wiki. Besides some mathematics on RG it also points out some interesting knowlege about pre-buffering and DSP settings

Audiophile’s Techno Thoughts — Playlist Generators

There are some thoughts I’d like to a. document but b. share, too. They are about — no, not about Techno music! — programming issues, or just ideas I would find useful if they where already realized. Maybe they are and only I don’t know about it jet. But also, I might be the one realizing them… let’s see.

Well, one thing that came to my mind by playing around with the MusicIP Mixer: Wasn’t it cool to have a free library like thing laying around that could easily be user by a software music player (or even extended to video, i.e. media player) to generate playlists similar to the MusicIP Mixer? To be more precise the features I’m thinking about are:

  • generate playlists by user selected music title from library, that is giving back a list of songs from the user’s music library that have similar mood, tempo, harmony, …
  • start playing tracks from the lib. The user can skip or pin it. Either one “pin” (as in MusicIP Mixer) is enough to start generating the list or maybe even keep on with more appropriate music (closer in the above terms). That’s especially useful for those audiophiles that just cannot remember artists or track names (like me). But also it’s faster in terms of I don’t have to sit down first and think about what I’d like to here right now but rather can just find out what I’d wanted to listen to in the first place.

To brake it down, the key note here is to have two methods generating playlists that are end-user orientated. You may ask: “Well, what the hack is so special about it? You mentioned MusicIP Mixer yourself, that does exactly that!” No, it does not. MusicIP Mixer, as fare as I know, is closed source. But more importantly the list generates are encapsulated within the Mixer software and cannot (again, as far as I know) be used by other software that MusicIP Mixer itself (I’ll digg in to find out when time permits). The generators that is; one can have the generated lists send to one’s music player of choice, easily. Which is fair dinkum.

An example: As of now I chose foobar2k to be my favorite music player as it is very customizable and produces good enough sound for my HiFi (gapless playback, replaygain, multiple format support, “tabbed playlisting“, has plugin API to name a few features and components). I runs more or less 24/7. Through the foo_playlist_tree I wrote a query which gives me 10h of random music, not yet listened to or not for more than two weeks. Of course there are also queries by audio format, … just by any combination of normal audio file tags plus more. But still if I might just want say 40 min of jogging music… mmh, that I could manage by all means, plus also the hassle of time consuming transfer to your portable player if your on the run… Not a good example. OK, imagine sitting on the couch with your fb2k (keyboard remote control configured, of course) ready to blast your ears away but you just can think of what so listen to. Even with my iPAQ running Minibar2k this would not really what ease my relaxed mood. Getting it? Comments, as always, appreciated!

Blogged with Flock

Tags: , ,