Base | Lab | iTunes

2006 01 29

Base | Lab | iTunes

22:52

Brainless backup

I recently had an email conversation with an otherwise happy user of my tip on how to manually move your music files, whilst retaining track usage data in which he mentioned backing up his files.

For one thing it has never occured to me to do that, but that is not uncommon for some other Best Practices that I can often be found beating mere mortals over the head with. What can I say... physician heal thyself, it seems.

Anyway, the conversation prompted me to think of ways in which one could have iTunes help out in this mundane task. Below are some thoughts, which are clearly just the ramblings of a sleep-deprived geek. Enjoy them as such. :)

Option 1: Full backups. For people with DVD-RW hardware or media to spare who also enjoy the sound of spinning burners in the background. (Or whose music collection is ludicrously modest by today's everyone-is-a-pirate standards.)

  1. Create a Smart Playlist that has everything in it. This might be 'Kind contains MP3' or 'Bit Rate is greater than 0 kbps' Whatever works for you.
  2. Burn your playlist to DVD. iTunes will ask you for more media if the file content is bigger than what can fit one disc.

I would suggest you always keep two sets of backup discs and toggle them.

Option 2: Full incremental.

This is basically the same idea as option 1, but by maintaining a Playlist that selects on 'Added after [date of your last backup]', you can save yourself the trouble of backing up everything for the sheer excitement of it.

Option 3: Automatic

Now this is the actual no-brainer I mentioned in the title. Works like this:

  1. Set up a Smart Playlist selecting on everything. (see above)
  2. Limit it to whatever medium you intend to use, say 4.6GB for DVD. (you may have to experiment a bit to get a good number; the 4.7GB on the DVD case is a bit misleading) and do so by most recently added.
  3. Burn away.

What this will do is grab the track you added most recently. If you merely buy a track every now and then or rip the occasional CD, this will result in a stack of DVDs that will have a fair number of tracks backup up several times over, as they get pushed along the queue when new files come in.

Caveats: if you add more than (say) 4.5GB inbetween backup sessions, you obviously miss out on some tracks. Increasing either the size of the playlist (for instance to the size of 2 DVDs) or the frequency of your backups may help.

Also, you may find that one track might be backed up dozens of times, where others exist on only one DVD. That may bother some, but not others.

To the question why you would even consider using iTunes to do your backups, rather than some file-based tool to just grab directories off disk, I would say that outside of niftiness and convenience, you will not have to hunt across your disk to find your files; iTunes will do that for you. Also, creating multi-disc backups is not the easiest thing in the world in some burn apps..

A definite downside is that you may or may not be able to control any form of directory structure on the backup discs, unless you agree to having iTunes manage your music files.

2005 12 14

Base | Lab | iTunes

21:04

Overlapping tastes

I share a living space with my wife and family (a concept not unheard of, I guess). For the time being and seeing as the eldest is about 2.5 years old, the children have very little to say about what songs get played. Actually, I make a concious effort to expose them to the full scope of my own taste in music (which has in the past attracted comments like 'stretches the meaning of eclectic')

In any case, the only relevant tastes I need to deal with are my own and my wife's. Fortunately, mine encompasses all of hers.

What I did to create the possibility to switch between her taste and mine, wilst maintainting the cool dynamic of my Shuffle playlist, is to assign meaning to four ratings:

no stars
This track has not been assessed yet for this purpose.
1 star
Definitely outside my wife's taste
2 stars
Possibly inside my wife's taste
3 stars
Definitely inside my wife's taste

For my wife's automatic music playing needs, I use a smart playlist that is exactly the same as TheShuffle, but with the added rule that tracks must be rated two stars or higher.

One nice aspect of this mechanism is that by adjusting the rating, she can influence tracks appearing in her playlist or not. Another (deliberate) effect is that I can codify my unsureness about her liking a certain track.

Last (but rather obscure) benefit of the overall setup is that my main playlist reacts not only to itself being played (in the intened way), but also to her playlist being used, or any other track selection mechanism, for that matter.

Base | Lab | iTunes

19:26

Controlled randomness

As promised earlier, I will tell something about the rather elaborate stack of (smart) playlists that I use in iTunes.

First, let me state the Ultimate Goal that started me on this road: to have a playlist that I could blindly use, but that would both be as random as possible, but with a moderate preference for newer (i.e. more recently added) tracks. I got the idea off a mailing list, where someone had his currently playing tune mentioned in his email footer. Someone commented on the appearance that he had but one or two CDs and was apparently listening to them non-stop.

He explained in an answer to that comment that it was merely a coincidence, but not a strange one, as his playlists favoured newer tracks. It was never explained how he he achieved such a dynamic, but I was intrigued to the point of being hooked. In fact, I ditched WinAmp in favour of iTunes the very same day.

Over the course of time since then, I have tried a variety of algorithms, improving and tweaking as I went along. I should admit to trial-an-error, spurred on by the occasional insight, rather than systematic research. In any case, here is my thinking, so that you can either save yourself the trouble or correct my thinking.

The first thing to note is that any setup should be somewhat tuned to the amount of tracks in your database. Not as much the higher numbers, but rather the starting point of someone ripping all five of their CDs.

Second is the notion that to do anything above creating a very basic dynamic involves layering Smart Playlists. This will undoubtly result in a few playlists that you will use and just as many that merely form the building blocks. Recent incarnations of iTunes have a 'folder' feature, that I use, for this very reason, to hide those building blocks, since I will never use them for playing as they are.

OK, building blocks, then.

The most crucial playlist I use is one called 'Recent'. It's 'formula':

  • Match any of the following rules:
  • kind contains MPEG
  • Limit to 2 hours selected by most recently played
  • Math only checked songs
  • Live updating

In English this means: any MP3s that are considered active and have been played for the last two hours of playing.

Next up is a list called 'Exclusion'. The name is pretty accurate: it consists of a list of rules that are all like 'Playlist is not [Foo]. The idea is that in my library, there are a number of tracks that I do want to keep, but that I really only ever want to hear on purpose and thus never as part of any Smart Playlist. I manage this by maintaining a number of manual (i.e. normal) playlists, such as 'Spoken Word', 'Children's music', 'Stuff only my wife actually likes', etc. I find that using membershp of a certain playlist is easier to manage than maintaining multi-value 'Genre' tags. For one thing, I can use the interface with drag & drop for track management. Other than that, the difference is negligible.

In any case, to enable the concept of 'never automatically play any of these tracks' I collect these playlists in one called 'Exclusion'.

The next playlist I have called 'potential', but the name is rather meaningless in the current context. Its 'formula':

  • Match all of the following rules:
  • Playlist is not Exclusion
  • Playlist is not Recent
  • Limit to 10 hours selected by least often played
  • Match oly checked songs
  • ive updating

In English: figure out which tracks I want to hear at all, but leave out the ones that recently have been played. From all those, grab ten hours of the ones played the least often.

Whilst playing tracks, the dynamic emerges that tracks get played, get excluded for some time (determined by the limit in 'Recent') and then re-enter the pool. At some point, however, they are played too often and drift outside of the time limit set in 'Potential'. This is where the size of the library comes in: if you have just a dozen hours or so total playing time to work with (or you aplpy this mechanism to a subset of your library), you should use smaller values. Beware that large values (resulting in playlist with thousands of tracks) require lots CPU resource every time a track ends and mulitple playlists get (re)processed.

Obviously this dynamic is not nearly enough for the demanding bastard that I am. :)

The final playlist (and the one I actually use the most) is called 'TheShuffle' and looks like this:

  • Match the following rule:
  • Playlist is Potential
  • Limit to 54 songs selected by random>
  • Match only checked songs
  • Live updating

The net effect in terms of the playing dynamic, is that over time, the play counts of the tracks in the database levels out. In the meantime, by picking 54 songs (the amount which fills one screen on my laptop with iTunes maximised) out of ten hours playing time at random, I very much avoid repeating patterns in the order in which tracks are played.

The last element to throw into the mix (and somewhat of the reason I add the randomising layer) is that I play the 54-song list in descending order of track time, with the list set to repeat. Because of the 'live updating' set on all Smart Playlists, replacing songs can get added above or below the current one playing. It may thus take anywhere from 4 hours (a typical total playing time for 54 songs) to ten hours (in case the newly added songs are biased towards the 4-5 minute range).

I would be interested to do a study in larger libraries, but I think there is a weak correlation between song length (playing time in iTunes parlance) and genre. My technique thus groups genres dynamically based on that weak correlation.

2004 09 09

Base | Lab | iTunes

09:35

Avoid very large playlists

A while ago, I had intermittent but persistent problems with iTunes (on WinXP) maxing out the CPU to the point where the music would start skipping playback.

At first (as I recorded here) I thought the issue was with a USB vs FireWire connection to the external drive on which the music files are stored. Using USB instead of the FireWire surely made the problem worse, as did the fact that the file system was severely fragmented.

So, I defragmented the drive and things were better--but not over just yet.

Really by coincidence, I found out what the actual culprit must have been: automatic playlists that have around 8000 songs each (largely overlapping, off course).

I had about a dozen of those lists and they used the 'Last played' data as a criterium. In effect, after every song, each of those lists would have to be updated. The process of updating involves (from what I can tell from file movements and memory usage) building a catalog of songs in memory, wich then get written to temporary files, which then, in turn, get written out to the XML file in the iTunes directory.

Open that file (or a copy thereof) in your favourite text editor some time. You will find that not just the 'recipe' for your automatic playlists is listed, but also a list of the the actual songs that make up the list. Not very elegant, I agree.

I removed the playlists (they were a mere byproduct of playing around with iTunes anyway) and my CPU maxing problems disappeared.

2004 06 18

Base | Lab | iTunes

09:29

Avoid USB with large libraries in iTunes

For as long as I have been using it, I have been having problems with iTunes stopping playback due to lack of CPU resources. I have long thought that I was just hammering the machine too hard and that it was annoying, but maybe just to be expected. Not anymore.

My music library resides on an external 250GB drive, with USB2.0 and FireWire connection options. Mainly because I keep the USB hub 'n cable spaghetti in a bunch at the back of my desk, I opted for the USB connection, because the FireWire port is located on the left side of the laptop, too close to the keyboard for comfort.

Just acting on a hunch, I decided that since I have been using a wireless USB keyboard anyway, as well as a PC-card connected CD-ROM drive sticking out on the same side, the comfort argument is not valid anymore and that it might be a good idea to switch the drive to FireWire.

The hypothesis is that the high CPU burn of iTunes and other applications has something to do with the fact that I have a ton of gadgets hanging of USB ports. It all works, but it eats cycles. since FireWire is a standalone bus system, it will need a lot less resources. I don't expect a big speed gain, other than what is achieved by not having the CPU clipping at 100%, but it is still worth a try.

A reboot later and I am now due to smack myself on the head for not figuring this out sooner. iTunes' CPU usage used to jump to 90-100% on almost every use of the interface, and then again at every change of song. It now barely gets over 35%. I managed to push it to 65% by having it scan the entire library for new songs.

Moral of the story: if you use iTunes and your collection is of a significant size (which seems to make almost all operations expensive for some reason), avoid USB connections to your storage media.

(note: Apple users can stop their sniggering in the back, now. I know you will have been using FireWire as a first choice anyway)

2004 06 10

Base | Lab | iTunes

10:29

How to preserve library data when moving music files

Updated June 2007!

I think iTunes is great as a player, because it allows me to build all kinds of so-called 'Smart' playlists, using some easy rules that can be combined to wonderfully complex things, all just in order to come up with the right song to play next.

Problem is, this being Apple software, it has a tendency to give you the choice of:

  1. letting it control your life (shut up and be happy) or
  2. giving you absolutely nothing to work with if you decide to opt out of preconfigured bliss.

Specifically: if you allow iTunes to 'organise your music folder' (meaning: totally butcher any ideas that you might have yourself about what makes up a logical way of organising your music files) you are allowed to move the library (i.e. the folder(s) that hold the music files) around to another place on your drive, to other disks or even to a network drive.

The reverse is the actual problem: move a music file and iTunes will treat it as a new file, which obviously resets play counts and 'Last Played' dates.

Here are a few things I tried and found NOT to work:

  • Edit the 'iTunes Music Library.xml' file with your favourite text editor only to find it ignored first and overwritten second.
  • Delete the binary 'iTunes Music Library.itl' file and your entire library will vanish.
  • Import a previously exported or a modified XML file and the entries will be treated as new files.

I searched a lot of fora, but all relevant posts and discussions I could find are either OS X oriented, using AppleScript that (AFAIK) does not run on Windows or advising you to take the bliss option outlined above. I did read something about AppleScripts possibly working on Windows, but it would be confined to within iTunes itself, meaning that moving stuff around in the Windows file system is out of reach. Running AppleScripts that try to do things with the Finder (the OS X interface to the file system) are likely to do nothing if you're lucky and serious damage if you are not.

Admittedly by means of sheer luck on a what-does-this-button-do-mission I have stumbled on to a working solution by myself! Hurrah!

The trick is very simple: do not delete the 'itl' file, but rather damage it. To be sure, just empty it. Upon startup, iTunes will try to load its database and fail (because you have damaged it). In an effort to repair it, it will (re)import the XML file--including our precious play statistics. A message follows that the itl file has unfortunately become corrupt and all that could be done has been tried. It even saves a copy of your emptied file. :)

Given that the XML stores not only music data, but also the playlist definitions, you are good to go from this point on.

Please see below for a number of gotchas and observed side-effects.

Instructions for technically timid folk (on Windows!):

  1. Shut down iTunes
  2. Make a backup copy of these files:
    • iTunes Music Library.xml (on my system, it is located in My Documents/My Music/iTunes)
    • iTunes Music Library.itl (on my system, it is located in My Documents/My Music/iTunes)
    • If you have the disk space to spare and you are truly paranoid, backup your actual music files as well.
  3. Open the itl file in Notepad (or any other text editor, but not in Word or any other word processor).
  4. Select all (Ctrl+A) and hit Delete. You should now have an empty file; save it and close the text editor.
  5. In Explorer, verify that the file size is indeed now 0Kb. I haven't tried partially damaging the file, but this method is a sure-fire way.
  6. If you haven't already, do whatever you want to do to your music folder(s): move them all to a new drive, shuffle some of them around, etc.
  7. Open the xml file in your favourite text editor and modify the paths as they should be, reflecting the new location of your music files. Global search and replace comes to mind as a quick way of making these changes. Don't worry about mistakes--you did take a backup, did you?
  8. Save the file and fire up iTunes. Instead of starting normally, it will import the xml file. I have some 6500-odd entries in it and a dozen playlists. With the files on an external USB2.0 drive, it takes a few minutes for this to complete, which is indicated by the warning message about how iTunes discovered that your library file had gotten damaged and how it tried to save the day. Click OK.
  9. If all is well, you should now be looking at your library, with all data intact and the locations of the music files updated.

Some additional notes:

The location of files is dependant on your system defaults and any specific choices you may have made during installation of iTunes. In any case, you should be aware of the following:

  • The iTunes database files (the .itl and .xml files) are installed in your system's 'My Music' folder. Typically, this is a subfolder of 'My Documents'. Note that this location can be different for different users of the same computer. I do not know what happens in a situation where there is a 'Shared Documents' type of setup.
  • The music files (typically, your mp3 files) can be anywhere on your system. If you have let iTunes 'manage' your music library, it will have put (a copy of) those files in subfolders of the top-level library folder.
    This is where the confusion sets in: the default configuration is for the top-level library folder to be the same folder in which the database files reside. You CAN (easily) change the location of the library folder (in the preferences) but you CAN NOT change the location of the database files. At least, not from within iTunes and not without mucking about with profile defaults in Windows itself.
  • Changing the top-level library folder through the preferences only tell iTunes where to expect existing files and where to put new ones. That is, if you have configured it to manage your files for you. If you have not told it to manage your files, changing the preference does (I think) nothing for the music files.
  • Since the Album Art functionality was introduced, there is another tree of subfolders underneath the database folder, containting the image files for your albums. I honestly don't know how the location of those files changes (or not) if you change preferences. Just be aware that they are there, for the time being.
  • Moving the whole folder around (someone moved their entire 'My Music' folder to an external drive) confuses the hell out of iTunes. The actual program files (the software, rather than the data) are stored elsewhere (on Windows, 'C:\Program Files' by default). When starting iTunes, it can tell (from the Registry on Windows, in some similar way on OS X) where the database files are supposed to be located. The database files will point the way to the music files.
    At best, iTunes might be blunt and recreate the database files (empty!) at the place from where you have just moved them. Other weird side effects may occur. Bottom line: do NOT move the database files, but ONLY the music files themselves.

Some more notes and comments, partially inspired by emails I have received over the years:

  • Beware of some text editors that are unicode aware or have any other reason to stick (normally) invisible characters in the first few bytes of your files. People have reported crashes using my method and at least one person mentioned that UltraEdit had done just this. Removing the extra garbage from the text file using another editor solved the crashing.
  • If you have a large collection, NotePad is likely not up to the task. If you find that you can not open your XML with NotePad, try a text editor of heavier caliber, such as UltraEdit or LargeEdit. I personally use vim, which I suppose is akin to long range missiles in terms of caliber. :)
  • I have the setting enabled to normalise the volume of all tracks. Upon re-importing the xml, iTunes will 'Determine song volume' on all the songs. Depending on the size of your library, this might take a while, but it is otherwise harmless. Just let it do its thing.
  • Since iTunes started handling album art work, the setting for 'automatically downloading missing art work' might trigger a similar automatic process, taking another good bit of time to complete.
  • The same applies to the option for 'determining gapless playback'.
  • My method works on both Windows and OS X.
  • I have had confirmation of the method working on just about every version of iTunes up until the latest (at the time of writing this, June 2007). I have reason to believe that it will remain working for a while longer, because of the following: the xml file exist for a purpose, which is to allow Spotlight and other applications on the OS X platform access to iTunes' data. This is a cornerstone of cross-application functionality in OS X, on which a good many third party programs depend. I speculate that the existence of the XML file is pointless on Windows, but since iTunes is developed for the two platforms simultaneously, it has been overall easier for the development team to just leave it in. (rather than having to maintain two different architectures). Also, it seems to me that the 'repair' facility depends on the xml file to be present. My best guess is that as long as the cross-application API on OS X continues to use the XML mechanism, my method will keep working.

  • I have briefly looked into the way playlists are stored in the XML file. At first glance, it seems to be in the format of actually listing the IDs of tracks that belong to a particular playlist, along with copies of the metadata. This holds true even for dynamic playlists. When you change around some of the meta-data (such as the location of the file) these lists seem to get upset. I have had reports of playlists only partially surviving the process described above.
  • Podcast subscriptions are reported to break. I don't use them myself, so I can not give you any more specifics than that.
  • Someone (named Greg) mentioned that the 'Date added' data element will be reset, since the repairing is seen as re-importing. I have not verified this and actually have my doubts about it.
  • Someone (named Paul) moved an external drive from one machine to the next (I gather a new installation) and after installation dropped in his 'old' database files. Since all the paths to the music files were kept identical between installations, it worked straight away.
  • Be sure to exit out of iTunes cleanly at least once, so that it will rebuild the itl and xml files to be in sync again. I can't say what will happen if you don't, but based on past experience, I would say that only Bad Things can happen from that.
  • I have received feedback from a handful of happy users whose playlists have been saved from forced re-arrangement. If you use the tip above, do me a favour and send me a message at schmolle@pobox.com. Oh, and mention iTunes or SchmolleWorld in the subject, so that you don't get caught in my spamfilter straight away. :)
  • I can't claim having done a proper test, but the timing of the feedback suggests that the above should work on at least the last version or two of iTunes. Specifics in your feedback are welcome.
  • From one or two fora I can deduce that this trick will work on Mac OS X. There is no itl file, though. Rather, an iTunes Library and an iTunes Music Library.xml file. You need to have you Finder set to exposing file extensions to see this, obviously.
  • In a recent version, Apple introduced this method. All good and well--if you let iTunes organise your library. So bummer about that one.