Real-time mp3 recording, part 2

ArticleCategory: [Artikel Kategorie]

Applications

AuthorImage:[Bild des Autors]

TranslationInfo:[Author and translation history]

original in nl

nl to en

AboutTheAuthor:[Über den Autor]

Soon, he will get his Ph.D. at the University of Nijmegen on the subject of chemometrics. He enjoys working with Linux and writes down his experiences on a regular basis.

Abstract:[Zusammenfassung]

This article is the follow up of an earlier publication concerning making real-time mp3 recordings of e.g. radio broadcasts. In the previous paper, it is demonstrated how a large mp3-file is obtained, e.g. 650 MB for 12 hours or 1.3 gigabyte of 24 hours of music approaching CD quality (128 kbit/s; stereo). How to handle such a large file? Is it possible to make even longer recordings or to compress the music even better? What about the quality of the mp3 recordings? We will focus on answering the questions above.

ArticleIllustration:[Titelbild des Artikels]

ArticleBody:[Der eigentliche Artikel]

Editing mp3-files

Assume that you made a beautiful recording of 24 hours, e.g. the Eighties Top 300 as broadcasted by Loostad radio, and you want to split this file in smaller, thus more suitable, files. During the splitting process, you also want to remove advertisements and news broadcasts. How can you do this best? One way to do it would be:

Decode the mp3-file to the wave-format.
Use a wave-editor for the better 'cut and paste' job.
Encode the edited wave-file back to an mp3-file.

This procedure would theoretically work and as opposed to common believe the re-encoding of a decoded mp3 signal with the same mp3 algorithm does not result in a loss of quality. It has however in practice a few problems:

Decoding an 650 MB mp3-file to the wave-format results in a 6.5 gigabyte file (compression is about a factor 10). I am not even discussing the required processor time!
During decoding, the original 650 MB file remains. For editing the decoded wave-file, some more disk-space is needed. If you want to cut the whole file in parts of one hour, you will need 2x6.5=13 gigabyte!

In short, you need a program that is capable of editing the mp3-files THEMSELVES! Such programs do exist, but not (for the moment) for Linux. Quite a number of such Windows programs are available, but most of these programs have one big problem: they cannot handle an mp3-file with a size of 650 MB! I have found one program that is able to do the job, if you are patient: 'Manns' mp3 edit'. I use this program in Windows98. In the graphical interface, you can indicate the begin and the end of the desired piece from the original mp3-file. You can copy this piece to a new file with the 'trim' command. This program is 'donationware', which means that you are not obliged to pay for it, but you are kindly asked to do so if the program suits your needs. This program has two drawbacks: it is awful SLOW on a 650 MB file and it can only handle CONSTANT bit-rates!!! Searching the beginning of the final hour, e.g. in an mp3-recording of 24 hours, takes a long time and includes heavy disk access. Trimming the desired piece of mp3-file, on the other hand, is very fast (without losing mp3-quality). Until now, I have not encountered any problem with mp3-file playback after using this editor. Nevertheless, I always check the final trimmed files by loading them in the mp3-editor. If loading works correctly, the trimmed files are O.K. During my first experiments with this editor, I already clicked on the 'trim'-button while still heavy disk access occurred. This resulted in damaged mp3-files that cannot be played back! To give you an indication: editing a 650 MB file takes me roughly 1 hour, mainly caused by waiting (which can be quite frustrating sometimes)!

Making long mp3-recordings (>24 hours)

It is no problem to make 24 hours recordings from a radio broadcast. The file remains reasonable small (e.g. 1.3 gigabyte for 24 hours) and can easily be stored on the harddisk. This problem changes if you want to make a recording for e.g. for a week. Presumable, it won't fit on your harddisk and you want to overcome this problem. I encountered this problem with the broadcast of the 'Alltimes Top 2000' during Christmas and new year 2000 (144 hours of music!), broadcasted by a Dutch national and public radio-station: Radio 2. My harddisk was on beforehand far too small to store all recordings, so I had to solve the matter in a different way. Here you have the recipe:

Create an empty file that precisely fits on a 700 MB CD-recordable (80 minutes CD).
Create on this empty file the Linux ext2-filesystem.
Mount this file (or more precisely: this image), so it can be accessed as a separate Linux disk.
Make an 12 hour mp3-recording on this image (128 kbit/s; stereo; about 670 MB).
Unmount this image, mount a second image, continue the recording on this second image.
Copy the first image to CD (using the CD recorder). This is not any problem during real-time mp3-recording on my computer, a Pentium II 450 MHz (no XWindows running).
If the CD has been created successfully, delete the mp3-file (in the image; not the image itself!). This image can be used to make a new recording.
Repeat this procedure until the recording is finished.

I have used the above procedure to record the whole Top2000 completely and without any problem! I used 12 backup-CDs that will never be used anymore, but you have to make some expenses to get these kind of successful recordings. How does it all work? I obtained the idea reading the CD-Burning-Howto and the necessary steps are explained below:

I created 4 images on a separate partition. I happened to have 2.9 gigabyte free.
I created 4 images, respectively called image_1, image_2, image_3, and image_4:
dd if=/dev/zero of="image_1" bs=1024k count=700
I repeated this command for all 4 images. Creating the images takes some time!
Create the Linux ext2-filesystem on the image:
shell> /sbin/mke2fs -b 2048 image_1
empty_file is not a block special device.
Proceed anyway? (y,n) y
Note: the ext2-filesystem also uses some disk space, so you do not have the full 700 MB available for storing the mp3-recording!
To mount the image, you need to use the loopback device. This requires support for the loopback device in the kernel (or making the loopback device available as a module).
mount -t ext2 -o loop=/dev/loop1 image_1 /mnt
You can access the image via the /mnt-directory. Now make any recording you like using this image!
Unmounting the image and creating the CD:
umount /mnt
cdrecord -v speed=2 dev=0,1,0 -data image_1
Please note that the command 'dev=0,1,0' depends on your system configuration and might be different for your particular situation (cdrecord -scanbus, will give you a clue what it should be). The image is being copied to CD. In the mean-time, you are already continuing the mp3-recording on the second image (hopefully).
Testing of the created CD:
mount -t ext2 /dev/scd0 /cdrom
If everything went right, you can see the recorded mp3-file via the /cdrom directory! I want to remark that I encountered some problems with kernels 2.4.0 to 2.4.2. I was unable to mount these CDs (with the ext2-filesystem) and I encountered problems with the loopback device. In kernel 2.4.3, all these problems are properly solved and everything functions correctly (as is the case with the 2.2.x-kernels).

The benefit of this approach is the limited processor usage, needed for creating the mp3-CDs. The images already exist! You can automate the whole procedure by using a script: mounting the images, starting the mp3-recording, finishing the mp3-recording and unmounting of the image after 12 hours of recording, mounting a second image and continuing the mp3-recording, and so on. I burned the CDs by hand, just in case problems occured (which was true once, due to faulty media). For this reason, I created 4 images: 3 spare images in case I encountered CD burning problems (giving me maximally 36 hours to solve the problem). The script I used looks like this:


#!/bin/sh
echo "loading modules for sound support..."
modprobe -k snd-card-sb16
modprobe -k snd-pcm-oss
modprobe -k snd-mixer-oss
echo "setting mixer..."
amixer set Line 31 unmute capture; amixer set Master 31; amixer set PCM 31
cd /where/ever/you/stored/the/images/
for i in 1 2 3; do
        for ii in 1 2 3 4; do
        echo "mounting image_$ii on /mnt ..."
        mount -t ext2 -o loop=/dev/loop1 image_$ii /mnt
        echo "starting mpegrec ..."
        mpegrec -b 128 -x -mj -o /mnt/top2000_$ii.mp3
        echo "umount /mnt"
        umount /mnt
        done
done

I did everything WITHOUT XWindows, using the Bash-prompt. I loaded the sound support by hand, as well as the OSS emulation (necessary to get correctly functioning ALSA-drivers) and the mixer driver (of course you can automate this by using a pre-install/post-install trigger in /etc/conf.modules). T The 4th line configures the mixer correctly: recording source is 'line-in' and the volume has been maximally opened. After setting these options, the script handles the mounting and unmounting of the images. Concerning the mpegrec options: '-mj' indicates a joined stereo recording. I did this because I expected smaller mp3-files using this option (it should all fit on 670-680 MB). This appeared to be untrue: it does not matter if you use stereo or joined-stereo recordings! When I make new recordings, I will use ordinary stereo, The 'mpegrec' program should stop after 12 hours. To achieve this task, I adapted the source code and changed the maximally 24 hours (which is simply stated in the source code) to 12 hours. Unfortunately, this approach did not work: the program continued recording after 12 hours! I had to break the recording every 12 hours (you won't get it for free...). The script took care for the remaining parts. I do not understand a thing from 'mpegrec' anymore: I should study the source code carefully...

Note:
The most recent version of cdrecord (cdrecord-1.9) caused me problems with creating the CDs. The previous version, cdrecord-1.8.1, operated without any problem. I solved these problems by upgrading the firmware of my CD recorder, a Philips CDD 3610, from version 3.02 to version 3.09. Initially, I expected that the CD burning problems were caused by kernel 2.4.3, but this appeared to be untrue!

The quality of the mp3-recordings

Due to the feedback on my previous article via the English Talkback page, I received a link to a website containing interesting information concerning mp3-recording. Especially the mp3 quality and the connection with the mp3-bitrate is excellently explained, focusing on the program 'lame'. The address is: http://www.r3mix.net/.

Summarizing the most important facts:

A sampling rate of 128 kbit/s is NO CD-quality. A listening panel could successfully detect all 128 kbit/s mp3-files. For making recordings of radio-broadcasts, this should not be any problem: radio broadcasts are not in CD quality and 128 kbit/s is more than sufficient for this specific situation!!!
A sampling rate of 256 kbit/s gives you REAL CD-quality, in other words: it is not possible to separate a 256 kbit/s mp3-stream from the original CD. Despite this fact, it is still possible to detect some compression artifacts, but only if you know which music-aspects to consider (as an audiophile)!
Lame recordings with a variable bitrate (VBR) are recommended. The standard bitrate of 192 kbit/s (a value which I considered 'weird' in my previous publication) is a good starting value. Depending on the 'difficulty' of the song, the sampling rate is increased or decreased. A second benefit is the application of filters, giving a better mp3-file compared to the 256 kbit/s recording (without these aforementioned filters). It is indicated that Philips specified these filters during the development of the CD recording technique and that these filters can be used without any problem!

All this is discussed in detail in the above mentioned website!

Creating as small as possible mp3-recordings

Below, I will discuss a script that has been submitted by a reader (via email). The script looks as follows:


#!/bin/bash

echo "record $1 named ($3) for $2 seconds"
aumix -f ~/.aumixrc
/usr/bin/yes x | /usr/bin/radio $1 >/dev/null 2>/dev/null
sox -V -r 44100 -c 2 -t ossdsp -w -s /dev/dsp -t wav - 2>/dev/null |
lame - ~/$3.`/bin/date "+%H%M.%m-%d-%y"`.mp3 --resample 16 -b 8 -a -m m -h --lowpass 4 --highpass .12 --voice&
sleep $2
kill -9 $!
# copy the output to a different direcory. If you want you can
# replace cp by rcp or scp and copy it to a different computer:
cp $3.*`/bin/date "+.%m-%d-%y"`.mp3 /tmp/outputdir

The script's main goal is to obtain understandable radio recordings that only need 3.5 megabyte/hour diskspace! With understandable I mean that one should be able to interpret the recorded speech. Off course, the quality of the music is awful! The first line in the script configures the mixer and starts a radio-card program and 'sox'. The 'trick' resides in the 'lame'-options, which we will investigate now:

--resample 16 -b 8 -a -m m -h --lowpass 4 --highpass .12 --voice

--resample 16	Redefine the sampling frequency from the output file from 44 kHz to 16 kHz.
-b 8	Set the sampling rate on 8 kbit/s (gasp!).
-a	Combine the stereo stream to a mono recording.
-m m	Force mono mp3 encoding.
-h	An option to use high-quality mp3-encoding (slower).
--lowpass 4	Set the lowpass filter on a value of 4 kHz. All signals (frequencies) higher than 4 kHz are removed.
--highpass .12	Set the highpass filter on a value of .12 kHz. All signals (frequencies) below .12 kHz are removed.
--voice	Use an mp3-compression technique optimized for speech.

It is clear that a high compression can be obtained for speech. Most of the times, this will be undesired. Now, at least the possibility is indicated!

A standard script to create mp3-recordings

This script was sent via the TalkBack pages (I am grateful for that; everyone can profit from this information!). This is the large benefit of the TalkBack pages that are made available for the previous article (and also for the current article). Everyone can post his or her experiences, ask advice to other visitors if troubles arise, in short: a tremendous amount of information is brought together and is (hopefully) made available for everyone! The script, called 'mp3_record', looks as follows:

#!/bin/sh 
 /usr/local/bin/mpegrec \ 
 -l 5400 \ 
 -x '--preset fm \ 
 --tt "Computers in Your Life" \ 
 --tl "Open Line with Tom Wieble" \ 
 --ta "Tom Wieble" --ty `date +%Y` \ 
 --tg "Speech" \ 
 --tc "Copyright (c) 1997 - `date +%Y` WOSU" -c' \ 
 -o open_line_`date +%d%b%y`_`date +%H%M`.mp3

The option '-l 5400' sets the recording time on 90 minutes. All options after '-x' are directly piped to 'lame' (lines 4 to 9) and the '-o' option saves the file as 'date_and_time.mp3'. These lines can be adapted as one likes. For this purpose, it might be useful to consider the 'lame'-options more closely. The current options are explained in the table below:

--preset fm	Lame knows some standard settings. 'fm' is meant for making fm-broadcasts recordings, but I dislike the recording-quality compared to the standard sampling rate of 128 kbit/s. The main cause are the applied filters in 'lame'.
--tt "Computers in Your Life"	'--tt' define the title of the recording (30 characters max).
--tl "Open Line with Tom Wieble"	'--tl': album, or: original source (30 characters max).
--ta "Tom Wieble" --ty `date +%Y`	'--ta': artist (30 characters max).
--tg "Speech"	'--tg': genre (name or number).
--tc "Copyright (c) 1997 - `date +%Y` WOSU" -c"	'--tc': comment (additional information; 30 characters max).

In short: a nice way of creating an mp3-file, including recording date and time and incorporating all kinds of additional information. It requires some time to write everything down into a script, but it is nice to see that all these things are possible!