User:Mjb/Workspace

Misc

• The Windows Workgroup on my LAN was having some problems. I have since reinstalled W2K and the problems have gone away.
• I was writing up a report on Oracle XSLT processor bugs.
• I was writing up the answers to the OASIS XSLT Developer Questionnaire.
• I'd really like to rewrite the proposed Wikipedia attribution policy. My suggestions were shot down and the proposed policy bit the dust as well.
• I'm going to keep track of DJ Beppe Loda mixes and links.
• I have a misc. to-do list as well. Probably not up to date.

Font related

• On Discogs we need to figure out the best fonts to use for the phonorecord copyright symbol "℗".
• I'm keeping track of my GDI++ testing here. GDI++ allows Windows apps to use FreeType2 for font rendering, rather than Windows' built-in GDI libraries.
• On Discogs, more and more people are using "N" followed by the degree symbol to approximate a numero sign. I used to be successful at converting these to "No." but moderators are becoming increasingly argumentative about this not being "as on the release". My position is that there's no way to use Unicode characters alone to approximate the kind of formatting found in a proper English numero sign, just as one can't produce 1^st or certain other superscript/subscript characters.

Replacing Tahoma and Microsoft Sans Serif

On Windows 2000 and Windows XP, these TTF files are watched by Windows File Protection (WFP):

• %systemroot%\fonts\micross.ttf
• %systemroot%\fonts\tahoma.ttf
• %systemroot%\fonts\tahomabd.ttf

In addition, tahoma.ttf is always "in use" and cannot be deleted. The other two can be deleted, but get replaced by WFP right away.

You can turn off WFP for specific files by following directions at the bottom of Jeremy Collake's Hacking Windows File Protection page. Specifically, here is what to do:

1. Get a hex editor that can search for UTF-16LE text strings, like the freeware XVI32.
2. Copy %systemroot%\system32\sfcfiles.dll to a temporary location.
3. Edit it using the hex editor. Search for micross, and change the "m" so that it is hex 00, not 6D. Do this by clicking the "6D" and then typing "00". Be careful not to change the "m" to "0" (hex 30).
4. Do the same for the two tahomas. NOTE: There are multiple instances of each filename. Change them all!
5. Save the file and exit.
6. Extract PEChksum.exe from pesuite.zip and run it from the command line:
   PEChksum.exe sfcfiles.dll
7. Extract MoveLatr.exe from movel_101.zip and run it from the command line:
   MoveLatr.exe sfcfiles.dll %systemroot%\system32\sfcfiles.dll (answer Y when prompted to overwrite).
8. It seems like it'd be a good idea to do this for the cached copy as well:
   copy sfcfiles.dll sfcfiles2.dll
MoveLatr.exe sfcfiles2.dll %systemroot%\system32\dllcache\sfcfiles.dll
9. Reboot.
10. You should now be able to delete and replace micross.ttf and tahomabd.ttf without having them become replaced right away. If you then try to delete tahoma.ttf, you might notice the dialog box telling you it's busy now uses your updated micross.ttf!
11. Replacing tahoma.ttf requires booting from another partition and OS from which you can access the file and replace it manually, or you could just use MoveLatr.exe again:
    MoveLatr.exe modifiedTahoma.ttf %systemroot%\Fonts\tahoma.ttf

file URI related

Important links

• IETF and the RFC Standards Process, from The Art of Unix Programming by Eric Steven Raymond, emphasizes that Internet RFCs and standards tend to be based more on actual implementation than pie-in-the-sky theory
• File system info by Chris Giese in addition to providing various FAT technical details, gives additional details about encodings, legal characters, and limitations of FAT12, FAT16, VFAT, FAT32, NTFS, ext2, ISO9660, Joliet, and HFS+
• Wikipedia:en:Comparison of file systems covers a lot of ground, and links to separate articles about each file system. Check the discussion page as well.
• This page from IBM's WebSphere CORBA documentation is an example of an implementation expecting to see ":" and "\" in a 'file' URL
• This Lynx documentation shows how an implementation might treat '~' specially in a 'file' URL

Mailing list posts and notable quotes

'file' URI conventions (13 July 2004) - Mike Brown brings up many issues that complicate the mapping of file system paths to URIs

What to do about file: (19 August 2004) - Paul Hoffman points to a now-expired Internet-Draft that was just RFC 1738's 'file' URI section pulled out into a separate document, and asks about courses of action:

• Publish it as-is (which would accomplish nothing other than hastening the retirement of RFC 1738)
  • "no" — Larry Masinter
• Prescribe what implementations SHOULD do, knowing that such a prescription is bound to break many/most existing implementations
  • "this would be useful if it were accompanied by documentation of the caveats." — Larry Masinter
• List many more interpretations that current implementations use, but not say whether or not to do them
  • "I propose a variant of this: list the interpretations known to be in use, labeled with who uses them, in an informative section." — John Cowan
• Say more about the wide variety of interpretations, but don't list them so as not to confuse readers

An RFC that says, essentially, "Internet Explorer on post-4.0 versions on Windows platforms does X, while Gecko-based engines on linux platforms do Y, on Windows platforms do Z, while the popular LWP perl library does W, java.net.URI does U…" would feel profoundly weird to me. — Tim Bray [1]

Not all RFCs prescribe standards, and this is information that would be profoundly useful to the Internet community. … It would be excellent to have a single reasonably authoritative place to go, rather to have to run one's own experiments all the time. — John Cowan [2]

RE: What to do about file: (19 August 2004) - Larry Masinter suggests some topics to cover

File system path to URI

When converting from any file system path to a URI, questions to consider include the following.

For what kind of file system is the path?

• MS-DOS and Windows: FAT16, VFAT, FAT32, NTFS
• Unix-like OSes: UFS, UFS2, ext2, ext3, ReiserFS V3, Reiser4
• Legacy Mac OS: HFS+

There are differences in how these file systems store directory entries, what characters they allow, how paths manifest in internal APIs, and how paths manifest to the end user of the OS.

If the path's file system is not known…

…what should you do?

• Assume the path is from a default file system for the local OS? Many OSes offer a choice of file systems. How can you be sure you got it right? Is there a "good enough", file system-agnostic fallback?
• Maybe just reject the path? IOW, just say that the file system type must be known.

If the path's file system is not recognized…

…what should you do?

• Reject the path?
• Use a default algorithm, like just prepending 'file:' and doing whatever percent-encoding is required?

Is the path 'absolute'?

• If it's a UNIX path, whether it starts with "/" is the only qualification, I believe.
• If it's a Windows path, it could be absolute if it matches the regular expression ^(\\|[A-Za-z]:) - that is, it either starts with "\" or a drivespec (an ASCII-range letter followed by ":").

If the path is not absolute…

…what should you do?

• Reject it?
• Create a relative URI reference? ('the/path')
• Create an RFC 3986-compliant, but RFC 1738-offending, URI like 'file:the/path'?
• Attempt to make the path absolute by interpreting it to be relative to the local host's 'current working directory', if such a concept exists in the local OS? What if the path is for some other file system?
• And do you make it absolute according to the file system's conventions first, or do you do an RFC 3986 conformant resolution of a relative URI reference ('the/path') against the base URI that is derived from the current working directory?

Does the path contain same- or parent- (. or .., for example) references?

Do you attempt to collapse dot segments (or equivalent) in the path or in the resulting URI? Does it depend on whether the path or URI is absolute? A reason to collapse dot segments in an absolute URI is so that the URI can be suitable for use as a base URI for RFC 3986 conformant resolution.

Is the mapping between segments in the filesystem path and segments in the path component of the URI well-defined?

On Unix file systems, it should be sufficient to percent-encode all non-unreserved characters. Note that '/' may appear *within* a segment, though (you can put a slash in a filename), so be sure to apply percent-encoding to each segment individually.

On Windows, complications abound. (I think I cover these below)

If the path purports to be for a particular OS, but does not match that OS's syntax for a path, e.g. 'C:/autoexec.bat' on Windows…

• Reject the path?
• Be as lenient as possible, e.g. replace '/' with '\' for Windows?
• What about '9:\autoexec.bat' on Windows (bad drivespec)? acceptable?

If the path is provided as a sequence of Unicode characters…

• Form the URI by leaving unreserved characters as-is, and percent-encoding the rest, using UTF-8 as the basis? (RFC 3986 default)
• Use some other encoding more appropriate to the path's OS?

If the path is provided as a sequence of bytes

(not Unicode characters, with no additional info about encoding)…

• Reject it because it can't be decoded to Unicode?
• Assume a default encoding? based on...? How confident can you be about, say, a file system default encoding? (probably not very, on Unix)
• Attempt no decode; just form the URI by converting to unreserved characters only those bytes that, when decoded as ASCII, correspond to unreserved characters, and percent-encoding the rest of the bytes individually?

For a Windows path, is it in the form of a local path or a UNC path?

("local" may not be the right term)

• local, absolute, with drivespec: C:\autoexec.bat
• local, absolute, no drivespec: \autoexec.bat
• local, relative: the\path
• UNC: \\host\share\autoexec.bat

• Do you map the UNC host name to the authority component? Don't forget to percent-encode.

• Do you leave the UNC share name as the first segment of the path component, or..? And don't forget to percent-encode.

Exceptional UNC paths

Networked instances of Windows do weird things like refer to network printers like this: '\\http://192.168.0.1/printername', and refer to shared drives like this: '\\server\d$\autoexec.bat'. When are these conventions used? I saw the former today, and the latter a few years back on NT4 systems. Are they documented anywhere, and do you want to attempt to deal with them?

Update 2006-04-03: I found out that "$" at the end of a share name is a naming convention that causes it to be hidden from network browsers and 'net view'. [3] The format of a UNC path is \\server\share\path\filename.

Windows case normalization

For a Windows path, do you do any case normalization, e.g. in the drivespec? ('c:' -> 'C:')

Windows and colon characters

Windows uses ":" in the drivespec (and nowhere else, currently). ":" is a reserved character in a URI, but does not need to be percent-encoded in a path segment. Therefore, 'file:///C:/autoexec.bat' is acceptable as a URI, and is equivalent to 'file:///C%3A/autoexec.bat.

There is a convention of using "|", e.g. 'file:///C|/autoexec.bat', I believe because of the ambiguities that arise when you have situations like 'C:/foo' as a relative URL being resolved against, say, 'file:/autoexec.bat' or 'file:C:/autoexec.bat' and so on - things that appear in the wild and may(?) have been canon at one time, but don't play nicely with any relative resolution algorithms.

I haven't much sympathy for "|" and feel it should be deprecated as much as possible. Resolvers should continue to accept it and treat it as synonymous with a drivespec ":". On that note, though, should they treat all "|" as ":", or just those that appear to be a drivespec?

If ":" or "|" ever become legal characters in Windows paths… then what.

Empty path segments

Empty segments in the path: collapse them? Depends on OS?

This gets tricky round-tripping on Windows with UNC paths.. I'd have to experiment again to give you some good examples though. I decided not to worry about it too much).

MP3 related

Lossless MP3 editing on Windows

To be absolutely safe:

1. mp3packer -b 320 -r inputfile.mp3 tempfile.mp3 to all but eliminate bit reservoir usage
2. Edit tempfile.mp3 in mp3DirectCut
3. mp3packer -s -t -z tempfile.mp3 outputfile.mp3 to recompress to VBR

However it's generally safe just use mp3DirectCut on the files, if all you're doing is trimming silence/noise from the ends. Generally. Problems:

• If you cut off frames from the beginning, then the new first frame(s) might not be decodable because their main data began up to 511 bytes back, not counting frame headers. That is, it was in the frames that got cut off. It would be nice if mp3DirectCut would color such frames so you can make an informed decision as to which ones are safe to cut, if any.
• If you cut off the last frame, then the last 529 samples of the last frame you kept won't be decoded because there's no more padding.
• Due to granule overlap, 288 samples at each truncated end will be about inaccurate and too quiet because the rest of their info was in a frame that's no longer there.

I'm not sure how mp3DirectCut deals with encoder padding and delay in the LAME tag.

There's another lossless editor called mpgedit. There's a console version (mpgedit) and a graphical version (xmpgedit), and it is available for multiple platforms. However, when I tried it on Windows, it didn't really work at all. Its editing interface isn't intuitive, and mp3DirectCut seems to just be a little better all around.

I used to use mpTrim for trimming silence from the ends, but the freeware version only works on smallish files, and it has nothing resembling a wave editor, so you have to 'earball' it. I now prefer mp3DirectCut.

MP3 splitting

MP3 splitting, such as with a cue sheet, is a can of worms. MP3 frames always output a fixed number of samples (1152, if 44.1 kHz). Thus, cutting on a frame boundary means cutting on an 1152-sample boundary, usually. You can't really cut with any more precision than that. However if you include an extra frame beyond the one you're cutting in, and then adjust encoder padding & delay values, your "gapless playback"-aware decoder can output just the samples you wanted. AFAIK, the only splitting tool that will do this correctly is called pcutmp3, a java command-line program. mp3DirectCut does let you overlay a cue sheet, but it only cuts on frame boundaries, doesn't include overlapping frames, and it messes up the padding & delay tags.

MP3 gapless playback

LAME always stores padding & delay values in the LAME tag, so it's always "gapless" in the sense that the number of samples output will be correct, given a decoder that supports those values. But it's my understanding that if you just encoded the tracks one at a time, LAME didn't have access to the previous and next track's frames, which are needed in order to know how to best encode the frames near the boundaries of the current track. From what I've read, it seems 288 samples on either side of the boundary between a pair of tracks will be at half volume (not confirmed, though). The psychoacoustic model may also be affected for adjacent frames (I think 2 frames before & after). I can't say with certainty what really happens, but you don't get the same samples at the boundaries when encoding separate tracks as when encoding the whole album at once. So the proper thing to do, to get split tracks, is to encode the entire set of tracks all at once by invoking LAME with the --nogaptags --nogap options. The problem, though, is that the resulting files will have no padding (except in the last file), which is a situation that players (even foobar2000) generally don't account for. These unpadded files won't be played to the very end. What a player has to do, to support playback of such files, is start feeding the next file's frames to the decoder when the end of the current file is reached. As far as I know, very few players do it. Rockbox supposedly does. Do any others?

Anwyay, the HydrogenAudio forums have a number of posts suggesting that --nogap is deprecated or obsolete, but unless I'm mistaken, that's clearly not the case. Abandoned, yes. Deprecated for basic gapless playback, yes. But obsolete, no. If properly supported in players, it should offer superior gapless playback, at least from a technical perspective, if not an audible difference.

I'd be interested to see the results of splitting a pair of tracks with pcutmp3, as compared to encoding them with --nogaptags --nogap. Does it produce the same frames and padding/delay values? Even if it doesn't, does it produce the same output when decoded?

MP3 SHOUTcast stream transcoding

Before I switched to AAC+, I was looking into offering a low-bitrate MP3 version of my audio stream by connecting to the regular stream as a client, transcoding it, and feeding it to another server. This isn't necessary when using recent versions of the SHOUTcast DSP for Winamp; it can feed multiple streams at different bitrates to different servers. But I was using an older DSP so that I could use an external MP3 codec, and had to come up with a workaround. I didn't get very far in my research, but did find several options for grabbing a stream to a file on Unix:

mpg123 -b 4096 -s http://212.72.165.24:9154/ > out.mp3
gnetcat -l -p 8081 212.72.165.24 9154 | lame --mp3input -b 64 - out.mp3
mpg123 -b 4096 -s http://212.72.165.24:9154/ | lame --mp3input -b 64 - out.mp3

And then there was also fIcy: an icecast/shoutcast stream grabber suite.

I don't remember if I tried any of these. The idea though was that if I could send the data to a file, the file could actually be a named pipe into icecast or something. However it didn't occur to me that I'd probably lose the protocol data (stream info and song titles). So it was starting to look like the only real option was going to be streamTranscoder, last updated in 2004.

I don't care anymore. I'm just keeping this info for possible future reference, because I remember it was hard to find some of this info.

Info about my old SHOUTcast setup

My previous setup used the 'Radium' pirated Fraunhofer MP3 codec at 96 kbps CBR, which provided excellent sound quality due to its lowpass filter discarding frequencies above 11.5 kHz, and its ability to force the use of L/R stereo mode rather than poorly-tuned M/S. MP3 codecs, especially Fraunhofer's, do very well at sub-128 kbps bitrates, especially when their lowpass filters are set to cut off the bandwidth-intensive higher-frequency audio.

Why didn't I use LAME? Well, I tried it several times, but for various reasons, I was required to use a codec packaged as Windows .acm file, and LAME's ACM build isn't nearly as tunable as the command-line or DLL versions. The biggest problem is that they've set its lowpass filter a bit too high, around 14 kHz, I think, with no way to change it. The result is inaccurate, noisy upper midrange, which is tolerable for speech, but that's too much high end to try to cram into a 96 kbps MP3 stream of music; something's got to give. They try to make up for it a bit by requiring joint stereo, which lets the encoder decide whether to use L/R or M/S in each frame. A lot of effort has gone into tuning LAME at higher bitrates so that it makes a good guess for that decision, but at 96 kbps CBR, it's no good; the result is too much noise and/or poor stereo separation. If I could do the encoding with LAME's command-line interface, it would be no problem, because I could just force L/R stereo mode and a ~12.5 kHz lowpass, which give it better high end than the Radium codec, with no degradation. If the stream wasn't limited to CBR, I'd use VBR and joint stereo, for which LAME has been tuned very well.

I use AAC+ now. Screw MP3.

Custom compiling AACGain

I'm going to try to build an SSE-optimized (Pentium III and up) version of AACGain. I'll trace my steps here.