2007-01-28

A difference in mindset

Some of my favourite literature are unix man pages. When I got started on unix, the system (a 3B2 w/ SVR2.0 IIRC) we used at the vakgroep AIV only had printed documentation. But it had all of its man pages in neat little red binders. The format fit exactly the way I absorb information, a quick overview, followed by a linear list of features.

Commands or options listed on the left hand side, with their explanation on the right. As a quick reader, I've never had any trouble distilling the information given in a man page. If I don't understand something, I just keep on going until I either find an explanation or get lost entirely. No problem. A second reading will often put things in a different perspective. And a third or fourth careful reading may sometimes clear up some assumptions or delusions as well.

I read the man page for sh(1), ksh(1) and ksh93(1) at least once a year. And I learn from it. I know most of the features supported by the original Bourne shell, plus the differences with the non-existing POSIX shell, Sun's ksh and AT&T's ksh93. I must admit I'm not confident with most bashisms, though.

The funny thing is that, until I encountered the dummy guides, I had never considered that some people do not like to read through a full description of all the features of a piece of equipment or a piece of software. It came as quite a revelation to me that most people, in fact, do not want to understand things, they just want them to work.

So the type of documentation a geek like me prefers is more like:
option x will do ...
option y does ...
etc., while a non-technical person will prefer text like:
To perform action X, select/press/dial ...
If you want to ..., do ...
The difference is like the P/J difference in the Meyers-Briggs typology. An old style hacker/nerd prefers to be given a set of options to explore (P), where another would prefer a more goal-oriented(J) approach.

I'm not saying one approach is better than another. I'm too much of an observer for that. I like to watch and see how people differ, than take a position on one side or another.

Let us be different.

Jim Gray tests Thumper

Jim Gray has tested Thumper on windows server 2003 and ntfs, in collaboration with Johns Hopkins U.
The summary is very positive:
This is the fastest Intel/AMD system we have ever benchmarked. The 6+ GB/s memory system (4.5GB/s copy) is very promising. Not reported here, but very promising is that we repeated most of the SkyServer Query log analysis on this system – performance was 3x to 100x what we experienced on previous systems – largely due to the 64-bit SQL and to the 16GB of RAM. We hope to report the SkyServer query results soon.
Nice box,Thumper
(fixed typo)

More Adaptive Replacement Cache Algorithm

Based on a conversation during the recent nlosug meeting, I've updated the wikipedia article for the ARC with a better explanation of the algorithm. The language is now more tangible, and the terms used are closer to the original literature.





2007-01-20

Adaptive Replacement Cache in ZFS

Last week, I could not reach the OpenSolaris source browser. I was looking for an explanation of what is called the 'ARC', or Adaptive Replacement Cache, in ZFS.

In contrast to the venerable UFS and NFS, ZFS does not use the normal solaris VM subsystem for its page cache. Instead, pages are mapped into the kernel address space, and managed by the ARC.

Looking through the zfs-discuss archives, I did not find any explanation of the ARC, except for references to the solaris Architecture Council, which is useful enough in itself, but does not deal specifically with paging algoritms...

Googling around, I finally found some useful references: Roch Bourbonnais explains the acronym, and refer to the IBM Almaden research lab, where the Adaptive Replacement Cache algorithm was developed.

In the original IBM version, it uses a cache directory twice as large as needed for the cache size. The extra space is used to keep track of recently evicted entries, so we know if a cache miss actually refers to
a recently used page or not.

After I created the wiki entry I came up with this visualisation of the cache directory:

. . . [1 hit,evicted <-[1 hit, in cache <-|-> 2 hits, in cache]-> 2 hits, evicted] . . .

and the following for a modification in Solaris ZFS, which knows in advance that it
should not throw out certain pages:

. . . [1 hit,evicted <-[1 hit, in cache <-|non-evictable|-> 2 hits, in cache]-> 2 hits, evicted] . . .

The inner brackets represent the actual cache, while the outer brackets show the virtual directory, referring to evicted entries. The total size for the cache is of course fixed, but it moves freely between the outer brackets. In addition, the divider in the middle can also move around, favouring recent or frequent hits.

Because the cache is mapped into kernel memory, this puts quite some stress on 32bit (x86) systems, as the 4GB address space on that architecture is shared by kernel and user space. Space used by the cache limits the size of user processes. Don't run your DBMS on one of these.

Links:
Wikipedia: Adaptive_Replacement_Cache