bits, bytes, packets, scripts...

Some devops links and resources

2011-11-03T20:22:00.002+01:00

At a recent Amsterdam Middleware Meetup I promised I would post some links about the devops movement. I decided to replicate that post here for easier reference.

Start with the devopsdays.org site created by Patrick Debois, who coined the term 'devops'.

It's not easy to come up with a good definition. Devops is a movement where system administrators and developers work together to continuously build and maintain a single system including application, infrastructure and process. It builds on old and recent ideas from software development and lean systems thinking.

Some key concepts and a few links:

Bringing together people and practice from development and operations.
Infrastructure as Code
Continuous Improvement
Kanban for development
Kanban for operations
Criticism (we're over half a year further, and there are now devops workshops, and books:

    Jez Humble and David Farley, Continuous Delivery: Reliable Software Releases through Build, Test, and Deployment Automation (informit.com) takes the concept of fully automated deployment to its extreme. It covers practically every part of the deployment pipeline, including the various stages of automated testing, the enormous impact of distributed version control system, and a good management summary in the last chapter.

    Michael C. Feathers' book Working Effectively with Legacy Code is all about dealing with code that has no (automated) tests. Feathers has a very simple, pragmatic definition for legacy code: The word Legacy here says nothing about age or quality alone. To him, Legacy is code without tests.

   Code without tests will resist change. It will fight back. If you don't have tests, you will never know the impact of any change you dare to make. And when you do have tests, better make them run every time you change something.

    The bulk of the book deals with various methods for bringing code under test and start refactoring. This is where this book differs from many others on testing; Feathers actually comes with strategies to increase test coverage, instead of just promoting Test Driven Development or writing tests as 'a good idea'. Some of the methods he proposes are not pretty, but they get the job done. They prepare code for change, and that might actually include getting rid of significant parts of the code base.
It is very inspirational: It led me to see how I could write tests for deployment scripts (in Korn shell), and make me explore the various testing frameworks in python/jython (with junit thrown in for reference). I learned something.

    Donald A. Reinertsen
    The Principles of Product Development Flow: Second Generation Lean Product Development

   While Lean and the Toyota Production System are typically associated with manufacturing, Reinertsen shows how the conditions of product development change the playground. By focusing on Economic principles first, he shows that a lean approach in development may mean something different for development than for production. For instance, while variability is undesired in production and manufacturing, it is something you want to seek out in product development.

Original posted on the Middleware Meetup Message Board

Michael Nygard on "Agile Operations in the Enterprise" (InfoQ)

2010-05-22T21:56:00.000+02:00

InfoQ has an excellent article on Agile Operations in the Enterprise by Michael Nygard. (posted 2010-05-21)

In the introduction, Nygard defines Agile Operations, and how they are all about the principles, not just tools or procedures.

One paragraph in the intro stands out, as it addresses a major misconception about agile operations (or #devops):

Like agile software development, agile operations emphatically does not equate to cowboy administrators running amok on the systems, without plan or documentation. Quite the opposite, agile operations requires great self-discipline. Operators must commit to putting everything into version control. They must accept nothing less than 100% automation. Manual actions must never be permitted.

The article goes on about typical enterprise issues like Audit and Compliance (SoX), and ITIL.
And the quote above already implies about how Agile Operations deals with these challenges.

Nygard continues with the bigger challenges of History and Culture before concluding with a couple of warnings.

[... ]a team without the agile principles can emulate the practices but will not derive the benefits

Focus on the agile principles and automate everything.

Automated Acceptance Tests are not worth the cost

2010-04-09T14:21:00.001+02:00

InfoQ cites Jim Shore (author of The Art of Agile Development), who in has abandoned Automated Acceptance tools because they're not worth the cost, as he reported in February, in response to a question from Gojko Adzic. This was followed up by Jim Shore with a summary of Alternatives to Acceptance Testing, and a twitter debate about Fit recorded and commented on by Eric Lefevre.

So, everyone re-emphasizes that getting business folks together with developers and having them talk through examples is still a must-do, whew. But regarding automating these examples, Shore, Rainsberger, and Marick say no. Others argue yes. -- An interesting debate indeed. What say you? -- InfoQ

So, I'm not in application development, I've only recently arrived at Agile via the #devops movement, and I've been gathering many references to things I learned about in Gent.

What strikes me is that Jim Shore does not say no to automated testing. Number one in his list of alternatives is still his 'defect-elimination workhorse' of Test-Driven Development (TDD).

And that number one actually consists of three seperate practices: unit test, focused integration tests, and end-to-end integration tests.

Those are all part of the Continuous Integration paradigm: ship early, ship often. And all of these tests are essential for such practices as refactoring. (Or we will never be able to fulfill one primary requirement for the latter: unchanged behaviour).

Automated builds and automated testing are all about keeping the feedback loop short.

Acceptance tests come down to one simple thing, or just one single word...

Trust

And trust is based on the relation that we have with the customer. We build that by sitting down with the customer and getting agreement on requirements - frequently. That is a different kind of feedback loop, in which we demonstrate we understand those requirements.

Now this is serendipitous - it's friday:

jamesshore Just released full text of "Trust" from Art of Agile Development: http://bit.ly/ccwnN

I swear - I didn't peek.

Q42 Techops - devops reinvented, or parallel evolution?

2010-04-04T16:23:00.002+02:00

A foaf (actually, more the soaf - son-of-a-friend) just started his new job at q42, a young team of programmers building solutions focused more on the user than on the customer.

All employees (except the office manager and general manager) are interaction engineers, with a passion for all aspects of web development.

And q42 has no separate IT department or System Administrators...

That task is shared amongst the staff: (paraphrasing from Dutch)

So what is included in Technical Operations? Everything. Think about managing servers, network, security, accounts, bugtrackers, our time tracking en planning tools, subversion, software, licencing, hardware, desktop installation, internet connectivity, WiFi, phone exchange, backups, office alarm, et cetera.
That is indeed a lot. Who's going to do it all? All of us. Yes, everyone indeed. But Q’ers don't have to worry, they won't be saddled with a task if they haven't got the time or know nothing about the subject. From time to time it will happen that a Q'er will get a Techops job assigned, or get to answer a question. Most TechOps work is done in pairs. It's more fun and and automatically ensures the sharing of knowledge.

Sounds like a nice application of devops principles.

How they deal with switching priorities may need some help in order to scale:

If a request has extreme urgency, in nine times out of ten, the requestor is a fool: he or she should have thought about it earlier. If indeed the matter is urgent, one can always go to the work planner, who decides who's best suited to pick up the issue and won't let the same person fall prey to such interruption.

The shop looks pretty agile to me - without the buzzwords. I'm interested in seeing how they develop.

"Agile does not solve your problems, it just makes them painfully clear"

2010-03-26T18:20:00.000+01:00

Serge Beaumont Xebia explains why failing a sprint is good:

"Agile does not solve your problems, it just makes them painfully clear."

This is the essential element of Continuous Improvement, be it Lean, Agile, or whatever: Failure enables Learning. Only by exposing failure can we improve.

This is also why ITIL has processes for Problem Management and Error Control. They allow an organisation (team, department, company, or enterprise) to inspect its own practices and improve on them. This is why Agile is the Lean approach to software development.

So let's all fail, frequently... and show how we learn, recover, and improve.

Avatar, with spoilers

2010-01-09T01:21:00.002+01:00

Just watched James Cameron's Avatar last Jan 8 with my eldest Son(18).

It's an experience.

This was on the big, big screen, with the full 3d effects. And it really felt different. The whole movie was built to make people feel immersed into the world of Pandora. I enjoyed it. Over two and a half hours, no breaks, and action, stereotypes, action, wonderful effects, fine actors, action, and action.

Blunt spoilers ahead after the jump...

My first impression at the start of the movie when we see the space ship with Jake Sully arrive at Pandora was a feeling of losing my innocence. What I saw there did not need any suspension of disbelief - it was out there, and it was real, even though I knew it was just a better model than many others I had seen before.

I was ten when Star Trek aired here first, followed by Space 1999, saw Kubrick's 2001, then Star Wars, Alien, and Blade Runner. Star Trek focused on the people, the story and the concepts, effects took the place of props and sets, because they were less expensive. Space 1999 was another space opera, with even more footage being reused from one episode to another. The original Star Wars trilogy tried to tell an old story with innovative design and effects, and probably was the first SF/Fantasy movie trying to create a different world, the next attempt that I remember well must have been Brian Froud and Jim Henson's World of the Dark Crystal.

Alien and Blade Runner created a credible picture of the future, not by presenting vast landscapes and beautiful vistas, but by going the 'noir' way - focusing on the inevitable grittiness of basic human life, and thereby avoiding any speculation about the advanced technology we are supposed to have created in the far future.

Avatar perhaps uses the same trick of spending more screen time on things we're not supposed to know or understand:

We see the corporation's headquarters and the labs with the 3d high-resolution screens - so the viewer can gape and wonder how we will be doing that in tens of years time, but it's all there - obviously created by people, and we see people using it.
We see a strange world with a strange biology - and nothing is explained - even the 'science crew' knows almost nothing - we can only watch and wonder.
And we're given plenty of time to enjoy the sights.

What's less...

There are silly and annoying bits, and I'm not talking about Sigourney Weaver smoking inside. That's just a good joke on us. Smoking in the 22nd century apparently is a rare luxury (and is done inside - outside will kill you).

No, I'm talking about other things. The things that threw me out of Pandora back into my seat were things like:

"The Avatars are hybrids created out of human and Na'vi DNA"

Advise to future moviemakers - dont even hint at an explanation, unless you lampshade it. "Of course Pandora has a different genetic code, so we can't just mix our DNA with theirs..." Don't make up some fake concept that's completely off the mark (midi-chlorians?)

Either avoid the issue entirely, or make it part of the plot and give it real screentime.

The close up of a blue hand - with a human thumb

Ok, the Na'vi are obviously graceful, very humanlike, bipedal mammals, or we would not be able to establish them as in our mind as 'people'. So they have only 4 digits instead of five. They've got blue skin - which numbs our mind's eye - we don't look for detail in blue (Ever watched the sky up close?), and Cameron deftly avoids the uncanny valley, by making the Na'Vi both sufficiently different and similar to us as required for the story.
But that thumb somehow put me off.

Fortunately, these moments were few, and the movie pulled me back in quickly enough.

I'll probably want to see it again soon.

[edited to repair a broken line - Jan 11]

devopsdays 2009, the missing talk (OpenQRM)

2009-11-09T22:48:00.003+01:00

A private openQRM Cloud use-case for a developer team

In my post about devopsdays I left out Matt Rechenburg's presentation on OpenQRM, a tool for provisioning appliances within any kind of virtualisation, or just on bare metal. The company that created QRM stopped developing it just after Matt had convinced the owners to make the product open source. Matt stayed on as coordinator for the open source project. Currently OpenQRM does not have commerical support, but the developers are available on a time and materials basis.

OpenQRM starts from the principle that a kernel, a program, a directory, or anything you can install on a computer system is basically just a file. It's just data. But this data is the most important piece that you need to make the system function, and you would then store this data on the weakest component of the system - a mechanical device inside a random piece of hardware.

OpenQRM therefore assumes that all code and configuration is stored centrally, and can be accessed/mounted across the network when required.

Taking the everything-is-a-file-concept to an extreme, OpenQRM allows you to build a complete application stack by dragging and dropping some components, and some memory into a single image and then schedule that to be deployed for an interval of a few days, weeks, or just for a couple of hours. The tool then schedules the necessary resources, and shortly before the requested timeframe, activates the image in a physical or virtual environment (OpenQRM supports all major virtualisation techniques, and new ones can be added as plugins.)

As an example, Matt showed a webdeveloper's appliance and suggested that you'd schedule its deployment for five minutes before you're arrival in the office :-)

More can be found online in Matt's presentation (ODF):

http://www.openqrm.com/userfiles/Private_Cloud_Computing_Use_Case.odp

Or watch an earlier demo of OpenQRM on Youtube

ZFS dedup done

2009-11-03T00:54:00.001+01:00

Devopsdays 2009

2009-11-01T21:21:00.008+01:00

I returned home from Devopsday late saturday night, and having been digitally challenged during the trip I will try to summarise my impressions of the weekend over the next couple of days.

Devopsdays was a small conference about a couple of emerging themes combining Development and Operations:

The first theme is the realisation that if you want to build a scalable infrastructure, you need to automate deployment and administration of that infrastructure and the applications that run on it. System configurations becomes just another type of code to be developed, tested, integrated and deployed. Deployment becomes Release, Configuration becomes development, the ITIL processes for Incident, Problem Management become debugging. Change management becomes release management.

Second, another recent development in the last ten years has been the advent of Agile. I've only recently encountered the Agile Development movement, and although it's far from a Silver Bullet, it does appear to address some of the essential issues in software engineering in terms of Fred Brooks' original analysis.

Originally intended as a light-weight alternative to the waterfall model of software development, Agile transposes the stages of the waterfall model into concurrent processes, introducing feedback everywhere. Requirements analysis continues long after coding starts. Rapid prototyping, user stories, continuous integration, test first design are just a few methods used in Agile to shorten the feedback loop for developers. But while methods and processes are important, the real focus of the Agile movement is on communication and collaboration, in the end making developers and users jointly responsible for the end result.

The Devops Concept (for want of a better name) is about merging the two approaches and how to apply Agile principles to System Administration and how to get people in Operations and Development to collaborate on deployment.

At the conference, the two day programme was split in two: Talks and Presentations in the morning, and free-form discussion/presentations in OpenSpace format in the afternoon.

I would say that the talks and discussions focused on three themes:

(Open Source) Tools for automating IT Operations
Collaboration between Development and Operations
Agile methods and principles for Operations.

Speakers and participants of the conference were from all parts of the industry - developers, operations, consultants, small startups, and large enterprise.

Lindsay Holmwood explained his work on cucumber-nagios - combining 'cucumber', a tool/language for expressing tests in almost human readable scripts with the nagios monitoring tool, resulting in behaviour-driven monitoring. This was a very fast-paced presentation.

Teyo Tyree of Reductive Labs talked about the principles behind Practical Infrastructure Automation, referring to the "James White" Manifesto on Infrastructure (now up at github). Of course he focused on tools like Cfengine, Chef, and Reductive Labs' own Puppet, but he also sketched the challenges for the big enterprise with a multitude of services, commercial application stacks, and many platforms. He strongly suggest to start with baby steps: Implement Configuration tools like Puppet in a reporting state first, and use its reporting mechanism to create a history of change from within the system. Leverage the legacy CMDB, and work within established change control policies.

Agile coach Rachel Davies focused on the Agile principles, methods and tools, in particular about User stories and how to use those to identify non-functional requirements (requirements that do not add measurable value to the product, but that improve the product by reducing risk.)

Mattias Skarin presented a case study of using Kanban that helped to get Operations and Development collaborating closer. The key to Kanban is twofold: visualise task planning and put a hard limit on the amount of work in progress. The important thing here is that there is no single best design for a Kanban board - the team has to create what's best for them. After a couple of iterations, or sprints, the team may decide to add a category of work, or drop a phase from the progress axis.

Chris Read of Thoughtworks told us about Build pipelines, and how to take Continuous Integration several steps further into Continuous Deployment.

During Openspace, Jochen Maes started a hot discussion about the merits of distributed version control - and how to minimise the risk of branching: Even when developing in their own copy of the repository, he expects his developers to check in frequently, and at the same time rebase with the main repository at least once an hour. This way, individual developers run their own tests frequently, and then merge in any updates to the main source tree from other sources into their own copy without polluting the upstream. This requires strict discipline, but the result is that any time a change causes the build to fail, you can always fall back to the previous build. Also, because merging code in a distributed version control system like git or mercurial involves merging the complete history (and not just the current state), it is easy to identify which code change was responsible.

UPDATE: It so happens that George Neville-Neil just posted an article about this in his Kode Vicious column at ACM's Queue: Merge Early, Merge Often.

[edited]

OpenSolaris bijeenkomst, Amersfoort, 8/10/2009

2009-09-28T07:47:00.001+02:00

Zegt het voort...

De eerstvolgende bijeenkomst van de NLOSUG (OpenSolaris User group NL) zal plaatsvinden op donderdag 8 october, waar we weer te gast zijn bij Sun in Amersfoort.

Om onze gastheer en de catering een goede schatting te geven van het aantal bezoekers, graag aanmelden via email op nlosug@sun.com, onder vermelding van: Aanmelding-NLOSUG-okt2009

AGENDA

19:00 Welkom

19:30 Opening

Update on NLOSUG
Bart Muijzer
Operating Systems Ambassador - Sun Microsystems
What's New in OpenSolaris
Boekbespreking
Jan E. Kuba van Bijnen
Unix/Solaris system & network consultant
Confused by Solaris-es
Bart Muijzer
Operating Systems Ambassador - Sun Microsystems

PAUZE

Contributing to OpenSolaris Repositories
Eric R. Reid
Staff Engineer at ISV engineering - Sun Microsystems

22:00 Afsluiting

Locatie:
Sun Nederland B.V.,
Saturnus 1,
3824 ME Amersfoort ( Route )

NL Opensolaris User Group

Robin Harris surprised about Apple dropping ZFS

2009-08-29T23:24:00.007+02:00

Robin Harris regretfully eports: "Apple kicks ZFS in the butt", and speculates why Apple did not ship ZFS as part of Snow Leopard, its latest incarnation of Mac OS X:

What did it in? Maybe it was a schedule problem - file systems require a lot of testing - and rewriting all the other bits took precedence. NIH - Not Invented Here - syndrome is another possibility. Or perhaps the uncertainty of Sun’s future led Apple to pull back.

Or maybe they just decided customers wouldn’t know enough to care, so why bother? Whatever the reason it is a major step backwards for the PC industry.

I can think of a few practical reasons myself.

For now, I'll try to focus on one: Apple's not ready for it. And perhaps, neither are the users.

Rule #1: Apple designs and sells systems that are supposed to just work. No hassle, no jumping through hoops, no bells, no whistles. Pure form, pure function.

ZFS was designed to do one thing really well. You give it your storage and your data, and it will go to extreme ends to protect your data. It will need at least two disks in order to do that.

Apple's desktop systems and notebook computers still come with only one disk inside.

Use an external disk for ZFS redundancy? The ultimate Rule #1 violation. The whole point of an external disk is that it can be disconnected. The whole point with ZFS redundancy is that you don't want to even create a hint that one of its disks could be disconnected.

After all, there is only one storage pool, and ZFS will take care of that, thank you kindly, sir. The firewire/USB/eSata cable is just the rope that the user needs. Allow them to disconnect the drive, and friendly as Mac OS is, you can provide sufficient automation to recognise that the cable was disconnected, show a kind Applely warning that "Mac OS cannot protect your data if you do not reconnect the external volume."

People are just not ready for this yet. You don't want to run ZFS with h/w that can be disconnected on a whim, or purely by accident, it's just asking for trouble. After three or four friendly warnings, people will ignore them. Yes, I know! Stop nagging me! The ease with which ZFS could recover from this will only encourage people to become careless, annoyed, or both.

ZFS will be ready for consumer use when all the volumes in a storage pool will reside together in the same device. Detachable storage is great for backups, especially with a notebook, but would have to be redundant itself. So now we're talking about at least four disks: two inside the computer, and two outside to protect against physical loss. Let's just stop there.

My conclusion is that Apple probably has taken the right decision business-wise, but I hate them for not having the hardware to support it. Maybe they will get back to it, and I look for the day when they will have notebooks and iMacs with an even numbers of disk slots.

How many languages were used to build that?

2009-08-19T02:27:00.004+02:00

I spent two days tracking a nasty bug in PRM, a performance monitoring module in Sun Management Center, and wondered how many languages were involved with creating the product.

So far, I've found the following:

java
tcl
sh
awk
sql
perl
C/C++
XML
ASN-1

Are there any missing?

How many languages can you use for product development without things getting too complex?
Some of the languages have a specific focus, like sql for db queries, sh for system adminstration (start/stop components), and the data description languages XML for SMF and ASN-1 for describing snmp mibs.

Automated Configuration Management

2009-06-25T16:40:00.003+02:00

Paula Rooney talks about the advantages of Puppet, an open source tool for automated configuration management.

These tools are essential if you want to scale up deployment of servers, especially now that more services are being hosted on virtual servers, making deployment of the hardware, O/S, and application completely independent.

"One senior systems engineer at Digg.com was able to rebuild 60 [virtual] machines from scratch in two hours [using Puppet] that would have taken two full days of work if done manually. 'And I was largely a spectator,' said that engineer, Paul Lathrop, of Digg. 'Now that’s automation.' ”

Puppet is not the only game in town, if you're interested in these tools, you also need to look at cfengine, isconf, and bcfg2. Interestingly enough, two of projects appear to use trac for release management. These are all open source projects, with various levels of commercial support. The commercial nova edition of cfengine, adds some extra features for a price.

The value of these tools is that they offer automatically enforced policies, with implicit reporting of any exceptions.

Een kandidaat voor ZFS at home?

2009-03-06T19:54:00.003+01:00

Willem de Moor van Tweakers.net Nieuws de presentatie van de Asus Eee Station PC NAS, op Cebit deze maand.

Het doosje is wat steviger van opzet dan de gebruikelijke Home NAS appliance. Zo bevat het vier Gigabit Ethernet pporten, 2GB aan DDR2 Ram geheugen en een Intel Atom N270processor, geklokt op 1,6 GHz. Er is ruimte voor twee harde schijven in een raid0-, raid1-, of jbod-configuratie. Het prijskaartje komt op een 700 dollar.

Het systeem draait onder Linux, vanuit een 512MB Flash geheugen. Hoe veel moeite zou het kosten om dat te vervangen door OpenSolaris en ZFS?

Starslip by Kris Straub - rebranded

2009-01-21T21:52:00.006+01:00

(via Scott Kurz and webcomics.com)

Kris Straub has revamped his web comic Starslip - a science fiction comic about a museum starship that's been recommissioned for battle. As he describes on webcomics.com, he has gone through a long and painful process going back and forth about the decision and finally decided to get done with it.

The comic has a new look, a new title, the artwork has been improved, and the storyline is restarting as well. I've long been wanting to add it to my daily web comic routine, but as of today, it is in.

Go have a look!

Packet Construction Set

2009-01-01T12:43:00.002+01:00

Here's something I'm going check out. A toolkit for creating protocol stacks in Python called the Packet Construction Kit. The admin for the project is George Neville Neil of freebsd and ACM Queue KV fame.

Best wishes to all for 2009!

Auto-scaling your cloud services - benefits and risks

2008-12-07T15:16:00.002+01:00

Read George Reese on Why I Don't Like Auto-Scaling in the Cloud first.
It's a good story about some of the risks of cloud-computing.

However, Rob La Guesse makes a valid point.

I guess the key phrase here is Everything in Moderation.

A personal example -- on a really small scale -- follows

I pay fastmail to manage my mailbox with numerous features for a fixed fee a year.
I get a bandwidth allowance tied to the size of my mailbox - at 1GB per month. Since first signing up I got 2GB of spare bandwidth - which I haven't ever needed yet, but I could buy more spare capacity in advance if I ever expect to need more.

This example illustrates what the customer and the provider expect: predictable service in an unpredictable world.

The prepaid spare capacity allows the service to pick up any peak load, and allows the customer to put a cap on any expense.

If additional load would translate directly into additional revenue, you have the time to decide to divert some of that revenue back into more capacity and availability.

Henk

Free Multipage Icon Editor found

2008-10-31T22:29:00.005+01:00

I downloaded the greenfish icon editor and was able to create a new icon for uwin/sh in minutes, combining it with the original 16x16 into a single file. On the desktop and in explorer uwin/ksh now shows the 32x32 icon, while the title bar still displayes the smaller one.

Mission accomplished

The files have been updated.

A new little icon for the UWIN ksh console

2008-10-24T01:18:00.011+02:00

I've been using the korn shell for over twenty years now. Not only on solaris/opensolaris, but I also use it on windows, through the Uwin project, a port of the ast toolkit - a portable set of all standard unix tools, including the shell, but also nawk and the usual cat,sed,tr, list of tools.

Today, I finally created an icon for the UWIN shell console - the original icon, with the letters UWIN in bright colours is not displayed on the console window title bar. It is too large, and the title bar reverts to the standard C:\ icon.

I never paid much attention to it, but tonight I figured out the issue - the title bar icon should be no larger than 16x16 pixels. Once I realised that, a new fitting icon was created quickly, and here it is:

Yes, it's tiny. It's also just what I want for the shell window in my windows quick launch toolbar.

Here's a bigger view of the design. Getting the dollar sign sharp at three pixels wide was a bit of a challenge, but I was quite satisfied.

There's a zip here with the icon as a windows .ico, plus the .png above, and the development version with an odd colour in the background, which can easily be turned transparant when you save with Irfanview.

I've modeled the icon after the 16x16 Windows CMD icon. That is, I've used the measurements of the frame, although the pixels are mine.

Author: Henk Langeveld, shell icon, Oct 2008: zip
Copyleft: This is a free work, you can copy, distribute, and modify it under the terms of the Free Art License http://artlibre.org/licence/lal/en/

In addition, the files can be distributed under the Common Public License
http://www.opensource.org/licenses/cpl1.0.txt

Update:
Windows actually uses multi-page ICO files. For the title bar and toolbars it'll look for a 16x16 bitmap or substitute another icon of that size. For the desktop it will use the largest icon available. Does anyone know of a simple method of combining icons in a single file? Free tools preferred. I did see a method using Photoshop, which seems a bit of overkill...

Enjoy

Chimps, elephants, and now magpies...

2008-10-24T00:19:00.006+02:00

Last year I attended the Tinbergen Lecture in Leiden, where prof Frans de Waal told us about his research on animal behaviour. He is well known for his work with chimps, but he also reported how they were finally able to demonstrate Mirror Self Recognition behaviour with Elephants. Previous attemps had failed, apparently because testers had been using small mirrors, which would not have large enough for an elephant to even see a significant portion of itself.

Now the same test has been performed with birds - magpies to be precise - not by de Waal, but he comments on it at PloS: doi:10.1371/journal.pbio.0060201.
He explains the significance of the test: It's not about intelligence or about the concept of self-identity as such, but a side-effect of the species developing stronger social skills. Primates, elephants and magpies all have complex social lives and that requires them to be be able to distinguish each other and put oneself in another's position to understand them.

And who knows what they think about each other...
And that's what Dave Kellet thought as well.

The trouble with /var in unix systems

2008-09-21T22:18:00.002+02:00

Mike Gerdts suggests using /var/share for sharing data between boot environments in his March 2008 post Future of OpenSolaris Boot Environment management.

The trouble with unix /var is that it is a grabbag, in that it is used both for storing system data (i.e., identity), as for storing application data.

/var/ is intended for variable data that should persist through a reboot: short lived temp files, spool directories for transient files (printing,mail), and whatever data applications might wish to store (/var/opt).

However, as said above, it also contains data about a systems 'identity', specific settings like cronjobs, printer settings, network configuration, definitions of services that altogether make up what the box 'is'. This also includes the list of installed software in /var/sadm. At some point these all moved into /var from /usr or /etc in ancient unix past.

Root was supposed to be small then, and did not contain a lot that could change, limiting the risk of a long fsck if there were a lot of modified files in the root fs. With just enough stuff to allow the system to boot, if you lost any other filesystem, if you had at least a running system you could attempt to repair or restore whatever was broken.

Mike's suggestion is very sound, as it does what is required. System identity is kept in the root fs (/), while application data, whether it's persistent or transient, goes into /var/share, keeping /var as part of the root fs, even though the 'share' name may be a bit of an unlucky choice - it's used in /usr/share and other places for architecture-neutral data that can be shared with other systems (nfs clients).

lplist - show solaris "lp-lite" queues matching pattern

2008-08-05T00:14:00.007+02:00

This is something I've been using for years in various variations. "lp lite" was introduced as an alternative for the complexity of the sysv lpsched printer daemon. Instead of directories with printer filters and interface scripts, the assumption was that you'd be talking to a remote BSD print spooler.
For that, you needed the printer address and printer name, nothing more. Paper is typically A4 or Letter, depending on geography. What we print is typically postscript, or plain text. All of those are assumed to be preset on the other end.

This little script will scan your printers.conf (using lpget list) and dump all printer definitions matching your supplied regular expression pattern one line at a time, giving the queue name first, followed by all 'lpset -a' keyword=value settings for that queue.

By putting the queue name first in the output, this makes it easy input for the next script in a pipe, as you can simply say

lplist hp | while read hpq more; do ...; done

to process all print queues with "hp" in their name.

The empty pattern will just dump all queue names.

#!/bin/ksh -p
#
# usage: lplist [pattern]
#
#
PATH=/usr/bin

lpget list |
nawk -F: -v pattern=$1 '\
/:/ {
if (!pattern || q~pattern) {
print q, def
}
q=$1
def=""
next
}
!pattern || q ~ pattern {
def=def" -a "$0
}
END { if (def) print q,def }'

Exercise for the reader: Rewrite in pure awk. That is, drop the shell.

Publishing and author's rights

2008-06-06T20:59:00.007+02:00

Scott Kurz of pvponline blogs about author's rights in the comic industry. The traditional American comic used to be funded through syndication. You got a deal with the syndicate and from then on you get paid and they own what you create. Forever. The music industry works pretty much the same way, except you cut a deal for a certain period or a certain number of albums, and they retain an awful lot of influence over what you actually create for them. Scott is one of those, who has been able to find a different way.

Scott Kurtz:
I am shocked at how many cartoonists I’ve talked to who think that a creator’s rights automatically revert back to the creator if something goes tits up. Unless that’s specified that is NOT the case. And even if that is specified in a contract, those rights can be tied up in negotiations or legal battles for years.

Ownership of your work is the single greatest asset you have as a creator. Never willingly sign that ownership away. I can never say that enough. Write that in a notepad 10 times every day and then show me that notepad at an upcoming convention so I know you’re paying attention.
more...

The problem as I* see it is with copyright law and the so-called work for hire. Publishers will make sure that they word any contract as a piece of work they contract out to you. You don't hire them to publish your work, they they hire you to create theirs. This is a standard deal, and unless you make any other provision, you will only get paid once, and never anything more. Modern copyright law requires that a writer/artist can get their moral rights asserted, but that does not change the business arrangement.

Let me get it straight: I have friends in the publishing industry and the life of a publisher isn't easy. A small publisher (of games/comics) with tiny print runs cannot afford any risk. I remember at least one that required you to sign a waiver before you where even allowed to send them a manuscript. It said in so many words: "We may or may not be working on products with content similar to your submission. If we don't accept your work, you may still see a product like it published by us in the future, and will not pay you.
By submitting your manuscript you agree that we can do this and you will not ever have any legitimate claim."

And you know what? It is the only way for a small publisher to survive. If you want a different deal, make sure you have already made a name for yourself. Then you can call the shots. Until then, don't make any assumptions.

*I am not a lawyer, find one yourself if you need legal advise, etc., etc.

oracle on t2000

2008-06-04T14:19:00.005+02:00

Since the issue came up I checked out some references on the performance of oracle (10g) on t1000/t2000. Here is some wisdom found via google.

Basic oracle stuff:
* oltp scales well on niagara - no floating point, typically i/o bound
* a bit of tuning to exploit maximal parallelism may be necessary
* lots of parallel queries change the profile from i/o bound to cpu bound

jvm
* an obscure oracle jvm trap could hang the installation procedure
* don't copy/clone an existing sparc deployment onto a t1/t2
* do you want oracle apps running on the same h/w?

* the t2k does well in the space/watts/performance arena

Bletchley Park short on funding...

2008-05-16T16:59:00.004+02:00

Bletchley Park, now a museum about its own role during WW2 is lacking funds to continue for more than another couple of years.

The estate used to house the team dealing with intercepting and deciphering German communications during the war, and it was here that the enigma code was broken.

Update 30/5: Bruce Schneier is looking into organising some relief...
Maybe he read it in the Reg...

July 6 Update: The Heritage Lottery appears to consider adopting the park...

Arthur C. Clarke

2008-03-19T12:39:00.003+01:00

Arthur C. Clarke is no more. I read Childhood's End around 1976, and it must have been among the first English books that I read, together with Asimov and Herbert.

Kitty Squid Crocheting

2008-03-15T23:17:00.003+01:00

Bruce Schneier's blog is about Security, except on fridays, when it's about squid.
(Although I suspect his friday squid blogs don't get posted until saturday)

Chronicles of a Yarn Obsession: Kitty Squid

This looks like a great toy for our LaPerms.
I never learned crocheting, but I may get Karin
to make this.

bash vi mode converging with ksh

2007-12-28T22:50:00.002+01:00

One of the great features of ksh (93 and sun's) is the option to invoke an external editor on the current command line, while preserving multi-line layout.

In vi-editing mode, the ESC-'v' keystroke invokes an external editor ${VISUAL:-${EDITOR:-vi}} on the current command line. Bash does this as well, but on exit, all newlines are replaced with semicolons, while ksh preserves them.

It's one of my minor nits with bash, but for me it's important. I like my 'one-liners' readable, even if they span half a screen... This is nice to have, as many of the script that I've written were prototyped on the command line.

Shortly after my original post I read the bash(1) man page and found me the 'lithist' shell option.

It works of sorts, but it's not the behaviour I'm familiar with in ksh.

When scrolling back in history, ksh truncates long lines and lets you scroll horizontally, while bash will wrap them across multiple lines. Not bad, but counter-intuitive in vi-mode. Maybe there's another option to escape newlines.
After invoking vi, and re-evaluating the command line, bash returns to edit-mode. Ksh returns to the default prompt.
Bash specific shell options are set with 'shopt' in contrast to 'set -o ' for traditional longname options. I'm not sure why there's two methods for a similar feature, a good guess is that it prevents collisions with other shell options. It beats the csh method of storing options in shell variables.
Minor nit: after completion, the shell prompt command counter ( ! in posix mode ) is not incremented until an extra newline is entered. The counter then jumps by two.

Another case for CIFS and ZFS?

2007-12-27T23:30:00.000+01:00

Microsoft has released kb/946676, detailing a problem with Windows Home Server shared folders.

When you use certain programs to edit files on a home computer that uses Windows Home Server, the files may become corrupted when you save them to the home server.

The article warns about certain applications that are not supported with shared folders. Users should copy their files to local storage before opening them with any of the suspect applications.

That basically halves the functionality of WHS, which is being touted as a NAS/backup appliance.

I wonder though, whether the problem described here is inherent to the way windows applications use their datafiles. The typical approach I remember is that applications do live updates to the original files, after making a temporary backup file. This in contrast to the traditional unix way of life, where you first create a working copy and when done use that to replace the original.

I really should build my own home NAS based on Solaris/CIFS server and ZFS next year. Let's see what kind of budget I have...

Update: It appears to be a reliability issue under heavy load, and was hard to reproduce. I claim it would not happen with NFS. That's why NFS (and any NAS) write performance can suffer badly, if you don't have the hardware to help it along.

Links:
mswhs.com UK fansite
Computerworld article

ksh93 performance through builtins - a small example

2007-12-18T00:22:00.000+01:00

I recently held a presentation for the Dutch OpenSolaris Users Group about the work of Roland Mainz on integrating ksh93. I focused on the history of unix shells, how ksh93 was accepted in the OpenSolaris project, specifically highlighting the OpenSolaris ARC process.

I did not discuss the relative merits and reasons for getting ksh93 included, but today I'll mention one reason in particular: shell script performance. Ksh93 is faster than any other POSIX conforming shell in executing code, not in the least because many standard unix commands have been included as builtins in the shell, allowing scripts to bypass fork() and exec().

Here is a comparison of two lines of code in ksh93. Their effect is exactly the same: Ten thousand times, they create and remove a unique directory. The difference is that the second time around, the ksh builtins for mkdir and rmdir are used.

glorantha 10 $ time for i in {0..9999}; do /bin/mkdir $i; /bin/rmdir $i; done

real 0m35.63s
user 0m1.44s
sys 0m1.65s
glorantha 11 $ time for i in {0..9999}; do mkdir $i; rmdir $i; done

real 0m1.44s
user 0m0.33s
sys 0m1.09s

And this explains why a builtin matters. Both the internal and external command are just as efficient if they're only invoked once. When you have to invoke an external command inside an inner loop, the unix fork()/exec() overhead add up.

But there are other ways of improving the above bit of code, if you're satisfied with a slightly different sequence of actions, and non-POSIX code due to the compact {start .. end } notation.

What I like about this code is that it's compact, and quite clear in its intention.

glorantha 12 $ time mkdir {0..9999} && rmdir {0..9999}

real 0m0.53s
user 0m0.02s
sys 0m0.51s
glorantha 13 $ time /bin/mkdir {0..9999} && /bin/rmdir {0..9999}

real 0m0.53s
user 0m0.03s
sys 0m0.48s

More later

Document the why in your code

2007-12-05T15:56:00.000+01:00

This article in ACM Queue reminded me of the most valuable
argument I once heard in favor of code comments.

You don't explain what the code does, you tell why it does it.

I'm all for the concept of self-documenting code, and except
for the obvious iterator i and counter n, I do try to find descriptive
names for my data structures.

The strongest argument against comments is that they can be wrong.

There's nothing as frustrating as a comment that contradicts the code.
Especially when you believe it.

So yes, comments are good, but let them tell why.

E8 - Lisi's Theory of Everything

2007-11-22T10:18:00.000+01:00

Noticed this thing in New Scientist this morning.

A fitting universal theory of everything, with no strings attached,
except for a couple of puns.

An_Exceptionally_Simple_Theory_of_Everything

26400 hits and counting... 'beowulf uncanny-valley'

2007-11-22T09:55:00.000+01:00

I saw the trailers for beowulf yesterday, and the first thing that sprang to mind was 'uncanny valley'.

The characters as a whole look larger than life, but all personality has been removed from their expression.

A real waste of talent.

I'm far from the first to realise this, as a quick search shows.

26,400 and counting, I'm sure...

party time? IPv6 is done!

2007-09-26T22:44:00.000+02:00

I found this in my mailbox yesterday, Sep 25. It's been a long time.
This message somehow touched me. People have been working on
ipng/ipv6 for over twelve years, and will continue to do so.

But how long before people will start using ipv6 in greater numbers?
What will be the killer app for ipv6? P2p? Games? HPC? Social Networking?

High Performance Computing will use ipv6, just because it provides ways of
handling massive amounts of data. No window scaling or other hacks required.
But HPC is not for the masses.

The killer app will be used by millions at a time. And it will be big enough to let
people want to make the jump, and join their friends. But it will only happen
when enough providers actually offer the technology for end-users.

But why would end-users want ipv6? To give up their little NAT router that incidently
protects them against prying eyes?

Or maybe that's it - give every internet subscriber a billion ip addresses and see how
ports canners deal with that. Hide your little pc in a sea of duds. Yeah, as if people
won't just choose the very first address out of the pool - most will actually let dhcp
do that for them.

Not that it matters, as port scanners are not where the risk is. Most malware on the
internet gets transported through human vectors anyway. 'Click here'. No, that's not a link.
I don't even pretend it's one.

Whatever it will be, it should require no more than one or two links to follow in order
to get connected to the internet mark 2, with glorious 128 bit addressing.

Cheers, mine's a half

Henk

To: ietf-announce at ietf.org
Subject: WG Action: Conclusion of IP Version 6 (ipv6)
From: IESG Secretary <iesg-secretary at ietf.org>
Date: Tue, 25 Sep 2007 14:30:02 -0400

The IP Version 6 Working Group (ipv6) in the Internet Area has concluded.

The IESG contact persons are Jari Arkko and Mark Townsley.

+++

A new Working Group, 6MAN, has been created to deal
with maintenance issues arising in IPv6 specifications.
The IPv6 WG is closed. This is an important milestone
for IPv6, marking the official closing of the IPv6
development effort.

The ADs would like to thank everyone -- chairs, authors,
editors, contributors -- who has been involved in the effort
over the years. The IPv6 working group and its predecessor,
IPNGWG, produced 79 RFCs (including 5 in the RFC queue).

Issues relating to IPv6 should in the future be taken up in
6MAN if they relate to problems discovered during
implementation or deployment; V6OPS if they relate to
operational issues; BOF proposals, individual submissions
etc. for new functionality.

The mailing list of the IPv6 WG stays alive; the list will
still be used by the 6MAN WG in order to avoid people
having to resubscribe and/or adjust their mail filters.

--------------------------------------------------------------------
IETF IPv6 working group mailing list
ipv6 at ietf.org


Administrative Requests: https://www1.ietf.org/mailman/listinfo/ipv6
--------------------------------------------------------------------

Windows Is Free...

2007-08-16T10:48:00.001+02:00

... at least people behave like it is. Sharing is seen as a social lubricant, if not a requirement to belong.

read more | digg story

The memory could not be "read" ...

2007-03-08T10:40:00.000+01:00

Windows keeps killing apps with this message:

The instruction at '...' referenced memory at '...'.
The memory could not be read.
Click OK to terminate the program.

After reading the MS KB article, I realised this is not such a strange message after all.
It's actually a very familiar old friend

no trains this morning due to bad disk?

2007-03-01T12:16:00.000+01:00

Just heard on the radio that the failure of the railway systems around Amsterdam this morning had been caused by a bad disk...

Curious.

Friends of Jim Gray suspend search

2007-02-18T02:18:00.000+01:00

Jim Gray is a pioneer in database research, and contributed greatly to the concept of transactions with roll-back and roll-forward, all very commonplace nowadays. Late january I posted about his test of Sun's thumper, apparently on the same day he went out sailing solo out of San Fransico Bay into the Pacific. He's been lost since. All efforts (and they've involved some very high-tech methods) so far have not resulted in any trace of Jim or his vessel, the Tenacious.

In the last several days, the Friends of Jim group has reviewed all the data with Coast Guard officials. The fact is that we have no evidence as to what has happened to Tenacious or to Jim Gray. Neither we nor the Coast Guard can come up with a surface search plan that is likely to find either Tenacious or Jim, given everything that has been done already.
Accordingly, the Friends of Jim group is suspending its active effort to find Tenacious that has been centered here at the blog. For both the Coast Guard and the Friends of Jim, “suspension” means that the active search has been discontinued due to exhausting all present leads and the lack of new information. Of course, should we or the Coast Guard receive any new information, we will investigate it.

Opensolaris zfs + dtrace guides available in pt_BR translation

2007-02-17T01:44:00.000+01:00

A few opensolaris enthousiasts have translated the guides to dtrace and zfs
into Brasilian Portuguese.

OpenSolaris i18n Forums: pt_BR translation zfs + dtrace guides available for review

I'm pleased to announce that the Brazilian Portuguese SGML & PDF version of the following book are now available in the Download Center:

Solaris Dynamic Tracing Guide ( Solaris 10 3/05 : SGML & PDF)
Solaris ZFS Administration Guide ( Solaris 10 11/06 : SGML & PDF)

Note to self: I need to share this with our DBA's...

HP announces support for Solaris-10 on Xeon

2007-02-11T23:19:00.000+01:00

The Unix Guardian says: HP announces support for Solaris 10 on Xeon. The reasoning being that Sun has no Xeon h/w of its own at this time. Interesting, as in my job the x86/x64 server h/w is all HP, running either Windows or Linux. Our ERP systems all run on SPARC/Solaris, though.

And the same article mentions Transitive chalked up HP for its h/w emulation technology, adding them to the matrix of multi-platform cross-overs.

A difference in mindset

2007-01-28T22:36:00.000+01:00

Some of my favourite literature are unix man pages. When I got started on unix, the system (a 3B2 w/ SVR2.0 IIRC) we used at the vakgroep AIV only had printed documentation. But it had all of its man pages in neat little red binders. The format fit exactly the way I absorb information, a quick overview, followed by a linear list of features.

Commands or options listed on the left hand side, with their explanation on the right. As a quick reader, I've never had any trouble distilling the information given in a man page. If I don't understand something, I just keep on going until I either find an explanation or get lost entirely. No problem. A second reading will often put things in a different perspective. And a third or fourth careful reading may sometimes clear up some assumptions or delusions as well.

I read the man page for sh(1), ksh(1) and ksh93(1) at least once a year. And I learn from it. I know most of the features supported by the original Bourne shell, plus the differences with the non-existing POSIX shell, Sun's ksh and AT&T's ksh93. I must admit I'm not confident with most bashisms, though.

The funny thing is that, until I encountered the dummy guides, I had never considered that some people do not like to read through a full description of all the features of a piece of equipment or a piece of software. It came as quite a revelation to me that most people, in fact, do not want to understand things, they just want them to work.

So the type of documentation a geek like me prefers is more like:

option x will do ...
option y does ...

etc., while a non-technical person will prefer text like:

To perform action X, select/press/dial ...
If you want to ..., do ...

The difference is like the P/J difference in the Meyers-Briggs typology. An old style hacker/nerd prefers to be given a set of options to explore (P), where another would prefer a more goal-oriented(J) approach.

I'm not saying one approach is better than another. I'm too much of an observer for that. I like to watch and see how people differ, than take a position on one side or another.

Let us be different.

Jim Gray tests Thumper

2007-01-28T17:04:00.000+01:00

Jim Gray has tested Thumper on windows server 2003 and ntfs, in collaboration with Johns Hopkins U.
The summary is very positive:

This is the fastest Intel/AMD system we have ever benchmarked. The 6+ GB/s memory system (4.5GB/s copy) is very promising. Not reported here, but very promising is that we repeated most of the SkyServer Query log analysis on this system – performance was 3x to 100x what we experienced on previous systems – largely due to the 64-bit SQL and to the 16GB of RAM. We hope to report the SkyServer query results soon.

Nice box,Thumper
(fixed typo)

More Adaptive Replacement Cache Algorithm

2007-01-28T08:46:00.000+01:00

Based on a conversation during the recent nlosug meeting, I've updated the wikipedia article for the ARC with a better explanation of the algorithm. The language is now more tangible, and the terms used are closer to the original literature.

Adaptive Replacement Cache in ZFS

2007-01-20T11:54:00.000+01:00

Last week, I could not reach the OpenSolaris source browser. I was looking for an explanation of what is called the 'ARC', or Adaptive Replacement Cache, in ZFS.

In contrast to the venerable UFS and NFS, ZFS does not use the normal solaris VM subsystem for its page cache. Instead, pages are mapped into the kernel address space, and managed by the ARC.

Looking through the zfs-discuss archives, I did not find any explanation of the ARC, except for references to the solaris Architecture Council, which is useful enough in itself, but does not deal specifically with paging algoritms...

Googling around, I finally found some useful references: Roch Bourbonnais explains the acronym, and refer to the IBM Almaden research lab, where the Adaptive Replacement Cache algorithm was developed.

In the original IBM version, it uses a cache directory twice as large as needed for the cache size. The extra space is used to keep track of recently evicted entries, so we know if a cache miss actually refers to
a recently used page or not.

After I created the wiki entry I came up with this visualisation of the cache directory:

. . . [1 hit,evicted <-[1 hit, in cache <-|-> 2 hits, in cache]-> 2 hits, evicted] . . .

and the following for a modification in Solaris ZFS, which knows in advance that it
should not throw out certain pages:

. . . [1 hit,evicted <-[1 hit, in cache <-|non-evictable|-> 2 hits, in cache]-> 2 hits, evicted] . . .

The inner brackets represent the actual cache, while the outer brackets show the virtual directory, referring to evicted entries. The total size for the cache is of course fixed, but it moves freely between the outer brackets. In addition, the divider in the middle can also move around, favouring recent or frequent hits.

Because the cache is mapped into kernel memory, this puts quite some stress on 32bit (x86) systems, as the 4GB address space on that architecture is shared by kernel and user space. Space used by the cache limits the size of user processes. Don't run your DBMS on one of these.

Links:
Wikipedia: Adaptive_Replacement_Cache

The CS system (6809)

2006-09-02T12:09:00.000+02:00

At last I found a reference to the group around the CS system, a SWTPc clone designed by Cees Schoenmaker.

The group frequently met at the monthly Delft Hobby Computer Club meetings in the cellars the Department of Electrical Engineering of Delft University in the early 80s.

server3.fastmail.fm has been down all day

2006-09-01T02:00:00.000+02:00

I can't get to my private mail.

Otherwise, the fastmail.fm service is pretty nice and reliable. Not today.

A typical case of Fsck You
Time to switch to Solaris/ZFS

2006-09-01T01:45:00.000+02:00

Bits form bytes

Bytes form packets

Without packets no digital networks

For traffic monitoring I prefer bar-charts

Bandwidth is a lovely vague word

You cannot change the speed of light

Every transaction requires one synchronous event