Scripting Filenames with Spaces: A for Replacement

Okay, this will be a quickie. Really.

I use for loops in scripts all the time, but for chokes on files with spaces in the name. After years of finding ways to avoid this problem any way I could, I've finally found what I believe to be the solution.

The for command treats any whitespace as a line break. So if you try to use for on a list of files (or folders or whatever) the loop will not function properly on items with spaces in the name. The for command will treat everything after the space as a new item.

Let's say you want to copy every file that starts with "Systems" in the folder "Start" to a folder called "Finish." The "Start" folder contains the items:

Systems Boy

Systems Girl

Systems Baby

James Bond

Bat Man

Using for would go something like this:

for item in `ls Start | grep Systems`

   do cp "$item" Finish

done

The for command, however, will treat "Systems" and "Boy" as two separate items, and when the cp command is issued it will complain:

cp: Systems: No such file or directory

cp: Baby: No such file or directory

cp: Systems: No such file or directory

cp: Boy: No such file or directory

cp: Systems: No such file or directory

cp: Girl: No such file or directory

The solution is to forego for and pipe your command to a while read statement like in the following example:

ls Start | grep Systems | while read item

   do cp "$item" Finish

done

The quotes around the $item variable are essential. Without them the script will fail in the same way as for. With them, the script works like you wish the for loop did and is able to properly copy files with spaces in the names.

Nice. And by "nice" I mean halle-fuckin-lujah!

I should give some credit for this tip. I originally read about it here at MacGeekery, but didn't quite understand it until I read some posts on various forums. Of course, there's nothing like a real-world problem to solve, and I had this as well to aid my understanding of this method.

"Go Away..." Redux: Simplifying the Complex

Well, it's official. The "Go Away or I Shall Replace You with a Very Small Shell Script" post is the runaway hit of the season. It's the most popular — or at least the most commented on — post on this site since "Getting Back to (Search) Basics," which is odd considering how different the two articles are: One is a simple, utilitarian shell script; one is an opinionated rant. Strange what strikes a nerve. But that's not really what this post is about.

In the "Go Away..." post I postulated, in essence, that perhaps this latest generation of users seems to be less tech-savvy than mine because they've had technology handed to them on a silver platter. I was pissed off and in my annual funk, or bitch-mode, or whatever the Hell it is that makes me so irritable 'round this time of year, and I needed to vent, and my logic skills were not at their finest. My brilliant theory may have been somewhat half-baked (although I still think it's kind of interesting). In the comments, people either agreed with me or didn't, and if you disagreed with me and I flamed you in any way, I want you to know that, though I'm not exactly apologizing for it, your comments are truly appreciated. I try to respond to every comment on the site, because responding forces me to rethink and restate my position on things, and that tends to get my brain a-workin' and that's always good (even when it keeps me writing 'til 4 AM). So on thinking about my theory in "Go Away..." and the the comments therein, some things occurred to me that I'd not considered in the original rant, and these lead down some interesting systems-philisophical roads.

The basic idea that I want to talk about here is simplicity. I realized that as time has gone on, maybe users are less tech savvy not because they're jaded and spoiled — no, maybe they haven't changed at all. Maybe it's because technology has grown so much more complex. I know that in our lab this is the case. When I first started this job, nearly six years ago, we were using Mac OS 9. Overall, as you probably know, OS 9 was a far less capable OS in terms of network capabilities, multi-user capabilities and reliability. But, man, was it easy to administer! Mac OS X is vastly more useful — has vastly more capabilities — but it's also infinitely more complex, and therefore more difficult to setup and maintain. Apple has done a great job of making OS X almost as easy to use as OS 9, from a user standpoint, but the simple addition of a multi-user environment, for instance, (not to mention the UNIX layer, networked home accounts, permissions and the like) has made it more difficult to use than previous Mac operating systems. And this additional complexity has spread throughout the entire lab, thus making the lab itself more difficult to understand and use. I once had a girlfriend who preferred McDonald's to Wendy's because the Wendy's menu offered too many options. In a complex world there is a certain appeal to simplicity.

Apple has done an absolutely amazing job in the realm of simplification. Mac OS X, as incredibly complex as it is, is still the easiest OS in the world to use. Mac hardware — even the pro hardware — is unbelievably easy to set up. And the unmitigated success of the iPod over literally all its competition — even cheaper, more feature-laden players — is proof positive that simple sells, and that Apple is the master of this philosophy. I first understood this when Safari was introduced a few short years ago. Here was a browser from which they'd stripped out numerous features, but which (after the release of the tab-enabled version) I found myself using as my primary browser. Though I did recently switch, for certain unavoidable professional reasons, I still prefer the Safari experience to that of any other browser. And part of the reason is because of its simplicity. Safari is a joy, in large part because it's simple and easy as Hell to use. You find this approach in every Apple product. It's a hallmark of the brand, and it's the reason Mac users are so loyal.

When I started this job I was a Macintosh SysAdmin. Mac only, in a mid-sized, very cross-platform networked lab with a separate SysAdmin for each platform. Over time we've restructured the staff, and in recent months responsibility for the whole lab has suddenly come under my purview. I now spend a lot of time thinking about how the lab is currently constructed and how I want it to be in the future. Right now, things work pretty well overall, and our network is quite powerful. There's a lot a user can do to leverage the power of this network if they have some understanding of how it works. The problem is, there's no way they can possibly understand how it works with their limited time and access. In "Go Away..." I said that, to a certain extent, people need to understand the tools of their trade, and I stand by this. But how well do they need to understand? How deep should their knowledge run? Should the car mechanic have a complete and thorough understanding of the way each and every individual car part is machined or manufactured? Should the car owner? Probably not. And I would argue that users of our lab should not need to have much understanding of the nuts and bolts of its construction in order to make use of the majority of its power. The functionality of our lab, like with an Apple product, should be almost self-explainatory. This is what I think about when I think about how I want our lab to be: How do I make the complex simple?

I look to Apple for ideas along these lines. How does Apple streamline and simplify the use of a product or a piece of software? There are a few ways. Firstly, they remove the unnecessary. If it's something people aren't using, they strip it out or they hide it. Apple is really good at deciding what the most important features of a product are and stripping out — or at least hiding to a certain degree — everything else. Everything you need to know is right there in front and working as 99% of people would expect, and more advanced functionality is hidden away in places where more advanced users know to look. So that's step one: remove the cruft. Take out everything we don't need and present a limited set of the most essential options.

Step two is what I guess I'd call sensibility: presenting things in a logical and intuitive manner. Once we've boiled functions down to the essentials, how do we present them to the user in a straightforward and easily understandable way? This step is hard because it requires a good understanding of users in general, and of our particular user base specifically. How do our users tend to use the computers in our lab? A great way to find this out is to ask them. I try, when I'm implementing a new feature — or updating an existing one — to ask a sample of users how they'd use the feature, or how they'd expect it to function. I also look to Apple again for inspiration. How does this feature (or a similar one) work in a Mac product? Why did they choose this method? What about it makes sense? If you can get a sense of how users are using your lab and the computers therein, you can then implement new features in a way that's easy for them to learn and remember.

Step three is consistency. Final Cut Pro is a great example of Apple's consistency. I've been using FCP since version 1. We're on version 5 now, and yet I've almost never had to relearn how to do anything in the program. With all the myriad new functions added in the last seven or eight years, almost every command functions exactly the same way it did in version 1. The interface layout is almost the same in every way. Apple may add features, but they're insanely consistent when it comes to maintaining usability from version to version of FCP. It's this kind of consistency that makes continued use of Apple products a very organic and easy affair. Similarly, in the lab, o nce we implement a new feature in a sensible way, maintaining the way that feature works from semester to semester should be as consistent as possible. This is not always easy, and sometimes even Apple themselves are to blame for broken consistency. God knows, differences even between Panther and Tiger have complicated the consistency problem for me and my users, and there's not a lot I can do about this sort of thing. But as long as general consistency is considered and maintained whenever possible, and combined with sensible implementations, it goes a long way to simplifying use and creating an environment that "Just Works." An example of where sensibility and consistency have broken down in our lab is the numerous password servers we employ. We have seven or eight (I can't even remember anymore) different password servers for the various platforms and web application on out intranet. This means that passwords for Macs are stored on a separate server than those for Windows or Linux or any of the other authenticated systems in our lab. This is neither sensible nor consistent. Users' passwords match at the outset of their enrollment here, but a change on any one platform will render the set of passwords inconsistent between platforms, and this is extremely confusing to users. I've spent untold hours working on (and writing about) this problem and plan to implement at least part of my solution this summer. Suffice to say, inconsistency in user data and workflow does little to simplify lab operations, and does a great deal to complicate them. And this makes the lab much harder to use than it should be.

The final step in simplifying the lab is education. Once intuitive, sensible, consistent features are implemented, we need to tell people how they work. Fortunately, if your implementations are truly intuitive, sensible and consistent, this shouldn't be that difficult, and should be something you don't have to do over and over again. Instructions on how to use the lab should be simple, clear and to the point. They should be illustrated whenever possible. And they should be documented online somewhere that is intuitively, sensibly and consistently accessible. Relearning the network should be as easy as opening and skimming a web page.

What we're really talking about here is the user experience. I think, personally, my job as a SysAdmin and Lab Administrator, at its heart, is really all about creating a user experience. Right now, in many ways, the user experience in this lab is more akin to Windows than it is to Mac OS X. There are too many steps to do simple tasks, things don't work as you'd expect, and there's a lot of confusion about how things work. Hopefully, it's getting better. Unfortunately, there is probably no way for me to make every change I want to make in one fell swoop. Changes to our existing infrastructure need to made, to some extent, with legacy users in mind. Plus there just isn't time to plan and implement everything at once, nor would it necessarily be wise to do so. So we move in baby steps. Occasionally leaps and bounds. But always piecemeal. In the end we hope to have a vastly simplified and, as such, vastly improved user experience that allows our users to focus less on how our lab is set up behind the scenes and more on how they can use it to do their work. That is what I see as "my job."

Thanks again to readers and commentors on this site. You've given me a lot to think about and I appreciate it.

UPDATE:
Another blogger, software developer Daniel Jalkut, posts some similar (and similarly long-winded) thoughts on simplification from a software development perspective on his Red Sweater Blog.

UPDATE 2:
Yet more. Seems to be a hot topic these days. (Via Daring Fireball.)

Apple Software RAID Tests or: What to Do if You Don't Have an Intel Mac

Yes, I got very excited about what I saw in Boot Camp, though for maybe different reasons than most, and now I'm over it. I don't have an Intel Mac, so I couldn't really do much with the Boot Camp Assistant, but I did get turned on to the latest software RAID capabilities included with the Mac OS. Specifically, two items I was previously unaware of: concatenated RAID and the new resizeVolume verb included with the latest version of the diskutil command. Since I promised I'd report about anything I found, and since I did take the time to perform a battery of tests on all things Apple Software RAID, I thought I'd share what I found, though there's not much here that's very exciting, or perhaps even very enlightening. Still, what follows is a fairly thorough explanation of the various RAID options available in Mac OSX. For PPC, that is.

First of all, software RAIDs cannot be partitioned, which I actually didn't realize, though it does make sense.

Attempting to Partition a Software RAID: No Go

(click for larger view)

The two most common and useful types of RAID are Striped and Mirrored. A RAID stripe combines two drives for increased storage and performance. A mirror combines two drives for redundancy. Here are some details on creating RAID stripes and mirrors on Mac OS X.

Striped RAID

  • Increase in capacity
  • Increase in performance
  • No redundancy
  • Works best with disks of equal size/type
  • Can be used with disks of different sizes
    • Consequences unknown
  • Deletes disk contents on creation

Mirrored RAID

  • Half capacity of combined disks
  • No performance increase
  • Redundancy for disk failures
  • Works best with disks of equal size/type
  • Can be used with disks of different sizes
    • Total size/redundancy is equal to the smaller of the two members
    • Other consequences unknown
  • Deletes disk contents on creation

Most of this I already knew, though I did discover that the RAID mirror can also be set to automatically rebuild in the event that one of the disks fails. Unfortunately, there is no notification for disk failure. So you can automatically rebuild the set, but you have to manually check its status. This is obviously putting the cart before the horse. Kind of stupid. But whatever.

RAID Mirror AutoRebuild: I Didn't Know it was Broken

(click for larger view)

What I did not already know about, however, was something called concatenated RAID. Turns out concatenated RAID (or "JBOD" for Just a Bunch Of Disks) is the oldest form of RAID, and has generally fallen out of fashion. No one uses concatenated RAID anymore. With the current size of hard drives, there's little reason to.

Basically, concatenated RAID does only one thing. It combines a set of drives for increased capacity. There is no performance increase. Concatenated RAID can be used with combinations of drives of any size and can be added to dynamically. If you add a disk to a concatenated RAID, the added disk is erased but the contents of the RAID remain untouched. There is also no redundancy with concatenated RAID, so if you lose one drive in the set, you lose all the data on the RAID. Here's how concatenated RAID works in Apple's Disk Utility.

Creating a Concatenated RAID Set: Data Will be Deleted

(click for larger view)

Concatenated RAID

  • Increase in capacity
  • No increase in performance or redundancy
  • Works with any set of disks
  • WILL delete disk contents on creation
  • Unmounting one member of the set will cause the RAID to fail completely; RAID must be mounted/unmounted all at once
    • Rebooting fixes the broken RAID
  • Adding to the set deletes data on the new disk, but keeps data on the existing set
Updating a Concatenated RAID Set: RAID Data is Preserved

(click for larger view)

Resizing Partitions with diskutil resizeVolumes

Finally, in Mac OS X 10.4.6 there is a new option for diskutil, the command-line version of the Disk Utility application. This new verb is called resizeVolumes and is intended for doing just what it says: taking existing partitions and resizing them dynamically. Unfortunately, as indicated in the comments of this MacGeekery article, though the option is included in the PPC version of 10.4.6, it only works on GPT (GUID Partition Table) formatted partitions which are not normally created by PPC Macs. PPC Macs use the APM (Apple Partition Map) format for their partitions, so on my machine the command failed with the following errors:

Attempting to Resize an APM Volume on PPC: Fat Chance!

(click for larger view)

This appears to be more than just a partition format problem though. The new Disk Utility in 10.4.6 allows for creating partitions with the GUID scheme:

Choosing a Partition Scheme: No Help for the Intel-Challenged

(click for larger view)

But attempting to resize these with diskutil still fails thusly:

Attempting to Resize a GPT Volume on PPC: Nice Try!

(click for larger view)

So, it would appear that, although there are some intriguing new partitioning tidbits in Mac OS X 10.4.6, they are not to be enjoyed by the Intel Mac-less. Alas! Such is life on an educational salary. If I can convince someone to buy me an Intel Mac, I'll surely mess around with this stuff some more. Until then I guess we'll leave the real cutting edge stuff up to the early adopters.

Lucky bastards!

To Repair or not to Repair

I don't really understand why everyone's all of a sudden interested in the Repair Permissions function of Disk Utility. But lately there's been a flurry of writings on the subject of whether or not you should repair permissions before and/or after a software update. It started way the hell back in god-knows-when (actually it was in May of 2005) with an article by a guy called Rosyna who happens to make Unsanity's haxies. This was only recently followed up with two posts by the brilliant author of Daring Fireball, John Gruber. In his second post he blasts MacFixIt for recommending the procedure. And now we have a rather thorough response from them as well.

Why, it's a regular permissions flame war, I tell ya.

Well, I wanted to weigh in on this from my own personal experience, briefly. As someone who manages rooms full of Macintosh computers that are chock full of every imaginable piece of software — both from Apple and third-party vendors — on a daily basis, I have to say, repairing permissions before and/or after a software update — or on a regular basis for that matter — is a perfectly reasonable precautionary measure. I can't tell you how many times a simple permissions repair has fixed an errant problem on a workstation or a client computer. Repair Permissions has saved my ass on countless occasions. And as far as repairing permissions affecting software updates goes, it seems to me that if an incorrect permission can affect overall system or application behavior, it's quite likely it could affect a software update. What happens when the software update package runs and hits a file it's supposed to modify or replace but can't because the permissions are incorrect? Well, seems to me like that could screw things up and leave you with an incomplete install of the update. And that could be bad. This doesn't sound like voodoo to me; it sounds like common sense.

So why doesn't Apple recommend repairing permissions before/after software updates? Well, I have long thought they should. In fact, I think it would be really smart if Apple's software updaters checked the permissions of any dependencies needed by the updater and alerted you to any inconsistencies, then allowed you to choose whether or not to proceed with a repair of the incorrect permissions, and finally proceeded with the update. I doubt this will happen, but a boy can dream. I think the reason Apple doesn't recommend repairing permissions on any sort of regular basis, or before or after a software update, is because they are loathe to admit the generally sad state of permissions settings in the OS. Permissions get changed on a very regular basis. Epson printer software regularly changes permissions on my systems. Flash Player updates do too. And that's just off the top of my head. The fact is, just about any third-party app that installs with an installer (as opposed to drag and drop) can change permissions on any file it wants, and this happens routinely. I've even seen Apple software updates change permissions on my systems (though, admittedly, this hasn't happened for a long time). Basically, Apple just doesn't want to say up front, "Look, this software update might break something, or a third-party app might have changed something that will break something when you install this update, so you should take certain precautions," because doing so would be like saying, "there are huge problems with our permissions model which has gotten much better, but is still seriously flawed in many respects."

So there's an unspoken rule that you should do some general permissions repair on a semi-regular basis. Actually, it's not even unspoken, it's just buried in Apple's Knowledge Base. The fact that the Repair Permissions function exists at all is proof that there is wonkiness in the permissions system, and that running this routine can actually fix stuff. It sure makes a lot of sense to me to perform whatever fixes you can before running a software or OS update, and if that includes Repair Permissions, then that's fine. It takes about one minute of my time and, if nothing else, is well worth the peace of mind it provides.

My question is, why now, guys? Why all of a sudden is this such an issue? The Repair Permissions function has been around for years, and MacFixIt has recommended using it for almost as long. And why John Gruber cares what I do to my system before and after a software update is beyond me. But frankly, I think I trust the MacFixIt guys — and myself — on this one. Don't get me wrong. I love Daring Fireball. It's one of my favorite sites. But MacFixIt — like myself — is in the business of troubleshooting. They/we know about what works and what doesn't. Out there. In the big, bad, complicated world of multiple computers with every imaginable combination of hardware and software, where things most definitely and most decidedly go very, very wrong. I mean, where do you go for troubleshooting advice?

Anyway, I think this whole thing is silly. I respect both these sites immensely, and it pains me to see them argue over such a trivial piece of advice.

Guys, can't we all just get along?

Boot Camp: Shut Up Already

Okay, sure, maybe there's a hint of sour grapes here, my being still Intel Macless and all, but really, please, just stop it already. If I read one more article about Boot Camp — this one included — I'm liable to puke. Just stop. Stop writing about it. It's all been said. It's all been done. You're just repeating yourselves. And it's driving me crazy.

I just clicked a link entitled "Apple to complete Intel transition by end of 2006" on MacFixIt which, rather than leading me to an article about the Apple Intel transition, actually lead to a five page write up on — yup, you guessed it — Boot Camp. I assume somewhere in the five page article is a line or two about the Intel transition and its imminent completion, but if you think I'm going to wade through another opinion piece on how well Windows runs on the Mac, you're freakin' out of your freakin' mind.

People have been running Windows XP on hardware from vendors the world over for, like, seventy billion years now. I can't imagine the experience would be much different on a Mac. And from everything I've read, this is indeed the case.

Sure, it's great we can do it. Just, please, stop writing about it. You're killing me.