Apparently, I’m not the only one who has started commuting by bicycle lately, as this article in the most recent Economist points out. I find it very easy to imagine some of the bicyclist/driver altercations they describe.

I was almost nailed at a four-way stop at the beginning of the summer by a guy who rolled through a stop sign while talking on his cellphone. He didn’t just roll; he looked like he was going to come to a complete stop, and then gunned it just as I was crossing. He didn’t even know I was there (despite my safety-orange shirt). Just last week some lady in a minivan decided that she really wanted to make the green light that we were approaching. We were on a four-lane city street with no shoulder. Traffic was flowing in the left lane, but I was occupying part of the right lane and she wasn’t sure if she should try to squeeze between me and the traffic. As the green light got more stale, she decided to try it. She gunned her engine, accelerating madly until she was right next to me, only to brake abruptly as she doubted she could get through after all. She practically sideswiped the car to her left and practically knocked me off the road. People like her knock the mirrors off parked cars by forgetting how wide the piece of steel they’re piloting is. I would really hate to be a casualty of someone’s misplaced overconfidence in their spatial reasoning. Such a pointless way to go. The risks of a driver’s misjudgment are asymmetric: He risks his paint job, while he risks my life, or at least my skin and bones.

Bumpy Roads

WITH petrol the price it is, more and more people are riding a bicycle to work. In Broward County, Florida, about 35,000 people a month typically put their bicycles on a bus bike-rack, thereby shortening a cycle commute. In May of this year, 68,000 people did so. Denver saw 25,000 people register for a recent “bike to work” day, up from 15,000 a year ago. In Seattle cyclists complain about a shortage of bike stands, while in Portland, Oregon, some 6,000 cyclists cross just one of the city’s many bridges each morning.

Bicycle-boosters are thrilled with the sudden popularity of their humble machine. “Ridership is just skyrocketing,” says Elizabeth Preston of the League of American Bicyclists, a Washington, DC, advocacy group (even cyclists have lobbyists these days). Performance Bicycles, a retailer with shops in 15 states, says bicycle sales in June were the highest ever recorded.

But cycling’s popularity has a downside. The people of Portland, for instance, have been entertained over the past few days by a series of altercations between bicyclists and motorists. In one, a motorist and cyclist came to blows after the motorist berated the pedal-pusher for ignoring a stop sign. The enraged cyclist used his bike to batter the motorist’s car until a bystander punched him.

In Seattle, meanwhile, two cyclists were arrested after they attacked a motorist during a so-called “Critical Mass” ride, events where large groups of cyclists ride through city streets to demonstrate their right to the road. New York cyclists are up in arms about an incident in which a police officer, for no apparent reason, knocked a cyclist off his bike and then arrested him and tried to pretend the man had run into him until a video recording proved him wrong. And in Colorado, reports the Los Angeles Times, cyclists have been feuding with the sheriff of Larimer County for his aggressive—cyclists say unreasonable—enforcement of bike-related traffic laws. More seriously, most bicycle advocates say cycling deaths are sharply up, although the National Highway Transportation Safety Administration has no figures as yet for 2008.

After years of federal and local spending on bike routes and other amenities, most cities are ready to handle more cyclists. But many motorists simply don’t see their two-wheeled brethren or, when they do, find them aggravating. Managing more cyclists is going to take more than new bike paths or fresh stripes on the roads. It looks as though there is a need, on both sides, for a revolution in manners.

DC, by the way, is not one of the cities that has invested in infrastructure and is “ready to handle more cyclists.” Many of the thoroughfares do not have bike lanes or shoulders of any kind. I often have to resort to all sorts of side streets and shortcuts to get where I’m going, and some places I just can’t go by bike. Until roads are built with an extra foot or two for the rest of us, riders will keep getting killed in pointless ways (an intern was run down by a garbage truck just a couple of weeks ago), and people will keep throwing their bicycles through people’s windshields.

• Annocent. Sort of wishes it were Eno.
• Manic Toybox. Sort of wishes it were Radiohead.
• Prophecies Make the World Go 'Round (Do-Over).
• Taxi Bender.
• Fratchy Fro Froo Five.

The good mileage is not the main reason I’ve gotten so into biking lately, but it is certainly a bonus. Neither are the health benefits my main motivation, though they also are profound. No, the main reason I’ve been riding all over and around the capital of our fair nation is that it’s fun as hell.

These days, though, I’m catching myself getting just a bit smug on my daily commute. It takes me about 30 minutes to drive to work in traffic, though it’s only a six mile drive. Lights and traffic make it a chore, partly because DC has never heard of traffic sensors. Every single light is on a timer. Even if no car has approached 16th street from some minor side street in seventeen hours, that side street a green light every 45 seconds anyway. The 65 cars lined up on 16th street wait the 30 seconds, just in case a driver should happen to drive down that street someday, and then they continue on their way until they hit the next red light a couple of blocks down. Hurry up and wait. Hurry up and wait. I hate driving in this town.

I can bike to work in about 35 minutes. Part of that speed comes from being able to blow off red lights and stop signs when no cars are coming, as God intended for us all to do, and part comes from being able to hop up on the sidewalk and circumvent the long lines of cars waiting at stoplights. I’m usually the first vehicle at the intersection even if I pulled up last. Sometimes I have private moments of gloating when the same car passes me at several sequential stop lights. “See, buddy?” I find myself thinking, “You should ditch the car and ride with me. It’s just as fast and a hell of a lot less annoying.”

Now with gas prices as they are, I have another reason to gloat. As the poor souls who bought Lincoln Navigators and Ford Expeditions trudge down the road, watching the gas gauge dip visibly every block, I pedal along for free. I’ve filled up my car once since the beginning of June. It’s due for another sometime soon, but not until I next get in it, which could be late next week or even later. It only needs a fill-up now because I made two trips to and from Dulles airport a couple of weeks back. Instead of burning expensive gasoline, clogging up the air with my exhaust and clogging up the roads with a 1.5-ton,15-foot by six-foot steel contraption that transmits but a single piece of human cargo, I zip along on a 21-pound, two-foot by six-foot vehicle that runs on body fat. What’s not to love?

I had the good fortune of growing up about a mile from the Illinois Prairie Path, a miles-long rail-to-trail path that runs among Chicago’s western suburbs and connects to several other area trails. It’s possible to ride several different configurations of 40 to 60 mile loops, crossing very few roads, and without ever retracing your route. It became a summer pastime for me to set off on many mornings into the interstitial green spaces of the suburban sprawl around me.

In the summer of 1994, having dropped out of school the second time, I was living in rural Illinois, working as an apprentice cabinet maker for my older brother and, probably due to feeling trapped and hopeless in many aspects of my real life, I fell in love more than ever before with the limitless freedom I could achieve riding along the slick straight flat black country highways for hours. At the end of the summer, just a couple of weeks before I’d planned to set out on a two-week ride from Polo, Illinois to New Orleans, I was hit from behind by a car while out riding near dusk. I broke a vertebra and ruptured a disk in my back. My bike was mangled and one of the pedals was completely sheared off. Given that the driver was doing 45, and that I wasn’t wearing a helmet and for most of the 50 feet I rolled my head was above pavement, it could have been much worse. Nonetheless, it completely derailed my travel plans and about a year of my life. But I told that story already, a couple of years ago, one evening after a long ride on the road between Brno and Vienna.

And then, four years ago, I moved to Washington, DC. For the first three of those years, I don’t think I got on my bike more than twice or thrice each summer. I’m not sure what I forgot. It’s equivalent to loving ice cream, or golfing, and just forgetting to eat ice cream or golf for three years. I have no idea what happened.

But this year, it has all come back. Part of it was meeting a retired Canadian couple on a ferry from Sardinia to Naples in May. We were traveling Italy by train and Ryanair; they were doing it by bicycle. They had just ridden down the western coast of Sardinia, and were on their way to ride another week or so on the Amalfi coast. C was really inspired by them, in a way I haven’t seen her inspired by very many things. I was too, but only to the extent that it reminded me how much I love doing exactly what they were doing.

Immediately after we returned to DC, C left for Arizona for the summer to study Navajo. I started riding my bike to work, but the 13 year-old Trek was simply not cutting it for me. It made things a chore. Plus they have these things called hills here, a feature conveniently removed from my native northern Illinois landscape by the movement of glaciers ten thousand years ago or so. And several large examples of these hills have been placed between my house and the building where I work. So one Saturday in early June, I headed over to City Bikes on Connecticut Avenue to see what’s on offer these days. I didn’t mean to buy anything; I just wanted to window shop and do some test driving.

Unfortunately, I fell in love with a Jamis cyclocross bike that was just too much fun to ride. Compared to my huge heavy old Trek hybrid with the rear pannier rack, this thing was slick, small, nimble, and quick. I spent four hours test driving other bikes that were more reasonably priced. It went on so long that I had to take a lunch break from my bike shopping. But every few bikes, I would take the Jamis back out and head a mile or so down the Capital Crescent Trail just to see if it still felt so right. And it did. So I took it home. We’ve traveled 300-odd miles so far and we have many more to go.

The hills, they are nothing to me now. Soon I will conquer mountains. And soon after that, the farthest reaches of outer space. There are no limits when you get infinity MPG.

1. I bought the newest version of Photoshop Elements, since my old version won't run in Leopard.
2. I cropped, edited, improved contrast and saturation, corrected camera distortion, rotated, fill-flashed, and otherwise improved the pictures for your viewing pleasure. But they still maintain that in-the-moment rawness that you crave.
3. I installed Gallery (open source photo gallery software) on the website to add new functionality (but an inferior aesthetic) to the photo page. However, this is a tradeoff for the greatly reduced amount of time I have to spend manually editing HTML and tweaking CSS code. And you can view the pictures in multiple resolutions.
4. I have painstakingly written terse and cursory captions for over half of the photos.

So I hope you enjoy the presentation. It is here: Paul, Corinne, and Thea's Italian Adventure 2008.]]> 2008-06-29T16:54:12Z 2008-06-29T03:26:06Z tag:www.strangeproportion.com,2008://1.183 2008-06-29T03:26:06Z

We have a nice circuitous route planned which gives us ample time for visiting archeological sites, seeing art and architectural what-have-ya’s, not to mention a few days of quality beach time on the Mediterranean (my favorite place so far in the whole world). We’re taking our snorkels. We’re taking advantage of cheap RyanAir flights booked well in advance. We’re spending one night on a ferry from Sardinia to Naples. It’s got everything. The only thing it doesn’t have is a whole summer. Damn this whole “vacation time” thing.

I don’t have anything new to say about the film, at least not now, at midnight on a weeknight, but I did want to share the beautiful Carter Burwell song that plays as the credits roll. Once the song started, I decided to sit through the credits until I saw what it was. Only two songs are listed, one of them a Mexican traditional, and this isn’t one of them.

Carter himself (here) tells us that there are only 16 minutes of music in the movie, almost six of which are this as-the-credits-roll song. A little searching turns up the song title and artist, but finding more than the short excerpt he posts on his site is very difficult. It’s not on iTunes, and there doesn’t seem to be a soundtrack for sale. In the end, I dug through the HTML around his flash player and found the link to the mp3, so you can listen to it without having to pay the $9 to see the movie again. Although that wouldn’t be the worst thing in the world.

The song is called “Blood Trail.” As it fades in, you think it’s just someone walking through the sand. Then you realize it’s a bit too regular to be footsteps, but you’re still thinking about Tom Ed Bell’s dream to pay much attention. If you stand up and leave right away, you’re probably out of the theater before the guitar, bass, and drums come in around the 1:45 mark. That would be your loss.

What I noticed was that a French pet adoption website was loading something many hundreds of times a day, completely dwarfing everything else happening on my site. The webmaster had apparently set the background image on every page to load a photo from my server. And the particular photo? Well, you see, a couple of years ago, I wrote a StrPrpn post about blanketing my cubicle at work with hundreds of pictures of kittens (don’t ask). And so I found this picture on the web somewhere, copied it to my server, and stuck it in a post:

It’s not my picture. I’m not attached to it. But when a web designer sets it as the background image on another website and it gets loaded many hundreds of times a day, that bandwidth can really add up. So I composed a polite email to the person, in French. Well, that’s not true. I composed it in English and then fed it to Google Translate. I studied French once, long ago. If you’d like me to translate any Rousseau for you, or perhaps some Flaubert or Pascal, I could possibly do it. But compose a straightforward email asking someone to stop linking to a picture on my website? You don’t learn the words for that stuff reading 18th century philosophy. And C was studying, so I didn’t want to bother her to translate it. So I fed it to Google Translate, which does a passable job. It probably came across sounding a bit like this to the French person at the other end:

Hello, excuse please my french bad.

You use a picture of my web site as your foundation. Near half of bandwidth is used as readers of your Web site load this file on my server. The image is not up to me, so I do not worry if you move to your server. But do not please link to my site. The image that you link is:

http://www.strangeproportion.com/images/friends1024X768.jpg.

It is kittens.

Thank you.

I never got a response, and after a few days, I checked again and saw that this picture was still being loaded hundreds of times a day. So I decided to activate the .htaccess. This is a file you can put on your website to control what gets served up in response to various types of requests. With a little internet research, I figured out how to set it up to serve up a substitute picture when anyone links to an image file that’s stored on my server.

I spent a while trying to pick just the right substitute. I have some funny anti-Bush stuff lying around, and some inappropriate humor and whatnot. But these seemed like overkill. I wanted to send a clear message—“Hey, quit linking to my photo”—but do it with a sense of humor. I wanted to be on the edge of inappropriate without veering into lewd or tasteless.

Google had some choice images in response to my “hairy man in a speedo” query, and after perusing a few pages of hairy men, I found that one that called my name. David Hasselhoff in a speedo with a bright red background. What more could you ask for?

I grabbed it, loaded it up, and then checked the results. As perfect as I could have hoped. I was very pleased with myself.

(Click for a better view.)

They’ve since taken the picture down. But for at least a few hours, I bet there were a few French potential pet adopters who were a bit confused.

This list is a bit long because I went through a period of time between the ages of, say, 15 and 23 when I never kept a job longer than three months. They just got so tedious, you know? So I’d stop going. Like the guy in Office Space. His concept wasn’t particularly novel to me, and I couldn’t figure out why he spent most of the movie debating whether or not to stop going to a job he hated. Just stop going, dude. It’s easy. (Though, admittedly, he was an office dude, and he’d gone to college, so his job was less disposable than all the clerk jobs betwixt which I flitted more flitty-like than a honeybee in a clover field.

So, without further ado:

1. lawn mowing kid
2. video store clerk
3. ice cream store clerk
4. convenience store clerk
5. TJ Maxx clerk
6. Burger King clerk (2 weeks)
7. grocery clerk (produce)
8. plastic container factory worker (1 day)
9. convenience store clerk (different store)
10. street sign changer and tree trimmer
11. apprentice woodworker
12. Barnes and Noble clerk
13. convenience store clerk (different store)
14. gas station clerk (graveyard shift)
15. convenience store clerk (different store)
16. driver for furniture designer
17. bike messenger
18. van messenger
19. temp worker
20. Tower Records clerk
21. focus group participant (intermittent)
22. driver for demolition company
23. recording engineer
24. indie-rock bass player
25. lightbulb changer
26. graphic designer
27. textbook typesetter
28. bookstore clerk again
29. assistant bookkeeper
30. Thrifty Nickel layout guy
31. Visiting Foreign Lecturer
32. Business Analyst
33. Senior Financial Engineer

You think you can tell where in there I finally finished college, but you’d be wrong. I actually finished college just before “bookstore clerk again.” There was a little lull in there, and the job market in Santa Fe, NM for liberal arts graduates wasn’t the best. I couldn’t find a full-time job to save my life. So I found four part-time jobs instead.

When I was a twenty-something slacker, we always used to make fun of Systems Analysts. Not the people themselves, just the job title, because it meant absolutely nothing to us. One of the people in The Onion’s photo opinion section is usually a Systems Analyst, or at least that was the case in the mid-1990s. Maybe there are so many Systems Analysts in The Onion’s target demographic now that they’ve had to make some changes. Regardless, though, I am one of those people, from other people’s point of view, anyway.

But I think that what I do for a living rocks. It’s so much fun that some days I can’t believe they pay me to do this stuff. (On other days, when I’m doing tedious, bureaucratic business stuff—like updating spreadsheets that track the last time we audited process X, and trying to find the link to the drive where we stored the evidence of the audit, and then pasting the link into a spreadsheet and sending it to the guy who archives the spreadsheets—I remember exactly why they do have to pay me.) But primarily, the work I do, the part that I consider to be the part that rocks, is that I get to work on computer models. These models consume enormous quantities of data and try to resolve that data into very straightforward relationships among the quantities involved to predict useful things that you would never be able to deduce if you looked at a few hundred or even a few thousand of the data records. Instead, you follow this algorithm:

1. Dig into the data using various kinds of “computer programs”
2. Use “math” and “statistics” to find some relationships between the quantities
3. Design a model (an “equation”) that describes the relationships you found
4. Throw all the data at the model and see what comes out
5. Lather, rinse, repeat

Of course, there’s much more finesse involved in the process, but you probably don’t care too much about the details, so I spared you them.

Anyway, the reason I bring this all up is that The Economist recently featured an article that describes almost, but not quite exactly, what I get to play with all day long. The article focuses largely on analyzing consumer data and process-generated data to improve systems on the fly, while the models I work on are a little more static and straightforward. But the concept is the same: Distilling meaningful relationships out of piles upon piles of numbers and using those relationships to your advantage.

Algorithms: Business by Numbers
Sep 13th 2007
From The Economist print edition

Consumers and companies increasingly depend on a hidden mathematical world

ALGORITHMS sound scary, of interest only to dome-headed mathematicians. In fact they have become the instruction manuals for a host of routine consumer transactions. Browse for a book on Amazon.com and algorithms generate recommendations for other titles to buy. Buy a copy and they help a logistics firm to decide on the best delivery route. Ring to check your order's progress and more algorithms spring into action to determine the quickest connection to and through a call-centre. From analysing credit-card transactions to deciding how to stack supermarket shelves, algorithms now underpin a large amount of everyday life.

Their pervasiveness reflects the application of novel computing power to the age-old complexities of business. “No human being can work fast enough to process all the data available at a certain scale,” says Mike Lynch, boss of Autonomy, a computing firm that uses algorithms to make sense of unstructured data. Algorithms can. As the amount of data on everything from shopping habits to media consumption increases and as customers choose more personalisation, algorithms will only become more important.

Algorithms can take many forms. At its core, an algorithm is a step-by-step method for doing a job. These can be prosaic—a recipe is an algorithm for preparing a meal—or they can be anything but: the decision-tree posters that hang on hospital walls and which help doctors work out what is wrong with a patient from his symptoms are called medical algorithms.

This formulaic style of thinking can itself be a useful tool for businesses, much like the rigour of good project-management. But computers have made algorithms far more valuable to companies. “A computer program is a written encoding of an algorithm,” explains Andrew Herbert, who runs Microsoft Research in Cambridge, Britain. The speed and processing power of computers mean that algorithms can execute tasks with blinding speed using vast amounts of data.

Some of these tasks are more mechanistic than others. For instance, people often make mistakes when they key in their credit-card numbers online. With millions of transactions being processed at a time, a rapid way to weed out invalid numbers helps to keep processing times down. Enter the Luhn algorithm (see below), named after its inventor, Hans Luhn, an IBM researcher. The numbers on a credit card identify the card type, the issuer and the user's account number. The last number of all is set to ensure that the Luhn algorithm produces a figure divisible by ten. If it is, the card number has been properly entered and the processing can go ahead.

The Luhn algorithm performs a simple calculation. But the real power of algorithms emerges when they are put to work on much more complex problems. As far as most businesses are concerned, these problems typically fall into two types: improving various processes, such as how a network is configured and a supply chain is run, or analysing data on things such as customer spending.

UPS uses algorithms to help deliver the millions of packages that pass through its transportation network every day in the most efficient way possible. The simplest routes are easy to draw up. If a driver has only three destinations to visit, he can take only six possible routes. But the number of possible routes explodes as the destinations increase. There are more than 15 trillion, trillion possible routes to take on a journey with just 25 drop-off points—and an average day for a UPS driver in America involves 150 destinations. The picture is further complicated by constraints such as specified drop-off and pick-up times for drivers or runway lengths and noise restrictions for aircraft. “Algorithms provide benefits when the choices are so great that they are impossible to process in your head,” says UPS's Jack Levis.

Go here, go there

Solving this “travelling-salesman problem” means a lot to UPS. For its fleet of aircraft in America, the company uses an algorithm called VOLCANO (which stands for Volume, Location and Aircraft Network Optimiser). Developed jointly with the Massachusetts Institute of Technology (MIT), it is used by three different planning groups within UPS—one to plan schedules for the following four to six months, one to work out what kind of facilities and aircraft might be needed over the next two to ten years, and one to plan for the peak season between Thanksgiving and Christmas. Getting the scheduling wrong imposes a heavy cost: flying half-empty planes or leasing extra aircraft is an expensive business. UPS reckons that VOLCANO has saved the company tens of millions of dollars since its introduction in 2000.

Logistics firms are far from the only ones working on “optimisation” algorithms. Telecoms operators use algorithms to establish the quickest connections for phone calls through their networks or to retrieve web pages speedily from the internet. Manufacturers and retailers use them to fine tune their supply chains. Call centres decide where to place an incoming call, based on things such as the customer's location, the length of queues that operators have to deal with and the reason for people calling.

Jeff Gordon, who looks after innovation for Convergys, a call-centre operator, says that the efficiency of algorithms is as crucial to his industry as the quality of call agents: “If you get the algorithm wrong and put customers into the wrong hands you degrade the experience. No one likes being handed off to someone else.”

The most powerful algorithms are those that cope with continual changes (see article). The delivery schedules for online grocers have huge “feedback loops” in which the delivery times chosen by customers affect the routes that vans take, which in turn affects the choice of delivery slots made available to customers. UPS is working on a real-time algorithm for its drivers that can recalibrate the order of deliveries on the fly, in much the same way that satellite-navigation systems in cars adjust themselves if a driver chooses to ignore a suggested route.

In the world of the internet, operators are looking at ways of marrying up the algorithms that find the shortest path through a network and those that control the speed with which information flows. At the moment, the routing algorithm does not talk to the flow-control algorithm, which means paths do not change even when there is congestion. According to Marc Wennink, a researcher at Britain's BT, combining the algorithms would mean that tasks such as downloading files could become much more resilient to network disruption. It would also allow BT to make better use of its existing network capacity.

Airports also have a keen interest in dynamic algorithms. Passengers at London's Heathrow and other congested airports often sit in a long queue of planes waiting near the runway to depart. Delays happen because air-traffic controllers need to leave a safety margin between aircraft as they take off. This margin depends on the size and speed of an aircraft, and re-ordering the queue can minimise the delay before all the planes get into the air (mathematicians call this the departure problem). Air-traffic controllers have always reordered planes in the departure queue manually, but researchers are working on algorithms that would be more efficient.

Just as optimisation algorithms come in handy when people are swamped by vast numbers of permutations, so statistical algorithms help firms to grapple with complex datasets. Dunnhumby, a data-analysis firm, uses algorithms to crunch data on customer behaviour for a number of clients. Its best-known customer (and majority-owner) is Tesco, a British supermarket with a Clubcard loyalty-card scheme that generates a mind-numbing flow of data on the purchases of 13m members across 55,000 product lines. To make sense of it all, Dunnhumby's analysts cooked up an algorithm called the rolling ball.

It works by assigning attributes to each of the products on Tesco's shelves. These range from easy-to-cook to value-for-money, from adventurous to fresh. In order to give ratings for every dimension of a product, the rolling-ball algorithm starts at the extremes: ostrich burgers, say, would count as very adventurous. The algorithm then trawls through Tesco's purchasing data to see what other products (staples such as milk and bread aside) tend to wind up in the same shopping baskets as ostrich burgers do. Products that are strongly associated will score more highly on the adventurousness scale. As the associations between products become progressively weaker on one dimension, they start to get stronger on another. The ball has rolled from one attribute to another. With every product categorised and graded across every attribute, Dunnhumby is able to segment and cluster Tesco's customers based on what they buy.

Where to put the biscuits

The rolling-ball algorithm is in its fourth version. Refinements occur every year or two, to add new attributes or to tweak the maths. All these data then feed into a variety of decisions, such as the ranges to put into each store and which products should sit next to each other on the shelves. “All this sophisticated data analysis and it comes down to where you put the biscuits,” laments Martin Hayward, director of consumer strategy at Dunnhumby.

Fraud detection has a touch more glamour to it. SPSS, another data-analysis firm, uses algorithms to scrutinise customer data and to build propensity scores that predict how people will behave. One of its clients is ClearCommerce, which provides payment-processing services to online merchants. SPSS helped ClearCommerce to build a system that looks at a customer's past transactions and learns what hints at fraud—it might be the amount of money being spent, the shipping details and the time of day, and so on. Transactions then get a fraud-propensity score based on these characteristics; merchants decide which scores should ring alarm bells and how to respond.

Algorithms are most commonly associated with internet-search engines. “The tussle between MSN, Google and Yahoo! is about whose algorithm produces the best results to a query,” observes Microsoft's Mr Herbert. Ask.com, another search engine, has even tried to popularise the term in an advertising campaign. Few other types of companies are so obviously dependent on algorithms for success, but the role that they play is rising in importance for two reasons.

The first is the sheer amount of data that is now available to companies. The information floodwaters are rising everywhere. Smart meters give utility firms data on consumption patterns inside households. Digital media will make it easier for firms such as Dunnhumby to see how what people read online and watch on television affects what they buy.

Online shopping means that internet merchants now know what customers are browsing as well as buying. Search engines are mining their own information on the relationship between queries and clickthroughs so as to improve their ranking algorithms. “For the first time in business history there is more information than many organisations' capacity to deal with it,” says Dunnhumby's Mr Hayward. Algorithms are a way to cope.

The second reason why algorithms are becoming more important is that companies inevitably want to use all this new data to do more complicated things. In particular, they want to respond to each customer in a personalised way. Tesco does this by using its analysis to tailor direct-marketing offers to each Clubcard member. As well as segmenting its customers on how they live, the data also enable the supermarket rapidly to spot shifts in their consumption patterns (caused by children going to university, say). Tesco's response rates to such targeted marketing stands at 10-20%, against an industry average of only around 1%.

Convergys wants to bring more real-time data to the operation of call-centres. Mr Gordon gives the example of a customer who calls an electricity utility from an area that has suffered a power failure and, because of where they are speaking from, is automatically put through to an operator who can deal with his queries. Such algorithms help firms to tease simplicity from complexity.

Algorithms are not for everyone. Some companies will always generate more data than others, of course: retailers, utilities and telecoms firms process many more transactions than house insurers, whose deals tend to happen once a year. Some will also be more focused than others on how algorithms can shave costs or maximise capacity. Firms that enjoy high margins and strong demand are going to be less worried about the efficiency of their supply chains, says Hau Lee, of Stanford Graduate School of Business.

Rocket science for non-boffins

What is more, lots of things have to fall into place for algorithms to work. They tend to be highly complex: it is not easy to find people with the right skills to develop and refine them. The systems within which the algorithms run—the user interface—need to be intuitive to non-boffins. “This is rocket science but you don't have to be a rocket scientist to use it,” says Jack Noonan, boss of SPSS. The inputs have to be right. One UPS planning model routed all the packages in the system through Iowa, which perplexed everyone until they found an error in the data that made it appear to be free to send packages via Iowa. The algorithm was right, in other words, but the data were wrong. Mr Noonan says that SPSS's “secret sauce” lies in its ability to deal with missing or unreliable data, rather than the algorithms themselves.

Above all, human judgment still has a role—a point perhaps reinforced by the recent performance of algorithmically driven quantitative funds in the financial markets. In fraud detection, for example, algorithms can eliminate the majority of transactions that are above suspicion but a human is still best placed to analyse the dodgy ones. Dunnhumby is trying to overlay attitudinal research on top of purchasing data to understand why people buy things as well as what they buy. Even so, Autonomy's Mr Lynch is convinced that algorithms are on the march. Algorithms process data to arrive at an answer. The more data they can process the more accurate the answer. For that reason, he says, “they are bound to take over the world”.

Certain kinds of rare events are called black swans, after Nassim Taleb’s term to describe the kind of unexpected, rare events that have significant and unanticipated consequences (the Internet, September 11, and so on). But not all rare events are black swans. Some are just rare, simply tail events. In any distribution of possible outcomes, many are probable and a few are outliers. Flip a coin 400 times. Odds are that you’ll see heads roughly 200 times, give or take. Yet there is a nonzero probability that you'll flip that coin 399 times before you see the first head. It’s a very small probability, but it could happen.

I mention this by way of introduction to my recent tail event. I mentioned in my last post that I’d just made a huge amateur investing mistake and that in all likelihood I had just flushed down the toilet a pretty huge sum of money. Huge for me, anyway.

In real time, I spent almost two weeks sitting on a lot of virtually worthless stock options, trying to come to terms with the stupidity and/or incompetence of a novice in way over his head. As I said at the time, “I’d gladly have given [that money] to any number of charities or individuals, but that is not an option available to me now.” Even though I have slowly but steadily come to regard myself as an atheist during the past seven or eight years, I found myself occasionally making promises to “the air, or any kind of supernatural entity that may or may not exist, given appropriate caveats, and of course assuming that said entity, given its existence, is the kind of entity that steps in to micromanage human affairs based on its own selfish wishes” that, if things worked out such that I didn’t lose all that money, I would gladly donate it to a collection of mindfully chosen charities whose goal it is to improve the lives of people having a difficult time in the world.

And as a result of making such pacts, or more precisely, as a result of allowing for the possibility that someone might be listening to them, I found myself falling into that simplistic mindset wherein one starts implicitly viewing events in the world as responses on the part of an all-powerful and ever-active deity to the unseen, internal sequence of thoughts, motives, beliefs, and doubts, that pass through a person’s awareness during the course of a day. Knowing full well that God does not manipulate the stock market to answer the selfish wishes of a non-believer, I nonetheless caught myself on at least one occasion wondering if he might. Realizing how weak-minded this was, I tried to turn the pacts inward and vowed to myself to follow through regardless of whether some entity had been listing, to follow through simply because it would be the right thing to do. This was the First Axiom of Morality that my mother had beat into my skull from as far back as I can remember: Do the right thing, even if no one is watching. But even so, as the stock price began to creep up and I realized that these things might not end up completely worthless after all, the deal gradually weakened: from donating all of the money, to only the profit, to just the percentage of a typical tithe. I justified each of these transitions admirably, but justified them nonetheless. A supplicant will say certain things that he might conveniently forget when he’s been given shelter, a warm bath, and a hot meal.

And, in fact, were God testing the sincerity of my resolutions, he could have chosen no better way than to make the bet pay off, and pay off big. That’s exactly what happened. On one hand, it was quite fortunate for me. It’s like I fell down a hole and found a bag of buried cash with my name on it. “To Mr. Proportion, from The Fates.” On the other hand, I now have that most difficult of decisions to make. Knowing (in my renewed certainty of God’s complete lack of desire or ability to micromanage human affairs) that there will be no consequences, I could conveniently forget to give any of it away. The First Axiom of Morality pretty much prevents me from completely reneging on the deal I made, but the Theorem of Rationalization, along with the associated Lemma of Justification of What’s Convenient and Self-serving, makes it quite easy to give away much less than the standard tithing percentage and still argue that it’s better than if I’d given nothing.

Thanks to the miracle of the Internet, it all happened on a Friday I spent sitting on various airplanes, far away from Wall Street and my computer. Knowing that I’d be away from the computer and unable to obsessively track the price of this particular option throughout the day, I programmed a series of automatic trades to sell various increments at various prices if the price went up while I was away. I had bought these things for 15 cents each, so I set up the trades to sell some at 20 cents, some at 25 cents, and so on up to 40 cents. These may seem like trifling amounts, but I had accidentally bought quite a lot of them, and if you look at the prices as ratios, you see that selling them for 30 cents each would actually double my money. I thought it really unlikely that the price would go that high, but set up the trades as a precaution and expected still to own most, if not all, of these options when I returned on Monday.

So I was quite surprised, when I got to my destination and ended up somewhat randomly spending an hour at a public library waiting to be picked up, to see what I saw when I reserved 15 minutes at the public internet terminal and logged into my account. The stock price had skyrocketed that day. There was now an enormous amount of cash sitting in my account, and I no longer owned a single option. Quite the tail event.

I’m proud to say that I’ve identified four worthwhile charities this week and, again thanks to the miracle of the Intertubes, it took approximately 25 seconds to fire off donations to each of them. But my samaritanosity is starting to feel satiated, and I haven’t even reached half the standard tithing percentage yet. Did my mother fail? Or is okay to be only human? Or is it none of the above?

My mistake concerns another aspect of options that I didn’t encounter in my research. The crucial detail is that a single option is actually an option to buy a bundle of shares, usually 100. The option also costs 100 times as much to buy as the amount listed. So when I placed the order for y options costing $x each, my account was immediately charged $100x, and I was granted the option of buying 100y shares of stock. This is not intuitive behavior at all, which I understand now, in retrospect, is one of the many reasons why every page that mentions options on the online trading company’s website has a huge warning plastered at the top: “Options are not suitable for all investors, as the special risks inherent to options trading may expose investors to potentially rapid and substantial losses.” Or, in short, “Chumps best watch out.”

For me, $y was pocket change, because I knew I was making a practice transaction. But $100y was not pocket change. And there is no way to undo a trade. So I’m fairly like to lose a lot of money when this particular option expires in late October. The upside is that there’s a tiny but nonzero chance that I’ll become a multithousandaire instead. Given that this is the only basket available to me, it’s the one into which I’ve put all my eggs. If either of my readers knows anything about manipulating stock prices, drop me a note. We’ll chat.

The thing that bothers me most about this loss is not that the money will be gone. That’s painful, but the real pain comes from the giant flushing sound that indicates where it has gone, and all because I didn’t read far enough into the manual. I’d gladly have given $100y to any number of charities or individuals, but that is not an option available to me now. No pun intended.

Corinne found me banging my head against a wall (figuratively) and asked what was wrong. I admitted my gaffe, and by way of making me feel better, she told me about this Guatemalan gentleman who recently lost a lot of money. He came to the US illegally and worked for 11 years as a dishwasher, making $5.50 an hour for most of it. Over the course of 11 years, he managed to save $59,000. If you’ve ever made $5.50 an hour, you know that it’s very difficult to cover all your expenses on such a wage, let alone save over $5000 a year. From this we know that Pedro Zapeta is a very frugal man, and probably did not enjoy many of the material comforts that were available to him in his adopted home. I suspect that he worked more than eight hours a day, and he perhaps shared a small apartment with several other people not related to him, probably more than one to a bedroom. I further suspect that he did not particularly enjoy his life here, but instead got through many days by thinking of the life he would have many years hence, back in Guatemala, living off the wad of cash he was working so hard to save. 11 years. That’s a very long time to live a life in the name of diligent delayed gratification. For the average person, that’s 20 percent of an adult life.

One day, Pedro decided he’d saved enough to realize his dreams, so he packed up his $59,000 into a duffle bag and headed for the airport. He apparently didn’t pick up much English while he was in the US, and as a result didn’t understand the part of the customs paperwork that informed him he was required to declare any cash in excess of $10,000. He didn’t declare it, and when it was discovered, it was seized. At that point they realized that he was in the country illegally, so they called in the INS, who promptly deported him.

My day was a walk in the country compared to this story, but hearing it didn’t make me feel much better. In fact, I think it made me feel worse. When his case became public last year, folks who sympathized chipped in and raised $10,000 in donations for him. However, he is not allowed access to that money for a reason that is not explained in the article. For me, though, the clincher of Pedro’s story is how the federal prosecutors assigned to his case tried to buy him off. According to CNN, “Robert Gershman, one of Zapeta's attorneys, said federal prosecutors later offered his client a deal: He could take $10,000 of the original cash seized, plus $9,000 of the donations as long as he didn't talk publicly and left the country immediately.” But Pedro stuck to his guns, which explains why his case is still wending its way through the courts two years after the incident. Go Pedro!

Possibly more disturbing than the ass-headed reaction of the authorities in this case (who are, admittedly, just following the ass-headed laws), is the reaction of the ass-headed right wing (in this case, self-appointed exposers of Liberal Media Bias), who are outraged that this man admits to not paying income taxes the whole time he was in the country, that he was illegal, that he never learned English, that he is brown. Um, excuse me, right-wing. Having spent several years earning $5.50 an hour, I can tell you confidently that, come tax time, you always get 100% of your deductions back, due to convenient reality that the federal minimum wage places you safely below the poverty line.

Nonetheless, the bloggers (who are far more kind than the commenters, as you’ll see if you dare read down to the kind of hateful, xenophobic comments posted by the kind of people who read right-wing blogs) have things like this to say about Pedro:

It is an insult to all law-abiding Americans to frame Zapeta's “mistake” as being his failure to successfully flee the country with his ill-gotten, untaxed cash. Zapeta’s “mistake” was entering this country illegally in the first place. Zapeta compounded his error by failing to pay income taxes—which over 11 years would likely comprise a significant portion of the $59,000. Wouldn’t we all like to keep 100% of our earnings for the next 11 years?    

Now I’m definitely more sad than I was. I can handle losing some cash. I can even handle the ass-headed way in which I lost it. But I am becoming less and less able to handle the fact that public discourse in my country, whose founding rhetoric is among the most beautiful and inspiring rhetoric ever crafted, is ever more dominated by small-minded, hate-filled regurgitators of bullshit rhetoric that would have made certain hate-filled fascists of the twentieth century proud. Like this commenter:

Had you came legally and attempted to learn english you wouldn't have this problem. You are a crimminal, be happy that we are only keeping the money and sendng you home, In my world we would keep the money, sentence you to hard labor, force you to pay back taxes and any tax payer paid services you scammed. We woyld also shave your head and brand you with a huge "I" in the middle of your forehead. We would allow you to choose what the "I" stood for though, illegal or idiot, your choice.

—former Arkansas Governor Mike Huckabee

He was speaking at a GOP debate at historically black Morgan State University in Maryland, commenting on the fact that McCain, Romney, Giuliani, and Thompson decided not to attend, citing “scheduling conflicts.”

Full article here. I don’t feel like I have to add a thing.