'Sunk costs' and the war

University of San Francisco economics professor Bruce Wydick has an exceptionally clear op-ed piece in the 15 August 2007 issue of USA Today regarding America's involvement in the Iraq war.

It's Wikilicious!

The hype du jour is that a clever analysis tool for wikipedia entries and interesting/entertaining sources that edit/massacre/delete entries that pertain to themselves now exists.

* (I don't recall where I read this first today but to keep things on the up-and-up it was either Slashdot or Wired through my feed reader).

It seems that one can data-mine *.wikipedia.org via online search or download of a snapshot image with all previous edits, a la a useful audit trail. OK, that doesn't seem too interesting. Until...one looks at the sources of the edits and the effects and affects of these edits.

This is a perfect example, without drawing attention to myself, of the difference between data and information and the relation between the two. Data is available...it's up to individuals or processes to transform, and hopefully protect, it into information.

The New Age of Ignorance

As a follow-up to my last post concerning the systemic and intentional attempts to dissuade students from studying mathematics, I thought I would just offer a pointer to this, by way of the Edge, article from The Guardian:

The New Age of Ignorance

We take our young children to science museums, then as they get older we stop. In spite of threats like global warming and avian flu, most adults have very little understanding of how the world works.

Mathematics: Seriously, Just Say "No!"

In a previous post, I commented on a trend in the UK where students were being discouraged from taking "unnecessary," higher level mathematics classes. By way of Slashdot, it seems that is is an issue in Australia as well. The poster states and asks:

"The claim is that Australian schools are actively discouraging students from taking upper level math courses to boost their academic results on school league tables. How widespread is this phenomenon? Are schools taking similar measures in the US and Canada?"

I ask again: Will science education in the classroom be dead by the time my kids go to school?

Data v. Information

I had fun with a previous post regarding a data visualization tool called many eyes, from IBM alphaWorks Services. There are some nice graphing templates available but pretty graphs simply do not the wonderful experience make. OpenOffice CALC and Microsoft Excel can produce a multitude of graphs in a variety of canned formats but do they really assist in helping one understand the data being presented to them.

Are they capable though, as tools, to transform data into information? The distinction may or may not be a subtle but the implications are huge. We're generally over-run with data and consider so much of it to be throw-away. Information, however - information being data with some sort of context applied to it - one holds onto as long as possible because the context applied to the data, the transform or function applied to some data set, increases the data's value and elevates it to that of information.

Consider a couple of simple examples:

What does this string of data mean, if anything: 011903124555555

1. Well, it could be a random string of 16 digits and not very interesting (highly likely).
2. Out-of-country phone dialing number? (yes, US Embassy in Turkey)
3. Credit card number? (same format for Visa/MasterCard but not a valid number)
4. USPS/FedEx/UPS/DHL tracking number? (UPS if you drop their "1Z" prefix)
   
5. US social security number? (Massachusetts SSN with some cruft appended to the end).
6. Product SKU (I seem to recall that there are standardized SKU formats)
   

We just don't know, without any context applied to it. Now, what if we thought about another string of digits in the context of identity theft:

• 034011234,Last,First,Acct#

Huh...that looks important and maybe should be protected. Maybe it's a person with an account # and MA SSN on-record. The problem though, is that if the suspect data were changed to be:

• Acct#,034011234,Last,First

It could become meaningless because the transformation changed through simple re-ordering of data elements and the context may no longer be identifiable therefore leaving the data as data. There's a good chance, however, in this specific case that the context could be inferred. What happens if we eliminate the comma delimiters and just spew a line of text in the hope that it will be properly caught and processed?

• Acct#034011234LastFirst

Here we have an example where Acct# and SSN have been concatenated and probably lose meaning outside of the process that knows to stop reading the Acct# field after X characters and read the next nine characters as the SSN. First and last names can be extremely difficult to distinguish without capitalization and/or localized knowledge of standard names. Michael Smith may mean nothing to a non-English speaker.

So what does this mean from a practical point of view? Without waxing philosophical, from an information security and protection standpoint, it is an extremely compelling reason to give serious consideration to Translucent Databases, which I will post about at a future point in time.

Finally...A Use For Twitter

Until recently, I've found Twitter to be the single most annoying and useless fad since NeoPets and Tamagotchi toys. I still have absolutely no interest in, nor do I understand the fascination with virtual pets. I may be coming around to Twitter however.


The Lifehacker folks are atwitter for Twitter and using it as a productivity-boosting tool as opposed to a sort of micro-blogging platform:

Twitter is for staying in touch and keeping up with friends no matter where you are or what you’re doing. For some friends you might want instant mobile updates—for others, you can just check the web. Invite your friends to Twitter and decide how connected you want you to be.

I personally don't have much interest in what my friends or family are doing every second of the day or where they are at any arbitrary time when I am not in direct contact with them. Maybe I'm just stubborn and still prefer email, a telephone call, maybe some Blackberry love, or - heaven forbid - face-to-face interactions.

However, following two previous posts concerning access to interesting data sets and an interesting data visualization tool, I noticed My Mile Marker.

Being a bit of a data wonk, I immediately jumped onto the Twitter bandwagon so I could send SMS messages to M3 from the gas pump and fulfill my need for immediate gratification (which may seem like a strange thing when you're generating and collecting data for future analysis). M3 account creation was easier than easy.

M3 has a nice interface and provides some tracking and reporting tools.

I like this idea because I usually compute my fuel consumption when I fill my gas tank but it's nothing I track beyond each fill. I notice trends such as getting 50 fewer miles range per tank or averaging 2 MPG more than usual. I like the idea of viewing fuel consumption trends and identifying when, for example, I added air to the car's tires, changed the engine oil or the gas station began adding more ETOH as it gets colder in New England.

We'll see how this goes.

On Virtualization

Now, I don't know how far it has been considered in academia or industry but, excuse me if I am naive or simply unaware, virtualization, specifically paravirtualization, could be immensely useful to systems requiring a 1) minimal OS footprint with the ability to swap images in near-real-time, 2) an "online OS" or 3) an OS running on a virtual machine.

1. Set-top boxes such as (my favorite) the Moxi box or a TiVo could run multiple virtualized guest images, for the sake of a manufacturer upgrading feature sets or remediating flaws, over a consistent, hardened host image that only provides a bootstrapping facility the guest images. Through paravirtualization, presumably, hardware sharing and optimization functions, such as video capture and display/replay, could also be virtualized and provide for a transparent user experience as the various operating images are propagated to customers (presumably over multicast...I mean cable pay-per-view). Additionally, it could also offer rapid roll-back in the event of an image push and a call center flooded with complaints about set-top boxes that no longer work.


2. Ah, the "online OS." Everything Google does has an entire industry wondering if they are preparing for a browser-based, online OS. Browser-based with bootstrapping through a browser's existing JVM (or something similar)? Others are obviously thinking about this more topically than myself:
   

   
   "a bunch of startups like YouOS , Goowy, DesktopTwo, Xin and open source eyeOS are already tackling this exact problem"

   
   Maybe In a way, Sun is already doing it with their Sun Secure Global Desktop Software (wicked demo, BTW). Sun licensed Microsoft's Remote Desktop Protocol (RDP) technology in order to do some of this and the result is as smooth as a MS Terminal Services/RDP session but with dedicated, thin client hardware and a java-based smart card that allows a user to disconnect and later reconnect, at an arbitrary location, and have their last session in-place.


3. In a way, an arbitrary OS running on a virtual machine is sort of cool but mostly, "this is what virtualization is all about, slow to the table commentator." The Sun solution mentioned above can be considered in this light. Microsoft Terminal Services can use a thick client or browser-based end-user environment and still provide seamless connectivity.

So, I'm probably well behind the curve on this one as far as experimenting with the technologies but at least, to the uninitiated, there seem to be some useful implementation possibilities.

So You Say It's Your Birthday?

Today I am at the age where I:

• may not experience another birthday defined by the product of twin primes
• will not experience another birthday defined by the product of twin Mersenne primes
• will hopefully experience another birthday defined by the product of cousin primes
• probably will not experience the first three digit palindromic prime
• may not experience another birthday defined by the sum of n^th and n^th+1 primes

I'm not sure this trivia makes me any happier :(

Remember OpenDoc?

I stumbled on this hysterical spoof of the "I'm a Mac and I'm a PC" ads a while ago and nearly wet myself laughing. Almost everyone has an obligatory Newton reference when recalling great Apple ideas that were a bit too far ahead of their times. I'll skip the Newton reference and just start in with OpenDoc.

First off, Apple even seems a wee bit bitter when referencing their ideas and tech that never went anywhere. This is the best you can get from them on OpenDoc:

Reference Library

Legacy Technologies

Legacy documents consist of guides, references, sample code, and other resources that have become irrelevant for current product development. Some of these documents describe features, functions, classes, or methods that are no longer supported. Other legacy documents describe currently supported technologies, but the approaches and development techniques described in these documents are no longer recommended.

. . .

OpenDoc is a cross-platform technology that replaces conventional applications with user-assembled groups of software components. OpenDoc allows users to create virtually any kind of custom software solution. OpenDoc is not supported in Carbon.

OpenDoc met its end in a stand-off with Java that never really was a stand-off; they were complementary in the sense that you could develop OpenDoc objects in Java or what every language you preferred (Think C/C++ anyone?). Apple was having a bad day (OK, a series of years) financially, Java was the sweetheart of the internet du jour (second only to Helen) and Microsoft was first to market with OLE (object linking and embedding).

The true beauty of OpenDoc, IMO, was the ability to embed arbitrary objects into arbitrary "containers" as well as extending the container objects. Abstraction, encapsulation, inheritance, and polymorphism; traditional object-oriented topics, implemented well ahead of their time, as far as data formats are concerned.

The Palm Pilot (and its derivatives) was to the Newton as XML (and its extenstions) was to OpenDoc.

Microsoft Goes All Wishy-Washy?

Microsoft has historically been considered a rough-and-tumble company who chews up and spits out others firms like a lumberjack with a mouth full of tobacco. Three fairly recent instances however have me a bit confused about their image.

1. Microsoft claims Linux infringes on hundreds of its patents. The FOSS community goes bonkers. Microsoft refuses to state which patents have been infringed upon. Still no group hug to make up.
   
   So...Microsoft is emulating SCO but with a substantially larger legal war chest?


2. Via BetaNews, "In a policy document specifically timed for release this afternoon, Microsoft's general managers for interoperability, Tom Robertson and Jean Paoli, make a play for ownership of the standards issue facing users of competing document formats, by saying the company would support ratification of its own Open XML format along with OpenDocument Format (ODF) as ISO standards, if and only if doing so would promote choice among the world's consumers."

No comment here. I'm still miffed over Massachusetts gutting its CIO's power and appointing Brian Burke, a MS anti-ODF lobbyist to the Governor's Technology Advisory Group.


3. "Silverlight is a cross-browser, cross-platform plug-in for delivering the next generation of media experiences and rich interactive applications (RIAs) for the Web. The Silverlight 1.0 Beta has a go-live license that implies it can be used for commercial purposes."
   
   It implies it can be used for commercial purposes??? I was happy, however, to see the following on the installation page:
   
   "For Mozilla Firefox users:
   Save Silverlight.1.0beta.exe to your hard disk. Once the download is finished, click Open. The installation starts."

Fun With Data Visualization

My last post noted Data360.org and the impressive number of data sets they have available. I just stumbled on a new data visualization tool from IBM called many eyes so....

I grabbed an oil reserve data set from the US Energy Information Administration, by way of Data360.org, massaged it a bit in OpenOffice and fed it into many eyes. Click, click, click and you're provided with a pretty slick visualization.

Go. Go and play.

Fun Site for Data Wonks

Loads of fun. It's like Wikipedia for data and analysis.

Is 740 the new 520?

Recently, sub-prime mortgage lending has made the news amid fears that there may be as many as 2.2 million new forclosures in the US in the coming year. In the Boston Globe a while back, Thomas Callahan summarize:

"NOW THAT everyone is paying attention, let's review how we got here: Subprime loans became the mortgage flavor of the day several years ago. The worst kind of lenders preyed on the universal yearning for the American Dream -- homeownership -- and too often targeted African-Americans and Latinos with alluring, affordable, and ultimately deceptive sales pitches.

. . .

These are not just run-of-the-mill high cost loans. Subprime lenders keep coming up with new products and new gimmicks. That is why there has been a proliferation of 'stated income' loans and interest-only loans. These products have been around awhile but they had a limited use until brokers and mortgage companies started selling them to average consumers who didn't have a realistic chance of paying the mortgage."

In a way, sub-prime lending in the mortgage market is a bit like...like what? Let's construct a shady example that unfortunately, I've been told happens more often than it should.

A mortgage broker woos potential borrowers with wonderful terms and immediately flips the loan after closing it. OK, perhaps the broker over-sold to an under-qualified borrower but that's not their problem any longer. Nor is it their problem really when they out and out misrepresent a potential borrower. The result is a lender that now has an at greater risk borrower that it would have accepted otherwise. What to do? Flip the loan. This cycle can continue, based on other criteria as well, until the loan is paid-off or the borrower defaults. Musical chairs or hot potato with a large loan.

Excluding the boring case where the borrower pays the loan off in-full, without incident, the end result is a loan in default because a borrower was misrepresented to a lender. Most people still need to borrow in order to finance large purchases, such as houses. Sub-prime mortgage lending will continue to persist in some manner because of this; people need houses. Less than ideal borrowers will need to, in some way, boost their all-important credit scores since, no matter what anyone says, you are just a score.

There are positive financial incentives to every party involved except the unfortunate lender holding the note when the risk-deflated borrower defaults. Not knowing when or if that event will occur is the risk. If there was no risk involved, there would be no sub-prime lending markets; all borrowers would be identical. Since this is not the case, borrowers need to be differentiated in some manner and this will continue to be the credit score, for better or for worse.

The potential problem at hand then is the renormalization of misrepresented credit scores and how that could affect better borrowers and risk-adverse lenders. Borrowers that previously would not qualify for certain lending options would have their scores inflated in order to qualify. Typical borrowers would remain the same or possibly be a bit inflated as well. In other words, the typical ranges of credit scores would begin to contract while the upper bound would remain the same.

520 would be scaled to 720.

Recapitulation Theory and Second Life

Ontogeny recapitulates phylogeny

Internet Timeline: everything pre-Mosaic can be considered to be proto-web...extremely necessary from an evolutionary perspective but just not that sexy (like a caveman in a Speedo).

Second Life Timeline: in the beginning there was Real Life and it was fine but we got bored and started creating MUDs and MUSHes and the like but our kids thought we were weird until we installed SimCity for them so they would stay off our backs but it backfired and we ourselves became obsessed with Godzilla laying waste to our hard work for a few years until there was no reason to limit ourselves to simulating a city or life or an ant colony, no, we could simulate ourselves and everything we hoped and aspired to be and those around us in The Sims which taught us all how comfortable we could feel in drag or engaged in virtual infidelity or strafing a city block while trying to steal the helicopter...wait, wrong game, back up...all of this kept us satiated until we discovered (drum roll followed by hush)...Second Life.

Breathe...deeply...slowly...repeat....

So we have First Life (to those of us who think that art imitates life) and Second Life (to those of us who think that life imitates art). The internet as we know it slowly evolved until it reached a tipping point, i.e. the creation and release of the Mosaic web browser. After that, to many, it all went spiraling down the toilet, what with the massively increasing onslaught of spam, porn, commercialization of something that was pure, etc, etc, etc.

I don't completely agree nor disagree with the idea that the Internet has devolved into something crass and no longer meaningful. What I find terribly interesting, however, is the way in which SL is to a large extent, mimicking the evolution of the Internet as a whole.

Porn, rampant capitalism, porn, spam glorious spam, porn, activism and/or terrorism, porn....

So, is SL ontogeny to the Internet's phylogeny?

Why Is Gas So Freakin' Expensive?

Excellent essay describing the various forces at play regarding gasoline pricing (by way of the Freakonomics Blog).

Google Trends - Part 2 - American Idol

OK, I've been tangentially aware of American Idol for the past few years and have seen it a few times so I'm not interested in getting on a soapbox and critique "reality TV" or offer commentary on the competitors. What I am interested in, however, is how the recent rise in popularity in Google's Trends tracking, could act as a predictive analysis tool, aimed at larger...things....

So...American Idol has already aired (presumably concluded) for the night here on the east coast of the US so I'll fill-in a bit of background for the uninitiated (like myself, really): after some recurring period of group talent contests, two contestants are left in competition, after some number have been voted off of the show by popular opinion, one week at a time. The winner of the contest will be decided by popular opinion as well.

And...the winner is...huh, nothing posted to the official American Idol site. I guess I'll post more about Google Trends and predictive analysis as soon as I have some sort of official results to go with for verification.

Shame...I was getting interested but now tired as the night progresses. Sigh....

---

Update!

OK, Blake Lewis and Jordin Sparks were the final two contestants on the show and Ms. Sparks won by popular vote. What happens if we plug both of their names into Google Trends? Well, currently, the results are disappointing as Google Trends is still a beta/lab product and there isn't any current, searchable trend data for yesterday. Let's see if there is anything to be gleaned from the 100 hottest searches for 23 May 2007.

OK, a search for "jordin sparks" was the 12th most popular search on the day of the show's season finale with search activity peaking around 2100 EDT. "blake lewis" is nowhere to be found in the top 100. Does this mean anything as far as being a potentially useful prediction tool? Is it more or less significant that guest performers on the show - Smokey Robinson (#1), Gladys Knight (#5), Tony Bennett (#6), Bette Midler (#7) and Joe Perry (#10) - were more popular searches, between 2000-2100 EDT, than Jordin Sparks?

Perhaps Ms. Sparks became the clear popular favorite during the day. Let's see what the previous day had for searches related to Ms. Sparks and Mr. Lewis.

Nothing.

The closest search was for "american idol winner" (#9 on 22 May and #2 on 23 May). So, maybe search trends aren't terribly useful for predictive analysis. Maybe there isn't enough data available at the moment. Let's see who the finalists were last season and plug them into Google Trends...

Taylor Hicks won the season 5 contest and Katherine McPhee was runner-up on the 24 May 2006 season finale. The Google Trends results appear to clearly support Mr. Hicks the days leading into the competition:

"taylor hicks" "katherine mcphee"

Let's look at season 4 for the fun of it. Carrie Underwood won the season 4 contest and Bo Bice was runner-up on the 25 May 2005 season finale. Shaking the Google Magic 8 Ball shows...nothing useful:

"carrie underwood" "bo bice"

They're tied until the next day when everyone is trying to find out more about the winner. Having access to the vote tallies could describe this if the popular count goes nearly 50%-50% for the contestants.

Might as well look at the 2004 season since Google has the data to play with. On the 26 May 2005 season finale, Fantasia Barrino won over Diana DeGarmo. Reading the Google tea leaves shows...more nothing.

"fantasia barrino" "diana degarmo"

If anything, Ms. DeGarmo may have had a slight popular lead entering the vote.

Have we learned anything? No, probably not; at least not from a predictive analysis point of view. If I think of it in the coming months I'll come back and see if Google has search/trend data available to cover this most recent season.

Google Trends - Part 1

Huh...loads of news regarding Google Trends' new functionality: regularly updated info listing the most popular search trends/terms. In one manner, this is obvious - Google has the search data available to them at any given point in time through, presumably, numerous search entry points or via a search funnel. It also makes perfect sense that this functionality, the ability to see what your peers or organization or group is searching for, would be pushed down to the Google Search Appliance at some point.

That's really the interesting addition in my opinion: by localizing search trends, through the GSA or adding a new search primitive, such as fromDomain:, fromIpAddress: or fromNetBlock: (in the same way that the site: or url: search primitives are available now), search/trend consumers would have the ability, from a social networking perspective, to learn what is being searched for, relative to themselves in their own localized search bubble.

Let's consider an example:

Say you're a faculty member at a university and you're vaguely aware of a new collective bargaining vote that's upcoming and you want to know more. You search Google and find press releases and maybe a few local area newspaper items that have been indexed. This is the way in which we normally search or engage in absolute searches, modulo intentional deletions or censoring of search results, etc. Say you're a Google Master and break-out your Google-fu and attempt a more granular search through the selective use of one or two Google search primitives, for example, site:GoogleU.edu. Great! You weren't interested in what was being reported through channels on the other side of the country anyway.

Is this enough? Are the results sufficient in order for a search consumer to best utilize the data provided? Maybe, but it could more than likely be better by taking into account the context of the search results.

Now, say that advanced, localized trending were available to the faculty member. S/he would be able to search, relative to the university community or a specific department or a specific demographic, and determine what his/her peers are searching for, in the hope of learning about key issues being addressed at the collective bargaining vote. Add in the capabilities for search trends and results to be presented in such a way that they can be ordered or grouped and the faculty member may find him/herself better in the know and ready to cast an educated vote.

So...search, trends and localization step up as the latest social networking platform.

I wonder what I spend most of my time searching for and if anyone else cares?

BSO - Boston Pops!

Last night was the opening night of the latest Film Night series with John Williams returning to the Pops to conduct.

"This year’s 'Film Night' concerts also feature the iconic music of musicals that have become movies, tunes that resonate on stage as well as the silver screen. The program will be performed at 8 p.m., May 22-26, at Symphony Hall. The program opens with a tribute to Academy Award-winning composer Bernard Herrmann and his music from such legendary films as Citizen Kane, Vertigo, and North by Northwest. Mr. Williams also leads 'America’s Orchestra' in the timeless music that has made the leap from Broadway to Hollywood, including 'All That Jazz' from Chicago and highlights from Fiddler on the Roof with Boston Pops concertmaster Tamara Smirnova.

Mr. Williams, a favorite at Symphony Hall for more than a quarter-century, closes the Boston Pops’ 'Film Night' programs leading his own music from such blockbusters as Superman and the Harry Potter series. In a special Pops tribute to George Lucas and Steven Spielberg, Mr. Williams also conducts music from Jaws, Star Wars, Raiders of the Lost Ark, and E.T., accompanied by film clips from these memorable movies."

It was a wonderful evening with wonderful music and a wonderful meal at Brasserie JO in the Boston Colonnade Hotel.

Where Does Your Paycheck Go? - Update

I previously blogged about the allocation of my personal income as well as the US government's revenues. The Simple Dollar explores the same idea in a 20 May 2007 post, Figuring Out Exactly How Much Your Time Is Worth.

They started the year with an interesting series: 31 Days To Fix Your Finances. The Simple Dollar comes to us by way of the always useful and entertaining Lifehacker.

Language Skills

A friend lamented yesterday that he didn't know a language other than his native tongue and that he had never really left his home country. Up until a few years ago, I was in the same boat. While I can't do much for sending someone to foreign locales, here are a few options for acquiring new language skills:

• Foreign Language Lesson Podcast Collection at Open Culture
• BBC Languages
• BBC's Muzzy language courses for kids
  
• Foreign Languages and Literatures through MIT's OpenCourseWare
• Rosetta Stone language software
• Living Language audio and text collections

If anyone else has suggestions, I'd love to know them.