Monday, April 11, 2011

Addicted to the Internet

According to a new global study by the International Center for Media and Public Affairs using IBM's Many Eyes software for data visualization children are increasingly developing symbiotic relationships with their technology.

Throughout the study technology was repeatedly referred to in metaphorical terms comparing it to an appendage or a drug.  Many young people spoke of acute withdrawal symptoms.

As you can see below young people from Chile, China, Mexico, Lebanon, Slovakia, the U.S. and Great Britain spend 3.5 to 5.5 hours a day on the Internet at least half of which is spent wandering around social networks.



Somewhat surprising is the lack of smartphone penetration in quite unexpected markets.  Notice that both Lebanon and Mexico have wider penetration than the United States.



Students were encouraged to abandon their devices for 24 hours.  Although some students reported benefits of their unplugging experiment the majority felt lost, irritable and bored without their electronic appendages.



This is perhaps attributable to the near constant presence of technologies in their lives as seen in this final graph detailing their first age when using key technologies such as the smartphone, the PC and the internet.

Sunday, March 27, 2011

How Much Money is Rebecca Black's Worst Song Ever Earning?

Currently clocking in at 58 million views on YouTube, many are asking themselves if in aiding and abetting the viral spread of what nearly universal consensus is calling the 'Worst Song Ever, they may have inadvertently enabled the flow of money to its makers.

The bad news is the answer is yes.  The good news is, well, not that much.

The video first entered the social fray in mid-February.  Around the second weekend of March, Tosh.0, The Daily What and scores of Twitter users propelled it into notoriety.

More recently, in one of the worst misreportings since 75% of American news sources claimed Mubarak was the richest man in the world a few weeks back, Forbes Chris Barth asserted that Black was at least a millionaire, if not on the way to becoming one multiple times over.

PC Magazine and a score of a media outlets were quick to say, "Not so fast."  Barth himself even issued a mea culpa, claiming that in his hectic blogger life he had mistakenly sourced information from huge Bloomberg and Reuters competitor for economic news, Good Morning America who had confused YouTube views with iTunes downloads.

Let's break down the two biggest revenue streams:  iTunes downloads and YouTube views.  Forbes' Barth is claiming that since it is assumed YouTube makes $1 for every 1000 page views and content creators like Black and Ray William Johnson take approximately 68% of the profit, Friday should have raked in $20,000 around the 30 million view mark.

Unfortunately, he seems to confirm this figure with public disclosures from viral-video-making-rock-group OKGO's lead singer Damien Kulash.  Of course, Barth fails to take into account the fact that OKGO's This Too Shall Pass, a video made famous for its promethean mechanical-dominos game received 128800 likes and a mere 1512 dislikes (which is really a limit approaching zero in an anonymized world filled with contrarian-haters like YouTube or virtually any message board out there).






On the other hand, Black's Friday has received an awe-inspiring, record-breaking 969,304 dislikes over a 115,834 likes (probably composed strictly of the aforementioned contrarians and 'it's so bad it's good'-people).   Although I doubt that many YouTube viewers understand the Google company's business model perfectly, I have enough faith in humanity to believe that most people know not to click on an ad from a video that they are watching out of a derisive, condescending wish to belittle a horrible song/singer/production company.

iTunes figures aren't so hard to pin down though.  PC Magazine reports that with "Friday" sales as of last Friday were roughly 37,000 and that Ark Music/Rebecca Black are taking in about 70 cents per download, or 25,900 dollars.

The two wildcards in the equation are the split made between the Black family and Ark Music and whatever she's getting for media appearances.  Although I'm not clear on whether Black won an audition or her family merely sought out Ark Music, Black's mom shelled out $2000 to record this song and retain partial rights after being given that choice or that her daughter could record the song free with no rights.  Although after the fact it seems like a good decision, I still believe her mom is to be belittled for it, as I don't think people should be recognized for profiting off of, excuse the probability pun, Black Swan statistical outliers.  Also, Black recently appeared on the Jay Leno show, which ought to have netted her something.

So the bottom line...about $63,000, most of which I assume will be going to Ark Music who retains rights over the song and produced the video (without which I doubt any of us would know this song).  Luckily, although the singer of the worst song ever and a rather annoying personification of the idealized overprivileged California teen, Black knows how to keep her musical karma clean, and in return for exposing us all to, on all dimensions -production, lyrics, performance, etc - the most heinous song in history, she's decided to donate the proceeds to relief in Japan.

I am happy to report that in the true American spirit of innovation, appropriation and optimism, nothing is beyond saving.

YouTube account HeyMikeBauer has made an especially entertaining version of Friday, recorded as a pre-electric Bob Dylan and amassed about 1.8 million views.  Why this video has not been monetized is a true mystery to me.




Conan and Team CoCo attacked claiming copyright infringement on their earlier hit: Thursday.





Slightly funny is the parody made by these two superaffluent suburban kids that's clocked 2.25 million views, although I ultimately find it wanting.  If they had been two low-income innercity kids I feel like I could have gotten behind it, but when two eleven year olds have a music studio in their gigantic housemansion, high quality video equipment and apparently an iPhone 4, I really expect perfection. No clicks for you kiddies.



Finally, I find this dubstep version INCREDIBLE:



Saturday, March 26, 2011

iRobot in Japan

Japan, long home of bipedal dancing robots, has caved to American robot imports to handle their nuclear issues.  For years, people have criticized Boston-based iRobot's products as unromantic, dull and unimaginative.  Yet in times of crisis, Honda's Asimo, Toyota's Running/Violin-Playing Robot and Sony's robodog Aibo haven't shown to be of much utility.  On the other hand, iRobot's Packbot and Warrior have.



In age where everyone moans about American deindustrialization and pins all hopes of a comeback on a revaluation of the Yuan, Colin Angle and Co at chez iRobot can be seen as a tremendous example for the rest of us.  After all, in a world where you can export robots to Japan, I wouldn't be surprised if you could export snow to Russia.

It was Japan's Self Defense Forces that asked iRobot to aid in controlling the situation at Fukushima Dai-1, the nuclear plant left unstable in the wake of last week's earthquake/tsunami.  iRobot has sent their iRobot 510 Packbot and Warrior 710, until recently most popular for their work in Iraq (and a knockoff of one of their products' cameo in "the Hurt Locker") neutralizing bomb threats.

In addition to supplying them, iRobot will be providing service and training for the robots, an increasingly important revenue stream for the company reiterated by CEO Colin Angle in the last few Earnings Calls for the company.

In a fun side note, iRobot's devices are plug and play which means that they can be remote controlled by Xbox and Playstation controllers.  Halo anybody?




Sunday, March 20, 2011

The Monopoly Power of the Motorola Xoom


With the Motorola Xoom hitting the shelves soon in a Wi-Fi incarnation at $599, I think it's the perfect time to step back and criticize some of these critics who have been going nuts over the initial 3G model's $799 price point.  
According to just about everybody, the model was priced far too ambitiously and didn't stand a chance in a head to head battle with the iPad.  Nobody implied that the Xoom was too expensive given its features (as it cost a mere 80 dollars more than the nearest comparable iPad) but rather that without a $499 entry model it wouldn't stand a chance of competing with the iPad.
I posit an entirely different view on this matter.  I begin by assuming that the people who purchase a Xoom do not consider an iPad a viable substitution. We're talking IT nerds, computer programmers, gadget freaks, open source acolytes, etc. Therefor, although surely the Xoom is part of the larger "Tablet Market", it is not truly in open competition with the iPad.  
It is, on the other hand, in competition with the other Android tablets.  And those are.....
Exactly.  There are no other Android tablets.  The Samsung Galaxy Tab is a cute little machine, but it's running an operating system designed for a phone and not the tablet-optimized Honeycomb.  The Dell Streak may have counted as a tablet when it was first dropped in mid-2010, but only in lieu of the fact that, besides the iPad, it was the only other Slate-style tablet from a major manufacturer (sorry Archos!) on the market.  Today, with two iPads, the Xoom and the Tab making waves, and the huge HTC Evos and Samsung Galaxy S phone models en vogue, I'm loathe to call the 5'' device a tablet and more a "gigantic waffle phone".  
So coming back to markets, anyone who ever took a little Microeconomics knows that it behooves producers to price discriminate (charge different groups of consumers different prices for the same product).  Let's think about this for a moment in the context of the Xoom.  You've got the iPad 2 to contend with (and the marked down to less than 50% of  the Xoom price iPad 1).  Should you really try to undercut a company like Apple?  Apple is making money off of iTunes, off of the AppStore, off of in-App subscriptions.  Additionally due to economies of scale, monopsony issues and the First-Mover Advantage, Apple is making tablets at less cost than other manufacturers.  Is it realistic to try to undercut a company that can make devices cheaper than you can and additionally derives profit from more than just the sale of said machine?
I side with Motorola on this one.  All those oddball Android fanboys who need a tablet need a tablet PRONTO and they don't want anything from Apple's closed garden.  A lot of these guys have decent incomes, and what's more, they dedicate inordinate fractions of said income to the purchase of gadgets.  Why not then, knowing that they're boxed into the Android Honeycomb market, of which you, Motorola, are the only supplier, charge a ridiculous price for your tablet?  These guys are going to pay it anyways.  You can maximize your Producer's Surplus by literally stealing away all of these early adapter's Consumer Surplus in a way that pop-hit Apple could only dream of.  

Then, on that faraway day that Samsung drops a Galaxy Tab 10'', HTC comes through with a 10'' Evo or whatever they are currently cooking up, and Archos throws their hat into the 'legit-tablet market', Motorola can lower price to $600 or $500 or whatever the equilibrium price turns out being.  In the meantime though, why not suck as much money out of early-Android-adapters as you can?
In sum I generalize to this rule: There is a tablet market, but its participants do not cross clearly delineating borders in the market.  I, myself, have considered buying both the Xoom and the iPad 2 but I am an outlier.  I know Archos and Xoom owners ecstatic for the next Galaxy tablet, and I know people fretting over whether to buy the iPad2 or a refurb'ed iPad 1 for half the price, but outside of myself I know no one in the interesection of these two groups. 
I applaud Motorola for understanding this and pricing accordingly.  

Monday, March 14, 2011

Quake to Push Up Component Prices

The Wall Street Journal and Engadget are reporting that the Japanese quake will be pushing up component costs.  As of 9:56 AM Sandisk (SNDK) is up 3.15 %.  Good news for tech component investors, bad news for tech buyers.

Hopefully your capital gains on Sandisk will be enough to cover your exorbitant tablet, smartphone and PC refresh costs later this year.

Watch out for movements in Apple (AAPL) who has leveraged their 50 billion cash stockpile to snatch up components in advance and act as a virtual monopsony in the tablet component market thus far.

Engadget on Component Crisis

Wall Street Journal on Component Crisis


Saturday, March 12, 2011

Joshua Topolsky Leaves Engadget

Apparently Paul Miller was not an isolated experience.  I am sad to now know that I shall never attend an Engadget show here in New York.  Joshua Topolsky, aka Editor in Chief, is now out of the AOL box too.  Is Nilay next?  Hopefully they shall be coming back in a Huff-free incarnation pronto.

Engadget Loses Its CPU

*CORRECTION: He's apparently still hosting the show.

*JUST-IN: Topolsky is from Pittsburgh, Pennsylvania.

A look at some of the SuperSuperHappy 1990s Pop Anthems and the Nasdaq Tech Bubble. Big Boom and Bubble and Bubblicious Major Chord Insanely Happy Problemless People Music


Monday, March 7, 2011

The Worst Tech Blogger Alive/The Worst Wall Street Blogger Alive...at the same time!??!?!?!?

I don't know how many of you have experienced the same abject disappointment I have going through this, but I'm going to go off about it anyways.  MarketWatch, a blog of a blog of a blog of the Wall Street Journal features this guy named Cody Willard with The Cody Word (I'm not even going to sink to linking it).  He is not only the worst tech blogger alive (sorry MG Siegler) yet also possibly the worst financial blogger in history....simultaneously.

I don't doubt that perhaps he is a good investor as he seems totally immersed in mobile (and who among us who has owned AAPL, RVBD, ARMH etc could doubt the power of mobile?).  What scratches my back the wrong way are his thoroughly ridiculous blogs which amount to nothing more than hard-core used-car-salesman-type pitches to buy his "Investment Report" on the state of mobile internetzzzz.

It's always something along the lines of, "We are REVOLUTION investors.  Do you want to make a bajillion dollars off of MOBILE INTERNET?  I made a load of money before and surely past performance is indicative of my kickass investments.  I know mobile and will make you a millionaire...."

You think I'm exaggerating?

Click Finding the Next Apple or Google Before Wall Street Does Only to Find Out it's a Blog About the Greatness of Cody Willard

Or maybe try this horrible of horribles: I Explain to You Profit-Taking by Giving You an Allegory About My Teenage BMX Bike Adventures, No Seriously I Do

Okay, we all know the internet is filled with this kind of rollicking 'prose', but what I dislike is that the majority of the times I get linked to his page, it's because the link read the regularly reputable and personal favorite of mine, the Wall Street Journal.  This is pure carrot-stick swindling and my clicks will no longer stand for it.

I don't want you to dangle Walter Mossberg in front of my face and give me this drivel.  Rupert Murdoch, you've got some splaining to do!!!!

Tech ETFs For the Guy Smart Enough to Admit He Can't Predict the Future

Once the whole Libyan crisis, huge income inequalities, protestors in Wisconsin, Republican Congress foolishly cutting all the wrong spending and possible hyperinflation worries subside you may want to buy some tech stocks...of course today's Arm Holdings can quickly become yesterday's pets.com so exponential technological development is no promise of profits.   Here's a handful tech ETFs for the Random Walk Down Wall Street index investor.













*I'm not advocating investing in any of these.  I am not acting as an investment advisor.  I'm just saying looking at these graphs is fun.   Past performance is not indicative of future results. I do not guarantee any specific outcome or profit. You should be aware of the real risk of loss in following any strategy or investment discussed on this website. Strategies or investments discussed may fluctuate in price or value. Investors may get back less than invested. Investments or strategies mentioned on this website may not be suitable for you. 

Sunday, March 6, 2011

Nexus S Coming to Sprint

The folks at chez Engadget are speculating about Sprint's huge booking block at CTIA being an indication of a flurry of new devices, including an HTC tablet, a 3D HTC and a Nexus S.  Let's cross our fingers, I want to dump my Samsung Moment I have with them ASAP.  My iPhone doesn't like sharing a pocket with something so obsolete.

Sprint to Bring out the Big Guns at CTIA?!?!?!?!????!!?!???!???!?!?!

Thursday, March 3, 2011

Great Bill Gates Quote


Says Bill:
One thing I want to understand better is to what degree innovation that improves the human condition is reflected in economic figures.
Let’s say that someone invents a new bug spray that stops mosquitoes from biting people and is super cheap.
Or let’s say someone comes up with a way to replace hearts for people over 80 that enables them to live longer but that costs a lot of money.
I think that innovation does generally map to economic growth over time. But taking a particular innovation and quantifying its direct economic impact is hard.
For example I believe that because the Internet lets people find a lot more information a lot more easily and quickly, it is a much bigger benefit than the economic figures show.
So when people say middle class salaries have not gone up much I think that the “basket of goods” approach to measuring improvement inherently understates how much better life is now than it was in the past.

Friday, February 4, 2011

Capitalism: Cult of a Pack of Wild Hobbyists

It's a Friday night at 12:20.  In Manhattan.  Of course I'm married so I'm at home living conservador.  But that doesn't mean I can't get drunk on Rioja and hit the blogosphere with a crazy futurist fantasy.

I think we've all heard a bit of political noise from the Tea Party over the past two years.  These people are furious about spending right?  Just not military spending.  Or spending in the form of unnecessary tax cuts for the super rich.  Or subsidies for churches.  Additionally, the Tea Party can't even vaguely conceptualize the macroeconomic processes via which cuts in spending could contribute to the deficit.

The deficit, after all, isn't what bothers them.  These people are people who weren't happy with Welfare, and that was before folks started collecting unemployment for years at a time.  I can even vaguely emphasize with them.  There is a species of anger that the assiduously productive instinctively feel toward the leisurely of our race.  It was probably breed into us sometime shortly after the arrival of the city-state.

The interesting question is what will the world be like when the furious working poor and lower middle class leave the job market too?  These people aren't indignant over deficits, they're indignant because they slave away at jobs they loathe while other people freeride for their salary at a purely negligible discount.

But all that is about to change.  Within 10-20 years these people will not being working either because A) they're living upper middle-class lives in 2010 dollars not working and B) they will have essentially nothing to contribute to our society.

There's this thing in humans where we're never satisfied.  If you told someone from the 16th century they could live in the Projects, get occasional medicare and receive foodstamps without working they'd probably think they had passed on into the celestial heavens.  But, since human progress is a continuous process and not a discrete one, the human greed function transforms every "quantity of consumption" input into a "not enough" output even though the selfsame human clamoring for more would probably have dropped out of the labor force if promised the same goods and services for not working if you had offered him the entire package 10-15 years previously.

Ultimately though, the week is linear and human beings tend to sleep.  There will come a time when welfare, and I don't mean checks and government cheese, I mean additionally quantities of social good other humans are no longer ambitious enough to try to grasp, will float around in something resembling a uniform distribution.  Personalized services, crowdsourcing, robotics, natural language processing, gigantic amounts of free and opensource media, 3-dimensional printing:  I love and believe in the human species but I seriously doubt 75-95% of people would work if they could eat gourmet food and experience a panoply of rich virtual environments for a price who's limit is approaching zero.

And what jobs would they do if they indeed had the urge to work?  They express fury at all the cash register clerks and manufacturing workers lining up at the socialhandoutline now but do they really think that low-level office workers and carpenters are not automatable?

In a future filled with robots and houses where your appliances talk to you, in a world where a small mounted headset will fulfill any wish you ask of it, where cars drive themselves and not a human on earth works in manufacturing, manual labor or agriculture there will be a strange group of individuals, something like the personal computer hobbyists of the 70s, the Capitalists.

Capitalist 1:  The Super-Greedy.  Gordon Gekko.  The 4th-generation superrich kid who must become an investment banker.  The guy who planned to retire at 30, achieved the sufficient means to do it comfortably...and worked fevorishly until the day he died.  For some people, nothing will ever be enough.  Although the unemployed welfare recipient of her age can tour the entire solar system for free, she will insist on vacationing at Alpha Centauri and Vega.  Maybe Gordon had a point, Greed, for lack of a better word is -

Capitalist 2:  L'artiste:  Vincent Van Gogh didn't achieve much in his lifetime.  Much less Kafka.  Kerouac wrote over 90% of his work before the publication of his third novel.  Although certainly not all, there is some significant fraction of the population of world artists who would continue working without even the slightest chance of income.

Capitalist 3:  The Creative Technologist:  The most consistent flaw in my belief in the American Dream is apparent when I see how genuinely happy Richard Stallman looks and how cramped and uncomfortable Steve Jobs and Bill Gates look.  Linux.  Firefox.  The Natural Language Toolkit.  I don't think I have to convince you that people will write software for free....

Capitalist 4: The Gamer.  Measures achieving some level of societal good will be programmed into enjoyable game-like formats.  Play with your hand-controller, play with Kinect, play in immersive virtual reality environments, but no matter how you play, the derivative of your playtime will be some type of identified task that needed done for furthering of human society.  This will all be done quite smoothly such that you don't have the slightest sentiment that you're working....

Friday, November 12, 2010

The Top Secret Economic Boom of Generation Z


March 30 1999, the Dow Industrial Average passed 10000 for the first time.   Late August/Early September 2010 the Dow Industrial Average passed 10000 half a dozen times.  The Nasdaq in the same period has been cut roughly in half.  The recent Census finds that 1 in 7 Americans, the most since the 1960s, are now living in poverty.  Not just dull manufacturing jobs have been shipped overseas, but a hoard of IT and Customer Service too.  

My generation shouldn't have a rosy view of the economy.  We came of employment age when the dot-com bubble burst and graduated from college in the midst of the sub-prime-mortgage-turned-credit-turned-financial-turned-economic crisis.  As Napster morphed to Soulseek to Limewire to iTunes and bitTorrent we all realized we were the first generation since the era of Ragtime that would not have "rock" stars, just professional musicians barely living off of the long-tail economy.  Waiters, some nurses, doorman, stock brokers, human resources departments and a whole host of other professional and not-so-professional positions will be held by robots within 12 years.  Many of us have five or six digits worth of student debt.  Most of us live with our parents.  Most of our parents only significant savings is in the form of home-ownership and hordes of them are getting foreclosed on.   Those that haven't seen foreclosure have seen shriveled net worth and credit limits.  The list of negative Dust-Bowl-sounding gloom and doom items is endless.

And yet we feel like millionaires.  All of us.  As long as we can reach the critical salary mass to afford a fast computer, broadband, a mobile computing device and an unlimited data plan.  Why?  Because all we ever wanted was information and now information is free.  Don't get me wrong, we still have a couple of stragglers whose greatest wishes include SUVs and fine wines and other things that are impossible (at least today) to transmit via 0s and 1s.  Pound for pound though, we are a people whose interest is media and increasingly so.  

Remember the late 1990s?  Remember the money you spent on media?  I had a friend (a high school student working in Pizza Hut, mind you) who bought about $10,000 worth of CDs in 1998 from ColombiaHouse.  I remember planning on taking two summer jobs, one day and one night with about a half-hour of leisure budgeted daily when I was in 9th grade with the sole purpose of buying the ultimate collection of DVDs.   A hot CD in 1999, like Juvenile's 400 Degreez or that Santana record that got played to death (think Rob "Matchbox 20" Thomas and that Maria, Maria song) sold for about 18.99 or more at your local mall.   The graver robbery was the singles, 5 to 7.99 to get the hit, the remix of the hit and the probable next single (provided the test groups give it the go).  Movies, forget about it!  I lost my cousin's VHS copy of "Jerry Maguire" in 1997 and my dad made me pay over half my life savings, 22.97 to replace it.  Renting a movie from Blockbuster for 3 days nearly cost $6.  

Flash-forward to 2010.  I've got stations like Pandora and LastFM that generate music based algorithms to please my most particular favorite traits, be the variables mood, chronological, genre, tempo or instrumentation.  I even start BlackHat Pandora accounts with the sole purpose of bending the station to playing the songs I want.  Via a few well placed BookMarks and ThumbsUp I have a station that plays just The Strokes three records and assorted singles, about 57.97 worth of value in 1998 dollars.  File-sharing is inconceivably various , so I can peer-to-peer with best friends, go to sites, swap Flash drives, share over a common wi-fi or even just bump smartphones. High-speed internet means pirates can get their hands on whatever movies they want and get them fast.  It also means those people in the street who used to sell $10 copies of crummy hand-held-captured films now sell $5 high-def BlueRay discs and carry portable players so you can check for Quality Assurance before purchase.  A $20 Netflix subscription lets you stream endless content in 30-odd languages and order two physical discs at a time from a nearly universal catalogue.  

If the quantitative facts about changes in media are so significant, it's really shocking to sit back and feel the qualitative ones as well.  YouTube means anyone can put up content anytime, anywhere.  File-sharing means artists like Mos Def and The Strokes can be some of the defining artists of early 2000s without ever really getting heard on the radio.  The Bed Intruder phenomenon was an incredible rebuff to the record industry.  The greatest hit of the summer of 2010 was a clip from the news auto-tuned by some brothers and ensuing ocean of multi-genre covers of the auto-tuned news clip.  The funniest things of our generation aren't sitcoms or standup or variety shows but YouTube classics like Greatest Cry Ever and David After Dentist.  

As monumental as all that is, its really just information we used to store in physical objects like tapes and CDs made digital.  More interestingly, now even processes are becoming digital or at least partially so, and therefore depreciating as well.  Dating, for instance, remains a process in which it is virtually impossible to avoid all costs.  Yet hope is not lost.  Dating Services, with their broad connectivity and matching algorithms, serve to eliminate those frequent dating experiments with no ROI.  Facebook, 4square, Google Latitude, Yelp's Monocle and other location-based services help create the kind of coordination and environmental knowledge that makes dining, partying or getting a haircut much more successful endeavors. 

My personal favorite is the OpenCourseWare.  Since I was about 11 I've seen the tragic look on people's faces when they find out they can't attend a particular university.  I saw a kid cry when his mom told him he couldn't go to MIT and I saw a coworker at a Fortune 10 banking institution crushed when he received word neither of the Ivy-League MBA's he'd applied to had accepted him.  Those who do settle for universities on their budget constraint line while seriously yearning for a superior system can develop misanthropic habits and miss out on majors ill-explained to them.  At best, they waste a lot of money just for a diploma and sign a contract for a much higher lifetime opportunity cost of networking.  I don't think I even have to talk about high schools.  
OpenCourseWare amends so much of this. August 24th, CNET reported iTunes U's 300 millionth download.  That's about as many university lectures downloaded as there are American citizens. I've personally felt the power of this. I majored in Spanish Linguistics and Economics in college, thinking  that knowing Linguistics and Economics while also speaking Spanish was the best I could do with my life, my budget, my time constraints and those of college programs.  I went to City University of New York, A) because it's the most esteemed public university in America (Gordon Gekko, Andy Grove and Colin Powell went there) and B) because in most other states people (like employers) get the weird impression that you went to NYU when you say City University of New York and it's not lying if your response is always, "NYU is a great school".  

Needless to say, I felt cheated.  I felt abused by market forces.  I felt like I got felt up by Adam's Smith's invisible hand.  I wanted to be a Computer Science Major at Stanford.  I wanted to study Philosophy at Harvard.  I wanted to study Linguistics at MIT.  And I wanted to do them all at once.  By working and not taking a loan I didn't even have a schedule that would permit me to study Engineering at CUNY.  I was jealous of the rich and my envy hurt me.  I kept my bitterness inside as I noticed quickly that bringing up class's correlation with success is not appreciated in conversation, especially amongst the successful.  I soon learned I wasn't alone though.  I heard my suspicions echoed in offices and bars and publics places of all kinds.  I realized most people were dissatisfied with their college education and the few that had nothing to complain of their school generally filled that void in conversation by ruminating on their tremendous debt.  

Then one day Google saved my educational life.  I was watching YouTube, cruising through some programming tutorials, a recent discovery of mine, jumping from C++ to Python, and Primitive Operators to Lists.   Great short tutorials that taught me about the syntax of computer programming languages but not great sources for contemplating the spectacularity of their possibilities.  See a three-minute video on an implementation of a simple list and realizing how to type it correctly such that you get the data structure is nothing like the truly incredible consequences lists can have on your life.  What I really needed was a conceptual tour of the key ideas of software building.  One day I was prompted to a page of one of the true-content producers of YouTube, this guy named thenewboston, who has around 955 tutorials (as of 9/22/10) in multiple languages, most under ten minutes, perfect for median-intelligence people and great for above-intelligence folks new to programming who know how to fast-forward tactfully and realize when the guy is just mumbling incoherently.  Gazing momentarily rightward, in my YouTube Suggestions I caught sight of a blackboard, a man and the words MIT.  The mouse attacked for me:

I discovered MIT's Intro to Computer Science, one of the most ample offerings in OpenCourseWare.  This MIT course includes 24 lecture videos, 2 Complete Texts, Selections of Others, Assignments and Exams.  People are now even forming informal study groups so they can execute the class socially.  Although much OpenCourseWare is of this par of completeness just seeing the videos never hurts either.  The most charming, effective teacher I've ever known is Mehren Sahami doing Programming Methodology at Stanford whose videos I recommend to people of all ages, seriously entertaining by force of his tremendous personality to watch as though it were a sitcom about learning JAVA.  I've taken Computer Science, Mathematics and Economics Courses at Stanford, Harvard, MIT, CMU and UC Berkeley.  I sat through graduation speeches by both Steve "the Ninja" Jobs and Bill "the Borg" Gates.
The democratizing element of OpenCourseWare is a great social thing in the industrial world, but when it hits the third world things will never be the same again.  When broadband and tablets become cheap enough for distribution throughout the poorest of the nations (within 3-5 years) the endogenous growth functions attributable to the percentage of the population involved in STEM fields of those countries are going to explode.  Countries that never so much as had a research facility will suddenly be teaming with experts in Physics, Information Sciences, Mathematics and Engineering.  It is a note of national pride for Americans that we've innovated nearly everything (with a little help from Europe) over the past 200 years.  What most people don't realize is that losing our technological edge is not absolutely negative.  Imagine you're diagnosed with Lung Cancer….will you care that the cure was invented in China?  You lose your leg in accident…is it relevant enough to worry that it's bionic replacement is product of Indian Engineers?

The etymology of the OpenSource Movement (and a frequently-heard phrase of its exponents) is that it's "free as in free speech, not free as in free beer".  I beg to differ.  As automation progressively takes care of all our animal needs and bioinformatics our health, people will continue to create meaningful content without ever occurring to them that they should monetize it.  This utopian day may be very far away but as consumers that doesn't really matter to us, as long as people are willing to distribute content for ad revenue from businesses.  It's a beautiful thought that people someday would create content to inspire, inform and educate other people.  It's a beautiful life we live in today, in 2010, because our inspiration, information and education is largely being subsidized by businesses chasing super-specified ad revenue enabled by companies like Google and Facebook. 

I close with a crude-back-of-the-napkin-calculation.  In 1998-dollars I'm making hundreds of thousands of dollars just living.  Not everyone can be as lucky as me.  If some people had such a large stipend, they might spend it on other things like wine and women. I luckily, would spend it all on content anyways, content as data, content as information and content as knowledge.  The first two are nearly always free and the third is getting there.  On the other hand, I don't pirate much at all, just make due with manipulating the legal distribution channels so I'm sure there's some regular Joes, some store clerks and some Government Assisted-Living people out there pulling in as much as millions of yearly media-salary.  

I run through 6-8 films a week.  At 5.68 1998 Blockbuster dollars, that's $~2000 in income a year.  As I almost only watch documentaries and foreign films and would certainly have to resort to buying (and some cases spending significant time searching for) some, I apply a premium and multiply it times 1.25, to 2500 of 1998 Video Content Dollars.  Musical Albums I love.  With age I've grown and lost some of the ritualistic attachments American teenagers develop with pop music.  When programming, writing, studying or researching I like to find an album that is just pleasing enough to play from start to finish.  Since I do these four activities many of my waking hours, I'm consuming at least 4 albums a day (if the outlier super-music filled weekends are counted and the fact that music I consume mutually with friends was probably obtained freely too (unless by those saints who actually buy songs on iTunes)).   Some are certainly popular enough to cost the $18.99 in popular 1998 album dollars, some not-popular enough to cost less and yet some so unpopular in a physical object market like that of Compact Discs to cost a lot more so we'll make the price of a CD 17 in 1998 dollars. Assuming music was a simple as listening to albums from start to finish, I'd have had to have budgeted $24,820 strictly to the purchase of LP discs. Of course, when I feel like I need to be inspired and just look the song up on YouTube, when I leverage the fact that Android FroYo on my htc Evo 4G has Flash so I can go to any number of music-vendor sites that let you listen to whole songs for free without purchase, when I stream some artist's I love/hate's new single to sample, when I consider that Pandora is often turning me on to songs I'd never have discovered with 1998 radio I'm buying like 10 1998 singles a day.  That's another 23500.    The 2010 GDP per capita of the United States is about 46,400.  The mean-American earner would be stuck with about a $5000 deficit after buying as much 1998-priced media as I do.  I assumed he'd be hard-pressed having a good time and getting a good education.  I wasn't.  I don't keep accurate enough books to tell you how much money I've blown in bars and restaurants but I live in Manhattan, New York and I'm sure with 4Square, Facebook, Twitter, Google Maps, Yelp, a Smartphone and the assurance that everyone I know has one too, I'm saving myself at least 8 spend-a-lot-of-money-yet-have-no-fun-nights who in Manhattan often cost at least 100 before you realize you got to take that cab home (another 20 bucks), so right there you have it, I earn the entire GDP per-capita seated and then some in front of a screen in leisure activities. 
Saved Educational expenditures probably more than double my content-income as I doubt I have to remind remind  you that taking 12 classes at 5 Ivy League Schools this year probably dwarfs all the opportunity cost dollars I just laid out in the last paragraph...

Great Natural Language Processing Videos

Practical Applications of NLP

Speech Recognition and Retrieval

Incremental Bayesian Networks

The Next Generation in Neural Networks

Using Evolution to Design Artificial Inteligence

Bay Area NLTK User Group

Google Wave and NLP

Google Python Class: Regular Expressions

Machine Learning and Machine Translation

Machine Learning @ Stanford

Knowledge-Based Information Retrieval with Wikipedia

Saturday, October 30, 2010

Let's Outsource Reading

Dear Friends,

I love to read.   You love to read.  Why not?  A novel is great experience.  You live in it.  You feel it.  It becomes you.  You become its author or characters or mood.  The words in the book become the words that you come to define your life with and the very nature of your life changes because of the semantic meaning embedding in everything you see or do.

Sometimes though, we are forced to read against our will.  We are made to extract meaning from texts we have no intention of loving.  Worse yet, even if we love the books we are obligated to read, how are we realistically expected to emerge from it's reading with some objective idea of its plot, its language and its moral when its plot is endlessly assaulted by our subjective memories of life and media representing life, it's language assaulted by our idea of language and its moral assaulted by our generally-firmly-established moral compasses?

Any of us in high school or college know the pain of being forced to read novels we would never have read.  Worse still many teachers in the "liberal arts" don't grade based on execution but based on the size of the intersection of their ideology and the student's.  Now if you can cull a few key phrases from your teacher you can interpret a book according to their ideology without every having to abandon your own!

Those of us in the world of  non-matriculated adults wish we could feign a knowledge of books to impress coworkers, possible romantic mates, etc.   The opportunity cost of a book is roughly a week's salary, nearly $1,000 if you are the average American.  Would you like 40 hours and/or $1,000?

There are also those of you who love to read but for whom reading isn't enough.  You read a book five times and then you read all the existing criticism of it yet you still wish you could decipher the nexus of this book more.  Well my friend, I can help you apply the same exactitude Mechanical Engineers or Investment Bankers apply to their respective crafts with a cornucopium of tools you never imagined could be applied to literature.

A quick demonstration:  Let's say you had to read "A Tale of 2 Cities" by Charles Dickens.  Why you just paste the url of a website containing the book's text into my prebuilt function like so:

Please tell me the website containing the text you wish to analyze:
http://www.gutenberg.org/files/98/98-h/98-h.htm


Having pasted in the address of Project Gutenberg's free copy of the book the program outputs a series of facts with which one could easily create arguments for an essay with perhaps just the assistance of the book's description on the back.  There is no doubt that the aid of just the introduction or CliffNotes would be more than enough to proceed with this output to an incredible and insightful essay on the even the graduate level.

A few interesting characteristics:

The first output is that of collocations, or simply the most common two-word combinations of words in the book.  This is a great tool for figuring out main characters and combined with a little Google can really help you decipher character development in a cinch.  Also some of the non Proper Noun combos can be extremely telling of the plot.

The 100 most common two word sets in the book are: 
Building collocations list
Mr. Lorry; Miss Pross; Madame Defarge; Mr. Cruncher; Saint Antoine; Doctor Manette; said Mr.; Charles Darnay; young lady; Mr. Stryver; Mr. Lorry.; Miss Manette;Jacques Three; Sydney Carton; Young Jerry; Miss Pross.; Old Bailey; said Miss; Gutenberg-tm electronic;  n't know; one another; honest tradesman; electronic works;Archive Foundation; Mr. Darnay; Mr. Barsad; Monsieur Defarge; Temple Bar; golden hair; thousand seven; Mrs. Cruncher; seven hundred; Jarvis Lorry; Monsieur Gabelle; Mr. Carton; long time; Mr. Attorney-General; Good day; last night; Doctor Manette.; Madame Defarge.; electronic work; set forth; blue cap; Tellson 's.; North Tower

The limit to this function is your hunger for two-word combos.  I have set it to 100 for demonstration purposes.  You could set it to 1 or 1000.

 Let's move on to some other significant output:


Words used in similar contexts to monstrous are 


Building word-context index...
big blameless certainly complete early easily exactly familiar far
final freely good heavily here ignorant impracticable late litter
little long


Here you can toss a word in and see all the other words in the text that appear in a similar syntactic position of the sentence to that of your word and follow or are followed by the same or similar words.  This tool can be very useful for breaking down an author's style, or perhaps even analyzing the speeches of a political leader or CEO.  Applying it to Pride and Prejudice for example, we find the word 'monstrous'  for Jane Austen had the same sense as 'very' or 'extremely'.   Please note that 'monstrous' was a demonstration pick and you can use any word of your choice.


The lexical diversity score of the book is:  11  

The lexical diversity is the total number of words divided by the number of unique words in the text, the smaller the number the greater variety used by the author.  Nearly no books have scores under 8 or 9 (it would be hard to get a book out saying "the" just once so you can consider that range of "Slightly Below Ten" as extremely diverse.  Other great ways to apply it are to compare one author to others from their period or the book in question of the author with his other works.  Perhaps the "young" books of the author are filled with more lexical diversity as his/her primary concern was dazzling the world with his/her incredible and perplexing descriptions.  Perhaps the "old" books of the author exhibit low lexical diversity because now plot, character development and meaning have grown to take center stage in the work and now words are used like motifs in symphony, repeating throughout the text so as to create a definitive belief system without every explicitly stating it.  These claims are big, but with quantitatively sound lexical diversity measures you can at least dare to make them without your teacher responding with his "instant-write-off-ability" powers.

Now imagine you're reading "Great Expectations" but you saw your teacher at the video store once with an Obama shirt on and a copy of "Steal This Movie: The Life of Abbie Hoffman" in his hand, walking by his desk once you saw a thick tome of Karl Marx's "Kapital" and his hybrid auto has a Dennis Kucinich bumper sticker.  Let's be real.  Unless he is an incredibly detached and objective person - fairly rare in this world - writing a paper that interprets GreatEx from the perspective that Adam Smith's Invisible Hand and Reagan's Supply-Side Economics is not going to get you an A.

Instead you can develop a thesis dealing with Dickens' Marxist side manifesting itself through uses of the words "poverty" and "justice" throughout the work.  But how to know them all without reading it?  Boom:


The word injustice occurs in these contexts:
Building index...
Displaying 8 of 8 matches:
ock at my Lord Chief justice himself , and pulled
ge of the Lord Chief justice in the Court of King
s , the Tribunals of justice , and all society ( 
believe it. I do you justice ; I believe it. " Hi
 love of Heaven , of justice , of generosity , of
er of death , to his justice , honour , and good 
 love of Heaven , of justice , of generosity , of
 mind to impeach the justice of the Republic. She
None 


the word poverty occurs in these contexts: 
Displaying 7 of 7 matches:
ed the air , even if poverty and deprivation had 
abandoned her in her poverty for evermore , with 
ave repented them in poverty and obscurity often 
ations as their bare poverty yielded , from their
 in their children , poverty , nakedness , hunger
ss of twenty , whose poverty and obscurity could 
n the south country. poverty parted us , and she 




Naturally you can modify the amount of the words surrounding your search-word, asking for just a handful of them to get a few stylistic details or thousands at a time to get the whole page of excerpt featuring the search-word in question.

What about all those sources you need to quote?  Building a critical background for your essay is as easy as applying to the same search method to the Journal Articles you've thrown in your bibliography.  Do a quick run-through on "poverty"/"justice" in a handful of articles and you'll have the framework of an critical literary essay worthy of Harper's without ever having read the book nor the criticism you're quoting!

In the starter kit program I have provided these few key operations you can perform on a text.  I know hundreds more. Their output is not simply in the form of sentences and word lists but even graphs and charts or useful functions like search.  You can plot the fifty most used words in a text, do pie charts of word ratios to each other, or comb through thousands of pages with an algorithm much faster than the 'command f' you've been living by.

This program with the output above (legitimately enough for an essay writing) is on sale for only eight dollars.  I can also give you hour training/support sessions for 50$.  We can do as many such sessions are you desire to learn or my knowledge on Natural Language Processing provides.    Or you can just pick some of the aforementioned other methods and I will add them to the code for you just 4 dollars each (price negotiable as number of other methods passes 10).  Your choice.  

We let computers free of us childish basic arithmetic, algebra, geometry and statistics years ago and we  proceeded to an incredible degree of understanding of the maths and sciences.  Who would really believe then that we can't eliminate some of the more shallow reading functions and come away with more profound knowledge of texts?