Fractals of Change

FeedBlitz, Stage II

Founder and CEO Phil Hollows of FeedBlitz (in which I’m an investor and of which I’m a board member) announced that the company will make all its features available free to publishers who permit the placement of small ads at the bottom of the emails sent to subscribers. Publishers who prefer not to run ads pay FeedBlitz for its services according to their readership. This is similar to the way that Google licenses search for use on a company website: it’s free if you let Google run ads and you pay for it if you don’t want Google ads.

Formerly FeedBlitz had a free version with limited features and earned revenue from upgrades - another respectable Internet model. This was a choice born of necessity for a startup – you can’t get advertiser attention  for a tiny circulation. But, with over 2.5 million subscriptions served, some advertisers are investing resources in figuring out how to use FeedBlitz.

Note that this is very different from the well-known business of sending ads as email (SPAM if you’re not interested in the product or service being offered). These small ads are attached to the bottom of emails which have content that readers subscribed to (and verified that they subscribed to). No additional emails are generated for advertisers. The advertisers don’t have access to the readers’ email addresses. These are analogous to the ads which appear in paper subscription products and help pay keep the costs of newspapers and magazines down. They may turn out to be more effective than plain web ads as advertisers learn how to use them since they are attached to valued content but advertisers needed a large enough potential audience to justify learning.

FeedBlitz will discontinue its feature-poor version next month and upgrade all publishers using it to the full-featured model with ad support unless they’d rather pay for service directly (or elect to take their subscribers and leave).

When to seek revenue from the user base and how are tough questions for any web service company. If you don’t have a critical mass of users, you don’t have a business. If you charge users who, themselves, are not directly making money by using your service, you’re likely not to have very many users. Professional users of FeedBlitz paid (happily, I think) for the features they needed. Non-professional users, with some exceptions, were reluctant to pay for service out of their non-existent blog revenue.

Some services simply grow their user base without revenue until somebody buys the service for the users and that somebody – Google or Yahoo, for example – already has an inventory of ads and tries to figure out how to serve them as part of the newly acquired service. In fact, that hasn’t been a bad strategy for some although monetizing the acquisitions is often more difficult for the acquirer than it expected.

Our guess as the FeedBlitz board was that the market for email (and Twitter and SMS and IM etc.) distribution of content is still in its infancy. We think growing with that market’ll be exciting.

Many more small blogs and small websites are about to spring into existence to serve existing groups which, up to now, couldn’t use the web as the basis of member communication because broadband penetration among members was too low. The cost of sending an e-newsletter is much, much less than a physical one even if you don’t put a price on the time spent stuffing and stamping envelopes. An e-newsletter, particularly if it actually is generated by posts to a website, is also more timely and more accessible than its physical equivalent. And readers don’t have to worry about  putting it in the physical recycle bin or suddenly needing it after it left on the recycling truck.

Some of these new blogs will want (we think) to pay for FeedBlitz service and be ad-free or run their own ads.  Others will be glad to have FeedBlitz provide them with the capabilities they need and obtain advertising which small blogs wouldn’t be able to obtain on their own to pay the cost of the service.

Watching companies you invest in come of age and make choices is almost (but not quite) as scary as watching children grow up. In both cases you get to give advice but you can’t determine the outcome or even be sure your advice is right. 

Short Fall Charges – Verizon Style

We generally make our landline long distance calls on Vonage. As a backup and partly out of lethargy, I left Verizon in Stowe as the LD carrier on the one Verizon line we have there. Carefully chose the Verizon plan which says it has no monthly fees but has high rates because didn’t intend to use it anyway - the plan that’s meant for impoverished and friendless hermits.

This was actually a good deal for Verizon both because we did occasionally use it on purpose and, more often, because we dialed out on the wrong line and ended up paying ten cents/minute for domestic LD and astronomical amounts for international.

But we haven’t been in Stowe for the last months so I was very surprised top see that we still had a $2.18 charge for LD. It’s all itemized on my bill: this is a shortfall charge:

“Your Long Distance qualifying charges this month are 0.00. When Long Distance charges are less than $2.00, the shortfall charge is the difference between those charges and $2.00.” Of course there’s an eighteen cent Federal Universal Service charge on top of the two buck charge for NOT making any calls.

Why Verizon can’t just say there’s a two dollar minimum is a mystery. Shortfall charge? come on. Am I being punished?

This is the residual culture of a politically well-connected monopoly. They can charge me for NOT being listed in the directory. They can charge me for NOT making long distance calls. This is rent-collecting in spades. It’s like the fee John D. Rockefeller got the railroads to “agree” to pay to Standard Oil for every barrel of oil they hauled for Standard Oil’s competitors.

But they’re not a fully-protected monopoly any more. I’d already abandoned them for LD and all but one local line. The short fall charge reminded me to simply cancel the LD service from them. Slowly – painfully slowly – competition will win out.

BTW, I can no longer find the plan I signed up for on Verizon’s website. I’m sure I saw it the other day listed as “no monthly charge”. I was looking to see whether “shortfall charges” were properly disclosed.

Recovery Log – Still Not Done

Plan had been to do a retrospective on our hard drive failure and recovery experience in this post. Problem is recovery is not yet done so too soon for a complete retrospective. Mr. Murphy’s been around enforcing his law.

We’ve been trying to download the two 3.6 Gigabyte files created by Mozy restore (1.3 Gig file was downloaded by the time of yesterday’s post). At the sustained effective download rate of about 1.0 Megabits per second (which is about what we’re getting over our nominal 3.0 megabit connection), we’re downloading about 100-125 Kilobytes of data per second (eight bits per byte for non-nerds plus some extra bits per packet for technical reasons). That’s eight hours of downloading if all goes well.

But all doesn’t always go well for eight hours.

Last night Windows downloaded a patch for itself and decided it needed to restart. Apparently this machine is set up to let Windows reboot whenever it wants to so it did. Didn’t care that we were well along in the overnight download. Need to fix that setting, especially on patch Tuesdays.

Restarted this morning. About .5 Gig in our Internet connection went down. Rebooting the router seemed to fix it. Some hours later, the connection is down again. Apparently something going on at our usually reliable ISP.

It’s my birthday so we went boating.

Connection apparently stable now. Have restarted the download.

Will get to more of it in the retrospective but this is poor design on Mozy’s part. The download should preserve however much of the file has been successfully downloaded and let us restart from there. This is very common technology. In fact. Mozy upload which does the daily backup does work this way. Microsoft downloads can recover from failure and go on.

If I have to I’ll work around this by only requesting some of Mary’s “lost” folders at a time to create smaller downloads but that’ll be a pain. Will give 3.6 Gig download one more chance.

Oh yeah, got an email from Mozy tech support today (Tuesday) morning answering my weekend question “isn’t there an easier way to do recovery?”. Short answer: no. In fact, they don’t even suggest breaking it up into smaller chunks which would have been a better way to start.

But I’ll get to the retrospective when this is behind us. Conclusion and conclusions are here.

Recovery (I hope) Log

Bottom line, if you don’t care to read about my mistakes, is that using offsite backup service Mozy did save our bacon (or at least our data) when a hard drive went bad. The process is more difficult than it needs to be but does work. More lessons and suggestions in the next post.

8:10 AM: Mary’s six year old Dell is showing a blue screen of death. Message says Windows is terminating to avoid damage to her machine.

8:15 AM: Reboot seems to work but the hard drive or fan (please let it be the fan) is making a funny noise.

8:17 AM: Mouse response is erratic.

8:18 AM: Blue screen of death. Same message.

8:20 AM: Reboot fails with a similar but not identical message. This one suggest removing newly installed devices. There aren’t any but I did move some USB connections around in the last couple of days.

8:21 AM: Try rebooting from “last good configuration”. Same message.

8:22 AM: Take the machine out of the docking station (isolates from all devices) and reboot. Noise is still there. Message says to run CHKDSK (hard to do on a computer that won’t boot).

8:24 AM: Reboot consistently says it can’t find the boot drive. Time for Plan B.

In theory I could reboot from one of the CDs that came with the machine and then run CHKDSK, perhaps even repair the drive. That’s pretty theoretical though because I didn’t bring the CDs (assuming I still have them) to our summer place. Besides the noise from the drive probably means it’s really kaput.

However, we have a desktop machine which I’ve been meaning to install as the house computer for guest use. Most important, we’ve been using a service called Mozy to backup critical data from our machines to somewhere in cyberspace. Now’s the time to see whether I can actually restore from there (Yes, yes, should’ve tested this when it wasn’t critical but didn’t).

8:35 AM: Have scrounged together a monitor, keyboard and mouse and snaked a cable out of the utility room to the box (this machine doesn’t have WiFi installed). Machine boots normally although it complains about its internal battery being run down (hasn’t been used in a year).

8:40 AM: Brief panic. Did I store the Mozy password for Mary on Mary’s machine? No, it’s on mine which also means it’s on my phone in case my machine should be down.

8:41 AM: Mozy says that Mary’s files were backed up last night. I request recovery. Mozy says it’ll take a while to get the files ready for download (I’m not sure why) and that it’ll send me an email when they’re ready. Also offers to sell us a DVD with the files on them as an option.

8:50 AM: No email from Mozy. Go the account screen and it says that 400 of the 4000 odd files have been gathered and it’s working on the request. A refresh tells me the number is incrementing although slowly (still can’t figure out what it’s doing but it seems to be OK).

9:00 AM: Figure I might as well run Windows Updates on the machine while waiting for Mozy. It has been a year. Of course Updates has to install a new Updater and has to verify our copy of Windows (I was afraid it would ask me for the code on the Windows CDs which aren’t here but it didn’t).

9:10 AM: Updater complains that the clock is wrong. Sure enough, machine thinks it’s 2004, probably because it’s internal battery ran down.

9:12 AM: 67 Updates! This’ll take a while. Well, Mozy’s not ready yet anyway and I haven’t had breakfast.

11:30 AM: Windows says its done updating and wants a reboot. Done. McAfee wants to refresh itself.

11:40 AM: McAfee refresh crashes into a script debugger but McAfee now says it’s current.

12:44 PM: Mozy emails that the restores have been completed and are ready to download. Turns out there are three files of 3.6 Gig, 3.6 Gig. and 1.3 Gig respectively. No idea why there are three files but decide to see if I can find some way just to download Mary’s outlook.pst so she can get back on email properly.

1:00 PM: Nothing specific in the Mozy doc about restoring from a dead machine to a live one. Would think that would be a fairly common situation. But it looks like the Mozy client can look at the backed up files as if they were a virtual disk.

1:05 PM: Windows decides it has 10 more updates it needs badly, badly. Let it start doing that.

1:10 PM Download and install the Mozy client. Since this is a new machine, I have to pay for now having three machines registered. But turns out that this client – logically, I guess – wants to backup the new machine, not find and restore files from the old machine. Decide to try live chat with a tech rep but that’s only available Monday to Friday during office hours. Yuk! Not good for an emergency service.

1:12 PM: Look on the Mozy virtual drive but it is empty (of course) on the new machine and no apparent way to shift it to be the backup of the dead machine.

1:30 PM: Start downloading 1.3 Gig file. Better to find any problem with that than after waiting to get the bigger files. Hope I don’t need all three before I can do anything. Doc doesn’t say why there are three files but I suspect it is to avoid even huger single downloads. My connection (nominally 3 meg) usually tests out at a download speed between 1.0 and 1.5 and it looks like I’m getting close to one meg. Still, this’ll take a while. Sent an email to Mozy support to see if I can get some hints this way.

3:27 PM: Mozy robot responds to my email: “Thanks for contacting Mozy Support.  This is an auto response, just to let you know that our hours are between 7:00AM and 5:00PM Mountain Time Monday through Friday.” Wonder what the robot was doing between 1:30 and now?

6:30 PM: Have the 1.3 Gig file down. It’s self–extracting and a number of Mary’s files from her old machine reappear. I look through the files and don’t see the mail pst so start to download one of the 3.6 Gig files. That’ll take until tomorrow morning.

Next morning 8AM: The good news is that the download seems to have accelerated overnight and has finished. The bad news is that I apparently clicked run instead of save (the doc says to click save) when the download began last night. For some reason the file can’t unpack (probably because it’s in the temporary Internet folder) and promptly erases itself. Yuk!

8:15 AM: Because the objective is to get Mary’s email up as fast as possible and she can wait for the rest of her backed up folders, I do what I should have done in the beginning and see if I can isolate just her pst files and download them. Mozy has a nice search capability of what it has in storage. No problem finding both outlook.pst and archive.pst and queuing them for restore. They’re just a few hundred meg so shouldn’t take long.

8:20 AM: While waiting for the pst files to download, I decide to search the new machine for any existing pst files both to know what to replace and to double-check that there’s nothing good in them. What I find is that the pst files from Mary’s dead machine ARE on this machine. How can this be? I didn’t see them in the unzipped stuff downloaded yesterday.

8:22 AM: OK, I’m stupid. Of course I didn’t see them. The Windows default is not to show system folders and files and that’s what they are and where they are. They did come in the first download (just good luck) but I couldn’t see them until I did a search (which I had told to look for system files and in system folders). I drag them into the folder where Outlook expects to find them after renaming the existing outlook.pst (doesn’t look like it has anything relevant in it).

8:24 AM: Outlook opens fine. Mary’s mail is there, A little configuration and it is attached to our mail host from which it downloads about 2000 old emails for some reason but new stuff as well.

8:30 AM. Mary has a lot of stuff to delete that she deleted once before but she’s back in business (or back doing public sector, non-profit stuff which is what she does now). In the background, the 3.6 Gig download is running again.

Next post, lessons learned and suggestions everyone with data at risk (that’s everyone on a computer, of course) and Mozy.

Mary's Tough New Assignment

See below for why I'm especially proud of my wife Mary today. The position which she was selected for by Governor Jim Douglas, House Speaker Gaye Symington, and Senate President Pro Tem Peter Shumlin is both tough and crucial to Vermont. She will be the first chair of the Telecom Authority set up in the e-state legislation passed last Spring. The Authority's mission, which she has accepted, is to make sure that there is no spot in Vermont which does not have both cellular access and broadband connectivity by 2010. She can't move the Green Mountains but I'm sure she'll find a way to work around them.

State of Vermont

For Immediate Release:

Friday, August 24, 2007

Contacts:  Office of the Speaker, (802) 828-2245

Office of the President Pro Tem, (802) 828-3806

Office of the Governor, (802) 828-3333

Speaker, Pro Tem and Governor Announce Appointments to Vermont Telecommunications Authority

Montpelier, Vt. – House Speaker Gaye Symington, Governor Jim Douglas and Senate President Pro Tem Peter Shumlin today announced their appointments to the Vermont Telecommunications Authority (VTA).

The 11-member Authority is a key component of the e-State Initiative—a plan to make Vermont the first state to provide universal cellular and broadband coverage everywhere and anywhere within its borders, and building on earlier legislative initiatives to encourage broadband access in rural areas of the state.

The product of comprehensive telecommunications legislation endorsed by Governor Douglas and the Legislature in the last session, the VTA is empowered to partner with service providers, private investors, municipalities, and federal agencies to make investments in telecommunications infrastructure and advance the spread of broadband and cellular services in every area of the state. Authorized to deploy up to $40 million in state bond backing, the Authority hopes to leverage $200 million or more in private investments.

The Governor, President Pro Tem and Speaker each appointed two members and jointly appointed the Chair and Vice Chair.

Vermont Telecommunications Authority Membership:

• Mary Evslin, Chair (Jointly Appointed)

Evslin has an extensive technology background and currently works as a consultant.  She was co-founder of ITXC Corporation, the Fastest Growing Technology Company in North America according to Deloitte & Touche in 2002.  Her company had presence in over 175 countries and was considered one of the world’s largest international wholesale carriers of telephone calls and the largest Internet telephone carrier with 350 employees at its peak.  Among other community service, Evslin has served on the boards of the Helen Day Art Center in Stowe, the American Red Cross and the Vermont Humanities Council.  She is also co-founder of the Internet for All, Now Committee.

• Peter Meyer, Vice Chair (Jointly Appointed)

Prior to his retirement from state government, Meyer was the Executive Director and Environmental Analyst for the Vermont Public Service Board, where he served for 17 years.  Before joining the PSB staff, he was the Chief Coordinator for the Vermont Environmental Board for 10 years.  From 1995 to 2003, Meyer served as the Treasurer for the Central Vermont Memorial Civic Center where he helped raise funds for, build and operate a recreational facility for the Central Vermont community.  He is currently on the Board of ECHO, Lake Aquarium and Science Center at the Leahy Center for Lake Champlain. 

• Jeb Spaulding, Treasurer, State of Vermont (Ex-Officio)

Spaulding is serving his third term as Vermont State Treasurer. During his tenure, he has emphasized continuous improvement in treasury fundamentals such as timely and accurate bank reconciliations, enhanced customer service, expanded outreach and compliance in the unclaimed property division, and proactive investment management and oversight. In addition, he has initiated a new focus in the Treasurer’s Office to leverage State assets to encourage responsible corporate citizenship and to support community and economic development within Vermont.  Treasurer Spaulding served eight terms in the Vermont State Senate from 1985-2000.

• Jo Bradley, Vermont Economic Development Authority (Ex-Officio)

Bradley has served as Chief Executive Officer of the Vermont Economic Development Authority since February, 1997.  Her previous professional experience includes the positions of Deputy Secretary at the Vermont Agency of Commerce and Community Development, Commercial Loan Officer at the Vermont Economic Development Authority, and Vice President at Solomon Brothers, Inc. in Boston. She has also served as the State Government representative on the national Advisory Committee of the Export-Import Bank of the United States (Ex-Im Bank), and serves on numerous state and local Boards.

• Al Brisard (Pro-Tem Appointee)

A proven manager with a unique combination of technical and business experience, Brisard has served in a wide range of operations, marketing and leadership positions in companies specializing in technology and telecommunications.  His experience includes the design and implementation of multi-location LAN and WAN, network monitoring and the analysis and delivery of voice over IP, audio conferencing, wireless services and other network elements. 

• Jerry Johnson (Governor Appointee)

Johnson is a senior executive with a proven record of effectively leading organizations ranging in size from less than five employees to over one thousand. With 25 years experience in telecommunications, Johnson held positions as General Manager of Integrated Planning and Vice President of Network Design at NYNEX. He later served as President and Chief Operating Officer for Hill Associates Inc in Colchester, a leading corporate telecommunications training company.  He also created an independent consulting company, Daybreak Intellectual Capital Solutions Inc. in Williston, specializing in telecom and Internet applications, technical training, and executive coaching with a focus on small companies and startup operations. Johnson served as the President of the Vermont Telecommunications Advancement Center (VTAC) established in conjunction with Gov. Dean and served on the Vermont Council on Rural Development's Telecommunication's Infrastructure Committee.

• Robin Lane (Pro-Tem Appointee)

Lane is a full-time member of the faculty in the e-business management program at Champlain College. A proven manager with a keen understanding of next generation technologies, she was CEO of the software publishing company Microseeds Publishing and CEO of Together Networks Internet service provider.

• Thomas Murray (Secretary of Admin. Appointee)

Murray was appointed state’s Chief Information Officer and Commissioner of the Department of Information and Innovation in March 2006.  For 16 years, Murray worked in the telecommunications industry with TDS Telecom and RCC/Unicel, serving in a variety of roles from marketing, regulatory compliance, technical support and upper management.  In 2003, he joined the state of Vermont, to serve as the Director of Telecommunications Infrastructure, working to improve cellular and broadband services in the state. He also served as Deputy Commissioner of the Department of Economic Development.

• Dawn Terrill (Governor Appointee)

Terrill is the owner and operator of Janitech Inc. in Burlington Vermont.  Previously she served as Secretary of Transportation and Deputy Secretary of the Agency of Commerce and Community Development. Terrill has also participated in a wide range of state initiatives and organizations, including the Vermont Economic Development Authority, the Vermont Broadband Council and the Vermont Council on Technology.  Prior to these roles, Terrill served as President and CEO of Hill Associates in Colchester, Vermont.

• CJ Vadnais (Speaker Appointee)

An advocate for universal broadband and its economic value, Vadnais was one of the driving forces behind the Southern Vermont Broadband Cooperative.  The Southern Vermont Broadband Co-op is the only successful broadband Co-op in the state.  He is currently a computer programmer at Williams College in Massachusetts.

• David Zahn (Speaker Appointee)

Zahn is the president of Signal Advertising, an electronic advertising agency in Montpelier.  Since 1985, Zahn has worked in the computer graphics field and interactive marketing, beginning with their company, Wordgraphics in Stowe where they ran one of Vermont’s first graphics studios and helped pioneer the use of computers in advertising and design.  In 1991, he and his wife created the full service electronic advertising agency they run today.  Zahn has also worked on interactive media and hyper-text based presentation systems since 1988, and was an active developer in the CD-ROM publishing business in the early 1990’s.  He is also co-founder of the Internet for All, Now Committee.

###

LocalReplay PrePlayed!

LocalReplay, a stealth-mode startup which I invested in, was just outed on Alarm Clock and other sites on the basis of a LinkedIn help-wanted. So much for secrets in cyberspace. The bot’s are ever vigilant.

LocalReplay is a hyper-local site for youth sports and quietly began hyper-locally in North Carolina and vicinity with intense coverage including video and stills of games and players, StarCards, TeamCards, and many other kinds of cards as well. The cards are inspired by what we used to collect from bubblegum packs and have not only pictures but also videos associated with them. Players fill out their own StarCards; TeamCards are composites with parts filled in by coaches and other parts filled in by fans and players,

Here’s the StarCard for basketball player Jamius Gunsby of the LaGrange, GA Wolverines.

Use of the service has been spreading as one team plays (and talks to and emails) another. Professional and amateur photographers and writers, fans, and players all contribute content. The result is a rich archive full of stuff that people really care about. Evidence includes eight minute average visit time and 45% repeat visits.

LocalReplay is an example of what I believe will be the next wave of Web successes – services which strengthen and enhance existing local groups. Broadband penetration has now increased to the point where even a local group can count on most of its members having access. FaceBook (see earlier post) is a prime example of this new breed of application: it didn’t create the campuses, obviously; but it does serve them and, in its initial stage, spread virally campus to campus rather than person to person.

LocalReplay is also an example of how local news can be much richer in the days of the web. Broadcast video of local sports has been almost non-existent, sometimes appearing briefly on local cable channels or over the air. Yet watchers of local sports have been recording it ever since the super-8 movie camera made that possible. The first video-cam I ever used was one we rented to tape a basketball game daughter Kelly was playing in. Now all that recording can be made broadly available to the people who care about it. Everyone who wants to be is a video sports reporter. Proud grandpa in Grand Rapids can watch (and rewatch) the touchdown in Topeka. Last year’s game is as accessible as this one.

Success for these hyper-local sites will come a community at a time. It is possible to succeed in some places and not in others because local participation defines success or failure. Scale is needed to support software development and the platforms but 10% penetration of members of 99% of the targeted local groups is failure while 60% penetration of 20% of the groups could be a wild success (although allows room for competitors to steal a march). It’s better to get it right locally than blow out fast everywhere.

You can read more about the outing and LocalReplay on the blog of founder Galeal Zino.

Related posts:

For Web 2.0 Success - Think Local, Act Local

Web 2.0 – The Global Opportunities in Local

Local – The First Life Opportunity

The Newbies are Coming

Web 2.0 is Stuck at a Local Optimum

New My Way Blog

The newest member of the My Way network of blogs by entrepreneuers is boxee blog by avner ronen. I was convinced to add this blog to the network as soon as I read the post below which is called wanted: a code poet.

***************************************************************************

we are trying to recruit a developer in Israel.

i asked Vulkan if he can give me a short job description, so that i can ask people if they know someone. so being the methodical, process oriented, bureaucratic animal that he is he told me "you know. someone that can really code. a shark."

so here is what we’re looking for:

someone that can really code.
a shark.
a code poet.
a lazy bastard.
a hacker.

someone that can write PHP with two hands tied behind his back,
but dreams in C++.

favorite beach book reading: design patterns (hard cover edition, signed by the authors)
favorite debate topic: LAMP vs. J2EE (prefers LAMP, but can argue both sides of the issue)
favorite animal: penguin.
hobbies: none.

(social skills are not a requirement)

if you think you fit the bill please send us an email (don’t forget to mention your favorite compilation flags for the Linux kernel).

*************************************************************************

I understand what avner's looking for: the great programmer that's worth fifty good ones. Find him or her and social skills are certainly optional (and would probably only lead to distraction).

In Defense of Bubbles

The press is full of stories of people who’ve been hurt by the housing bubble, people who borrowed too much and now face losing their homes because both the housing bubble and the sub-prime credit bubble are – not coincidently – ending at the same time. What’s missing from the press I read are stories of people who bought homes before the bubble, didn’t refinance extravagantly or sold and took profits, and now have a nice nest egg. What’s also missing are the stories of people who got credit even though they would not have qualified previously and now own partially-paid-for homes as a result.

There is no question that there are families in real trouble because of over-easy credit. There is no question that lenders and those who finance them got greedy and extended credit they never should have extended. There may be a social purpose in helping some of the former; there is no social or economic reason to help the latter. It would be bad policy to do so.

But we don’t want to eliminate future bubbles; bubbles do more harm good than good harm.

You wouldn’t believe that assertion if you read the story in this morning’s NY Times about American’s adjusted growth income from 2000 to 2005 headlined 2005 Incomes, on Average, Still Below 2000 Peak. The story contains this dismal graph:

Sure enough, if you measure from peak year 2000 when there was a huge stock market sell off (meaning many, many people were taking gains even if they weren’t selling at highs), you find that adjusted gross income (inflation adjusted) hasn’t recovered.

It took a little work since The Times didn’t deign to give URLs of its source; but I went back to IRS reports that describe the period covered in this graph (http://www.irs.gov/pub/irs-soi/05intba.xls and http://www.irs.gov/pub/irs-soi/01intba.xls) and extended back in time to get some perspective (http://www.irs.gov/pub/irs-soi/97ina.xls). The resulting graph tells a different story:

 

Incomes started to rise rapidly in the late 1990s as the bubble got underway. They did collapse FROM THEIR HIGHS as the bubble burst; they never fell back to the pre-bubble level (inflation adjusted) of 1996. If anything the bubble seems to have spurred a faster long-term trend in income growth (but this graph alone doesn’t prove that). It’s a fact that the bubble of late 1990s paid for the infrastructure on which the Internet has grown.

And, with hindsight, I think we’ll find that the housing bubble financed a permanent upward bump in the balance sheet of many American families AND made home ownership – continued home ownership possible for people lenders would have turned away.

Snarky notes for math nerds: when you put numbers together from different sources and double-check them, you find funny things. The NY Times (or its unnamed source) think that $7,422,495,663,000 (total US AGI in 2005) rounds to $7.43 trillion. The IRS incorrectly inflation-adjusted AGI for 1997 in some of the spreadsheets cited above (I corrected).

The first post ever on Fractals of Change was about the great Internet bubble and the reason why this blog’s motto is “nothing great is ever accomplished without irrational exuberance”.

P2P – Boon, Boondoggle, or Bandwidth Hog? – Is Metering the Answer?

Reader Aswath posted a comment on Thursday’s post suggesting that charging users explicitly for both upload and downloads pricing is “an equitable  solution” to the congestion problem ISPs claim is caused when peer to peer (P2P) services use some of each user’s “unlimited” Internet capacity to serve other users and as a substitute for the service provider having to buy more Internet bandwidth itself. Aswath’s a smart guy and knows his telecom; he’s right that this is a solution to the problem; but I think it’s the wrong solution.

P2P does involve shifting bandwidth usage (and usage of other resources) from the service provider to the users. If users choose to let Skype or BBC or Napster use some of their CPU power or disk storage to serve other users (as they actually do when they sign up for Skype calling, BBC iPlayer, or Napster sharing), that’s a personal choice (as long as they’re informed that’s what they’re doing).

You could say the same thing about “their” access bandwidth; they are paying for it. They should be allowed to use it as they please. But ISPs do rely on the fact that most access bandwidth isn’t being used most of the time. If they assumed it would all be used all the time, backbone networks would have to be many times larger than they are today and Internet access would cost us all many times what it costs today (especially in markets which are more competitive than the US).

If, as Aswath suggest, we all paid for what we actually used, we’d be very careful of what was done with “our” access bandwidth and not place more load on the backbone than we’re willing to pay for. We’d be very sure that applications we choose to run, especially P2P applications, don’t run up huge bills behind our back

“Granted,” Aswath writes, “users need to be retrained from the flat-rate pricing, but it eliminates ISPs' cries of wolf.” Seems to me that usage pricing for Internet use is to draconian a cure to the problem, particularly if the problem IS just ISPs crying wolf.

I think unmetered use constrained only by the size of the access pipe one chooses to buy (and the actual capacity you can obtain over that pipe in true life) has helped to give the Internet the enormous utility it has. Several reasons for this:

1. usage billing systems are themselves expensive and drive up the cost of service. Lots of records have to be kept of lots of little things. Detailed bills need to be produced; expensive conversations have to be had to resolve billing disputes. What if I download 100meg of a 101meg file and my Internet connection blinks (or I think it did) and I have to start all over? Am I entitled to a credit? Do I have to pay for spam?
2. people are afraid of accidentally running up a large bill so refrain from many activities. How do you know how many bytes is a VoIP call is going to be? Are you gonna tell your kids not to send you big pictures? Videos?
3. people hate complexity. As long as you’re going to be billed at all, there’s nothing simpler than having your bill be the same amount each month. You don’t have to reconcile it; you don’t have to worry about being overcharged. There are no unknowns in your budget. My experience is that people will pay a premium for simplicity (see this old post).

Moreover, simply charging per byte transferred doesn’t reflect the load each user places on the network as a whole. In general, data transferred during non-peak periods cause no incremental cost to the ISP nor do they cause congestion for other users. It’s only when PEAK loads grow that congestion is experienced and ISPs need to buy more backbone. Those users who transfer data to other users offpeak are actually increasing the productive use of the network at not cost to the ISP and making the network more useful to other users. Should they be penalized or rewarded for providing this service using their own machines and bandwidth?

Even during peak periods, it’s not clear that users running P2P applications actually add to total network load. Take the example of BBC iPlayer whose use involves copies of TV shows being legally transferred from one user to another. If this service really is popular, many of the transfers will traverse only a small distance on a single network. If the transfers were all being done from central BBC servers, all of those people who do not use the same ISPs as BBC will cause long cross-network transfers. ISPs would argue, however, that BCC wouldn’t accept the cost of running the servers and buying the access necessary for running all of these transfers directly so they simply wouldn’t happen.

Net, as always I appreciate Aswath’s thinking. Don’t agree this time with the conclusion and will try to present a solution to the possibility of P2P arbitrage in a future post.

The first post of this P2P series is here.

P2P – Boon, Boondoggle, or Bandwidth Hog? – The Dark Side

Yesterday’s post explained how peer-to-peer (P2P) applications use the processing power, bandwidth, and storage capacity of participants in a service rather than centralized resources. This makes such applications generally less subject to catastrophic failure, much less subject to running out of resources (since each new user brings new capacity as well as new demand), and much cheaper FOR THE PROVIDER of the application in terms of hardware and bandwidth required.

It’s the FOR THE PROVIDER part that’s the rub. Let’s consider the case of BBC’s iPlayer service. For a seven days after most broadcasts, UK residents over 16 years old can download the show free and store it 30 days on their PCs for later viewing which can be offline. The current version doesn’t even download ads with the shows.

Sounds great, right? Just what TV should become on the Internet. Not so fast, according to British ISPs. They complain that this new service will overload their networks and that BBC has no right to do this (although the British regulator has given them permission). The ISPs say they may have to throttle the number of people who can get the shows or “protect” themselves in some other unspecified ways. Usually (by Fractals of Change, at least) the UK is held up as a model of a competitive market for broadband services where issues of net neutrality don’t raise their ugly heads. What’s going on here?

I suspect that the fact that iPlayer is a P2P service is at the root of the problem that ISPs have with it. If BBC downloaded shows to those who requested them directly from its own servers and if the service proves as popular as it might, BBC would have to buy huge (or huger) pipes of its own into the Internet and ISPs would get some revenue from that. However, in a P2P implementation, it is likely that only a few seed copies will be downloaded directly from BBC to the first people who ask for a particular show. Subsequent requestors will get their copies from the first requestors. BBC describes it this way in their terms of services (TOS):

“When you install the BBC iPlayer Library you will also install peer-to-peer file sharing software from Verisign Inc. This software has a file share feature that enables other BBC iPlayer users to download BBC Content through your personal computer (using part of your upload bandwidth), via a secure link, to their personal computers. … When you use BBC iPlayer Library you shall not have the option to 'switch off' the peer-to-peer functionality as this is a core component of the BBC iPlayer Library.”

BBC also warns: “… you are responsible for paying all expenses that you may incur in connection with your access to and use of BBC iPlayer including your internet service provider charges and any excess charges to that provider if you have a cap on downloads and/or uploads…”

OK, fair warning if you’re in the habit of reading TOS carefully. But this warning does NOT appear in any of the marketing information for the service which I saw.

So why should the ISPs be upset? If the users upload too much, they’ll be charged more and the ISPs’ll get paid more – not by BBC but by the users themselves. The problem is that not all ISPs have upload or download caps. Those that do usually don’t advertise them very prominently if at all. Moreover, most users may be well below their caps now – almost certainly are. So the users will use bandwidth which, from their point of view has been sitting idle, and either won’t pay the ISPs any more or will be outraged if they are presented with a bill or thrown off an “unlimited” service for overuse.

It is highly unlikely that you use more than a small fraction of the bandwidth on your connection to your ISP most of the time – usually your connection is idle. But ISPs count on the fact that most connections are idle most of the time; there isn’t nearly enough backbone capacity to handle all the traffic which would result if all the local connections were busy all the time.

“Oversubscription” isn’t fraud; it’s the correct way to design networks which are inexpensive enough for their users to use. The phone network wouldn’t work if all phones (or even more than a small fraction of phones) were offhook at the same time. The highway system wouldn’t work if every driveway were disgorging and engorging its maximum capacity 7x24. The backbone of most networks is designed assuming that most spurs will be idle most of the time.

So if BBC succeeds in low cost distribution of its content without buying new capacity of its own and users substantially increase their use of download and upload capacity without paying extra themselves, the ISPs face either providing degraded service or a sudden need to upgrade their networks (or higher charges for uses of other providers’ networks).

Are the ISPs right to try to make BBC pay directly or indirectly for use of enduser bandwidth? Next post a smart reader suggests that Internet access be metered to avoid even the perception of P2P arbitrage at the expense of ISPs.

Timely note: Ironically, as I write this, P2P network Skype is experiencing a rare outage. P2P services are generally more outage resistance than services which depend on a centralized bank of servers which can be subject to all sorts of catastrophes but they’re not invulnerable. Some networks which perform most of their functionality between peers may still rely on a central server to coordinate. Also new software is just as likely to act up in an unanticipated situation when it’s run on a distributed network as it is when it’s run at a central site – this appears to be Skype’s problem; in fact, it’s a bit harder to back out a bad upgrade on a P2P service than a centralized one.