Fractals of Change

Fun With Google Maps

If you have a list of locations, eventually you'll want to see them on a map – count on it.

An organization I work with gathered addresses and other useful information relevant to their mission with a web form and neatly downloaded all the information into a Google Docs spreadsheet. Eventually they decided they wanted to see it on a map. Turns out they can do that rather easily – no programming involved – by using one of the paid versions of Google Earth.

All spreadsheets can save their content as a .csv file (cells in a row are separated by commas). The paid versions of Google Earth have an import function for CSV files, which asks you which columns contain which address elements and then puts a dot on its map for each row. It works whether you have each element of the address in a separate column or have the address elements all run together as you'd enter them into a query for Google Maps. Non-address data from the spreadsheet can be displayed in the balloons for the entries when they're clicked on. Google Earth Plus is $20/year and can import up to 100 addresses at a time; Google Earth Pro is $400/year but will import as much as your machine has stomach for. Once the addresses have been loaded by a paid version of Google Earth, they can be saved as a KML file which even the free version of Google Earth can open.

There are a few drawbacks to this approach:

1. If the addresses are entered incorrectly initially in the form that gathered them, they don't get coded so they don't appear on the map. "Incorrect" is whatever the map provider Google is using says it is so some addresses are bound to get rejected no matter what.
2. You need to pay for one copy of Google Earth to create the KML files.
3. Whoever is going to view the map with the points on it needs to have Google Earth (free version is fine). However, there are web-based viewers for KML available. One is here; only hitch is I haven't gotten it to work and don't have time to debug right now. But, if this works, it gives you an easy way to make your location data available on the web.
4. Address-based geocoding is only available for certain countries.

If you can do it, it's better to gather your data on a map so the user has a chance to verify the location and correct any errors or negotiate a compromise with what the map provider expects as far as street names are concerned. That's what we did with Vermont Telecommunications Authority's "who has what broadband where" map. But collecting on a map takes some programming and may not be worth it for your project. With limitations, you can gather data on a form and make it visible on a map – no programming required.

JavaScript or PHP – Why It Sometimes Matters

It's all about whose computing power gets used – yours the application provider or yours the end user. If you're the user, you probably don't care since you are waiting for your browser to show you something in either case and you already paid for your computer. If you're the application developer, you may care a lot because you have to pay for server processing cycles one way or the other but you don't have to pay for user machine cycles.

If you are not involved in providing web applications in any way, you may want to bail out of the rest of this post at the end of this paragraph. Before you go an interesting factoid is that PHP stands for "PHP: Hypertext Preprocessor" – the acronym is recursive. Of course, if you're not a nerd you may not think this is interesting so I'll give you one more factoid. You may have noticed that many web addresses end with ".php". These are web pages that are created on the fly on servers using PHP. Pages which are more static OR are made dynamic on the user machine often end with ".htm" or ".html". Now you know.

OK; only us nerds are left now. You all know that a page which is pure HTML is pretty static; appropriate for lots of things but pretty static. But you also know that you can add some JavaScript and have the page become interactive; even better, you can use AJAX (Asynchronous JavaScript and XML) and load all kinds of data into an existing page to create an interesting and dynamic application. This data doesn't even have to be XML.

Pretty much anything you can do with JavaScript on a user's computer you can also do with PHP on your own server. The two languages are similar so the complexity's about the same either way. One huge exception is that you can't do anything with PHP that requires lightening fast interaction like reacting to a mouse-over, a click, or a key-press instantaneously. All PHP interactions gotta go from the user's computer to the host and back and that takes noticeable time.

One thing that you can do in PHP on your server that you can't do in JavaScript is load data or XML from domains other than the one the current page or iFrame was served from. That can be a significant limitation. You also can't keep your code secret if it's JavaScript; it's there in source form on every user's computer. You may want to keep it secret either because it's where your value comes from OR because it would be a security risk to you or the user to expose the code.

Many APIs like the Facebook API or the Amazon S3 API assume that you have a server at your disposal. I did manage to use S3 and maintain necessary security with only a client application and no server but had to jump through hoops to do it (see Amazon S3 – Backstory for Nerds - Part 1 and Amazon S3 – Backstory for Nerds – Part 2). 

In many ways it's easier to test and debug using PHP than JavaScript. The problem with JavaScript is that different browsers treat it differently. You sometimes have to ask in the code what browser you're supporting. You have to do extensive cross-browser testing. I spent (wasted) most of today on a problem caused by the fact that Firefox fills in the "Content-Type" header in an HTTPRequest and IE doesn't. The PHP you run on your server is going to execute the same way no matter what browser the user has – of course, the resultant HTML may not image the same way but that's a problem no matter how you do you development.

You can mix JavaScript and PHP so you can use each to get around the limitations of the other.

That gets us back to whose CPU cycles do you want to burn. If you're planning to have a blockbuster virally-spreading world-changing application (which, of course, you aren't going to charge for) then you don't want to have to buy a huge server farm. You can outsource to Amazon EC2 or perhaps use the Google apps server; but millions of users are still gonna be expensive to support. That'll often be a good reason to use JavaScript and the user's cycles rather than PHP and cycles you pay for. The user isn't hurt by this because she's already waiting for her machine to do something and just wants it done fast – it may, in fact, happen faster with local JavaScript execution than if an interaction with the server is required.

The P2P services, including Skype and BitTorrent and Napster, all rely on user resources – both bandwidth and computing power. My guess is that many web services running in browsers will be successful because scalability is instant and "free" when each user's machine provide the resources to handle most of the incremental load that user represents and when the bandwidth for communication between application and server is used sparingly. That's a big reason for doing as much in JavaScript as you can.

Vista Misery and Mysteries

Wasn't planning to install Vista on my new Toughbook CF-30 because it's been hard to deal with on Mary's new HP; but I did and now have both some misery and an Internet Explorer protected mode mystery which I'd be glad for nerd help with.

I liked the fact that the new Toughbook came with XP installed plus recover DVDs for both Vista Business and XP. That meant, I thought, that I could give Vista a while to stabilize and then install it. Unlike Mary's preinstalled Vista Home, I knew I could even uninstall Vista and reinstall XP since I have the business edition and Panasonic supplied me with both recover DVDs including all the crucial drivers needed to go back and forth.

But then the fine print: you have to wipe your hard drive of all content to install Vista. Do I want to spend weeks getting things the way I want them on a new machine, run a couple of months with XP, then start all over again with Vista and a "clean" machine? No, I decided, since I have nothing of mine on the machine now, this is the time to install Vista. Then I'll get the Vista version of everything.

Right now I'm regretting the decision. And looking for nerd help with a mystery.

Vista is running and isn't noticeably slow on my new machine; it hasn't crashed. That's the good news.

But Vista seems determined to protect me from myself even though I run with administrator privileges.

At first I couldn't get any ActiveX extensions to install. The yellow bar above the browser window which usually warns me that I've clicked on something which wants to install an ActiveX extension now didn't give me the options of installing; it just told me that my security settings didn't permit ActiveX extensions to be installed; this despite the fact that I'd deliberately clicked an option to prompt for permission before installing an extension. Couldn't even get Microsoft's own software verification extension to install to get the latest fixes to Office 2007.

Turning off "Protected Mode" in the browser let extensions install (but without a warning which I don't like either). Then, while trying to figure out why there were no time-wasting games like minesweeper around, I discovered (by Googling, of course) that you can go through Control Panel/Programs/Programs and Features to "Turn Windows feature on and off". This not only lets you turn games on; it also lets you turn on the ActiveX installer service. I've verified that games instantly appeared; haven't stumbled across an uninstalled ActiveX component since so not sure this is working properly.

But here's the mystery:

Web pages that used to work fine including basic Facebook pages now SOMETIMES break because, according to IE, it can't load a DLL (doesn't say which DLL). This never happens when Protected Mode is turned off. It doesn't get cured by a reload but sometimes the same page WILL load without an error much later even when I know the HTML of the page hasn't changed. It never happens in Firefox but Firefox doesn't have a Protected Mode. If it happened in Firefox, it'd be easier to debug because of Firebug. It doesn't seem to happen in Protected Mode on Mary's machine because some of these are pages she goes to often and she hasn't complained (but she's running Vista Home). I can't run a parallel test or compare all the settings on our machines because she and her machine are traveling.

It's not a solution for me to just run Firefox or run in unprotected mode because, as a developer, I need to know why pages sometimes break. If they break for me, they'll break for other people as well.

Haven't been able to Google my way to a solution or find one on Microsoft's site.

Ideas anyone?

Why Old Computers Get Slow

My Toughbook CF-29, bought back in 2004 when I left the corporate world and expected to spend many days in physically tough places, has been getting slow. Mouse clicks and resulting actions are increasingly far apart. Windows open blank and don't fill in for eons. Programs go into non-responding status; sometimes they recover; sometimes they don't. Rebooting temporarily speeds things up but it takes longer and longer to shut down and restart.

Why? You ask. The speed of light hasn't changed. The circuits can't be slower than on the day you bought the thing. Why are old computers slow computers.

A small part of the problem is barnacles. Most of the problem is that programs are written for the average two year old machine and its capabilities. The old computer bogs down under limits of memory and processing power when asked to do tasks designed for its more modern successors.

The barnacles are all the stuff you installed over the years and probably aren't even using anymore. Inactive programs don't do any harm except take up disk space but some of what you installed, printer and communication programs and drivers for example, have a small part which loads itself at startup and absorbs resources all the time the machine is running. With some trouble (and some tuneup utilities I've never tried), you can find these and turn them off. If you don't, they absorb more and more memory and processing power. There is some "printer subsystem" on my old Toughbook which reports its own failure every couple of hours and asks permission to tell Microsoft about the problem. Microsoft doesn't offer any solutions when told and the failure of this subsystem doesn't seem to affect my printing; but it's there somewhere.

When I bought the old Toughbook I installed as much RAM (memory) as it would take: 512 megabytes; seemed huge to a guy who wrote programs for the 128 kilobyte Mac (the first Mac) and 16kb TRS-80 model 1. (a megabyte is a thousand kilobytes). "My" first computer was an IBM 7090 the size of a basketball court with the equivalent of about 156kB. It cost millions of dollars. But I've digressed to my age… back to the old computer.

The CF-29 has a single processor which runs (walks by today's standards) at 1.3GHz. It has a 40 gigabyte hard drive which is almost full.

When I first used the machine I had one window open on the Internet most of the time. Now I have at least six tabs open in Explorer – because I can. I've gotten a bigger screen and, when at my desk, I have windows spread across two screens for easier cross reference and cutting and pasting or perhaps because I have code in one window and I'm watching how it runs in another window. Whatever, I've got more stuff running at one time.

There isn't enough memory on my machine for all this stuff to be running at once so Windows puts some of it in "virtual memory" – really on disk from which it must be reloaded before use. Often it is clear that my computer is "thrashing" – the technical term for swapping pages in and out of memory so often that it can't do any real work. Picture a hundred people trying to work on something in a room with only space for ten; every time someone else is needed for the task at hand; someone has push his way out so someone else can push her way in. But the guy who went out hadn't finished so he reenters and pushes someone else out. Pretty soon it's all elbows and no productivity. That's my computer.

Programs like Google Earth and Sketchup assume that my computer can draw lightening fast. It can't at the resolutions they're feeding it. Turning the scroll wheel for zooming a graphic image results in a jerky ascent or descent like a rocket with misfiring engines.

Web pages now contain incredibly elaborate Javascript – easy to write; cool to use; but a big burden on an old CPU to interpret and execute. Then there's Flash. And videos embedded in everything. You can almost hear that lonely old CPU groan.

So that's why it's time for a new computer. My new Toughbook CF-30 (I still plan to go to all those tough places) has 4 gigabytes (4 billion bytes) of RAM (although Windows can only see 3.3 gig). It has an 80 gig hard drive. It has TWO processors each screaming along at 1.6GHz. It has more auxiliary graphics processors to take the load off the main processor. It has an embedded EV-DO radio and GPS to reduce the clutter in my gadget bag.

And it has Vista… So far not at all to my liking.

Testing Office 2007

Meant to start this series on moving to a new computer (gulp) and upgrading to Office 2007 (gulp, gulp) and Vista Pro (GULP!) in an orderly way with a post on how computers get old and slow. But when I told the new Microsoft Word that comes with Office 2007 that I wanted to create a new document, it asked me if I really meant a document or a blog post. Well, this is meant to be a blog post so…

Word asked me, reasonably, who my blog provider is – it's TypePad – and then for my TypePad ID and password; it warned me that these would be transmitted in the clear. Since I have several test blogs on TypePad as well as Fractals of Change, Word asked which one I wanted. Maybe I should have started with one of the test blogs but what the hell… let's give it a try.

If you read this post, it is possible to go directly from Word to a post without cutting and pasting. I'll let you know what steps intervene after I click the Publish icon.

Reset#1: After I clicked Publish, I noticed two typos and that there is an Insert Category button. Clicked cancel. Word became unresponsive for a while according to Vista but did recover and say that the blog service provider wasn't responding.

Clicked Insert Category; got back my category list from Typepad (nice) but can only choose one category even though TypePad makes it possible to choose several (not nice) but will continue the experiment.

Reset#2: OK. It did publish; in fact, so did my previous attempt. There is an Open Existing option so have done that and now am presumably editing and able to repost. Tried to use this option to delete the earlier post but can't do that (no big deal). Will try reposting by clicking Publish again. Then will go directly into TypePad to add more categories and Technorati Tags (too bad but not critical).

Just saw an option for handling pictures (imbedded, I hope) by uploading them to the blog provider (default) or somewhere else (which the documentation says can be anything with a public URL including Flickr). Will insert a picture below and leave the default to have it hosted at TypePad. Hmm.. scaling only gave a choice of 1% (maybe for a thumbnail?) but could simply specify a new height and the width adjusted to keep the aspect ratio. We'll see what happens.

 

Auditioning for New Roles

Below I am practicing to supplant pigs and toasters as a screensaver.

In case that doesn't work, I could always be a demo dolly (what's the masculine of that?). I'll buy a BUG like the one I'm demoing below for anyone who can correctly identify the polo shirt I'm wearing in this picture before the end of April.

If all else fails, it's back to email.

 

All pictures are from Kinnernet 2008. Great fun as usual.

Third Life: Social Networking Breakfast with Jeff Pulver

“It’s like Facebook only it’s not online,” someone at Jeff Pulver’s social networking breakfast in Tel Aviv explained to somebody else. “There’s tagging and everything but it’s not virtual.” That’s a pretty accurate description of these real world events which build on not only connections but also techniques learned online.

Look at this picture of Jeff Pulver, himself:

The elements’ll be familiar to you if you use Facebook. The stickers on Jeff’s right side are his wall, meant for other people to write on. One way the ice gets broken between strangers is that they put tags on each others walls. You can see a yellow sticky someone put on Jeff and there are some little white ones as well.

The sticker on Jeff’s right side has his name and what is meant to be a conversation starting line he wrote about himself. His says “I take having fun seriously.” He does. Below that Jeff tagged himself, accurately, as a geek (unlike me, I’m a nerd).

The printed tag in the middle of Jeff’s chest is from the Marker COM.Vention in which this particular breakfast was imbedded. Craig Newmark of Craigslist and superblogger Robert Scoble were both at the COM.Vention, at Kinnernet which preceded it, and at the breakfast. Most of Jeff’s breakfasts are standalone; most guests are not famous. If you’d like to attend a breakfast near you, best things to do are either read Jeff’s blog or follow him on twitter. I’m trying to convince him to have one in Burlington, Vermont.

Just because the social networking breakfasts are live doesn’t mean technology goes away. At this one, and I suspect most others, there a huge variety of high tech cameras being used almost recursively. Note people taking pictures of each other and of others taking pictures of others and me (not seen) taking this picture and possibly someone taking a picture of me.

If you’d like to hear more about the social networking toolkit, which Jeff says he’s gonna patent, watch the inventor explain in the video below:

Introducing the BUG at Kinnernet

There are about 250 nerds gathered here at Yossi Vardi’s Kinnernet camp on the shore of the Sea of Galilee; there couldn’t be a better place to introduce the BUG to the Israeli development community. By the grace of the demo gods, the BUG worked flawlessly and I didn’t make any mistakes I couldn’t recover from and I avoided a shower (more on that below).

The BUG is going to revolutionize consumer electronics; it’s going to enable a huge wave of innovation in consumer devices just as the PC unleashed software innovation I’m an investor in BUGLabs, BTW; not only because I believe in the product but also because, once I heard about the BUG, I realized it’s something I always wanted.

See the BUG lets software hackers who can’t solder become hardware hackers; now we can invent and build new gadgets as easily as we’ve been able to program personal computers. I’m building the world’s best anchor alarm so sailors  can sleep at night in an anchorage knowing they’ll be awakened if the anchor drags or even might drag. Every cruising sailor’ll have to have one.

So what’s a BUG?

It’s a 128MB Linux box about the size of a pack and a half of cigarettes. What makes it special are four ports onto which modules can be snapped – that’s where the hardware building capability comes from. The four modules available today are an LCD, a GPS, an accelerometer/motion detector, and a camera; many more are coming soon. Until BUG, you couldn’t get a consumer-usable box put together with the components you need for your application unless you were willing to order 100,000 of them or so – that’s limited innovation in consumer electronics pretty much to companies which can take gambles of this size and have the marketing muscle to make at least some of them work.

The pieces I need to build my gadget cost me $525 – early adopter price.  Presumably once am buying in some quantity, I can get these kits for even less per unit, load my application on to them, and sell to my sleepless sailor market at a price point below $1000 and pay for ads and marketing and still make money.

My plan had been to demo my own app at Kinnernet – that would be cool and re-earn my nerd credentials before my younger peers. The BUG software developer kit (SDK), which I’m doing a workshop on at Kinnernet, is easy to use (if you know Java) and even features a virtual BUG so you can test without hardware. Both BUGlabs and the growing developer community are helpful. Everything is open source and interface specs published so anyone can develop software or hardware for the BUG. There are lots of sample apps and my Hello World app was running in just a day or so (I had to learn Java).

But Baby Jack got born early and got priority (good excuse, huh?) so my app not ready. Quickly BUG coding genius John Connolly put together an impressive demo using the camera, LD and accelerometer – shake’n’show. It was fun; it made the point; and, above all, it worked.

On stage here is a little dicey. Thirty seconds before your time is up, there’s an ominous roll of thunder; at fifteen seconds a shower above the presenter’s head starts to mist; at zero time there’s a deluge. I might have been willing to take a shower to make a point but couldn’t afford to get the BUG wet. Just stepped forward towards the audience in time to avoid the flood and still deliver a closing line.

I’ve been doing small demos since. Most frequent reaction is “where do I get one?” (remember, this is still a nerd device; it needs to be programmed to do anything; and this is a very skillfully nerdy group). The fun thing is watching people immediately come up with cool gadgets that they’ll now be able to build. Next week I’ll get back to work on mine.

Findability in the (un)Social Directory

If you don’t want to be found, you should have the right to remain a hermit. I can’t bring myself to pay Verizon several dollars a month to take me OUT of its 411 and white pages but I don’t want to be there, even though – or especially because – the number they list is attached to the alarm system and neither rings in the house nor has voice mail.

Even in the relative privacy of social networks, most of us do leave ways to find us open. The key distinction is that finding us doesn’t mean reaching us. For example, if you’ve registered for Facebook, even people who aren’t your certified friends can find you. They can ask permission to be your friends; they can try to communicate with you. But you don’t have to respond to these requests or accept communication from them.

Similarly someone can find you on Skype if they make a few lucky guesses or know your email address – but they can’t see your online status or complete a call to you without your permission. Some people may want to remain truly anonymous, visible only to those to whom they have explicitly given contact permission. Most people, I suspect, want to be findable enough so that a stranger or a long lost friend can ask permission to reach them - something I’d very much welcome your comments on.

Assuming that you do want to be findable, then it makes little sense that a seeker have to wander through multiple walled gardens looking for you. Are you on Skype? Facebook? MySpace? Are you a friend of a friend?

The (un)social directory we are working on at FWD will span social networks in an attempt to find you using only that information that you have already made public through those social networks. If someone does find you through us, all he will be able to do is ask your permission to contact you. You can, of course, say no or ignore the request. You can also tell us not to forward anymore requests from this individual or from anyone. If she is someone you want to hear from, you can specify how she is allowed to reach you – text, voicemail, voice on your landline or cellphone etc.

As we (with your permission) know more about your social graph, we’ll also be able to tell you how many contacts you and this person have in common across a variety of social networks and who those mutual contacts are.

At the moment, this is vaporware not yet implemented in software. Consider this post a request for comments.

This thinking about findability owes much to old friend and one time competitor Jacob Ner-David, cofounder of Delta Three and now, among many other things, the Managing Partner of Jerusalem Capital.

The (un)Social Directory

I want to be in an (un)social directory. I want to be accessible to some people, want to be findable by most people, but want to keep complete control of who communicates with me by what method.

The (un)social directory is the inverse of the kind of directories we’ve been living with throughout our lives; that makes it hard to think about initially.

A traditional directory is a collection of information about other people which you own - your Outlook directory, your collection of business cards, a phonebook, the phonebook in your mobile phone. Each entry gives you one or more ways to reach those other people. The information is static. If it changes, you have to both know about the change and take the time to enter the change or the directory will be out of date.

When you give people information about yourself to put in their directories, you are implicitly granting them permission to access you with that information. Once someone knows your mobile number he knows it; you can’t revoke that. You can refuse to answer when you get a caller ID you aren’t interested in but you have to change the number to revoke the privilege of calling it. On the other hand, if you do make a change, you have to find a way to notify all the people whom you do want to be able to reach you that the information changed.

Prepare to invert.

You maintain one copy of your master contact information in the (un)social directory – all of the possible ways to contact you. Everyone else has a similar master contact page which is visible only to her. When two people meet and exchange (un)social contact info, what they are actually doing is exchanging permissions but NOT contact information. Permissions are always revocable. This needs an example.

We meet for the first time at a tradeshow. You decide that you want to allow me email access to you because you may want to buy what I’m selling. I want to allow you both email and phone access as well as IM because I’m very eager for you to buy what I have to sell. We both do something online or on our mobile phones (UI TBD) to grant each other these permissions. Note that we do NOT exchange actual email addresses, IM handles, or mobile numbers.

We now each have two entries in our personal directory. The contact entry I use to reach you has nothing but permissions in it and the address of your contact page (which I can’t see but can get connected to you through). The other entry is the permissions I granted you which are to a subset of the possible ways to reach me. I can enhance, change or revoke these at any time – like if you don’t buy anything but keep calling to tell me about your golf game.

If I change phone numbers or email addresses, it makes no difference to you because you didn’t know what they were in the first place. As long as the address of my contact page remains the same, you’ll be able to get to me. And vice versa.

When you want to call me, you click on my name (whatever name you gave me) and a connection is made through my contact page. There’s some smarts in the directory application so you get the best permitted connection given the media you want to use – real time voice in this case – and the media I’ve permitted you and am available through at the minute. Maybe you’d like to leave voice mail if you can’t get me in real time; maybe you want to IM or email me. You may want me to call you back and can give me temporary permission to do that (remember, you didn’t give me general permission to call). But, since you are leaving just a permission and not a call back number, you don’t have to worry that I’ll pester you forever just because you asked me to call once.

How we get to the nirvana of the (un)social directory is coming up. So is findability.

Posted on my professional interest in this at My New Gig.

Fellow FWD-er CEO Daniel Berninger has more to say about directories here.

Aswath of EnThinnai shares the vision but differs on the business path to implementation.

Amazon S3 – Backstory for Nerds – Part 2

Note to non-nerds – back to English tomorrow; you may want to skip this one.

Here’s the problem: My broadbandwiki browser application does two things for users: it shows them what broadband access people around them are using and it enables them to add themselves to the map so that their information can be helpful to others. Clearly the data the users provide needs to be stored. However, only some of this data is meant to be visible to other users. Street addresses, for example, are used to generate latitude and longitude but are not meant to be publicly accessible even though they may later be used for engineering studies.

Controlling who has access to what data would be relatively straightforward if there were an application running on a server somewhere between the app running in the browser and the Amazon S3 servers where the data’s stored – but there is no such server application in this case. Nevertheless, the needed security CAN be maintained using available S3 tools.

Data on S3 is stored in buckets. The model is simple; you put objects (lumps of data) in buckets. Each object has exactly one key by which it can be retrieved from the bucket. Objects can also have reasonably elaborate headers.

You, the owner of the S3 buckets, get to decide who can read or write (or change the permissions) on a bucket by bucket basis. Each object also has an object control list (ACL) associated with it which it does NOT inherit from its bucket; the ACL is set when the object’s created and may be changed later by those who have permission to change it.

So I set up a bucket called broadbandwiki for my beta; anyone can read this bucket but only I can write to it or change its permissions. Reading the bucket, however, doesn’t mean reading the contents of the objects in the bucket (they have their own permissions). In practice, the right to read the bucket means the right to read the index of objects stored in the bucket which includes keys and creation dates for objects but does NOT include headers or payload.

Offline I used my secret key (which should never be transmitted) to generate very specific permissions to write a very specific type of object to this specific bucket. The permission restricts the access policy of created objects so it can’t be used either to create objects which are user accessible or which I can’t access. These encoded permissions signed with my secret key are safe to imbed in a browser page and to transmit.

The keys of the objects created and stored by the browser app include all the data which is supposed to be visible to ordinary users. The payload of the objects contain the data meant to be protected. When a user launches the browser, it reads the index of the broadbandwiki bucket (which anyone can read) and uses that to put pins on the map representing data supplied by prior users. A beneficial side effect is that there are usually a thousand index records returned per read of the directory. Much cheaper to get the data most users need out of the index than read each of a thousand records (because Amazon charges for GETs).

When users put their own pin in the map, the presigned permission is used to make sure this is only the kind of object we want here and that other users will not have access to this data.

Problem pretty much solved. The protected data – actually all the data – is retrieved by an administrative utility – whose user must know and input the secret key – but which never transmits that key. The administrative program converts the data to an XML file before downloading so it can be fed into nearly any analysis tool. Excel works fine.

An invaluable tool for anyone doing S3 development is a free Firefox addin called S3 Firefox Organizer. I did donate to the tip jar, though.

More on the broadbandwiki project is here.

Nerd tips on using AJAX GET, PUT, and DELETE with S3 are here.

Economics of S3 and possible implications are here.

And the code for this browser app is here.

Amazon S3 – Backstory for Nerds - Part 1

It’s practical to use Amazon’s Simple Storage Service (S3) as the backend for a web application WITHOUT any intervening application server. I have the code to prove it and I’m sharing it. If you’re a nerd, read on otherwise you may want to skip this post

S3 clearly was designed to for use with an application server between browser and desktop applications and the actual data store. To Amazon’s credit, that server doesn’t have to be Amazon’s Elastic Computing Cloud (although pricing somewhat favors this) . The Getting Started Guide, for example, gives examples only in PHP, C#, Java, Perl, Ruby and Python because they assume that you’ll be using one of those languages on a server. The implication is that you can’t do it in JavaScript/AJAX – but you can.

The Developers Guide also gives short shrift to JavaScript. It does explain that you can let “your users” upload certain data directly (bypassing your server) through HTML forms although it assumes that access keys will be delivered to the browser by a server just prior to upload for all non-trivial cases. There is a also a description of a way to prepackage a very specific authorized retrieval request so that the browser can get data directly from S3 – but, again, the assumption is that the browser first asks “mother may I?” and receives a specific and time-limited key from the server before making an XMLHttpRequest GET.

My application – broadbandwiki – lets users put pins on a map in order to indicate what type of Internet access they have available at their locations. S3 is the place that the locations, access types, and access providers are remembered. Google’s examples assume that, if you want to save the locations you gather with an application like this, you’ll have a LAMP stack operating on a server somewhere. They give good examples of the PHP and mySQL you’d use to do this.

My problems were both that I don’t have a server on which to run said LAMP stack and I would have to learn a lot about Linux, Apache, mySQL, and PHP. The first problem could easily be solved by any of a number of hosting services including EC2; but all the learning seemed a lot to wade into when I just wanted to store a little data and have access to it later. Probably ended up being harder to get S3 to do what I wanted than if I’d just swallowed my medicine and learned the server-side stuff but I got stubborn.

The good news is that, once you know the tricks, this is easy.

Here’s the main thing you need to know even though the documentation doesn’t say it. XMLHttpRequest works for not only GET but also PUT and DELETE (That’s all I’ve tested). I didn’t need PUT and DELETE in the user application but I did need them in the batch-like administrator application. Note that the administrator application does all sorts of privileged stuff but it can still be run safely in a browser because the Amazon Secret ID is supplied by the administrator at run time. It is used to develop a time-limited hash key on the administrator’s computer but the secret key itself is NEVER transmitted.

Note that XMLHttpRequest is usually limited to making requests from the same domain as the web page was served from and this is no exception. The solution is to host the web page on your Amazon S3 account (it’s got to live somewhere, anyway) so that there is no cross-domain violation. However, the page hosted on Amazon can be in a frame of a page hosted elsewhere if that’s important to the look and feel of what you’re doing.

Sample code for the administrator app (which borrows liberally from every other bit of sample code I could find) is here. It won’t run for you until you have an Amazon S3 account and fix the places where I bound the name of my own bucket on S3 in the code but it is meant to provide an example of how to use XMLHttpRequests with S3. It is NOT a sample of clean, well-documented  code, however; it’s a work in progress. It also needs more functionality and more error checking so I will post later versions as I clean it up.

Next nerd post on this subject will be on how to let browser users write data, some of which is meant to be seen by other browser users and some of which isn’t – all without a server of your own.

Once I have done some cleaning up and packaging, I’ll also post the user application although all you good hackers know you can already get it just by downloading it from its web address or viewing source in a browser.

Phew… Made the Deadline

Faithful readers know that I promised Mary to have an app ready for her to beta at town meeting tomorrow to collect information on who has what kind of broadband in Stowe and from whom they get it.

I used the Google maps API to build this. It runs as an application on my laptop rather than as a window in a browser because it’s not until my next project that I figure out what many of you already know – how to configure a server to receive all this information. The survey results are saved as an XML file which can go into Excel and lots of other places.

Took me at least forty hours of work to do this simple app. But that’s not a reflection on the Google tools; it just shows how much I had to learn about many things that working programmers already know. Reversing the usual, the first 10% took 80% of the time; the last 90% was a lot easier because I increasingly knew what I was doing.

I could do a new task with these tools of similar complexity in three or four hours.

BTW, the Google sample code was very helpful and got incorporated wherever I could.

At the last minute it looked like we might not be able to use the app. Town Meeting (oh yeah, and primary day, too) are at the High School. No way at the last minute to get my Internet access through the school. Oh, oh. We went up to the school to test today. Despite the fact that cellphone coverage is marginal there (ask the kids), Verizon EVDO with my new battery-boosted USB modem AND the antenna I bought managed to see a bar or two and that was enough for this to work as long as I don’t put the maps in bandwidth-hungry satellite map view.

Wonder if it’ll crash on its first real outing. Did put lots of care into making sure data will not be lost in that case; I’d be in lots of trouble if that happened.

Great Time to be a Nerd

My friend Jeff Pulver was just telling me that it’s a great time be a nerd. Right on cue I walked past the poster above which I think advertises a new TV show.

On the same theme, Mary asked the other day what all us nerds would’ve done for a living if there weren’t computers or other electronic devices for us to play with. It’s a very frightening thought. I thought I’d be a writer like my father but met my first computer, an IBM 7090 mainframe, in the computer lab at college when I was nineteen (1962) and found this was a much better way to make a living. Didn’t get around to writing fiction again until a couple of years ago when I wrote hackoff.com: an historic murder mystery set in the Internet bubble and rubble – not surprisingly, nerd Dom Montain is a major character of that book.

So what would we have done? Took some thought. Not many of would have been gymnasts, super warriors, craftsman (takes fine motor skills), or very good farmers.

We would’ve counted, added, subtracted, multiplied and divided. Before there were computers, there were lots of numbers that had to be crunched by hand. When you go over the George Washington Bridge or ascend the Empire State Building, remember that the only mathematical tools for all the engineering calculations were adding machines and slide rules (easy to use for a nerd); presumably the calculations were done at least twice.

All the accounts of everything had to be done by hand. All monthly statements manually totaled (at best on an adding machine).

So we would’ve been accountants or engineers or employed by such.

It’s much more fun to teach computers to do all that work (what programming is all about); we never have to do the same thing twice – at least until we reimplement it for a new computer or operating system. And we get great toys to play with.

Jeff’s right.

Nerding Again: It's Hard

I’m back writing code. Despite the fact that I first programmed 45 years ago and used to be a top gun (if I say so myself), it’s hard getting re-started after a 17 year hiatus in which I’ve coded nothing more complex than some VBA in Excel macros an