Life is Beautiful

Tokyo life-style vs. Fukushima life-style

Amazon's Cloud Reader vs. my CloudReaders (TM)

I have just found out that Amazon has released "Kindle Cloud Reader" this month (August 2011), which is a web-based e-Book application for iPad and other mobile devices. 

Amazon Kindle Cloud Reader

Release by amazon.com Inc. in August 2011

Web-based e-Book reader application for iPad and other devices

Amazon has acquired cloudreader.com in July 2011 (reference)

 

This is a little bit annoying to me, because I have an iPad application called "CloudReaders pdf,cbz,cbr", which is also an e-Book reader application. I have released this application when Apple came up with the first iPad (April 2010), and have more than three quarter million users world-wide. 

CloudReader pdf, cbz, cbr

Release by Satoshi Nakajima (me) in April 2010

Native e-Book reader application for IPad and iPhone

I have acquired cloudreaders.com in March 2010. 

Even though there is a difference (web-based vs. native), both apps are e-Book readers mostly targeting iPad. Names and logos are too similar, and I see a conflict. 

Since I am not a big fan of legal battle, I have decided to just post this entry for now to find out how other people perceive this event (not necessarily from legal point of view, but as a regular business practice as well).   

Facebook API on Google App Engine: JavaScript SDK vs. Python SDK

In my previous post, I talked about the app-id/secret-id pair you get when you register your app to Facebook. The documentation provided by Facebook, however, is a little bit unclear about the usage model of this pair under various SDKs. In this post, I am going to write some tips for App Engine developers who are building Facebook applications using this app-id/secret-id pair. 

In a nut shell, 

(1) It is possible to build your application only with JavaScript SDK. In this case, you don't need a secret key to access Facebook API. 

(2) This JavaScript-only above approach is fine for client-only application (such as a small widget application that presents Facebook news feed for the user), but not appropriate if you want to use Facebook API as the authentication/identification mechanism to access your database (such as per-user storage).

(3) The best approach is a hybrid application, which uses JavaScript API for log-in and various non-critical API calls (such as fetching the list of friends to present it to the user), but uses Python SDK for authentication and some critical API calls.

I built fruence.com with this hybrid architecture. Here are some code from it:

main.js (client-side):

    window.fbAsyncInit = function() {
        SNBinder.get('/api/appid', null, true, function(result, data) {
            FB.init({appId: result.appId, status: true, cookie: true,
                     xfbml: true});
            FB.Event.subscribe('auth.login', function(response) {
                main.init();
            });      
            FB.Event.subscribe('auth.logout', function(response) {
                main.login();
            });      
            main.init();
        });
    };

SNBinder.get is a wrapper of HTTP-GET and fetches the app-id by accessing /api/appid, then calls FB.init() to initialize the API. Notice that you don't need the secret-key in this case, and you don't want to (otherwise, it is no longer a secret).

When the user clicks the log-in button (provided by Facebook) and authorizes the access to the app (this UI will be provided by Facebook as well), the application receives the 'auth.login' event. 

Then (within the main.init()), the application accesses '/api/user' with a cookie attached by Facebook API, which will execute the following code on the server side.

gdispatch.route(lambda: ('/api/user', ApiUserHandler))
class ApiUserHandler(webapp.RequestHandler):
    @login_required
    def post(self):
        self.response.out.write('{"success":true, "result":%s}' % self.current_user.json)

The login-required decorator performs the server-side authentication. 

def login_required(original_func):
    def decorated_func(rh, **kw):
        if User.current_user(rh):
            return original_func(rh, **kw)
        else:
            return rh.response.out.write('{"success":false, "login_required":true}')
    return decorated_func

User.current_user is defined as follows.

class User(db.Model):
    ....
    @classmethod
    def current_user(cls, rh):
        if not hasattr(rh, "current_user"):
            rh.current_user = None
            (id, secret) = cls.facebook_ids(rh)
            cookie = facebook.get_user_from_cookie(rh.request.cookies, id, secret)
            if cookie:
                key_name = "fb:"+cookie["uid"]
                user = cls.get_by_id(key_name)
                if not user:
                    graph = facebook.GraphAPI(cookie["access_token"])
                    profile = graph.get_object("me")
                    user = cls(key_name=str(key_name),
                                name=profile["name"],
                                profile_url=profile["link"],
                                access_token=cookie["access_token"],
                                cached_logs=[])
                    user.put()
                else:
                    if user.access_token != cookie["access_token"]:
                        user.access_token = cookie["access_token"]
                        user.put()
                rh.current_user = user
        return rh.current_user

Notice that you need to give the secret-id to Facebook API (facebook.get_user_from_cookie) in this case, but your secret-id is secure because this code is on the server side. 

 

Debugging Facebook AppEngine application locally

There is a good discussion around "how to use Facebook log-in from Google App Engine", but there is one tricky issue dealing with Facebook's app-id/secret-key pair, which is generated for a specific URL.

If you hard-code the id/key pair for your applicatin's URL in your source code, you won't be able to debug it until you deploy. 

I've encountered this issue when I was developing fruence.com (groupware for Facebook users), and came up with a simple work around.

Here is the code (the actual app-id and secret-code are replaced by "{app-id}","{secret-code}" to hide them):

@memoize
def _facebook_ids(domain):
    dict = {
        "www.fruence.com": ("{app-id}","{secret-code}"),
        "un-url.appspot.com": ("{app-id}", "{secret-code}"),
        "staging.latest.un-url.appspot.com": ("{app-id}", "{secret-code}"),
        "localhost:8104": ("{app-id}", "{secret-code}"),
        "10.0.1.100:8104": ("{app-id}","{secret-code}"),
    }
    return dict[domain]

class User(db.Model):
    ...
    @classmethod
    def facebook_ids(cls, rh):
        url = rh.request.url
        (a, domain, b, c, d, e) = urlparse.urlparse(url)
        return _facebook_ids(domain)

    @classmethod
    def current_user(cls, rh):
        if not hasattr(rh, "current_user"):
            rh.current_user = None
            (id, secret) = cls.facebook_ids(rh)
            cookie = facebook.get_user_from_cookie(rh.request.cookies, id, secret)
            if cookie:
                ... 

Notice that I have generated five app-id/secret-key pairs, including staging environment and local environment (localhost as well as LAN), which allows me to debug my application in any environment.

To see this mechanism in action, access my app from both www.fruence.com and un-url.appspot.com. Each URL has an unique app-id/secret-key pair, but both are connected to the same version of the application. As a Facebook user, you need to "approve" each app separately because they are two different apps from Facebook's point of view (and cookies are associated with each domain), but my application recognize you correctly because the "current_user" is associated with facebook id. 

Notice that Facebook allows you to register local URLs like "http://localhost:8104". As far as I can tell, Facebook generates a unique app-id/secret-key pair for each registration, therefore, you don't need to worry about collision. 

Viva Google App Engine!

Google App Engine: Updating Your Model's Schema (schema-versioning)

Unlike relational databases, the bigtable is a schema-less database. Therefore, adding new properties is quite easy. You just need to add new property definition(s) like this: 

class Foo(db.Model):
    existingProperty = db.StringProperty()
    newProperty = db.StringProperty()

When you db.Model.get() one of existing entities (which are created before you define this new property), the newProperty will be automatically set to None (because the default value of db.StringProperty is None).

This is because the bigtable is schema-less database, and it does not require all entities to have the same set of properties. When you see existing entities using the Datastore Viewer, you see <missing> as the value of this new property. 

It's very easy, isn't it? 

There is one caveat, though. You can't query against those missing properties. The following query will NOT find those entities with <missing> property value.

query = Foo.all().filter('newProperty', None)

This is a little bit counter-intuitive because you actually see "None" as the property value when you db.Model.get() those entities. You need to understand that this "None" is not coming from the database, but simply coming from the default value of the property definition. 

Then, how can I find those entities with missing properties? This is a common questions among app-engine newbies, you see a lot of discussions around this topic, such as:

AppEngine: Query database for records with <missing> value

There is even an article about this topic written by one of Google engineers. 

Updating Your Model's Schema

This article is, however, a bit outdated (written in 2008 before the introduction of taskqueue) and not generalized. 

After creating quite a few appengine aplications, I came to the conclusion that it is much easier to use schema-versioning technique instead.

Here is the quick summary of this technique. 

1. Define the "version" property when you define a model object

class Foo(db.Model):
    VER = 1 
    version = db.IntegerProperty(default = VER)
    existingProperty = db.StringProperty()

2. Increment the version number when you change the schema

class Foo(db.Model):
    VER = 2 
    version = db.IntegerProperty(default = VER)
    existingProperty = db.StringProperty()
    newProperty = db.StringProperty()

3. Define a migration method, which typically looks like this (notice that we don't need to explicitly set the values of new properties as long as the default is appropriate - None in this case) 

class Foo(db.Model):
    ...
    @classmethod
    def migrate(cls):
        query = cls.all().filter("version <", cls.VER)
        items = query.fetch(1)
        if items:
            item = items[0]
            item.version = cls.VER
            item.put()
            deferred.defer(cls.migrate)

4. Execute this method from your own admin screen (via admin url)

gdispatch.route(lambda: ('/admin/api/migrate', AdminMigrateHandler, 'admin')
class AdminMigrateHandler(webapp.RequestHandler):
    def get(self):
        Foo.migrate()
        return self.responce.out.write('{"success:true"})

Once you run this migration code, the property values of existing entities will become None (or the default value you specified in the property definition), and you can query against those property values. 

 

Handing and Testing DeadlineExceededError

Last week was a quite exciting week for me because of various discussion around Google App Engine, such as "Goodbye Goole App Engine" and "Why we're really happy with AppEngine". I also have some opinions about it, but I much prefer to write code than participating in this kind of debate. 

Instead, I am going to share my solution to deal with one of those AppEngine specific issues - an occasional DeadlineExceededError. This solution may not apply to everybody, because I am just solving this issue for my own application - but I hope sharing this kind of idea helps the community as well as myself. 

To deal with this issue, I came up with a following decorator (yes, I am writing my code in Python, and decorator is my favorite feature of this language). 

@memoize
def catch_dee(ratio):
    def decorator(original_func):
        def decorated_func(rh, *args, **kwargs):
            try:
                if os.environ['SERVER_SOFTWARE'].startswith('Development/') and random.random() < ratio:
                    raise DeadlineExceededError
                return original_func(rh, *args, **kwargs)
            except DeadlineExceededError:
                user_agent = rh.request.headers['User-Agent']
                rh.response.set_status(500)
                return rh.response.out.write('{"success":false, "retry":true, "errors":["deadline_exceeded_error"]}')
        return decorated_func
    return decorator

And this is a piece of code from my application (sorry for additional decorators unrelated to this post -- all my code looks like this). 

gdispatch.route(lambda: ('/blob/delete', DeleteHandler))
class DeleteHandler(webapp.RequestHandler):
    @mysession.require_login
    @config.catch_dee(0.333)
    @gdispatch.kwargs
    def post(self, ids):
        ...

This decorator does two things.

(1) It catches the DeadlineExceededError exception, and reports this error in the format my application (iPhone application) understands.

(2) It randomly (33.3%) generates this exception when I run this application under SDK, which helps me to test my client-side of code, which performs some retries. 

I hope this post helps somebody. Feedback is welcome especially if you notice something I am missing. 

Cheer to Google App Engine developers!

Dear users of CloudReaders(TM)

One of our users has pointed our that the "support page" of CloudReaders(TM) is not in English, and it's hard to give me feedback. It was my mistake, and here is the place for all of CloudReaders(TM) to give me feedback, encouragements and feature requests (no refund-request please - CloudReaders(TM) is free!). 

Why HTML5 is a big threat to Adobe

Adobe made a couple of interesting announcements today.

• Adobe Unveils First Full Flash Player for Mobile Devices and PCs
• Adobe Labs: Applications for iPhone

Neither announcement is a surprise to us, but it is very important to notice the timing and the implications of these announcement. 

First of all, Adobe has been avoiding to publicly admit that Flash lite is useless in 99.9% of cases because Flash lite is so different from Flash for PC. 

The marketing team of Adobe has been smartly using the statement "Over 800 million (mobile) devices shipped with Flash" over and over again, trying to convince us that Flash is already de facto standard, ignoring the fact that most of Flash contents on web-site today are written for Flash 8 or 9, and Flash lite on those mobile devices are not able to display those contents. 

As far as I know, this is the first time Adobe publicly admitted that putting "full" Flash Player (Flash 10) on majority of smartphone is very important for Adobe to become the true de facto standard in the mobile industry. 

As all of us already know, the biggest challenge for Adobe is iPhone. Even though nobody besides a small number of executives in Adobe and Apple know the real reason, Apple won't (or can't) put Flash on iPhone and will remain this way for a very long time (if not forever). Along with the fact that the most of web traffic from mobile devices are coming from iPhone, this is making virtually impossible for Adobe to dominate the smartphone market. 

While iPhone itself is enough pain in the neck, here comes HTML5. It started with a small "hack" made by engineers in Apple who put <canvas> tag into Safari browser for Apple Dashboard. It was very awkward extension - mixing the "immediate" mode into the "retained" mode, but made it possible to create flash-like web applications inside HTML pages. Later, Google chose WebKit as the rendering engine for Chromium. Both Firefox and Opera started supporting <canvas> tag, and it became a part of proposed HTML5 standard. 

The <canvas> tag is just a small piece of HTML5. Many other features in HTML5, such as <video> tag, CSS animation, SVG, and WebSocket are threats to Flash. Once HTML5-compatible browsers became the norm, the technical advantage of Flash player will disappear. 

There is even a rumor that Apple is working on HTML5 authoring tool, which would directly compete with Adobe's Flash authoring tool. This makes sense. If I were inside Apple today, I would definitely propose such a product. 

The biggest hurdle for HTML5 is Internet Explorer, which still has a very large market share. Microsoft just made an announcement that they will support HTML5, but I think it takes years for them to catch up. In addition, supporting HTML5 is a double-edged sword for Microsoft. While pretending to support HTML5, Microsoft may even (incorrectly) think there is a chance to make Silverlight more relevant than Flash while Adobe is busy fighting with HTML5. 

If we just look at PC market, it probably takes at least three years for HTML5-compatible browsers to become the majority (unless something really drastic happens within a year). Adobe does not need to worry about this market too much. 

On the other hand, the smartphone market is very different. Mobile Safari is already the #1 browser in the market because of iPhone and its traffic (iPhone users are much more active than Blackberry users). We also know that a flood of Android-based smartphone will hit the market in later this year and 2010 - most of them will have a Webkit-based browser. This fact - Webkit is becoming the de facto standard of smartphone browsers, will accelerate the adaption of HTML5 by web developers in mobile mobile - way faster than PC market.

Let's pretend you are a web developer. If your client ask you to create smartphone version of their web-site, which looks great on iPhone and also works other smartphones such as Blackberry, Palm Pre and Andoroid phones, which technology should you use? The answer is obvious - HTML. If your client ask you to make it animated, interactive or multi-media rich, you'd probably choose one of those new features in HTML5. 

This is obviously a big threat to Adobe. Considering the fact that more and more people access web from their smartphones than from their PCs, this is a REALLY BIG threat. 

Sticker shock of iLike acquisition

One of Seattle start-ups, iLike, was acquired by MySpace for $20.5 million ($13.5 to buy the company, and $6 million to retain talents). While I am glad to see an exit for Hadi (we used to work together in Microsoft building Internet Explorer), this price tag bothers me a lot (I run Big Canvas Inc., which offers "Big Canvas PhotoShare" - a real-time photo sharing social networking service application to iPhone users).

According to "Why is iLike selling out to MySpace for $20 million?", iLike has 50 million registered users and 10 million active users. It means the value per registered user is only 40 cents.

I still remember that Microsoft has acquired Hotmail for $400 million when they have only 9 million registered users at the peak of the first Internet bubble - $44 per each user. 

Yes, we are in the middle of financial crisis, but it makes it very difficult to invest money into social networking business, if this is a fair market value today. iLike has raised $16 million from external investors, and they get only $13.5 millon - a negative return. 

When I presented our business at the business plan competition (where we won a "best consumer business" award) in April this year (2009), a lot of people, especially venture capitalists, suggested me to raise money -- millions of dollars just like iLike did -- and aggressively acquire users before competitors do. 

The price tag of iLike is yet another proof that those venture capitalists are wrong. Building web services became very cheap these days because of crowd computing services like Amazon's ec2. A couple of thousands dollars is enough to bootstrap the service, and venture capitalists should put money into those companies who can actually generate revenue (even from a small number of users) and clearly present the way to lead to profitability. Putting millions of dollars into a company too early, does not necessary help the business. 

The statement like "we don't need to worry about revenue today. We just need to acquire enough users, and sell this company to Google before we run out of money" by CEO is probably acceptable during the bubble, but is just plain wrong in this economy. 

A good entrepreneur should be able to say "The gross margin per active user is A cents today. If we are able to keep our fixed/overhead cost at B, we will turn profitable once we get C users. We typically need to spend $D to acquire each user. Therefore, we need to raise $E million as a working capital to get to a profitability." 

This is very much like the way people run restaurants and movie theaters, and I think the Internet business should be run just like that. 

What does it take to launch Apple Tablet?

Until very recently, I had a very hard time believing that Apple would release the Tablet version of Mac (or iPhone) and have never spend time thinking about it. Tablet PC market is too small for Apple to go after, and eBook market is far from attractive yet.

The photo contest by Gizmodo, however, forced me to think about it further. I put myself into shoes of somebody who is in charge of this device, directly reporting to Jobs. What does it take to convince Jobs to approve the launch of this device?

I came up with one possible answer, which the picture above represents. Instead of going after traditional eBook market, I think Apple should directly go after schools - just like Apple did it with iTunes store and iPhone. Here is the pitch.

"It is great to see you guys to take advantage of technology, offering online classes and registrations and networking entire campus with WiFi, but there is one thing all the schools are a decade behind - text books. It does not make sense to waste all the trees, spend extra cost in printing and distributing, and forcing your students to carry tens of pounds of textbooks in their backpacks. I think it's time to switch. Just like we've eliminated CDs with iPod, we have created a perfect device to eliminate text books.

(And show the device, and demonstrate its various capabilities)

We have already talked to all major publishers, and they have agreed to offer all their text books in the new eBook format we've defined. In order to protect their business, we have created a new DRM mechanism, which makes it impossible to illegally copy text books. We also added a mechanism to expire books after a specified period, so that publisher has a full control over the business model.

It is going to cost $600 for each student to buy this device, but it allows them to "rent" text books at half the price of paper-based text books. This cost saving alone justifies this up-front cost, but it also allow school to offer online classes with video chat capabilities and gives a great flexibilities in the type of materials you offer.

Do you want to join thousands of other schools in the united states to join this exciting opportunities?" 

I think this level of effort (and involvement of publishers and schools as the result) is absolutely necessary to successfully launch the Apple tablet. I think it's time to take advantage of "reality distortion field" by Mr. Jobs.