Gesture recognition in smartphones

This piece from the Aberdeen Group shows accelerometers and gyroscopes becoming universal in smartphones by 2018.

Accelerometers were exotic in smartphones when the first iPhone came out – used mainly for sensing the orientation of the phone for displaying portrait or landscape mode. Then came the idea of using them for dead-reckoning-assist in location-sensing. iPhones have always had accelerometers; since all current smartphones are basically copies of the original iPhone, it is actually odd that some smartphones lack accelerometers.

Predictably, when supplied with a hardware feature, the app developer community came up with a ton of creative uses for the accelerometer: magic tricks, pedometers, air-mice, and even user authentication based on waving the phone around.

Not all sensor technologies are so fertile. For example the proximity sensor is still pretty much only used to dim the screen and disable the touch sensing when you hold the phone to your ear or put it in your pocket.

So what about the user-facing camera? Is it a one-trick pony like the proximity sensor, or a springboard to innovation like the accelerometer? Although videophoning has been a perennial bust, I would argue for the latter: the you-facing camera is pregnant with possibilities as a sensor.

Looking at the Aberdeen report, I was curious to see “gesture recognition” on a list of features that will appear on 60% of phones by 2018. The others on the list are hardware features, but once you have a camera, gesture recognition is just a matter of software. (The Kinect is a sidetrack to this, provoked by lack of compute power.)

In a phone, compute power means battery-drain, so that’s a limitation to using the camera as a sensor. But each generation of chips becomes more power-efficient as well as more powerful, and as phone makers add more and more GPU cores, the developer community delivers new useful uses for them that max them out.

Gesture recognition is already here with Samsung, and soon every Android device. The industry is gearing up for innovation in phone based computer vision with OpenVX from Khronos. When always-on computer vision becomes feasible from a power-drain point of view, gesture recognition and face tracking will look like baby-steps. Smart developers will come up with killer applications that are currently unimaginable. For example, how about a library implementation of Paul Ekman’s emotion recognition algorithms to let you know how you are really feeling right now? Or, in concert with Google Glass, so you will never again be oblivious to your spouse’s emotional temperature.
Update November 19th: Here‘s some news and a little bit of technology background on this topic…
Update November 22:It looks like a company is already engaged on the emotion-recognition technology.

iPhone 4S not iPhone 5

Technically the iPhone 4S doesn’t really pull ahead of the competition: Android-based phones like the Samsung Galaxy S II.

The iPhone 4S even has some worse specifications than the iPhone 4. It is 3 grams heavier and its standby battery life is 30% less. The screen is no larger – it remains smaller than the standard set by the competition. On the other hand the user experience is improved in several ways: the phone is more responsive thanks to a faster processor; it takes better photographs; and Apple has taken yet another whack at the so-far intractable problem of usable voice control. A great benefit to Apple, though not so much to its users, is that the new Qualcomm baseband chip works for all carriers worldwide, so Apple no longer needs different innards for AT&T and Verizon (though Verizon was presumably disappointed that Apple didn’t add a chip for LTE support).

Since its revolutionary debut, the history of the iPhone has been one of evolutionary improvements, and the improvements of the iPhone 4S over the iPhone 4 are in proportion to the improvements in each of the previous generations. The 4S seems to be about consolidation, creating a phone that will work on more networks around the world, and that will remain reliably manufacturable in vast volumes. It’s a risk-averse, revenue-hungry version, as is appropriate for an incumbent leader.

The technical improvements in the iPhone 4S would have been underwhelming if it had been called the iPhone 5, but for a half-generation they are adequate. By mid-2012 several technologies will have ripened sufficiently to make a big jump.

First, Apple will have had time to move their CPU manufacturing to TSMC’s 28 nm process, yielding a major improvement in battery life from the 45 nm process of the current A5, which will be partially negated by the monstrous power of the rumored 4-core A6 design, though the Linley report cautions that it may not be all plain sailing.

Also by mid-2012 Qualcomm may have delivered a world-compatible single-chip baseband that includes LTE (aka ‘real 4G’).

But the 2012 iPhone faces a serious problem. It will continue to suffer a power, weight and thin-ness disadvantage relative to Samsung smartphones until Apple stops using LCD displays. Because they don’t require back-lighting, Super AMOLED display panels are thinner, lighter and consume less power than LCDs. Unfortunately for Apple, Samsung is the leading supplier of AMOLED displays, and Apple’s relationship with Samsung continues to deteriorate. Other LCD alternatives like Qualcomm’s Mirasol are unlikely to be mature enough to rely on by mid-2012. The mid-2012 iPhone will need a larger display, but it looks as though it will continue to be a thick, power hungry LCD.

Sharing Wi-Fi Update

Back in February 2009 I wrote about how Atheros’ new chip made it possible for a phone to act as a Wi-Fi hotspot. A couple of months later, David Pogue wrote in the New York Times about a standalone device to do the same thing, the Novatel MiFi 2200. The MiFi is a Wi-Fi access point with a direct connection to the Internet over a cellular data channel. So you can have “a personal Wi-Fi bubble, a private hot spot, that follows you everywhere you go.”

The type of technology that Atheros announced at the beginning of 2009 was put on a standards track at the end of 2009; the “Wi-Fi Direct” standard was launched in October 2010. So far about 25 products have been certified. Two phones have already been announced with Wi-Fi Direct built-in: the Samsung Galaxy S and the LG Optimus Black.

Everybody has a cell phone, so if a cell phone can act as a MiFi, why do you need a MiFi? It’s another by-product of the dysfunctional billing model of the mobile network operators. If they simply bit the bullet and charged à la carte by the gigabyte, they would be happy to encourage you to use as many devices as possible through your phone.

WiFi Direct may force a change in the way that network operators bill. It is such a compelling benefit to consumers, and so trivial to implement for the phone makers, that the mobile network operators may not be able to hold it back.

So if this capability proliferates into all cell phones, we will be able to use Wi-Fi-only tablets and laptops wherever we are. This seems to be bad news for Novatel’s MiFi and for cellular modems in laptops. Which leads to another twist: Qualcomm’s Gobi is by far the leading cellular modem for laptops, and Qualcomm just announced that it is acquiring Atheros.

Dual Mode Phone Trends Update 4

We are half way through the year, so it’s time for another look at Wi-Fi phone certifications. Three things jump out this time. First, a leap in the number of Wi-Fi phone models in the second quarter of 2010. Second, the arrival of 802.11n in handsets, and third Samsung’s market-leading commitment to 802.11n. According to Rethink Wireless “Samsung’s share of the smartphone market was only about 5% in Q1 but it aims to increase this to almost 15% by year end.” Samsung Wi-Fi-certified a total of 73 dual mode phones in the first six months of 2010, three times as many as second place LG with 23. In the 11n category, Samsung’s lead was even more dominating: its 40 certifications were ten times either of the second place OEMs.

Here is a chart of dual mode phones certified with the Wi-Fi Alliance from 2008 to June 30th 2010. We usually do this chart stacked, but side-by-side gives a clearer comparison between feature phones and smart phones. Note that up to the middle of 2009, smart phones outpaced feature phones, but then it switched. This is a natural progression of Wi-Fi into the mass market, but may also be exaggerated by a quirk of reporting: of HTC’s 17 certifications in the first half of 2010, it only categorized one as a smart phone.
Dual mode phones by quarter 2008-2010

The chart below shows the growth of 802.11n. It starts in January 2010 because only one 11n phone was certified in 2009, at the end of December. As you can see, the growth is strong. I anticipate that practically all new dual mode phone certifications will be for 802.11n by the end of 2010.

802.11n phones 2010 by month

Below is the same chart sliced by manufacturer instead of by month. The iPhone is missing because it wasn’t certified until July, and the iPad is missing because it’s not a phone. With only one 802.11n phone, Nokia has become a technology laggard, at least in this respect. The RIM Pearl 8100/8105 certifications are the only ones with STBC, an important feature for phones because it improves rate at distance. All the major chips (except those from TI) support STBC, so the phone OEMs must be either leaving it disabled or just not bothering to certify for it.

802.11n phones 2010 by manufacturer

Samsung GT-S8500 is first with 11n, BT 3.0 certifications

Engadget reports that the Samsung GT-S8500 is the first phone to support Bluetooth 3.0. A look at the Wi-Fi Alliance website reveals that it was also the first feature phone to gain 802.11n certification.

The certificate is dated December 28th 2009, the same date that the first smartphone was certified for 802.11n – the LG Veri/VS750. The VS750 Wi-Fi appears to be more advanced than the Samsung, since it is certified for short guard interval and WMM Power Save.

While these are the first phones to gain Wi-Fi certification for 802.11n, they may not be the first to market.

Dual mode phone trends update 3

I last looked at dual mode phone certifications on the Wi-Fi Alliance website almost a year ago.

Here’s what has happened since, through the first three quarters of 2009:
Wi-Fi Alliance Dual-Mode Phone Certifications 2005-2009

There are still no certifications for 802.11 draft n, and almost none for 802.11a.

Here’s another breakdown, by manufacturer and year. Click on the chart to get a bigger image. This shows that the Wi-Fi enthusiasts have been pretty constant over the years: Nokia, HTC, Motorola and Samsung. Then more recently SonyEricsson and LG. Note that the 2009 figures are only through Q3, so the growth is even more impressive than it seems from this chart.
Wi-Fi Alliance Dual-Mode Phone Certifications 2005-2009 by OEM

The all-time champion is Samsung, with a total of 84 phone models certified for Wi-Fi, followed by Nokia with 68, then HTC with 54. This changes if you look just at smartphones, where Nokia has 61 total certifications to HTC’s 34 and Samsung’s 29.

Nokia no longer the only VoWi-Fi friendly phone maker

Until now, Nokia has been top of the heap in the category of VoIP-friendliness. When I spoke with Richard Watson, CTO of DiVitas, last year in the course of my test drive of the DiVitas system, he pointed out that dual-mode phones are not normally VoIP-friendly. At that time the only phone he recommended was the Nokia E71. There are several reasons for this, primarily the treatment of the voice path and the ease of integration of the VoIP software with the built-in phone software user interfaces. Since then, DiVitas has been working closely with Samsung, and now Richard says several Samsung phones are well suited to Voice over Wi-Fi. Let’s hope this shakes the other phone OEMs loose and gets them working on improving Voice over Wi-Fi performance.

A not so perfect Storm

The Verizon Storm may be heading for failure in more than one way. A raft of reviewers, led by David Pogue of the New York Times are trashing its usability. This means that even with the marketing might of Verizon behind it it may not fulfill its goal of being a bulwark against the iPhone in the enterprise.

But the Storm was an experiment in another way by Verizon. The other three major American mobile network operators have capitulated to Wi-Fi in smartphones. Against the new conventional wisdom, Verizon decided to launch a new flagship smartphone without Wi-Fi. The Storm looks like a trial balloon to see whether Wi-Fi is optional in modern smartphones. If the Storm is a success, it will demonstrate that it is possible to have credible business smartphones without Wi-Fi. But if it turns out to be a flop because of other factors, it will not be a proof point for Wi-Fi either way.

But Wi-Fi is a closed issue by now for all the network operators, perhaps even including Verizon. Phones have lead times of the order of a year or so, and controversies active back then may now be resolved. Verizon covered its bets by launching three other smartphones around the same time as the Storm, all with Wi-Fi (HTC Touch Pro, Samsung Omnia, Samsung Saga).

Before its launch, AT&T hoped that the iPhone would stimulate use of the cellular data network. It succeeded in this, so far beyond AT&T’s hopes that it revealed a potential problem with the concept of 3G (and 4G) data. The network slows to a crawl if enough subscribers use data intensively in small areas like airports and conferences. Mobile network operators used to fear that if phones had Wi-Fi subscribers would use it instead of the cellular data network, causing a revenue leak. AT&T solved that problem with the iPhone by making a subscription to the data service obligatory. T-Mobile followed suit with the Google phone. So no revenue leak. With the data subscription in hand, Wi-Fi is a good thing for the network operators because it offloads the 3G network. In residences and businesses all the data that goes through Wi-Fi is a reduction in the potential load on the network. In other words, a savings in infrastructure investment, which translates to profit. This may be some of the thinking behind AT&T’s recent acquisition of Wayport. The bandwidth acquired with Wayport offloads the AT&T network relatively cheaply. AT&T’s enthusiasm for Wi-Fi is such that it is selling some new Wi-Fi phones without requiring a data subscription.

The enterprise market is one that mobile network operators have long neglected. It is small relative to the consumer market, and harder to fit into a one-size-fits-all model. Even so, in these times of scraping for revenue in every corner, and with the steady rise of the Blackberry, the network operators are taking a serious look at the enterprise market.

The device manufacturers are way ahead of the network operators on this issue: the iPhone now comes with a lot of enterprise readiness Kool-Aid; Windows Mobile makes manageability representations, as does Nokia with its Eseries handsets. RIM, the current king of the enterprise smartphone vendors also pitches its IT-friendliness.

Wi-Fi in smartphones has benefits and drawbacks for enterprises. One benefit is that you have another smart device on the corporate LAN to enhance productivity. A drawback is that you have another smart device on the corporate LAN ripe for viruses and other security breaches. But that issue is mitigated to some extent if smartphones don’t have Wi-Fi. So it’s arguable that the Storm may be more enterprise-friendly as a result of its lack of Wi-Fi. Again, if the Storm becomes a hit in enterprises that argument will turn out to hold water. If the Storm is a flop for other reasons, we still won’t know, and it will have failed as a trial balloon for Wi-Fi-less enterprise smartphones.

Google phone alliance members

The Open Handset Alliance was announced today by Google and 30 or so other companies. Until now the highest-profile open source handset operating environment was OpenMoko.

The list of participants has no real surprises in it. Nokia isn’t on the list, most likely because this project competes head on with Symbian. This may also help to explain why Sony Ericsson isn’t a supporter yet, either. But the other three of the top five handset manufacturers are members: Motorola, Samsung and LG. All of these ship Symbian-based phones, but they also ship Windows based phones, so they are already pursuing an OS-agnostic strategy. Open standards are less helpful to a market leader than to its competitors.

Of course the other leading smartphone OS vendors are also missing from the list: Microsoft, Apple, Palm and RIM.

Ebay is there because this massively benefits Skype.

Silicon vendors retain more control of their destiny when there is a competitive software community, so it makes sense that TI is aboard even though it is the market leader in cellphone chips. Intel is another chip vendor that is a member. Intel can normally be relied on to support this type of open platform initiative, and although Intel sold its handset-related businesses in 2006, its low power CPU efforts may evolve from ultra-mobile PCs down to smartphones in a few years.

Among MNOs Verizon and AT&T Mobile are notorious for their walled-garden policies, so it makes sense that they aren’t on the list, though Sprint and T-Mobile are, which is an encouraging indication.

At the launch of the iPhone Steve Jobs said that the reason there would be no SDK for the iPhone was that AT&T didn’t want their network brought down by a rogue application. I ridiculed this excuse in an Internet Telephony column. Even so, the carriers do have a valid objection to completely open platforms: their subscribers will call them for support when the phone crashes. For this reason, applications that use sensitive APIs in Symbian must be “Symbian signed.” When he announced the iPhone SDK, Steve Jobs alluded to this as a model that Apple may follow.

So Sprint’s and T-Mobile’s participation in this initiative is very interesting. Sprint’s press release says:

Unlike other wireless carriers, Sprint allows data users to freely browse the Internet outside its portal and has done so since first offering access to the Internet on its phones in 2001.

Open Internet access is actually available from all the major US MNOs other than Verizon; AT&T ships the best handset for this, the iPhone. But the iPhone doesn’t (officially) let users load whatever software they want onto the phone. Symbian and Windows-based phones generally do, and again all the major MNOs ship handsets based on these operating systems. An open source handset goes a big step further, but who benefits depends on what parts of the source code are published, and what APIs are exposed by the proprietary parts of the system. As a rule of thumb, one would think that giving developers this greater degree of control over the system will increase their scope for innovation.

Mossberg revisits UMPC

I wrote last month about a WSJ article panning the Flipstart UMPC.

Today Mossberg is a lot kinder; writing about the Samsung Q1 Ultra, he says it takes over four minutes to reboot it, the battery lasts about 3 hours, and it costs too much at $800. But his conclusion is: “if you don’t do a lot of document creation, and value small size and weight enough to put up with some hassles, the UMPC is finally an acceptable choice.”

This has to be a huge relief to Paul Otellini. Intel is banking on the UMPC becoming a mainstream device.