Clearing the Cloud for Reliable, Crystal-Clear VoIP Services

The compelling advantage of VoIP is that it is far cheaper than circuit switched technology. But VoIP calls often sound horrible. It doesn’t have to be this way. Although VoIP is intrinsically prone to jitter, delay and packet loss, good system design can mitigate all these impairments. The simplest solution is over-provisioning bandwidth.

The lowest bandwidth leg of a VoIP call, where the danger of delayed or lost packets is the greatest, is usually the ‘last mile’ WAN connection from the ISP to the customer premises. This is also where bandwidth is most expensive.

On this last leg, you tend to get what you pay for. Cheap connections are unreliable. Since businesses live or die with their phone service, they are motivated to pay top dollar for a Service Level Agreement specifying “five nines” reliability. But there’s more than one way to skin a cat. Modern network architectures achieve high levels of reliability through redundant low-cost, less reliable systems. For example, to achieve 99.999% aggregate reliability, you could combine two independent systems (two ISPs) each with 99.7% reliability, three each with 97.8% reliability, or four each with 94% reliability. In other words, if your goal is 5 minutes or less of system down-time per year, with two ISPs you could tolerate 4 minutes of down-time per ISP per day. With 3 ISPs, you could tolerate 30 minutes of down-time per ISP per day.

Here’s a guest post from Dr. Cahit Jay Akin of Mushroom Networks, describing how to do this:

Clearing the Cloud for Reliable, Crystal-Clear VoIP Services

More companies are interested in cloud-based VoIP services, but concerns about performance hold them back. Now there are technologies that can help.

There’s no question that hosted, cloud-based Voice over IP (VoIP) and IP-PBX technologies are gaining traction, largely because they reduce costs for equipment, lines, manpower, and maintenance. But there are stumbling blocks – namely around reliability, quality and weak or non-existent failover capabilities – that are keeping businesses from fully committing.

Fortunately, there are new and emerging technologies that can optimize performance without the need for costly upgrades to premium Internet services. These technologies also protect VoIP services from jitter, latency caused by slow network links, and other common unpredictable behaviors of IP networks that impact VoIP performance. For example, Broadband Bonding, a technique that bonds various Internet lines into a single connection, boosts connectivity speeds and improves management of the latency within an IP tunnel. Using such multiple links, advanced algorithms can closely monitor WAN links and make intelligent decisions about each packet of traffic to ensure nothing is ever late or lost during communication.

VoIP Gains Market Share

The global VoIP services market, including residential and business VoIP services, totaled $63 billion in 2012, up 9% from 2011, according to market research firm Infonetics. Infonetics predicts that the combined business and residential VoIP services market will grow to $82.7 billion in 2017. While the residential segment makes up the majority of VoIP services revenue, the fastest-growing segment is hosted VoIP and Unified Communications (UC) services for businesses. Managed IP-PBX services, which focus on dedicated enterprise systems, remain the largest business VoIP services segment.

According to Harbor Ridge Capital LLC, which did an overview of trends and mergers & acquisitions activity of the VoIP market in early 2012, there are a number of reasons for VoIP’s growth. Among them: the reduction in capital investments and the flexibility hosted VoIP provides, enabling businesses to scale up or down their VoIP services as needed. Harbor Ridge also points out a number of challenges, among them the need to improve the quality of service and meet customer expectations for reliability and ease of use.

But VolP Isn’t Always Reliable

No business can really afford a dropped call or a garbled message left in voicemail. But these mishaps do occur when using pure hosted VoIP services, largely because they are reliant on the performance of the IP tunnel through which the communications must travel. IP tunnels are inevitably congested and routing is unpredictable, two factors that contribute to jitter, delay and lost packets, which degrade the quality of the call. Of course, if an IP link goes down, the call is dropped.

Hosted, cloud-based VoIP services offer little in the way of traffic prioritization, so data and voice fight it out for Internet bandwidth. And there’s little monitoring available. IP-PBX servers placed in data centers or at the company’s headquarters can help by providing some protection over pure hosted VoIP services. They offer multiple WAN interfaces that let businesses add additional, albeit costly, links to serve as backups if one fails. Businesses can also take advantage of the various functions that an IP-PBX system offers, such as unlimited extensions and voice mail boxes, caller ID customizing, conferencing, interactive voice response and more. But IP-PBXes are still reliant on the WAN performance and offer limited monitoring features. Thus, users and system administrators might not even know about an outage until they can’t make or receive calls. Some hosted VoIP services include a hosted IP-PBX, which typically include back-up and storage and failover functions, as well as limited monitoring.

Boosting Performance through Bonding and Armor

Mushroom Networks has developed several technologies designed to improve the performance, reliability and intelligence of a range of Internet connection applications, including VoIP services. The San Diego, Calif., company’s WAN virtualization solution leverages virtual leased lines (VLLs) and its patented Broadband Bonding, a technique that melds various numbers of Internet lines into a single connection. WAN virtualization is a software-based technology that uncouples operating systems and applications from the physical hardware, so infrastructure can be consolidated and application and communications resources can be pooled within virtualized environments. WAN virtualization adds intelligence and management so network managers can dynamically build a simpler, higher-performing IP pipe out of real WAN resources, including existing private WANs and various Internet WAN links like DSL, cable, fiber, wireless and others. The solution is delivered via the Truffle appliance, a packet level load balancing router with WAN aggregation and Internet failover technology.

Using patented Broadband Bonding techniques, Truffle bonds various numbers of Internet lines into a single connection to ensure voice applications are clear, consistent and redundant. This provides faster connectivity via the sum of all the line speeds as well as intelligent management of the latency within the tunnel. Broadband Bonding is a cost effective solution for even global firms that have hundreds of branch offices scattered around the world because it can be used with existing infrastructures, enabling disparate offices to have the same level of connectivity as the headquarters without the outlay of too much capital. The end result is a faster connection with multiple built-in redundancies that can automatically shield negative network events and outages from the applications such as VoIP. Broadband Bonding also combines the best attributes of the various connections, boosting speeds and reliability.

Mushroom Networks’ newest technology, Application Armor, shields VoIP services from the negative effects of IP jitter, latency, packet drops, link disconnects and other issues. This technology relies on a research field known as Network Calculus, that models and optimizes communication resources. Through decision algorithms, Application Armor monitors traffic and refines routing in the aggregated, bonded pipe by enforcing application-specific goals, whether it’s throughput or reduced latency.

VoIP at Broker Houlihan Lawrence – Big Savings and Performance

New York area broker Houlihan Lawrence – the nation’s 15th largest independent realtor – has cut its telecommunications bill by nearly 75 percent by deploying Mushroom Networks’ Truffle appliances in its branch offices. The agency began using Truffle shortly after Superstorm Sandy took out the company’s slow and costly MPLS communications network when it landed ashore near Atlantic City, New Jersey last year. After the initial deployment to support mission-critical data applications including customer relationship management and email, Houlihan Lawrence deployed a state-of-the-art VOIP system and runs voice communications through Mushroom Networks’ solution. The ability to diversify connections across multiple providers and multiple paths assures automated failover in the event a connection goes down, and the Application Armor protects each packet, whether it’s carrying voice or data, to ensure quality and performance are unfailing and crystal clear.

Hosted, cloud-based Voice over IP (VoIP) and IP-PBX technologies help companies like Houlihan Lawrence dramatically reduce costs for equipment, lines, manpower, and maintenance. But those savings are far from ideal if they come without reliability, quality and failover capabilities. New technologies, including Mushroom Networks’ Broadband Bonding and Application Armor, can optimize IP performance, boost connectivity speeds, improve monitoring and shield VoIP services from jitter, latency, packet loss, link loss and other unwanted behaviors that degrade performance.

Dr. Cahit Jay Akin is the co-founder and chief executive officer of Mushroom Networks, a privately held company based in San Diego, CA, providing broadband products and solutions for a range of Internet applications.

QoS meters on Voxygen

The term “QoS” is used ambiguously. The two main categories of definition are first, QoS Provisioning: “the capability of a network to provide better service to selected network traffic,” which means packet prioritization of one kind or another, and second more literally: “Quality of Service,” which is the degree of perfection of a user’s audio experience in the face of potential impairments to network performance. These impairments fall into four categories: availability, packet loss, packet delay and tampering. Since this sense is normally used in the context of trying to measure it, we could call it QoS Metrics as opposed to QoS Provisioning. I would put issues like choice of codec and echo into the larger category of Quality of Experience, which includes all the possible impairments to audio experience, not just those imposed by the network.

By “tampering” I mean any intentional changes to the media payload of a packet, and I am OK with the negative connotations of the term since I favor the “dumb pipes” view of the Internet. On phone calls the vast bulk of such tampering is transcoding: changing the media format from one codec to another. Transcoding always reduces the fidelity of the sound, even when transcoding to a “better” codec.

Networks vary greatly in the QoS they deliver. One of the major benefits of going with VoIP service provided by your ISP (Internet Service Provider) is that your ISP has complete control over QoS. But there is a growing number of ITSPs (Internet Telephony Service Providers) that contend that the open Internet provides adequate QoS for business-grade telephone service. Skype, for example.

But it’s nice to be sure. So I have added a “QoS Metrics” category in the list to the right of this post. You can use the tools there to check your connection. I particularly like the one from Voxygen, which frames the test results in terms of the number of simultaneous voice sessions that your WAN connection can comfortably handle. Here’s an example of a test of ten channels:

Screen shot of Voxygen VoIP performance metrics tool

VoIP on the cellular data channel

In a recent letter to the FCC, AT&T said that it had no objection to VoIP applications on the iPhone that communicate over the Wi-Fi connection. It furthermore said:

Consistent with this approach, we plan to take a fresh look at possibly authorizing VoIP capabilities on the iPhone for use on AT&T’s 3G network.

So why would anybody want to do VoIP on the cellular data channel, when there is a cellular voice channel already? Wouldn’t voice on the data channel cost more? And since the voice channel is optimized for voice and the data channel isn’t, wouldn’t voice on the data channel sound even worse than cellular voice already does?

Let’s look at the “why bother?” question first. There are actually at least four reasons you might want to do voice on the cellular data channel:

  1. To save money. If your voice plan has some expensive types of call (for example international calls) you may want to use VoIP on the data channel for toll by-pass. The alternative to this is to use the voice channel to call a local access number for an international toll by-pass service (like RebTel.)
  2. To get better sound quality: the cellular voice codecs are very low bandwidth and sound horrible. You can choose which codec to run over the data network and even go wideband. At IT Expo West a couple of weeks ago David Frankel of ZipDX demoed a wideband voice call on his laptop going through a Sprint Wireless Data Card. The audio quality was excellent.
  3. To get additional service features: companies like DiVitas offer roaming between the cellular and Wi-Fi networks that makes your cell phone act as an extension behind your corporate PBX. All these solutions currently use the cellular voice channel when out of Wi-Fi range, but if they were to go to the data channel they could offer wideband codecs and other differentiating features.
  4. For cases where there is no voice channel. In the example of David Frankel’s demo, the wireless data card doesn’t offer a voice channel, so VoIP on the data channel is the only option for a voice connection.

Moving on to the issue of cost, an iPhone unlimited data plan is $30 per month. “Unlimited” is AT&T’s euphemism for “limited to 5GB per month,” but translated to voice that’s a lot of minutes: even with IP packet overhead the bit-rate of compressed HD voice is going to be around 50K bits per second, which works out to about 13,000 minutes in 5GB. So using it for voice is unlikely to increase your bill. On the other hand, many voice plans are already effectively unlimited, what with rollover minutes, friend and family minutes, night and weekend minutes and whatnot, and you can’t get a phone without a voice plan. So for normal (non-international) use voice on the data channel is not going to reduce your bill, but it is unlikely to increase it, either.

Finally we come to the issue of whether voice sounds better on the voice channel or the data channel. The answer is, it depends on several factors, primarily the codec and the network QoS. With VoIP you can radically improve the sound quality of a call by using a wideband codec, but do impairments on the data channel nullify this benefit?

Technically, the answer is yes. The cellular data channel is not engineered for low latency. Variable delays are introduced by network routing decisions and by router queuing decisions. Latencies in the hundreds of milliseconds are not unusual. This will change with the advent of LTE, where the latencies will be of the order of 10 milliseconds. The available bandwidth is also highly variable, in contrast to the fixed bandwidth allocation of the voice channel. It can sometimes drop below what is needed for voice with even an aggressive variable rate codec.

In practice VoIP on the cellular data channel can sometimes sound much better than regular cellular voice. I mentioned above David Frankel’s demo at IT Expo West. I performed a similar experiment this morning with Michael Graves, with similarly good results. I was on a Polycom desk phone, Michael used Eyebeam on a laptop, and the codec was G.722. The latency on this call was appreciable – I estimated it at around 1 second round trip. There was also some packet loss – not bad for me, but it caused a sub-par experience for Michael. Earlier this week at Jeff Pulver’s HD Connect conference in New York, researchers from Qualcomm demoed a handset running on the Verizon network using EVRC-WB, transcoding to G.722 on Polycom and Gigaset phones in their lab in San Diego. The sound quality was excellent, but the latency was very high – I estimated it at around two seconds round trip.

The ITU addresses latency (delay) in Recommendation G.114. Delay is a problem because normal conversation depends on turn taking. Most people insert pauses of up to about 400 ms as they talk. If nobody else speaks during a pause, they continue. This means that if the one-way delay on a phone conversation is greater than 200 ms, the talker doesn’t hear an interruption within the 400 ms break, and starts talking again, causing frustrating collisions.
The ITU E-Model for call quality identifies a threshold at about 170 ms one-way at which latency becomes a problem. The E-Model also tells us that increasing latency amplifies other impairments – notably echo, which can be severe at low latencies without being a problem, but at high latencies even relatively quiet echo can severely disrupt a talker.

Some people may be able to handle long latencies better than others. Michael observed that he can get used to high latency echo after a few minutes of conversation.

Wi-Fi certification for voice devices

In news that is huge for VoWi-Fi, the Wi-Fi Alliance announced on June 30th a new certification program, “Voice-Personal.” Eight devices have already been certified under this program, including enterprise access points from Cisco and Meru, a residential access point from Broadcom, and client adapters from Intel and Redpine Signals.

Why is this huge news? Well, as the press release points out, by 2011 annual shipments of cell phones with Wi-Fi will be running at roughly 300 million units. The Wi-Fi in these phones will be used for Internet browsing, for syncing photos and music with PCs, and for cheap or free voice calls.

The certification requirements for Voice-Personal are not aggressive: only four simultaneous voice calls in the presence of data traffic, with a latency of less than 50 milliseconds and a maximum jitter of less than 50 milliseconds. These numbers will produce an acceptable call under most conditions, but a network round-trip delay of 300 ms is generally considered to approach the limit of acceptability, and with a Wi-Fi hop at each end running at the limit of these specifications there would be no room in the latency budget for any additional delays in the voice path. The packet loss requirement, 1% with no burst losses, is a very good number considering that modern voice codecs from companies like GIPS can yield excellent sound quality in the presence of much higher packet loss. This number is hard to achieve in the real world, as phones encounter microwave ovens, move through spots of poor coverage and transition between access points.

Since this certification is termed “Voice-Personal,” four active calls per access point is acceptable; a residence is unlikely to need more than that. Three of the four access points submitted for this certification are enterprise access points. They should be able to handle many more calls, and probably can. The Wi-Fi Alliance is planning a “Voice-Enterprise” certification for 2009.

There are several things that are good about this certification. First, the WFA has seen fit to highlight voice as a primary use for Wi-Fi, and has set a performance baseline. Second, this certification requires some other certifications as well, like WMM power save and WMM QoS. So far in 2008, of 99 residential access points certified only 6 support WMM power save, and of 52 enterprise access points only 13 support WMM power save. One of the biggest criticisms of Wi-Fi in handsets is that it draws too much power. WMM power save yields radical improvements in battery life – better than doubling talk time and increasing standby time by over 30%, according to numbers in the WFA promotional materials.

WSJ on FMC

Today’s Wall Street Journal has a good article about T-Mobile’s UMA trial in Seattle. It says that T-Mobile may be rolling it out nationally as early as next month, despite some trial particpants’ complaints about handoff and battery life issues. T-Mobile will be offering a home router to help with QoS and battery life. I presume that for the battery life this is just WMM Power Save (802.11e APSD) since that is what the phones in the trial (Samsung T709 and Nokia 6136) support. For QoS side I expect these APs will support WMM (802.11e EDCF), but they could also support some proprietary QoS on the WAN access link, the way that the AT&T CallVantage routers do, which would be interesting.

There is some background on the trial here.

The article goes on to put the trial into the context of other FMC deployments, from BT Fusion, Telecom Italia and Orange. The article quotes a Verizon Wireless spokesman saying that they aren’t convinced that Wi-Fi can deliver high enough voice quality to carry Verizon branded calls. This is amusing bearing in mind the usual quality of a cellular call in a residence.

The article also quotes Frank Hanzlik, the head of the Wi-Fi Alliance as saying that business FMC may have more potential than consumer. I agree.

Wi-Fi Interference Experiments

Interesting new series of white papers on Wi-Fi interference from Craig Mathias of the Farpoint Group. He set up a couple of clients and attempted various activities (file transfer, VoIP, video streaming) in the presence of interference from various sources (microwave oven, cordless phone, DECT phone, another AP, a Bluetooth headset) and characterized the impairments. His conclusions were that some interference sources can completely shut down some uses (almost all of them shut down video), but that interference can be managed and does not present a long term stopper to Wi-Fi.

Missing from the tests was 802.11n. This should make a huge difference, for several reasons. First, its MIMO operation is intrinsically more resistant to interference, second 11n operates both in the 2.4 GHz frequency range (like 11b/g) and in the 5 GHz frequency range (like 11a) . The 5 GHz waveband is immune from microwave oven interference, and most of the cordless phone interference. Its disadvantage of shorter range is mitigated by the multi-path amplification effect of MIMO.