Clearing the Cloud for Reliable, Crystal-Clear VoIP Services

The compelling advantage of VoIP is that it is far cheaper than circuit switched technology. But VoIP calls often sound horrible. It doesn’t have to be this way. Although VoIP is intrinsically prone to jitter, delay and packet loss, good system design can mitigate all these impairments. The simplest solution is over-provisioning bandwidth.

The lowest bandwidth leg of a VoIP call, where the danger of delayed or lost packets is the greatest, is usually the ‘last mile’ WAN connection from the ISP to the customer premises. This is also where bandwidth is most expensive.

On this last leg, you tend to get what you pay for. Cheap connections are unreliable. Since businesses live or die with their phone service, they are motivated to pay top dollar for a Service Level Agreement specifying “five nines” reliability. But there’s more than one way to skin a cat. Modern network architectures achieve high levels of reliability through redundant low-cost, less reliable systems. For example, to achieve 99.999% aggregate reliability, you could combine two independent systems (two ISPs) each with 99.7% reliability, three each with 97.8% reliability, or four each with 94% reliability. In other words, if your goal is 5 minutes or less of system down-time per year, with two ISPs you could tolerate 4 minutes of down-time per ISP per day. With 3 ISPs, you could tolerate 30 minutes of down-time per ISP per day.

Here’s a guest post from Dr. Cahit Jay Akin of Mushroom Networks, describing how to do this:

Clearing the Cloud for Reliable, Crystal-Clear VoIP Services

More companies are interested in cloud-based VoIP services, but concerns about performance hold them back. Now there are technologies that can help.

There’s no question that hosted, cloud-based Voice over IP (VoIP) and IP-PBX technologies are gaining traction, largely because they reduce costs for equipment, lines, manpower, and maintenance. But there are stumbling blocks – namely around reliability, quality and weak or non-existent failover capabilities – that are keeping businesses from fully committing.

Fortunately, there are new and emerging technologies that can optimize performance without the need for costly upgrades to premium Internet services. These technologies also protect VoIP services from jitter, latency caused by slow network links, and other common unpredictable behaviors of IP networks that impact VoIP performance. For example, Broadband Bonding, a technique that bonds various Internet lines into a single connection, boosts connectivity speeds and improves management of the latency within an IP tunnel. Using such multiple links, advanced algorithms can closely monitor WAN links and make intelligent decisions about each packet of traffic to ensure nothing is ever late or lost during communication.

VoIP Gains Market Share

The global VoIP services market, including residential and business VoIP services, totaled $63 billion in 2012, up 9% from 2011, according to market research firm Infonetics. Infonetics predicts that the combined business and residential VoIP services market will grow to $82.7 billion in 2017. While the residential segment makes up the majority of VoIP services revenue, the fastest-growing segment is hosted VoIP and Unified Communications (UC) services for businesses. Managed IP-PBX services, which focus on dedicated enterprise systems, remain the largest business VoIP services segment.

According to Harbor Ridge Capital LLC, which did an overview of trends and mergers & acquisitions activity of the VoIP market in early 2012, there are a number of reasons for VoIP’s growth. Among them: the reduction in capital investments and the flexibility hosted VoIP provides, enabling businesses to scale up or down their VoIP services as needed. Harbor Ridge also points out a number of challenges, among them the need to improve the quality of service and meet customer expectations for reliability and ease of use.

But VolP Isn’t Always Reliable

No business can really afford a dropped call or a garbled message left in voicemail. But these mishaps do occur when using pure hosted VoIP services, largely because they are reliant on the performance of the IP tunnel through which the communications must travel. IP tunnels are inevitably congested and routing is unpredictable, two factors that contribute to jitter, delay and lost packets, which degrade the quality of the call. Of course, if an IP link goes down, the call is dropped.

Hosted, cloud-based VoIP services offer little in the way of traffic prioritization, so data and voice fight it out for Internet bandwidth. And there’s little monitoring available. IP-PBX servers placed in data centers or at the company’s headquarters can help by providing some protection over pure hosted VoIP services. They offer multiple WAN interfaces that let businesses add additional, albeit costly, links to serve as backups if one fails. Businesses can also take advantage of the various functions that an IP-PBX system offers, such as unlimited extensions and voice mail boxes, caller ID customizing, conferencing, interactive voice response and more. But IP-PBXes are still reliant on the WAN performance and offer limited monitoring features. Thus, users and system administrators might not even know about an outage until they can’t make or receive calls. Some hosted VoIP services include a hosted IP-PBX, which typically include back-up and storage and failover functions, as well as limited monitoring.

Boosting Performance through Bonding and Armor

Mushroom Networks has developed several technologies designed to improve the performance, reliability and intelligence of a range of Internet connection applications, including VoIP services. The San Diego, Calif., company’s WAN virtualization solution leverages virtual leased lines (VLLs) and its patented Broadband Bonding, a technique that melds various numbers of Internet lines into a single connection. WAN virtualization is a software-based technology that uncouples operating systems and applications from the physical hardware, so infrastructure can be consolidated and application and communications resources can be pooled within virtualized environments. WAN virtualization adds intelligence and management so network managers can dynamically build a simpler, higher-performing IP pipe out of real WAN resources, including existing private WANs and various Internet WAN links like DSL, cable, fiber, wireless and others. The solution is delivered via the Truffle appliance, a packet level load balancing router with WAN aggregation and Internet failover technology.

Using patented Broadband Bonding techniques, Truffle bonds various numbers of Internet lines into a single connection to ensure voice applications are clear, consistent and redundant. This provides faster connectivity via the sum of all the line speeds as well as intelligent management of the latency within the tunnel. Broadband Bonding is a cost effective solution for even global firms that have hundreds of branch offices scattered around the world because it can be used with existing infrastructures, enabling disparate offices to have the same level of connectivity as the headquarters without the outlay of too much capital. The end result is a faster connection with multiple built-in redundancies that can automatically shield negative network events and outages from the applications such as VoIP. Broadband Bonding also combines the best attributes of the various connections, boosting speeds and reliability.

Mushroom Networks’ newest technology, Application Armor, shields VoIP services from the negative effects of IP jitter, latency, packet drops, link disconnects and other issues. This technology relies on a research field known as Network Calculus, that models and optimizes communication resources. Through decision algorithms, Application Armor monitors traffic and refines routing in the aggregated, bonded pipe by enforcing application-specific goals, whether it’s throughput or reduced latency.

VoIP at Broker Houlihan Lawrence – Big Savings and Performance

New York area broker Houlihan Lawrence – the nation’s 15th largest independent realtor – has cut its telecommunications bill by nearly 75 percent by deploying Mushroom Networks’ Truffle appliances in its branch offices. The agency began using Truffle shortly after Superstorm Sandy took out the company’s slow and costly MPLS communications network when it landed ashore near Atlantic City, New Jersey last year. After the initial deployment to support mission-critical data applications including customer relationship management and email, Houlihan Lawrence deployed a state-of-the-art VOIP system and runs voice communications through Mushroom Networks’ solution. The ability to diversify connections across multiple providers and multiple paths assures automated failover in the event a connection goes down, and the Application Armor protects each packet, whether it’s carrying voice or data, to ensure quality and performance are unfailing and crystal clear.

Hosted, cloud-based Voice over IP (VoIP) and IP-PBX technologies help companies like Houlihan Lawrence dramatically reduce costs for equipment, lines, manpower, and maintenance. But those savings are far from ideal if they come without reliability, quality and failover capabilities. New technologies, including Mushroom Networks’ Broadband Bonding and Application Armor, can optimize IP performance, boost connectivity speeds, improve monitoring and shield VoIP services from jitter, latency, packet loss, link loss and other unwanted behaviors that degrade performance.

Dr. Cahit Jay Akin is the co-founder and chief executive officer of Mushroom Networks, a privately held company based in San Diego, CA, providing broadband products and solutions for a range of Internet applications.

ITExpo West — Achieving HD Voice On Smartphones

I will be moderating a panel discussion at ITExpo West on Tuesday 5th October at 11:30 am in room 306B: “Achieving HD Voice On Smartphones.”

Here’s the session description:

The communications market has been evolving to fixed high definition voice services for some time now, and nearly every desktop phone manufacturer is including support for G.722 and other codecs now. Why? Because HD voice makes the entire communications experience a much better one than we are used to.

But what does it mean for the wireless industry? When will wireless communications become part of the HD revolution? How will handset vendors, network equipment providers, and service providers have to adapt their current technologies in order to deliver wireless HD voice? How will HD impact service delivery? What are the business models around mobile HD voice?

This session will answer these questions and more, discussing both the technology and business aspects of bringing HD into the mobile space.

The panelists are:

This is a deeply experienced panel; each of the panelists is a world-class expert in his field. We can expect a highly informative session, so come armed with your toughest questions.

ITExpo West — Building Better HD Video Conferencing & Collaboration Systems

I will be moderating a session at ITExpo West on Tuesday 5th October at 9:30 am: “Building Better HD Video Conferencing & Collaboration Systems,” will be held in room 306A.

Here’s the session description:

Visual communications are becoming more and more commonplace. As networks improve to support video more effectively, the moment is right for broad market adoption of video conferencing and collaboration systems.

Delivering high quality video streams requires expertise in both networks and audio/video codec technology. Often, however, audio quality gets ignored, despite it being more important to efficient communication than the video component. Intelligibility is the key metric here, where wideband audio and voice quality enhancement algorithms can greatly improve the quality of experience.

This session will cover both audio and video aspects of today’s conferencing systems, and the various criteria that are used to evaluate them, including round-trip delay, lip-sync, smooth motion, bit-rate required, visual artifacts and network traversal – and of course pure audio quality. The emphasis will be on sharing best practices for building and deploying high-definition conferencing systems.

The panelists are:

  • James Awad, Marketing Product Manager, Octasic
  • Amir Zmora, VP Products and Marketing, RADVISION
  • Andy Singleton, Product Manager, MASERGY

These panelists cover the complete technology stack from chips (Octasic), to equipment (Radvison) to network services (Masergy), so please bring your questions about any technical aspect of video conferencing systems.

ITExpo West — The State of VoIP Peering

I will be moderating a session at ITExpo West on Monday 4th October at 2:15 pm: “The State of VoIP Peering,” will be held in room 304C.

Here’s the session description:

VoIP is a fact – it is here, and it is here to stay. That fact is undeniable. To date, the cost savings associated with VoIP have largely been enough to drive adoption. However, the true benefits of VoIP will only be realized through the continued growth of peering, which will keep calls on IP backbones rather than moving them onto the PSTN. Not only will increased peering continue to reduce costs, it will increase voice call quality – HD voice, for instance, can only be delivered on all-IP calls.

Of course, while there are benefits to peering, traditional carriers have traditionally not taken kindly to losing their PSTN traffic, for which they are able to bill by the minute. But, as the adoption of IP communications continues to increase – and of course the debate continues over when we will witness the true obsolescence of the PSTN – carriers will have little choice but to engage in peering relationships.

This session will offer an market update on the status of VoIP peering and its growth, as well as trends and technologies that will drive its growth going forward, including wideband audio and video calling.

The panelists are:

This is shaping up to be a fascinating session. Rico can tell us about the hardware technologies that are enabling IP end-to-end for phone calls, and Mark and Grant will give us a real-world assessment of the state of deployment, the motivations of the early adopters, and the likely fate of the PSTN.

HD Voice, Peering and ENUM

The most convenient route between telephone service providers is through the PSTN, since you can’t offer phone service without connecting to it. Because of this convenience telephone service providers tend to consider PSTN connectivity adequate, and don’t take the additional step of delivering IP connectivity. This is unfortunate because it inhibits the spread of high quality wideband (HD Voice) phone calls. For HD voice to happen, the two endpoints must be connected by an all-IP path, without the media stream crossing into the PSTN.

For example, OnSIP is my voice service provider. Any calls I make to another OnSIP subscriber complete in HD Voice (G.722 codec), because I have provisioned my phones to prefer this codec. Calls I make to phone numbers (E.164 numbers) that don’t belong to OnSIP complete in narrowband (G.711 codec), because OnSIP has to route them over the PSTN. If OnSIP was able to use an IP address for these calls instead of an E.164 number, it could avoid the PSTN and keep the call in G.722.

Xconnect has just announced an HD Voice Peering initiative, where multiple voice service providers share their numbers in a common directory called an ENUM directory. When a subscriber places a call, the service provider looks up the destination number in the ENUM directory; if is there, it returns a SIP address (URI) to substitute for the phone number, and the call can complete without going over the PSTN. About half the participants in the Xconnect trial go a step further than ENUM pooling: they interconnect (“peer”) their networks directly through an Xconnect router, so the traffic doesn’t need to traverse the public Internet. [See correction in comments below]

There are other voice peering services that support this kind of HD connection, notably the VPF (Voice Peering Fabric). The VPF has an ENUM directory, but as the name suggests, it does not offer ENUM-only service; all the member companies interconnect their networks on a VPF router.

Some experts maintain that for business-grade call quality, it is essential to peer networks rather than route over the public Internet. Packets that traverse the public Internet are prone to delay and loss, while properly peered networks deliver packets quickly and reliably. In my experience, this has not been an issue. My access to OnSIP and to Vonage is over the public Internet, and I have never had any quality issues with either provider. From this I am inclined to conclude that explicit peering of voice networks is overkill, and that if you have a VoIP connection all that is needed for HD voice communication is to list your phone number in an ENUM directory. Presumably the voice service providers in Xconnect’s trial that are not peering share this opinion.

Xconnect’s ENUM directory is enormous, partly because it is pooled with Pathfinder – the GSMA ENUM directory administered by Neustar. Xconnect’s ENUM directory had over 120 million numbers in it as of 2007.

Xconnect and the VPN only add to their ENUM directories the numbers owned by their members. But even if you are not a customer of one of their members, you can still list your number in an ENUM directory, e164.org. This way, anybody who checks for your number in the directory can route the call over the Internet. Calls made this way don’t need to use SIP trunks, and they can complete in HD voice.

If you happen to have an Asterisk PBX, you can easily provision it to check in a list of ENUM directories before it places a call.

HD Voice – state of deployment

At the HD Voice Summit in Las Vegas last week, Alan Percy of AudioCodes gave a presentation of the state of deployment of HD Voice, citing three levels of deployment: announced interest, trials and service deployment.

Percy’s take was that in the “Crossing the Chasm” technology adoption lifecycle, HD Voice is right at the chasm.

Here is his list, augmented with input from Jan Linden of GIPS,Tom Lemaire of FT/Orange, Doug Mohney of HD Voice News and Dave Erickson of Wyde Voice:

Category Company Stage
PC VoIP Skype >500 m downloads
QQ (China) >500 m downloads
Gizmo5 (now Google)
Wireline telco France Telecom 500K HD users
British Telecom Trials
FT/Orange Spain Deployed 1Q09
FT/Orange Poland Deploys 1Q10
Mobile Orange (Moldova) Production
Orange (UK) Deploys 3Q10
Orange (Belgium) Deploys 2010
CLEC VoIP Alteva Production
SimpleSignal Production
Ooma 25K HD users
8×8 >70K HD users
OnSIP Production
Phone.com Trials
US MSOs CableVision/Lightpath Limited Trials
Conferencing ZipDX Production
ClearOne Production
Citrix Production
FreeConferenceCall.com Production
Global Crossing Limited Trials

The main codecs in each of these deployments are: Skype:SILK; QQ, Citrix, Freeconferencecall:iSAC; mobile:AMR-WB; all others: G.722.

Alan pointed out the conspicuous lack of involvement of the cable companies (MSOs), even though Cable Labs has done a good job of creating HD specifications for them.

HD Voice Numbers

50% of consumers say they would change their telephone service provider to get better sound quality, according to Tom Lemaire, Sr. Director of Business Development at Orange/France Telecom North America, speaking at the CES HD Voice Summit this week (Orange/France Telecom has the largest deployment of HD Voice of any traditional telco). Rich Buchanan, Chief Marketing Officer at Ooma, said at the same session that his surveys show that 65% of consumers would change their provider to get better voice quality.

Bearing in mind that we know from observation that consumers value both mobility and price above call quality, these survey numbers fall into the “interesting if true” category.

Lemaire and Buchanan also said that their logs show that the average call in HD lasts longer than the average narrowband call, though they didn’t give numbers.

VoIP on the cellular data channel

In a recent letter to the FCC, AT&T said that it had no objection to VoIP applications on the iPhone that communicate over the Wi-Fi connection. It furthermore said:

Consistent with this approach, we plan to take a fresh look at possibly authorizing VoIP capabilities on the iPhone for use on AT&T’s 3G network.

So why would anybody want to do VoIP on the cellular data channel, when there is a cellular voice channel already? Wouldn’t voice on the data channel cost more? And since the voice channel is optimized for voice and the data channel isn’t, wouldn’t voice on the data channel sound even worse than cellular voice already does?

Let’s look at the “why bother?” question first. There are actually at least four reasons you might want to do voice on the cellular data channel:

  1. To save money. If your voice plan has some expensive types of call (for example international calls) you may want to use VoIP on the data channel for toll by-pass. The alternative to this is to use the voice channel to call a local access number for an international toll by-pass service (like RebTel.)
  2. To get better sound quality: the cellular voice codecs are very low bandwidth and sound horrible. You can choose which codec to run over the data network and even go wideband. At IT Expo West a couple of weeks ago David Frankel of ZipDX demoed a wideband voice call on his laptop going through a Sprint Wireless Data Card. The audio quality was excellent.
  3. To get additional service features: companies like DiVitas offer roaming between the cellular and Wi-Fi networks that makes your cell phone act as an extension behind your corporate PBX. All these solutions currently use the cellular voice channel when out of Wi-Fi range, but if they were to go to the data channel they could offer wideband codecs and other differentiating features.
  4. For cases where there is no voice channel. In the example of David Frankel’s demo, the wireless data card doesn’t offer a voice channel, so VoIP on the data channel is the only option for a voice connection.

Moving on to the issue of cost, an iPhone unlimited data plan is $30 per month. “Unlimited” is AT&T’s euphemism for “limited to 5GB per month,” but translated to voice that’s a lot of minutes: even with IP packet overhead the bit-rate of compressed HD voice is going to be around 50K bits per second, which works out to about 13,000 minutes in 5GB. So using it for voice is unlikely to increase your bill. On the other hand, many voice plans are already effectively unlimited, what with rollover minutes, friend and family minutes, night and weekend minutes and whatnot, and you can’t get a phone without a voice plan. So for normal (non-international) use voice on the data channel is not going to reduce your bill, but it is unlikely to increase it, either.

Finally we come to the issue of whether voice sounds better on the voice channel or the data channel. The answer is, it depends on several factors, primarily the codec and the network QoS. With VoIP you can radically improve the sound quality of a call by using a wideband codec, but do impairments on the data channel nullify this benefit?

Technically, the answer is yes. The cellular data channel is not engineered for low latency. Variable delays are introduced by network routing decisions and by router queuing decisions. Latencies in the hundreds of milliseconds are not unusual. This will change with the advent of LTE, where the latencies will be of the order of 10 milliseconds. The available bandwidth is also highly variable, in contrast to the fixed bandwidth allocation of the voice channel. It can sometimes drop below what is needed for voice with even an aggressive variable rate codec.

In practice VoIP on the cellular data channel can sometimes sound much better than regular cellular voice. I mentioned above David Frankel’s demo at IT Expo West. I performed a similar experiment this morning with Michael Graves, with similarly good results. I was on a Polycom desk phone, Michael used Eyebeam on a laptop, and the codec was G.722. The latency on this call was appreciable – I estimated it at around 1 second round trip. There was also some packet loss – not bad for me, but it caused a sub-par experience for Michael. Earlier this week at Jeff Pulver’s HD Connect conference in New York, researchers from Qualcomm demoed a handset running on the Verizon network using EVRC-WB, transcoding to G.722 on Polycom and Gigaset phones in their lab in San Diego. The sound quality was excellent, but the latency was very high – I estimated it at around two seconds round trip.

The ITU addresses latency (delay) in Recommendation G.114. Delay is a problem because normal conversation depends on turn taking. Most people insert pauses of up to about 400 ms as they talk. If nobody else speaks during a pause, they continue. This means that if the one-way delay on a phone conversation is greater than 200 ms, the talker doesn’t hear an interruption within the 400 ms break, and starts talking again, causing frustrating collisions.
The ITU E-Model for call quality identifies a threshold at about 170 ms one-way at which latency becomes a problem. The E-Model also tells us that increasing latency amplifies other impairments – notably echo, which can be severe at low latencies without being a problem, but at high latencies even relatively quiet echo can severely disrupt a talker.

Some people may be able to handle long latencies better than others. Michael observed that he can get used to high latency echo after a few minutes of conversation.

HD Communications Project

As part of the preparation for the fall HD Communications Summit, Jeff Pulver has put up a video clip promoting HD Voice for phone calls. It goes over the familiar arguments:

  • Sound quality on phone calls hasn’t improved since 1937. Since most calls are now made on cell phones, it has actually deteriorated considerably.
  • The move to VoIP has made it technically feasible to make phone calls with CD quality sound or better, yet instead VoIP calls are usually engineered to sound worse than circuit-switched calls (except in the case of Skype.)
  • Improved sound quality on phone calls yields undisputed productivity benefits, particularly when the calls involve multiple accents.
  • Voice has become a commodity service, with minimal margins for service providers, yet HD Voice offers an opportunity for differentiation and potentially improved margins.

The HD Communications Summit is part of the HD Connect Project. The HD Connect Project aims to provide a coordination point for the various companies that have an interest in propagating HD Voice. These companies include equipment and component manufacturers, software developers and service providers.

Among the initiatives of the HD Connect Project is a logo program, like the Wi-Fi Alliance logo program. The logo requirements are currently technically lax, providing an indicator of good intentions rather than certain interoperability. Here’s a draft of the new logo:

HD Connect Draft Logo

Another ingredient of the HD Connect project is the HDConnectNow.org website, billed as “the news and information place for The HD Connect Project.”

It is great that Jeff is stepping up to push HD Voice like this. With the major exception of Skype almost no phone calls are made with wideband codecs (HD Voice). Over the past few years the foundation has been laid for this to change. Several good wideband codecs are now available royalty free, and all the major business phone manufacturers sell mostly (or solely) wideband-capable phones. Residential phones aren’t there yet, but this will change rapidly: the latest DECT standards are wideband, Gigasets are already wideband-capable, and Uniden is enthusiastic about wideband, too. As the installed base of wideband-capable phones grows, wideband calling can begin to happen.

Since most dialing is still done with old-style (E.164) phone numbers, wideband calls will become common within companies before there is much uptake between companies. That will come as VoIP trunking displaces circuit-switched, and as ENUM databases are deployed and used.