Being a developer with a focus on SIP I take a lot of interest in what’s going on in the Voice 2.0 space (I actually dislike the whole Web/Voice 2.0 terms but they get the message across). Over the last 6 months I would say there has been a marked pick up in activity with new web-based voice services coming out on an almost weekly basis. Two years ago you would have been lucky if could provision a SIP account online or update your voicemail to email address. These days drag and drop GUIs for building IVRs and REST APIs for managing voice resources are becoming the norm.

This morning I came across a blog article that waxed lyrical about a new piece of software from a company called Twilio. The post was very complimentary towards the software but it was two claims that the post made that roused my skepticism. First it claimed that the software was what Asterisk should have been. The second claim was that Twilio and Voxeo are competing neck and neck for developer mind share.

I have a love hate relationship with Asterisk. In a previous existence it allowed me to start a service provider business and I lived and breathed Asterisk for 4 years. When I started Asterisk was pre its 1.0 release so it really was frontier stuff and great fun. As the business grew the shortcomings of Asterisk became more and more acute and the disinterest of the Digium developers to fix defects that were critical for providers using Asterisk and to incorporate features that were needed was extremely frustrating. Don’t get me wrong I wasn’t asking for a free ride and was looking to either sponsor developement or do it myself but in the ned I couldn’t get buy in from Digium and patches that don’t get absorbed into the main source tree wither and die. All that aside there is no doubt whatsoever that Asterisk is a brilliant product that I’m pretty sure is responsible for enabling more people on the planet to adopt VoIP than any other piece of software except Skype. So to claim that a new piece of software is better than Asterisk is a big claim. I’ve had a quick look over the new Twilio product. I also downloaded it but didn’t install it as the dependencies would take some time to set up. In essence it seems like a sort of client web application to the Twilio service offering albeit requires PHP and MySQL to run. It doesn’t process media let alone ISDN/SS7/MGCP etc. etc. as Asterisk does and in fact appears intended to operate with Twilio’s service as the only provider of incoming and outgoing calls. I can’t pass judgement on how good Twilio’s new software is but I can very confidently say that it’s not even close to the capabilities and flexibility of Asterisk and to claim it is what Asterisk should have been is completely absurd.

As I’ve posted a few times previously Voxeo’s Tropo service is the best voice application development platform available. The feature set and depth it gets from building on top of Voxeo’s Prophecy platform coupled with the fact that it’s very developer friendly with usage being completely free pre-production mean it’s very impressive to work with. It’s not perfect by any means and I’m more than happy to moan about the fact that they don’t allow SIP calls to be easily taken back once sent to them and thereby leave the meter running at $0.03 per minute. But at least they support SIP which is something Twilio don’t do. The majority of developers writing voice applications are going to focus on SIP. Why? Because Skype aside it’s the protocol with the biggest user base and developers develop for users. Sure some applications will be better suited by the narrower constraints and pre-canned features offered by Twilio and equivalent services, of which there are quite a few, but if there’s an alternative service that’s cheaper to develop on, more open in its protocols and more powerful in its features where are developers likely to end up. I know of a number of sipsorcery users that have messed around with bits and pieces on Tropo and even come up with some pretty powerful applications. I doubt there are any sipsorcery users who have written an application on Twilio and if they have they would have to go out onto the PSTN to be able to use it. So while Twilio and Voxeo may be slightly closer than Twilio’s new software is to Asterisk they are still a very very long way apart. Rather than neck and neck Twilio is not even on the same lap.

The title of this post about something funny being in the ether is because the blog post about Twilio’s new software was not isolated and in fact it cropped up on 2 other voice/VoIP related blogs I follow. I only follow about 10 so that’s a 30% coverage rate which is pretty impressive and one of those 3 blogs linked off to another article so that’s 4 seemingly independent posts within 24 hours. The other blogs didn’t make the same absurd claims as the ones discussed above but they did give glowing reviews to what is not exactly a revolutionary product. There have definitely been more worthy recipients of such coverage in the recent past, Anveo’s Call Builder tool or even Voxeo’s biometrics integrations to name just two, but they generally didn’t rate a mention. I suspect such glowing coverage stems from a bit of a chummy relationship between the influential bloggers and Twilio. Personally I don’t have a problem with that, I’m happy enough to run my eyeballs over any new voice related service or product, but if a blogger is going to claim that a product is good enough to pants both Asterisk and Voxeo then I’d expect something pretty special. That is definitely not the case here. I wish Twilio all the success in the World and hope that they will one day support SIP but I’m guessing at this point their marketing skills are far outweighing their technical ones.

Update 22 Jun 2010: Stumbled across another new voice application service provider called Teleku that seem to have not only duplicated Twilio’s API but have also put up instructions on how to commandeer their new open source product; it’s getting nasty out there.

Over the last 4 months I’ve been working on a new sipsorcery client application. It’s really the culmination of work that’s been going on for a lot longer than that and in fact I tried to write this sort of application 3 years ago in 2007. At that time the server software, which was then known as mysipswitch, wasn’t really up to the task and the attempt to write the client using AJAX also came up short due to limitations with javascript and the browser sandbox.

MySIPSwitch Call Management Screen Shot

And I almost forgot about the very crude prototype effort I did in August last year.

The application will be released for beta testing next week but I thought I’d provide a little taster of the interface in advance.

The client application is a SIP switchboard that leverages the power and flexibility of the Ruby dialplans as well as the ability of the sipsorcery SIP stack to be able to run entirely within a Silverlight browser application. The sole goal of the SIP Sorcery Switchboard is to manage inbound calls in much the same manner as many advanced PBX phone systems allow receptionists to do. Being a web application the SIP Sorcery Switchboard can do a lot of things a push button phone based switchboard can’t.

A word of warning as well. The switchboard application will be the first time that the sipsorcery project is going to offer a paid service offering. In other words while the beta will be free when the application goes into production there will be a charge for it and the intention is that a portion of the proceeds will go into supporting the infrastructure for the free service.

Anyway more details will be forthcoming next week. For the time being here’s a screenshot of the SIP Sorcery switchboard processing a couple of calls (the icons were whipped by me on my tablet and are for the beta test only, the final version will have a proper graphic designer’s polish).

SIP Sorcery Switchboard User Interface

Transfer functionality was added to sipsorcery at the start of the year. I’ve also previously blogged about using blind transfers with Tropo and other similar SIP application services using a roundabout HTTP mechanism. Now for the first time I’m able to post about one advanced SIP application service that supports blind transfers the way it should be done; using SIP REFER requests.

The service is Anveo and even though at the time of writing they are the newest SIP application service to arrive on the scene their offering is extremely polished and at least as far as my own testing over the last week very robust. The really neat feature and key innovation that Anveo has come up with is their graphical tool for building voice call flows, possibly the term IVR could be used but IVR’s are typically associated with “press 1 for sales, 2 for support…” and Anveo’s tool lets you build solutions way beyond that. The only other tool I know of that’s comparable to Anveo’s Call Flow Builder is Voxeo’s Prophecy Designer. I must admit I haven’t played around much with the Prophecy Designer as I have with Voxeo’s other offering, Tropo, but from what I have seen of it it’s much more geared towards building a traditional IVR and collecting input from callers using speech recognition grammars and DTMF. Anveo’s solution on the other hand seems more designed to do as its name suggests and build a call flow that provides a more diverse range of ways to process the call rather than just collecting input. In a lot of cases, traditional IVR’s for example, collecting input is exactly what’s required and I have no doubt Voxeo’s platform are one of if not the best in that regard.

Back to the blind transfer story. As pointed out in the blind transfers with Tropo post the desire to use blind transfers stems from not wanting to stay in complete control of the media path of the call irrespective of whether it’s for cost, quality, reliability or some other concern. It’s a powerful concept to be able to receive a call, send it off to a server somewhere to do some processing and then take the call back and perhaps send it off to a different server with different features somewhere else before eventually forwarding it to its final destination. That sort of scenario is not very easy to accomplish with most VoIP providers where they will allocate an incoming DID and if they do allow it to be forwarded to a SIP URI very rarely provide any facility to transfer the call to a different SIP URI. A simple example would be a DID provider receiving an incoming call that gets sent to an IVR provider which asks who the caller would like to speak to, once the caller has provided their choice the call goes back to the DID provider who forwards it directly to a SIP client or if the the person required is busy the call gets forwarded to a voicemail provider who takes a message and transcribes and delivers it via email. Nothing particularly fancy there except that the call involved 3 different providers, trying to accomplish that in the real World would be a challenge. However if each of the providers supported transfers, which really are a fairly fundamental concept, it would be simple. The problem is most providers don’t support transfers and if they do it’s normally an attended transfer meaning they always want to keep the call on their system, usually so they can keep charging you by the minute for simply bridging the media. Enough of that and back to Anveo who are a provider that will allow you to take your call back without having to resort to any HTTP or other trickery.

Below is the very rudimentary call flow I built in Anveo. I copied one of the existing flows and then after seeing how it worked was able to delete the unwanted items and add it the ones I wanted to test in no time. So far I’ve had to spend no time at all fighting with the interface which is a testament to a good design. The flow works by answering the call, playing some arbitrary message using the Robotalk control, then prompting the user to enter an extension and then based on that extension the call is either forwarded to a SIP URI, sent to voicemail or a blind transfer is initiated.

Anveo Call Flow Builder with SIP transfer control

Forwarding to a SIP URI is what this post is about avoiding since it introduces all the shortcomings outlined in the paragraph above. However sometimes forwarding to a SIP URI is what’s required and it’s certainly good to have the control available. However the really cool control is the transfer control.

SIP transfer control configuration

This control will send an in dialogue REFER request back to the server. The sipsorcery server can then decide that the call leg to Anveo is finished and that the call should now be sent off somewhere else. The key point is that the incoming call leg between a SIP client and sipsorcery can stay oblivious to this and therefore doesn’t need to be configured to handle the transfer. The SIP clients I have played around with usually have quirks with processing transfers, for example when my Cisco 7960 phone gets a REFER request it will place a call to the URI specified in the Refer-To header as expected but if the REFER request is for line 2 on the phone it will always place the call on line 1 which is a problem as line 1 and line 2 are connected to different servers. That sort of issue can be avoided by having the sipsorcery server handle the REFER processing and all that the client will get is a re-INVITE request to inform it where it should switch its RTP to.

If you’re thinking about playing around with blind transfers you need to read the next paragraph.

Like any good saga there is a dark side to blind transfers and that’s undoubtedly got a lot to do with why there is so little support for them amongst commercial SIP providers, well that and the fact that Asterisk, the software of choice for a lot of providers, is too poorly designed to be able to correctly generate CDRs for transfers despite literally years of attempts but I digress. Back to the dangers of blind transfers, consider that I could be sitting at my desk minding my own business when I receive a SIP call over the internet that shows up on my phone as a friendly name. I answer the call and the next thing I hear is one ring and then a sultry voice informs me I’ve called the XXX hotline and am being billed $10/minute. What happened? The person or machine calling me waited until I answered and then immediately sent a REFER request with a Refer-To header containing the XXX hotline number. My phone obediently processed the REFER request and dialled the number through my VoIP provider. My VoIP provider would not see anything different about the call so happily connected it. Before I knew it I’d been pinged for a call that I didn’t want or authorise. It’s worth noting that the same problem applies to SIP 3xx redirect responses which incidentally are another thing that VoIP providers tend to block and is why the “forward when busy” of “forward all” type buttons on IP phones often don’t have the desired outcome.

As a consequence of the danger blind transfers pose sipsorcery does not accept them by default. If the sipsorcery receives a REFER request and the call leg that created the call did not have an explicit option to allow transfers then it will be rejected.

Anveo REFER request being blocked.

To allow transfers and have them passed through to the SIP device at the other end of the call use the dial string [tr=p] option as shown below. Note that while using the pass thru transfer mode is safe enough with Anveo since the call flow control specifies the transfer destination extra caution is needed when using this transfer mode where the other end of the call is untrusted, see the paragraph above about the dark side of blind transfers.


Anveo REFER request being passed through to calling SIP device

To process the blind transfer on the sipsorcery server a dial string option of [tr=c] should be used AND a new dialplan called transfer must be created. The transfer dial plan is where the Refer-To header will be sent for processing. It’s essentially the same as any other sipsorcery dialplan but the key thing to be aware of to be extra careful what destinations are allowed.


The transfer dialplan:

sys.Log("blind transfer starting...")
case req.URI.User
  when /^hold$/ then sys.Dial("")
  else sys.Dial("#{req.URI.User}@local")

Anveo REFER request processed on sipsorcery server.

To summarise the new SIP Transfer call flow control from Anveo opens up some very powerful and flexible calling scenarios for sipsorcery users but it’s absolutely essential to have a good understanding of how blind transfers work and the risks involved before rushing headlong into it. Being able to use the Anveo control is all about providing extra choices when it comes to building a voice application. At the time of writing it costs $0.08 to use the control which compares with $0.03 per minute to have a call bridged on Tropo or Cloudvox. Arguably if it’s likely to be a short call and all else is equal it may be better not to transfer the call and avoid the $0.08 but if the caller is likely to be put on hold for 5 minutes waiting for an agent then it’s a saving to do the transfer. Extra choices can only be good.

While Tropo is probably still a more powerful all round tool it’s really only suitable for programmers. Anveo’s Call Flow Builder attempts to cross the chasm to non-programmers and let them quickly build voice applicationsp; in a way it reminds me somewhat of what Frontpage did for building web sites by removing the need for people to know HTML.

Special thanks and kudos to Denis Chukhryaev, the founder of Anveo, for both agreeing to come up with a blind transfer control and churning it out in less than a week! I’ve been very impressed with the Anveo Call Flow Builder so far and fully expect to see it take off and become a big success.

A new dedicated server has now been purchased and is ready to take over the sipsorcery service from the flakey Amazon EC2 server instances. The sipsorcery DNS records will be updated in approx 8 hours at 24 May 2010 00:00 UTC. There should be minimal disruption unless anyone has hard coded in an IP address of one of the existing servers.

The new sipsorcery deployment consists of a single dedicated server and the IP address is

Even though the new deployment is going back to a single server deployment I do anticipate that it will be significantly more reliable than the existing dual Amazon EC2 virtual server deployment. As discussed in numerous previous posts the EC2 virtual Windows instances are particularly flakey and on average fail 2 to 3 times per week. Before moving to Amazon’s EC2 the precursor to sipsorcery, the mysipswitch service, was hosted on a single dedicated server which did not suffer a single operating system failure in well over a year.

In the event that there is a failure the sipsorcery service can be installed and configured from a new Windows server in approximately 30 minutes. The new hosting provider has a 100% service level agreement on hardware and network and grants 100 minutes of credit for every 1 minute of downtime. That doesn’t mean failures won’t happen but it means they are going to be doing everything they can to avoid any outages and should the sipsorcery dedicated server fail it will be repaired or an alternative server will be provided very quickly.

The sipsorcery web site has been moved to a new host and DNS updated. The old site which is hosted on the Windows Azure cloud is still running but I will be shutting it down in a day or two once the DNS entry has had time to fully propagate. In addition the sipsorcery Silverlight client has been updated to Silverlight version 4 which means unless you’ve already updated the Silverlight plugin you will get prompted to do so next time you visit the sipsorcery web site. The update should be completely painless and take less than a minute.

These two changes are not related to yesterday’s IP address dramas or the future move to a new host for the sipsorcery SIP servers. They are related to some work I have been doing on a new product for sipsorcery and also the relatively high running costs of Windows Azure.

Apart from the Silverlight update there should be no impact to people from these two changes.

Update: The issue with the new account confirmations page failing has now been fixed.

Update: After allocating the new IP addresses I could not get the server on to consistently acquire a database connection. I tried rebooting it several times, swapping the IP address to the alternative server, waiting for over an hour but all to no avail. In the end I ditched the IP address and allocated a new one which is so far working fine. Just another quirk of EC2 I guess. The IP addresses for sip1 and sip2 shown below have changed from what was originally posted.

Thirty minutes ago I attempted the straight forward operation of swapping the IP addresses on sip1 and sip2 so as to do an operating system and software upgrade on sip1 (sip1 is by far the busier of the two servers). Unfortunately I clicked the “Release IP address” button instead of the “Disassociate IP address” button in the ElasticFox tool and I lost the two static IP addresses that the sip1 and sip2 servers were using. I was instictively staying away from the button with the big red X on it but in this case that was the correct button. The consequence of the malfunction by me is that the IP addresses of sip1 and sip2 will now be different. I’ve updated DNS and the since the A and SRV records were all set with timeouts of 1 hour they should have already started to propagate. At 1000, about half the normal number, device registrations are back and calls are going through.

The new IP addresses for anyone that needs to hard code them into their ATA or provider settings are: and both resolve to sip1.


Many thanks to the nearly 50 people who donated what they could towards the proposed new hosting trial. The amount rasied was USD $825 which unfortunately didn’t quite get to the amount of USD $1169.89 needed for the first month’s installment so it won’t be possible to proceed with the original proposal in full.

Since the donations started another development has meant that the current sipsorcery hosting arrangements need to be halted at the end of this month. Consequently it’s now my intention to use the donations to purchase a single dedicated server with an alternative host and run the sipsorcery service from it. The donations total should cover the set up cost and the first two months and if a little more trickles in even the first three months. I am working on a plan to move the service to a self sustaining footing but for at least the next few months the donations will be critical.

The plan would be to over time incorporate the F5 load balancing and second server into the hosting arrangements, as funding permits, and enhance the the service and hopefully get to the magical five nines reliability. For the interim I’m confident that running the service from a single dedicated server will actually be a substantial improvement on the current situation from the current Amazon virtual servers which are now at a point where they are going down so often I’ve given up even keeping a log of them.

This new arrangement is a deviation from the original proposal for donations so if anyone would like a refund please email me at and I will be happy to oblige. I would ask that refund requests are made within the next week by Sunday the 23rd of May otherwise the money will have been committed.

I have been researching an alternative provider to Amazon’s EC2 and have identified a promising option. Unfortunately the sipsorcery sponsorship only covers Amazon’s EC2 so an alternative source of funding would need to be found to move hosting providers. To that end I have put up a Donations page which runs through the costs and arrangements.

One of the questions the sipsorcery project has been aimed at answering is whether it’s possible to run a telco in the cloud. By cloud I mean a public platform offering on demand resources at an operating system or database level. The sipsorcery service moved from a single server hosted in private infrastructure to a single server in the Amazon EC2 cloud in July 2009 from which point on the sipsorcery service could be said to have become a cloud hosted telecoms service. For a service like sipsorcery the advantages of operating in the cloud were envisaged primarily as cost, scalability and reliability.

Within about 6 weeks of migrating to the EC2 cloud, and after sorting out a few software issues within the sipsorcery code, the Amazon EC2 failures started and have continued ever since. The symptoms of the failure were that the underlying Windows virtual server hosting the sipsorcery software just dropped off the network and stopped responding to any type of network traffic: pings, RDP, HTTP, SIP, everything. Initially I suspected it must have been a problem in the sipsorcery software but after a lot of effort over 3 or 4 months and from keeping an eye on the EC2 forums for people having similar experiences I became suspicious that it was more than likely an issue with something related to Amazon’s infrastructure either at the network, firewall or host level. Since that initial suspicion I’m now 99% sure the problem is at the host level and that the network driver in the Xen virtualisation software is failing under certain conditions. The strongest evidence for this is that when a sipsorcery instance drops off the network it will miraculously start working again around 3 hours later assumedly after the underlying Xen software clears out its network connections cache or something.

Since 2006 I have invested a lot of time and effort in working with Amazon’s EC2 and S3 products and when the sipsorcery EC2 failures started causing a lot of pain I rationalised that an alternative cloud service provider would most likely have their own issues and it was better to stay with the devil you know. To that end I signed up to EC2’s premium support (at USD$100 per month) so I could log a ticket to resolve the issue. Unfortunately that proved fruitless and the advice I received was to try the latest Windows image provided by Amazon and see if that helped, reasonable advice but it the updated images exhibited exactly the same issue. I cancelled the premium support and resorted to vociferous whinging on the EC2 forum. After a month and half of that approach I eventually got an email from an Amazon engineer who did look into it for me and while not so much acknowledging that the issue was Xen related suggested I try a Windows image that was backed by Elastic Block Store (EBS) storage because it had some additional updated Xen drivers. So with high hopes I toddled off again and built up the sipsorcery image, which I had down to a fine art by now. It actually took a few weeks for the EBS backed instance to fail and I’d actually started to believe it had solved the problem but eventually it did fail and the Amazon engineer didn’t have any further ideas.

So back to the 3 advantages of operating in the cloud, the reliability of the service certainly diminished a lot after moving from a dedicated server but what about cost and scalability? To overcome the repeated failures of the sipsorcery EC2 instance a second failover instance was utilised meaning there were now two sipsorcery EC2 instances required. In addition the size of both the instances needed to be upgraded from small to medium to cope with the increase in sipsorcery users. And often during troubleshooting the failures it was necessary to leave one or two failed instances running in case Amazon ever wanted to check them. Suffice to say the average monthly cost for the sipsorcery EC2 servers is 5 or 6 times what had been originally forecast and is 2 or 3 times more than a dedicated server with equivalent specifications. That leaves scalability. The first scalability problem for sipsorcery was the database. In the initial single EC2 instance deployment a MySQL database sitting on the same instance was used, the very first instance failure and some a misconfiguration of the MySQL database by me resulted in the very first sipsorcery users losing their data. To scale the database a better deployment model was needed. About that time two new products came out, one was Amazon’s Relational Database Service (RDS) the other was Microsoft’s SQL Azure. Amazon’s RDS is based on MySQL and since it’s probably in the same data center as the EC2 infrastructure my preference was to use it. However it didn’t take long to realise it was a poor solution, there was no replication, no clustering the product was simply a single MySQL server sitting on its own instance which was not much different from what sipsorcery already had, no thanks. Microsoft’s SQL Azure was unsurprisingly based on Microsoft’s SQL server and was about 100ms of network away from the sipsorcery EC2 servers but it was still compelling because it claimed to solve two of the most difficult database problems by handling data replication for fault tolerance and having a deployment model that allowed it to scale demand across the SQL Azure cloud. But it was a new product and if the EC2 experience was anything to go by was it worth risking. In December of 2009 I did move the sipsorcery data to SQL Azure and while there have been a few issues that have caused outages to the sipsorcery service in general it has worked extremely well. Even more importantly on one of the issues that I experienced and posted on the SQL Azure forums about I got an email from the Microsoft SQL Azure product manager who got his engineers involved to identify the root cause. And in this case, unlike with Amazon, the Microsoft engineers were able to provide a good explanation and more importantly that particular issue hasn’t re-occurred.

Starting from November 2009 I have kept track of all the sipsorcery failures related to the “clouds” and while I won’t list them all a summary is interesting.

SQL Azure

  • Between 16 Dec 2009 and 25 Mar 2010 there were 22 detected outages totalling 26.9 minutes with the longest being 2 minutes and 43s,
  • On the 28th Mar 2010 an approximately 3 minute outage occurred,
  • On the 9th of April 2010 an approximately 3 hour outage occurred however it is not clear whether this was caused by a network issue or an SQL Azure issue. SQL Azure engineers investigated and found no evidence of a problem and during the outage I was able to connect to the database from outside the EC2 network.


  • Between 1 November 2009 and 13 Apr 2010 there were 28 detected outages totalling over 32 hours with the longest being 8.5 hours,
  • EC2 outages to date have not occurred simultaneously to both servers (sip1 and sip2) so the outage times apply only to a single server instance and not to the overall service,
  • EC2 outages require a server instance to be manually rebooted which takes a minimum of 15 minutes. I am notified of outages via SMS and email with 30s but depending on my circumstances remedial action can vary between instant and up to 8 hours, on average it’s generally less than 30 minutes.

To answer the original question relating to can a telco run in the cloud my answer is “probably” but if part of that cloud is Amazon’s EC2 then the answer is “probably not”. I know of another SIP based service that has recently started on Amazon’s EC2, unlike sipsorcery they are Linux based, so it will be interesting to follow their experience.

As for sipsorcery I’ve identified a promising alternative to EC2 that I hope to migrate the service to at some stage. One big advantage of this alternative provider is that they have F5 load balancers which would allow the sipsorcery service to be deployed reliably without having to depend on SIP SRV records which have been shown to be pretty poor as a failover mechanism for SIP clients, when a sip1 outage occurs approximately two thirds of the sipsorcery clients drop off and don’t fail over to sip2. However there are some funding challenges involved in migrating sipsorcery from EC2 and I need to come up with some way to pay for the new cloud host.


Over the last few weeks I have been working on implementing the SIP Event RFC’s, focusing on the dialog and presence event packages. The main motivation for the work is a new application I am working on but one useful side effect is that SIP user agents that support presence can now do so with sipsorcery SIP accounts.

SIPSorcery Presence