Freeswitch HA mode based on Keepalived.

Idea to have 2 instances of Freeswitch in Master-Slave mode, if one node dies, second will automatically takeover calls with 2-3 seconds lag. Yep, media would be restored.

Main scheme is looks like this

We need 3 IP's. 2 for nodes and 1 for floating.

Solution based on this, but with a bit of additional config.
Used Debian 8x64 as a host OS.
Assume, that we have FreeSWITCH'es already installed.

Install keepalived on both nodes.

apt-get install keepalived

Create files on both nodes

/etc/keepalived/keepalived.conf

global_defs {
    router_id FREESW
}

vrrp_script chk_fs {
    script "/etc/keepalived/scripts/ka-status.pl"
    interval 1
}

vrrp_instance VI_FREESW {
    # replace with SLAVE on a slave node
    state MASTER
    # change to SLAVE on slave node
    interface <YOUR_INTERFACE_HERE>
    #interface eth0
    virtual_router_id <YOUR_ROUTER_ID>
    # virtual_router_id 15
    # higher is preferred for master
    # disable to have failover be sticky
    priority 1
    advert_int 1
    unicast_src_ip <CURRENT_NODE_IP>
    #unicast_src_ip 192.168.10.10
    unicast_peer {
        <SLAVE_NODE_IP>
        #192.168.10.11
    }
    authentication {
        auth_type PASS
        auth_pass YourPassHere
    }
    notify "/etc/keepalived/scripts/ka-notify.pl"
    virtual_ipaddress {
        <FLOATING_IP/CIDR> dev <YOUR_INTERFACE_HERE>
       # 192.168.0.15/24 dev eth0 - Example
    }
    track_script {
        chk_fs
    }
}

/etc/keepalived/scripts/ka-notify.pl
#!/usr/bin/perl

# INSTANCE|VI_FREESW|BACKUP|50
my ($what,$id,$state,$prio) = @ARGV;
open(STDOUT, "|/usr/bin/logger -t ka-notify");

print "what($what) id($id) state($state) prio($prio)\n";

if ( $state eq "MASTER" )
{
    print "Instance went to master, issuing sofia recover.\n";
    system("/usr/bin/fs_cli", "-x", "sofia recover");
}


/etc/keepalived/scripts/ka-status.pl
#!/usr/bin/perl

use Sys::Syslog;
openlog "ka-status", "ndelay,pid", "local0";

my @required = ("internal", "external");

my %saw = ();
open(my $in, "-|") || exec("/usr/bin/fs_cli", "-x", "sofia xmlstatus");
while ( defined(my $line = <$in>) )
{
    if ( $line =~ m|<name>(.*)</name>|o )
    {
        $saw{$1} = 1;
    }
}
close($in);

foreach my $profile ( @required )
{
    if ( ! $saw{$profile} )
    {
        syslog(LOG_INFO, "sip profile $profile not found, marking failure");
        exit(1);
    }
}
exit(0);


chmod +x /etc/keepalived/scripts/*.pl
echo "net.ipv4.ip_nonlocal_bind = 1" >>  /etc/sysctl.conf 

Make sure, that FreeSWITCH'es uses same runtime database and uses same hostname in switch.conf Also profiles you control, need to listen on FLOATING_IP address.
Better reboot here.

service keepalived start

To check state, best is to use ip addr show. On which node FLOATING_IP address is, this is master :)

You can play with priority parameter in keepalived.conf file to make one node Master at all cases.
Point, there is a issue, when you restore calls on one node, you can't get same calls back with sofia recover.

Vultr HA. Some additions to existing article.

So, now I've switched from DigitalOcean completely to Vultr. To be honest - mostly of price, cause it's a bit cheaper. And accepts debit cards.
And also had a chance to set up HA on this cloud provider followed on this article.
But some additions to it for complete algo.
1. Order a BGP with support ticket with Vultr provided AS.
2. Purchase a Reserved IP address, but not assign it to none of instances
3. Order 2 instances in same location
4. Set up network on both instances according to article.
 4.1 Use Reserved IP address as dummy1 interface IP
 4.2 Use /32 subnet mask regardless on what you see in Reserved IP settings.
 4.3 Use Bird settings from server instance BGP page (BGP configuration link) not to mistaken with AS addresses and passwords.

FusionPBX(Freeswitch) HA sync gateways across servers

There was a little task to do.
Assume, we have several FusionPBX in HA (shared database incl. runtime data is on). On "master managing" server we add a gateway to external profile. To add them on other servers we have to go to each server and push rescan on external profile. Eh... To lazy for this.
We will use script, that will subscribe to CUSTOM sofia::gateway_add and gateway_delete event and run sofia profile external rescan over other servers.
We will do this over xmlrpc, so no need to add ssh keys or so.

"Slave" servers:

• Go to Advanced -> Modules and enable XML RPC module
• Go to Advanced -> Settings on each "slave" server and put common data for XMLRPC connection, like username, pass and port
• Restart mod_xml_rpc on each "slave" server

"Master" server

• Create files /usr/local/freeswitch/scripts/app/gateway_states/resources/gateway_states_add.lua and /usr/local/freeswitch/scripts/app/gateway_states/resources/gateway_states_delete.lua

(I can't get add and delete event work from a single file)

• gateway_states_add.lua

--description

--monitor custom sofia:gateway_* event and run sofia profile external rescan over other servers

--protect xmlrpc using a firewall on the server to limit access by ip address

--iptables

-- /sbin/iptables -I INPUT -j ACCEPT -p tcp --dport 8080 -s x.x.x.x/32

--define the servers running freeswitch do not include local

servers = {}

x = 0;

servers[x] = {}

servers[x]['username'] = "freeswitch";

servers[x]['password'] = "work";

servers[x]['hostname'] = "10.0.0.2";

servers[x]['port'] = "8080";

x = x + 1;

servers[x] = {}

servers[x]['username'] = "freeswitch";

servers[x]['password'] = "work";

servers[x]['hostname'] = "10.0.0.3";

servers[x]['port'] = "8080";

--x = x + 1;

--servers[x] = {}

--servers[x]['username'] = "ccc";

--servers[x]['password'] = "xxxxx";

--servers[x]['hostname'] = "127.0.0.3";

--servers[x]['port'] = "8080";

--subscribe to the events

--events = freeswitch.EventConsumer("all");

events = freeswitch.EventConsumer("CUSTOM", "sofia::gateway_add");

        -- events = freeswitch.EventConsumer("CUSTOM", "sofia::gateway_delete");

        -- Modify line above to get gateway_states_delete.lua

--prepare the api object

api = freeswitch.API();

--get the events

for event in (function() return events:pop(1) end) do

api_command_argument = "profile%20external%20rescan";
  -- event_gateway = event:getHeader("Gateway");
  -- api_command_argument = "profile%20external%20killgw%20"..event_gateway;
  -- Modify line above to get gateway_states_delete.lua

for key,row in pairs(servers) do

url = [[http://]]..row.username..[[:]]..row.password..[[@]]..row.hostname..[[:]]..row.port..[[/webapi/sofia?]]..api_command_argument;

response = api:execute("curl", url);

freeswitch.consoleLog("INFO", "[notice] ".. url .. " "..response.."\n");

end

end

• Modify /usr/local/freeswitch/conf/autoload_configs/lua.conf.xml

    <param name="startup-script" value="app/gateway_states/resources/gateway_states_add.lua"/>

    <param name="startup-script" value="app/gateway_states/resources/gateway_states_delete.lua"/>

• Restart "master" server. At this point, every gateway add or delete will trigger an event, that will go across all our servers and command them pull new config from common database.

This script was modified from memcache.lua, that flushes memcaches over all Fusion servers.