Sipsak → sipexer for small SIP tests

I think a lot of peoples who are working with SIP knows and use sipsak. It's a Swiss-army knife for SIP, as it states. But not anymore. In a way migrating to OpenSSL 3.0 and newer distributions proper TLS transport support was lost. Or, in another words, I cannot make it work on Alma 9.

 

So, meet a replacement - sipexer from Kamailio author. Doing mainly the same (for me) and supports TLS and WebSocket out of the box. As a bonus for simple testing, you can get SIP reply code as a return code of the program itself.

So, instead of checks like

 

sipsak -s sip:check_server_health@localhost:5061 --transport=tls

if [ $? != 0 ]; then

  restart_kamailio

fi

 

you can do 

 

sipexer -timeout-write 500 -timeout 500 -vl 0 -mt options -ruser check_server_health tls:localhost:5061 >> /dev/null

if [ $? != 200 ]; then

  restart_kamailio

fi

 

Quite a simple replacement and due to sipexer being written on Go, you can just get the binary and use it right away.

I like it when things become simpler.

Importance of creating dialog in Kamailio right

I know, the things described here might be obvious and considered "for beginners", but everyone can make stupid errors. Even the best of us :)
So, what I'm using Kamailio dialog module for?

• Keepalives with ka_timer parameter, mainly cause of the nature of mobile clients that are unstable due to the mobile network nature
• Presence PUBLISH'es via pua/pua_dialoginfo modules to separate presence server

With the second one actually, I've issued some "problems" due to configuration.

Most examples (at least what I've seen) on Kamailio configs, that are operating with the dialogs, have more-or-less this structure:

request_route {
   route(REQINIT);

   if (is_method("INVITE")) {
       dlg_manage();
   }

   route(WITHINDLG);

   ...

   route(AUTH);

   ...

}

There is a problem with this code. And it is, that this code creates dialogs for EVERY INVITE, even if not authed. 

So, the usual flow INVITE - 401 - INVITE (w/auth) - 200 will create 2 dialogs in this case. The first one will be not terminated correctly with connection with pua module. It will generate PUBLISH with the Trying state,  but on 401 - ACK, there would be no corresponding PUBLISH for the terminated state. And as usual, Expire here is 3600, and you will have a lot of "ringing" devices on your presence server.

The answer would be: create dialogs only for authed (and valid) INVITEs. This will also save some CPU not to create dialogs for spammers.

request_route {
   route(REQINIT);

   route(WITHINDLG);

   ...

   route(AUTH);

   if (is_method("INVITE")) {
       dlg_manage();
   }

   ...

}

Adding a bit of privacy to Kamailio's pua_dialoginfo module

Kamailio is a great product. With really good customisation possibilities.One of examples will follow here.

Task is the following: we have a setup with 2 different servers, proxy and presence server. Presence server is simple and just processing SUBSCRIBE/PUBLISH messages and generating NOTIFY's. Mainly for BLF's.  But users wants to have some privacy in a question, than not everyone can subscribe to anyone and more interesting - monitor who's calling whom. But some of the users, especially in boss-assistant scenarios want to have full information about calls. Like assistant of the boss usually should know who's calling to boss phone and answer (pickup) in some cases. 

Kamailio allows to achieve this scenario, but with a bit of customisation.

Idea is to have concept of 2 groups. 

Group1 for ACL about possibility of user monitor other user in general, second one - having ACL for seeing call details.

Here I'll point on idea with some code examples for proxy part. And just to mention, pickup functions are implemented not on proxy server, so it's not broken for me here.

Proxy

kamailio.cfg

loadmodule "pua.so"
loadmodule "pua_dialoginfo.so"
...
modparam("pua", "db_url", DBURL)
modparam("pua", "outbound_proxy", "sip:<PRESENCE_SERVER>:5060")
modparam("pua", "db_mode", 0)
...
modparam("pua_dialoginfo", "include_callid", 0)
modparam("pua_dialoginfo", "include_tags", 0)
# we want to have info on who's calling
modparam("pua_dialoginfo", "include_localremote", 1)
... 
# Table to allow subscription one on each other. If users share the same group - they can subscribe one to each other.
# By default all shares the same group = 1. If group == 0, that means nobody can see your state.
# Format is <extension>=><list_of_groups_comma_separated>
modparam("htable", "htable", 'presence_view_grp=>size=15;')
# Table to allow subscribers view each other call details data. By default it's forbidden.
modparam("htable", "htable", 'presence_view_details_grp=>size=15;')
# You can fill these tables via database or whatever else method you prefer

...
route(PRESENCE_PRIVACY);
route(WITHINDLG);
...

route(AUTH)
route(PRESENCE)
...
# Presence server route
route[PRESENCE] {
    if(!is_method("PUBLISH|SUBSCRIBE")) {
        return;
    }
    # Check if user is registered
    if (is_method("SUBSCRIBE") && !(registered("location", "$fu", 4) == 1)) {
        append_to_reply("Retry-After: 10\r\n");
        send_reply("500", "Retry Later");
        exit;
    }

   $du = "sip:<PRESENCE_SERVER>:5060";
   
    # By default - allow anyone subscribe to anyone.
    $var(presence_grp_from) = "1";
    $var(presence_grp_to) = "1";

    if ($sht(presence_view_grp=>$fU) != $null) {
        $var(presence_grp_from) = $sht(presence_view_grp=>$fU);
    }

    if ($sht(presence_view_grp=>$tU) != $null) {
        $var(presence_grp_to) = $sht(presence_view_grp=>$tU);
    }


    if (is_method("SUBSCRIBE")) {
        if ((str)$var(presence_grp_to) == "0") {
            # Presence group "0" means fully private.

            send_reply("404", "Subscriber not exists");
            exit;
        }

        $var(group_1) = $var(presence_grp_to);
        $var(group_2) = $var(presence_grp_from);

        if (!route(CHECK_GROUP_INTERSECTION)) {
            # No common groups for From/To
            send_reply("404", "Subscriber not exists");
            exit;
        }
    }

    route(RELAY);
}

route[PRESENCE_PRIVACY] {
    # We're processing only NOTIFY's with Dialog XML data here
    if (!is_method("NOTIFY") || !has_body("application/dialog-info+xml")) {
        return;
    }

    if ($sht(presence_view_details_grp=>$fU) == $null || $sht(presence_view_details_grp=>$tU) == $null) {
        # By default all call data is private
        route(PRESENCE_CLEAR_CALL_DETAILS);
        return;
    }

    $var(group_1) = $sht(presence_view_details_grp=>$tU);
    $var(group_2) = $sht(presence_view_details_grp=>$fU);

    if (!route(CHECK_GROUP_INTERSECTION)) {
        # No common groups for From/To
        route(PRESENCE_CLEAR_CALL_DETAILS);
        return;
    }

    # At this point all is ok.

}

route[PRESENCE_CLEAR_CALL_DETAILS] {

    # Forming new body of the presence info
    # For replace trick see https://lists.kamailio.org/pipermail/sr-users/2022-February/114185.html
    $xml(body=>doc) = $(rb{s.replace,xmlns=,xyzwq=});
   
    # Forming new XML
    $var(new_body) = "<?xml version=\"1.0\"?>\n";
    $var(new_body) = $var(new_body) + "<dialog-info xmlns=\"" + $xml(body=>xpath:/dialog-info/@xyzwq) + "\"";
    $var(new_body) = $var(new_body) + " version=\"" + $xml(body=>xpath:/dialog-info/@version) + "\"";
    $var(new_body) = $var(new_body) + " state=\"" + $xml(body=>xpath:/dialog-info/@state) + "\"";
    $var(new_body) = $var(new_body) + " entity=\"" + $xml(body=>xpath:/dialog-info/@entity) + "\">\n";
    $var(new_body) = $var(new_body) + "  <dialog id=\"" + $xml(body=>xpath:/dialog-info/dialog/@id) + "\"\n";
    $var(new_body) = $var(new_body) + " direction=\"" + $xml(body=>xpath:/dialog-info/dialog/@direction) + "\">\n";
    $var(new_body) = $var(new_body) + "    <state>" + $xml(body=>xpath:/dialog-info/dialog/state/text()) + "</state>\n";
    $var(new_body) = $var(new_body) + "  </dialog>\n</dialog-info>";

    replace_body_atonce("^.+$", $var(new_body));
}

route[CHECK_GROUP_INTERSECTION] {
# Checks if contents of $var(group_1) and $var(group_2) that are comma-separated list of groups have at least one in common
    $var(group_1_param_count) = $(var(group_1){s.count,,});

    while ($var(group_1_param_count) >= 0) {
        $var(group_1_param) = $(var(group_1){s.select,$var(group_1_param_count),,});

        if (in_list("$var(group_1_param)", "$var(group_2)", ",")) {
            return 1;
        }
        $var(group_1_param_count) = $var(group_1_param_count) - 1;
    }
    return -1;
}

Idea here is we're stripping local/remote data from NOTIFY XML that indicates a dialog. In this case we're indicating fact of the call, but not exposing any sensitive data.

VOLTS - framework for testing VoIP deployments

 Last year I was working on VOLTS - framework I'm using for testing VoIP deployments in automated way. 

Right now it combines tools like voip_patrol and sipp in one tool with some extra features like database write/delete and media file check.

Not to repeat here all README, two links - github and the following video will tell it all.

 

Do you test your deployments?

Protect Kamailio from TCP/TLS flood

 After stress-testing Kamailio with sipflood tool from sippts suite (which deserves another article), not so good outcome was faced.

Using CentOS 7 default OpenSSL library (1.0.2k-fips) with using Kamailio 5.4-5.6 with TLS transport, it's quite easy to get a segfault inside tls routines. I've found that roughly 10 000 OPTIONS packets with 200 threads is enough to ruin Kamailio process.

Basically, you can DoS the whole server regardless of it's power just with a single mid-range computer.

Solution was found with using Kamailio 5.6, but with tlsa flavour and latest openssl 1.1.x compiled.

Turns out it's a really simple process. 

As we're gonna need to compile Kamailio anyway, assume, that we have all necessary packets for build already on the system.

First - we need to get openssl sources:

# cd /usr/src

# wget https://www.openssl.org/source/openssl-1.1.<latest>.tar.gz

# tar xvf https://www.openssl.org/source/openssl-1.1.<latest>.tar.gz

# cd  openssl-1.1.<latest>

#  ./config

# make

(Optionally) Here we can make sure that this release is passing tests

# yum install perl-Test-Simple

# make test

Next step - point Kamailio to newly compiled openssl

# cd /usr/src

# wget https://www.kamailio.org/pub/kamailio/5.6.<latest>/src/kamailio-5.6.<latest>_src.tar.gz

# tar xvf kamailio-5.6.<latest>_src.tar.gz

# cd kamailio-5.6.<latest>

#  sed -i "s?LIBSSL_STATIC_SRCLIB \?= no?LIBSSL_STATIC_SRCLIB \?= yes?g" ./src/modules/tlsa/Makefile

# sed -i "s?LIBSSL_STATIC_SRCPATH \?= /usr/local/src/openssl?LIBSSL_STATIC_SRCPATH \?= /usr/src/openssl-1.1.<latest>?g" ./src/modules/tlsa/Makefile

...

Than goes your usual Kamailio compiling and don't forget to replace all "tls" module mentions in kamailio.cfg to "tlsa"

Results are much better. But than I've faced, that it's possible to "eat" all TCP connections on Kamailio server with this type of flood.

First - ulimit. Never underestimate defaults.  

# ulimit -n unlimited

Next steps - tune TCP stack.

Disclamer: next provided options are discussable and was not found by me and need to be adjusted to your case

kamailio.conf

...

tcp_connection_lifetime=3605
tcp_max_connections=4096
tls_max_connections=4096
tcp_connect_timeout=5
tcp_async=yes
tcp_keepidle=5
open_files_limit=4096

...

/etc/sysctl.conf

...

# To increase the amount of memory available for socket input/output queues
net.ipv4.tcp_rmem = 4096 25165824 25165824
net.core.rmem_max = 25165824
net.core.rmem_default = 25165824
net.ipv4.tcp_wmem = 4096 65536 25165824
net.core.wmem_max = 25165824
net.core.wmem_default = 65536
net.core.optmem_max = 25165824

# To limit the maximum number of requests queued to a listen socket
net.core.somaxconn = 128

# Tells TCP to instead make decisions that would prefer lower latency.
net.ipv4.tcp_low_latency=1

# Optional (it will increase performance)
net.core.netdev_max_backlog = 1000
net.ipv4.tcp_max_syn_backlog = 128
...

This will help, but not fully (at least in my case, I've must miss something and comments here are really welcomed)

As the second part I've decided to go with Fail2Ban and block flood on iptables level.

Setup is quite simple as well.

First - make sure Kamailio will log flood attempts:

kamailio.conf

...

 loadmodule "pike.so"

modparam("pike", "sampling_time_unit", 2)
modparam("pike", "reqs_density_per_unit", 30)
modparam("pike", "remove_latency", 120)

...

if (!pike_check_req()) {
            xlog("L_ALERT", "[SIP-FIREWALL][FAIL2BAN] $si\n");

            $sht(ipban=>$si) = 1;
            if ($proto != 'udp') {
                tcp_close_connection();
            }
            drop;
        }

...

Next - install and configure Fail2Ban

# yum install -y fail2ban

 /etc/fail2ban/jail.local

[DEFAULT]
# Ban hosts for one hour:
bantime = 3600

# Override /etc/fail2ban/jail.d/00-firewalld.conf:
banaction = iptables-multiport
action      = %(action_mwl)s

[cernphone-iptables]
enabled  = true
filter   = mypbx
action   = iptables-mypbx[name=mypbx, protocol=tcp, blocktype='REJECT --reject-with tcp-reset']
           sendmail[sender=<sender_addr>, dest=<dest_addr> sendername=Fail2Ban]
logpath  = <your_kamailio_logfile>
maxretry = 1
bantime  = 3600s
findtime = 10s
 

 /etc/fail2ban/action.d/iptables-mypbx.conf

[INCLUDES]

before = iptables-common.conf

[Definition]

actionstart = <iptables> -N f2b-<name>
              <iptables> -A f2b-<name> -j <returntype>
              <iptables> -I <chain> -p <protocol> -j f2b-<name>

actionstop = <iptables> -D <chain> -p <protocol>  -j f2b-<name>
             <actionflush>
             <iptables> -X f2b-<name>


actioncheck = <iptables> -n -L <chain> | grep -q 'f2b-<name>[ \t]'

actionban = <iptables> -I f2b-<name> 1 -s <ip> -p <protocol> -j <blocktype>

actionunban = <iptables> -D f2b-<name> -s <ip> -p <protocol> -j <blocktype>

 

 /etc/fail2ban/filter.d/mypbx.local

[Definition]
# filter for kamailio messages
failregex = \[SIP-FIREWALL\]\[FAIL2BAN\] <HOST>$
 

# systemctl enable fail2ban

# systemctl start fail2ban

In this case we will get host banned on iptables level.

Functional testing of VoIP infrastructure

 Sometimes, it's really hard to test VoIP infrastructure in an automated way. The usual way of testing any telco system is to take a phone, make some calls and that's it. Maybe to look at the logs. 

Some of the engineers going further and using famous SIPP, which is quite good in testing low-level SIP, but really could be a pain in some closer-to-world scenarios.

Another approach is to write some scripts and automating Asterisk or FreeSWITCH to do some test calls. And it's a good approach, but sometimes writing something simple could take a lot of time. And attention.

But there is one more way - use standalone SIP libraries like baresip or pjsip and control them via API or CLI.

Exactly this way was taken by Julien Chavanton in his voip_patrol project. After a short time of playing around with it, I can say it's a really good combination of simplicity of things it can test and a way it's configuring.

In my current position, I'm dealing mostly with opus/SRTP media (in DTLS-SRTP flavor) and I was able to add support of this  to voip_patrol (in this branch) and it's working well with rtpengine provided SRTP.

And here is just an example of voip_patrol scenario to register TLS endpoint, make a call with SRTP and catch it back. So, a simple register-call test as a good starting point for more complex scenarios. 

 

<config>
  <actions>
  <action type="codec" disable="all"/>
    <action type="codec" enable="pcma" priority="250"/>
    <action type="codec" enable="pcmu" priority="249"/>
    <action type="register" label="Register 88881"
            transport="tls"
            account="88881"
            username="88881"
            password="XXXXX"
            registrar="XXXXX"
            realm="XXXXX"
            expected_cause_code="200"
            srtp="dtls,sdes,force"
    />
    <action type="wait" complete="true"/>
    <action type="accept" label="Receive all calls"
            account="default"
            hangup="10"
            code="200" reason="OK"
            transport="tls"
            srtp="dtls,sdes,force"
    />
    <action type="call" label="Call to 88881"
            transport="tls"
            expected_cause_code="200"
            caller="88882@XXXXX"
            callee="88881@XXXXX"
            from="sip:88882@XXXXX"
            to_uri="88881@XXXXX"
            max_duration="20" hangup="16"
            username="88882"
            password="XXXXX"
            realm="XXXXX"
            rtp_stats="true"
            max_ringing_duration="15"
            srtp="dtls,force"
            play="/git/voip_patrol/voice_ref_files/reference_8000_12s.wav"
    />
    <action type="wait" complete="true"/>
  </actions>
</config>

 

For more advanced, I suggest to look on issues [1, 2] on GitHub, where author expands some concepts.

And as a good quickstart - video from author

UPDATE: Based on this tool, templating and reporting suite VOLTS was created. Stay tuned for more cool features to come!

Kamailio and delayed CANCEL on iOS

One who is working with SIP aware of a problem with mobile devices. Long story short, the only reliable and proper way to deliver SIP calls to mobile devices is push notification.

A usual algo here, the server sends push notification to a device (via Apple or Google PNS respectively) and after mobile device woke up an application, it registers to SIP server and only after this SIP server sends INVITE to application. And last part here - device starts ringing.

So, SIP server have to store info on this call and wait for a device to register. To achieve this, Kamailio offers tsilo module, which is working great. If you haven't seen it or having problems with it, there is a great presentation which is a good starting point.

But there is one problem. After the call is canceled or answered, no more info is stored in this module. So, if push arrives later, than the call was answered, the app will not receive any info after REGISTER. Usually, it's ok, but with iOS > 13, there is a new mechanism of push notifications using for VoIP calls. CallKit (iOS developers may correct me here, cause I'm not). Idea is the following. When iOS device receives push notification for the VoIP call, it first will show a screen with info, that call has arrived. Later, after the app is already wakened up and running, the app is taking control over the calling screen to update caller info, show Accept/Reject buttons, etc. Sounds good. But if the call was canceled (or answered elsewhere) at the moment when Call Screen is shown already, but the app is not ready yet? The app is woken up, sends REGISTER to SIP server, but no INVITE is following. The call is already not this device business. 

Usually, this problem is solved on an application level. Like if an app is not receiving INVITE after push within X seconds, it asks for a remote server for info and shows (or not) corresponding missed call info.

But that's not my case, unfortunately. I'm using slightly rebranded Linphone and not an expert in iOS programming. So, need to resolve everything fully on a server side, where Kamailio is my SIP server.

The idea of the solution is following - if iOS device registers within X seconds after the call start and the call is already canceled or answered - a fake call is simulated to this endpoint. X could be any, but we come to 15 seconds. For simulating call the famous sipp is used.

Here following some parts of the code

kamailio.conf

# Flag to define that the call was locally generated via SIPP

#!define FLT_LOCALGEN 4

...

# Save info iOS CallKit workaround
modparam("htable", "htable", "ios_reg=>size=5;autoexpire=2;")
modparam("htable", "htable", "ios_state=>size=5;autoexpire=15")

...

request_route {

...

    if (is_method("CANCEL")) {
        # inbound CANCEL

        xlog("L_INFO", "[REQUEST_ROUTE] Detected CANCEL, saving state\n");

        $sht(ios_state=>$rU) = "state=canceled;callid=" + $ci + ";fromnumber=" + $fU;
    }

    # Save info in a case if call was answered to preserve callerID/destination

    if (is_method("INVITE") && !has_totag()) {

        $avp(orig_fU) = $fU;
        $avp(orig_rU) = $rU;

    }

...

onreply_route {

     if ($rs == 200) {
        # Save call state fr
        xlog("L_INFO", "[MANAGE_REPLY] Call $avp(orig_fU) -> $avp(orig_rU) was answered\n");

        $sht(ios_state=>$avp(orig_rU)) = "state=answered;callid=" + $ci + ";fromnumber=" + $avp(orig_fU);
    }

...

route[AUTH]

...

   if ($si == 'MY_IP_ADDR') {
        xlog("L_NOTICE", "[AUTH] Packet from ourselves\n");

        setflag(FLT_LOCALGEN);
        return;
    }

...

# We don't want  this INVITE to be processed via tsilo routine.

route[INVITE_STORE] {

    if (isflagset(FLT_LOCALGEN)) {
        return;
    }

...

 route[PUSH_NOTIFICATION_ON_INVITE] {
    if (!is_method("INVITE") || isflagset(FLT_LOCALGEN)) {
        return;
    }

...

# Small extra route to be called after succesful save("location")

route[REGISTER_IOS] {
    if (!($ua =~ "LinphoneiOS*")) {
        xlog("L_INFO", "[REGISTER_IOS] Not IPhone, not interesting...\n");

        return;
    }

    xlog("L_INFO", "[REGISTER_IOS] IPhone registered, saving contact for 2 sec\n");

#  Saving location info for later
    $var(ios_contact) = "sip:" + $fU + "@" + $si + ":" + $sp + ";transport=" + $pr;

# Adding some random to support several iOS devices per same account.
    $var(ios_reg_hash_id) = $fU + "_" + $RANDOM;

# information on contact is stored in hash memrory, and the hash key would return to us via X-IOS-Contact-Hash header provided by SIPP.
    $sht(ios_reg=>$var(ios_reg_hash_id)) = $var(ios_contact);

    $var(ios_callstate) = $sht(ios_state=>$fU);

    if (not_empty("$var(ios_callstate)")) {
        $var(call_state) = $(var(ios_callstate){param.value,state});
        $var(call_callid) = $(var(ios_callstate){param.value,callid});
        $var(call_from) = $(var(ios_callstate){param.value,fromnumber});

        xlog("L_INFO", "[REGISTER_IOS] Registration from iOS is late. Call state is $var(call_state)\n");

        # Here we actually start call via SIPP

        exec_msg("/usr/bin/nohup launch_sipp.sh $var(call_callid) $fU $var(ios_reg_hash_id) $var(call_state) $var(call_from) >/dev/null 2>& 1 &");
    }
}

...

# Own lookup("location"), but with info saved on [REGISTER_IOS]

route[LOCATION_INTERNAL] {
    if (!isflagset(FLT_LOCALGEN) || !is_present_hf("X-IOS-Contact-Hash")) {
        return;
    }

    $var(ios_reg_hash_id) = $hdr(X-IOS-Contact-Hash);
    $var(ios_contact) = $sht(ios_reg=>$var(ios_reg_hash_id));

    remove_hf("X-IOS-Contact-Hash");

    if (!not_empty("$var(ios_contact)")) {
        xlog("L_WARN", "[LOCATION_INTERNAL] Locally generated packet, but no contact was saved\n");

        send_reply("404", "Saved contact not found");
        exit;
    }

    $ru = $var(ios_contact);
    $du = $ru;

    route(RELAY);
}

launch_sipp.sh

#!/bin/bash

if [ -z "${1}" ]; then
    echo "CallID is not specified"
    exit 1
fi
CALL_ID=${1}

if [ -z "${2}" ]; then
    echo "To number is not specified"
    exit 1
fi
TO_NUMBER=${2}

if [ -z "${3}" ]; then
    echo "HASH ID is not specified"
    exit 1
fi
HASH_ID=${3}

REASON=${4:-answered}
FROM_NUMBER=${5:-MY_PBX}
FROM_NAME=${6:-${FROM_NUMBER}}

# Create CSV file for SIPP first
/usr/bin/echo -e "SEQUENTIAL\n${TO_NUMBER};${FROM_NUMBER};'${FROM_NAME}';${HASH_ID};${REASON};" >> /tmp/invite_cancel_$$.csv

# Run SIPP with provided CallID, as it should be same as declared in Push Notification
/usr/bin/sipp localhost -sf invite_cancel.xml -inf /tmp/invite_cancel_$$.csv -m 1 -cid_str ${CALL_ID}
rm /tmp/invite_cancel_$$.csv

invite_cancel.xml

 <?xml version="1.0" encoding="UTF-8" ?>

<!-- This sipp scenario combines 2 types of CANCEL with different reasons
Reason "Call Completed Elsewhere" will not get into a missed calls list
CANCEL without a reason will get there
Which CANCEL would be sent, controlled by field 4 in CSV line
If this field == 'answered', call is considered answered and will not get into missed call list
In all other cases call considered missed -->

<scenario name="iOS CallKit canceler">
 <send>
  <![CDATA[

    INVITE sip:[field0 line=0]@[remote_ip]:[remote_port] SIP/2.0
    Via: SIP/2.0/[transport] [local_ip]:[local_port]
    From: <sip:[field1 line=0]@[local_ip]:[local_port]>;tag=[call_number]_sipp_call_canceler_[call_number]
    To: <sip:[field0 line=0]@[remote_ip]:[remote_port]>
    Call-ID: [call_id]
    Cseq: [cseq] INVITE
    Allow: OPTIONS, SUBSCRIBE, NOTIFY, PUBLISH, INVITE, ACK, BYE, CANCEL, UPDATE, PRACK, REGISTER, MESSAGE, REFER
    Supported: 100rel, timer, replaces, norefersub
    Session-Expires: 1800
    Min-SE: 90
    Contact: sip:asterisk@[local_ip]:[local_port]
    Max-Forwards: 70
    X-IOS-Contact-Hash: [field3 line=0]
    Content-Type: application/sdp
    Content-Length: [len]

    v=0
    o=- 558046395 558046395 IN IP[media_ip_type] [media_ip]
    s=Asterisk
    c=IN IP[media_ip_type] [media_ip]
    t=0 0
    m=audio 16170 RTP/AVP 107 8 0 101
    a=rtpmap:107 opus/48000/2
    a=fmtp:107 useinbandfec=1
    a=rtpmap:8 PCMA/8000
    a=rtpmap:0 PCMU/8000
    a=rtpmap:101 telephone-event/8000
    a=fmtp:101 0-16
    a=ptime:20
    a=maxptime:20
    a=sendrecv
    m=video 14074 RTP/AVP 100
    a=rtpmap:100 VP8/90000
    a=sendrecv

  ]]>
 </send>

 <!-- Wait for provisional responces and if something wrong (404 or timeout) - end scenario -->
 <recv response="100" optional="true" timeout="5000" ontimeout="12"/>
 <recv response="404" optional="true" next="12"/>
 <recv response="180" timeout="5000" ontimeout="12" />

<!-- Check if field4 == 'answered' and assign bool result to $3 -->

 <nop>
  <action>
    <assignstr assign_to="1" value="[field4 line=0]" />
    <strcmp assign_to="2" variable="1" value="answered" />
    <test assign_to="3" variable="2" compare="equal" value="0" />
  </action>
 </nop>

<!-- If $3 == true, go to label 10  -->
 <nop next="10" test="3" />

 <send>
  <![CDATA[

    CANCEL sip:[field0 line=0]@[remote_ip]:[remote_port] SIP/2.0
    Via: SIP/2.0/[transport] [local_ip]:[local_port]
    From: <sip:[field1 line=0]@[local_ip]:[local_port]>;tag=[call_number]_sipp_call_canceler_[call_number]
    To: <sip:[field0 line=0]@[remote_ip]:[remote_port]>
    Call-ID: [call_id]
    Cseq: [cseq] CANCEL
    Max-Forwards: 70
    Content-Length: 0

  ]]>
 </send>

 <!-- Simple goto "11" after receiving 200 -->
 <recv response="200" next="11"/>

 <label id="10"/>

 <send>
  <![CDATA[

    CANCEL sip:[field0 line=0]@[remote_ip]:[remote_port] SIP/2.0
    Via: SIP/2.0/[transport] [local_ip]:[local_port]
    From: <sip:[field1 line=0]@[local_ip]:[local_port]>;tag=[call_number]_sipp_call_canceler_[call_number]
    To: <sip:[field0 line=0]@[remote_ip]:[remote_port]>
    Call-ID: [call_id]
    Cseq: [cseq] CANCEL
    Max-Forwards: 70
    Reason: SIP;cause=200;text="Call completed elsewhere"
    Content-Length: 0

  ]]>
 </send>

 <recv response="200" />

 <label id="11"/>

 <recv response="487" />

 <send>
  <![CDATA[

    ACK sip:[field0 line=0]@[remote_ip]:[remote_port] SIP/2.0
    Via: SIP/2.0/[transport] [local_ip]:[local_port]
    From: <sip:[field1 line=0]@[local_ip]:[local_port]>;tag=[call_number]_sipp_call_canceler_[call_number]
    [last_To:]
    Call-ID: [call_id]
    Cseq: [cseq] ACK
    Contact: sip:asterisk@[local_ip]:[local_port]
    Content-Length: 0

  ]]>
 </send>

 <label id="12" />

</scenario>

A bit dirty solution, but it's working.

Adding call quality parameters to Asterisk CDR

Following presentation from AsterConf 2016 (yes, I know, it was 5 years ago), I've decided to collect similar info on CDR. But this presentation was given a long ago and many things have changed. 

So, a bit updated version.

I'm omitting here connection procedure to the database via odbc, don't want to write the same procedure once again.

And few things to mention - it's for PJSIP. And I'm not using setvar= routine, insead doing it all explicitly in the dialplan.

So...

cdr_adaptive_odbc.conf

[asterisk]
connection=asterisk
table=cdr_extra
usegmtime=no
alias start => calldate
alias sip_callid => sip_callid
alias rtcp_a_all => rtcp_a_all
alias rtcp_a_all_jitter => rtcp_a_all_jitter
alias rtcp_a_all_loss => rtcp_a_all_loss
alias rtcp_a_all_rtt => rtcp_a_all_rtt
alias rtcp_a_txjitter => rtcp_a_txjitter
alias rtcp_a_rxjitter => rtcp_a_rxjitter
alias rtcp_a_txploss => rtcp_a_txploss
alias rtcp_a_rxploss => rtcp_a_rxploss
alias rtcp_a_rtt => rtcp_a_rtt
alias rtcp_b_all => rtcp_b_all
alias rtcp_b_all_jitter => rtcp_b_all_jitter
alias rtcp_b_all_loss => rtcp_b_all_loss
alias rtcp_b_all_rtt => rtcp_b_all_rtt
alias rtcp_b_txjitter => rtcp_b_txjitter
alias rtcp_b_rxjitter => rtcp_b_rxjitter
alias rtcp_b_txploss => rtcp_b_txploss
alias rtcp_b_rxploss => rtcp_b_rxploss
alias rtcp_b_rtt => rtcp_b_rtt

database_cdr_extra.sql

 CREATE TABLE cdr_extra (
        calldate datetime NOT NULL default '0000-00-00 00:00:00',
        clid varchar(80) NOT NULL default '',
        src varchar(80) NOT NULL default '',
        dst varchar(80) NOT NULL default '',
        dcontext varchar(80) NOT NULL default '',
        channel varchar(80) NOT NULL default '',
        dstchannel varchar(80) NOT NULL default '',
        lastapp varchar(80) NOT NULL default '',
        lastdata varchar(80) NOT NULL default '',
        duration int(11) NOT NULL default '0',
        billsec int(11) NOT NULL default '0',
        disposition varchar(45) NOT NULL default '',
        amaflags int(11) NOT NULL default '0',
        accountcode varchar(20) NOT NULL default '',
        uniqueid varchar(32) NOT NULL default '',
        userfield varchar(255) NOT NULL default '',
        sip_callid varchar(255) NOT NULL default '',
        rtcp_a_all varchar(255) NOT NULL default '',
        rtcp_a_all_jitter varchar(255) NOT NULL default '',
        rtcp_a_all_loss varchar(255) NOT NULL default '',
        rtcp_a_all_rtt varchar(255) NOT NULL default '',
        rtcp_a_txjitter decimal(10,6),
        rtcp_a_rxjitter decimal(10,6),
        rtcp_a_txploss int(11) NOT NULL default '0',
        rtcp_a_rxploss int(11) NOT NULL default '0',
        rtcp_a_rtt decimal(10,6),
        rtcp_b_all varchar(255) NOT NULL default '',
        rtcp_b_all_jitter varchar(255) NOT NULL default '',
        rtcp_b_all_loss varchar(255) NOT NULL default '',
        rtcp_b_all_rtt varchar(255) NOT NULL default '',
        rtcp_b_txjitter decimal(10,6),
        rtcp_b_rxjitter decimal(10,6),
        rtcp_b_txploss int(11) NOT NULL default '0',
        rtcp_b_rxploss int(11) NOT NULL default '0',
        rtcp_b_rtt decimal(10,6)
);

 

extensions.conf

 ...

; usual Dial

same => n,Set(CHANNEL(hangup_handler_push)=qos_leg_a,s,1)

same => n,Dial(${destinationStr},,b(set_qos_handler_b^s^1))
...

[qos_leg_a]
exten => s,1,Noop(Channel: ${CHANNEL(name)}, QoS stats RTCP: ${CHANNEL(rtcp,all)})
    same => n,Set(CDR(sip_callid)=${CHANNEL(pjsip,call-id)})
    same => n,GotoIf($["${CHANNEL(rtcp,all)}"==""]?end)
    same => n,Set(CDR(rtcp_a_all)=${CHANNEL(rtcp,all)})
    same => n,Set(CDR(rtcp_a_all_jitter)=${CHANNEL(rtcp,all_jitter)})
    same => n,Set(CDR(rtcp_a_all_loss)=${CHANNEL(rtcp,all_loss)})
    same => n,Set(CDR(rtcp_a_all_rtt)=${CHANNEL(rtcp,all_rtt)})
    same => n,Set(CDR(rtcp_a_txjitter)=${CHANNEL(rtcp,txjitter)})
    same => n,Set(CDR(rtcp_a_rxjitter)=${CHANNEL(rtcp,rxjitter)})
    same => n,Set(CDR(rtcp_a_txploss)=${CHANNEL(rtcp,txploss)})
    same => n,Set(CDR(rtcp_a_rxploss)=${CHANNEL(rtcp,rxploss)})
    same => n,Set(CDR(rtcp_a_rtt)=${CHANNEL(rtcp,rtt)})
    same => n(end),Return()

[qos_leg_b]
exten => s,1,Noop(Channel: ${CHANNEL(name)}, QoS stats RTCP: ${CHANNEL(rtcp,all)})
    same => n,GotoIf($["${CHANNEL(rtcp,all)}"==""]?end)
    same => n,Set(CDR(rtcp_b_all)=${CHANNEL(rtcp,all)})
    same => n,Set(CDR(rtcp_b_all_jitter)=${CHANNEL(rtcp,all_jitter)})
    same => n,Set(CDR(rtcp_b_all_loss)=${CHANNEL(rtcp,all_loss)})
    same => n,Set(CDR(rtcp_b_all_rtt)=${CHANNEL(rtcp,all_rtt)})
    same => n,Set(CDR(rtcp_b_txjitter)=${CHANNEL(rtcp,txjitter)})
    same => n,Set(CDR(rtcp_b_rxjitter)=${CHANNEL(rtcp,rxjitter)})
    same => n,Set(CDR(rtcp_b_txploss)=${CHANNEL(rtcp,txploss)})
    same => n,Set(CDR(rtcp_b_rxploss)=${CHANNEL(rtcp,rxploss)})
    same => n,Set(CDR(rtcp_b_rtt)=${CHANNEL(rtcp,rtt)})
    same => n(end),Return()

[set_qos_handler_b]

exten => s,1,Set(CHANNEL(hangup_handler_push)=qos_leg_b,s,1)
    same => n,Return()

As a bonus you have the SIP Call-ID in this CDR, which is useful for debugging SIP traces.

Kamailio and mobile TCP endpoints. Unregistering

The modern world is mobile. You may like it or not. But it is. So, with all these technologies that are about saving your battery life, all application reachability goes to vendor-lock push servers.

 

But the problem emerges in the following. SIP client registers on registrar (Kamailio) with TCP. Yes, you can do SIP keepalives here, but why to have em if we have a built-in mechanism of keepalive in TCP itself? Plus, additional packets over the network will drain the battery faster. And iOS, when putting the app to the background, just cut the network of app off. Literally killing it.

 

But we somehow need to know which actual state of the endpoint is. And here Kamailio is to help us with tcpops module.

 

The idea is quite simple. On each query to location table, we will clean-up it from "dead" TCP connections. So, after save(). And the only actual state of endpoints would be taken in the branch and calling process. Yes, it will give extra initial INVITE on already 'dead' connection, but appears to be, it sometimes cleaning just arrived REGISTER's so it's acceptable.

# Save info for TCP connections for unregister on close/timeout/error
loadmodule "htable.so"

modparam("htable", "htable", "tcpconn=>size=15;autoexpire=7200;")

...

# Handle SIP registrations
route[REGISTRAR] {

   ...

    save("location");
    route(TRACK_TCP_STATE);

    $var(processed_subscriber) = $fu;

    route(TCP_REGISISTER_CLEANUP);

   ...

}

 

route[TRACK_TCP_STATE] {
    if (proto == UDP) {
        return;
    }
    xlog("[TRACK_TCP_STATE] Saving state for future disconect track of $fu\n");

    $sht(tcpconn=>$conid) = $fu;

    tcp_enable_closed_event();
}

# Make sure you set up $var(processed_subscriber) before calling this route

route[TCP_REGISISTER_CLEANUP] {

    if ($var(processed_subscriber) == $null) {
        return;
    }

    xlog("[TCP_REGISISTER_CLEANUP] Processing subscriber $var(processed_subscriber)\n");

    # Getting registered endpoints for this AoR
    if (!reg_fetch_contacts("location", "$var(processed_subscriber)", "subscriber")) {
        $var(processed_subscriber) = $null;
        xlog("[TCP_REGISISTER_CLEANUP] No registered contacts for $var(processed_subscriber)\n");
        return;
    }

    $var(i) = 0;

    # Loop through registered endpoints
    while ($var(i) < $(ulc(subscriber=>count))) {

        $var(stored_subscriber_conid) = $(ulc(subscriber=>conid)[$var(i)]);

        # Make sure proto is TCP
        if ($var(stored_subscriber_conid) != $null) {

            # Check if entry is still active TCP connection. Unregister otherwise.
            if ($var(stored_subscriber_conid) == -1 || !tcp_conid_alive("$var(stored_subscriber_conid)")) {

                $var(stored_subscriber_ruid) = $(ulc(subscriber=>ruid)[$var(i)]);
                $var(stored_subscriber_address) = $(ulc(subscriber=>addr)[$var(i)]);

                xlog("[TCP_REGISISTER_CLEANUP]: Unregistering entry $var(i)/$var(stored_subscriber_conid) -> $var(stored_subscriber_address)\n");

                if (!unregister("location", "$var(processed_subscriber)", "$var(stored_subscriber_ruid)")) {
                    xlog("[TCP_REGISISTER_CLEANUP]: Unregistering entry $var(i)/$var(stored_subscriber_conid) -> $var(stored_subscriber_address) FAILED!\n");
                }
            }
        }
        $var(i) = $var(i) + 1;
    }
    $var(processed_subscriber) = $null;
}

event_route[tcp:closed] {
    xlog("[TCP:CLOSED] $proto connection closed conid=$conid\n");

    $var(processed_subscriber) = $sht(tcpconn=>$conid);

    if ($var(processed_subscriber) != $null) {
        route(TCP_REGISISTER_CLEANUP);
        $sht(tcpconn=>$conid) = $null;
    }
}

event_route[tcp:timeout] {
    xlog("[TCP:TIMEOUT] $proto connection timeout conid=$conid\n");

    $var(processed_subscriber) = $sht(tcpconn=>$conid);

    if ($var(processed_subscriber) != $null) {
        route(TCP_REGISISTER_CLEANUP);
        $sht(tcpconn=>$conid) = $null;
    }
}

event_route[tcp:reset] {
    xlog("[TCP:RESET] $proto reset closed conid=$conid\n");

    $var(processed_subscriber) = $sht(tcpconn=>$conid);

    if ($var(processed_subscriber) != $null) {
        route(TCP_REGISISTER_CLEANUP);
        $sht(tcpconn=>$conid) = $null;
    }
}

Simple and efficient solution. Yes, some of "dead" endpoints would be present in table during expiration time, but I'm ok with that.

Janus Audio/Video plays around

WebRTC got to me. I can't say it was the first time, but most deep at the moment.

Small disclaimer. I'm not a web developer and only now got, that browser is OS by itself.

And the way to get media from the browser is WebRTC. So, let's make one more bicycle to have WebRTC - VoIP bridge.

There are many projects around to fill this gap. FreeSwitch (mode_verto or mod_sofia), Kamailio + RTPEngine are just to mention some.

And Janus was chosen. To try something new and get some skills in bright'n'shiny world of web development. And I like, that Janus has already built-in SIP plugin.

The idea of this small proof of concept - get audio and video streams separated and treated separately. Why - to have audio part handled by SIP (passed to Asterisk, more precisely) and video (or screen sharing part in the future) - by Janus SFU unit (VideoRoom plugin)

Project is just a set of 3 docker containers running in network host mode (just don't want to play around with NAT and all this stuff) and glued together with docker-compose. Nginx as a web server, Janus as Janus WebRTC server, and Asterisk as SIP application server.

Web part is just reworked and gloriously copy-pasted from original Janus demos.

This project is intended to be used as a playground for future experiments and donor of code for other projects.

Repo with project on GitHub.

Some of the pictures not to be so boring:

Audio is passed to Asterisk via SIP plugin, and Video is handled by EchoTest plugin.

Audio is mixing by Asterisk (SIP plugin as well), Video is handled by VideoRoom plugin.

Point, this one is tested on a VPS with a Let'sEncrypt certs to test it mostly "in the wild" and also there are many restrictions in modern browsers about WebRTC and security, which means even over LAN you need to have pure HTTPS.

SIP Monitoring based on SIP3 and Heplify

Few words upfronts.

 - What is SIP3?

Follow the link :) If you don't know why you need it (or similar system) - you don't.

 - Why SIP3 and not Homer? 
Homer 3 was really good. It actually covers all my needs in SIP traffic monitoring. Than Homer 5 with new GUI was introduced. And things slowly start getting worse. Mostly cause of really strange changes in UI, which can stop working at any moment. No, it will respond, show something, but sometimes not the real picture. The usual answer - log out and back in. With Homer 7 things went even worse. Recommendation from developers is using the latest alpha version. If set up the system with a recommended setup (set of docker containers), it's not a "set and forget" solution. No. It will stop processing traffic at some time without any warning. For monitoring system which I need not every day (actually I need mostly post-mortem traces), it always ends up with log into the system and finds out it stops working like a week ago. Not the best option.
But it's really my experience. Maybe I'm not that lucky at all.

 - Why heplify and not captain (which is a part of SIP3)?
The answer is quite simple. Heplify - single binary, captain (at the moment) - a full zoo of tools, like docker, ansible, python, etc. Yes, I'm aware, that captain also can provide RTCP stats, but overhead is too big.

So, getting to the good part.

Installation and setting up

Central server

I'm gonna use CentOS 8 (minimal). Not like I'm a big fan of it, but developers like it. So, most tested one

1. Install docker

# yum install -y yum-utils   device-mapper-persistent-data  lvm2
# yum-config-manager --add-repo https://download.docker.com/linux/centos/docker-ce.repo
# containerdSource=https://download.docker.com/linux/centos/7/x86_64/stable/Packages/containerd.io-1.2.6-3.3.el7.x86_64.rpm
# dnf install $containerdSource -y
# dnf install docker-ce docker-ce-cli -y
# systemctl disable firewalld
# systemctl start docker
# systemctl enable --now docker

 

2. Install ansible, pip, git

# yum install -y epel-release
# yum install -y ansible git python3-pip
# pip3 install docker-py

3. Install SIP3 backend

# cd /usr/src/
# git clone https://github.com/sip3io/sip3-ansible.git

Note on taking metrics from public network services.

/usr/src/sip3-ansible/roles/sip3-salto/templates/application.yml.j2
#! Server
server:
  uri: udp://0.0.0.0:15060


#! Management socket
management:
  uri: udp://0.0.0.0:15090

#! Metrics
metrics:
  logging:
    step: 1000
  influxdb:
    uri: http://sip3-influxdb:8086
    db: sip3
    step: 1000

#! MongoDB
mongo:
  uri: mongodb://sip3-mongodb:27017
  db: sip3
  bulk-size: 1
  collections:
    - prefix: attributes
      indexes:
        ascending: [name]
      max-collections: 7
    - prefix: sip_register_index
      indexes:
        ascending: [created_at, terminated_at, src_addr, src_host, dst_addr, dst_host, caller, callee, state]
        hashed: [call_id, x_call_id]
      max-collections: 7
    - prefix: sip_register_raw
      indexes:
        ascending: [created_at]
        hashed: [call_id]
      max-collections: 7
    - prefix: sip_call_index
      indexes:
        ascending: [created_at, terminated_at, src_addr, src_host, dst_addr, dst_host, caller, callee, state, error_code, error_type, duration, setup_time, establish_time]
        hashed: [call_id, x_call_id]
      max-collections: 7
    - prefix: sip_call_raw
      indexes:
        ascending: [created_at]
        hashed: [call_id]
      max-collections: 7
    - prefix: sip_message_index
      indexes:
        ascending: [created_at, terminated_at, src_addr, src_host, dst_addr, dst_host, caller, callee, state]
        hashed: [call_id, x_call_id]
      max-collections: 7
    - prefix: sip_message_raw
      indexes:
        ascending: [created_at]
        hashed: [call_id]
      max-collections: 7
    - prefix: sip_options_index
      indexes:
        ascending: [created_at, terminated_at, src_addr, src_host, dst_addr, dst_host, caller, callee, state]
        hashed: [call_id, x_call_id]
      max-collections: 7
    - prefix: sip_options_raw
      indexes:
        ascending: [created_at]
        hashed: [call_id]
      max-collections: 7

#! Application
sip:
  message:
    x-correlation-header: X-Call-ID
    exclusions: [MESSAGE, SUBSCRIBE, NOTIFY, OPTIONS, REGISTER]
  calls:
    aggregation-timeout: 120000

attributes:
  record-ip-addresses: false
  record-call-users: false

It's not to index in the database all IP addresses and usernames are seen on the system, cause the production system usually resides on public networks, where are tons of spam/attack calls and packets. Also retention period for database is set to 7 days, which is ok for me.

# ansible-playbook /usr/src/sip3-ansible/playbooks/trial/sip3-backend.yml

Relax and wait for system installs.

4. Add autostart

 /etc/systemd/system/sip3-backend.service

[Unit]
Description=Start Ansible Playbook SIP3
After=network.target

[Service]
Type=simple
ExecStart=/usr/bin/ansible-playbook /usr/src/sip3-ansible/playbooks/trial/sip3-backend.yml
TimeoutStartSec=10

[Install]
WantedBy=default.target



# systemctl daemon-reload
# systemctl enable sip3-backend

5. (Optional )Add hosts aliases to db

To get more friendly trances to read. Point, no spaces in names are allowed.

# docker exec -it sip3-mongodb mongo
# use sip3
# db.hosts.insert({"name" : "my_softswitch", "sip": ["X.X.X.X", "10.10.0.1"]})

Installing heplify (capture agent)

Log in to the server you want to collect SIP traffic from

# cd /usr/local/bin/
# wget https://github.com/sipcapture/heplify/releases/download/1.56/heplify
# chmod +x heplify

/etc/systemd/system/heplify.service
[Unit]
Description=Captures packets from wire and sends them to SIP3
After=network.target

[Service]
WorkingDirectory=/usr/local/bin
ExecStart=/usr/local/bin/heplify -i any -m SIP -hs SIP3_SERVER_ADDRESS:15060
ExecStop=/bin/kill ${MAINPID}
Restart=on-failure
RestartSec=10s
Type=simple

[Install]
WantedBy=multi-user.target



# systemctl daemon-reload
# systemctl enable heplify
# systemctl start heplify

System using

Log into your newly installed system on
http://SIP3_SEVER_ADDRESS with admin:admin credentials.

Sometimes, just after login, you will need to refresh your page. Actually, make sure Go! button is active

It's your search string. Simple, but yet powerful. It really looks like Wireshark one.
You can add search critera there based on sip.

or ip.

criteria
ip.src_host and ip.dst_host are hosts we added at step 5 installing Central Server paragraph.

Changing password is quite tricky, cause SIP3 relies on Grafana auth (Grafana is shipped as well) and to change password, head to

 http://SIP3_SEVER_ADDRESS/grafana

and use it interface to change password. As a bonus, you have Grafana dashboard with some useful metrics. And for sure you can add one you like. How - read Grafana manual :)

Adding batteries

One is a key feature of SIP3 is the ability to glue call legs in a 1 call flow. Usual algorithm is based on a strict match From and To numbers, but it's really rare case (at least, in my scenarios). There is a possibility to write user-defined functions, but I'll leave it to official manual. Another way to match call is to use custom SIP headers to match call legs. The default SIP3 header is X-Call-ID. So, if one leg holds Call-ID standart header matching other call X-Call-ID header, they would consider as one call and glue together.

In my case, I'm using FusionPBX, the solution is really simple. Add dialplan entry:

It will add X-Call-ID header to call equal to current Call-ID, if it's not present. Also, FreeSwitch preserves X-Headers over call legs. which is perfect for us.

So, we will have a picture like this

Upgrading

# cd /usr/src
# git pull
# ansible-playbook /usr/src/sip3-ansible/playbooks/trial/sip3-backend.yml

Happy using!