Adding call quality parameters to Asterisk CDR

Following presentation from AsterConf 2016 (yes, I know, it was 5 years ago), I've decided to collect similar info on CDR. But this presentation was given a long ago and many things have changed. 

So, a bit updated version.

I'm omitting here connection procedure to the database via odbc, don't want to write the same procedure once again.

And few things to mention - it's for PJSIP. And I'm not using setvar= routine, insead doing it all explicitly in the dialplan.

So...

cdr_adaptive_odbc.conf

[asterisk]
connection=asterisk
table=cdr_extra
usegmtime=no
alias start => calldate
alias sip_callid => sip_callid
alias rtcp_a_all => rtcp_a_all
alias rtcp_a_all_jitter => rtcp_a_all_jitter
alias rtcp_a_all_loss => rtcp_a_all_loss
alias rtcp_a_all_rtt => rtcp_a_all_rtt
alias rtcp_a_txjitter => rtcp_a_txjitter
alias rtcp_a_rxjitter => rtcp_a_rxjitter
alias rtcp_a_txploss => rtcp_a_txploss
alias rtcp_a_rxploss => rtcp_a_rxploss
alias rtcp_a_rtt => rtcp_a_rtt
alias rtcp_b_all => rtcp_b_all
alias rtcp_b_all_jitter => rtcp_b_all_jitter
alias rtcp_b_all_loss => rtcp_b_all_loss
alias rtcp_b_all_rtt => rtcp_b_all_rtt
alias rtcp_b_txjitter => rtcp_b_txjitter
alias rtcp_b_rxjitter => rtcp_b_rxjitter
alias rtcp_b_txploss => rtcp_b_txploss
alias rtcp_b_rxploss => rtcp_b_rxploss
alias rtcp_b_rtt => rtcp_b_rtt

database_cdr_extra.sql

 CREATE TABLE cdr_extra (
        calldate datetime NOT NULL default '0000-00-00 00:00:00',
        clid varchar(80) NOT NULL default '',
        src varchar(80) NOT NULL default '',
        dst varchar(80) NOT NULL default '',
        dcontext varchar(80) NOT NULL default '',
        channel varchar(80) NOT NULL default '',
        dstchannel varchar(80) NOT NULL default '',
        lastapp varchar(80) NOT NULL default '',
        lastdata varchar(80) NOT NULL default '',
        duration int(11) NOT NULL default '0',
        billsec int(11) NOT NULL default '0',
        disposition varchar(45) NOT NULL default '',
        amaflags int(11) NOT NULL default '0',
        accountcode varchar(20) NOT NULL default '',
        uniqueid varchar(32) NOT NULL default '',
        userfield varchar(255) NOT NULL default '',
        sip_callid varchar(255) NOT NULL default '',
        rtcp_a_all varchar(255) NOT NULL default '',
        rtcp_a_all_jitter varchar(255) NOT NULL default '',
        rtcp_a_all_loss varchar(255) NOT NULL default '',
        rtcp_a_all_rtt varchar(255) NOT NULL default '',
        rtcp_a_txjitter decimal(10,6),
        rtcp_a_rxjitter decimal(10,6),
        rtcp_a_txploss int(11) NOT NULL default '0',
        rtcp_a_rxploss int(11) NOT NULL default '0',
        rtcp_a_rtt decimal(10,6),
        rtcp_b_all varchar(255) NOT NULL default '',
        rtcp_b_all_jitter varchar(255) NOT NULL default '',
        rtcp_b_all_loss varchar(255) NOT NULL default '',
        rtcp_b_all_rtt varchar(255) NOT NULL default '',
        rtcp_b_txjitter decimal(10,6),
        rtcp_b_rxjitter decimal(10,6),
        rtcp_b_txploss int(11) NOT NULL default '0',
        rtcp_b_rxploss int(11) NOT NULL default '0',
        rtcp_b_rtt decimal(10,6)
);

 

extensions.conf

 ...

; usual Dial

same => n,Set(CHANNEL(hangup_handler_push)=qos_leg_a,s,1)

same => n,Dial(${destinationStr},,b(set_qos_handler_b^s^1))
...

[qos_leg_a]
exten => s,1,Noop(Channel: ${CHANNEL(name)}, QoS stats RTCP: ${CHANNEL(rtcp,all)})
    same => n,Set(CDR(sip_callid)=${CHANNEL(pjsip,call-id)})
    same => n,GotoIf($["${CHANNEL(rtcp,all)}"==""]?end)
    same => n,Set(CDR(rtcp_a_all)=${CHANNEL(rtcp,all)})
    same => n,Set(CDR(rtcp_a_all_jitter)=${CHANNEL(rtcp,all_jitter)})
    same => n,Set(CDR(rtcp_a_all_loss)=${CHANNEL(rtcp,all_loss)})
    same => n,Set(CDR(rtcp_a_all_rtt)=${CHANNEL(rtcp,all_rtt)})
    same => n,Set(CDR(rtcp_a_txjitter)=${CHANNEL(rtcp,txjitter)})
    same => n,Set(CDR(rtcp_a_rxjitter)=${CHANNEL(rtcp,rxjitter)})
    same => n,Set(CDR(rtcp_a_txploss)=${CHANNEL(rtcp,txploss)})
    same => n,Set(CDR(rtcp_a_rxploss)=${CHANNEL(rtcp,rxploss)})
    same => n,Set(CDR(rtcp_a_rtt)=${CHANNEL(rtcp,rtt)})
    same => n(end),Return()

[qos_leg_b]
exten => s,1,Noop(Channel: ${CHANNEL(name)}, QoS stats RTCP: ${CHANNEL(rtcp,all)})
    same => n,GotoIf($["${CHANNEL(rtcp,all)}"==""]?end)
    same => n,Set(CDR(rtcp_b_all)=${CHANNEL(rtcp,all)})
    same => n,Set(CDR(rtcp_b_all_jitter)=${CHANNEL(rtcp,all_jitter)})
    same => n,Set(CDR(rtcp_b_all_loss)=${CHANNEL(rtcp,all_loss)})
    same => n,Set(CDR(rtcp_b_all_rtt)=${CHANNEL(rtcp,all_rtt)})
    same => n,Set(CDR(rtcp_b_txjitter)=${CHANNEL(rtcp,txjitter)})
    same => n,Set(CDR(rtcp_b_rxjitter)=${CHANNEL(rtcp,rxjitter)})
    same => n,Set(CDR(rtcp_b_txploss)=${CHANNEL(rtcp,txploss)})
    same => n,Set(CDR(rtcp_b_rxploss)=${CHANNEL(rtcp,rxploss)})
    same => n,Set(CDR(rtcp_b_rtt)=${CHANNEL(rtcp,rtt)})
    same => n(end),Return()

[set_qos_handler_b]

exten => s,1,Set(CHANNEL(hangup_handler_push)=qos_leg_b,s,1)
    same => n,Return()

As a bonus you have the SIP Call-ID in this CDR, which is useful for debugging SIP traces.

Cancel a calls in an Asterisk dialsting

Sometimes life gives strange tasks.

Imagine that you have a dialstring in Asterisk like

Dial(PJSIP/dest1&PJSIP/dest2...)

But you need to stop the call at the moment when one of the destinations cancels the call. It's against Asterisk logic, but may be useful sometimes, if all destinations (devices) belongs to a same person.

So, the solution is to use a combination of pre-dial handlers and hangup handlers. 

The code actually - extensions.conf

MAX_CHILD=10

[dial_uas]

exten => _X.,1,NoOp(Call from UAS received)
    same => n,Dial(PJSIP/${EXTEN}@sipp_uas1&PJSIP/${EXTEN}@sipp_uas2,,b(arm_simultaneous_dial^s^1))

[arm_simultaneous_dial]

exten => s,1,NoOp(I am on channel:${CHANNEL})
    same => n,Set(CHANNEL(hangup_handler_push)=simultaneous_dial_hangup_channel,s,1)
    same => n,Set(CHILD_COUNT=1)
    same => n,While($[("x${MASTER_CHANNEL(CHILD_${CHILD_COUNT})}" != "x" && $[${CHILD_COUNT} <= ${MAX_CHILD}])])
    same => n,Set(CHILD_COUNT=$[${CHILD_COUNT} + 1])
    same => n,EndWhile()
    same => n,Set(MASTER_CHANNEL(CHILD_${CHILD_COUNT})=${CHANNEL})
    same => n,Return

[simultaneous_dial_hangup_channel]

exten => s,1,NoOp(Hangup on channel:${CHANNEL})

; you may put here analyze of hangup code, so not to hangup on failed destination, etc.
    same => n,Set(CHILD_COUNT=1)
    same => n,While($[("x${MASTER_CHANNEL(CHILD_${CHILD_COUNT})}" != "x" && $[${CHILD_COUNT} <= ${MAX_CHILD}])])
    same => n,GoToIf($[ "x${CHANNEL}" = "x${MASTER_CHANNEL(CHILD_${CHILD_COUNT})}"]?skip_myself)
    same => n,SoftHangup(${MASTER_CHANNEL(CHILD_${CHILD_COUNT})})
    same => n(skip_myself),Set(CHILD_COUNT=$[${CHILD_COUNT} + 1])
    same => n,EndWhile()
    same => n,Return

Idea is to save child channels id's in sequential CHILD_X variables of the master channel and on hangup one of child's - softhangup other child's.

Full proof of concept is available as docker-compose containing Asterisk and SIPP's on my GitHub.

sexpect - expect alternative in shell

While looking for solution on terminal window resize using expect, found an alternative tool - sexpect. Despite the naming, a great tool actually.

Actually helps me to solve many issues and more of all - combine shell and expect-like scripts in 1 file.

Just to show an simple example (yes, I'm aware of ssh keys):

shell + expect:

#!/bin/bash

...
SSH_PASS=XXXXX
SSH_ADDR=YYYYY

expect <(cat << EOD
    spawn ssh $SSH_ADDR
    expect "Password:"
    send -- "${SSH_PASS}\n"
    interact
EOD
)

shell + sexpect

#!/bin/bash

...
SSH_PASS=XXXXX
SSH_ADDR=YYYYY

export SEXPECT_SOCKFILE=/tmp/sexpect-$$.sock

type -P sexpect >& /dev/null || exit 1

echo "Connecting to $SSH_ADDR..."

sexpect spawn -idle 120 -t 60 $SSH_ADDR

 if [[ $? != 0 ]]; then
    echo "Spawn failed!"
    exit 1
fi

sexpect expect -ex "Password:"
if [[ $? != 0 ]]; then
    echo "Nowhere to enter password!"
    exit 1
fi

sexpect send -enter "$SSH_PASS"

sexpect set -idle 5
sexpect interact

As a bonus - terminal window resize work as expected.

Loop protection on Kamailio/Asterisk

 Sometimes, due to incorrect configuration, even not on the system we control, we can have call loops. Imagine a situation with loop call forward with one of the endpoints is a cell phone and forward is made via another operator. Usually, you can't get all these loops relying only on the internal pre-save mechanism.

 

But there is a simple and efficient method to avoid such loops. Idea is to limit calls from A to B, if there are more than X such calls per second.

 

Proposed solutions are Kamailio and Asterisk's implementation of these scenarios.

Kamailio method is based on htable

...

# Not more than FROM_TO_PER_SECOND_RATIO calls per second from X number to Y number. If it is set to 3,  4th call would be blocked
#!define FROM_TO_PER_SECOND_RATIO 3

...

loadmodule "htable.so"

modparam("htable", "htable", "fu_tu=>size=5;autoexpire=5;")

...

request_route{

...

     # Process only new calls
    if (is_method("INVITE") && !has_totag()) {

        $var(from_to_hash) = $fU + '_' + $tU + '_' + $timef(%H_%M_%S);

        if ($sht(fu_tu=>$var(from_to_hash)) != $null) {
            $sht(fu_tu=>$var(from_to_hash)) = $sht(fu_tu=>$var(from_to_hash)) + 1;
            xlog("L_NOTICE", "Possible loop detected on call $fu -> $tu ($sht(fu_tu=>$var(from_to_hash)) total calls during last second)\n");
        } else {
            xlog("L_INFO", "Loop protection enabled on call $fu -> $tu with $var(from_to_hash) hash)\n");
            $sht(fu_tu=>$var(from_to_hash)) = 1;
        }

        if ($sht(fu_tu=>$var(from_to_hash)) > FROM_TO_PER_SECOND_RATIO) {
            xlog("L_WARN", "Loop detected on call $fu -> $tu (IP:$si:$sp)\n");
            send_reply("482", "Loop detected");
            exit;
        }
    }

...

}

Asterisk method is based on GROUP()

[dial_all]

exten => _X.,1,NoOp(Main call context)

  same => n,GoSub(loop_protection, ${EXTEN}, 1)

...

 

[loop_protection]
; Provide loop protection based on calls per second on same from/to numbers.
exten => _X.,1,NoOp(Loop protection...)
  same => n,Set(GROUP_HASH=${CALLERID(num)}_${EXTEN}_${STRFTIME(${EPOCH},,%Y_%m_%d_%H_%M_%S)})
  same => n,Set(GROUP()=${GROUP_HASH})
  same => n,GoToIf($[${GROUP_COUNT(${GROUP_HASH})}<=3]?limit_ok)
  same => n,NoOp(Total Calls From ${CALLERID(num)} to ${EXTEN} during this second exceeded limit of <%= @loop_cps_ratio %>)
  same => n,Hangup(25)
  same => n(limit_ok),NoOp(Total Calls From ${CALLERID(num)} to ${EXTEN} during this second is: ${GROUP_COUNT(${GROUP_HASH})})
  same => n,Return()

The mechanism is fairly simple on both. Have a hash with the following name - <from_number>_<to_number>_<date_with_second>, calculate it for every new call, and if his hash name already exists - increase it by 1 till limit is reached. After this - drop the call, considering this is a loop.

 

Maybe not the best method, but reliable. Only thing I did not investigate much - possible memory leak on Asterisk with group names. I consider this is something to be tested.

Kamailio and mobile TCP endpoints. Unregistering

The modern world is mobile. You may like it or not. But it is. So, with all these technologies that are about saving your battery life, all application reachability goes to vendor-lock push servers.

 

But the problem emerges in the following. SIP client registers on registrar (Kamailio) with TCP. Yes, you can do SIP keepalives here, but why to have em if we have a built-in mechanism of keepalive in TCP itself? Plus, additional packets over the network will drain the battery faster. And iOS, when putting the app to the background, just cut the network of app off. Literally killing it.

 

But we somehow need to know which actual state of the endpoint is. And here Kamailio is to help us with tcpops module.

 

The idea is quite simple. On each query to location table, we will clean-up it from "dead" TCP connections. So, after save(). And the only actual state of endpoints would be taken in the branch and calling process. Yes, it will give extra initial INVITE on already 'dead' connection, but appears to be, it sometimes cleaning just arrived REGISTER's so it's acceptable.

# Save info for TCP connections for unregister on close/timeout/error
loadmodule "htable.so"

modparam("htable", "htable", "tcpconn=>size=15;autoexpire=7200;")

...

# Handle SIP registrations
route[REGISTRAR] {

   ...

    save("location");
    route(TRACK_TCP_STATE);

    $var(processed_subscriber) = $fu;

    route(TCP_REGISISTER_CLEANUP);

   ...

}

 

route[TRACK_TCP_STATE] {
    if (proto == UDP) {
        return;
    }
    xlog("[TRACK_TCP_STATE] Saving state for future disconect track of $fu\n");

    $sht(tcpconn=>$conid) = $fu;

    tcp_enable_closed_event();
}

# Make sure you set up $var(processed_subscriber) before calling this route

route[TCP_REGISISTER_CLEANUP] {

    if ($var(processed_subscriber) == $null) {
        return;
    }

    xlog("[TCP_REGISISTER_CLEANUP] Processing subscriber $var(processed_subscriber)\n");

    # Getting registered endpoints for this AoR
    if (!reg_fetch_contacts("location", "$var(processed_subscriber)", "subscriber")) {
        $var(processed_subscriber) = $null;
        xlog("[TCP_REGISISTER_CLEANUP] No registered contacts for $var(processed_subscriber)\n");
        return;
    }

    $var(i) = 0;

    # Loop through registered endpoints
    while ($var(i) < $(ulc(subscriber=>count))) {

        $var(stored_subscriber_conid) = $(ulc(subscriber=>conid)[$var(i)]);

        # Make sure proto is TCP
        if ($var(stored_subscriber_conid) != $null) {

            # Check if entry is still active TCP connection. Unregister otherwise.
            if ($var(stored_subscriber_conid) == -1 || !tcp_conid_alive("$var(stored_subscriber_conid)")) {

                $var(stored_subscriber_ruid) = $(ulc(subscriber=>ruid)[$var(i)]);
                $var(stored_subscriber_address) = $(ulc(subscriber=>addr)[$var(i)]);

                xlog("[TCP_REGISISTER_CLEANUP]: Unregistering entry $var(i)/$var(stored_subscriber_conid) -> $var(stored_subscriber_address)\n");

                if (!unregister("location", "$var(processed_subscriber)", "$var(stored_subscriber_ruid)")) {
                    xlog("[TCP_REGISISTER_CLEANUP]: Unregistering entry $var(i)/$var(stored_subscriber_conid) -> $var(stored_subscriber_address) FAILED!\n");
                }
            }
        }
        $var(i) = $var(i) + 1;
    }
    $var(processed_subscriber) = $null;
}

event_route[tcp:closed] {
    xlog("[TCP:CLOSED] $proto connection closed conid=$conid\n");

    $var(processed_subscriber) = $sht(tcpconn=>$conid);

    if ($var(processed_subscriber) != $null) {
        route(TCP_REGISISTER_CLEANUP);
        $sht(tcpconn=>$conid) = $null;
    }
}

event_route[tcp:timeout] {
    xlog("[TCP:TIMEOUT] $proto connection timeout conid=$conid\n");

    $var(processed_subscriber) = $sht(tcpconn=>$conid);

    if ($var(processed_subscriber) != $null) {
        route(TCP_REGISISTER_CLEANUP);
        $sht(tcpconn=>$conid) = $null;
    }
}

event_route[tcp:reset] {
    xlog("[TCP:RESET] $proto reset closed conid=$conid\n");

    $var(processed_subscriber) = $sht(tcpconn=>$conid);

    if ($var(processed_subscriber) != $null) {
        route(TCP_REGISISTER_CLEANUP);
        $sht(tcpconn=>$conid) = $null;
    }
}

Simple and efficient solution. Yes, some of "dead" endpoints would be present in table during expiration time, but I'm ok with that.

Developing for Kamailio locally using Docker.

 The plan was to participate in KWC2020 in the flesh. But pandemic add a correction to this plan. 

Talk on how to have different development environements on a local PC/Mac using Docker and docker-compose.

 

 And links from the presentation:

1. https://github.com/igorolhovskiy/kamailio-dev-dc - Look for the branches

2. https://docs.docker.com/config/containers/runmetrics/

3. https://engineering.squarespace.com/blog/2017/understanding-linux-container-scheduling

4. https://sysdig.com/blog/container-isolation-gone-wrong/

5. https://discuss.linuxcontainers.org/t/cpu-limits-confusion/1837

6. https://docs.docker.com/config/containers/resource_constraints/#configure-the-realtime-scheduler

7. https://lwn.net/Articles/549580/

FusionPBX - VTiger 7 Integration

Every telecom engineer should build a PBX, CallCenter, and CRM Integration.

So, my time for CRM integration came. The idea is quite simple.

1. Calls are registering in CRM with call recording
2. Calls are being notified with popup while using CRM
3. Calls from CRM directly. Click 2 Call actually

Integration is done for popular opensource CRM VTiger 7 and FusionPBX. Have to say, not plain, reworked, my fork actually.

This project consists of 3 parts.

1. PBXManager  - module for VTiger CRM, built on top of the original PBXManager module made for Asterisk.
2. VFusion Daemon - Docker Image that is running with FusionPBX (FreeSwitch) and listening to FreeSwitch events and making requests to VTiger CRM. Made on NodeJS
3. FusionPBX - Actually my fork of it. Version 4.4

I actually tried to avoid daemons or so, resides on FusionPBX server, but really can't avoid it totally. So, architecture is not perfect. Some of the requests are made from FreeSwitch via mod_curl, some - in v_xml_cdr app by Fusion, some - in VFusion daemon. Not clean, but as of now it's working.
So, how it looks like?

1. Call from unknown contact

2. Call from a known contact

 

3. Call details records

yes, you can listen to the recordings directly from CRM.


Click 2 Call is made quite simple. Just click on a phone number inside CRM and you will have a callback to your extension. After picking up, a call to that number would be established.

As it's intended to be a commercial product, code is open. But documentation is still not. Maybe in the future documentation also would be open.

If you have any interest in deploying this product - pls contact me at support <at> consertis.at. Or here.