Friday, October 15, 2021

Functional testing of VoIP infrastructure

 Sometimes, it's really hard to test VoIP infrastructure in an automated way. The usual way of testing any telco system is to take a phone, make some calls and that's it. Maybe to look at the logs. 

Some of the engineers going further and using famous SIPP, which is quite good in testing low-level SIP, but really could be a pain in some closer-to-world scenarios.

Another approach is to write some scripts and automating Asterisk or FreeSWITCH to do some test calls. And it's a good approach, but sometimes writing something simple could take a lot of time. And attention.

But there is one more way - use standalone SIP libraries like baresip or pjsip and control them via API or CLI.

Exactly this way was taken by Julien Chavanton in his voip_patrol project. After a short time of playing around with it, I can say it's a really good combination of simplicity of things it can test and a way it's configuring.

In my current position, I'm dealing mostly with opus/SRTP media (in DTLS-SRTP flavor) and I was able to add support of this  to voip_patrol (in this branch) and it's working well with rtpengine provided SRTP.

And here is just an example of voip_patrol scenario to register TLS endpoint, make a call with SRTP and catch it back. So, a simple register-call test as a good starting point for more complex scenarios. 


  <action type="codec" disable="all"/>
    <action type="codec" enable="pcma" priority="250"/>
    <action type="codec" enable="pcmu" priority="249"/>
    <action type="register" label="Register 88881"
    <action type="wait" complete="true"/>
    <action type="accept" label="Receive all calls"
            code="200" reason="OK"
    <action type="call" label="Call to 88881"
            max_duration="20" hangup="16"
    <action type="wait" complete="true"/>


For more advanced, I suggest to look on issues [1, 2] on GitHub, where author expands some concepts.

And as a good quickstart - video from author

UPDATE: Based on this tool, templating and reporting suite VOLTS was created. Stay tuned for more cool features to come!

Thursday, May 20, 2021

Asterisk and importance of filtering of numbers dialed

 Recently my collegaue found an interesting vector of possible attacks on Asterisk.

Imagine having following construction


exten => _X.,1,GoTo(check_rights,${EXTEN},1)



exten => _X.,1,AGI(my_check.agi)

 same => 2,GoTo(all_ok,${EXTEN},1)

I know, looks a bit dully, but quite common situation. You may believe, that all calls from [endpoints] context will bypass your script of checking auth. my_check.agi, actually.

But imagine calling not to 12345, for example, but 12345,2

What will happen, that line

exten => _X.,1,GoTo(check_rights,${EXTEN},1)

will evaluate to

Goto("PJSIP/anonymous-00000579", "check_rights,12345,2,1") 

So, you can attach desired priority to your number and in example above - just bypass auth.

I'm not saying it's very common case, but don't forget to use something like FILTER

Possible good idea would be using something like

GoToIf($[ "${EXTEN}" == "${FILTER(+0-9,${EXTEN})}" ]?number_ok:number_not_ok)

to filter only + and digits.

Tuesday, March 9, 2021

Kamailio and delayed CANCEL on iOS

One who is working with SIP aware of a problem with mobile devices. Long story short, the only reliable and proper way to deliver SIP calls to mobile devices is push notification.

A usual algo here, the server sends push notification to a device (via Apple or Google PNS respectively) and after mobile device woke up an application, it registers to SIP server and only after this SIP server sends INVITE to application. And last part here - device starts ringing.

So, SIP server have to store info on this call and wait for a device to register. To achieve this, Kamailio offers tsilo module, which is working great. If you haven't seen it or having problems with it, there is a great presentation which is a good starting point.

But there is one problem. After the call is canceled or answered, no more info is stored in this module. So, if push arrives later, than the call was answered, the app will not receive any info after REGISTER. Usually, it's ok, but with iOS > 13, there is a new mechanism of push notifications using for VoIP calls. CallKit (iOS developers may correct me here, cause I'm not). Idea is the following. When iOS device receives push notification for the VoIP call, it first will show a screen with info, that call has arrived. Later, after the app is already wakened up and running, the app is taking control over the calling screen to update caller info, show Accept/Reject buttons, etc. Sounds good. But if the call was canceled (or answered elsewhere) at the moment when Call Screen is shown already, but the app is not ready yet? The app is woken up, sends REGISTER to SIP server, but no INVITE is following. The call is already not this device business. 

Usually, this problem is solved on an application level. Like if an app is not receiving INVITE after push within X seconds, it asks for a remote server for info and shows (or not) corresponding missed call info.

But that's not my case, unfortunately. I'm using slightly rebranded Linphone and not an expert in iOS programming. So, need to resolve everything fully on a server side, where Kamailio is my SIP server.

The idea of the solution is following - if iOS device registers within X seconds after the call start and the call is already canceled or answered - a fake call is simulated to this endpoint. X could be any, but we come to 15 seconds. For simulating call the famous sipp is used.

Here following some parts of the code


# Flag to define that the call was locally generated via SIPP

#!define FLT_LOCALGEN 4


# Save info iOS CallKit workaround
modparam("htable", "htable", "ios_reg=>size=5;autoexpire=2;")
modparam("htable", "htable", "ios_state=>size=5;autoexpire=15")


request_route {


    if (is_method("CANCEL")) {
        # inbound CANCEL

        xlog("L_INFO", "[REQUEST_ROUTE] Detected CANCEL, saving state\n");

        $sht(ios_state=>$rU) = "state=canceled;callid=" + $ci + ";fromnumber=" + $fU;

    # Save info in a case if call was answered to preserve callerID/destination

    if (is_method("INVITE") && !has_totag()) {

        $avp(orig_fU) = $fU;
        $avp(orig_rU) = $rU;



onreply_route {

     if ($rs == 200) {
        # Save call state fr
        xlog("L_INFO", "[MANAGE_REPLY] Call $avp(
orig_fU) -> $avp(orig_rU) was answered\n");

        $sht(ios_state=>$avp(orig_rU)) = "state=answered;callid=" + $ci + ";fromnumber=" + $avp(




   if ($si == 'MY_IP_ADDR') {
        xlog("L_NOTICE", "[AUTH] Packet from ourselves\n");



# We don't want  this INVITE to be processed via tsilo routine.


    if (isflagset(FLT_LOCALGEN)) {


    if (!is_method("INVITE") || isflagset(FLT_LOCALGEN)) {


# Small extra route to be called after succesful save("location")

    if (!($ua =~ "LinphoneiOS*")) {
        xlog("L_INFO", "[REGISTER_IOS] Not IPhone, not interesting...\n");


    xlog("L_INFO", "[REGISTER_IOS] IPhone registered, saving contact for 2 sec\n");

#  Saving location info for later
    $var(ios_contact) = "sip:" + $fU + "@" + $si + ":" + $sp + ";transport=" + $pr;

# Adding some random to support several iOS devices per same account.
    $var(ios_reg_hash_id) = $fU + "_" + $RANDOM;

# information on contact is stored in hash memrory, and the hash key would return to us via X-IOS-Contact-Hash header provided by SIPP.
    $sht(ios_reg=>$var(ios_reg_hash_id)) = $var(ios_contact);

    $var(ios_callstate) = $sht(ios_state=>$fU);

    if (not_empty("$var(ios_callstate)")) {
        $var(call_state) = $(var(ios_callstate){param.value,state});
        $var(call_callid) = $(var(ios_callstate){param.value,callid});
        $var(call_from) = $(var(ios_callstate){param.value,fromnumber});

        xlog("L_INFO", "[REGISTER_IOS] Registration from iOS is late. Call state is $var(call_state)\n");

        # Here we actually start call via SIPP

        exec_msg("/usr/bin/nohup $var(call_callid) $fU $var(ios_reg_hash_id) $var(call_state) $var(call_from) >/dev/null 2>& 1 &");


# Own lookup("location"), but with info saved on [REGISTER_IOS]

    if (!isflagset(FLT_LOCALGEN) || !is_present_hf("X-IOS-Contact-Hash")) {

    $var(ios_reg_hash_id) = $hdr(X-IOS-Contact-Hash);
    $var(ios_contact) = $sht(ios_reg=>$var(ios_reg_hash_id));


    if (!not_empty("$var(ios_contact)")) {
        xlog("L_WARN", "[LOCATION_INTERNAL] Locally generated packet, but no contact was saved\n");

        send_reply("404", "Saved contact not found");

    $ru = $var(ios_contact);
    $du = $ru;



if [ -z "${1}" ]; then
    echo "CallID is not specified"
    exit 1

if [ -z "${2}" ]; then
    echo "To number is not specified"
    exit 1

if [ -z "${3}" ]; then
    echo "HASH ID is not specified"
    exit 1


# Create CSV file for SIPP first
/usr/bin/echo -e "SEQUENTIAL\n${TO_NUMBER};${FROM_NUMBER};'${FROM_NAME}';${HASH_ID};${REASON};" >> /tmp/invite_cancel_$$.csv

# Run SIPP with provided CallID, as it should be same as declared in Push Notification
/usr/bin/sipp localhost -sf invite_cancel.xml -inf /tmp/invite_cancel_$$.csv -m 1 -cid_str ${CALL_ID}
rm /tmp/invite_cancel_$$.csv


 <?xml version="1.0" encoding="UTF-8" ?>

<!-- This sipp scenario combines 2 types of CANCEL with different reasons
Reason "Call Completed Elsewhere" will not get into a missed calls list
CANCEL without a reason will get there
Which CANCEL would be sent, controlled by field 4 in CSV line
If this field == 'answered', call is considered answered and will not get into missed call list
In all other cases call considered missed -->

<scenario name="iOS CallKit canceler">

    INVITE sip:[field0 line=0]@[remote_ip]:[remote_port] SIP/2.0
    Via: SIP/2.0/[transport] [local_ip]:[local_port]
    From: <sip:[field1 line=0]@[local_ip]:[local_port]>;tag=[call_number]_sipp_call_canceler_[call_number]
    To: <sip:[field0 line=0]@[remote_ip]:[remote_port]>
    Call-ID: [call_id]
    Cseq: [cseq] INVITE
    Supported: 100rel, timer, replaces, norefersub
    Session-Expires: 1800
    Min-SE: 90
    Contact: sip:asterisk@[local_ip]:[local_port]
    Max-Forwards: 70
    X-IOS-Contact-Hash: [field3 line=0]
    Content-Type: application/sdp
    Content-Length: [len]

    o=- 558046395 558046395 IN IP[media_ip_type] [media_ip]
    c=IN IP[media_ip_type] [media_ip]
    t=0 0
    m=audio 16170 RTP/AVP 107 8 0 101
    a=rtpmap:107 opus/48000/2
    a=fmtp:107 useinbandfec=1
    a=rtpmap:8 PCMA/8000
    a=rtpmap:0 PCMU/8000
    a=rtpmap:101 telephone-event/8000
    a=fmtp:101 0-16
    m=video 14074 RTP/AVP 100
    a=rtpmap:100 VP8/90000


 <!-- Wait for provisional responces and if something wrong (404 or timeout) - end scenario -->
 <recv response="100" optional="true" timeout="5000" ontimeout="12"/>
 <recv response="404" optional="true" next="12"/>
 <recv response="180" timeout="5000" ontimeout="12" />

<!-- Check if field4 == 'answered' and assign bool result to $3 -->

    <assignstr assign_to="1" value="[field4 line=0]" />
    <strcmp assign_to="2" variable="1" value="answered" />
    <test assign_to="3" variable="2" compare="equal" value="0" />

<!-- If $3 == true, go to label 10  -->
 <nop next="10" test="3" />


    CANCEL sip:[field0 line=0]@[remote_ip]:[remote_port] SIP/2.0
    Via: SIP/2.0/[transport] [local_ip]:[local_port]
    From: <sip:[field1 line=0]@[local_ip]:[local_port]>;tag=[call_number]_sipp_call_canceler_[call_number]
    To: <sip:[field0 line=0]@[remote_ip]:[remote_port]>
    Call-ID: [call_id]
    Cseq: [cseq] CANCEL
    Max-Forwards: 70
    Content-Length: 0


 <!-- Simple goto "11" after receiving 200 -->
 <recv response="200" next="11"/>

 <label id="10"/>


    CANCEL sip:[field0 line=0]@[remote_ip]:[remote_port] SIP/2.0
    Via: SIP/2.0/[transport] [local_ip]:[local_port]
    From: <sip:[field1 line=0]@[local_ip]:[local_port]>;tag=[call_number]_sipp_call_canceler_[call_number]
    To: <sip:[field0 line=0]@[remote_ip]:[remote_port]>
    Call-ID: [call_id]
    Cseq: [cseq] CANCEL
    Max-Forwards: 70
    Reason: SIP;cause=200;text="Call completed elsewhere"
    Content-Length: 0


 <recv response="200" />

 <label id="11"/>

 <recv response="487" />


    ACK sip:[field0 line=0]@[remote_ip]:[remote_port] SIP/2.0
    Via: SIP/2.0/[transport] [local_ip]:[local_port]
    From: <sip:[field1 line=0]@[local_ip]:[local_port]>;tag=[call_number]_sipp_call_canceler_[call_number]
    Call-ID: [call_id]
    Cseq: [cseq] ACK
    Contact: sip:asterisk@[local_ip]:[local_port]
    Content-Length: 0


 <label id="12" />


A bit dirty solution, but it's working.

Thursday, March 4, 2021

Adding call quality parameters to Asterisk CDR

Following presentation from AsterConf 2016 (yes, I know, it was 5 years ago), I've decided to collect similar info on CDR. But this presentation was given a long ago and many things have changed. 

So, a bit updated version.

I'm omitting here connection procedure to the database via odbc, don't want to write the same procedure once again.

And few things to mention - it's for PJSIP. And I'm not using setvar= routine, insead doing it all explicitly in the dialplan.



alias start => calldate
alias sip_callid => sip_callid
alias rtcp_a_all => rtcp_a_all
alias rtcp_a_all_jitter => rtcp_a_all_jitter
alias rtcp_a_all_loss => rtcp_a_all_loss
alias rtcp_a_all_rtt => rtcp_a_all_rtt
alias rtcp_a_txjitter => rtcp_a_txjitter
alias rtcp_a_rxjitter => rtcp_a_rxjitter
alias rtcp_a_txploss => rtcp_a_txploss
alias rtcp_a_rxploss => rtcp_a_rxploss
alias rtcp_a_rtt => rtcp_a_rtt
alias rtcp_b_all => rtcp_b_all
alias rtcp_b_all_jitter => rtcp_b_all_jitter
alias rtcp_b_all_loss => rtcp_b_all_loss
alias rtcp_b_all_rtt => rtcp_b_all_rtt
alias rtcp_b_txjitter => rtcp_b_txjitter
alias rtcp_b_rxjitter => rtcp_b_rxjitter
alias rtcp_b_txploss => rtcp_b_txploss
alias rtcp_b_rxploss => rtcp_b_rxploss
alias rtcp_b_rtt => rtcp_b_rtt


 CREATE TABLE cdr_extra (
        calldate datetime NOT NULL default '0000-00-00 00:00:00',
        clid varchar(80) NOT NULL default '',
        src varchar(80) NOT NULL default '',
        dst varchar(80) NOT NULL default '',
        dcontext varchar(80) NOT NULL default '',
        channel varchar(80) NOT NULL default '',
        dstchannel varchar(80) NOT NULL default '',
        lastapp varchar(80) NOT NULL default '',
        lastdata varchar(80) NOT NULL default '',
        duration int(11) NOT NULL default '0',
        billsec int(11) NOT NULL default '0',
        disposition varchar(45) NOT NULL default '',
        amaflags int(11) NOT NULL default '0',
        accountcode varchar(20) NOT NULL default '',
        uniqueid varchar(32) NOT NULL default '',
        userfield varchar(255) NOT NULL default '',
        sip_callid varchar(255) NOT NULL default '',
        rtcp_a_all varchar(255) NOT NULL default '',
        rtcp_a_all_jitter varchar(255) NOT NULL default '',
        rtcp_a_all_loss varchar(255) NOT NULL default '',
        rtcp_a_all_rtt varchar(255) NOT NULL default '',
        rtcp_a_txjitter decimal(10,6),
        rtcp_a_rxjitter decimal(10,6),
        rtcp_a_txploss int(11) NOT NULL default '0',
        rtcp_a_rxploss int(11) NOT NULL default '0',
        rtcp_a_rtt decimal(10,6),
        rtcp_b_all varchar(255) NOT NULL default '',
        rtcp_b_all_jitter varchar(255) NOT NULL default '',
        rtcp_b_all_loss varchar(255) NOT NULL default '',
        rtcp_b_all_rtt varchar(255) NOT NULL default '',
        rtcp_b_txjitter decimal(10,6),
        rtcp_b_rxjitter decimal(10,6),
        rtcp_b_txploss int(11) NOT NULL default '0',
        rtcp_b_rxploss int(11) NOT NULL default '0',
        rtcp_b_rtt decimal(10,6)




; usual Dial

same => n,Set(CHANNEL(hangup_handler_push)=qos_leg_a,s,1)

same => n,Dial(${destinationStr},,b(set_qos_handler_b^s^1))

exten => s,1,Noop(Channel: ${CHANNEL(name)}, QoS stats RTCP: ${CHANNEL(rtcp,all)})
    same => n,Set(CDR(sip_callid)=${CHANNEL(pjsip,call-id)})
    same => n,GotoIf($["${CHANNEL(rtcp,all)}"==""]?end)
    same => n,Set(CDR(rtcp_a_all)=${CHANNEL(rtcp,all)})
    same => n,Set(CDR(rtcp_a_all_jitter)=${CHANNEL(rtcp,all_jitter)})
    same => n,Set(CDR(rtcp_a_all_loss)=${CHANNEL(rtcp,all_loss)})
    same => n,Set(CDR(rtcp_a_all_rtt)=${CHANNEL(rtcp,all_rtt)})
    same => n,Set(CDR(rtcp_a_txjitter)=${CHANNEL(rtcp,txjitter)})
    same => n,Set(CDR(rtcp_a_rxjitter)=${CHANNEL(rtcp,rxjitter)})
    same => n,Set(CDR(rtcp_a_txploss)=${CHANNEL(rtcp,txploss)})
    same => n,Set(CDR(rtcp_a_rxploss)=${CHANNEL(rtcp,rxploss)})
    same => n,Set(CDR(rtcp_a_rtt)=${CHANNEL(rtcp,rtt)})
    same => n(end),Return()

exten => s,1,Noop(Channel: ${CHANNEL(name)}, QoS stats RTCP: ${CHANNEL(rtcp,all)})
    same => n,GotoIf($["${CHANNEL(rtcp,all)}"==""]?end)
    same => n,Set(CDR(rtcp_b_all)=${CHANNEL(rtcp,all)})
    same => n,Set(CDR(rtcp_b_all_jitter)=${CHANNEL(rtcp,all_jitter)})
    same => n,Set(CDR(rtcp_b_all_loss)=${CHANNEL(rtcp,all_loss)})
    same => n,Set(CDR(rtcp_b_all_rtt)=${CHANNEL(rtcp,all_rtt)})
    same => n,Set(CDR(rtcp_b_txjitter)=${CHANNEL(rtcp,txjitter)})
    same => n,Set(CDR(rtcp_b_rxjitter)=${CHANNEL(rtcp,rxjitter)})
    same => n,Set(CDR(rtcp_b_txploss)=${CHANNEL(rtcp,txploss)})
    same => n,Set(CDR(rtcp_b_rxploss)=${CHANNEL(rtcp,rxploss)})
    same => n,Set(CDR(rtcp_b_rtt)=${CHANNEL(rtcp,rtt)})
    same => n(end),Return()


exten => s,1,Set(CHANNEL(hangup_handler_push)=qos_leg_b,s,1)
    same => n,Return()

As a bonus you have the SIP Call-ID in this CDR, which is useful for debugging SIP traces.

Tuesday, February 9, 2021

Cancel a calls in an Asterisk dialsting

Sometimes life gives strange tasks.

Imagine that you have a dialstring in Asterisk like


But you need to stop the call at the moment when one of the destinations cancels the call. It's against Asterisk logic, but may be useful sometimes, if all destinations (devices) belongs to a same person.

So, the solution is to use a combination of pre-dial handlers and hangup handlers

The code actually - extensions.conf



exten => _X.,1,NoOp(Call from UAS received)
    same => n,Dial(PJSIP/${EXTEN}@sipp_uas1&PJSIP/${EXTEN}@sipp_uas2,,b(arm_simultaneous_dial^s^1))


exten => s,1,NoOp(I am on channel:${CHANNEL})
    same => n,Set(CHANNEL(hangup_handler_push)=simultaneous_dial_hangup_channel,s,1)
    same => n,Set(CHILD_COUNT=1)
    same => n,While($[("x${MASTER_CHANNEL(CHILD_${CHILD_COUNT})}" != "x" && $[${CHILD_COUNT} <= ${MAX_CHILD}])])
    same => n,Set(CHILD_COUNT=$[${CHILD_COUNT} + 1])
    same => n,EndWhile()
    same => n,Return


exten => s,1,NoOp(Hangup on channel:${CHANNEL})

; you may put here analyze of hangup code, so not to hangup on failed destination, etc.
    same => n,Set(CHILD_COUNT=1)
    same => n,While($[("x${MASTER_CHANNEL(CHILD_${CHILD_COUNT})}" != "x" && $[${CHILD_COUNT} <= ${MAX_CHILD}])])
    same => n,GoToIf($[ "x${CHANNEL}" = "x${MASTER_CHANNEL(CHILD_${CHILD_COUNT})}"]?skip_myself)
    same => n,SoftHangup(${MASTER_CHANNEL(CHILD_${CHILD_COUNT})})
    same => n(skip_myself),Set(CHILD_COUNT=$[${CHILD_COUNT} + 1])
    same => n,EndWhile()
    same => n,Return

Idea is to save child channels id's in sequential CHILD_X variables of the master channel and on hangup one of child's - softhangup other child's.

Full proof of concept is available as docker-compose containing Asterisk and SIPP's on my GitHub.

Thursday, February 4, 2021

sexpect - expect alternative in shell

While looking for solution on terminal window resize using expect, found an alternative tool - sexpect. Despite the naming, a great tool actually.

Actually helps me to solve many issues and more of all - combine shell and expect-like scripts in 1 file.

Just to show an simple example (yes, I'm aware of ssh keys):

shell + expect:



expect <(cat << EOD
    spawn ssh $SSH_ADDR
    expect "Password:"
    send -- "${SSH_PASS}\n"

shell + sexpect



export SEXPECT_SOCKFILE=/tmp/sexpect-$$.sock

type -P sexpect >& /dev/null || exit 1

echo "Connecting to $SSH_ADDR..."

sexpect spawn -idle 120 -t 60 $SSH_ADDR

 if [[ $? != 0 ]]; then
    echo "Spawn failed!"
    exit 1

sexpect expect -ex "Password:"
if [[ $? != 0 ]]; then
    echo "Nowhere to enter password!"
    exit 1

sexpect send -enter "$SSH_PASS"

sexpect set -idle 5
sexpect interact

As a bonus - terminal window resize work as expected.