Asterisk queues, local channels and transfers

 Here comes the same old story about Asterisk queues and using Local channels within it. But with PJSIP flavour.

So, it is known, that local channels in Asterisk can be tricky and might require disable of the optimizations when used in Queues. Yes, famous /n at the end of the channel name so the Queue application can trach the status of the channel

But the drawback of this method is also known and this is transfers calls from the agents. Means when you do a transfer, initial channel is not freed.

Or outgoing calls from the agents. Like if agent do an outbound call and still resides in a queue, it would not be considered as busy.

Many methods to workaround this, and I'm not exception to build own based on GROUP and GROUP_COUNT originating from Asterisk: The Definitive Guide book, but with some modifications regarding new ways of setting variables on PJSIP channels

queues.conf

...

[QUEUE_A]

member => Local/AGENT_A@agents

...

extensions.conf

...

[incoming]

exten => QUEUE_A,1,Queue(QUEUE_A)

[agents]

; First to  check if there are active calls on this agent

exten => AGENT_A,1,ExecIf($[ ${GROUP_COUNT(${EXTEN}@agents)} >= 1 ]?Congestion())

    ; We  need to set GROUP on outgoing channel as current channel will be destroyed upon answer

    same => n,Dial(PJSIP/AGENT_A@trunk,,b(set_group^${EXTEN}^1)

[agents_outgoing] 

; Make sure we're not getting calls to agent when it's on outgoing call

exten => OUT_NUM,1,Gosub(set_group,${CALLERID(num)},1)

    same => n,Dial(PJSIP/ ${EXTEN}@out_trunk)

[set_group]

exten => AGENT_A,1,Set(GROUP(agents)=${EXTEN})

    same => n,Return()

Just small warning, it's more pseudo-code, but just to give overall idea

Importance of creating dialog in Kamailio right

 I know, the things described here might be obvious and considered "for beginners", but everyone can make stupid errors. Even the best of us :)
So, what I'm using Kamailio dialog module for?

• Keepalives with ka_timer parameter, mainly cause of the nature of mobile clients that are unstable due to the mobile network nature
• Presence PUBLISH'es via pua/pua_dialoginfo modules to separate presence server

 

With the second one actually, I've issued some "problems" due to configuration.

Most examples (at least what I've seen) on Kamailio configs, that are operating with the dialogs, have more-or-less this structure:

request_route {
   route(REQINIT);

   if (is_method("INVITE")) {
       dlg_manage();
   }

   route(WITHINDLG);

   ...

   route(AUTH);

   ...

}

There is a problem with this code. And it is, that this code creates dialogs for EVERY INVITE, even if not authed. 

So, the usual flow INVITE - 401 - INVITE (w/auth) - 200 will create 2 dialogs in this case. The first one will be not terminated correctly with connection with pua module. It will generate PUBLISH with the Trying state,  but on 401 - ACK, there would be no corresponding PUBLISH for the terminated state. And as usual, Expire here is 3600, and you will have a lot of "ringing" devices on your presence server.

The answer would be: create dialogs only for authed (and valid) INVITEs. This will also save some CPU not to create dialogs for spammers.

request_route {
   route(REQINIT);

   route(WITHINDLG);

   ...

   route(AUTH);

   if (is_method("INVITE")) {
       dlg_manage();
   }

   ...

}

Adding a bit of privacy to Kamailio's pua_dialoginfo module

Kamailio is a great product. With really good customisation possibilities.One of examples will follow here.

Task is the following: we have a setup with 2 different servers, proxy and presence server. Presence server is simple and just processing SUBSCRIBE/PUBLISH messages and generating NOTIFY's. Mainly for BLF's.  But users wants to have some privacy in a question, than not everyone can subscribe to anyone and more interesting - monitor who's calling whom. But some of the users, especially in boss-assistant scenarios want to have full information about calls. Like assistant of the boss usually should know who's calling to boss phone and answer (pickup) in some cases. 

Kamailio allows to achieve this scenario, but with a bit of customisation.

Idea is to have concept of 2 groups. 

Group1 for ACL about possibility of user monitor other user in general, second one - having ACL for seeing call details.

Here I'll point on idea with some code examples for proxy part. And just to mention, pickup functions are implemented not on proxy server, so it's not broken for me here.

Proxy

kamailio.cfg

loadmodule "pua.so"
loadmodule "pua_dialoginfo.so"
...
modparam("pua", "db_url", DBURL)
modparam("pua", "outbound_proxy", "sip:<PRESENCE_SERVER>:5060")
modparam("pua", "db_mode", 0)
...
modparam("pua_dialoginfo", "include_callid", 0)
modparam("pua_dialoginfo", "include_tags", 0)
# we want to have info on who's calling
modparam("pua_dialoginfo", "include_localremote", 1)
... 
# Table to allow subscription one on each other. If users share the same group - they can subscribe one to each other.
# By default all shares the same group = 1. If group == 0, that means nobody can see your state.
# Format is <extension>=><list_of_groups_comma_separated>
modparam("htable", "htable", 'presence_view_grp=>size=15;')
# Table to allow subscribers view each other call details data. By default it's forbidden.
modparam("htable", "htable", 'presence_view_details_grp=>size=15;')
# You can fill these tables via database or whatever else method you prefer

...
route(PRESENCE_PRIVACY);
route(WITHINDLG);
...

route(AUTH)
route(PRESENCE)
...
# Presence server route
route[PRESENCE] {
    if(!is_method("PUBLISH|SUBSCRIBE")) {
        return;
    }
    # Check if user is registered
    if (is_method("SUBSCRIBE") && !(registered("location", "$fu", 4) == 1)) {
        append_to_reply("Retry-After: 10\r\n");
        send_reply("500", "Retry Later");
        exit;
    }

   $du = "sip:<PRESENCE_SERVER>:5060";
   
    # By default - allow anyone subscribe to anyone.
    $var(presence_grp_from) = "1";
    $var(presence_grp_to) = "1";

    if ($sht(presence_view_grp=>$fU) != $null) {
        $var(presence_grp_from) = $sht(presence_view_grp=>$fU);
    }

    if ($sht(presence_view_grp=>$tU) != $null) {
        $var(presence_grp_to) = $sht(presence_view_grp=>$tU);
    }


    if (is_method("SUBSCRIBE")) {
        if ((str)$var(presence_grp_to) == "0") {
            # Presence group "0" means fully private.

            send_reply("404", "Subscriber not exists");
            exit;
        }

        $var(group_1) = $var(presence_grp_to);
        $var(group_2) = $var(presence_grp_from);

        if (!route(CHECK_GROUP_INTERSECTION)) {
            # No common groups for From/To
            send_reply("404", "Subscriber not exists");
            exit;
        }
    }

    route(RELAY);
}

route[PRESENCE_PRIVACY] {
    # We're processing only NOTIFY's with Dialog XML data here
    if (!is_method("NOTIFY") || !has_body("application/dialog-info+xml")) {
        return;
    }

    if ($sht(presence_view_details_grp=>$fU) == $null || $sht(presence_view_details_grp=>$tU) == $null) {
        # By default all call data is private
        route(PRESENCE_CLEAR_CALL_DETAILS);
        return;
    }

    $var(group_1) = $sht(presence_view_details_grp=>$tU);
    $var(group_2) = $sht(presence_view_details_grp=>$fU);

    if (!route(CHECK_GROUP_INTERSECTION)) {
        # No common groups for From/To
        route(PRESENCE_CLEAR_CALL_DETAILS);
        return;
    }

    # At this point all is ok.

}

route[PRESENCE_CLEAR_CALL_DETAILS] {

    # Forming new body of the presence info
    # For replace trick see https://lists.kamailio.org/pipermail/sr-users/2022-February/114185.html
    $xml(body=>doc) = $(rb{s.replace,xmlns=,xyzwq=});
   
    # Forming new XML
    $var(new_body) = "<?xml version=\"1.0\"?>\n";
    $var(new_body) = $var(new_body) + "<dialog-info xmlns=\"" + $xml(body=>xpath:/dialog-info/@xyzwq) + "\"";
    $var(new_body) = $var(new_body) + " version=\"" + $xml(body=>xpath:/dialog-info/@version) + "\"";
    $var(new_body) = $var(new_body) + " state=\"" + $xml(body=>xpath:/dialog-info/@state) + "\"";
    $var(new_body) = $var(new_body) + " entity=\"" + $xml(body=>xpath:/dialog-info/@entity) + "\">\n";
    $var(new_body) = $var(new_body) + "  <dialog id=\"" + $xml(body=>xpath:/dialog-info/dialog/@id) + "\"\n";
    $var(new_body) = $var(new_body) + " direction=\"" + $xml(body=>xpath:/dialog-info/dialog/@direction) + "\">\n";
    $var(new_body) = $var(new_body) + "    <state>" + $xml(body=>xpath:/dialog-info/dialog/state/text()) + "</state>\n";
    $var(new_body) = $var(new_body) + "  </dialog>\n</dialog-info>";

    replace_body_atonce("^.+$", $var(new_body));
}

route[CHECK_GROUP_INTERSECTION] {
# Checks if contents of $var(group_1) and $var(group_2) that are comma-separated list of groups have at least one in common
    $var(group_1_param_count) = $(var(group_1){s.count,,});

    while ($var(group_1_param_count) >= 0) {
        $var(group_1_param) = $(var(group_1){s.select,$var(group_1_param_count),,});

        if (in_list("$var(group_1_param)", "$var(group_2)", ",")) {
            return 1;
        }
        $var(group_1_param_count) = $var(group_1_param_count) - 1;
    }
    return -1;
}

Idea here is we're stripping local/remote data from NOTIFY XML that indicates a dialog. In this case we're indicating fact of the call, but not exposing any sensitive data.

VOLTS - framework for testing VoIP deployments

 Last year I was working on VOLTS - framework I'm using for testing VoIP deployments in automated way. 

Right now it combines tools like voip_patrol and sipp in one tool with some extra features like database write/delete and media file check.

Not to repeat here all README, two links - github and the following video will tell it all.

 

Do you test your deployments?

Protect Kamailio from TCP/TLS flood

 After stress-testing Kamailio with sipflood tool from sippts suite (which deserves another article), not so good outcome was faced.

Using CentOS 7 default OpenSSL library (1.0.2k-fips) with using Kamailio 5.4-5.6 with TLS transport, it's quite easy to get a segfault inside tls routines. I've found that roughly 10 000 OPTIONS packets with 200 threads is enough to ruin Kamailio process.

Basically, you can DoS the whole server regardless of it's power just with a single mid-range computer.

Solution was found with using Kamailio 5.6, but with tlsa flavour and latest openssl 1.1.x compiled.

Turns out it's a really simple process. 

As we're gonna need to compile Kamailio anyway, assume, that we have all necessary packets for build already on the system.

First - we need to get openssl sources:

# cd /usr/src

# wget https://www.openssl.org/source/openssl-1.1.<latest>.tar.gz

# tar xvf https://www.openssl.org/source/openssl-1.1.<latest>.tar.gz

# cd  openssl-1.1.<latest>

#  ./config

# make

(Optionally) Here we can make sure that this release is passing tests

# yum install perl-Test-Simple

# make test

Next step - point Kamailio to newly compiled openssl

# cd /usr/src

# wget https://www.kamailio.org/pub/kamailio/5.6.<latest>/src/kamailio-5.6.<latest>_src.tar.gz

# tar xvf kamailio-5.6.<latest>_src.tar.gz

# cd kamailio-5.6.<latest>

#  sed -i "s?LIBSSL_STATIC_SRCLIB \?= no?LIBSSL_STATIC_SRCLIB \?= yes?g" ./src/modules/tlsa/Makefile

# sed -i "s?LIBSSL_STATIC_SRCPATH \?= /usr/local/src/openssl?LIBSSL_STATIC_SRCPATH \?= /usr/src/openssl-1.1.<latest>?g" ./src/modules/tlsa/Makefile

...

Than goes your usual Kamailio compiling and don't forget to replace all "tls" module mentions in kamailio.cfg to "tlsa"

Results are much better. But than I've faced, that it's possible to "eat" all TCP connections on Kamailio server with this type of flood.

First - ulimit. Never underestimate defaults.  

# ulimit -n unlimited

Next steps - tune TCP stack.

Disclamer: next provided options are discussable and was not found by me and need to be adjusted to your case

kamailio.conf

...

tcp_connection_lifetime=3605
tcp_max_connections=4096
tls_max_connections=4096
tcp_connect_timeout=5
tcp_async=yes
tcp_keepidle=5
open_files_limit=4096

...

/etc/sysctl.conf

...

# To increase the amount of memory available for socket input/output queues
net.ipv4.tcp_rmem = 4096 25165824 25165824
net.core.rmem_max = 25165824
net.core.rmem_default = 25165824
net.ipv4.tcp_wmem = 4096 65536 25165824
net.core.wmem_max = 25165824
net.core.wmem_default = 65536
net.core.optmem_max = 25165824

# To limit the maximum number of requests queued to a listen socket
net.core.somaxconn = 128

# Tells TCP to instead make decisions that would prefer lower latency.
net.ipv4.tcp_low_latency=1

# Optional (it will increase performance)
net.core.netdev_max_backlog = 1000
net.ipv4.tcp_max_syn_backlog = 128
...

This will help, but not fully (at least in my case, I've must miss something and comments here are really welcomed)

As the second part I've decided to go with Fail2Ban and block flood on iptables level.

Setup is quite simple as well.

First - make sure Kamailio will log flood attempts:

kamailio.conf

...

 loadmodule "pike.so"

modparam("pike", "sampling_time_unit", 2)
modparam("pike", "reqs_density_per_unit", 30)
modparam("pike", "remove_latency", 120)

...

if (!pike_check_req()) {
            xlog("L_ALERT", "[SIP-FIREWALL][FAIL2BAN] $si\n");

            $sht(ipban=>$si) = 1;
            if ($proto != 'udp') {
                tcp_close_connection();
            }
            drop;
        }

...

Next - install and configure Fail2Ban

# yum install -y fail2ban

 /etc/fail2ban/jail.local

[DEFAULT]
# Ban hosts for one hour:
bantime = 3600

# Override /etc/fail2ban/jail.d/00-firewalld.conf:
banaction = iptables-multiport
action      = %(action_mwl)s

[cernphone-iptables]
enabled  = true
filter   = mypbx
action   = iptables-mypbx[name=mypbx, protocol=tcp, blocktype='REJECT --reject-with tcp-reset']
           sendmail[sender=<sender_addr>, dest=<dest_addr> sendername=Fail2Ban]
logpath  = <your_kamailio_logfile>
maxretry = 1
bantime  = 3600s
findtime = 10s
 

 /etc/fail2ban/action.d/iptables-mypbx.conf

[INCLUDES]

before = iptables-common.conf

[Definition]

actionstart = <iptables> -N f2b-<name>
              <iptables> -A f2b-<name> -j <returntype>
              <iptables> -I <chain> -p <protocol> -j f2b-<name>

actionstop = <iptables> -D <chain> -p <protocol>  -j f2b-<name>
             <actionflush>
             <iptables> -X f2b-<name>


actioncheck = <iptables> -n -L <chain> | grep -q 'f2b-<name>[ \t]'

actionban = <iptables> -I f2b-<name> 1 -s <ip> -p <protocol> -j <blocktype>

actionunban = <iptables> -D f2b-<name> -s <ip> -p <protocol> -j <blocktype>

 

 /etc/fail2ban/filter.d/mypbx.local

[Definition]
# filter for kamailio messages
failregex = \[SIP-FIREWALL\]\[FAIL2BAN\] <HOST>$
 

# systemctl enable fail2ban

# systemctl start fail2ban

In this case we will get host banned on iptables level.

Kamailio dynamic logging level

 Yet another method to get dynamic logging level on Kamailio. Means to change logging level on the fly.

First to mention - already built-in method for Kamailio inside corex module. But this one could be very verbose.

Other method is to specify level in xlog command explicitly.

kamailio.cfg

#!KAMAILIO

# Level for realtime logging for messages. To see debug messages in realtime, set it to 2
realtime.debug_level=5

...

debug=2

...

request_route {

    $var(debug_level) = $(sel(cfg_get.realtime.debug_level){s.int});

    ...

    xlog("$var(debug_level)", "This is a debug message\n");

}

And than just adjust this debug_level config variable via shell

# kamcmd cfg.set realtime debug_level 2

to get all xlog messages to be printed, or set it to the greater value like

# kamcmd cfg.set realtime debug_level 5

to get em suppressed.  

You can expand this method to have bigger levels of verbosity via different variables, but usually it's enough like this.

P.S.: Also, as an alternative (or build-in method) there is a possibility to use corex module functionality

Small script to record calls on Asterisk on the fly

 Sometimes there is a need to record ongoing call on Asterisk, where usually there is no recording (due to GDPR or smth like this). Usual way with mixmonitor start in CLI requires a channel ID, which is usually not at hand.

 Small script that will get calls with caller and/or callee numbers (based on assumption, that these numbers are available in Asterisk channel name) and put recordings of them in /tmp directory. Separated for A/B legs and mixed.

Made for myself to simplify some tasks and not to look for it again.

Usage:

./call_record.sh <CALLER_NUM> [<CALLEE_NUM>]

call_record.sh

#!/bin/bash

NUM_1=${1:-"DUMMY"}
NUM_2=${2:-"DUMMY"}

CHANNELS=$(asterisk -rx 'core show channels' | grep -E "${NUM_1}|${NUM_2}" | awk '{print $1}')

if [ -z ${CHANNELS} ]; then
    echo "Cannot find channel"
    exit 0
fi

DATE=$(date '+%F-%H-%M-%S')
ID=1

for CHANNEL in $CHANNELS; do

    DIRNAME=/tmp/${DATE}-${NUM_1}-${NUM_2}

    mkdir -p ${DIRNAME}

    REC_COMMAND=`echo ${DIRNAME}/${ID}-mix.wav,r'('${DIRNAME}/${ID}-in.wav')'t'('${DIRNAME}/${ID}-out.wav')'`

    echo "Recording ${ID} at ${DIRNAME}"

    asterisk -rx "mixmonitor start ${CHANNEL} ${REC_COMMAND}"
    
    ID=$[ ${ID} + 1 ]

done