Last year I was working on VOLTS - framework I'm using for testing VoIP deployments in automated way.
Right now it combines tools like voip_patrol and sipp in one tool with some extra features like database write/delete and media file check.
Not to repeat here all README, two links - github and the following video will tell it all.
Do you test your deployments?
After stress-testing Kamailio with sipflood tool from sippts suite (which deserves another article), not so good outcome was faced.
Using CentOS 7 default OpenSSL library (1.0.2k-fips) with using Kamailio 5.4-5.6 with TLS transport, it's quite easy to get a segfault inside tls routines. I've found that roughly 10 000 OPTIONS packets with 200 threads is enough to ruin Kamailio process.
Basically, you can DoS the whole server regardless of it's power just with a single mid-range computer.
Solution was found with using Kamailio 5.6, but with tlsa flavour and latest openssl 1.1.x compiled.
Turns out it's a really simple process.
As we're gonna need to compile Kamailio anyway, assume, that we have all necessary packets for build already on the system.
First - we need to get openssl sources:
# cd /usr/src
# wget https://www.openssl.org/source/openssl-1.1.<latest>.tar.gz
# tar xvf https://www.openssl.org/source/openssl-1.1.<latest>.tar.gz
# cd openssl-1.1.<latest>
# ./config
# make
(Optionally) Here we can make sure that this release is passing tests
# yum install perl-Test-Simple
# make test
Next step - point Kamailio to newly compiled openssl
# cd /usr/src
# wget https://www.kamailio.org/pub/kamailio/5.6.<latest>/src/kamailio-5.6.<latest>_src.tar.gz
# tar xvf kamailio-5.6.<latest>_src.tar.gz
# cd kamailio-5.6.<latest>
# sed -i "s?LIBSSL_STATIC_SRCLIB \?= no?LIBSSL_STATIC_SRCLIB \?= yes?g" ./src/modules/tlsa/Makefile
# sed -i "s?LIBSSL_STATIC_SRCPATH \?= /usr/local/src/openssl?LIBSSL_STATIC_SRCPATH \?= /usr/src/openssl-1.1.<latest>?g" ./src/modules/tlsa/Makefile
...
Than goes your usual Kamailio compiling and don't forget to replace all "tls" module mentions in kamailio.cfg to "tlsa"
Results are much better. But than I've faced, that it's possible to "eat" all TCP connections on Kamailio server with this type of flood.
First - ulimit. Never underestimate defaults.
# ulimit -n unlimited
Next steps - tune TCP stack.
Disclamer: next provided options are discussable and was not found by me and need to be adjusted to your case
kamailio.conf
...
tcp_connection_lifetime=3605
tcp_max_connections=4096
tls_max_connections=4096
tcp_connect_timeout=5
tcp_async=yes
tcp_keepidle=5
open_files_limit=4096
...
/etc/sysctl.conf
...
# To increase the amount of memory available for socket input/output queues
net.ipv4.tcp_rmem = 4096 25165824 25165824
net.core.rmem_max = 25165824
net.core.rmem_default = 25165824
net.ipv4.tcp_wmem = 4096 65536 25165824
net.core.wmem_max = 25165824
net.core.wmem_default = 65536
net.core.optmem_max = 25165824
# To limit the maximum number of requests queued to a listen socket
net.core.somaxconn = 128
# Tells TCP to instead make decisions that would prefer lower latency.
net.ipv4.tcp_low_latency=1
# Optional (it will increase performance)
net.core.netdev_max_backlog = 1000
net.ipv4.tcp_max_syn_backlog = 128
...
This will help, but not fully (at least in my case, I've must miss something and comments here are really welcomed)
As the second part I've decided to go with Fail2Ban and block flood on iptables level.
Setup is quite simple as well.
First - make sure Kamailio will log flood attempts:
kamailio.conf
...
loadmodule "pike.so"
modparam("pike", "sampling_time_unit", 2)
modparam("pike", "reqs_density_per_unit", 30)
modparam("pike", "remove_latency", 120)
...
if (!pike_check_req()) {
xlog("L_ALERT", "[SIP-FIREWALL][FAIL2BAN] $si\n");
$sht(ipban=>$si) = 1;
if ($proto != 'udp') {
tcp_close_connection();
}
drop;
}
...
Next - install and configure Fail2Ban
# yum install -y fail2ban
/etc/fail2ban/jail.local
[DEFAULT]
# Ban hosts for one hour:
bantime = 3600
# Override /etc/fail2ban/jail.d/00-firewalld.conf:
banaction = iptables-multiport
action = %(action_mwl)s
[cernphone-iptables]
enabled = true
filter = mypbx
action = iptables-mypbx[name=mypbx, protocol=tcp, blocktype='REJECT --reject-with tcp-reset']
sendmail[sender=<sender_addr>, dest=<dest_addr> sendername=Fail2Ban]
logpath = <your_kamailio_logfile>
maxretry = 1
bantime = 3600s
findtime = 10s
/etc/fail2ban/action.d/iptables-mypbx.conf
[INCLUDES]
before = iptables-common.conf
[Definition]
actionstart = <iptables> -N f2b-<name>
<iptables> -A f2b-<name> -j <returntype>
<iptables> -I <chain> -p <protocol> -j f2b-<name>
actionstop = <iptables> -D <chain> -p <protocol> -j f2b-<name>
<actionflush>
<iptables> -X f2b-<name>
actioncheck = <iptables> -n -L <chain> | grep -q 'f2b-<name>[ \t]'
actionban = <iptables> -I f2b-<name> 1 -s <ip> -p <protocol> -j <blocktype>
actionunban = <iptables> -D f2b-<name> -s <ip> -p <protocol> -j <blocktype>
/etc/fail2ban/filter.d/mypbx.local
[Definition]
# filter for kamailio messages
failregex = \[SIP-FIREWALL\]\[FAIL2BAN\] <HOST>$
# systemctl enable fail2ban
# systemctl start fail2ban
In this case we will get host banned on iptables level.
Yet another method to get dynamic logging level on Kamailio. Means to change logging level on the fly.
First to mention - already built-in method for Kamailio inside corex module. But this one could be very verbose.
Other method is to specify level in xlog command explicitly.
kamailio.cfg
#!KAMAILIO
# Level for realtime logging for messages. To see debug messages in realtime, set it to 2
realtime.debug_level=5
...
debug=2
...
request_route {
$var(debug_level) = $(sel(cfg_get.realtime.debug_level){s.int});
...
xlog("$var(debug_level)", "This is a debug message\n");
}
And than just adjust this debug_level config variable via shell
# kamcmd cfg.set realtime debug_level 2
to get all xlog messages to be printed, or set it to the greater value like
# kamcmd cfg.set realtime debug_level 5
to get em suppressed.
You can expand this method to have bigger levels of verbosity via different variables, but usually it's enough like this.
Sometimes there is a need to record ongoing call on Asterisk, where usually there is no recording (due to GDPR or smth like this). Usual way with mixmonitor start in CLI requires a channel ID, which is usually not at hand.
Small script that will get calls with caller and/or callee numbers (based on assumption, that these numbers are available in Asterisk channel name) and put recordings of them in /tmp directory. Separated for A/B legs and mixed.
Made for myself to simplify some tasks and not to look for it again.
Usage:
./call_record.sh <CALLER_NUM> [<CALLEE_NUM>]
call_record.sh
#!/bin/bash
NUM_1=${1:-"DUMMY"}
NUM_2=${2:-"DUMMY"}
CHANNELS=$(asterisk -rx 'core show channels' | grep -E "${NUM_1}|${NUM_2}" | awk '{print $1}')
if [ -z ${CHANNELS} ]; then
echo "Cannot find channel"
exit 0
fi
DATE=$(date '+%F-%H-%M-%S')
ID=1
for CHANNEL in $CHANNELS; do
DIRNAME=/tmp/${DATE}-${NUM_1}-${NUM_2}
mkdir -p ${DIRNAME}
REC_COMMAND=`echo ${DIRNAME}/${ID}-mix.wav,r'('${DIRNAME}/${ID}-in.wav')'t'('${DIRNAME}/${ID}-out.wav')'`
echo "Recording ${ID} at ${DIRNAME}"
asterisk -rx "mixmonitor start ${CHANNEL} ${REC_COMMAND}"
ID=$[ ${ID} + 1 ]
done
I'm pretty sure most of us nowadays get familiar with 2-stage ringback on different OTT (Over-The-Top, yes it's a sort of common name) applications. Like Skype/WhatsApp/Telegram. It's when you have 1 ringback when the remote device is "search" or "trying" state (like locating, receiving push-notification) and actually "ringing" state.
And it's a handy thing to indicate these processes for the caller side
With a small hack, it's possible to do on Asterisk as well.
So, the overall schema will look like
Client A Asterisk SIP Proxy Client B
| INVITE | | |
| ---------------> | INVITE | |
| | -------------> | INVITE |
| | 100 - Trying | -----------> |
| Ringback 1 | <------------- | |
| <--------------- | | |
| | | 180 - Ringing|
| | 180 - Ringing | <----------- |
| Ringback 2 | <------------- | |
| <--------------- | | |
| | | |
| | | |
Between INVITE and 180 - Ringing on Client B can pass some seconds in case of a mobile device. I'm not touching here on how to make this schema on SIP Proxy, but not much changed since this presentation in 2015.
So, Asterisk recipe:
extensions.conf
[main_dial]
exten => _X.,1,Progress()
same => n,Playtones(trying)
same => n,Dial(PJSIP/${EXTEN}@my_trunk&Local/keep_ringback@keep_ringback)
[keep_ringback]
exten => keep_ringback,1,NoOp
indicators.conf
...
trying = 425/90,0/75,525/90,0/250,425/90,0/75,525/90,0/3000
The idea here is to add one more channel to Dial() command, which will die instantly after the start. But somehow will keep the Playtones() working.
Looks really hacky, but works. But also I've found a problem, that will get into conflict with Queue() command with r(ring) option in a case if queue members are defined as Local channels and got into this schema as well. So, a bit updated version of this script would look like
extensions.conf
[main_dial]
exten => _X.,1,Progress()
same => n,Playtones(trying)
same => n,Set(KEEP_RINGBACK=)
same => n,ExecIf($[ "${IS_CALL_WITHIN_QUEUE}" == "yes" ]?Set(KEEP_RINGBACK=&Local/keep_ringback@keep_ringback))
same => n,Dial(PJSIP/${EXTEN}@my_trunk${KEEP_RINGBACK})
This was done on Asterisk 13 (which is not supported at the moment), maybe on newer version of Asterisk this one will stop working.
Upd: confirmed it's needed and working on Asterisk 18
Sometimes, it's really hard to test VoIP infrastructure in an automated way. The usual way of testing any telco system is to take a phone, make some calls and that's it. Maybe to look at the logs.
Some of the engineers going further and using famous SIPP, which is quite good in testing low-level SIP, but really could be a pain in some closer-to-world scenarios.
Another approach is to write some scripts and automating Asterisk or FreeSWITCH to do some test calls. And it's a good approach, but sometimes writing something simple could take a lot of time. And attention.
But there is one more way - use standalone SIP libraries like baresip or pjsip and control them via API or CLI.
Exactly this way was taken by Julien Chavanton in his voip_patrol project. After a short time of playing around with it, I can say it's a really good combination of simplicity of things it can test and a way it's configuring.
In my current position, I'm dealing mostly with opus/SRTP media (in DTLS-SRTP flavor) and I was able to add support of this to voip_patrol (in this branch) and it's working well with rtpengine provided SRTP.
And here is just an example of voip_patrol scenario to register TLS endpoint, make a call with SRTP and catch it back. So, a simple register-call test as a good starting point for more complex scenarios.
<config>
<actions>
<action type="codec" disable="all"/>
<action type="codec" enable="pcma" priority="250"/>
<action type="codec" enable="pcmu" priority="249"/>
<action type="register" label="Register 88881"
transport="tls"
account="88881"
username="88881"
password="XXXXX"
registrar="XXXXX"
realm="XXXXX"
expected_cause_code="200"
srtp="dtls,sdes,force"
/>
<action type="wait" complete="true"/>
<action type="accept" label="Receive all calls"
account="default"
hangup="10"
code="200" reason="OK"
transport="tls"
srtp="dtls,sdes,force"
/>
<action type="call" label="Call to 88881"
transport="tls"
expected_cause_code="200"
caller="88882@XXXXX"
callee="88881@XXXXX"
from="sip:88882@XXXXX"
to_uri="88881@XXXXX"
max_duration="20" hangup="16"
username="88882"
password="XXXXX"
realm="XXXXX"
rtp_stats="true"
max_ringing_duration="15"
srtp="dtls,force"
play="/git/voip_patrol/voice_ref_files/reference_8000_12s.wav"
/>
<action type="wait" complete="true"/>
</actions>
</config>
For more advanced, I suggest to look on issues [1, 2] on GitHub, where author expands some concepts.
And as a good quickstart - video from author
UPDATE: Based on this tool, templating and reporting suite VOLTS was created. Stay tuned for more cool features to come!
Recently my collegaue found an interesting vector of possible attacks on Asterisk.
Imagine having following construction
[endpoints]
exten => _X.,1,GoTo(check_rights,${EXTEN},1)
....
[check_rights]
exten => _X.,1,AGI(my_check.agi)
same => 2,GoTo(all_ok,${EXTEN},1)
I know, looks a bit dully, but quite common situation. You may believe, that all calls from [endpoints] context will bypass your script of checking auth. my_check.agi, actually.
But imagine calling not to 12345, for example, but 12345,2.
What will happen, that line
exten => _X.,1,GoTo(check_rights,${EXTEN},1)
will evaluate to
Goto("PJSIP/anonymous-00000579", "check_rights,12345,2,1")
So, you can attach desired priority to your number and in example above - just bypass auth.
I'm not saying it's very common case, but don't forget to use something like FILTER
Possible good idea would be using something like
GoToIf($[ "${EXTEN}" == "${FILTER(+0-9,${EXTEN})}" ]?number_ok:number_not_ok)
to filter only + and digits.
