Installation

Uniloader is installed from a .tar.gz package, using the install.sh script it contains. This works on any system running Systemd, including CentOS-based and RPM-based PBX systems (FreePBX, PBXact, etc.), Debian and Ubuntu. If your system does not run Systemd, refer to the manual install below.

The Loway yum repository is no longer available, so yum install uniloader and yum update cannot be used anymore. If you installed Uniloader with yum in the past, see Migrating from the yum (RPM) installation.

Standard installation (all systems)

Installing Uniloader is very easy. First find the package you want to download under https://downloads.loway.ch/software/uniloader/

mkdir -p /opt/loway/uniloader
cd /opt/loway/uniloader
wget https://downloads.loway.ch/software/uniloader/uniloader-26.10.1.tar.gz
tar zxvf uniloader-26.10.1.tar.gz
cd uniloader-26.10.1

./install.sh

The automated install script will:

  • install Uniloader system-wide (so you can call it from any shell)

  • create configuration and init scripts for all services below

  • start the service uniloader (though it won’t work until you edit its credentials and restart it) and make it startable on reboots

  • install a script called ./check.sh that you can use to check the settings for each of the services, by using the configuration you specified for each unit. This makes it very easy to make sure it is correct.

For each service, we keep a separate configuration file under /etc, that is the one you are generally supposed to edit. If you make a change to the configuration files, make sure you issue a systemctl restart uniloader so that the new configuration can be picked up.

If you choose to enable any other service, it’s up to you to make sure that it is correctly restarted on boot.

To update, just download the new version of Uniloader as you did, and run the install script again. When updating, configuration files are not changed, while unit files are replaced with the current version.

A quick recap of Systemd commands might help:

  • systemctl restart uniloader will restart the service, so a new configuration can be picked up

  • systemctl status uniloader will show the status of the service (if it’s running or not), and the last lines of its logs.

  • journalctl -u uniloader will show a complete log, that you can then filter appropriately. You can pass multiple -u services that you want to see, and you’ll see the logs in the correct order.

  • systemctl enable uniloader will make sure that this service will start on boot

  • systemctl daemon-reload is needed if you make any change to the unit scripts. Usually not needed.

  • journalctl --disk-usage shows the size of the logs

Uniloader

  • Service name: uniloader

  • Configuration file: /etc/uniloader

  • Testing: ./check.sh upload that will use the current configuration and make sure the destination QM server is reachable.

We strongly suggest always using the HTTP driver to connect to QM - this makes it easy to send data to a local instance, a remote instance you control, or a QueueMetrics Live instance. Remember to enable the user webqloader on QM, and set a password for it.

URI=http://127.0.0.1:8080/queuemetrics
LOGIN=webqloader
UPASSWD=Secret1234
TOKEN=

PASS or UPASSWD? The uniloader program always receives the password through the environment variable UPASSWD (or through the --pass option). What changes between installations is the name used in the configuration file, and it depends on how the service unit reads it:

  • If the unit file contains the line Environment=UPASSWD=${PASS}, the password must be set as PASS= in the configuration file. This is the case for most older and manually-created units, and for the files of old RPM installations (/etc/sysconfig/uniloader).

  • If the unit file does not contain that line, set UPASSWD= in the configuration file, as in the example above.

To see which one applies to you, run systemctl cat uniloader and look for UPASSWD. Use only one of the two names, and do not rename it when you edit the file: just change its value. If the service starts but the log shows an authentication error even if the password is right, the wrong variable name is the most likely cause.

Make sure you restart the service after making any changes.

You can also turn-on the splitter for Uniloader by uncommenting the right line, and the editing the example splitter file and /etc/uniloader_splitter.json

Unitracker

  • Service name: unitracker

  • Configuration file: /etc/unitracker - you must set up the correct AMI credentials

  • Testing: ./check.sh unitracker will make sure that the AMI credentials used by Unitracker are working

  • This service must be enabled manually if needed.

By default, only MOH tracking is enabled. You may want to enable parkings or, more commonly, outbound tracking. In this case, do not forget to set up the "hidden channels" mask to decide which calls you don’t want to track.

AudioVault

  • Service name: audiovault

  • Configuration file: /etc/audiovault

  • Testing: ./check.sh audiovault that will try requesting a (non-existent) file as if it were an external client. You are expected to see no errors there.

  • This service must be enabled manually if needed

  • In order to run this effectively in production, it’s strongly advisable to put an HTTPS proxy in front, as per its documentation.

Freeswitch

  • Service name: uniloader-freeswitch

  • Configuration file: /etc/uniloader-freeswitch

  • Testing: ./check.sh freeswitch that will check ESL credentials

  • This service must be enabled manually if needed

  • When enabiling this service, you must make sure that the queue_log file that this service generates and the one being imported by Uniloader match - so do not forget to check /etc/uniloader as well.

  • This service is meant to run all of the time and never being restarted, as it will lose events if it is restarted during its operation.

Migrating from the yum (RPM) installation

If you installed Uniloader with yum install uniloader, you can no longer update it with yum. To upgrade, remove the RPM package and install the .tar.gz package as described in Standard installation (all systems). From then on, every update is done by running install.sh again. You need root access and a system running Systemd; plan a short maintenance window, as data is not uploaded while Uniloader is stopped.

The two installations place their files in different locations:

yum (RPM) installation tar.gz with install.sh

Updates

yum update

download the new package and run install.sh again

Program files

/usr/local/uniloader

installed system-wide by install.sh

Uniloader configuration

/etc/sysconfig/uniloader

/etc/uniloader

Unitracker configuration

/etc/sysconfig/unitracker

/etc/unitracker

Service control

service uniloader restart

systemctl restart uniloader

Follow these steps:

  1. Back up your configuration files. Save every file you use, and write down the installed version with uniloader --version:

    tar czf /root/uniloader-config-backup-$(date +%F).tar.gz \
      $(ls -d /etc/sysconfig/uniloader* /etc/sysconfig/unitracker* \
              /etc/uniloader* /etc/unitracker* /etc/audiovault* \
              /etc/defaults/uniloader* 2>/dev/null)

    If you start Uniloader in other ways (for example from /etc/rc.local or a cron job), save those files too.

  2. Remove the RPM package.

    service uniloader stop
    yum remove uniloader
    rpm -qa | grep -i uniloader

    The last command should print nothing. If you do not use the Loway repository for any other package, you can also delete /etc/yum.repos.d/loway.repo.

  3. Install the new package with install.sh, as described in Standard installation (all systems).

  4. Restore your settings in the new configuration files under /etc, using your backup as a reference. Copy the values one by one (URI, LOGIN, the password, TOKEN) instead of overwriting the new file with the old one: the new files contain comments that explain each setting. Pay attention to the name of the password variable: see the PASS or UPASSWD tip above. Settings that only served the old init script, such as the log, lock and pid file paths, are not needed anymore; logs are now available with journalctl -u uniloader.

  5. Start and check. Restart the service and test the connection:

    systemctl restart uniloader
    ./check.sh upload

    Then place a test call and make sure that it appears in QueueMetrics.

Only the uniloader service is started by install.sh. If you used other services, such as the tracking service, set up the configuration file under /etc, test it with ./check.sh <service> and enable it with systemctl enable --now <service>.

If you were using a direct MySQL connection (a URI starting with mysql:), you can keep it, copying URI, LOGIN, the password and TOKEN (the partition) exactly as they were. We suggest instead to switch to the HTTP driver with the webqloader user: see Switching from the MySQL connection to the webqloader user.

Switching from the MySQL connection to the webqloader user

Uniloader can upload data to QueueMetrics in two ways:

  • MySQL (the old way): Uniloader writes directly to the QueueMetrics database (URI starting with mysql:). This is the method used by older installations, and it is still supported.

  • HTTP/S (the new way): Uniloader sends data to QueueMetrics over HTTP/S, using the webqloader user of QueueMetrics (URI starting with http: or https:). This is now the recommended method.

The HTTP/S connection is better because:

  • the PBX does not need to reach the MySQL server, and does not need to know the database credentials;

  • it works in the same way with a local instance, a remote instance or QueueMetrics Live;

  • when running over HTTP, Uniloader can act as a proxy for the actions QueueMetrics sends to the PBX (log agents on and off, pause them, listen to calls, etc.);

  • data deduplication is available (it requires QueueMetrics 19 or later and the HTTP interface).

To switch:

  1. Enable the webqloader user in QueueMetrics. On QueueMetrics Live it is already enabled; on an on-premise installation, go to Users, find webqloader, tick Enabled, set a strong password and save. The user must hold the keys WQLOADER, USER and ROBOT.

  2. Edit /etc/uniloader. Comment out the MySQL settings and set the HTTP ones:

    #URI="mysql:tcp(127.0.0.1:3306)/queuemetrics?allowOldPasswords=1"
    #LOGIN=queuemetrics
    URI=http://127.0.0.1:8080/queuemetrics
    LOGIN=webqloader
    UPASSWD=Secret1234
    TOKEN=

    The URI is the base address of QueueMetrics, as you would use it in a browser, with https:// if QueueMetrics is behind SSL. For the name of the password variable, see the PASS or UPASSWD tip above.

  3. Mind the token. In MySQL mode the token holds the partition (such as P001); in HTTP/S mode it is usually left blank. In a cluster, it holds the name of the cluster member instead. If you were using a partition, check your cluster configuration before switching.

  4. Restart and check.

    systemctl restart uniloader
    ./check.sh upload

    Then place a test call and make sure it appears in QueueMetrics. If the login fails, check that webqloader is enabled and that the password variable has the right name.

Once everything works, the MySQL account that Uniloader used is no longer needed: if it was dedicated to the PBX, you can remove it and close access to the MySQL port from the PBX.

Manual installation

The Uniloader can be downloaded from https://downloads.loway.ch/software/uniloader/

The package contains:

  • Uniloader binaries for all supported architectures (i386, amd64, arm7, arm64),

  • A sample extensions_queuemetrics file,

  • A sample splitter file.

Just copy the file "uniloader_xxx" for your architecture (Intel 32 / 64 bit, ARM 7, ARM64) into your computer and make it executable:

cp ./bin/uniloader_arm7 ./uniloader
chmod a+x ./uniloader

To test it, run:

./uniloader -?

It should output a result like:

NAME:
   uniloader - the data upload companion for QueueMetrics and QueueMetrics Live.

USAGE:
   uniloader [global options] command [command options] [arguments...]

VERSION:
   22.11.3 - build: 1234-20230511.0849/8300761 - OS: darwin/arm64 - RT: go1.17.5

COMMANDS:
   help, h  Shows a list of commands or help for one command

   Diagnostics:
     pbxinfo  Loads PBX information
     test     Runs environment tests

   Services:
     upload, u  Uploads a source file to a QueueMetrics or QueueMetrics Live instance
     track, t   Tracks Asterisk events and creates relevant queue_log entries.
     fsw        Parses FreeSwitch mod_callcenter events
     av         Runs AudioVault - a media storage server for QM (experimental)

   Tools:
     cfgfile   Reads and writes a configuration file
     user      Creates or alters users
     qmdb      Performs direct operations on QueueMetrics' database
     asterisk  Performs direct operations on Asterisk queues

GLOBAL OPTIONS:
   --src value, -s value  The source queue_log file to be uploaded (default: "/var/log/asterisk/queue_log")
   --cacert value         An optional CA Cert file in .pem format, or 'insecure' to skip any certificate validation
   --verbose-back-end     Enables verbose back-end logging. Default: false.
   --read-pipe            The source file is a Unix pipe
   --help, -h             show help
   --version, -v          print the version

If it does, it is working.

Running in production

Uniloader produces a verbose log on STDOUT that should be redirected to a file and periodically rotated.

You should also make sure that Uniloader is started when the PBX boots and that in case it should crash it is automatically restarted.

We advise running Uniloader using 'nice' so that it has reduced access to scarce CPU resources in case of high load / contention with the PBX - while the PBX voice quality quickly degrades on a resource-starved system, Uniloader does not really care about small delays in data uploading.

We also advise adding Uniloader to the command path, so that its many debugging functions can be accessed easily.

Example: Uploading data to a local QueueMetrics system

This is the most common scenario when using a locally installed QueueMetrics system.

nohup nice \
./uniloader -s /var/log/asterisk/queue_log \
	upload --uri "mysql:tcp(1.2.3.4:3306)/queuemetrics?allowOldPasswords=1" \
    		--login queuemetrics --pass javadude --token P001 \
        	>> /var/log/uniloader.log &

Will upload the queue_log file located at /var/log/asterisk/queue_log to the remote "queuemetrics" database on server 1.2.3.4 with login "queuemetrics" and password "javadude", using the default partition "P001".

After you start it, check the file /var/log/uniloader.log to make sure there are no errors. The most common error is that you did not create the correct grants for your MySQL user to upload data remotely.

If everything seems to work, log in into QueueMetrics, select "Edit system parameters" and make sure that the default partition is P001.

# This is the default queue log file.
default.queue_log_file=sql:P001

At this point, log off; log on again and click on "Mysql storage information"; select partition P001 and select "Autoconfigure queues". Now the default queue "00 All" will include all of your queues and you can see your historical and real-time status.

Example: Uploading data to a QueueMetrics Live system

You received your QueueMetrics Live access information; and in the received email you find that your instance is called "ABCD" and the password is "1234".

nohup nice \
./uniloader --src=/var/log/asterisk/queue_log \
     upload --uri https://my.queuemetrics-live.com/ABCD \
            --login webqloader --pass 1234 \
            >> /var/log/uniloader.log &

At this point Uniloader will start feeding the remote database. You can login at any time by visting the address https://my.queuemetrics-live.com/ABCD and logging in as "demoadmin" and the password you were given.

The first time you log in, click on "System diagnostic tools" and then "Live DB inspector" to see data being uploaded. When your database is complete (this may take a few minutes, depending on how much data is on your PBX), go back to the home page and click on "Mysql storage information"; select partition P001 and select "Autoconfigure queues". Now the default queue "00 All" will include all of your queues and you can see your historical and real-time status.