Linux

What is a webhook? A webhook is an endpoint on your server which allows you to execute a particular task. Webhooks are usually triggered by some event. A good use-case for a webhook is running tests on a dedicated test server or deploying your latest master branch to staging/production. Github / Gitlab / Bitbucket allow you to specify a webhook URL in your repository settings. Github triggers the webhook which sends the event data on every push. Webhook server Webhook is a very useful golang project which runs any script you specify when a particular endpoint is hit. Download and extract the binary for your operating system from the releases page. For Linux, it is here. The program takes as config a hooks.json file: [ { "id": "hello-world", "execute-command": "/home/user/scripts/hello.sh", "command-working-directory": "/home/user/webhook" } ] Replace user with the username of your linux user. The hello.sh script. #!/bin/bash echo 'Hello!' Make the script executable by running chmod +x hello.sh Start webhook server as webhook -hooks hooks.json -hotreload -logfile webhooks.log. The server will run on port 9000 by default. You can check if everything is working by running curl http://localhost:9000/hooks/hello-world. This will print “Hello!” in the log file. ...

Setting display brightness using xrandr I auto-adjust the brightness on my display using a cron which runs at 7 pm everyday. xrandr --output eDP-1 --brightness 0.9 This sets the brightness to 90% of maximum brightness. Working with Linux Display environments in a cron The above command works when run from my terminal. But when run on a cron, it does not produce any effect. We need to explicitly mention the “display number” as an environment variable DISPLAY. 8 19 * * * DISPLAY=:0 xrandr --verbose --output eDP-1 --brightness 0.9 How to get the display number of your monitor? Running xrandr will give you a lot of output like Screen 0: minimum 320 x 200, current 1920 x 1080, maximum 16384 x 16384 eDP-1 connected primary 1920x1080+0+0 (normal left inverted right x axis y axis) In my case, the display number is :0 Adjusting backlight brightness vs perceived brightness If xrandr is unable to find backlight, it only changes “perceived” brightness i.e it does software color correction. If we want to save battery, we will need to reduce the backlight brightness. ...

Why Wireguard and How does it work? Simple (one config file on server and client and one command to run) Small codebase (5000 LOC) unlike OpenVPN and IPSEC Scheduled to be merged into Linux kernel soon Wireguard works by creating a tunnel to the server using a separate network interface on your machine. You do not need to worry about Proxy settings and all your applications will work without issue. Installation and Key Generation Do this on both server and client machines #Installation: For both machines sudo apt install wireguard wireguard-tools openresolv # openresolv is in case you are using systemd-resolved which is default for DNS resolution on Ubuntu # Generate private and public key pair : For both machines umask 077 wg genkey | tee privatekey | wg pubkey > publickey Configure Server Create file called /etc/wireguard/wg0.conf [Interface] PrivateKey = <Private Key> ListenPort = <UDP Port to listen on, by default it is 51820> Address = 192.168.2.1/24, fd86:ea04:1115::1/64 ListenPort = 51820 PostUp = iptables -A FORWARD -i wg0 -j ACCEPT; iptables -t nat -A POSTROUTING -o eth0 -j MASQUERADE; ip6tables -A FORWARD -i wg0 -j ACCEPT; ip6tables -t nat -A POSTROUTING -o eth0 -j MASQUERADE PostDown = iptables -D FORWARD -i wg0 -j ACCEPT; iptables -t nat -D POSTROUTING -o eth0 -j MASQUERADE; ip6tables -D FORWARD -i wg0 -j ACCEPT; ip6tables -t nat -D POSTROUTING -o eth0 -j MASQUERADE SaveConfig = true [Peer] PublicKey = <Client Public Key> AllowedIPs = 192.168.2.2/32 Note that you have to paste the contents of the private key in the conf file, not the path. The PostUp sets up forwarding rules when the wiregaurd interface is started. PostDown deletes the rules when the interface is shutdown. Packet forwarding is essential if you want to use your server as a VPN to the general internet. Enable it on your server by adding the following to /etc/sysctl.conf and run sudo sysctl -p to reload the configuration changes. ...

Today, I was curious to find out how many programs ending with cat I had on my system. I remember using cat, zcat and wanted to know if there are similar programs. Pressing TAB after cat only gives programs which start with cat. 1. apropos In my last post, I had mentioned that apropos is a way to search for what programs are available on your system. The search string for apropos can be any regex. So, apropos 'cat$' should solve the problem. cat$ means all words ending with ‘cat’ The output has STAILQ_CONCAT (3) - implementations of singly-linked lists, singly-linked tail queues, lists and tail queues OPENSSL_strlcat (3ssl) - Memory allocation functions .. bzcat (1) - decompresses files to stdout cat (1) - concatenate files and print on the standard output fc-cat (1) - read font information cache files gencat (1) - Generate message catalog Clearly, the top 2 do not look like programs. Why is apropos then returning them? Let’s have a look at the apropos manual ...

The command-line is often a daunting place for beginners. With nothing but a blinking cursor and an unfriendly dark screen staring back at you, despair sets in. Here are a few things to do if you are stuck: 1. How to find programs for specific tasks Many times, you want to do a particular task but you do not remember the name of the program which does that. Many shell utilities are not easy to remember at first with their cryptic 2 letter names. Worry not, there is a command called apropos to help you out! For example, you want to show a file with line numbers. You know that there is a program but have forgotten its name. You can just type apropos -a line number to get a list of programs. We use -a flag so that apropos will only return those programs which have the words line AND number in their description. On my computer, it gives the following output: ...

Many times, we have shell scripts which perform some important stuff like inserting into database, mailing reports, etc which we want to run exactly one instance of. Enter locks! A simple solution is to create a “lock file” and check if the file exists when the script starts. If the file is already created, it means another instance of that program is running, so we can fail with message “Try again later!”. Once the script completes running, it will clean-up and delete the lock file. LOCK_FILE=a.lock if [ -f "$LOCK_FILE" ]; then # Lock file already exists, exit the script echo "An instance of this script is already running" exit 1 fi # Create the lock file echo "Locked" > "$LOCK_FILE" # Do the normal stuff # clean-up before exit rm "$LOCK_FILE" This looks promising but there are issues with this approach. What happens if the script does not end correctly i.e it exits because of some failure before it reaches the clean-up part of the code? Or if it gets forcibly terminated with Ctrl+C or kill command? In both these cases, the created lock file will not be deleted. So next time you run the script, you will always get an error and will have to manually delete the file. ...

When your Linux machine runs out of memory, Out of Memory (OOM) killer is called by kernel to free some memory. It is often encountered on servers which have a number of memory intensive processes running. In this post, we dig a little deeper into when does OOM killer get called, how it decides which process to kill and if we can prevent it from killing important processes like databases. How does OOM Killer choose which process to kill? The Linux kernel gives a score to each running process called oom_score which shows how likely it is to be terminated in case of low available memory. The score is proportional to the amount of memory used by the process. The score is 10 x percent of memory used by process. So the maximum score is 100% x 10 = 1000. In addition, if a process is running as a privileged user, it gets a slightly lower oom_score as compared to same memory usage by a normal user process. In earlier versions of Linux ( v2.6.32 kernel), there was a more elaborate heuristic which calculated this score. ...

Many times, smoothly running processes stop working mysteriously. You open the logs and see what is happening, only to find that even the logs have stopped updating. But the process itself is running. You SSH to the server and type cd TAB. Bash weeps “Unable to create temporary file”. The machine is out of disk space… Here is a checklist to make disk space debugging easier, using standard Linux utilities so you can get started without having to install anything extra: df -h command gives you an overview in a readable format about the number of disks mounted and their total and available capacities. To get an idea of which folders/directories are eating up the maximum space, try out du -ch / | sort -h | tail -n 30. This gives you the 30 most space consuming directories. If you already know which directories generate maximum disk output e.g logs and temp files, you can replace the ‘/’ with your directory (DIR) and run the command as du -ch DIR | sort -h | tail -n 30 Now that we have identified the directories with maximum space consumed, we may need to delete some files and get our process going again. The rm command is your friend here. You can delete old logs and temporary files to free up space. Many times, the culprit is a single large file which is already in use by a program e.g catalina.out by Apache Tomcat. If you want to free up space without shutting down the process, the truncate command will help you out. Example: truncate -s0 BIG_LOG.log. This will truncate the file to 0 bytes and still allow the other process to use it without issues (standard Unix permissions apply) Sometimes, you delete files and still, the space does not seem to be recovered. This can be because some process is still holding on to the file descriptor of the deleted file. Once these processes are stopped, the space will be recovered. The lsof command will help you out here. It stands for list open files. You can find out which processes are using deleted files as follows: lsof | grep deleted | grep OLD_FILENAME. The lsof command gives you the process name and the process id so you can run kill on the process. If you do not know the name of the deleted file, you can still run lsof | grep deleted and see the output to check for any familiar file / process. Finally, keep in mind that disk space is one of the metrics you should monitor on your server. This checklist must be used in a pinch. If you find yourself constantly having disk space issues, the solution is to set up periodic deletion/rotation of old log files, alerts when the disk space reaches a particular threshold or to increase the disk size if your processes require a lot of disk space e.g Kafka, MySQL and other databases. ...