Linux / OS Fundamentals

Linux / OS Fundamentals en posts linux os devops A practical guide to processes, CPU, memory, disk, networking, permissions, SSH, package management, systemd, logs, cron, and shell scripting.

Overview

Linux is the foundation for understanding server operations and deployment environments. When incidents occur, you always end up checking processes, CPU, memory, disk, networking, logs, and service status.

This post organizes the topics you’ll frequently revisit in Linux / OS. Beginners can learn the concepts, while experienced engineers can quickly reference commands and decision criteria.

1. Processes vs Threads

Details

Concepts

A process is a unit of a running program. The OS allocates independent memory space (code, data, heap, stack) to each process. Processes don’t share memory by default, so one crashing doesn’t directly affect others.

A thread is an execution flow within a process. Threads in the same process share heap memory. This makes inter-thread communication fast, but one thread accessing bad memory can bring down the entire process.

How to check

# List all processes sorted by CPU/memory
# -e: all processes, -o: specify output columns
ps -eo pid,ppid,user,cmd,%cpu,%mem --sort=-%cpu | head -20

# Real-time monitoring
top
htop  # Press H in htop to see threads

# Check thread count for a specific process
# nlwp: Number of Light Weight Processes (thread count)
ps -o nlwp,pid,cmd -p <PID>

Good to know

Multi-process architecture (nginx workers, PHP-FPM): one dying doesn’t affect other requests
Multi-threaded architecture (Java, Node.js worker threads): memory-efficient but one issue can spread to the entire process
Zombie processes (Z state) piling up means the parent isn’t calling wait() properly

2. Investigating High CPU Usage

Details

Step 1: Identify which process is using CPU

# -b: batch mode (non-interactive), -n1: run once only
top -bn1 | head -20
ps -eo pid,ppid,user,cmd,%cpu,%mem --sort=-%cpu | head -10

Step 2: Narrow down why it’s high

# Check what system calls the process is making
# -c: summary statistics, -p: specify PID
strace -cp <PID>

# For Java, take a thread dump
kill -3 <PID>
jstack <PID>

# Check how many cores are being used (-P ALL: all cores, 1 5: every 1s, 5 times)
mpstat -P ALL 1 5

Step 3: Classify the cause

Cause	How to verify
Recent deployment	Compare deploy time with CPU spike time
Traffic surge	Load balancer metrics, access log count
Infinite loop	strace shows repeated system calls
Batch job	Check if cron execution time overlaps
External dependency delay	Check if I/O wait is high (`wa` in top)

Key point: Rather than just noting CPU is high, narrow down which process started consuming more and why.

3. Memory Usage and OOM

Details

Check overall memory status

free -h  # -h: human-readable (shows in GB, MB units)

Example output:

              total        used        free      shared  buff/cache   available
Mem:           7.8G        3.2G        512M        128M        4.1G        4.2G

used: Actually in-use memory
buff/cache: Memory used as disk cache (can be freed if needed)
available: Memory allocatable to new processes (free + reclaimable cache)

Low free is fine as long as available is sufficient.

Per-process memory usage

ps -eo pid,ppid,user,cmd,%mem,rss --sort=-%mem | head -10
# rss: Resident Set Size (actual physical memory usage in KB)

OOM (Out Of Memory)

When memory is truly exhausted, the Linux kernel’s OOM Killer forcibly terminates memory-hungry processes.

# Check if OOM occurred
dmesg | grep -i "oom"
journalctl -k | grep -i "oom"

# Which process was killed
dmesg | grep "Killed process"

How to respond

In container environments, exceeding memory limit causes container restart (exit code 137)
For JVM apps, set -Xmx to 70-80% of container limit
If memory leak is suspected, monitor RSS trends over time

4. Disk Full Incident Response

Details

Check status

# Filesystem usage (-h: human-readable)
df -h

# Find large directories (-s: summary, -h: human-readable)
du -sh /* 2>/dev/null | sort -rh | head -10

# Narrow down further
du -sh /var/log/* | sort -rh | head -10

Common causes

Cause	Location	Solution
Application logs	`/var/log/`, app log paths	logrotate config
Old deployment files	`/opt/`, `/home/deploy/`	Cleanup scripts
Docker images/volumes	`/var/lib/docker/`	`docker system prune`
Temp files	`/tmp/`	Periodic cleanup
Deleted but open files	—	Check with `lsof +L1`

Deleted file but space not freed

If a file is deleted while a process still has it open, disk space won’t be reclaimed.

# Find deleted but still-open files
lsof +L1

# Restart the process to free space

Prevention

Configure logrotate for automatic log file management
Set monitoring alerts at 80% disk usage
In Docker environments, periodically run docker system prune -f

5. Network Status Checks

Details

A significant portion of server incidents are network problems. If a process is alive but not responding, check networking.

Port checks — is the service listening?

# List open ports
ss -tlnp
# t: TCP, l: LISTEN, n: numeric display, p: process info

# Check specific port
ss -tlnp | grep :8080

# Older systems
netstat -tlnp

Connection status

# Count current TCP connections by state
ss -tan | awk '{print $1}' | sort | uniq -c | sort -rn

# Too many TIME_WAIT: suspect connection pool issues
# CLOSE_WAIT piling up: application not closing connections properly

External connectivity tests

# Basic connectivity
ping -c 4 google.com

# Test specific port connectivity
curl -v telnet://db-server:3306
nc -zv db-server 3306

# DNS check
nslookup example.com
dig example.com

# Route tracing (find where it's blocked)
traceroute example.com
mtr example.com  # More detailed route tracing

Firewall checks

# Check iptables rules
iptables -L -n

# In cloud environments, also check security groups/firewall rules
# AWS: Security Group, NACL
# GCP: Firewall Rules

Troubleshooting flow

Is the service port in LISTEN state? (ss -tlnp)
Can you connect locally? (curl localhost:8080)
Can you connect externally? (curl server-ip:8080)
If not, check firewall, security groups, DNS in that order

6. File Permissions and Ownership

Details

Reading permissions

ls -l app.sh
# -rwxr-xr-- 1 deploy deploy 1024 Jul 7 09:00 app.sh

-rwxr-xr--
│├─┤├─┤├─┤
│ │   │  └── Others: r-- (read only)
│ │   └───── Group: r-x (read + execute)
│ └───────── Owner: rwx (read + write + execute)
└──────────── File type (-: regular file, d: directory, l: symlink)

Numeric notation

r = 4, w = 2, x = 1

755 = rwxr-xr-x (owner: all, others: read+execute)
644 = rw-r--r-- (owner: read+write, others: read only)
600 = rw------- (owner only: read+write)

Changing permissions

# Change permissions
chmod 755 app.sh
chmod +x deploy.sh  # Add execute permission

# Change ownership
chown deploy:deploy app.sh

# Recursive change for all subdirectories
chown -R deploy:deploy /opt/app/
chmod -R 755 /opt/app/bin/

Common failures in production

Symptom	Cause	Fix
`Permission denied`	Missing execute permission	`chmod +x`
Deploy script fails	Owner mismatch	`chown deploy:deploy`
Can’t write log file	Directory lacks write permission	Add `w` to directory
SSH key rejected	Key file permissions too open	`chmod 600 ~/.ssh/id_rsa`
Cron job won’t run	Script missing execute permission	`chmod +x`

7. Environment Variables and PATH

Details

Concept

Environment variables are key-value settings referenced when a process runs. Applications often read DB addresses, API keys, and runtime modes from environment variables.

Checking

# All environment variables
env
printenv

# Specific variables
echo $PATH
echo $HOME
echo $LANG

PATH

PATH is the list of directories searched for binaries when you run a command.

echo $PATH
# /usr/local/bin:/usr/bin:/bin:/usr/sbin:/sbin

# Where a command is located
which python3
type python3

Setting variables

# Current session only
export DB_HOST=localhost
export PATH=$PATH:/opt/app/bin

# Persistent (loaded on login)
echo 'export DB_HOST=localhost' >> ~/.bashrc
source ~/.bashrc

# System-wide
sudo vi /etc/environment

Common issues in production

Symptom	Cause	Fix
`command not found`	Binary path not in PATH	Add to PATH
Env var missing after deploy	systemd services don’t use user shell env	Use `Environment=` or `EnvironmentFile=` in service file
Won’t run in cron	cron runs with minimal environment	Export directly in script or use full paths
Variable exists via SSH but not in service	Login shell vs non-login shell difference	Specify directly in systemd

Setting environment variables in systemd services

[Service]
Environment=DB_HOST=localhost
Environment=DB_PORT=5432

# Or separate into a file
EnvironmentFile=/etc/app/env

8. User and Group Management

Details

Why it matters

Applications on servers should run as dedicated users, not root. Separate deploy users from app execution users for security.

User management

# Create user (with home directory)
useradd -m -s /bin/bash deploy

# Set password
passwd deploy

# Check user info
id deploy
# uid=1001(deploy) gid=1001(deploy) groups=1001(deploy)

# Delete user
userdel -r deploy  # -r: also delete home directory

Group management

# Create group
groupadd devops

# Add user to group
usermod -aG devops deploy
# -a: append (keep existing groups), -G: supplementary group

# Check groups
groups deploy

sudo privileges

# Add to sudo group (Ubuntu)
usermod -aG sudo deploy

# Or edit sudoers directly
visudo
# deploy ALL=(ALL) NOPASSWD: ALL

Example setup

root        — System administration (direct login disabled)
deploy      — Deployment tasks (sudo capable, SSH key access)
appuser     — App execution only (no sudo, login disabled)

# Create a no-login app execution user
useradd -r -s /usr/sbin/nologin appuser

9. Package Management

Details

How to install and update software on servers. Commands differ by distribution.

Debian/Ubuntu (apt)

# Update package list
apt update

# Install package
apt install -y nginx

# Remove package
apt remove nginx
apt purge nginx  # Also removes config files

# Check installed packages
dpkg -l | grep nginx

# Upgrade all packages
apt upgrade -y

RHEL/CentOS/Amazon Linux (yum/dnf)

# Install package
yum install -y nginx     # CentOS 7
dnf install -y nginx     # CentOS 8+, Amazon Linux 2023

# Remove package
yum remove nginx

# Check installed packages
rpm -qa | grep nginx

# Update all
yum update -y

Production considerations

Don’t blindly run apt upgrade or yum update on production servers — dependencies can break
Pin specific versions with apt-mark hold <package> or yum versionlock
Automatic security updates: unattended-upgrades (Ubuntu) or dnf-automatic (RHEL)
In Docker, run apt update && apt install in a single layer during image build

10. SSH Access and Key Management

Details

SSH key generation

# Generate Ed25519 key (currently recommended)
ssh-keygen -t ed25519 -C "[email protected]"

# Default paths: ~/.ssh/id_ed25519 (private), ~/.ssh/id_ed25519.pub (public)

Register public key on server

# Method 1: ssh-copy-id
ssh-copy-id -i ~/.ssh/id_ed25519.pub user@server

# Method 2: Manual copy
cat ~/.ssh/id_ed25519.pub >> ~/.ssh/authorized_keys
chmod 600 ~/.ssh/authorized_keys

SSH connection

# Basic connection
ssh [email protected]

# Specify port
ssh -p 2222 deploy@server

# Specify key
ssh -i ~/.ssh/my-key deploy@server

SSH config for convenience

# ~/.ssh/config
Host prod-web
    HostName 10.0.1.50
    User deploy
    Port 22
    IdentityFile ~/.ssh/id_ed25519

Host staging
    HostName 10.0.2.50
    User deploy
    IdentityFile ~/.ssh/id_ed25519

# Now connect simply with
ssh prod-web

Security settings (/etc/ssh/sshd_config)

# Disable root login
PermitRootLogin no

# Disable password auth (key auth only)
PasswordAuthentication no

# Allow specific users only
AllowUsers deploy

# Restart after config changes
systemctl restart sshd

File permissions (SSH rejects connections if wrong)

~/.ssh/              → 700
~/.ssh/authorized_keys → 600
~/.ssh/id_ed25519    → 600
~/.ssh/id_ed25519.pub → 644

File transfer with SCP/SFTP

# Copy file
scp app.tar.gz deploy@server:/opt/app/

# Copy directory (-r: recursive)
scp -r ./dist deploy@server:/opt/app/

# rsync (-a: archive mode, -v: verbose, -z: compressed transfer)
rsync -avz ./dist/ deploy@server:/opt/app/dist/

11. systemd Service Management

Details

Basic commands

# Check service status
systemctl status nginx

# Start / stop / restart
systemctl start nginx
systemctl stop nginx
systemctl restart nginx

# Reload config only (no downtime)
systemctl reload nginx

# Enable auto-start on boot
systemctl enable nginx

# Disable auto-start
systemctl disable nginx

Service file structure (/etc/systemd/system/myapp.service)

[Unit]
Description=My Application
After=network.target

[Service]
Type=simple
User=appuser
Group=appuser
WorkingDirectory=/opt/app
ExecStart=/opt/app/bin/server
Restart=on-failure
RestartSec=5
Environment=NODE_ENV=production
EnvironmentFile=/etc/app/env

# Log output
StandardOutput=journal
StandardError=journal

[Install]
WantedBy=multi-user.target

# After modifying service file
systemctl daemon-reload
systemctl restart myapp

Checking logs

# Service logs
journalctl -u nginx -n 100
journalctl -u nginx -f          # Real-time
journalctl -u nginx --since "1 hour ago"
journalctl -u nginx --since "2026-07-07 09:00" --until "2026-07-07 10:00"

Production checklist

Restart=on-failure: Auto-restart on abnormal exit
RestartSec=5: Interval between restarts (too short adds load during incidents)
Did you enable it: #1 reason services don’t come up after reboot
Verify whether logs go to journal or to a file

12. Log Analysis Commands

Details

Frequently used commands

# Real-time log viewing
tail -f /var/log/app/app.log

# Last 100 lines
tail -n 100 app.log

# Filter errors only
grep "ERROR" app.log
grep -i "exception" app.log

# Filter by time range (depends on log format)
grep "2026-07-07 09:" app.log

# Multiple conditions
grep -E "ERROR|WARN" app.log

# Page through entire file
less app.log
# In less: /ERROR to search, n for next, q to quit

systemd journal

# Full system log
journalctl -xe

# Kernel log
journalctl -k

# Logs since boot
journalctl -b

# Specific time range
journalctl --since "2026-07-07 09:00" --until "2026-07-07 10:00"

Log analysis tips

Situation	Investigation order
Service down	`journalctl -u <service> -n 50` → check error messages
Intermittent errors	`grep -c "ERROR" app.log` → check frequency → narrow time window
Pinpointing incident time	Compare deploy time, traffic changes, external system response times
Disk full	`ls -lhS /var/log/` → check largest files first

Key point: Don’t just look at error messages — correlate with occurrence time, frequency, deploy timing, and traffic changes to narrow down the cause.

13. Cron and Batch Jobs

Details

Cron basics

# List current user's cron jobs
crontab -l

# Edit cron
crontab -e

# Check another user's cron (as root)
crontab -u deploy -l

Cron expression format

┌─── Minute (0-59)
│ ┌─── Hour (0-23)
│ │ ┌─── Day of month (1-31)
│ │ │ ┌─── Month (1-12)
│ │ │ │ ┌─── Day of week (0-7, both 0 and 7 = Sunday)
│ │ │ │ │
* * * * *  command

Examples

# Every day at 2 AM
0 2 * * * /home/deploy/backup.sh

# Every 5 minutes
*/5 * * * * /home/deploy/health-check.sh

# Every Monday at 9 AM
0 9 * * 1 /home/deploy/weekly-report.sh

# First day of every month at midnight
0 0 1 * * /home/deploy/monthly-cleanup.sh

Logging batch jobs

# Log both stdout and stderr
0 2 * * * /home/deploy/backup.sh >> /var/log/backup.log 2>&1

# Date-based log files
0 2 * * * /home/deploy/backup.sh >> /var/log/backup-$(date +\%Y\%m\%d).log 2>&1

Production considerations

Cron runs with minimal environment — use full paths since PATH is nearly empty
You must be able to distinguish between “ran but failed” vs “never ran at all”
Without logging, debugging is impossible
Watch for duplicate execution when same cron runs on multiple servers
Prevent duplicate execution with flock:

* * * * * flock -n /tmp/myjob.lock /home/deploy/job.sh

14. Shell Script Automation

Details

Basic template

#!/usr/bin/env bash
set -euo pipefail

# set -e: exit immediately on error
# set -u: error on undefined variables
# set -o pipefail: detect failures in pipeline middle

LOG_FILE="/var/log/deploy.log"

log() {
  echo "[$(date '+%Y-%m-%d %H:%M:%S')] $*" | tee -a "$LOG_FILE"
}

log "=== Deployment started ==="

# Work goes here
log "Done"

Conditionals

# Check file exists
if [ -f /opt/app/config.yml ]; then
  echo "Config file found"
fi

# Check directory exists
if [ ! -d /opt/app/logs ]; then
  mkdir -p /opt/app/logs
fi

# Check command success/failure
if systemctl is-active --quiet nginx; then
  echo "nginx is running"
else
  echo "nginx is stopped"
  systemctl start nginx
fi

Loops

# Run commands on multiple servers
SERVERS="web01 web02 web03"
for server in $SERVERS; do
  echo "=== $server ==="
  ssh deploy@"$server" "systemctl status app"
done

Example: Server health check script

#!/usr/bin/env bash
set -euo pipefail

echo "=== Server Health Check ==="
echo ""

echo "[Disk]"
df -h | grep -E "^/dev" | awk '{print $6, $5}'
echo ""

echo "[Memory]"
free -h | grep Mem | awk '{print "Used:", $3, "/", $2}'
echo ""

echo "[CPU Load]"
uptime | awk -F'load average:' '{print $2}'
echo ""

echo "[Key Services]"
for svc in nginx app-server redis; do
  if systemctl is-active --quiet "$svc" 2>/dev/null; then
    echo "  $svc: OK"
  else
    echo "  $svc: DOWN"
  fi
done

Script writing principles

Always start with set -euo pipefail
Log output so execution results are traceable
Add dry-run option before dangerous operations (rm, drop)
Use absolute paths (prevents PATH issues when run from cron)
Add status checks before and after critical operations

Summary

Linux / OS is the starting point for investigating operational issues. If you can check processes, CPU, memory, disk, networking, permissions, SSH, services, and logs in order, you can narrow down most server incidents faster.

Rather than memorizing each item, focus on remembering the flow — what sequence of commands to run when an incident occurs.