Linux: muhandis uchun first-principles qo'llanma

Bob 01Linux falsafasi va ekotizim

Linux — bu yadro (kernel). U bilan birga keladigan vositalar, paket menejer, init tizimi va sozlamalar — bu distributiv. Buyruqlar nima uchun shunday tuzilganini bilish, ularni eslab qolishdan ko'ra muhimroq.

Unix falsafasi

Bu to'rt tamoyil bugungi Linux haqida bilgan hamma narsangizni izohlaydi:

1. Bitta narsani yaxshi qil. Har bir vosita kichik va mas'uliyatli (grep faqat qidiradi, sort faqat tartiblaydi).
2. Vositalarni birlashtiring. stdout → stdin orqali pipeline yarating: ps aux | grep nginx | awk '{print $2}'.
3. Hammasi fayl. Disklar, tarmoq qurilmalari, jarayonlar, hatto kernel sozlamalari — barchasi fayl tizimida ko'rinadi (/dev, /proc, /sys).
4. Matn — universal interfeys. Konfiguratsiya, log, hatto IPC — odatda matn. Bu grep, sed, awk universal kuchini beradi.

Kernel va userspace

Kernel CPU rejimida (ring 0) ishlaydi: xotira, jarayon, qurilma, fayl tizimi, tarmoq stekini boshqaradi. Userspace ilovalar to'g'ridan-to'g'ri qurilmalarga tega olmaydi — ular kernel'dan system call orqali so'raydi: open(), read(), write(), fork(), execve(), mmap(), socket().

Bir buyruqning syscall'larini ko'rish uchun:

strace -c ls /tmp        # syscall statistikasi
strace -e openat ls      # faqat openat() syscalllari

Bu Linux'ni qora qutidan oq qutiga aylantiradigan asosiy fokus.

Distributiv nima?

Distro = kernel + GNU coreutils + libc (glibc/musl) + paket menejer + init tizimi + qaror qabul qilingan sozlamalar to'plami.

Konteyner image'lari ko'pincha Alpine (kichik, musl libc) yoki Debian-slim asosida. Production server'lar — odatda Ubuntu LTS yoki RHEL oilasi.

cat /etc/os-release
uname -a
hostnamectl

Bob 02Shell va terminal

Terminal — siz ko'radigan oyna; shell — buyruqlaringizni talqin qiluvchi dastur. Ularni ajratib bilish, muammolarni tezroq joylashtirishga yordam beradi.

TTY va PTY

TTY (teletypewriter) — fizik terminal qurilmasining nomi, hozir ham qoldi. PTY (pseudo-terminal) — terminal emulyatori (iTerm, GNOME Terminal, tmux) yaratadigan virtual juftlik. Sizning bash'ingiz /dev/pts/0'ga ulangan, tty buyrug'i buni ko'rsatadi.

Shell turlari

• sh — POSIX shell, minimal, skript yozish standarti.
• bash — Bourne Again Shell, Linux'da default. Skriptlaringiz ishlashi uchun universal nuqta.
• zsh — boy autocomplete, plugin (oh-my-zsh). macOS'da default.
• fish — eng do'stona, lekin POSIX-mos emas.

Production skriptlari uchun #!/usr/bin/env bash yoki yanada portativ #!/bin/sh yozing.

Buyruq qaerdan kelyapti?

Bash buyruq nomini ko'rganda, quyidagi tartibda qidiradi:

1. Alias (alias ll='ls -lah')
2. Function (siz yozgan shell funksiyasi)
3. Builtin (cd, echo, export, read, type)
4. Hash table (yaqinda topilgan yo'l)
5. $PATH kataloglari, chapdan o'ngga

type -a ls           # barcha aniqlanishlar
which ls             # faqat $PATH'dan birinchi
command -v ls        # POSIX-portativ variant

$PATH first principles

$PATH — ikki nuqta bilan ajratilgan kataloglar ro'yxati. Shell ularni chapdan o'ngga qidiradi.

echo $PATH | tr ':' '\n'
# /home/u/.local/bin → /usr/local/bin → /usr/bin → /bin

Yangi katalogni doimiy qo'shish: ~/.bashrc yoki ~/.profile'ga export PATH="$HOME/bin:$PATH". Boshiga qo'ying — shunda sizning versiyangiz ustun bo'ladi.

Quoting (juda muhim)

Eslab qoling: bo'sh joy o'rinda qolishi uchun har doim o'zgaruvchini qo'shtirnoqlang: "$file", hech qachon $file.

Job control

./long-task           # oldinda ishga tushirish
./long-task &         # orqada
Ctrl+Z                  # to'xtatib turish
bg                    # orqada davom ettirish
fg %1                 # oldinga qaytarish
jobs                  # joriy ishlar
nohup ./srv &         # shell yopilsa ham yashasin
disown %1             # shell uchun "ko'rinmas" qilish

mkdir -p ~/bin
printf '#!/usr/bin/env bash\necho "salom $USER"\n' > ~/bin/greet
chmod +x ~/bin/greet
export PATH="$HOME/bin:$PATH"
greet

Bob 03Fayl tizimi: FHS, inode, virtual fs

Linux'da "fayl" — bu nom, "ma'lumot" — bu inode. Bu farqni tushunish symlink, hard link, "Disk to'la" muammolari va konteyner volume'larini tushunish uchun zarur.

Filesystem Hierarchy Standard (FHS)

Virtual fayl tizimlari

Bu kataloglar diskda yo'q — kernel ularni xotirada hosil qiladi.

• /proc — har bir jarayon uchun katalog (/proc/<pid>) + kernel ma'lumotlari (/proc/cpuinfo, /proc/meminfo, /proc/mounts).
• /sys — sysfs, qurilmalar va kernel obyektlari (/sys/class/net/eth0/).
• /dev — qurilma fayllari (/dev/sda, /dev/null, /dev/urandom, /dev/zero).
• /run — runtime tmpfs.

cat /proc/cpuinfo | grep "model name" | head -1
cat /proc/meminfo | head -5
cat /proc/$$/status | head     # shell jarayoningizning holati

Inode: fayl nomi vs ma'lumot

Inode — diskda saqlangan struktura: fayl o'lchami, egasi, ruxsatlar, vaqt belgilari, va ma'lumot bloklariga ko'rsatkichlar. Fayl nomi — bu katalogdagi yozuv: (nom, inode_raqami). Bir necha nom bitta inode'ga ko'rsatishi mumkin — bu hard link.

ls -li file.txt        # inode raqami birinchi ustunda
stat file.txt          # to'liq metadata
df -i /                # inode usage (kichik fayllar ko'p bo'lsa, tugashi mumkin)

Hard link va symlink

Mount: filesystem'lar daraxti

Linux'da bitta birlashgan daraxt: turli disklar/partition'lar daraxtning turli nuqtalariga "ulanadi" (mount).

mount                          # barcha mount'lar
findmnt                        # daraxt ko'rinishi
mount /dev/sdb1 /mnt/data
umount /mnt/data
cat /etc/fstab                 # boot vaqtida nima mount bo'ladi

echo "asl" > a.txt
ln a.txt hard.txt
ln -s a.txt soft.txt
ls -li *.txt
rm a.txt
cat hard.txt    # hali ham ishlaydi
cat soft.txt    # xato: dangling symlink

Bob 04Foydalanuvchilar, ruxsatlar, capabilities

Ruxsat tizimi — Linux xavfsizligining birinchi qatlami. chmod 755 yodlash o'rniga, raqamlar nimani anglatishini va nega SUID root yoki capabilities'dan xavfli ekanligini bilish kerak.

Foydalanuvchi va guruh

Har bir foydalanuvchi UID bilan, har bir guruh GID bilan ifodalanadi. Nomlar — faqat insonlar uchun jadval. UID 0 — root.

id                          # siz: uid, gid, guruhlar
cat /etc/passwd | head -3   # nom:x:UID:GID:gecos:home:shell
sudo cat /etc/shadow | head # parol hash + policy
cat /etc/group | head -3

x — parol shadow'da. ! yoki * — hisob qulflangan.

Foydalanuvchi yaratish

sudo useradd -m -s /bin/bash -G sudo akmal
sudo passwd akmal
sudo usermod -aG docker akmal      # -a MUHIM: aks holda guruhlar o'chiriladi
sudo userdel -r akmal              # -r: home'ni ham o'chirish

Ruxsatlar: rwx

Har bir fayl uchun uchta sub'yekt: egasi, guruh, boshqalar. Har biri uchun r(4), w(2), x(1).

-rwxr-xr-- 1 ali dev 1234 May  6 12:00 script.sh
 │└┬┘└┬┘└┬┘
 │ │  │  └─ boshqalar: r--   = 4
 │ │  └──── guruh:    r-x   = 5
 │ └─────── egasi:    rwx   = 7
 └────────── tur: - fayl, d katalog, l symlink

chmod 750 file
chmod u+x,g-w,o= file       # symbolic
chmod -R 644 dir/           # rekursiv
chown ali:dev file
chgrp dev file

Maxsus bitlar: SUID, SGID, sticky

ls -l /usr/bin/passwd       # -rwsr-xr-x  → SUID
ls -ld /tmp                 # drwxrwxrwt  → sticky (oxirgi t)
chmod u+s file

umask: standart ruxsat

Yangi fayl maksimal 666, katalog 777'dan boshlanadi. umask bitlarni o'chiradi.

umask                  # 0022 odatda
# Fayl: 666 & ~022 = 644 (rw-r--r--)
# Katalog: 777 & ~022 = 755 (rwxr-xr-x)

ACL: kengaytirilgan ruxsat

rwx 3 sub'yektga cheklangan. Ko'proq nazorat kerak bo'lsa — POSIX ACL.

setfacl -m u:akmal:rw file
getfacl file
setfacl -d -m g:dev:rwx dir/   # default ACL: meros

sudo va /etc/sudoers

Hech qachon /etc/sudoers'ni to'g'ridan-to'g'ri tahrirlamang — visudo sintaksisni tekshiradi.

sudo visudo
# misol qatorlar:
ali ALL=(ALL:ALL) ALL
%docker ALL=(ALL) NOPASSWD: /usr/bin/docker
Defaults timestamp_timeout=15

Capabilities: root'ni bo'lish

Tarixan, "root yoki root emas" edi. Endi root kuchi 40+ capability'ga bo'lingan: CAP_NET_BIND_SERVICE (1024'dan past port'ni ochish), CAP_NET_ADMIN (tarmoqni sozlash), CAP_SYS_ADMIN (eng katta) va h.k.

getcap /usr/bin/ping       # cap_net_raw=ep
sudo setcap 'cap_net_bind_service=+ep' /opt/myapp
capsh --print              # joriy jarayon capabilities

sudo groupadd devs
sudo mkdir -p /srv/shared
sudo chgrp devs /srv/shared
sudo chmod 2775 /srv/shared
sudo -u akmal touch /srv/shared/test.txt
ls -l /srv/shared/test.txt   # guruh: devs

Bob 05Jarayonlar, signal, cgroup

Jarayon — bu kernel uchun ish birligi. fork(), exec(), signallar va cgrouplarni bilish — tizim muhandisligining yadrosi.

Jarayon qanday tug'iladi

Yangi jarayon — har doim ikki bosqich:

1. fork() — joriy jarayonning ko'chirmasini yaratadi (parent va child).
2. execve() — child o'zining xotira xaritasini yangi binary bilan almashtiradi.

Shu sababli har bir jarayon parent'ga ega (PPID). PPID 1 — systemd (yetim qolgan jarayonlarni qabul qiladi).

Jarayon holatlari

Yuqori D-state = disk yoki NFS bo'g'ilgan. Yuqori load average, lekin past CPU — odatda shu.

ps, top, htop, btop

ps aux | head            # BSD uslub: USER PID %CPU %MEM ...
ps -ef | head            # UNIX uslub
ps -eo pid,ppid,user,cmd,%cpu,%mem --sort=-%cpu | head
pstree -p $$             # shell ishtirokidagi daraxt
top                      # real-time, ASCII
htop                     # rangli, scrollable
btop                     # yanada zamonaviy

Signal: jarayonlarga xabar yuborish

kill 1234            # SIGTERM
kill -9 1234         # SIGKILL — oxirgi chora
kill -HUP 1234       # konfig reload
killall nginx
pkill -f "python.*worker"
pgrep -fl nginx

Niceness va prioritet

nice qiymati −20 (eng yuqori) dan +19 (eng past) gacha. Default 0.

nice -n 10 ./batch-job
renice -n 5 -p 1234
ionice -c 3 ./io-heavy        # I/O class: idle

cgroups: resurslar uchun nazorat

cgroup v2 — har bir jarayonni iyerarxik guruhlarga joylab, CPU/RAM/IO/PID limitlarini qo'yish vositasi. Konteynerlar va systemd shu ustida quriladi.

cat /sys/fs/cgroup/cgroup.controllers   # mavjud controller'lar
systemd-cgls                            # cgroup daraxti
systemd-cgtop                           # real-time
systemctl set-property nginx.service MemoryMax=512M CPUQuota=50%

/proc/<pid> chuqurligi

ls /proc/$$/
cat /proc/$$/status         # Name, State, Uid, VmRSS, ...
cat /proc/$$/cmdline | tr '\0' ' '
ls -l /proc/$$/fd           # ochilgan file descriptor'lar
cat /proc/$$/maps | head    # xotira xaritasi
cat /proc/$$/limits         # ulimit

(sleep 0.1 & exec sleep 30) &
sleep 1
ps -eo pid,ppid,stat,cmd | grep -E 'Z|defunct'

Bob 06Matn oqimlari, pipe, regex

Linux'da hamma narsa matn oqimi: log, konfig, buyruq chiqishi. grep, sed, awk sizning eng kuchli kombinatsiyangiz.

Uchta file descriptor

cmd > out.log              # stdout faylga
cmd >> out.log             # qo'shish
cmd 2> err.log             # stderr faylga
cmd > out.log 2>&1         # har ikkalasi bir joyga
cmd &> out.log             # bash qisqartmasi
cmd > /dev/null 2>&1       # jim
cmd < input.txt            # stdin fayldan
cmd <<< "matn"             # herestring
cmd <<EOF
ko'p qatorli matn
EOF

Pipe va process substitution

journalctl -u nginx | grep error | tail -20
diff <(sort a.txt) <(sort b.txt)        # process substitution
tee --help | head | tee saqlangan.txt   # bir vaqtda ekran + fayl

grep: regex bilan qidirish

grep "ERROR" app.log
grep -i error app.log              # case-insensitive
grep -v debug app.log              # inkor
grep -r "TODO" src/                # rekursiv
grep -E "^[0-9]{3} (404|500)" access.log     # ERE
grep -P '\d{4}-\d{2}-\d{2}' app.log          # PCRE
grep -A 3 -B 1 "error" log.txt     # atrofdagi qatorlar
grep -c "ERROR" *.log              # sanog'i
grep -l "TODO" *.py                # faqat fayl nomi

sed: oqim muharriri

sed 's/old/new/' file              # har qatordagi birinchi
sed 's/old/new/g' file             # har qatordagi hammasi
sed -i 's/old/new/g' file          # in-place
sed -i.bak 's/.../.../' file       # avval backup
sed -n '10,20p' file               # 10-20 qatorlar
sed '/^#/d' file                   # # bilan boshlanganlarni o'chirish
sed -E 's/([0-9]+)/\1<\/num>/g' file

awk: ustunli matn uchun

awk '{print $1}' file              # 1-ustun
awk -F: '{print $1}' /etc/passwd   # : separator
ps aux | awk '$3 > 50 {print $2,$11}'      # CPU > 50%
awk 'NR==5' file                   # 5-qator
awk 'END{print NR}' file           # jami qator
awk '{sum+=$1} END{print sum}' nums.txt
awk -F, 'BEGIN{OFS="|"} {print $1,$3}' data.csv

Boshqa muhim filtr'lar

cut -d: -f1,3 /etc/passwd
sort -k2 -n file
sort | uniq -c | sort -rn          # top-N pattern
wc -l file                         # qator soni
tr 'a-z' 'A-Z' < file
tr -d '\r' < win.txt > unix.txt    # CRLF tozalash
paste a.txt b.txt
join -1 1 -2 1 a.txt b.txt
comm -12 <(sort a) <(sort b)       # umumiy qatorlar

Eng foydali pipeline'lar

# Eng ko'p RAM yeydigan 5 jarayon:
ps aux --sort=-%mem | head -6

# Access log'dan eng tez-tez IP:
awk '{print $1}' access.log | sort | uniq -c | sort -rn | head

# Aniq port'da kim eshityapti:
ss -tulpn | awk '$5 ~ /:443$/'

# Kataloglardagi eng katta 10 ta fayl:
du -ah /var | sort -rh | head

awk '$9 ~ /^4/ {print $7}' access.log | sort | uniq -c | sort -rn | head -5

Bob 07Paket boshqaruvi

Distro tanlovi — bu birinchi navbatda paket menejer tanlovi. Ikki dunyoni biling: Debian/Ubuntu (apt) va RHEL oilasi (dnf).

Debian/Ubuntu: apt va dpkg

apt — yuqori darajadagi: repolarni biladi, dependency'larni hal qiladi. dpkg — past darajada: bitta .deb faylini o'rnatadi.

sudo apt update                    # indeksni yangilash
sudo apt upgrade                   # o'rnatilganlarni yangilash
sudo apt full-upgrade              # dependency o'zgarsa ham
sudo apt install nginx
sudo apt remove nginx              # konfig qoldi
sudo apt purge nginx               # konfig ham o'chdi
sudo apt autoremove                # keraksiz dependency'lar
apt list --installed | grep nginx
apt search redis
apt show redis-server
apt-mark hold nginx                # versiyani qotirib qo'yish

# Past daraja:
sudo dpkg -i package.deb           # dependency yo'q bo'lsa, xato
sudo apt install -f                # dependency'larni tuzatish
dpkg -l | grep nginx
dpkg -L nginx                      # paketning fayllari
dpkg -S /etc/nginx/nginx.conf      # bu fayl qaysi paketdan?

Repolar va GPG

Repolar /etc/apt/sources.list va /etc/apt/sources.list.d/'da. Har bir repo GPG kalit bilan imzolanadi — soxta paket'lardan saqlaydi.

# Modern usul (Ubuntu 22.04+):
sudo install -d /etc/apt/keyrings
curl -fsSL https://download.docker.com/linux/ubuntu/gpg | \
  sudo gpg --dearmor -o /etc/apt/keyrings/docker.gpg
echo "deb [arch=amd64 signed-by=/etc/apt/keyrings/docker.gpg] \
  https://download.docker.com/linux/ubuntu jammy stable" | \
  sudo tee /etc/apt/sources.list.d/docker.list
sudo apt update

RHEL oilasi: dnf va rpm

sudo dnf check-update
sudo dnf upgrade
sudo dnf install httpd
sudo dnf remove httpd
sudo dnf autoremove
dnf search nginx
dnf info nginx
dnf list installed | grep nginx
dnf history                        # tranzaksiya logi
sudo dnf history undo 42           # tranzaksiyani bekor qilish
dnf module list nodejs             # module stream'lar
sudo dnf module enable nodejs:20

# Past daraja:
sudo rpm -ivh package.rpm
rpm -qa | grep nginx
rpm -ql nginx                      # fayllar
rpm -qf /etc/nginx/nginx.conf      # bu fayl qaysi paketdan?
rpm -qi nginx                      # info

Universal: snap, flatpak, AppImage

• snap — Canonical, sandbox, har bir versiya alohida (auto-update).
• flatpak — desktop ilovalari uchun, Flathub repository.
• AppImage — bitta executable fayl, o'rnatish kerak emas.

Bob 08systemd

systemd — bu PID 1, ya'ni boot'dan keyin birinchi userspace jarayon. U parallel, dependency-aware, socket-activated servis menejer. Hozir hamma asosiy distro'larda standart.

Asosiy birliklar (units)

Joylashuv

systemctl: kundalik buyruqlar

systemctl status nginx
systemctl start nginx
systemctl stop nginx
systemctl restart nginx
systemctl reload nginx              # SIGHUP — qayta tushirmasdan
systemctl enable nginx              # boot'da yoqish
systemctl enable --now nginx        # yoq + hozir ishga tushir
systemctl disable nginx
systemctl mask nginx                # umuman ishga tushib bo'lmasin
systemctl is-active nginx
systemctl is-enabled nginx
systemctl list-units --type=service --state=running
systemctl list-unit-files --type=service
systemctl cat nginx                 # unit faylni ko'rsatadi
systemctl show nginx | grep Memory
systemctl daemon-reload             # unit fayl tahrirlangandan keyin

O'z servisingizni yozish

# /etc/systemd/system/myapp.service
[Unit]
Description=My Application
After=network-online.target
Wants=network-online.target

[Service]
Type=simple
User=myapp
Group=myapp
WorkingDirectory=/opt/myapp
ExecStart=/opt/myapp/bin/server --port 8080
Restart=on-failure
RestartSec=5s

# Xavfsizlik (sandboxing):
NoNewPrivileges=yes
PrivateTmp=yes
ProtectSystem=strict
ProtectHome=yes
ReadWritePaths=/var/lib/myapp

# Resurs cheklash:
MemoryMax=512M
CPUQuota=80%
TasksMax=200

[Install]
WantedBy=multi-user.target

sudo systemctl daemon-reload
sudo systemctl enable --now myapp

Type'lar

Drop-in: paket unit'ini buzmasdan o'zgartirish

sudo systemctl edit nginx
# /etc/systemd/system/nginx.service.d/override.conf yaratiladi
[Service]
LimitNOFILE=65536

journald va journalctl

journalctl                          # hammasi (page'lab)
journalctl -u nginx                 # bitta unit
journalctl -u nginx -f              # tail -f
journalctl --since "1 hour ago"
journalctl --since today -p err     # priority: err va undan yuqori
journalctl -k                       # kernel xabarlari (dmesg analogi)
journalctl -b                       # joriy boot
journalctl -b -1                    # oldingi boot
journalctl --disk-usage
journalctl --vacuum-time=7d         # 7 kundan eski'ni o'chirish

Targets (runlevel o'rnida)

systemctl get-default
sudo systemctl set-default multi-user.target
sudo systemctl isolate rescue.target

sudo tee /etc/systemd/system/boot-stamp.service <<'EOF'
[Unit]
Description=Boot timestamp
After=multi-user.target

[Service]
Type=oneshot
ExecStart=/bin/sh -c 'date >> /var/log/boot-stamp'
RemainAfterExit=yes

[Install]
WantedBy=multi-user.target
EOF
sudo systemctl daemon-reload
sudo systemctl enable --now boot-stamp

Bob 09Tarmoq (chuqur)

Tarmoq muhandisligi — Linux'ning kuchli tomoni. ifconfig va netstat eskirgan; ip va ss — hozirgi standart. Quyidan yuqoriga: link → IP → routing → DNS → SSH → firewall → tcpdump.

TCP/IP qisqacha

Interfeys va manzil

ip link                       # MAC, holat (UP/DOWN)
ip -br link                   # brief
ip addr                       # IP manzillar
ip -br -c addr                # brief + rang
ip addr add 10.0.0.5/24 dev eth0
ip link set eth0 up
ip link set eth0 mtu 9000     # jumbo frame

Routing

ip route                      # asosiy jadval
ip route get 8.8.8.8          # shu IP qaysi yo'l bilan ketadi
ip route add 192.168.50.0/24 via 10.0.0.1
ip route add default via 10.0.0.1
ip route show table all       # barcha jadvallar (policy routing)

Default gateway, metric, tarmoqdagi paketlarning yo'li shu yerdan ko'rinadi.

ARP / neighbor

ip neigh                      # MAC ↔ IP keshi
ip neigh flush all

Socket'lar va portlar — ss

ss -tulpn                     # TCP+UDP, listening, jarayon, raqamli
ss -t state established       # aktiv TCP
ss -t '( dport = :443 )'
ss -s                         # umumiy statistika
ss -ti                        # TCP ichki info: rtt, cwnd

DNS resolution path

Domen nomi qanday IP'ga aylanadi:

1. /etc/nsswitch.conf — hosts: qatori tartibni belgilaydi (files dns...)
2. /etc/hosts — qo'lda yozilgan static yozuvlar
3. systemd-resolved — kesh va stub resolver (/etc/resolv.conf → 127.0.0.53)
4. /etc/resolv.conf'dagi upstream nameserver'lar

resolvectl status              # systemd-resolved holati
cat /etc/resolv.conf
dig example.com                # asosiy
dig +short example.com
dig +trace example.com         # root'dan boshlab
dig MX example.com
dig @1.1.1.1 example.com       # aniq DNS server
host example.com
nslookup example.com

Bog'lanish testlari

ping -c 4 8.8.8.8              # bog'lanish + RTT
traceroute example.com         # har hop
mtr example.com                # traceroute + ping (real-time)
nc -vz example.com 443         # port ochilganmi?
nc -l 9000                     # oddiy listener
curl -I https://example.com    # faqat header'lar
curl -v https://example.com    # to'liq tafsilot
curl --resolve x.com:443:1.2.3.4 https://x.com   # DNS'ni override
curl -w "@-" -o /dev/null -s https://x.com <<'EOF'
DNS:    %{time_namelookup}\nTCP:    %{time_connect}\nTLS:    %{time_appconnect}\nTTFB:   %{time_starttransfer}\nTOTAL:  %{time_total}\n
EOF

tcpdump: paketlarni ko'rish

sudo tcpdump -i eth0 -n         # nom emas, raqam
sudo tcpdump -i any port 443
sudo tcpdump -i eth0 host 10.0.0.5 and port 80
sudo tcpdump -i eth0 'tcp[tcpflags] & (tcp-syn) != 0'   # SYN'lar
sudo tcpdump -i eth0 -w cap.pcap                  # faylga yozish
tcpdump -r cap.pcap -nn         # keyin o'qish

Firewall

# ufw (Ubuntu):
sudo ufw allow 22/tcp
sudo ufw allow from 10.0.0.0/8 to any port 5432
sudo ufw enable
sudo ufw status numbered

# firewalld (RHEL):
sudo firewall-cmd --add-service=https --permanent
sudo firewall-cmd --add-port=8080/tcp --permanent
sudo firewall-cmd --reload
sudo firewall-cmd --list-all

# nftables (xom):
sudo nft list ruleset
sudo nft add rule inet filter input tcp dport 22 accept

SSH (eng muhim — har kuni ishlatasiz)

# Kalit yaratish (ed25519 — eng yaxshi tanlov):
ssh-keygen -t ed25519 -C "akmal@laptop"
ssh-copy-id akmal@server
ssh akmal@server

# SSH agent — kalitlarni xotirada:
eval "$(ssh-agent -s)"
ssh-add ~/.ssh/id_ed25519
ssh-add -l                       # yuklangan kalitlar

# Port forwarding:
ssh -L 8080:localhost:80 server      # lokaldan server'ga
ssh -R 9000:localhost:3000 server    # remote'dan o'zingizga
ssh -D 1080 server                   # SOCKS proxy

# Bastion orqali (jump host):
ssh -J bastion.example.com akmal@private-server

~/.ssh/config — ish hayotini soddalashtiradi

Host bastion
    HostName bastion.example.com
    User devops
    Port 22
    IdentityFile ~/.ssh/id_ed25519

Host prod-*
    User akmal
    ProxyJump bastion
    ServerAliveInterval 60

Endi ssh prod-db1 yetadi.

ssh -J bastion.example.com -L 3307:db.internal:3306 jump-user@bastion
# Endi: mysql -h 127.0.0.1 -P 3307 -u app -p

Bob 10Disklar, fayl tizimlari, LVM

Disk = blok qurilmasi. Partition = bo'lim. Filesystem = format. LVM = moslashuvchan qatlam. Bu to'rtta tushunchani aralashtirmang.

Blok darajasini ko'rish

lsblk                       # daraxt: disk → partition → mount
lsblk -f                    # filesystem va UUID bilan
ls /dev/sd* /dev/nvme*
blkid                       # UUID, fs turi
fdisk -l
parted -l

Partition jadvali: MBR vs GPT

sudo parted /dev/sdb
(parted) mklabel gpt
(parted) mkpart primary ext4 1MiB 100%
(parted) print
(parted) quit

Filesystem yaratish va mount

sudo mkfs.ext4 -L data /dev/sdb1
sudo mkfs.xfs /dev/sdc1
sudo mkdir /mnt/data
sudo mount /dev/sdb1 /mnt/data
sudo umount /mnt/data

/etc/fstab — boot'da mount

# <UUID>  <mountpoint>  <fs>  <options>       <dump> <pass>
UUID=abcd-1234  /mnt/data   ext4  defaults,noatime  0  2

UUID= ishlating, /dev/sdb1 emas — qurilma nomlari yuklash o'rtasida o'zgarishi mumkin.

sudo mount -a       # fstab'ni sinab ko'rish
findmnt --verify    # tekshirish

LVM: moslashuvchan disklar

Uchta qatlam: PV (physical volume) → VG (volume group) → LV (logical volume).

sudo pvcreate /dev/sdb /dev/sdc
sudo vgcreate data /dev/sdb /dev/sdc
sudo lvcreate -L 50G -n logs data
sudo mkfs.ext4 /dev/data/logs
sudo mount /dev/data/logs /var/log/app

# Hajmni oshirish (online):
sudo lvextend -L +20G /dev/data/logs
sudo resize2fs /dev/data/logs        # ext4 uchun
sudo xfs_growfs /var/log/app         # xfs uchun

# Snapshot:
sudo lvcreate -L 5G -s -n logs-snap /dev/data/logs

# Holat:
pvs ; vgs ; lvs
pvdisplay ; vgdisplay ; lvdisplay

Swap

sudo dd if=/dev/zero of=/swapfile bs=1M count=2048 status=progress
sudo chmod 600 /swapfile
sudo mkswap /swapfile
sudo swapon /swapfile
swapon --show
free -h
# Doimiy uchun /etc/fstab:
# /swapfile none swap sw 0 0

# Swappiness sozlash (0-100):
sysctl vm.swappiness
echo 'vm.swappiness=10' | sudo tee /etc/sysctl.d/99-swap.conf

Disk holati va sog'lig'i

df -h                       # bo'sh joy
df -i                       # inode usage
du -sh /var/*               # har kataloglarning hajmi
du -ah /var | sort -rh | head     # top fayllar
ncdu /var                   # interaktiv (ko'rsatkichli)
sudo smartctl -a /dev/sda   # SMART (disk sog'lig'i)
iostat -xz 1

Bob 11Xotira va kernel ichi

"RAM band, lekin tizim normal" — bu odatiy. Page cache, swap, OOM killer va virtual xotirani tushunish — false alarm'lardan saqlaydi.

Virtual xotira

Har bir jarayon o'zining 64-bit address space'ini ko'radi. Kernel page table'lar orqali bu virtual sahifalarni fizik RAM'ga yoki swap'ga xaritalaydi. Demand paging: sahifa kerak bo'lganda yuklanadi.

/proc/meminfo'ni tushunish

cat /proc/meminfo | head -10
free -h

Page cache

Linux disk'dan o'qilgan fayllarni RAM'da saqlab qoladi. Bo'sh RAM'ni "behuda" deb ko'rmang — kernel uni cache uchun ishlatib, kelgusi o'qishlarni tezlashtiradi. Ilova RAM so'raganda, kernel kerak bo'lsa cache'ni darhol bo'shatadi.

sudo sync; echo 3 | sudo tee /proc/sys/vm/drop_caches    # cache'ni tozalash (test uchun)

Swap va swappiness

vm.swappiness 0-100: yuqori = anonymous sahifalarni tezroq swap'ga jo'natish. Default Ubuntu'da 60. Database server'lar uchun 1-10 ga tushiring.

OOM killer

RAM tugaganda, kernel ilovani o'ldiradi. Qaysini? oom_score eng yuqori bo'lganni.

dmesg | grep -i "killed process"
journalctl -k | grep -i oom
cat /proc/<pid>/oom_score
echo -1000 | sudo tee /proc/<pid>/oom_score_adj   # o'ldirilmaslik

systemd unit'da: OOMScoreAdjust=-500.

cgroup memory limits

systemctl set-property myapp.service MemoryMax=512M MemoryHigh=400M
# MemoryHigh: yumshoq cheklov, throttling
# MemoryMax: qattiq cheklov, OOM

vm sysctl'lari

sysctl vm.overcommit_memory      # 0 default, 1 har doim ruxsat, 2 strict
sysctl vm.dirty_ratio
sysctl vm.dirty_background_ratio
sysctl vm.min_free_kbytes

stress-ng --vm 2 --vm-bytes 90% --timeout 30s
journalctl -k --since "1 minute ago" | grep -i oom

Bob 12Loglar va observability

"Kuzatuvsiz tizim — qora quti." USE va RED metodlarini, journald, perf va eBPF asoslarini bilish — incident vaqtida qutqaradi.

journald va syslog

Zamonaviy distro'lar journald'ni asosiy log oluvchi qilib qo'ygan. Klassik /var/log/ fayllari hali ham mavjud, lekin ko'pi rsyslog yoki syslog-ng yordamida journal'dan ko'chiriladi.

journalctl ish jadvali

journalctl -u nginx --since "10 min ago" -p err
journalctl -fu myapp                # live tail
journalctl _PID=1234
journalctl _UID=1000
journalctl -o json-pretty
journalctl -F _SYSTEMD_UNIT         # mavjud qiymatlar

logrotate

Eski log'larni avtomatik aylantiradi va siqadi.

cat /etc/logrotate.conf
ls /etc/logrotate.d/
sudo logrotate -d /etc/logrotate.d/nginx     # debug, hech narsa qilmaydi
sudo logrotate -f /etc/logrotate.d/nginx     # majburlab

Real-time process tools

top                        # universal
htop                       # yaxshi UI
btop                       # zamonaviy
atop                       # tarixiy yozuv
iotop                      # I/O
iftop                      # tarmoq
nethogs                    # tarmoq jarayon bo'yicha
nload                      # interfeys yuki

Statistik vositalar

vmstat 1                   # har soniya: cpu, io, swap, memory
iostat -xz 1               # disk per-device
mpstat -P ALL 1            # har CPU bo'yicha
pidstat 1                  # jarayon bo'yicha
sar -n DEV 1               # tarmoq interfeys
sar -n TCP,ETCP 1          # TCP statistika
free -h ; uptime
dmesg --since "10 min ago"

USE metodi (Brendan Gregg)

Har bir resurs uchun uchta savol:

• Utilization — qancha foiz vaqt band?
• Saturation — navbat bormi? (load avg, run queue)
• Errors — xatolar (drop, retransmit, ECC)?

Resurslar: CPU, RAM, disk, tarmoq.

RED metod (servislar uchun)

• Rate — sekundiga so'rov
• Errors — necha foiz xato
• Duration — javob latentligi (p50, p95, p99)

perf — kernel-darajadagi profiler

sudo perf top                              # real-time hot funktsiyalar
sudo perf record -F 99 -ag -- sleep 30     # 99 Hz, hammasi, 30s
sudo perf report
sudo perf stat -e cache-misses,instructions ./prog

eBPF / bpftrace — yangi davr

eBPF — kernel ichida xavfsiz dasturchalar ishga tushirish texnologiyasi. bpftrace — uning awk'ga o'xshash front-end'i.

sudo bpftrace -e 'tracepoint:syscalls:sys_enter_openat { @[comm] = count(); }'
# Qaysi jarayon nechta openat() qilyapti?

sudo bpftrace -e 'kprobe:do_sys_open { printf("%s %s\n", comm, str(arg1)); }'
# Real-time fayl ochilishi

biotop-bpfcc        # bcc'dan: top I/O jarayonlar
tcpconnect-bpfcc    # yangi TCP ulanishlarni live
execsnoop-bpfcc     # har exec()

lsof — kim nimani ochgan

sudo lsof -i :443                  # 443-portni kim eshityapti
sudo lsof -p 1234                  # jarayonning ochiq fayllari
sudo lsof /var/log/syslog          # bu faylni kim ochgan
sudo lsof | grep deleted           # o'chirilgan, lekin hali ochiq fayllar (disk to'lganda muhim)

uptime
dmesg | tail
vmstat 1 5
mpstat -P ALL 1 5
pidstat 1 5
iostat -xz 1 5
free -m
sar -n DEV 1 5
sar -n TCP,ETCP 1 5
top -bn1 | head -20

Bob 13Bash advanced

"Bash — bu interfaol shell" — yarmi to'g'ri. Yaxshi yozilgan bash skriptlari yillab production'da ishlay oladi. Sirri — strict mode va parameter expansion.

Strict mode (har skriptda boshlang)

#!/usr/bin/env bash
set -euo pipefail
IFS=$'\n\t'

Parameter expansion (cheatsheet)

${var:-default}        # bo'sh bo'lsa, default
${var:=default}        # bo'sh bo'lsa, default va ASSIGN
${var:?xato xabari}    # bo'sh bo'lsa, chiqadi
${var:+alt}            # bo'sh emas bo'lsa, alt
${#var}                # uzunligi
${var#prefix}          # eng qisqa prefiksni olib tashlash
${var##prefix}         # eng uzun
${var%suffix}          # oxiridan qisqa
${var%%suffix}         # oxiridan uzun
${var/old/new}         # birinchi almashtirish
${var//old/new}        # hammasi
${var^^}               # uppercase
${var,,}               # lowercase

Test [[ ]] vs [ ]

# [[ ]] bash kengaytirilgan — har doim afzal
[[ -f /etc/hosts ]] && echo "fayl bor"
[[ -d /tmp ]]
[[ -z "$var" ]]            # bo'shmi
[[ -n "$var" ]]            # bo'sh emasmi
[[ "$a" == "b" ]]
[[ "$str" =~ ^[0-9]+$ ]]   # regex
[[ "$a" -lt "$b" ]]        # raqamli

Boshqaruv tuzilmalari

if [[ $? -eq 0 ]]; then
  echo "muvaffaqiyat"
elif [[ -f file ]]; then
  echo "fayl bor"
else
  echo "boshqa holat"
fi

for f in /var/log/*.log; do
  echo "$f"
done

for i in {1..10}; do echo $i; done

while IFS= read -r line; do
  echo "qator: $line"
done < file.txt

case "$1" in
  start) ;;
  stop) ;;
  *) echo "noma'lum"; exit 1 ;;
esac

Funksiyalar

greet() {
  local name="${1:-do'st}"
  printf 'salom, %s\n' "$name"
}
greet "Akmal"

Massivlar

files=(a.txt b.txt "c d.txt")
echo "${files[0]}"
echo "${files[@]}"        # hammasi
echo "${#files[@]}"       # soni
files+=("e.txt")          # qo'shish
for f in "${files[@]}"; do echo "$f"; done

declare -A user
user[name]=Akmal
user[role]=admin
echo "${user[name]}"

Trap: tozalash kerak

tmpdir=$(mktemp -d)
cleanup() { rm -rf "$tmpdir"; }
trap cleanup EXIT
trap 'echo "interrupted"; exit 130' INT TERM

Ishonchli skript shabloni

#!/usr/bin/env bash
set -euo pipefail
IFS=$'\n\t'

readonly SCRIPT_NAME=$(basename "$0")

log() { printf '[%s] %s\n' "$(date +%H:%M:%S)" "$*" >&2; }
err() { log "XATO: $*"; exit 1; }

usage() {
  cat <<EOF
Foydalanish: $SCRIPT_NAME [-v] <path>
EOF
}

main() {
  local verbose=0
  while getopts ":vh" opt; do
    case $opt in
      v) verbose=1 ;;
      h) usage; exit 0 ;;
      *) usage; exit 1 ;;
    esac
  done
  shift $((OPTIND - 1))

  [[ $# -lt 1 ]] && { usage; exit 1; }
  local path="$1"
  [[ -e "$path" ]] || err "yo'q: $path"

  log "ishlanmoqda: $path"
}

main "$@"

Bob 14Cron, at, systemd timers

Reja bo'yicha topshiriqlarni ishga tushirishning uch yo'li bor. Tanlov holatga bog'liq.

cron sintaksisi

# min  hour  day-of-month  month  day-of-week  command
*/5 * * * *  /usr/local/bin/check.sh
0   3 * * *  /opt/backup.sh                # har kuni 03:00
0   0 * * 0  /opt/weekly.sh                # har yakshanba
@reboot      /opt/start.sh
@daily       /opt/cleanup.sh

cron joylari

crontab -e                 # foydalanuvchi cron'i
crontab -l
crontab -r                 # o'chirish
sudo crontab -e -u www-data

# Tizim cron'lari:
ls /etc/cron.d/
ls /etc/cron.{hourly,daily,weekly,monthly}/
cat /etc/crontab

Eng tez-tez xato qilinadigan narsalar

• $PATH cron'da minimal — har binary uchun to'liq yo'l yozing yoki PATH=... qatorini qo'shing.
• Output elektron pochta'ga yuboriladi — kerak bo'lmasa: >/dev/null 2>&1.
• % belgi cron'da maxsus — qochiring (\%).
• Skript bash bo'lsa, #!/usr/bin/env bash yozing va chmod +x.

at — bir martalik

echo "/opt/backup.sh" | at 03:00 tomorrow
echo "shutdown -h now" | sudo at "now + 1 hour"
atq                        # navbat
atrm 3                     # bekor qilish

systemd timer (zamonaviy yondashuv)

Timer faylni ishga tushiradigan bir xil nomli .service kerak.

# /etc/systemd/system/backup.service
[Unit]
Description=Daily backup

[Service]
Type=oneshot
ExecStart=/opt/backup.sh
User=backup
Nice=10

# /etc/systemd/system/backup.timer
[Unit]
Description=Run backup daily

[Timer]
OnCalendar=*-*-* 03:00:00
RandomizedDelaySec=15min
Persistent=true             # missed run'ni keyinroq bajarish

[Install]
WantedBy=timers.target

sudo systemctl daemon-reload
sudo systemctl enable --now backup.timer
systemctl list-timers --all
journalctl -u backup.service

cron vs systemd timer

Tavsiya: yangi servis — timer; eski + sodda — cron qoladi.

Bob 15Xavfsizlik

Xavfsizlik — qatlam-qatlam. Bitta vositaning yetishmasligi tizimni teshikka aylantirmaydi, lekin bir nechta zaif joy birga ekspluatatsiya'ga olib keladi.

SSH hardening

# /etc/ssh/sshd_config
PermitRootLogin no
PasswordAuthentication no
PubkeyAuthentication yes
ChallengeResponseAuthentication no
KbdInteractiveAuthentication no
MaxAuthTries 3
LoginGraceTime 30
AllowUsers akmal devops
ClientAliveInterval 300
ClientAliveCountMax 2

# Qayta yuklash:
sudo sshd -t                 # konfig tekshiruv (avval shu!)
sudo systemctl reload sshd

fail2ban

Auth log'larni kuzatib, ko'p marta xato urinish qilgan IP'ni firewall'ga banlaydi. Soddagina ko'rinsa-da, "ishga tushdi" deb hisoblash uchun arxitekturani bilish shart — aks holda log'lar bor, lekin ban hech qachon kelmaydi (eng tez-tez tushib qoladigan tuzog'i).

Arxitektura

Bitta Python daemon (fail2ban-server) + boshqaruv klienti (fail2ban-client) Unix socket orqali gaplashadi. Asosiy birlik — jail: bitta filter (regex) + log manbai + action to'plami + chegaralar (maxretry, findtime, bantime).

Pipeline: log qatori → sana ajratilishi → failregex orqali <HOST> qo'lga olinadi → har jail uchun IP hisoblagichi → chegara bosilsa, banaction firewall qoidasini qo'shadi → bantime'dan keyin avtomatik unban.

Fayllar joylashuvi (eng asosiy tuzoq shu yerda)

/etc/fail2ban/
├── jail.conf       # paket bilan keladi — TEGMANG
├── jail.local      # siz yozadigan joy
├── jail.d/         # drop-in fayllar
├── filter.d/       # regex filter'lar (sshd.conf, nginx-http-auth.conf, ...)
└── action.d/       # firewall integratsiyasi (iptables-multiport.conf, ufw.conf, ...)

Asosiy parametrlar

Minimal ishchi konfig (modern best practice)

# /etc/fail2ban/jail.local
[DEFAULT]
ignoreself = true
ignoreip   = 127.0.0.1/8 ::1 10.0.0.0/8 192.168.0.0/16 fc00::/7
bantime    = 1h
findtime   = 10m
maxretry   = 5
backend    = systemd

# Modern: eskirgan recidive jail o'rniga avtomatik eksponensial ban
bantime.increment = true
bantime.factor    = 2
bantime.maxtime   = 4w

# 2026: nftables — yangi distro'larda standart firewall
banaction          = nftables-multiport
banaction_allports = nftables-allports

[sshd]
enabled  = true
mode     = aggressive          # normal + ddos + extra patternlar
maxretry = 4

[recidive]
enabled  = true                # bantime.increment'dan tashqari ham — arzon insurance
bantime  = 1w
findtime = 1d

sudo systemctl enable --now fail2ban
sudo fail2ban-client reload

systemd backend (RHEL / Arch / Fedora)

Bu distro'larda sshd faqat journal'ga yozadi — fayl yo'q. Default file backend hech narsa ko'rmaydi: status'da pos=1, 0 failure, hech kim banlanmaydi. Yechim:

[sshd]
enabled = true
backend = systemd
journalmatch = _SYSTEMD_UNIT=sshd.service + _COMM=sshd

fail2ban-client: kundalik buyruqlar

fail2ban-client ping                  # server tirikmi
fail2ban-client status                # barcha jail'lar
fail2ban-client status sshd           # bittasi: file pos, totals, banlanganlar
fail2ban-client get sshd logpath
fail2ban-client get sshd actions

# Qo'lda ban / unban
fail2ban-client set sshd banip 203.0.113.42
fail2ban-client set sshd unbanip 203.0.113.42
fail2ban-client unban 203.0.113.42    # barcha jail'larda
fail2ban-client unban --all           # DESTRUCTIVE — DB'ni ham tozalaydi

# Konfigni qayta o'qish
fail2ban-client reload                # hammasi
fail2ban-client reload sshd

Custom filter va testlash

Filter — /etc/fail2ban/filter.d/<name>.conf. Eng asosiy makro: <HOST> (IPv4/IPv6/hostname) yoki <ADDR> (faqat IP — DNS lookup yo'q, xavfsizroq).

# /etc/fail2ban/filter.d/myapp.conf
[Definition]
failregex = ^Login failed for user \S+ from <ADDR>\s*$
            ^Bad token from <ADDR>\s*$
ignoreregex =

Ishga tushirishdan oldin har doim test qiling:

# Real log'ga regex'ni urinib ko'rish
fail2ban-regex /var/log/myapp.log /etc/fail2ban/filter.d/myapp.conf

# Journald'ga
fail2ban-regex systemd-journal /etc/fail2ban/filter.d/sshd.conf

# Verbose: har match'ni qator raqami bilan
fail2ban-regex -v /var/log/auth.log sshd[mode=aggressive]

IPv6

v0.10 (2017)'dan boshlab to'liq qo'llab-quvvatlanadi: built-in filter'lar, iptables-multiport, nftables-multiport, ufw, persistence. Custom filter'larda \d+\.\d+\.\d+\.\d+ o'rniga har doim <HOST> yoki <ADDR>'ni ishlating — aks holda v6 manzillar match bo'lmaydi va siz buni log'lardan sezmaysiz.

Eng tez-tez xatolar (90% ticket'larning sababi)

1. Backend noto'g'ri — RHEL/Arch'da default auto file backend'ni tanladi, sshd esa journal'ga yozadi. Status'da 0 failure, ban yo'q. Fix: backend = systemd.
2. Filter regex log formatga mos kelmaydi — ilova yangilanib formatni o'zgartirgan. fail2ban-regex -v bilan tekshiring.
3. jail.conf tahrirlangan — keyingi apt upgrade'da yo'q bo'ladi.
4. O'zingizni banladingiz — ignoreip'ga office/VPN/local CIDR qo'shilmagan. Console / IPMI orqali qutqaring.
5. Reload qilmaslik — .local tahriri konfigda bor, lekin runtime'da yo'q. fail2ban-client reload.
6. banaction firewall stack'ga mos kelmaydi — nftables tizimda iptables-multiport: jimgina kompat qatlamiga tushadi, qoidalar amalda emas.
7. usedns=yes + DDoS — DNS resolver bo'g'iladi. warn yoki no.
8. SELinux/AppArmor journal'ni o'qishni taqiqlagan — fail2ban.log'da permission denied. Tegishli policy modul yoki bool kerak.

sudo fail2ban-client status sshd
# "Banned IP list:" ostida o'z IP'ingizni ko'rishingiz kerak
sudo fail2ban-regex /var/log/auth.log /etc/fail2ban/filter.d/sshd.conf
sudo fail2ban-client unban <IP>

SELinux (RHEL oilasi)

Mandatory Access Control (MAC) — har fayl, jarayon, port'ga "kontekst" beradi va kim nima qila olishini policy belgilaydi.

getenforce                   # Enforcing | Permissive | Disabled
sudo setenforce 0            # vaqtinchalik permissive
sestatus
ls -Z /var/www/html
ps -eZ | grep nginx
sudo restorecon -Rv /var/www/html
sudo semanage port -a -t http_port_t -p tcp 8081
sudo audit2allow -a          # buzilgan policy uchun yangi qoida tavsiya

Maslahat: SELinux'ni o'chirmang. Kontekst muammoni setroubleshoot + audit2allow bilan tuzating.

AppArmor (Ubuntu/Debian)

sudo aa-status
sudo aa-enforce /etc/apparmor.d/usr.sbin.nginx
sudo aa-complain /etc/apparmor.d/usr.sbin.nginx       # buzmasdan log qiladi

Capabilities (yana)

SUID-root o'rniga aniq capability'larni bering. setcap cap_net_bind_service=+ep ./myapp dasturga 1024'dan past port ochishga ruxsat beradi, qolgan root kuchini bermaydi.

Linux namespaces (xavfsizlik nuqtai nazaridan)

Bu — konteynerlarning poydevori. Quyida 16-bobda batafsil.

auditd

sudo auditctl -w /etc/passwd -p wa -k passwd_changes
sudo auditctl -l
sudo ausearch -k passwd_changes
sudo aureport --summary

TLS sertifikat'ni tekshirish

echo | openssl s_client -connect example.com:443 -servername example.com 2>/dev/null \
  | openssl x509 -noout -subject -issuer -dates
curl -vI https://example.com 2>&1 | grep -E "subject|issuer|expire"

Sirlar (secrets)

• Repository'ga API kalit qo'ymang. git-secrets yoki pre-commit hooks ishlating.
• Production'da .env fayllar — faqat 600 ruxsat, root yoki servis user egaligida.
• HashiCorp Vault, AWS Secrets Manager, sealed-secrets — production sirlari uchun.

Bob 16Konteynerlar (first principles)

"Konteyner" — sehrli quti emas. Bu — oddiy jarayon, lekin bir necha kernel xususiyatlari bilan o'ralgan. Bu xususiyatlarni biling — Docker'ni achchiq xatolarsiz ishlatasiz.

Konteyner = nima?

Konteyner = jarayon + bu xususiyatlar:

• Namespaces — pid, net, mnt, uts, ipc, user, cgroup. Ko'rish doirasini cheklaydi.
• cgroups — CPU, RAM, IO chegaralari.
• chroot/pivot_root — fayl tizimi ildizi.
• Capabilities — root kuchining bo'laklari.
• Seccomp — syscall'larni filtrlash.

Image — overlayfs'da yotuvchi qatlamlar to'plami. Container ishga tushganda, eng ustiga yozilishi mumkin bo'lgan bo'sh qatlam qo'shiladi (copy-on-write).

Docker — kundalik buyruqlar

docker run hello-world
docker run -d --name web -p 8080:80 nginx
docker ps                          # ishlayotganlar
docker ps -a                       # tugaganlar ham
docker logs -f web
docker exec -it web bash           # konteynerga kirish
docker stop web ; docker rm web
docker images
docker pull alpine:3.20
docker rmi alpine:3.20
docker system df                   # diskda nima yeyayapti
docker system prune -a             # tozalash

Image qurish (Dockerfile)

# Multi-stage: final image kichik bo'ladi
FROM golang:1.22-alpine AS build
WORKDIR /src
COPY go.mod go.sum ./
RUN go mod download
COPY . .
RUN CGO_ENABLED=0 go build -o /out/app ./cmd/app

FROM gcr.io/distroless/static
COPY --from=build /out/app /app
USER 65532:65532
ENTRYPOINT ["/app"]

docker build -t myapp:1.0 .
docker build --no-cache -t myapp:1.0 .
docker tag myapp:1.0 registry.example.com/myapp:1.0
docker push registry.example.com/myapp:1.0

Volume va tarmoq

docker volume create pgdata
docker run -d --name db \
  -e POSTGRES_PASSWORD=secret \
  -v pgdata:/var/lib/postgresql/data \
  -p 5432:5432 \
  postgres:16

docker network create app-net
docker run -d --network app-net --name api myapp:1.0
docker run --network app-net alpine ping -c 2 api    # DNS by name

Compose — deklarativ

# docker-compose.yml
services:
  web:
    image: nginx:alpine
    ports: ["80:80"]
    depends_on: [api]
  api:
    build: ./api
    environment:
      DB_URL: postgres://app:s3cret@db:5432/app
    depends_on: [db]
  db:
    image: postgres:16
    environment:
      POSTGRES_PASSWORD: s3cret
    volumes:
      - pgdata:/var/lib/postgresql/data

volumes:
  pgdata:

docker compose up -d
docker compose logs -f api
docker compose down               # konteynerlar; volume qoladi
docker compose down -v            # volume ham o'chsin

Image hajmi va xavfsizlik

• Asosli image'ni kichkina tanlang: alpine, distroless, scratch.
• Multi-stage'dan foydalaning — build vositalari final image'da yo'q bo'lsin.
• USER'ni root'dan boshqasiga o'zgartiring.
• --read-only root filesystem + --cap-drop=ALL + kerakli capability'larni qaytarib qo'shing.
• Image'ni skanerlang: trivy image myapp:1.0.

Kubernetes — qisqa kirish

Kubernetes — bu konteynerlarni cluster bo'ylab orchestrate qiladigan tizim. Asosiy ob'yektlar:

kubectl get pods -A
kubectl get deploy,svc -n prod
kubectl describe pod web-xyz
kubectl logs -f web-xyz -c container-name
kubectl exec -it web-xyz -- bash
kubectl apply -f deployment.yaml
kubectl rollout status deploy/web
kubectl rollout undo deploy/web
kubectl port-forward svc/api 8080:80

docker run -d --name n nginx
docker exec n ps -ef        # konteyner ichidan: PID 1 = nginx
ps -ef | grep nginx          # host'da: katta PID

Bob 17Avtomatlashtirish va IaC

"Bir marta qildimu, qayta hech narsa eslayolmayman" — bu manual operatsion qarz. Git, Ansible va cloud-init — bu qarzni tushiradi.

Git — eng zarur 90%

git init
git clone <url>
git status
git diff
git diff --staged
git add file.txt
git add -p                       # interaktiv: qaysi qismlarini stage qilish
git commit -m "matn"
git log --oneline --graph --decorate --all
git switch feature-x             # yangi sintaksis (checkout o'rnida)
git switch -c yangi-branch
git merge feature-x
git rebase main                  # tarixni chiziqli qilish
git pull --rebase                # merge commit'siz
git push
git push -u origin yangi-branch
git stash                        # vaqtinchalik yashirish
git stash pop
git reset --soft HEAD~1          # oxirgi commit'ni bekor qilish, o'zgarishlar qoladi
git reset --hard origin/main     # local'ni remote'ga to'liq tenglashtirish
git reflog                       # "yo'qotgan" commit'larni topish
git blame file.txt
git bisect                       # regressiya'ni binary search bilan topish

Ansible — agentless avtomatizatsiya

Ansible target server'larga SSH bilan ulanadi va Python modul'larini yuborib bajaradi. Idempotent: bir xil playbook'ni 100 marta yuborsangiz ham, faqat birinchisi o'zgartiradi.

# inventory.ini
[web]
web1.example.com
web2.example.com

[db]
db1.example.com

[all:vars]
ansible_user=devops

Ad-hoc:

ansible all -i inventory.ini -m ping
ansible web -i inventory.ini -m apt -a "name=nginx state=present" --become
ansible db -i inventory.ini -a "df -h"

Playbook:

# site.yml
- hosts: web
  become: yes
  vars:
    nginx_port: 80
  tasks:
    - name: Install nginx
      apt:
        name: nginx
        state: present
        update_cache: yes

    - name: Deploy site
      copy:
        src: index.html
        dest: /var/www/html/index.html
        owner: www-data
        group: www-data
        mode: '0644'
      notify: reload nginx

    - name: Ensure nginx running
      systemd:
        name: nginx
        state: started
        enabled: yes

  handlers:
    - name: reload nginx
      systemd:
        name: nginx
        state: reloaded

ansible-playbook -i inventory.ini site.yml
ansible-playbook -i inventory.ini site.yml --check     # dry-run
ansible-playbook -i inventory.ini site.yml --diff
ansible-playbook -i inventory.ini site.yml --limit web1.example.com

cloud-init

Bulutli VM'lar (AWS, GCP, Azure, DigitalOcean) birinchi boot'da user-data'ni o'qiydi va sozlanadi.

#cloud-config
users:
  - name: devops
    sudo: ALL=(ALL) NOPASSWD:ALL
    ssh_authorized_keys:
      - ssh-ed25519 AAA...

package_update: true
package_upgrade: true
packages:
  - nginx
  - htop

runcmd:
  - systemctl enable --now nginx
  - ufw allow 80/tcp

Immutable infrastructure

Bir marta yaratilgan server hech qachon qaytadan o'zgartirilmaydi — yangi versiya kerak bo'lsa, yangi server quriladi va eskisi o'chiriladi. Konfiguratsiyaga "drift" yo'q. Vositalar: Packer (image qurish), Terraform (infrastruktura), GitOps (ArgoCD/Flux).

Bob 18Performance va troubleshooting

Incident vaqtida — sababni topish 80%, tuzatish 20%. Bir nechta runbook va metodologiyalarni yodda tuting.

60-soniyali tezkor audit

Brendan Gregg uslubidagi birinchi reaksiya:

uptime              # load avg trend
dmesg | tail        # oxirgi kernel xabarlari (OOM, ECC)
vmstat 1 5          # cpu, io wait, swap
mpstat -P ALL 1 5   # har CPU bo'yicha
pidstat 1 5         # jarayon bo'yicha
iostat -xz 1 5      # disk per device
free -m
sar -n DEV 1 5      # tarmoq
sar -n TCP,ETCP 1 5
top -bn1 | head

Stsenariy: yuqori CPU

1. top — qaysi jarayon yuqori? %us (user) vs %sy (system) vs %wa (I/O wait).
2. Foydalanuvchi kodi: perf top -p <pid> yoki pidstat 1.
3. Kernel kodi: perf top umumiy.
4. I/O wait yuqori: bu CPU emas — disk muammo (iostat -xz 1).

Stsenariy: yuqori load average, past CPU

Load avg = R + D state'dagi jarayonlar. CPU past, lekin yuqori load = ko'p D-state.

ps -eo pid,stat,wchan,cmd | awk '$2 ~ /D/'
iostat -xz 1
iotop -oP

Stsenariy: OOM

dmesg -T | grep -iE 'oom|killed process'
journalctl -k --since "1 hour ago" | grep -i oom
free -h
cat /proc/meminfo | head -10
# Eng RAM yeyayotgan jarayonlar:
ps aux --sort=-%mem | head

Stsenariy: disk to'la

df -h
df -i                                # inode tugamadimi?
sudo du -sh /var/* 2>/dev/null | sort -rh | head
sudo ncdu /
# Eng yashirin sabab — o'chirilgan, lekin ochiq fayllar:
sudo lsof | grep deleted | sort -k7 -n | tail
# Yechim: tegishli jarayonni qayta yuklash

Stsenariy: tarmoq muammosi

1. Bog'lanish bormi? ping -c 4 8.8.8.8 (IP, DNS muammosini chetlab o'tish).
2. DNS? dig example.com, resolvectl status.
3. Yo'l qaerda buziladi? mtr example.com (paket loss qaysi hop'da).
4. Port ochiqmi? nc -vz host 443.
5. Mahalliy firewall? sudo iptables -L, sudo ufw status.
6. Paket darajasida? sudo tcpdump -i any host X and port Y.

Stsenariy: SSH sekin ulanmoqda

Odatiy sabab: server'dan reverse DNS yoki GSSAPI'ga ishlamayotgan urinish.

# /etc/ssh/sshd_config:
UseDNS no
GSSAPIAuthentication no

Stsenariy: servis ishga tushmaydi

systemctl status myapp
journalctl -u myapp -n 100
journalctl -u myapp --since "10 min ago"
sudo -u myapp /opt/myapp/bin/server     # qo'lda ishga tushirish — to'g'ridan-to'g'ri xato ko'rinadi
sudo strace -f -p <pid>                  # aslida nima sodir bo'lyapti

Latency analysis

Vaqt qaerga ketadi? curl bilan tezkor parchalash:

curl -w '
DNS:  %{time_namelookup}
TCP:  %{time_connect}
TLS:  %{time_appconnect}
TTFB: %{time_starttransfer}
TOT:  %{time_total}
' -o /dev/null -s https://example.com

Runbook shabloni

Har bir kritik servis uchun yozilgan, takrorlanadigan qadam-ba-qadam yo'riqnoma:

1. Hodisa belgilari — qanday alarm/log ko'rinadi.
2. Tasdiqlash — hodisa haqiqiyligini tekshirish buyruqlari.
3. Triage qadamlari — birinchi 5 daqiqada nima qilish.
4. Eskalatsiya — kim bilan bog'lanish.
5. Kommunikatsiya — status sahifasi, mijoz xabari.
6. Postmortem — keyin yozilishi kerak.

Runbook'lar — Markdown, repository'da. Skript bilan avtomatlashtiring.

Bob 19tmux

Terminal multiplexer. SSH uzilsa ham, ish davom etadi. Bir necha sessiya, har birida window'lar va pane'lar — engineer'ning kundalik vositasi. Faqat klaviatura bilan boshqariladigan tab/window menejer deb tasavvur qiling, lekin server tomonida yashaydi.

Arxitektura: server → session → window → pane

To'rt darajali iyerarxiya:

• Server — har foydalanuvchi uchun bitta orqa fonda turuvchi jarayon (/tmp/tmux-$UID/default socket). Birinchi tmux chaqiruvi serverni ishga tushiradi; oxirgi sessiya tugaganda u o'ladi (exit-empty off qilmasangiz).
• Session — nomlangan ish maydoni. Bir necha bo'lishi mumkin.
• Window — sessiya ichidagi "tab".
• Pane — window ichidagi haqiqiy TTY (sizning shell yoki dasturingiz).
• Client — sessiyaga ulangan terminal (terminalingizning oynasi).

Eng muhim xususiyat — detach/attach: SSH yo'qolsa, server hech qaerga ketmaydi. ssh server qilib qaytasiz, tmux a deysiz va dnf upgrade'ingiz, deploy'ingiz, db migration'ingiz hech to'xtamagan. Bu — production'da tmux'ni almashtirib bo'lmaydigan qiluvchi yagona xususiyat.

Session boshqaruvi

tmux                              # yangi nomsiz sessiya, attach
tmux new -s deploy                # nomlangan
tmux new -d -s deploy 'htop'      # detached + buyruq bilan
tmux ls                           # sessiyalar ro'yxati
tmux a                            # oxirgisiga attach
tmux a -t deploy
tmux a -dt deploy                 # attach va boshqa client'larni detach
tmux kill-session -t deploy
tmux kill-server                  # hammasini o'ldirish
tmux info                         # diagnostika

Default keybinding'lar (prefix C-b)

Hamma narsa prefix orqali. Default — Ctrl-b, keyin alohida bir tugma. C-b ? butun ro'yxatni ko'rsatadi (q bilan chiqing).

CLI va skripting

Har binding shell'dan chaqiriladigan tmux <buyruq> ekvivalentiga ega. Demak, butun tmux skript qilinadi.

# Targeting sintaksisi: session:window.pane (har bir qism optional)
tmux new-window -t work -n logs
tmux split-window -v -t work:logs -p 30        # 30% balandlik
tmux send-keys -t work:logs.1 'tail -f /var/log/syslog' Enter
tmux source-file ~/.tmux.conf                   # konfigni qayta yuklash
tmux display-message "deploy started"
tmux capture-pane -p -S - -t work:logs.1 > /tmp/log.txt
tmux display -p '#{session_name}:#{window_index}.#{pane_index}'

Copy mode

C-b [ bilan kirasiz — endi tarix bo'ylab scroll, qidirish va copy qila olasiz. Ikki kalit jadvali bor: copy-mode (emacs, default) va copy-mode-vi (set -g mode-keys vi sozlasangiz). vi-mode key'lari (kim vim biladi):

Tizim clipboard'iga copy qilish uchun (zamonaviy terminallar OSC 52'ni qo'llab-quvvatlaydi):

set -g set-clipboard on

Yo'q bo'lsa, copy-pipe bilan xclip/pbcopy'ga yuboring:

bind -T copy-mode-vi y send-keys -X copy-pipe-and-cancel 'xclip -selection clipboard -in'

~/.tmux.conf — engineering uchun ishchi konfig

# Prefix: Ctrl-a — readline bilan to'qnashmaydi
unbind C-b
set -g prefix C-a
bind C-a send-prefix              # ichkari tmux'ga literal C-a yuborish

# Asosiy
set -g default-terminal "tmux-256color"
set -ga terminal-overrides ",xterm-256color:RGB,*:Tc"
set -s escape-time 10             # default 500ms — vim'da Esc lag
set -g history-limit 100000
set -g mouse on
set -g focus-events on
set -g base-index 1               # window'lar 1'dan, klaviaturaga moslashgan
setw -g pane-base-index 1
set -g renumber-windows on
set -g set-clipboard on

# vi'cha hamma joyda
setw -g mode-keys vi
set -g status-keys vi

# Copy mode vim'cha
bind -T copy-mode-vi v   send-keys -X begin-selection
bind -T copy-mode-vi C-v send-keys -X rectangle-toggle
bind -T copy-mode-vi y   send-keys -X copy-pipe-and-cancel

# Split'lar joriy katalogda ochilsin
bind '"' split-window -v -c "#{pane_current_path}"
bind %   split-window -h -c "#{pane_current_path}"
bind c   new-window      -c "#{pane_current_path}"

# Reload binding
bind r source-file ~/.tmux.conf \; display "reloaded"

# Pane navigatsiya — vim'cha
bind -r h select-pane -L
bind -r j select-pane -D
bind -r k select-pane -U
bind -r l select-pane -R

# Status line — minimal
set -g status-position top
set -g status-interval 5
set -g status-left  "#[bold] #S "
set -g status-right "#(whoami)@#H  %Y-%m-%d %H:%M "

Plugin manager (TPM)

git clone https://github.com/tmux-plugins/tpm ~/.tmux/plugins/tpm

# ~/.tmux.conf'ning OXIRIDA
set -g @plugin 'tmux-plugins/tpm'
set -g @plugin 'tmux-plugins/tmux-sensible'
set -g @plugin 'tmux-plugins/tmux-resurrect'
set -g @plugin 'tmux-plugins/tmux-continuum'
set -g @plugin 'tmux-plugins/tmux-yank'

set -g @continuum-restore 'on'
set -g @continuum-save-interval '15'

run '~/.tmux/plugins/tpm/tpm'

Reload'dan keyin: prefix + I — install, prefix + U — update.

Engineer dashboard pattern

Bir SSH server uchun 4-pane "vaziyatga ko'rish" sessiyasini tayyor qiluvchi idempotent skript:

#!/usr/bin/env bash
# ~/bin/tmux-dash <hostname>
HOST="${1:-localhost}"
SESSION="dash-${HOST}"

tmux has-session -t "$SESSION" 2>/dev/null && {
  exec tmux a -t "$SESSION"
}

tmux new -d -s "$SESSION" -n monitor
tmux send-keys    -t "$SESSION:monitor"   "ssh $HOST htop" Enter

tmux split-window -v -t "$SESSION:monitor" -p 50
tmux send-keys    -t "$SESSION:monitor.1" \
                  "ssh $HOST 'sudo journalctl -fu nginx'" Enter

tmux split-window -h -t "$SESSION:monitor.0"
tmux send-keys    -t "$SESSION:monitor.2" \
                  "ssh $HOST 'watch -n2 \"ss -tnp | head -30\"'" Enter

tmux split-window -h -t "$SESSION:monitor.1"
tmux send-keys    -t "$SESSION:monitor.3" "ssh $HOST" Enter

tmux select-pane -t "$SESSION:monitor.3"
exec tmux a -t "$SESSION"

Eng tez-tez xatolar

• escape-time default 500ms — vim/neovim ichida Esc dahshatli sekin sezmoqda. set -s escape-time 10 — masala hal.
• Noto'g'ri TERM — tashqarida xterm-256color, ichida tmux-256color bo'lishi kerak. Noto'g'ri TERM → singan ranglar, g'alati key'lar.
• Ichkari tmux ichkari tmux ichida (lokal tmux ichida SSH ichida tmux) — ichkari tmux prefix'ni yutadi. bind C-a send-prefix bo'lsa, C-a C-a ichkariga yetkazadi. Yoki ichkarining prefix'ini boshqacha qiling.
• True color yo'q — terminal-overrides ',*:Tc' kerak + terminal emulator uni qo'llab-quvvatlasin (iTerm2, kitty, alacritty, wezterm).
• exit-empty on default — oxirgi sessiya yopilsa, server o'ladi va auto-restore o'rganmoqda. Saqlash uchun: set -g exit-empty off.
• Ikki client geometriyasi turlicha — kichikrog'i butunni siqib qo'yadi. tmux a -d bilan boshqalarni detach qiling yoki setw -g aggressive-resize on.

screen vs tmux

GNU Screen (1987) va tmux (2007, OpenBSD) bir muammoni yechadi. 2026'da tmux yutadi: aktiv ishlanadi, vertikal/gorizontal split yadrosida, skriptlanadigan buyruqlar to'plami, true-color, OSC 52, plugin ekosistemasi (TPM). Eski POSIX server'larda screen qoladi, lekin yangi loyihalarda tmux — to'g'ri tanlov.

ssh server
tmux new -s longjob
# sessiya ichida:
sleep 600 && echo "tugadi" | tee /tmp/done.txt

# C-b d — detach
# SSH'ni yoping (^D yoki terminal yopish)

# Yangi terminal:
ssh server
tmux a -t longjob   # ish hali davom etmoqda

IlovaCheatsheet

Bir varaqda eng zarurlar.

Fayl/katalog

ls -lah        ll  cd -        pushd/popd
cp -a src dst  mv  rm -rf
find /var -type f -name "*.log" -mtime +7 -delete
find . -type f -size +100M
find . -newer ref.txt
ln -s target link
stat file ; file file

Matn / qidirish

grep -RIn "TODO" .
grep -E "^(GET|POST) /api"
sed -i.bak 's/foo/bar/g' file
awk -F: '$3 >= 1000 {print $1}' /etc/passwd
sort | uniq -c | sort -rn
diff -u a b ; comm -3 a b

Jarayon

ps -eo pid,ppid,user,%cpu,%mem,cmd --sort=-%cpu | head
pgrep -af nginx
kill -HUP <pid> ; pkill -f worker
nohup ./srv &disown
htop ; btop

Tarmoq

ip -br a ; ip r ; ip neigh
ss -tulpn ; ss -ti
dig +short A example.com ; dig MX example.com
mtr example.com ; nc -vz host 443
curl -I https://x ; curl -L --resolve x:443:1.2.3.4 https://x
sudo tcpdump -i any -n port 443

Disk

lsblk -f ; blkid
df -h ; df -i ; du -sh /var/*
mount | column -t
sudo mount /dev/sdb1 /mnt -o noatime
mkfs.ext4 -L data /dev/sdb1
findmnt --verify

systemd

systemctl list-units --state=failed
systemctl status / start / stop / restart / reload
systemctl enable --now svc
journalctl -u svc -f -p err --since "10 min ago"
systemd-analyze blame
systemctl list-timers --all

Paket

apt update && apt install pkg ; apt purge pkg
dpkg -l | grep pkg ; dpkg -L pkg ; dpkg -S file
dnf install pkg ; dnf history undo <id>
rpm -qa | grep pkg ; rpm -ql pkg ; rpm -qf /etc/file

Ruxsat

chmod 750 file ; chmod -R u=rwX,g=rX,o= dir
chown -R user:group dir
setfacl -m u:user:rw file
getcap binary ; setcap cap_net_bind_service=+ep binary
sudo visudo

Docker

docker run -d --name n -p 80:80 nginx
docker ps -a ; docker logs -f n ; docker exec -it n sh
docker images ; docker rmi img ; docker system prune -a
docker compose up -d ; docker compose logs -f
docker build -t app:1 . ; docker push registry/app:1

Git

git status ; git diff ; git diff --staged
git switch -c branch ; git commit -am "msg"
git pull --rebase ; git push -u origin branch
git log --oneline --graph --all
git reset --hard origin/main ; git reflog

tmux

tmux new -s <name> ; tmux ls ; tmux a -t <name>
C-b c / C-b , / C-b &        # window: yangi / nom / o'chirish
C-b % / C-b "                # split: chap-o'ng / yuqori-past
C-b z / C-b o / C-b x        # pane: zoom / o'tish / o'chirish
C-b d / C-b s / C-b [        # detach / session tanlash / copy mode
tmux send-keys -t s:w.p "cmd" Enter
tmux source-file ~/.tmux.conf

fail2ban

fail2ban-client ping ; status ; status sshd
fail2ban-client set sshd banip / unbanip <IP>
fail2ban-client unban <IP>       # har qanday jail'dan
fail2ban-client reload [jail]
fail2ban-regex /var/log/auth.log /etc/fail2ban/filter.d/sshd.conf

Performance — 60s audit

uptime ; dmesg | tail
vmstat 1 5 ; mpstat -P ALL 1 5 ; pidstat 1 5
iostat -xz 1 5 ; free -m
sar -n DEV 1 5 ; sar -n TCP,ETCP 1 5
ss -s ; top -bn1 | head -20

Tezkor regex eslatmalari

Manbalar (chuqurroq o'qish)

• man <buyruq> — har doim birinchi to'xtash joyi.
• Arch Wiki — eng aniq va dolzarb.
• Debian / Ubuntu rasmiy hujjatlar.
• labex.io — interaktiv lab'lar.
• Linux Journey — fundamentaldan progressiv.
• Brendan Gregg — performance, eBPF, USE method.
• LWN.net — kernel'ning yangiliklari.
• The Linux Programming Interface (Michael Kerrisk) — syscall darajasidagi bibliya.