Introduction to cURL
cURL (Client URL) is a command-line tool and library for transferring data with URLs. First released in 1997, it has become the standard for making HTTP requests from terminal environments across all operating systems. cURL supports a wide range of protocols including HTTP, HTTPS, FTP, SFTP, SCP, and many others, making it an essential tool for developers, system administrators, and IT professionals.
Installation and Verification
Linux
Most Linux distributions include cURL by default. To verify installation and check version:
curl --version
If not installed:
# Ubuntu/Debian sudo apt update && sudo apt install curl # RHEL/CentOS sudo yum install curl # Fedora sudo dnf install curl # Arch Linux sudo pacman -S curl
macOS
cURL comes pre-installed on macOS. For the latest version via Homebrew:
# Check current version curl --version # Install/upgrade via Homebrew brew install curl
Windows
Windows 10/11: Built into Command Prompt and PowerShell (build 17063+)
Manual Installation:
- Download from official website: https://curl.se/windows/
- Extract to a directory
- Add to PATH environment variable
Package Managers:
# Chocolatey choco install curl # Scoop scoop install curl # Winget winget install curl
Git Bash: Includes cURL out of the box
Basic Command Structure
The fundamental syntax of cURL is:
curl [options] [URL]
Testing with Free APIs
We’ll use several free APIs for demonstration:
- JSONPlaceholder – Fake REST API for testing
- httpbin – HTTP request and response service
- Postman Echo – API for testing requests
Basic HTTP Operations
GET Requests (Default)
# Simple GET request curl https://jsonplaceholder.typicode.com/posts # GET with specific resource curl https://jsonplaceholder.typicode.com/posts/1 # Multiple GET requests curl https://jsonplaceholder.typicode.com/posts/1 https://jsonplaceholder.typicode.com/posts/2
Saving Output to Files
# Save to file with specified name (all platforms) curl -o output.json https://jsonplaceholder.typicode.com/posts # Save with remote file name (all platforms) curl -O https://jsonplaceholder.typicode.com/posts # Append to existing file curl -a https://jsonplaceholder.typicode.com/posts/1 >> existing_file.json
Viewing Response Details
# Include HTTP headers in output curl -i https://jsonplaceholder.typicode.com/posts/1 # Display only headers curl -I https://jsonplaceholder.typicode.com/posts/1 # Verbose mode (shows connection details) curl -v https://jsonplaceholder.typicode.com/posts/1 # Silent mode (no progress meter) curl -s https://jsonplaceholder.typicode.com/posts/1 # Silent but show errors curl -sS https://jsonplaceholder.typicode.com/posts/1
HTTP Methods
POST Requests
# Basic POST
curl -X POST https://jsonplaceholder.typicode.com/posts
# POST with data
curl -X POST https://jsonplaceholder.typicode.com/posts \
-d "title=Test&body=Content&userId=1"
# POST with JSON data
curl -X POST https://jsonplaceholder.typicode.com/posts \
-H "Content-Type: application/json" \
-d '{"title": "Test Post", "body": "This is content", "userId": 1}'
PUT Requests
curl -X PUT https://jsonplaceholder.typicode.com/posts/1 \
-H "Content-Type: application/json" \
-d '{"id": 1, "title": "Updated Title", "body": "Updated content", "userId": 1}'
PATCH Requests
curl -X PATCH https://jsonplaceholder.typicode.com/posts/1 \
-H "Content-Type: application/json" \
-d '{"title": "Partially Updated Title"}'
DELETE Requests
curl -X DELETE https://jsonplaceholder.typicode.com/posts/1
Working with Headers
Custom Request Headers
# Single header
curl -H "Authorization: Bearer token123" https://api.example.com/data
# Multiple headers
curl -H "Content-Type: application/json" \
-H "Authorization: Bearer token123" \
-H "X-Custom-Header: Value" \
https://api.example.com/data
# Remove default header
curl -H "Host:" https://example.com
Viewing Request Headers in Verbose Mode
curl -v https://jsonplaceholder.typicode.com/posts/1
Authentication Methods
Basic Authentication
# With username and password curl -u username:password https://api.example.com/protected # Password prompt (hides password in command history) curl -u username https://api.example.com/protected
Bearer Token Authentication
curl -H "Authorization: Bearer eyJhbGciOiJIUzI1NiIsInR5cCI6IkpXVCJ9" \
https://api.example.com/data
Digest Authentication
curl --digest -u username:password https://api.example.com/protected
File Operations
Uploading Files with POST
Linux and macOS:
# Single file upload curl -X POST https://httpbin.org/post \ -F "file=@/path/to/local/file.txt" # Multiple files curl -X POST https://httpbin.org/post \ -F "file1=@/path/to/file1.txt" \ -F "file2=@/path/to/file2.jpg" # File with additional form fields curl -X POST https://httpbin.org/post \ -F "document=@/path/to/document.pdf" \ -F "description=Quarterly Report" \ -F "category=finance"
Windows:
:: Command Prompt curl -X POST https://httpbin.org/post ^ -F "file=@C:\Users\Username\Documents\file.txt" :: With additional fields curl -X POST https://httpbin.org/post ^ -F "file=@C:\Users\Username\Documents\file.txt" ^ -F "description=Test upload"
Windows PowerShell:
# PowerShell curl -X POST https://httpbin.org/post ` -F "file=@C:\Users\Username\Documents\file.txt" ` -F "description=Test upload"
Windows Git Bash (same as Linux):
curl -X POST https://httpbin.org/post \ -F "file=@/c/Users/Username/Documents/file.txt" \ -F "description=Test upload"
Binary File Upload
# Upload raw binary data curl -X POST https://httpbin.org/post \ -H "Content-Type: application/octet-stream" \ --data-binary @/path/to/file.bin
Downloading Files
# Download and save with remote filename curl -O https://example.com/files/document.pdf # Download with custom filename curl -o mydocument.pdf https://example.com/files/document.pdf # Resume interrupted download curl -C - -O https://example.com/large-file.zip # Download with progress bar curl --progress-bar -O https://example.com/large-file.zip
Data Formats and Processing
JSON Data
# Send JSON data
curl -X POST https://jsonplaceholder.typicode.com/posts \
-H "Content-Type: application/json" \
-d '{
"title": "Test Post",
"body": "This is the content",
"userId": 1
}'
# Pretty print JSON response (Linux/macOS with jq)
curl -s https://jsonplaceholder.typicode.com/posts/1 | jq .
# Windows PowerShell alternative
curl -s https://jsonplaceholder.typicode.com/posts/1 | ConvertFrom-Json
# Format JSON with Python (cross-platform)
curl -s https://jsonplaceholder.typicode.com/posts/1 | python -m json.tool
URL-Encoded Form Data
curl -X POST https://httpbin.org/post \ -H "Content-Type: application/x-www-form-urlencoded" \ -d "username=johndoe&password=secret123&action=login"
XML Data
curl -X POST https://httpbin.org/post \ -H "Content-Type: application/xml" \ -d '<?xml version="1.0"?><note><to>User</to><from>Admin</from></note>'
Advanced Features
Following Redirects
# Automatically follow redirects curl -L https://httpbin.org/redirect/2 # Maximum number of redirects curl -L --max-redirs 5 https://example.com/redirect
Rate Limiting
# Limit download speed (bytes per second) curl --limit-rate 100K -O https://example.com/largefile.zip # Limit upload speed curl --limit-rate 1M -T largefile.zip ftp://example.com/
Timeouts and Retries
# Connection timeout (seconds) curl --connect-timeout 10 https://example.com # Maximum time for operation curl --max-time 30 https://example.com # Retry on failure curl --retry 3 https://example.com # Retry with delay curl --retry 3 --retry-delay 5 https://example.com
Cookie Management
# Send cookies curl -b "session=abc123; user=johndoe" https://example.com/dashboard # Save cookies to file curl -c cookies.txt https://example.com/login # Load cookies from file curl -b cookies.txt https://example.com/dashboard # Combine save and load curl -c cookies.txt -b cookies.txt https://example.com/session
SSL/TLS Configuration
# Skip SSL certificate verification (NOT recommended for production) curl -k https://example.com # Specify SSL version curl --tlsv1.2 https://example.com # Use specific certificate curl --cert client.crt --key client.key https://secure.example.com # Specify CA bundle curl --cacert /path/to/ca-bundle.pem https://secure.example.com
Compression
# Request compressed response curl -H "Accept-Encoding: gzip, deflate" https://example.com/data # Compress request data curl --compressed -X POST https://example.com/api -d @data.json
Platform-Specific Considerations
Windows Path Handling
:: Command Prompt - Use caret for line continuation
curl -X POST https://httpbin.org/post ^
-H "Content-Type: application/json" ^
-d "{\"name\":\"test\"}"
:: Paths with spaces require quotes
curl -F "file=@\"C:\Users\My Name\file.txt\"" https://httpbin.org/post
# PowerShell - Use backtick for line continuation
curl -X POST https://httpbin.org/post `
-H "Content-Type: application/json" `
-d '{"name":"test"}'
# Escaping quotes in PowerShell
curl -X POST https://httpbin.org/post `
-H "Content-Type: application/json" `
-d '{\"name\":\"test\"}'
Linux/macOS Shell Features
# Using variables
API_URL="https://jsonplaceholder.typicode.com"
ENDPOINT="/posts"
curl "${API_URL}${ENDPOINT}"
# Reading data from file
curl -X POST https://httpbin.org/post \
-H "Content-Type: application/json" \
-d @data.json
# Using heredoc for multiline JSON
curl -X POST https://jsonplaceholder.typicode.com/posts \
-H "Content-Type: application/json" \
-d @- << EOF
{
"title": "Heredoc Test",
"body": "Content from heredoc",
"userId": 1
}
EOF
Practical Examples and Scripts
Example 1: API Health Check Script
#!/bin/bash
# Linux/macOS API health check
API_ENDPOINTS=(
"https://jsonplaceholder.typicode.com/posts"
"https://jsonplaceholder.typicode.com/comments"
"https://jsonplaceholder.typicode.com/albums"
)
echo "API Health Check"
echo "================"
for endpoint in "${API_ENDPOINTS[@]}"; do
echo -n "Testing $endpoint... "
http_code=$(curl -s -o /dev/null -w "%{http_code}" "$endpoint")
if [ "$http_code" -eq 200 ]; then
echo "OK ($http_code)"
else
echo "FAILED ($http_code)"
fi
done
Example 2: Batch File Upload Script
#!/bin/bash
# Cross-platform file upload script
UPLOAD_URL="https://httpbin.org/post"
UPLOAD_DIR="./uploads"
echo "Batch File Upload"
echo "================="
# Platform-specific directory creation
if [[ "$OSTYPE" == "msys" ]] || [[ "$OSTYPE" == "win32" ]]; then
# Windows
if not exist "%UPLOAD_DIR%" mkdir "%UPLOAD_DIR%"
UPLOAD_DIR="C:\Users\%USERNAME%\Documents\uploads"
else
# Linux/macOS
mkdir -p "$UPLOAD_DIR"
fi
# Find and upload files
find "$UPLOAD_DIR" -name "*.txt" -type f | while read -r file; do
echo "Uploading: $(basename "$file")"
if [[ "$OSTYPE" == "msys" ]] || [[ "$OSTYPE" == "win32" ]]; then
# Windows syntax
curl -X POST "$UPLOAD_URL" ^
-F "file=@\"$file\"" ^
-F "timestamp=$(date +%s)" > /dev/null 2>&1
else
# Linux/macOS syntax
curl -X POST "$UPLOAD_URL" \
-F "file=@$file" \
-F "timestamp=$(date +%s)" > /dev/null 2>&1
fi
if [ $? -eq 0 ]; then
echo " Success"
else
echo " Failed"
fi
done
Example 3: Download Manager with Progress
#!/bin/bash
# Download manager script
DOWNLOADS=(
"https://httpbin.org/image/jpeg"
"https://httpbin.org/image/png"
"https://httpbin.org/xml"
)
DOWNLOAD_DIR="./downloads"
# Create download directory
mkdir -p "$DOWNLOAD_DIR"
echo "Starting downloads..."
echo "---------------------"
for url in "${DOWNLOADS[@]}"; do
filename=$(basename "$url")
output_path="$DOWNLOAD_DIR/$filename"
echo "Downloading: $filename"
# Download with progress bar
if [[ "$OSTYPE" == "darwin"* ]] || [[ "$OSTYPE" == "linux-gnu"* ]]; then
# Linux/macOS with progress bar
curl --progress-bar -L -o "$output_path" "$url"
else
# Windows (Git Bash supports same syntax)
curl --progress-bar -L -o "$output_path" "$url"
fi
# Check file size
if [[ "$OSTYPE" == "darwin"* ]] || [[ "$OSTYPE" == "linux-gnu"* ]]; then
size=$(stat -f%z "$output_path" 2>/dev/null || stat -c%s "$output_path")
else
# Windows compatibility
size=$(wc -c < "$output_path")
fi
echo " Size: $size bytes"
echo " Saved to: $output_path"
echo
done
echo "All downloads completed."
Debugging and Troubleshooting
Verbose Output Analysis
# Complete verbose output curl -v https://httpbin.org/get # Save verbose output to file curl -v https://httpbin.org/get 2> debug.log # Trace detailed communication curl --trace trace.txt https://httpbin.org/get curl --trace-ascii trace_ascii.txt https://httpbin.org/get
Performance Testing
# Timing analysis
curl -w "
Time Summary:
------------
Name lookup: %{time_namelookup}
Connect: %{time_connect}
App connect: %{time_appconnect}
Pre-transfer: %{time_pretransfer}
Start transfer: %{time_starttransfer}
Total: %{time_total}
" https://httpbin.org/delay/2
# Speed test
curl -w "
Download Speed: %{speed_download} bytes/sec
" -o /dev/null https://httpbin.org/stream-bytes/1000000
Error Handling
# Continue on error (for scripts) curl --fail-with-body https://httpbin.org/status/404 # Show HTTP errors on stderr curl --show-error https://httpbin.org/status/500 # Retry logic curl --retry 3 --retry-delay 2 --retry-max-time 30 https://example.com
Integration with Other Tools
cURL with grep (Linux/macOS)
# Search in response curl -s https://jsonplaceholder.typicode.com/posts | grep -i "voluptatem" # Extract specific data curl -s https://jsonplaceholder.typicode.com/posts/1 | grep -o '"title":"[^"]*"'
cURL with awk (Linux/macOS)
# Process response data
curl -s https://jsonplaceholder.typicode.com/posts | awk -F'"' '/"title":/ {print $4}'
cURL in Python Scripts
import subprocess
import json
# Execute cURL from Python
result = subprocess.run(
['curl', '-s', 'https://jsonplaceholder.typicode.com/posts/1'],
capture_output=True,
text=True
)
if result.returncode == 0:
data = json.loads(result.stdout)
print(f"Title: {data['title']}")
cURL in Batch Scripts (Windows)
@echo off
:: Windows batch script with cURL
set API_URL=https://jsonplaceholder.typicode.com/posts/1
echo Making API request...
curl -s -o response.json %API_URL%
if %errorlevel% equ 0 (
echo Request successful
type response.json
) else (
echo Request failed with error: %errorlevel%
)
Security Best Practices
- Never store credentials in scripts
# BAD: Hardcoded credentials curl -u admin:password123 https://api.example.com # GOOD: Use environment variables curl -u "$API_USER:$API_PASS" https://api.example.com # BETTER: Use credential files or keychain curl --netrc-file ~/.netrc https://api.example.com
- Validate SSL certificates
# Avoid -k flag in production curl --cacert /path/to/cert.pem https://secure.example.com
- Sanitize input in scripts
# Validate URLs before using
if [[ "$URL" =~ ^https:// ]]; then
curl "$URL"
else
echo "Invalid URL"
fi
- Use secure protocols
# Always prefer HTTPS over HTTP curl https://secure.example.com # GOOD curl http://example.com # AVOID
Common Use Cases
1. API Development and Testing
- Rapid endpoint testing during development
- Automated API health checks
- Load testing and performance monitoring
2. Web Scraping and Data Collection
- Download web pages for analysis
- Extract data from APIs
- Monitor website changes
3. File Transfer Operations
- Upload files to servers
- Download logs and reports
- Backup and restore operations
4. System Administration
- Service health monitoring
- Configuration management
- Automated deployments
5. Network Troubleshooting
- Testing connectivity
- Verifying firewall rules
- Debugging HTTP issues
Conclusion
cURL is an indispensable tool in the modern developer’s toolkit. Its versatility across platforms, support for numerous protocols, and powerful feature set make it suitable for a wide range of tasks from simple API testing to complex automated workflows. By mastering cURL, you gain the ability to interact with web services and transfer data efficiently from the command line, regardless of your operating system.
The key to effective cURL usage lies in understanding its options, practicing with different scenarios, and integrating it into your daily workflows. Start with basic requests, gradually incorporate more advanced features, and soon you’ll find cURL to be one of your most frequently used tools.
Remember that with command-line tools comes responsibility. Always respect the terms of service of the APIs and websites you interact with, implement proper error handling in your scripts, and prioritize security in all your operations.
Further Resources
- Official Documentation: https://curl.se/docs/
- cURL Book: https://everything.curl.dev/
- HTTP Status Codes: https://developer.mozilla.org/en-US/docs/Web/HTTP/Status
- RFC Standards: https://datatracker.ietf.org/doc/html/rfc7231 (HTTP/1.1)
- Practice APIs:
- JSONPlaceholder: https://jsonplaceholder.typicode.com/
- httpbin: https://httpbin.org/
- Postman Echo: https://postman-echo.com/
By incorporating cURL into your development and administration workflows, you’ll work more efficiently and gain deeper insights into how web technologies communicate at the protocol level.
























Leave a Comment
Your email address will not be published. Required fields are marked with *