The Complete Guide to cURL Command: Mastering HTTP Requests from Command Line

The Complete Guide to cURL Command: Mastering HTTP Requests from Command Line

Introduction to cURL

cURL (Client URL) is a command-line tool and library for transferring data with URLs. First released in 1997, it has become the standard for making HTTP requests from terminal environments across all operating systems. cURL supports a wide range of protocols including HTTP, HTTPS, FTP, SFTP, SCP, and many others, making it an essential tool for developers, system administrators, and IT professionals.

Installation and Verification

Linux

Most Linux distributions include cURL by default. To verify installation and check version:

curl --version

If not installed:

# Ubuntu/Debian
sudo apt update && sudo apt install curl

# RHEL/CentOS
sudo yum install curl

# Fedora
sudo dnf install curl

# Arch Linux
sudo pacman -S curl

macOS

cURL comes pre-installed on macOS. For the latest version via Homebrew:

# Check current version
curl --version

# Install/upgrade via Homebrew
brew install curl

Windows

Windows 10/11: Built into Command Prompt and PowerShell (build 17063+)

Manual Installation:

  1. Download from official website: https://curl.se/windows/
  2. Extract to a directory
  3. Add to PATH environment variable

Package Managers:

# Chocolatey
choco install curl

# Scoop
scoop install curl

# Winget
winget install curl

Git Bash: Includes cURL out of the box

Basic Command Structure

The fundamental syntax of cURL is:

curl [options] [URL]

Testing with Free APIs

We’ll use several free APIs for demonstration:

  1. JSONPlaceholder – Fake REST API for testing
  2. httpbin – HTTP request and response service
  3. Postman Echo – API for testing requests

Basic HTTP Operations

GET Requests (Default)

# Simple GET request
curl https://jsonplaceholder.typicode.com/posts

# GET with specific resource
curl https://jsonplaceholder.typicode.com/posts/1

# Multiple GET requests
curl https://jsonplaceholder.typicode.com/posts/1 https://jsonplaceholder.typicode.com/posts/2

Saving Output to Files

# Save to file with specified name (all platforms)
curl -o output.json https://jsonplaceholder.typicode.com/posts

# Save with remote file name (all platforms)
curl -O https://jsonplaceholder.typicode.com/posts

# Append to existing file
curl -a https://jsonplaceholder.typicode.com/posts/1 >> existing_file.json

Viewing Response Details

# Include HTTP headers in output
curl -i https://jsonplaceholder.typicode.com/posts/1

# Display only headers
curl -I https://jsonplaceholder.typicode.com/posts/1

# Verbose mode (shows connection details)
curl -v https://jsonplaceholder.typicode.com/posts/1

# Silent mode (no progress meter)
curl -s https://jsonplaceholder.typicode.com/posts/1

# Silent but show errors
curl -sS https://jsonplaceholder.typicode.com/posts/1

HTTP Methods

POST Requests

# Basic POST
curl -X POST https://jsonplaceholder.typicode.com/posts

# POST with data
curl -X POST https://jsonplaceholder.typicode.com/posts \
  -d "title=Test&body=Content&userId=1"

# POST with JSON data
curl -X POST https://jsonplaceholder.typicode.com/posts \
  -H "Content-Type: application/json" \
  -d '{"title": "Test Post", "body": "This is content", "userId": 1}'

PUT Requests

curl -X PUT https://jsonplaceholder.typicode.com/posts/1 \
  -H "Content-Type: application/json" \
  -d '{"id": 1, "title": "Updated Title", "body": "Updated content", "userId": 1}'

PATCH Requests

curl -X PATCH https://jsonplaceholder.typicode.com/posts/1 \
  -H "Content-Type: application/json" \
  -d '{"title": "Partially Updated Title"}'

DELETE Requests

curl -X DELETE https://jsonplaceholder.typicode.com/posts/1

Working with Headers

Custom Request Headers

# Single header
curl -H "Authorization: Bearer token123" https://api.example.com/data

# Multiple headers
curl -H "Content-Type: application/json" \
     -H "Authorization: Bearer token123" \
     -H "X-Custom-Header: Value" \
     https://api.example.com/data

# Remove default header
curl -H "Host:" https://example.com

Viewing Request Headers in Verbose Mode

curl -v https://jsonplaceholder.typicode.com/posts/1

Authentication Methods

Basic Authentication

# With username and password
curl -u username:password https://api.example.com/protected

# Password prompt (hides password in command history)
curl -u username https://api.example.com/protected

Bearer Token Authentication

curl -H "Authorization: Bearer eyJhbGciOiJIUzI1NiIsInR5cCI6IkpXVCJ9" \
     https://api.example.com/data

Digest Authentication

curl --digest -u username:password https://api.example.com/protected

File Operations

Uploading Files with POST

Linux and macOS:

# Single file upload
curl -X POST https://httpbin.org/post \
  -F "file=@/path/to/local/file.txt"

# Multiple files
curl -X POST https://httpbin.org/post \
  -F "file1=@/path/to/file1.txt" \
  -F "file2=@/path/to/file2.jpg"

# File with additional form fields
curl -X POST https://httpbin.org/post \
  -F "document=@/path/to/document.pdf" \
  -F "description=Quarterly Report" \
  -F "category=finance"

Windows:

:: Command Prompt
curl -X POST https://httpbin.org/post ^
  -F "file=@C:\Users\Username\Documents\file.txt"

:: With additional fields
curl -X POST https://httpbin.org/post ^
  -F "file=@C:\Users\Username\Documents\file.txt" ^
  -F "description=Test upload"

Windows PowerShell:

# PowerShell
curl -X POST https://httpbin.org/post `
  -F "file=@C:\Users\Username\Documents\file.txt" `
  -F "description=Test upload"

Windows Git Bash (same as Linux):

curl -X POST https://httpbin.org/post \
  -F "file=@/c/Users/Username/Documents/file.txt" \
  -F "description=Test upload"

Binary File Upload

# Upload raw binary data
curl -X POST https://httpbin.org/post \
  -H "Content-Type: application/octet-stream" \
  --data-binary @/path/to/file.bin

Downloading Files

# Download and save with remote filename
curl -O https://example.com/files/document.pdf

# Download with custom filename
curl -o mydocument.pdf https://example.com/files/document.pdf

# Resume interrupted download
curl -C - -O https://example.com/large-file.zip

# Download with progress bar
curl --progress-bar -O https://example.com/large-file.zip

Data Formats and Processing

JSON Data

# Send JSON data
curl -X POST https://jsonplaceholder.typicode.com/posts \
  -H "Content-Type: application/json" \
  -d '{
    "title": "Test Post",
    "body": "This is the content",
    "userId": 1
  }'

# Pretty print JSON response (Linux/macOS with jq)
curl -s https://jsonplaceholder.typicode.com/posts/1 | jq .

# Windows PowerShell alternative
curl -s https://jsonplaceholder.typicode.com/posts/1 | ConvertFrom-Json

# Format JSON with Python (cross-platform)
curl -s https://jsonplaceholder.typicode.com/posts/1 | python -m json.tool

URL-Encoded Form Data

curl -X POST https://httpbin.org/post \
  -H "Content-Type: application/x-www-form-urlencoded" \
  -d "username=johndoe&password=secret123&action=login"

XML Data

curl -X POST https://httpbin.org/post \
  -H "Content-Type: application/xml" \
  -d '<?xml version="1.0"?><note><to>User</to><from>Admin</from></note>'

Advanced Features

Following Redirects

# Automatically follow redirects
curl -L https://httpbin.org/redirect/2

# Maximum number of redirects
curl -L --max-redirs 5 https://example.com/redirect

Rate Limiting

# Limit download speed (bytes per second)
curl --limit-rate 100K -O https://example.com/largefile.zip

# Limit upload speed
curl --limit-rate 1M -T largefile.zip ftp://example.com/

Timeouts and Retries

# Connection timeout (seconds)
curl --connect-timeout 10 https://example.com

# Maximum time for operation
curl --max-time 30 https://example.com

# Retry on failure
curl --retry 3 https://example.com

# Retry with delay
curl --retry 3 --retry-delay 5 https://example.com

Cookie Management

# Send cookies
curl -b "session=abc123; user=johndoe" https://example.com/dashboard

# Save cookies to file
curl -c cookies.txt https://example.com/login

# Load cookies from file
curl -b cookies.txt https://example.com/dashboard

# Combine save and load
curl -c cookies.txt -b cookies.txt https://example.com/session

SSL/TLS Configuration

# Skip SSL certificate verification (NOT recommended for production)
curl -k https://example.com

# Specify SSL version
curl --tlsv1.2 https://example.com

# Use specific certificate
curl --cert client.crt --key client.key https://secure.example.com

# Specify CA bundle
curl --cacert /path/to/ca-bundle.pem https://secure.example.com

Compression

# Request compressed response
curl -H "Accept-Encoding: gzip, deflate" https://example.com/data

# Compress request data
curl --compressed -X POST https://example.com/api -d @data.json

Platform-Specific Considerations

Windows Path Handling

:: Command Prompt - Use caret for line continuation
curl -X POST https://httpbin.org/post ^
  -H "Content-Type: application/json" ^
  -d "{\"name\":\"test\"}"

:: Paths with spaces require quotes
curl -F "file=@\"C:\Users\My Name\file.txt\"" https://httpbin.org/post
# PowerShell - Use backtick for line continuation
curl -X POST https://httpbin.org/post `
  -H "Content-Type: application/json" `
  -d '{"name":"test"}'

# Escaping quotes in PowerShell
curl -X POST https://httpbin.org/post `
  -H "Content-Type: application/json" `
  -d '{\"name\":\"test\"}'

Linux/macOS Shell Features

# Using variables
API_URL="https://jsonplaceholder.typicode.com"
ENDPOINT="/posts"
curl "${API_URL}${ENDPOINT}"

# Reading data from file
curl -X POST https://httpbin.org/post \
  -H "Content-Type: application/json" \
  -d @data.json

# Using heredoc for multiline JSON
curl -X POST https://jsonplaceholder.typicode.com/posts \
  -H "Content-Type: application/json" \
  -d @- << EOF
{
  "title": "Heredoc Test",
  "body": "Content from heredoc",
  "userId": 1
}
EOF

Practical Examples and Scripts

Example 1: API Health Check Script

#!/bin/bash
# Linux/macOS API health check

API_ENDPOINTS=(
    "https://jsonplaceholder.typicode.com/posts"
    "https://jsonplaceholder.typicode.com/comments"
    "https://jsonplaceholder.typicode.com/albums"
)

echo "API Health Check"
echo "================"

for endpoint in "${API_ENDPOINTS[@]}"; do
    echo -n "Testing $endpoint... "
    http_code=$(curl -s -o /dev/null -w "%{http_code}" "$endpoint")

    if [ "$http_code" -eq 200 ]; then
        echo "OK ($http_code)"
    else
        echo "FAILED ($http_code)"
    fi
done

Example 2: Batch File Upload Script

#!/bin/bash
# Cross-platform file upload script

UPLOAD_URL="https://httpbin.org/post"
UPLOAD_DIR="./uploads"

echo "Batch File Upload"
echo "================="

# Platform-specific directory creation
if [[ "$OSTYPE" == "msys" ]] || [[ "$OSTYPE" == "win32" ]]; then
    # Windows
    if not exist "%UPLOAD_DIR%" mkdir "%UPLOAD_DIR%"
    UPLOAD_DIR="C:\Users\%USERNAME%\Documents\uploads"
else
    # Linux/macOS
    mkdir -p "$UPLOAD_DIR"
fi

# Find and upload files
find "$UPLOAD_DIR" -name "*.txt" -type f | while read -r file; do
    echo "Uploading: $(basename "$file")"

    if [[ "$OSTYPE" == "msys" ]] || [[ "$OSTYPE" == "win32" ]]; then
        # Windows syntax
        curl -X POST "$UPLOAD_URL" ^
            -F "file=@\"$file\"" ^
            -F "timestamp=$(date +%s)" > /dev/null 2>&1
    else
        # Linux/macOS syntax
        curl -X POST "$UPLOAD_URL" \
            -F "file=@$file" \
            -F "timestamp=$(date +%s)" > /dev/null 2>&1
    fi

    if [ $? -eq 0 ]; then
        echo "  Success"
    else
        echo "  Failed"
    fi
done

Example 3: Download Manager with Progress

#!/bin/bash
# Download manager script

DOWNLOADS=(
    "https://httpbin.org/image/jpeg"
    "https://httpbin.org/image/png"
    "https://httpbin.org/xml"
)

DOWNLOAD_DIR="./downloads"

# Create download directory
mkdir -p "$DOWNLOAD_DIR"

echo "Starting downloads..."
echo "---------------------"

for url in "${DOWNLOADS[@]}"; do
    filename=$(basename "$url")
    output_path="$DOWNLOAD_DIR/$filename"

    echo "Downloading: $filename"

    # Download with progress bar
    if [[ "$OSTYPE" == "darwin"* ]] || [[ "$OSTYPE" == "linux-gnu"* ]]; then
        # Linux/macOS with progress bar
        curl --progress-bar -L -o "$output_path" "$url"
    else
        # Windows (Git Bash supports same syntax)
        curl --progress-bar -L -o "$output_path" "$url"
    fi

    # Check file size
    if [[ "$OSTYPE" == "darwin"* ]] || [[ "$OSTYPE" == "linux-gnu"* ]]; then
        size=$(stat -f%z "$output_path" 2>/dev/null || stat -c%s "$output_path")
    else
        # Windows compatibility
        size=$(wc -c < "$output_path")
    fi

    echo "  Size: $size bytes"
    echo "  Saved to: $output_path"
    echo
done

echo "All downloads completed."

Debugging and Troubleshooting

Verbose Output Analysis

# Complete verbose output
curl -v https://httpbin.org/get

# Save verbose output to file
curl -v https://httpbin.org/get 2> debug.log

# Trace detailed communication
curl --trace trace.txt https://httpbin.org/get
curl --trace-ascii trace_ascii.txt https://httpbin.org/get

Performance Testing

# Timing analysis
curl -w "
Time Summary:
------------
Name lookup:    %{time_namelookup}
Connect:        %{time_connect}
App connect:    %{time_appconnect}
Pre-transfer:   %{time_pretransfer}
Start transfer: %{time_starttransfer}
Total:          %{time_total}
" https://httpbin.org/delay/2

# Speed test
curl -w "
Download Speed: %{speed_download} bytes/sec
" -o /dev/null https://httpbin.org/stream-bytes/1000000

Error Handling

# Continue on error (for scripts)
curl --fail-with-body https://httpbin.org/status/404

# Show HTTP errors on stderr
curl --show-error https://httpbin.org/status/500

# Retry logic
curl --retry 3 --retry-delay 2 --retry-max-time 30 https://example.com

Integration with Other Tools

cURL with grep (Linux/macOS)

# Search in response
curl -s https://jsonplaceholder.typicode.com/posts | grep -i "voluptatem"

# Extract specific data
curl -s https://jsonplaceholder.typicode.com/posts/1 | grep -o '"title":"[^"]*"'

cURL with awk (Linux/macOS)

# Process response data
curl -s https://jsonplaceholder.typicode.com/posts | awk -F'"' '/"title":/ {print $4}'

cURL in Python Scripts

import subprocess
import json

# Execute cURL from Python
result = subprocess.run(
    ['curl', '-s', 'https://jsonplaceholder.typicode.com/posts/1'],
    capture_output=True,
    text=True
)

if result.returncode == 0:
    data = json.loads(result.stdout)
    print(f"Title: {data['title']}")

cURL in Batch Scripts (Windows)

@echo off
:: Windows batch script with cURL

set API_URL=https://jsonplaceholder.typicode.com/posts/1

echo Making API request...
curl -s -o response.json %API_URL%

if %errorlevel% equ 0 (
    echo Request successful
    type response.json
) else (
    echo Request failed with error: %errorlevel%
)

Security Best Practices

  1. Never store credentials in scripts
   # BAD: Hardcoded credentials
   curl -u admin:password123 https://api.example.com

   # GOOD: Use environment variables
   curl -u "$API_USER:$API_PASS" https://api.example.com

   # BETTER: Use credential files or keychain
   curl --netrc-file ~/.netrc https://api.example.com
  1. Validate SSL certificates
   # Avoid -k flag in production
   curl --cacert /path/to/cert.pem https://secure.example.com
  1. Sanitize input in scripts
   # Validate URLs before using
   if [[ "$URL" =~ ^https:// ]]; then
       curl "$URL"
   else
       echo "Invalid URL"
   fi
  1. Use secure protocols
   # Always prefer HTTPS over HTTP
   curl https://secure.example.com  # GOOD
   curl http://example.com           # AVOID

Common Use Cases

1. API Development and Testing

  • Rapid endpoint testing during development
  • Automated API health checks
  • Load testing and performance monitoring

2. Web Scraping and Data Collection

  • Download web pages for analysis
  • Extract data from APIs
  • Monitor website changes

3. File Transfer Operations

  • Upload files to servers
  • Download logs and reports
  • Backup and restore operations

4. System Administration

  • Service health monitoring
  • Configuration management
  • Automated deployments

5. Network Troubleshooting

  • Testing connectivity
  • Verifying firewall rules
  • Debugging HTTP issues

Conclusion

cURL is an indispensable tool in the modern developer’s toolkit. Its versatility across platforms, support for numerous protocols, and powerful feature set make it suitable for a wide range of tasks from simple API testing to complex automated workflows. By mastering cURL, you gain the ability to interact with web services and transfer data efficiently from the command line, regardless of your operating system.

The key to effective cURL usage lies in understanding its options, practicing with different scenarios, and integrating it into your daily workflows. Start with basic requests, gradually incorporate more advanced features, and soon you’ll find cURL to be one of your most frequently used tools.

Remember that with command-line tools comes responsibility. Always respect the terms of service of the APIs and websites you interact with, implement proper error handling in your scripts, and prioritize security in all your operations.

Further Resources

  • Official Documentation: https://curl.se/docs/
  • cURL Book: https://everything.curl.dev/
  • HTTP Status Codes: https://developer.mozilla.org/en-US/docs/Web/HTTP/Status
  • RFC Standards: https://datatracker.ietf.org/doc/html/rfc7231 (HTTP/1.1)
  • Practice APIs:
  • JSONPlaceholder: https://jsonplaceholder.typicode.com/
  • httpbin: https://httpbin.org/
  • Postman Echo: https://postman-echo.com/

By incorporating cURL into your development and administration workflows, you’ll work more efficiently and gain deeper insights into how web technologies communicate at the protocol level.

Posts Carousel

Leave a Comment

Your email address will not be published. Required fields are marked with *

Latest Posts

Most Commented

Featured Videos