Proxy with Python Requests: 3 Setup Methods Explained

Basic knowledge of Python and Requests library

• Ensure you have Python installed on your system. You can download it from the official Python website.
• The Requests library should be installed. If not, you can install it using pip:

pip install requests

Understanding of proxies

A proxy server acts as an intermediary between your client and the server you want to communicate with. It can help you mask your IP address, manage request limits, and access geographically restricted content.

Proxy server information

To use proxies effectively, you need the following information about the proxy server:

• Proxy Server Address: The proxy server address consists of the IP address and the port number of the proxy server. This address is crucial as it determines the server through which your requests will be routed.

• IP Address: This is a unique address that identifies the proxy server on the network.some text
  • Example: 123.456.789.012

• Port Number: This is a numerical identifier in networking used to distinguish different services or processes on the proxy server.some text
  • Example: 8080

Together, the IP address and port number form the proxy server address:

• Example: http://123.456.789.012:8080

• Authentication Details (if required): Some proxy servers require authentication to ensure that only authorized users can use their services. If your proxy server requires authentication, you'll need to provide:

• Username: The account identifier for the user authorized to access the proxy.some text
  • Example: myUsername
• Password: The secret key associated with the username to authenticate the user.some text
  • Example: myPassword

When combined with the proxy server address, the format for an authenticated proxy URL is:

• Example: http://myUsername:myPassword@123.456.789.012:8080

Proxy Type: Proxies come in different types, each suited for specific kinds of network traffic and security needs. Understanding the type of proxy your server supports is essential for proper configuration.

• HTTP Proxy:some text
  • Usage: Suitable for standard HTTP requests. It is commonly used for accessing websites and web scraping.
  • Example: http://123.456.789.012:8080
• HTTPS Proxy:some text
  • Usage: Supports encrypted (HTTPS) traffic, making it suitable for accessing secure websites.
  • Example: https://123.456.789.012:8080
• SOCKS5 Proxy:some text
  • Usage: Offers more flexibility and can handle various types of traffic, including HTTP, HTTPS, FTP, and more.
  • Example: socks5://123.456.789.012:1080

Method 1: Using static proxies

Using a static proxy in Python Requests is one of the simplest and most common methods. A static proxy is a single proxy server through which all your requests are routed. This method is particularly useful for straightforward use cases where you don’t need to switch proxies frequently. Let’s explore how to use both HTTP and HTTPS proxies, including how to handle proxy authentication.

import requests

# Define the proxy server addresses
proxy = {
    "http": "http://123.456.789.012:8080",
    "https": "https://123.456.789.012:8080"
}

# Function to make a request through the proxy and print the response
def fetch_ip(url):
    response = requests.get(url, proxies=proxy)
    print(response.json())

# Make requests through the proxy
fetch_ip("http://httpbin.org/ip")
fetch_ip("https://httpbin.org/ip")

import requests

# Define the proxy server addresses with authentication
proxy = {
    "http": "http://username:password@123.456.789.012:8080",
    "https": "https://username:password@123.456.789.012:8080"
}

# Function to make a request through the authenticated proxy and print the response
def fetch_ip(url):
    response = requests.get(url, proxies=proxy)
    print(response.json())

# Make requests through the authenticated proxy
fetch_ip("http://httpbin.org/ip")
fetch_ip("https://httpbin.org/ip")

import os
import requests

# Set environment variables for the proxy
os.environ["HTTP_PROXY"] = "http://123.456.789.012:8080"
os.environ["HTTPS_PROXY"] = "https://123.456.789.012:8080"

# Make a request through the proxy using Requests (it will automatically use the environment variables)
response = requests.get("http://httpbin.org/ip")
print(response.json())

response = requests.get("https://httpbin.org/ip")
print(response.json())

Method 2: Using a proxy list

In scenarios where you need to distribute requests across multiple proxies to avoid rate limits or IP bans, using a proxy list is an effective strategy. By manually changing proxies using a proxy list configuration, you can cycle through different proxies to ensure a more balanced and less detectable request pattern. This method is especially useful for large-scale web scraping and data extraction tasks.

# List of proxies
proxy_list = [
    "http://123.456.789.012:8080",
    "http://234.567.890.123:8080",
    "http://345.678.901.234:8080"
]

import random

def get_proxy():
    return random.choice(proxy_list)

import requests

# List of URLs to scrape
urls = [
    "http://httpbin.org/ip",
    "http://httpbin.org/user-agent",
    "http://httpbin.org/headers"
]

for url in urls:
    # Get a random proxy from the list
    proxy = get_proxy()
    
    # Define the proxy dictionary
    proxies = {
        "http": proxy,
        "https": proxy
    }
    
    # Make a request through the proxy
    response = requests.get(url, proxies=proxies)
    
    # Print the response
    print(f"Using proxy {proxy}")
    print(response.json())
    print("\n")

# List of proxies with authentication
proxy_list = [
    "http://username:password@123.456.789.012:8080",
    "http://username:password@234.567.890.123:8080",
    "http://username:password@345.678.901.234:8080"
]

def get_proxy():
    return random.choice(proxy_list)

# Rest of the code remains the same

Method 3: Using rotating residential proxies

Rotating residential proxies provide a powerful solution for web scraping and data extraction tasks where high anonymity and avoiding IP bans are crucial. Residential proxies route your requests through real residential IP addresses, making them less likely to be detected and blocked by websites. This method involves using a pool of residential proxies that rotate automatically, ensuring each request comes from a different IP address.

Using rotating residential proxies can be more complex and typically involves subscribing to a proxy service that provides the necessary infrastructure. Let’s explore how to implement this method in Python Requests.

import requests

# Example configuration from a residential proxy service
proxy = {
    "http": "http://username:password@proxy-service.com:12345",
    "https": "http://username:password@proxy-service.com:12345"
}

# URL to scrape
url = "http://httpbin.org/ip"

# Make a request through the rotating residential proxy
response = requests.get(url, proxies=proxy)

# Print the response
print(response.json())

import requests
import time

# Proxy service configuration
proxy_url = "http://username:password@proxy-service.com:12345"

# Function to make a request through the proxy
def fetch(url):
    proxies = {
        "http": proxy_url,
        "https": proxy_url
    }
    try:
        response = requests.get(url, proxies=proxies, timeout=10)
        return response.json()
    except requests.exceptions.RequestException as e:
        print(f"Request failed: {e}")
        return None

# List of URLs to scrape
urls = [
    "http://httpbin.org/ip",
    "http://httpbin.org/user-agent",
    "http://httpbin.org/headers"
]

# Iterate over the URLs and make requests
for url in urls:
    result = fetch(url)
    if result:
        print(result)
    time.sleep(2)  # Adding delay to avoid too many requests in a short time

Modifying request HTTP headers

Modifying HTTP headers can help you emulate real browser behavior and avoid detection by target websites. Common headers to modify include User-Agent, Referer, and Accept-Language.

Here’s an example of how to modify these headers in your requests:

import requests

# Proxy configuration
proxy = {
    "http": "http://username:password@proxy-service.com:12345",
    "https": "http://username:password@proxy-service.com:12345"
}

# Custom headers
headers = {
    "User-Agent": "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/91.0.4472.124 Safari/537.36",
    "Referer": "https://www.google.com",
    "Accept-Language": "en-US,en;q=0.9"
}

# URL to scrape
url = "http://httpbin.org/headers"

# Make a request through the proxy with custom headers
response = requests.get(url, proxies=proxy, headers=headers)

# Print the response
print(response.json())

In this example:

• The headers dictionary contains custom HTTP headers to mimic a real browser request.
• The requests.get method sends a GET request through the proxy with the specified headers.

Handling sessions with proxies

Using sessions in combination with proxies can maintain state across multiple requests, such as handling cookies or reusing TCP connections for improved performance.

Here’s how to use a session with a proxy:

import requests

# Proxy configuration
proxy = {
    "http": "http://username:password@proxy-service.com:12345",
    "https": "http://username:password@proxy-service.com:12345"
}

# Create a session
session = requests.Session()

# Set the proxy for the session
session.proxies.update(proxy)

# Set custom headers for the session
session.headers.update({
    "User-Agent": "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/91.0.4472.124 Safari/537.36",
    "Referer": "https://www.google.com",
    "Accept-Language": "en-US,en;q=0.9"
})

# URL to scrape
url = "http://httpbin.org/headers"

# Make a request through the session
response = session.get(url)

# Print the response
print(response.json())

# Close the session when done
session.close()

In this script:

• A requests.Session object is created and configured with the proxy and custom headers.
• The session maintains these settings across multiple requests, providing efficiency and consistency.
• The session is closed after use to free up resources.

Incorrect proxy format

Using an incorrect proxy format can lead to connection errors. Ensure your proxy URL follows the correct format:

• For HTTP/HTTPS proxies:some text
  • Without authentication: http://IP:PORT
  • With authentication: http://username:password@IP:PORT

proxy = {
    "http": "http://123.456.789.012:8080",
    "https": "https://123.456.789.012:8080"
}

Verify the correct format to avoid syntax errors.

Authentication issues

Authentication issues arise when incorrect credentials are provided or when proxies require authentication but none is provided. These issues can manifest through various error codes. Understanding these error codes and their solutions can help troubleshoot and resolve authentication problems effectively.

Incorrect proxy types

Using the wrong proxy type (HTTP vs. SOCKS5) can result in connection failures and hinder the effectiveness of your web scraping efforts. It's crucial to understand the differences between these proxy types to ensure seamless operation.

import requests

proxy = {
    "http": "socks5://username:password@123.456.789.012:1080",
    "https": "socks5://username:password@123.456.789.012:1080"
}

try:
    response = requests.get("http://httpbin.org/ip", proxies=proxy)
    print(response.json())
except requests.exceptions.RequestException as e:
    print(f"Request error: {e}")