© Webshare Proxy
payment methods
A proxy server’s job is to be an intermediary between a client and a destination server. It basically handles requests on behalf of the client, keeping the operator's network private. In proxy-dependent Node.js applications, proxies are essential for managing network traffic, improving security, and bypassing network restrictions. Jump straight to your preferred method of choice:
If you do not have proxies, feel free to use Webshare’s 10 free proxies - sign up here.
This article will explore five most common methods for setting up proxies in Node.js, along with advanced configurations, to help developers optimize their applications for performance and scalability.
Below, we have listed five common methods for setting up proxies in Node.js, each suited for different use cases.
The http-proxy library is a Node.js module designed to create proxy servers for handling HTTP and HTTPS requests. Its lightweight design and reliable feature set make it a popular choice for implementing custom proxy solutions. By using http-proxy, developers can intercept and manipulate requests and responses to enable scenarios like load balancing, caching, or secure API access.
Follow these steps to create an HTTP proxy server using http-proxy.
1) Install the http-proxy library.
npm install http-proxy2) Set up the server: Import the library and configure your proxy to route traffic through Webshare proxies.
3) Run the server: Use a code snippet like the one below to start the proxy.
const http = require('http');
const httpProxy = require('http-proxy');
const proxy = httpProxy.createProxyServer({
target: 'https://www.randomcompanyapi.com', // Target URL
changeOrigin: true,
proxyTimeout: 5000, // Adjust timeout as needed
headers: {
'Proxy-Authorization': 'Basic ' + Buffer.from('username:password').toString('base64')
}
});
const server = http.createServer((req, res) => {
proxy.web(req, res, { target: 'https://www.randomcompanyapi.com' });
});
server.listen(8000, () => {
console.log('HTTP Proxy Server running on port 8000');
});HTTP proxies are most effective for the following.
By using http-proxy and Webshare proxies, you can implement a reliable, high-performance HTTP proxy server customized to your needs.
The SOCKS proxy is an intermediary at the transport layer level, which supports practically every form of network traffic, ranging from HTTP and HTTPS to FTP and more. While HTTP proxies only work with web traffic, SOCKS provides far greater flexibility and added anonymity that makes it highly suitable in implementations where safe data transfer, bypassing restrictions in firewalls, P2P handling and streaming are involved. Furthermore, they provide faster data transmission and fewer restrictions compared to HTTP proxies.
To begin setting up a SOCKS proxy, first install the socks library.
npm install socksNext, configure the proxy with SOCKS proxy details.
const socks = require('socks');
const net = require('net');
const proxyOptions = {
proxy: {
ipaddress: '111.222.333.44', // Proxy IP
port: 1080, // SOCKS proxy port
type: 5, // SOCKS version
userId: 'username', // Proxy IP username
password: 'password' // Proxy IP password
},
target: {
host: 'randomcompanyapi.com', // Target server
port: 80 // Target server port
}
};
socks.createConnection(proxyOptions, (err, socket) => {
if (err) {
console.error('Error connecting through SOCKS proxy:', err);
return;
}
console.log('Connected to target through SOCKS proxy.');
socket.write('GET / HTTP/1.1\r\nHost: 'randomcompanyapi.com\r\n\r\n');
socket.on('data', (data) => {
console.log('Received:', data.toString());
socket.end();
});
});Node-fetch is a library for Node.js applied in executing HTTP requests, with its interface rather similar to the Fetch browser API. It is often utilized for tasks such as API calls and web scraping. Using proxies with it increases its ability to bypass geo-restrictions, evade rate limits, and ensure anonymity.
Now, to use proxies with node-fetch, you must integrate it with proxies like http-proxy-agent and socks-proxy-agent, depending on whether it's an HTTP or a socks proxy.
First, install the required packages.
npm install node-fetch http-proxy-agent socks-proxy-agentNext, configure the proxy agent for HTTP or SOCKS proxies. Here’s an example using Webshare’s HTTP proxy.
const fetch = require('node-fetch');
const { HttpProxyAgent } = require('http-proxy-agent');
const proxy = new HttpProxyAgent('http://username:password@proxy.webshare.io:8080');
fetch('https://jsonplaceholder.typicode.com/posts', { agent: proxy })
.then(response => response.json())
.then(data => console.log(data))
.catch(error => console.error('Error:', error));Express.js is a popular Node.js framework for building web applications and APIs because it's flexible and easy to use. With the addition of the http-proxy-middleware library, Express.js can integrate proxy features like routing, logging, and load balancing. This setup allows you to forward requests through a proxy server while improving security, performance, and scalability.
To set up proxy middleware in an Express.js app, first install the necessary dependencies.
npm install express http-proxy-middlewareNext, integrate the proxy middleware to route requests through a Webshare proxy.
const express = require('express');
const { createProxyMiddleware } = require('http-proxy-middleware');
const app = express();
const options = {
target: 'https://www.randomcompanyapi.com', // Target server
changeOrigin: true,
headers: {
'Proxy-Authorization': 'Basic ' + Buffer.from('username:password').toString('base64'),
},
onProxyReq: (proxyReq, req, res) => {
console.log(`Proxying request to: ${req.url}`);
},
};
app.use('/api', createProxyMiddleware(options));
app.listen(3000, () => {
console.log('Server running on http://localhost:3000');
});This setup proxies requests to /api through Webshare to the target server. Common use cases include web scraping. Using this method, developers can build efficient and secure backend solutions.
Koa.js is a minimalist Node.js framework designed for building web applications and APIs with increased flexibility. Unlike Express.js, it has a lightweight core that focuses on middleware, which improves performance and error handling with async/await. When paired with the koa-proxies library, Koa.js becomes a very useful tool for integrating proxy capabilities, ideal for scenarios such as API forwarding, web scraping, and load balancing.
To set up proxies in Koa.js, begin by installing the necessary dependencies.
npm install koa koa-proxiesThen, configure the proxy middleware.
const Koa = require('koa');
const proxy = require('koa-proxies');
const app = new Koa();
app.use(
proxy('/api', {
target: 'https://www.randomcompanyapi.com', // Target server
changeOrigin: true,
headers: {
'Proxy-Authorization': 'Basic ' + Buffer.from('username:password').toString('base64'),
},
logs: true, // Enable logging for debugging
})
);
app.listen(4000, () => {
console.log('Koa server running on http://localhost:4000');
});This setup proxies any requests to /api through Webshare to the target server. Koa's lightweight design ensures low overhead and is therefore perfect for microservices, custom proxy features like logging and rate limiting, and high-performance web scraping.
Advanced proxy configurations can easily improve your Node.js application's performance, security, and efficiency. This section explores three essential techniques, proxy rotation, configuring request headers, and integrating caching.
Proxy rotation is a good technique for staying anonymous and keeping away from detection, especially when dealing with web scraping or frequent requests to other services. By rotating the proxies for each request, you avoid IP bans and anti-bot measures. Webshare.io offers proxy pools that make rotating proxies simple and efficient.
Example:
const proxies = [
'http://username:password@proxy1.webshare.io:8080',
'http://username:password@proxy2.webshare.io:8080',
'http://username:password@proxy3.webshare.io:8080',
];
async function fetchWithRotatingProxy(url) {
const proxyUrl = proxies[Math.floor(Math.random() * proxies.length)];
const { HttpProxyAgent } = require('http-proxy-agent');
const proxyAgent = new HttpProxyAgent(proxyUrl);
const response = await fetch(url, { agent: proxyAgent });
const data = await response.json();
console.log(data);
}Properly configured headers are important in order for your requests to seem more human-like and not to get your requests flagged as bots. Custom headers like User-Agent, Accept-Language, and Connection help simulate browser-like requests to make your proxy requests less detectable.
Example:
const headers = {
'User-Agent': 'Mozilla/5.0...',
'Accept-Language': 'en-US,en;q=0.9',
'Accept-Encoding': 'gzip, deflate, br',
'Connection': 'keep-alive',
};
fetch('https://jsonplaceholder.typicode.com/posts', { headers, agent: proxyAgent })
.then(response => response.json())
.then(data => console.log(data));Caching is a powerful method to improve performance and reduce the number of API calls. By storing previously fetched responses, you can significantly reduce the load on external servers and minimize API costs. Caching is particularly useful for data that doesn’t change often.
Example:
const cache = new Map();
async function fetchWithCache(url) {
if (cache.has(url)) {
console.log('Serving from cache...');
return cache.get(url);
}
const response = await fetch(url, { agent: proxyAgent });
const data = await response.json();
cache.set(url, data);
return data;
}You can use these proxy server setups to optimize your Node.js apps for scalability, security, and performance.
In this article, we’ve explored various methods to set up proxies in Node.js, with libraries like http-proxy, socks, node-fetch, and frameworks such as Express.js and Koa.js. These methods offer flexibility for different use cases, from web scraping to handling API requests. Advanced configurations like proxy rotation, custom headers, and caching can further improve performance and security. Webshare.io stands out as a reliable and affordable proxy provider, which offers scalable solutions for developers. By using Webshare’s proxies, you can efficiently manage large-scale proxy needs while optimizing your Node.js applications.
What is Puppeteer? Popular Node.js Library Explained
