IN THIS ARTICLE

Updated on

March 25, 2024

User-Agent in Puppeteer: Random & Custom Methods Explained

When it comes to web scraping, automated testing, performance monitoring, capturing screenshots, and headless browsing with Puppeteer, one crucial aspect often neglected is user-agent. The user-agent is a piece of information sent by a web browser to the web server. It identifies not just the browser and its version, but also often includes details about the device's operating system, type, and sometimes even model.

This information plays a significant role in determining how a website responds to a request. While Puppeteer allows for the setting of a custom user-agent string, if one wants to switch between random user-agent strings, one would need to implement their own logic or use an external library.

In this article, we’ll dive deeper into the user-agent manipulation in Puppeteer. We will discuss the reasoning behind choosing a custom user-agent, how to implement best practices, and discuss some of the common challenges faced when working with user-agents in Puppeteer.

Should you choose a random or custom user-agent?

Choosing a random or custom user-agent depends on the specific use cases and tester goals.

For example in the case of web scraping, if you do not rotate the user-agent information, the target site can detect all the requests are coming from the same device. It's possible to fake that information by sending a valid user-agent but different agents with each request.

In another case, if you are using Puppeteer for testing automation, you may want static user-agents for easier troubleshooting of your web application.

How Do Random User-Agents Behave?

Using random user-agent helps you maintain anonymity during web scrapping or automation because when you use random user-agent strings in Puppeteer for each request, it means you are constantly changing the user-agent header with each request.

This lack of a consistent signal can confuse websites that attempt to identify automated traffic based on user-agent patterns.

However, when you use a different user-agent for each request in your web scrapping or automation script, it can lead to different responses from the website, which can make the behavior of your automation less consistent and harder to predict.

How Do Custom User-Agents Behave?

With a custom user-agent, you have complete control over the user agent’s header content. You can include specific information about the browser, operating system, and even additional details that mimic a real user’s browsing environment. This control allows you to make your requests appear as if they are coming from a genuine browser and device.

Using custom user-agents in Puppeteer can contribute to the predictability and reliability of your automation tasks. When your automation script consistently uses the same user-agent for all the requests, websites are more likely to treat your request in a predictable manner.

For these reasons, random user agents are preferred over custom user agents for web scraping and browser automation. Customer user agents are preferred over random user agents for benchmarking, performance testing, and testing specific user agents' behavior.

How to Set Up User-Agent in Puppeteer

Now we will be setting up a random user-agent in Puppeteer. If you're experienced with Puppeteer or JavaScript, here's a full code example that you can directly copy and edit.


const puppeteer = require('puppeteer');
const { UserAgent, random } = require('user-agents');

(async () => {
  const browser = await puppeteer.launch();
  const page = await browser.newPage();

  const userAgent = new UserAgent({ deviceCategory: 'desktop' });
  const randomUserAgent = userAgent.toString();

  await page.setUserAgent(randomUserAgent);

  await page.goto('https://example.com');
  // Perform web scraping or automation actions here.

  await browser.close();
})();

This code sets a random user-agent for your Puppeteer instance, allowing you to scrape websites with anonymity and predictability. For those curious about the ins and outs of this code, we'll dive into the details in 6 easy steps below.

Install Puppeteer

First, you need to install Puppeteer in your project if you haven’t. You can install it using npm or yarn, depending on your preference.


npm install puppeteer

Install Puppeteer and Other Dependencies

In your JavaScript or TypeScript file, import all the required dependencies, including Puppeteer and a library for generating random user-agents. One popular library for setting up random user-agent is the “user-agents” package. Install it by running the following command.


npm install user-agents

Then, import all the required dependencies:


const puppeteer = require('puppeteer');
const { UserAgent, random } = require('user-agents');

Create a Puppeteer Instance

Set up an instance for the Puppeteer browser as you typically do. For instance, here is a sample code.


(async () => {
  const browser = await puppeteer.launch();
  const page = await browser.newPage();
  // Your scraping or automation code goes here.
  // ...
  await browser.close();
})();

Generate Random User-Agent

Before navigating to a web page, generate a random user-agent using the “user-agents” library. Here is how you can do it.


const userAgent = new UserAgent({ deviceCategory: 'desktop' }); // You can specify the device category

const randomUserAgent = userAgent.toString();

The ‘randomUserAgent’ variable contains a randomly chosen user-agent string that you can use in your Puppeteer requests.

Set User Agent in Puppeteer Request

Now set the random user-agent in your Puppeteer request by using the ‘page.setUserAgent()’ method.


await page.setUserAgent(randomUserAgent);

This will set the user-agent for the current web page instance.

Navigate and Perform Actions

Upon setting the random user-agent, you can navigate to the web page and perform your web scraping or automation task using Puppeteer as usual. Please refer to the full code example I have added above.

How to Set Up Custom User-Agents in Puppeteer

Setting up a custom user-agent in Puppeteer means manually specifying the user-agent string for your requests. For those familiar with Puppeteer or JavaScript, here's a full code example to quickly get you started.


const puppeteer = require('puppeteer');

(async () => {
  const browser = await puppeteer.launch();
  const page = await browser.newPage();

  // Set a custom User-Agent string
  await page.setUserAgent(
    'Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/90.0.4430.212 Safari/537.36'
  );

  await page.goto('https://example.com');
  // Write your web scraping or automation actions here.

  await browser.close();
})();

In the above code snippet, a custom user-agent string is specified using the ‘page.setUserAgent()’ method. Puppeteer will use this user-agent for all requests made by the ‘page’ instance. You can tailor the user-agent to your specific needs or emulate different browsers or devices as necessary.

For those wanting a detailed breakdown of how this code works, let's walk through the process step by step.

Install Puppeteer

Make sure you have Puppeteer installed in your project using either npm or yarn as we’ve discussed earlier.


npm install puppeteer

Import Puppeteer

In your JavaScript or TypeScript file, import the Puppeteer module.


const puppeteer = require('puppeteer');

Create a Puppeteer Instance

Setup a Puppeteer browser instance as you typically do.


(async () => {
  const browser = await puppeteer.launch();
  const page = await browser.newPage();
  // Your scraping or automation code goes here.
  // ...
  await browser.close();
})();

Specify a Custom User-Agent

Before making a request with Puppeteer, set a custom user-agent using the ‘page.setUserAgent()’ method. Replace ‘Your_Custom_User_Agent’ with the user-agent string you want to create.


await page.setUserAgent('Your_Custom_User_Agent');

For instance, if you want to mimic the user-agent of a specific browser, first find the user-agent strings for common browsers and use them. Here is an example of setting a custom user-agent to mimic Google Chrome.


await page.setUserAgent(
  'Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/90.0.4430.212 Safari/537.36'
);

UserAgentString.com provides an easy to use list of different user-agents for you to copy.

Navigate and Perform Actions

Upon setting up a custom user-agent, you can navigate to web pages and perform your web scraping or automation tasks using Puppeteer. Puppeteer will use the specified user-agent for all the requests made by that page instance.

Refer to the full code I have given above. After the following code line you can write your automation code.


await page.goto('https://example.com');

Fixing Common Issues

Setting up the user-agents in Puppeteer is a crucial step in web scraping and automation, but it often comes with multiple challenges.

Here are some common issues you might face while setting up user-agent in Puppeteer and how to avoid them.

Getting Blacklisted by Websites

The website may still detect automation, even with custom or random user-agents. This detection can lead to an IP ban or CAPTCHA challenges.

Resolution

To prevent this issue from occurring, use anti-detection techniques such as IP rotation, request rate limiting, and mimicking human-like behavior. If you are encountering CAPTCHAs too often, implement a CAPTCHA-solving solution in your automation. To get a bigger lost of IP's for rotation, using proxies in Puppeteer is recommended.

Incorrect User-Agent Format

If you use an inappropriate user-agent format, websites might flag it as suspicious.

Resolution

Ensure your user-agent string adheres to the format of the website or device you are trying to emulate. You can easily find legitimate user-agent strings for common browsers and devices online.

Rate Limiting

Some websites have mechanisms in place to restrict the frequency and volume of requests made by automation script, irrespective of the user-agent string used in the request.

Resolution

Introduce delays between the requests to avoid overloading the website’s servers. Besides, monitor and respect the website’s terms of service.

Custom User-Agent Not Recognized

Some websites have strict user-agent requirements and expect incoming HTTP requests to include user-agent strings that correspond to well-known and commonly used browsers or devices.

Resolution

To avoid such an issue, you might need to use a user-agent string that closely matches a common browser, even if it’s not truly custom. However, you need to stay cautious as some websites may still detect automation.

API Changes

The APIs or libraries used for generating random user-agents or managing user-agent rotation may become obsolete.

Resolution

Keep updating external libraries and dependencies regularly. Check for updates in the package you use and make necessary adjustments in the code accordingly.

Conclusion

The choice between random or custom user-agent in Puppeteer is a critical decision particularly when you are engaged in web scraping or automation tasks. Each approach has its own upsides and downsides, and the decision should align with your project requirements and the nature of the website you want to interact with.