When you do website automation or web scraping, you encounter various challenges. One of them is that some web applications don’t load all the content at once when you visit the webpage, including images, text, scripts, and other resources. Instead, they utilize JavaScript to fetch content after the initial page load. This technique of fetching additional content without reloading the entire web page is referred to as asynchronous loading.

In such circumstances, you need to wait until the desired element is loaded before you proceed with your test or scraping. In Puppeteer, there is a method called waitForSelector for this. waitForSelector is a function in Puppeteer that allows you to pause script execution until a specific DOM element on a web page becomes available or meets certain conditions.

In this article, we’ll explore both the fundamental setup details and advanced configuration of waitForSelector in Puppeteer. From waiting for a single element to appear on a web page to modifying the timeout and handling multiple selectors, we’ll discuss all the details.

How does waitForSelector work?

To understand how waitForSelector works, consider the following example. Imagine you are automating a task on a webpage with a login form. You enter your credentials, and there's a "Login" button to submit the form. However, this button becomes visible a few seconds after the page loads, possibly due to animations or dynamic content loading. With Puppeteer, instead of trying to click the "Login" button right after the page loads, you would use the waitForSelector function. This function ensures that your script pauses and waits until the "Login" button is available and interactable. Once waitForSelector detects the button, the script can proceed to click it and carry on with subsequent tasks.

How to setup ‘waitForSelector’ function in Puppeteer

Let’s discuss how to set up the ‘waitForSelector’ function in puppeteer code. 

Step 1:  Launch Puppeteer

As usual, you need to import the Puppeteer library and launch a browser instance. 

const puppeteer = require('puppeteer');
const browser = await puppeteer. launch();

Step 2: Navigate to a Web Page

Next, create a new page in the browser and go to the desired webpage. 

const page = await browser.newPage();
await page.goto('https://example.com');

Step 3: Use waitForSelector

Now you can use ‘waitForSelector’ to wait for the specific DOM element to appear or meet certain conditions. 

await page.waitForSelector('your-selector');

‘your-selector’ should be replaced by a valid CSS selector or XPath selector that corresponds to the DOM element you’re waiting for. When this code is executed, Puppeteer will pause the script execution until the specified element appears on the page. 

‘waitForSelector’ method comes with an optional parameter that can include the following properties.

• timeout (number): The maximum time to wait for in milliseconds. Defaults value is 30000 (30 seconds).
• visible (boolean): Wait for the element to be present and visible.
• hidden (boolean): Wait for the element to be present and hidden.

Here is an example code line using these parameters. 

const element = await page.$('your-selector');
await element.click();

Step 5: Close the browser

Finally don’t forget to close the Puppeteer browser when you’re done.

await browser.close();

Step 6: Wrap the code in an async function

After you’ve written all the code, you'll notice that the await keyword is used. The await keyword can only be used inside an async function. This is necessary because await pauses the code execution until the Promise is resolved, preventing any blocking of subsequent code. To make your code work, wrap all the code lines starting from the first await used to the last await inside an async function as shown below. You can learn more about it in this guide. 

const puppeteer = require('puppeteer');

(async () => {
//  Your code goes here
})();

This is the basic configuration of the ‘waitForSelector’ function in Puppeteer. It means that your script waits for the specified element to appear on the web page before proceeding with further actions.

Example of waitForSelector in Puppeteer

Here’s a practical example demonstrating the use of the waitForSelector function in Puppeteer to wait for a specific element to appear on a webpage. For this example, we will wait until the donate button appear on python.org website and click it. 

const puppeteer = require('puppeteer');

(async () => {
  const browser = await puppeteer.launch();
  const page = await browser.newPage();
  await page.goto('https://www.python.org/');

  await page.waitForSelector('.donate-button');
  const donateButton = await page.$('.donate-button');
  await donateButton.click();

  console.log("Clicked the donate button!");

  await browser.close();
})();

In this script, after navigating to 'https://www.python.org/', the waitForSelector function waits for the "Donate" button, which has the class donate-button, to appear on the web page. When the "Donate" button is found, it's clicked. The console then logs a message to indicate that the "Donate" button has been clicked, providing a confirmation of the action. The browser is closed after this operation.

You can use this code and replace the website and the CSS selector according to your use case. 

Advanced configuration

In Puppeteer, the waitForSelector function offers advanced configuration options allowing for more customized usage based on varying scenarios. In the following sections, we are discussing about optional parameters it supports

Modifying timeout

By using the timeout option, you can tell Puppeteer how long to wait for a specific HTML element to appear on a webpage. Puppeteer will keep waiting until either the element shows up, or the timeout period ends. If the timeout ends and the element hasn’t shown up, Puppeteer will let you know by giving an error.

For example, in the above example, we can tell Puppeteer to wait for 15 seconds until the Donate button appears. Here’s an easy way to use it. 

await page.waitForSelector('.donate-button', { timeout: 15000 });

In the example above, we're telling Puppeteer to wait for the "Donate" button (which has the selector '.donate-button') for up to 15 seconds. If the "Donate" button doesn’t show up in 15 seconds, Puppeteer will stop waiting and give an error. Using the timeout options make your script more flexible and reliable. 

Wait until the element disappears

For cases when you need to wait until a specific element disappears from the page, you can use the {hidden: true} option in waitForSelector. Using this option tells Puppeteer to hold off until the particular element is hidden or removed from the webpage.

For the sake of simplicity, we will use the same example we used above. Suppose, you want to ask the Pupepeter code to wait until the Donate button disappears, you can use the following code line.

await page.waitForSelector('.donate-button', { hidden: true });

Passing {hidden: true} as an option tells Puppeteer to wait until the "Donate" button is no longer visible or has been removed from the webpage before letting your script move forward. (in this case, the Donate button doesn’t disappear but you can use this option whenever it suits your needs.)

Handling multiple selectors

When working with web pages, you might encounter situations where you need to wait for multiple elements to appear on the webpage. Puppeteer’s waitForSelector can handle this too.

Let’s take our example again. In our python.org website you can see two buttons called "Donate" and "Go".

If you want to handle both of these elements you can do it with the following codes. The class selector for the "Donate" button is “donate-button", and for the "Go" button, it's “search-button”.

Waiting for Multiple Selectors to Show

Consider the scenario where we want to ensure both the "Donate" and "Go" buttons are present on the webpage. You can do it with the following code.

const puppeteer = require('puppeteer');

(async () => {
  const browser = await puppeteer.launch();
  const page = await browser.newPage();
  await page.goto('https://www.python.org/');

  await Promise.all([
    page.waitForSelector('.donate-button'),
    page.waitForSelector('.search-button')
  ]);

  const donateButton = await page.$('.donate-button');
  await donateButton.click();
  console.log('Clicked the Donate Button.');

  await browser.close();
})();

In the above code, Puppeteer is programmed to wait until both the “Donate” and "Go" buttons are available on the web page before clicking the "Donate" button.

Waiting for at least one selector to show

In another situation, your script might need to proceed when at least one out of the two buttons appears on the webpage. Here's how to manage this.

const puppeteer = require('puppeteer');

(async () => {
  const browser = await puppeteer.launch();
  const page = await browser.newPage();
  await page.goto('https://www.python.org/');
 
 
  const button = await Promise.race([
    page.waitForSelector('.donate-button'),
    page.waitForSelector('.search-button')
  ]);
 
 
  if (await page.$('.donate-button')) {
    const donateButton = await page.$('.donate-button');
    await donateButton.click();
    console.log('Clicked the Donate Button.');
  } else {
    const goButton = await page.$('.search-button');
    await goButton.click();
    console.log('Clicked the Go Button.');
  }
 
 
  await browser.close();
})();

In this example, the Puppeteer script continues as soon as either the “Donate” or "Go" button is found on the webpage, and it clicks the button that appears first.

Conclusion

In website automation testing, accuracy is very important. Puppeteer’s ‘waitForSelector’ method is a must-have tool in your arsenal to write accurate and reliable tests. You learned how this ‘waitForSelector’ method allows developers to pause script execution until specific elements appear on the web page. In this article, we discuss the basic and advanced uses of ‘waitForSelector’ in Puppeteer, from waiting for an element to appear on a web page to handling multiple selectors.

You can simply modify the codes given in this article according to your needs. All you have to do is update the webpage link you need to go to and the CSS selectors you want to wait for, and then modify the action you need to perform on that webpage. You can also play with this code, adding more complex logic for more advanced automation or scraping use cases.