Let's first understand the difference between existing and active in the context of web page elements. If a certain element is present in the webpage DOM, we say it exists. If a certain element is currently interactive and visible, we say it is active. Checking if an element exists and is active is a very common task in web scraping and automation. Therefore, frameworks like Puppeteer provide an easy way to do that.

When using Puppeteer, you can efficiently handle tasks such as element matching to determine if a DOM element exists. This is particularly useful when you need to match multiple elements or verify visible elements on the page.

In this article, we'll discuss options you have to work with Puppeteer element exists and how to check whether they're in an active state.

Checking if an element exists

In browser automation, it's common to create conditional statements depending on whether a specific element exists. For instance, let's consider a scenario where we're scraping products from an online shopping website. We aim to identify products that are on sale by checking for the existence of a "Sale" badge next to them. In this case, we'll only collect information about products displaying the "Sale" badge and disregard those without it. Let's look at an example code on how to do it.

cconst puppeteer = require('puppeteer');

(async () => {
  const browser = await puppeteer.launch();
  const page = await browser.newPage();
  await page.goto('https://ecommerce.com/product-list');

  // Select all product containers
  const productContainers = await page.$$('.product-container');

  // Loop through each product container
  for (const container of productContainers) {
    const hasSaleBadge = await container.$('.sale-badge');

    // If a sale badge exists, gather information about the product
    if (hasSaleBadge) {
      const productName = await container.$eval('.product-name', el => el.textContent);
      const productPrice = await container.$eval('.product-price', el => el.textContent);

      console.log(`Product Name: ${productName}`);
      console.log(`Product Price: ${productPrice}`);
      console.log('This product is on sale!\n');
    }
  }

  await browser.close();
})();

This code sample demonstrates how to scrape an e-commerce product list to find items on sale. It navigates to the product list page, selects all product containers, and iterates through them. For each product, it checks if a "Sale" badge is present. If the badge exists, it extracts and logs the product's name and price, indicating that the product is on sale.

To learn more on how to select an element in Puppeteer for data extraction, be sure to check our guide on Get Element in Puppeteer.

Using page.waitForSelector()

To handle dynamic web pages, you might need to wait for specific elements to appear. Using a method like waitForSelector, you can specify a timeout to wait for a unique element. This ensures that your script interacts with elements only when they are ready. If the specified timeout is reached and the element is not found, the function will return false instead of a return element, indicating that the element does not exist. This is particularly useful in web scraping with Puppeteer and test automation to ensure reliable operations when dealing with dynamic content.

Let's build on the previous example and use page.waitForSelector() to check for elements.

const puppeteer = require('puppeteer');

(async () => {
  const browser = await puppeteer.launch();
  const page = await browser.newPage();
  await page.goto('https://ecommerce.com/product-list');

  const productContainers = await page.$$('.product-container');

  for (const container of productContainers) {
    try {
      // Wait for the sale badge to appear within the product container
      const saleBadge = await container.waitForSelector('.sale-badge', { timeout: 5000 });

      if (saleBadge) {
        const productName = await container.$eval('.product-name', el => el.textContent);
        const productPrice = await container.$eval('.product-price', el => el.textContent);

        console.log(`Product Name: ${productName}`);
        console.log(`Product Price: ${productPrice}`);
        console.log('This product is on sale!\n');
      }
    } catch (error) {
      console.log('Sale badge not found for this product.');
    }
  }

  await browser.close();
})();

Let's break down the code step by step to understand it better.

Step 1 - Navigating to the Page and Selecting Elements

The code launches a headless browser, creates a new page, and navigates to the product list URL. It then selects all product containers on the page.

Step 2: Checking for the "Sale" Badge with page.waitForSelector():

Within each product container, the code uses container.waitForSelector('.sale-badge', { timeout: 5000 }) to wait for the "Sale" badge to appear. The timeout ensures the script waits up to 5 seconds for the badge to be present, making it robust for dynamically loaded content. If the badge is found, it extracts the product name and price and logs them, indicating the product is on sale. If the badge is not found within the timeout, it logs that the sale badge is not present for that product.

Step 3 - Closing the Browser

After processing all product containers, the browser is closed to clean up resources.

Using page.$$(selector)

Another method to check if an element exists on a webpage is by using page.$$(selector). This approach allows you to search for multiple elements that match a specific selector, providing a straightforward way to check for their existence.

const puppeteer = require('puppeteer');

(async () => {
  const browser = await puppeteer.launch();
  const page = await browser.newPage();
  await page.goto('https://example.com');

  const elements = await page.$$('#myElement');

  if (elements.length > 0) {
    console.log('Element exists');
  } else {
    console.log('Element does not exist');
  }

  await browser.close();
})();

This code basically does two things.

• The line const elements = await page.$$('#myElement'); searches for all elements with the ID myElement. The $$ notation indicates that this method can find multiple elements matching the selector.
• The if statement if (elements.length > 0) checks if the elements array contains any items. If it does, it logs "Element exists"; otherwise, it logs "Element does not exist."

Using page.$$(selector) is a concise way to check for the existence of elements on a webpage. It allows you to handle multiple elements efficiently, making it a useful tool in both web scraping and test automation.

Checking if an element is active

Once you check whether an element exists, it's also important to check if the element is active before executing operations. This can help you wait for dynamic elements to be available before the script proceeds.

Using document.activeElement

One of the most straightforward and reliable methods to check the status of an element on your web page is using document.activeElement. This is particularly useful for elements like input fields, buttons, and links.

Let's take a look at an example code that uses document.activeElement.

const isActive = await page.evaluate(() => {
  const activeElement = document.activeElement;
  return activeElement === document.querySelector('selector');
});

if (isActive) {
  console.log('Element is active!');
} else {
  console.log('Element is not active.');
}

This code checks if the element specified by the selector is currently active. Let's break it down into steps.

Step 1: Starting the function

const isActive = await page.evaluate(() => {

The code block starts off by declaring a constant variable named ‘isActive’. The await keyword holds back the execution until the await Promise is resolved, and then assigns the resolved value to the isActive variable.

Step 2: Retrieving the active element

const activeElement = document.activeElement;

This line gets a reference to the element that currently has a focus (is active) on the webpage. This could be any element that the user is interacting with 

Step 3: Checking if the elements are the same

return activeElement === document.querySelector('selector');

This line uses the ‘document.querySelector’ method used for matching elements on the webpage that uses the selector as valuable data and match the given selector to compare the ‘activeElement’ with the element found.

Step 4: Conditional output

  if (isActive) {
    console.log('Element is active!');
  } else {
    console.log('Element is not active.');
  }

The final step is to implement the conditions, which as you can probably tell, logs a message to the console regarding the status of the console and tells you whether the element is active or not.

Error handling with try-catch blocks

As you would’ve noticed with the first code snippet for checking if an element exists, a try-catch block is used.

  try {
    const elementHandle = await page.waitForSelector('#myElement', { timeout: 5000 });
  } catch (error) {
    console.error('Element not found');

This try-catch block is a fundamental mechanism for graceful error handling. They allow you to write code anticipating potential errors and handle them without crashing the program. 

The try block would contain the code that you expect to throw an exception (potential errors) during execution due to issues like invalid user input, file access issues, etc. The catch block then catches the exception if it does end up occurring. This block contains exceptions to take when an exception is thrown.

If we used try-catch blocks for the last code we looked at to check if an element is active, it would be applied as follows.

try {
  const isActive = await page.evaluate(() => {
    const activeElement = document.activeElement;
    return activeElement === document.querySelector('selector'); 
  });

  if (isActive) {
    console.log('Element is active!');
  } else {
    console.log('Element is not active.');
  }
} catch (error) {
  console.error('An error occurred:', error);
}

Conclusion

Puppeteer as a library offers a range of functionalities that you can apply to your development projects. Methods to check if an element exists in a given web page, or to check if it is active would allow you to develop your automation to avoid errors and unexpected outcomes from your script. You can step this up by applying a try-catch block for better error handling. All of this comes together ultimately to help you undertake proper web scraping and build well-rounded automation.