BlogHow to wait For Page to Load in Puppeteer

Mon Jan 13 2025

Arjun Komath

In this article, we will discuss some strategies for ensuring a web page is fully loaded or has necessary content before performing an action like scraping content, capturing screenshot or capturing PDF.

At the high level, we will be comparing the following solutions:

• waitForNavigation: Ideal for actions like clicking links or form submissions. Example:

await Promise.all([
	page.waitForNavigation({ waitUntil: 'domcontentloaded' }),
	page.click('a[href="/newpage"]')
]);

• waitForSelector: Wait for specific elements to appear. Example:

const element = await page.waitForSelector('#specificElement', { visible: true, timeout: 5000 });

• page.goto with options: Adjust waitUntil settings for different page types:

• load: Waits for all resources (best for static sites).

• domcontentloaded: Waits for basic HTML to load.

• networkidle0: Waits for no network activity (best for SPAs).

Quick Comparison

Method	Best Use Case	Timeout
waitForNavigation	Full page loads, redirects	30 seconds
waitForSelector	Dynamic content, AJAX updates	10 seconds
page.goto options	Initial page loading	60 seconds
Combined methods	Complex SPAs, heavy content	15-30 seconds

Puppeteer Methods for Page Load Waiting

Using waitForNavigation

This method waits for the page to navigate to a new URL or reload.

await Promise.all([
	page.waitForNavigation({ waitUntil: 'domcontentloaded' }),
	page.click('a[href="/newpage"]')
]);

The waitUntil parameter specifies when the navigation is considered complete, helping avoid timing issues [2].

Using waitForSelector

The waitForSelector method waits for specific elements to appear on the page, making it a better alternative to using arbitrary timeouts [3]:

const element = await page.waitForSelector('#specificElement', {
	visible: true,
	timeout: 5000
});

By setting visible: true, you ensure the element is not only in the DOM but also visible on the page [4].

Configuring page.goto Options

The page.goto method allows you to control how Puppeteer waits for a page to load. Here are some common options:

Option	Description	Best Used For
load	Waits for the load event	Static websites with all resources loaded
domcontentloaded	Waits for the DOMContentLoaded event	Pages with basic HTML content
networkidle0	Waits until there are no network requests	Single-page applications
networkidle2	Waits until there are 2 or fewer network requests	Pages with heavy content

An example of using page.goto to wait for a page to load:

await page.goto('https://example.com', {
	waitUntil: 'networkidle0',
	timeout: 30000
});

These methods provide flexibility when dealing with different types of pages and their loading behaviors, ensuring smoother automation workflows.

Examples of Waiting for Page Load

Example: Waiting for a Specific Element to Load

When working with dynamic web apps / SPA, it is important to wait for specific elements to ensure automation runs smoothly. Here's an example of waiting for a donation button on Python's official website:

const page = await browser.newPage();
await page.goto('https://www.python.org/');

// Wait for the donate button to become visible
const donateButton = await page.waitForSelector('.donate-button', {
	visible: true,
	timeout: 5000
});

await donateButton.click();

This script waits for the button to appear and become clickable, with a timeout set to 5 seconds [4].

Example: Handling Page Navigation with waitForNavigation

In cases involving page navigation, combining waitForNavigation with click actions ensures smooth transitions. Check out this example for navigating MDN's JavaScript documentation:

// Navigate to JavaScript documentation
const page = await browser.newPage();
await page.goto('https://developer.mozilla.org/en-US/docs/Web/JavaScript');

// Prepare for navigation before triggering the click
const loginLink = await page.$('.login-link');
await Promise.all([
	page.waitForNavigation({
		waitUntil: 'domcontentloaded',
		timeout: 30000
	}),
	loginLink.click()
]);

By using Promise.all, the script ensures the click and navigation processes happen together, avoiding unnecessary delays. This approach is especially helpful for single-page applications (SPAs) with dynamic content [2].

For even more complex scenarios, combining these methods can make your automation scripts more reliable. Let's dive deeper into that.

Handling Complex Scenarios

Tackling complex situations in web automation often calls for advanced techniques. Let’s break down some practical methods to address these challenges effectively.

Combining Waiting Methods

Dynamic web applications frequently require multiple waiting conditions to ensure smooth automation. Take LinkedIn's job search page as an example:

await page.goto('https://www.linkedin.com/jobs', { waitUntil: ['networkidle0', 'domcontentloaded'] });

await Promise.all([
	page.waitForSelector('input[aria-label="Search jobs"]', { visible: true, timeout: 8000 }),
	page.waitForSelector('.jobs-search-results-list', { visible: true, timeout: 8000 })
]);

await page.waitForFunction(() => document.querySelectorAll('.job-card-container').length > 0, { timeout: 10000 });

This combination ensures that all necessary elements and dynamic content are fully loaded before the script proceeds [1][3]. While this method boosts reliability, setting the right timeouts is equally important for managing slow-loading pages without unnecessary delays.

Using Timeouts

Timeouts play a crucial role in handling pages that load slowly while keeping the process efficient. For instance, you might set a 30-second timeout for navigation and a shorter, 10-second timeout for dynamic content:

await page.waitForSelector('.dynamic-content', { visible: true, timeout: 10000 });

To handle potential errors, you can implement fallback strategies:

try {
	await page.waitForSelector('.dynamic-content', { visible: true, timeout: 10000 });
} catch {
	await page.reload({ waitUntil: 'networkidle0' });
	await page.waitForSelector('.dynamic-content', { visible: true, timeout: 15000 });
}

This method has been shown to work well in production environments, especially on content-heavy platforms [2][5]. By combining careful timeout settings with fallback mechanisms, you can address common challenges in complex automation tasks, keeping your scripts reliable and effective.