JavaScript web scraping with Node.js: a detailed guide

Modern websites increasingly rely on dynamic JavaScript rendering rather than static HTML pages. The content often loads after user interaction, such as scrolling or clicking.

This renders traditional JavaScript web scraping tools ineffective. Plus, most platforms then also use CAPTCHAs, IP blocks, JavaScript challenges, and geo-restrictions.

The good news is you can bypass these by building a custom scraper. In this guide, we'll teach you how to build scalable JavaScript scrapers with Node.js.

If you want a quicker setup for the same task, you can use SOAX's domain-specific scraper APIs or Web Unblocker.

But for now let's focus on the basics of how to build a custom scraper.

Why Node.js and JavaScript for web scraping?

JavaScript and Node.js offer a strong and flexible foundation for building scrapers that can scale and adapt. Their combination supports simple data extraction and advanced scraping from dynamic, interactive websites. These tools allow you to write asynchronous scripts that fetch data from multiple sources without delays. 

Since most websites rely on JavaScript, using the same language gives you an advantage.

Node.js, on the other hand, grants you access to browser-like APIs like fetch and querySelector, making transitions easier. You can also run scripts in parallel, significantly speeding up scraping when working with large datasets.

Below is more information on the benefits of each. 

JavaScript is native to the web

Modern websites rely heavily on JavaScript to generate and modify content after the initial page load. That means many vital elements don’t appear in the raw HTML you receive from a basic request.

Because JavaScript handles these updates, using it for scraping lets you better understand and interact with web pages. You can inspect behavior in the browser and replicate it with the same language in your script.

This alignment simplifies testing selectors and logic in browser developer tools before coding your scrapers. Once confirmed, you can reuse the rules inside a Node.js environment without rewriting them in another language.

Node.js uses event-driven, non-blocking I/O

One core advantage of using Node.js for web scraping is handling input and output (I/O) operations. Unlike traditional setups, Node.js handles these operations in an event-driven and non-blocking manner.

Instead of waiting for a request to finish, Node.js processes other tasks and responds when ready. This improves scraping speed, especially when working with many pages, files, or network calls.

The event loop makes sure your scraper remains responsive while collecting data, even on slow or complex websites. You can fetch, parse, and store information without freezing or delaying the rest of your script. 

Here’s a sample script showing how Node.js handles events without blocking other incoming actions:

// Import Node.js's built-in HTTP module
const http = require('http');

// Define the port your server will listen on
const port = 3000;

// Create an HTTP server using the event-driven model
const server = http.createServer((req, res) => {
  // Set the response status and content type
  res.writeHead(200, { 'Content-Type': 'text/plain' });

  // Send the plain text response
  res.end('Hello, World!\n');
});

// Start the server and begin listening for incoming requests
server.listen(port, () => {
  console.log(`Server running at http://localhost:${port}/`);
});

Explanation of the events: 

• require('http'): Imports Node.js’s HTTP module, which handles web requests without needing any extra libraries.

• http.createServer(...): Sets up an event listener that runs a callback every time a new request arrives.

• (req, res) => { ... }: Handles incoming HTTP requests.  Sets headers and sends responses.

• server.listen(...): Starts the server. Listens on the defined port and immediately frees the event loop for new tasks.

This script creates a basic HTTP server that responds to every incoming request with a short message. While waiting for one request to finish, Node.js simultaneously accepts new connections. 

The model is key to building scalable scrapers that handle thousands of pages. It listens, triggers responses, and moves forward without blocking the rest of your program.

Vast ecosystem of tools (npm modules)

Node.js offers a rich library ecosystem that simplifies scraping and reduces the need for additional coding. To match your project's needs, you must install and compile these tools through Node.js’s Node Package Manager (npm).

Rather than code from scratch, you can build scrapers using high-quality modules already proven and tested by the community. This approach saves time and reduces complexity, especially when working with dynamic or structured websites.

Some popular Node.js scraper npm tools include:

• Axios: A HTTP client that sends requests and handles responses without requiring extra setup. Supports proxy settings, custom headers, and error handling to manage connections reliably.

• Cheerio: A fast HTML parser that lets you use CSS selectors to extract content from web pages. Allows selection of elements just like in jQuery and retrieves their text, attributes, or structure.

• Puppeteer: A headless browser JavaScript tool that controls Chromium without opening a visible window. Loads dynamic content, executes scripts, and interacts with page elements like a real browser session.

• jsdom: A module that simulates a browser’s document model so that scripts can modify content without opening Chrome. Works well for simpler pages that need JavaScript but not the whole user interaction.

You can easily install any of these tools with a single npm command, but we’ll return to this later. 

Seamless use of browser-style APIs

JavaScript scraping works smoothly in Node.js because it supports many of the browser’s built-in APIs. Familiar methods like fetch, querySelector, and textContent are available without launching a full browser.

This consistency allows seamless transitions from browser testing to writing your scraper code with minimal changes. The same selectors used in Chrome DevTools will also work in your script environment.

Additionally, Cheerio and jsdom offer a virtual document structure that mirrors how browsers handle HTML and content selection. You can target elements, retrieve text, or extract attributes using the same commands as client-side code.

Cheerio also lets you load raw HTML and extract content using jQuery-like syntax such as $('h1') or .find('.price'). The interface feels familiar, and results are returned instantly without rendering the page.

Also, when sites use inline scripts to update content, jsdom simulates those changes and updates accordingly. After the scripts run, you can read modified values, detect inserted elements, and extract the final output.

Together, these libraries bring browser-like behavior into your Node.js environment without requiring a full headless browser. They offer a balance of speed, control, and flexibility for efficient JavaScript web scraping.

Prerequisites for JavaScript web scraping with Node.js

Before writing your first scraper, make sure you have the essential tools. Here are the things you need:

• Node.js version 18 or higher

• npm version 8 or higher

• Basic knowledge of modern JavaScript (ES6 and above)

• Familiarity with browser DevTools

• A code editor or IDE (e.g., VS Code or WebStorm)

• Access to a rotating proxy service (e.g., SOAX proxies) and credentials

As you can see, background knowledge of these tools is also vital for proper setup.

Picking your tools: HTTP clients and parsers

To scrape data effectively using Node.js, you need two types of tools, namely HTTP clients and parsers. The former fetches webpage content, while the latter extracts specific data from a page’s HTML.

Your choice depends on the target site, how it loads content, and the extent of control you need. Here's a comparison of common tools used for fetching and parsing data.

HTTP clients

These are the tools to send requests, receive responses, and manage connection behavior for target pages.

• Axios: A promise-based HTTP client with a simple syntax for sending requests and handling responses. Supports headers, timeouts, proxy settings, and error handling out of the box.

• Native fetch: A fetch that mimics the browser's fetch API. Is available in recent versions of Node.js. Offers less flexibility than Axios or other advanced libraries, but is lightweight.

• SuperAgent: A minimalist HTTP client that uses chaining syntax and built-in support for redirects and test environments. Works well for quick scripts but is less common in larger scraping projects.

These clients form the foundation of your scraper by retrieving raw HTML or structured data from external sources.

HTML Parsers

These tools help you extract specific elements, text, or attributes from the content returned by your client.

• Cheerio: A fast and flexible HTML parser with syntax similar to jQuery. Lets you select elements using CSS selectors and extract content directly from static HTML pages.

• jsdom: A complete DOM emulator that simulates how a browser processes and modifies content. Supports JavaScript-based updates and allows virtual scripts to run inside the page structure.

Each parser fits a different need, depending on whether your target site is static, dynamic, or script-modified.

In the next section, we’ll show how to combine these tools to extract structured data from real pages.

Integrating proxies

To use a proxy in your scraper, you must configure your HTTP client to route requests. This allows your scraper to communicate through a third-party connection while maintaining your existing setup.

Axios supports native proxy configuration through an embedded proxy object inside each request. Using environment variables, you can define the proxy host, port, and credentials in a structured format.

Here’s a sample of how it looks using SOAX’s proxy service:

// Axios + SOAX residential proxy example
await axios.get(url, {
  proxy: {
    host: PROCESS.ENV.SOAX_HOST,
    port: PROCESS.ENV.SOAX_PORT,
    auth: { username: PROCESS.ENV.SOAX_USER, password: PROCESS.ENV.SOAX_PASS }
  }
});

The above setup forwards your HTTP request to the proxy host defined in your environment settings. Axios then automatically uses those values to authenticate and connect through the proxy server.

You can reuse this configuration across all your requests by assigning the object to a shared variable. This keeps your scraper clean and avoids duplicating the code inside every request.

Once your proxy is up and running, you should start sending requests as you would normally. All existing response handling, parsing, and error logic will continue to work without further adjustments.

Tools comparison: Pros and cons

Each library has specific strengths that work best for different use cases. We’ve compared them side-by-side to help guide your decision-making.

You should choose the right combination based on how your target site structures its content and handles client-side behavior. Switch tools depending on whether the content appears in the HTML or loads later with JavaScript.

Scraping HTML with Axios and Cheerio

You can use Axios and Cheerio when your target site returns usable HTML without needing JavaScript to render content. Axios sends the request, while Cheerio lets you extract data using CSS selectors in a jQuery-like style.

The method suits blog articles, product listings, or any page containing content in the initial response. It’s easy to locate and extract page titles, headings, or quotes without using a full browser.

Do the following:

Step 1: Prepare the environment

Create a project folder, navigate to it, and initialize a Node.js project. Use this command. 

mkdir my-scraper
cd my-scraper
npm init -y

This creates a package.json file to manage dependencies. It makes your setup ready for JavaScript web scraping tasks.

Step 2: Install Axios and Cheerio

Install both libraries using npm to prepare your environment for sending requests and parsing HTML. Run the following code in your terminal.

npm install axios cheerio

The duo will power your scraper by handling data retrieval and HTML parsing.

Step 3: Fetch HTML and load into Cheerio

Use Axios to retrieve a webpage’s HTML content, then load it into Cheerio. Cheerio’s jQuery-like syntax makes parsing accessible for beginners learning to scrape.

const axios = require('axios');
const cheerio = require('cheerio');
axios.get('https://example.com').then(({ data }) => {
  const $ = cheerio.load(data);
});

You can test this code in a file named scraper.js using node scraper.js. It fetches and prepares HTML for extraction without requiring complex coding.

Step 4: Use CSS selectors to extract elements

CSS selectors in Cheerio let you target specific elements like titles or links. You can extract data by selecting elements with familiar syntax from browser DevTools.

For example, to grab all headings or specific paragraphs, use selectors like h1or p. This code snippet extracts and logs the main heading from a page.

const axios = require('axios');
const cheerio = require('cheerio');
axios.get('https://example.com').then(({ data }) => {
  const $ = cheerio.load(data);
  const title = $('h1').text();
  console.log('Title:', title);
}).catch(err => console.error('Error:', err.message));

You can modify the selector to target other elements like quotes, author names, or links.

Scraping HTML with jsdom

When a site relies on inline JavaScript to modify its content, Cheerio may not detect those changes. Instead of loading a full browser with Puppeteer, you can use jsdom to simulate a browser-like environment and execute those scripts virtually.

Start by installing jsdom:

npm install jsdom

Then, write a scraper that loads HTML into a virtual DOM and lets embedded scripts run:

const axios = require('axios');
const { JSDOM } = require('jsdom');

(async () => {
  const response = await axios.get('https://example.com');
  const dom = new JSDOM(response.data, { runScripts: 'dangerously', resources: 'usable' });

  // Wait for scripts to execute (may need adjustment for heavier pages)
  setTimeout(() => {
    const document = dom.window.document;
    const heading = document.querySelector('h1').textContent;
    console.log('Heading:', heading);
  }, 3000); // 3 seconds delay for scripts to finish
})();

This approach works well for pages that manipulate the DOM using inline scripts. It can handle content inserted via innerHTML or JavaScript variables.

Note that jsdom doesn’t support complex browser features like animations, event loops, or interactive input. Still, it's a lighter and faster option where full emulation isn’t necessary.

Using Puppeteer for JavaScript-heavy sites

As mentioned, some websites, including single-page apps and infinite scrolling pages, use JavaScript to generate content after loading. Static tools can’t scrape this data because it doesn’t exist in the initial HTML response.

Luckily, Puppeteer controls a headless browser, allowing you to open pages, click elements, and extract data. This makes it possible to scrape JavaScript-heavy sites that update content dynamically after user interaction.

Begin by launching the browser using puppeteer.launch() and then open a new page instance. Once the page loads, you can wait for specific elements to appear using waitForSelector.

Check the demonstration below:

const puppeteer = require('puppeteer');

(async () => {
  const browser = await puppeteer.launch();
  const page = await browser.newPage();

  await page.goto('https://example.com');
  await page.waitForSelector('h1');

  const heading = await page.$eval('h1', el => el.innerText);
  console.log('Heading:', heading);

  await browser.close();
})();

This script opens a headless browser, waits for an <h1> element, and extracts the visible text. You can adjust the selector to target other elements depending on your scraping goal.

Looping through multiple pages with Puppeteer

Many websites divide content across pages using pagination, requiring your scraper to visit each separately. You can handle this by updating the page URL or clicking a “Next” button to move forward.

Puppeteer lets you navigate between pages by modifying the URL or simulating user interaction. 

Both methods allow data collection from multiple pages without restarting the browser session.

You can use a loop if the site has a clear URL pattern, such as /page/1, /page/2, /page/3, and so on. The method gives you more control and works reliably on sites utilizing pagination.

Here’s a sample scraper for multiple pages using a numbered URL pattern:

const puppeteer = require('puppeteer');

(async () => {
  const browser = await puppeteer.launch();
  const page = await browser.newPage();

  for (let i = 1; i <= 3; i++) {
    await page.goto(`https://example.com/page/${i}`);
    await page.waitForSelector('.item');

    const items = await page.$$eval('.item', elements =>
      elements.map(el => el.innerText)
    );

    console.log(`Page ${i} items:`, items);
  }

  await browser.close();
})();

This script loads three pages in sequence, waits for items to appear, and logs each set of results. Consider adjusting the selector and URL pattern to match the structure of your target website.

If the site uses a “Next” button instead of numbered URLs, use page.click() to simulate the interaction. Combine it withwaitForSelector to wait for content before extracting it.

Check out how it works: 

const puppeteer = require('puppeteer');

(async () => {
  const browser = await puppeteer.launch();
  const page = await browser.newPage();

  await page.goto('https://example.com');
  for (let i = 0; i < 3; i++) {
    await page.waitForSelector('.item');

    const items = await page.$$eval('.item', elements =>
      elements.map(el => el.innerText)
    );

    console.log(`Page ${i + 1} items:`, items);

    const nextButton = await page.$('.next');
    if (!nextButton) break;

    await Promise.all([
      page.waitForNavigation(),
      nextButton.click()
    ]);
  }

  await browser.close();
})();

This script clicks the “Next” button, waits for the new page to load, and repeats the extraction. Adjust the .next selector to match the actual button on your target site.

Avoiding blocks and bot detection

JavaScript web scraping can trigger blocks if sites detect automated requests too quickly. You can reduce detection risk by combining multiple strategies across timing, identity, and request management.

Use random delays and rotating user agents

Sites often track browser identity by inspecting the User-Agent header on each request. Repeatedly using the same header increases the risk of detection and block.

You can install the complete list of user-agent strings using the user-agents package from npm. This library provides realistic browser profiles and updates automatically with new entries.

Install it by running this script in your terminal:

npm install user-agents

Then, load a new user-agent for each request and include it in your request headers. This approach gives your scraper a rotating browser identity that mimics real visitor traffic patterns.

Here’s an example using random user-agents and random delays:

const axios = require('axios');
const UserAgent = require('user-agents');

function getRandomUserAgent() {
  const userAgent = new UserAgent();
  return userAgent.toString();
}

function delay(ms) {
  return new Promise(resolve => setTimeout(resolve, ms));
}

(async () => {
  for (let i = 0; i < 5; i++) {
    const headers = { 'User-Agent': getRandomUserAgent() };
    const response = await axios.get('https://example.com', { headers });

    console.log('Request', i + 1, 'done with user-agent:', headers['User-Agent']);
    await delay(Math.floor(Math.random() * 2000) + 1000); // Delay between 1–3 seconds
  }
})();

Each request sends a new user-agent string and waits for a random delay before continuing. This helps your scraper blend into regular browser traffic and avoid detection.

Rotate through a pool of residential proxies

By rotating proxies, you spread your requests across different IPs, making  your scraper appear like multiple real users. This technique allows you to bypass rate limits.

The better your proxy service, the better this technique works. For example, with a service like SOAX you can access 155 million residential IP addresses. Once you get such a service, you can integrate it into your scraping code as follows:

const axios = require('axios');

const proxies = [
  { host: 'proxy1.soax.com', port: 10001, user: 'user1', pass: 'pass1' },
  { host: 'proxy2.soax.com', port: 10002, user: 'user2', pass: 'pass2' },
  { host: 'proxy3.soax.com', port: 10003, user: 'user3', pass: 'pass3' },
];

function getRandomProxy() {
  return proxies[Math.floor(Math.random() * proxies.length)];
}

(async () => {
  const proxy = getRandomProxy();

  await axios.get('https://example.com', {
    proxy: {
      host: proxy.host,
      port: proxy.port,
      auth: { username: proxy.user, password: proxy.pass },
    },
  });

  console.log('Request sent using proxy:', proxy.host);
})();

This method routes traffic through different IP addresses, selecting random proxies from your list.

Bypass geo-restrictions with location-specific proxies

Some websites restrict access to certain content based on the visitor’s location. If your scraper runs from an unsupported region, it may return different results or fail to load the page altogether.

To solve this, use residential proxies that let you choose the location of each IP. SOAX, for example, allows you to target 195+ countries, including regions and even cities. With this broad and deep proxy pool, you can access basically any location your scraper needs. 

Below is how to this implementation works: 

const axios = require('axios');

const proxy = {
  host: 'proxy-us.soax.com', // Replace with a region-specific host
  port: 10000,
  auth: {
    username: 'SOAX_USERNAME',
    password: 'SOAX_PASSWORD',
  },
};

(async () => {
  const response = await axios.get('https://example.com', {
    proxy: proxy,
  });

  console.log('Page content:', response.data);
})();

This setup sends the request through a US-based proxy, ensuring you receive the version of the site shown to visitors in the United States. You can swap the proxy configuration to match the target region whenever needed.

Handle CAPTCHAs with third-party services

When a site shows a CAPTCHA, your scraper must pause and send it to a solving service. Services like 2Captcha and Anti-Captcha offer APIs that return the CAPTCHA solution after submission.

const axios = require('axios');

async function solveCaptcha(siteKey, pageUrl, apiKey) {
  const submission = await axios.get('http://2captcha.com/in.php', {
    params: {
      key: apiKey,
      method: 'userrecaptcha',
      googlekey: siteKey,
      pageurl: pageUrl,
      json: 1,
    },
  });

  const requestId = submission.data.request;

  // Polling until CAPTCHA is solved
  let result;
  while (true) {
    await new Promise(r => setTimeout(r, 5000));
    const res = await axios.get('http://2captcha.com/res.php', {
      params: { key: apiKey, action: 'get', id: requestId, json: 1 },
    });

    if (res.data.status === 1) {
      result = res.data.request;
      break;
    }
  }

  console.log('CAPTCHA solved:', result);
}

The script submits a CAPTCHA and checks every few seconds until the third-party service returns a solved response.

Rate-limit your requests

Sending too many requests quickly can trigger blocks or reduce access to target websites. You can do this manually, but that takes a lot of time and effort. Instead, use a limiter to avoid constantly adjusting your scraper’s code.

You can use the bottleneck library to schedule requests at specific intervals. It handles timing, retries, and concurrency so you can focus on scraping logic.

First, install Bottleneck using npm:

npm install bottleneck

Then, apply it to your scraper to space out requests automatically:

const axios = require('axios');
const Bottleneck = require('bottleneck');

const limiter = new Bottleneck({
  minTime: 2000, // 2 seconds between requests
  maxConcurrent: 1, // one request at a time
});

const urls = [
  'https://example.com/page/1',
  'https://example.com/page/2',
  'https://example.com/page/3',
];

async function fetchUrl(url) {
  const response = await axios.get(url);
  console.log('Fetched:', url);
}

urls.forEach(url => limiter.schedule(() => fetchUrl(url)));

This setup runs one request every two seconds, keeping your scraper within rate limits. Feel free to adjust the timing and concurrency settings based on your needs.

But what if there was an easier way to do it?

Scrape complex pages with SOAX Web Unblocker

Managing proxies, user-agents, CAPTCHAs, and JavaScript rendering can slow down your workflow. Use SOAX’s Web Unblocker to simplify the process.

It combines proxy rotation, headless browser rendering, CAPTCHA solving, and geo-targeting into a single solution. You send one request, and it returns the fully rendered page, regardless of JavaScript complexity or anti-bot measures.

Here’s how to integrate it using Axios:

const axios = require('axios');

(async () => {
  try {
    const response = await axios.post('https://unblocker.soax.com/api/v1/scrape', {
      url: 'https://example.com',
      render: true
    }, {
      headers: {
        'Authorization': 'Bearer YOUR_SOAX_API_KEY',
        'Content-Type': 'application/json'
      }
    });

    console.log('Page HTML:', response.data);
  } catch (error) {
    console.error('Scraping failed:', error.message);
  }
})();

Use the Web Unblocker when your target site combines multiple scraping defenses. It's a faster, more reliable setup because you don't have to write extra logic for every challenge.

Structuring and exporting scraped data

After gathering content, you must organize and export it in a readable or usable format. A clean data structure makes it easier to analyze or import into another system later.

Follow these steps:

1. Collect objects into an array

After scraping, collect each result as an object with consistent keys for easy access. Push every object into an array to loop through or convert the data later.

Here’s a sample structure for collecting author quotes:

const quotes = [];

$('.quote').each((i, el) => {
  const text = $(el).find('.text').text();
  const author = $(el).find('.author').text();

  quotes.push({ text, author });
});

Each object in the array contains one quote with its corresponding author. This array becomes the core of your export logic.

2. Convert to JSON or CSV

If exporting the data, start by converting the array into a readable format. Use JSON.stringify() for a clean and widely supported output structure.

const json = JSON.stringify(quotes, null, 2);
console.log(json);

To export as CSV, use a helper like papaparse or build a basic converter function.

const fs = require('fs');
const { parse } = require('json2csv');

const csv = parse(quotes);
fs.writeFileSync('quotes.csv', csv);

These formats are easy to import into dashboards, spreadsheets, or bulk upload systems.

3. Save to a file or database

After collecting and formatting the data, you can write it to a file or store it in a database. Choose the option that best fits how you plan to access or use the information later.

To write your scraped data to a file, use the built-in fs module. This example saves the array of objects as a JSON file.

const fs = require('fs');
fs.writeFileSync('quotes.json', JSON.stringify(quotes, null, 2));

If you prefer to save your data in CSV format, install the json2csv package first:

npm install json2csv

From here, use the following script to convert your data and write it to a CSV file:

const { parse } = require('json2csv');
const fs = require('fs');

const csv = parse(quotes);
fs.writeFileSync('quotes.csv', csv);

This script converts the array of objects into CSV format and writes it to a file called quotes.csv. You can open it in Excel or any spreadsheet application for review or sharing.

Saving data in a database (MongoDB)

If you want to store your data in a database, install the necessary driver first. For example, to use MongoDB, configure its module with this command:

npm install mongodb

Here's a sample function that connects to a local MongoDB server and stores your data:

const { MongoClient } = require('mongodb');

async function saveToMongoDB(data) {
  const uri = 'mongodb://localhost:27017';
  const client = new MongoClient(uri);
  await client.connect();

  const db = client.db('scraperDB');
  const collection = db.collection('quotes');

  await collection.insertMany(data);
  console.log('Data saved to MongoDB');

  await client.close();
}

saveToMongoDB(quotes);

This script simultaneously connects to MongoDB, selects a database and collection, and inserts all scraped items. You can later query, update, or export the data using any MongoDB tool or client.

Wrapping up

You now have the foundation to run reliable JavaScript web scraping workflows using Node.js and other tools. You know how to choose the right tools for your project, bypass bot systems, and export scraped data.

Moving forward, you can expand your setup by rotating proxies, automating identity changes, and precisely controlling request timing. These adjustments help your scraper stay stable across different sites and technical environments.

You should also consider monitoring page structure changes to catch broken selectors or empty results before they affect output. Adding version checks and alerts helps you maintain scraper performance without manual intervention.

If you’re interested in advanced proxy usage in your setup, visit the SOAX documentation page. Our team is always happy to answer any questions you have. We can help you figure out the best way to tackle any web scraping challenge you’re facing.