ScrapeNinja Live Sandbox

Get started right from your browser: execute scraping requests, both JS and non-JS, submitting the form below.

URL to scrape:

Quick examples: auto.de results scraping HackerNews basic Airlines dump AJAX headers POST JSON

Evaluate JS Launches real browser. This is 5x slower. Only GET requests are supported with JS evaluation.

Catch subsequent AJAX call

Advanced. Can be used to dump HTTP headers of the sub-request, for later usage of headers in non-JS requests. See airlines scraper example. Use with caution: bad URL mask will result in timeouts because scraper will be waiting for the AJAX call until the timeout, and won't be able to dump required data!

Mask for AJAX request url:

Block Images Do not wait for images to load.

Block CSS and fonts Do not wait for css to load.

HTTP Method

Proxy geo

Number of retries

Wait For Selector [JS evaluation]:

Wait for DOM selector to appear

Headers to send (one per line)

Raw POST/PUT body

Examples of valid payloads:

JSON payload: {\"fefe\":\"few\"}
www-encoded payload: key1=val1&key2=val2

Add Content-Type: application/x-www-form-urlencoded to headers in case of www-encoded POST and Content-Type: application/json

Scrape it Scraping..

Raw ScrapeNinja Response:

Latency: {{ responseLatency }}ms HTTP Status: {{ responseBody.info.statusCode }}

Submit the form to scrape the URL.

                {{ responseBodyFormatted }}

Screenshot

Click to open full size in a new tab.

Unescaped target website response

Access it in your code via responseJson.body property. Use {{cmdKey}}+F for a quick search in the response body.

Generated code for ScrapeNinja:

import fetch from 'node-fetch';

const url = 'https://scrapeninja.p.rapidapi.com/{{sjUrl}}';

const PAYLOAD = {{ payloadString }};

const options = {
  method: 'POST',
  headers: {
    'content-type': 'application/json',
    // get your key on https://rapidapi.com/restyler/api/scrapeninja
    'X-RapidAPI-Key': 'YOUR-KEY',
    'X-RapidAPI-Host': 'scrapeninja.p.rapidapi.com'
  },
  body: JSON.stringify(PAYLOAD)
};

try {
  let res = await fetch(url, options);
  let resJson = await res.json();

  // Basic error handling. Modify if neccessary
  if (!resJson.info || ![200, 404].includes(resJson.info.statusCode)) {
      throw new Error(JSON.stringify(resJson));
  }

  console.log('target website response status: ', resJson.info.statusCode);
  console.log('target website response body: ', resJson.body);
} catch (e) {
  console.error(e);
}

Launching the scraper in your local Node.js:

Make sure you are using Node.js v16 node -v

Step #1. Create project folder and install node-fetch

mkdir your-project-folder && \
  cd "$_" && \
  npm i -g create-esnext && \
  npm init esnext && \
  npm i node-fetch -y

Step #2. Copy&paste the code above

Create new empty file like scraper.js and paste the code to this file.

Step #3. Launch

node ./scraper.js

Why ScrapeNinja.net?

Websites are getting harder to scrape. Web scraping protection evolved from checking user agent and other headers, to checking ip address of requester, and lately — to TLS fingerprint analysis. It is easy to start scraping in any programming language but only specialized solutions can provide the reliable way to scrape data on scale.

ScrapeNinja.net has proven to provide high scraping reliability due to state-of-art real browser TLS fingerprint emulation, rigorous realtime stability monitoring of responses, and advanced tech which allows us to find and rotate best proxies.

Detailed post about why ScrapeNinja was created.

FAQ

Is ScrapeNinja.net any better than using Chrome extension XYZ?

Scraping Chrome extensions might be good when getting started, to do some light scraping, but for heavy duty and better performance you'd better use cloud-based solutions like ScrapeNinja.

Why not just use Puppeteer/Selenium to emulate Chrome?

When your Python or Node.js scraper fails to get data, using Puppeteer or Selenium is usually the most obvious alternative. But, these two, while doing wonders in terms of automation and flexibility, are slow, and it is very expensive in terms of hardware resources to keep 'em running on a scale, and it requires a lot of work to support it.

Is ScrapeNinja.net any better than Luminati Web Unlocker?

Luminati is a big player on scraping market, but their pricing is through the roof. It is easy to spend $2k or $3k on scraping via Luminati - and ScrapeNinja is a good affordable alternative.

ScrapeNinja Live Sandbox

Examples of valid payloads:

Raw ScrapeNinja Response:

Screenshot

Unescaped target website response

Generated code for ScrapeNinja:

Launching the scraper in your local Node.js:

Step #1. Create project folder and install node-fetch

Step #2. Copy&paste the code above

Step #3. Launch

Why ScrapeNinja.net?

FAQ

Is ScrapeNinja.net any better than using Chrome extension XYZ?

Why not just use Puppeteer/Selenium to emulate Chrome?

Is ScrapeNinja.net any better than Luminati Web Unlocker?

Project integrations:

ScrapeNinja

restyler/scrapeninja-api-php-client

Online tools:

cURL Converter

Scraper Sandbox

Contact Us:

[email protected]

Telegram: @instalyze