Puppeteer and Pipedream

Need to grab a secured iFrame JSON response that is limited to single origin Cross-Origin Resource Sharing CORS

Pipedream has a free level of access that combined with their own puppeteer module allows me to open a secured webpage with a headless browser and then intercept and respond back with the target JSON.

This means I am scraping a secure private REST API blocked by CORS with a headless browser and watching for a specific url

import { puppeteer } from '@pipedream/browsers';

export default defineComponent({
  async run({steps, $}) {
    const browser = await puppeteer.browser();    
    const page = await browser.newPage();
    let quoteDetails = null;
    page.on('response', async response => {
    if (response.url().startsWith("https://<protected CORS internal API request>"))
      if (response.status() === 200) {
        quoteDetails = await response.json();
      }
    });
    await page.goto(steps.trigger.event.body.url);
    const content = await page.content();
    await $.respond({
      status: 200,
      body: quoteDetails
    });
    await browser.close();
  },
})

Let's me get an internal JSON response like this

Example curl request

curl --location 'https://<deployed_handler>.m.pipedream.net' --header 'Content-Type: application/json' --data '{ "url": "https://target_url/4a10fafefdc74568a47fff484882b8d5?or_arrival=2024-03-03&or_departure=2024-03-06&or_adults=2" }'

And the response

{
    "succeeded": false,
    "total": 667.95,
    "totalFormatted": "$668",
    "totalFormatted2": "$667.95",
    "nights": 3,
    "errors": [
        "A minimum of 4 nights is required."
    ],
    "charges": [
        {
            "type": "Rent",
            "description": "Rent",
            "rate": 550,
            "rateIsPercent": false,
            "amount": 550,
            "amountFormatted": "$550.00"
        },
        {
            "type": "Surcharge",
            "description": "Service Fees",
            "rate": 13.5,
            "rateIsPercent": true,
            "amount": 74.25,
            "amountFormatted": "$74.25"
        },
        {
            "type": "Tax",
            "description": "Taxes",
            "rate": 43.7,
            "rateIsPercent": false,
            "amount": 43.7,
            "amountFormatted": "$43.70"
        }
    ],
    "currencyThreeLetter": "USD",
    "gaEvent": {
        ...
    }
}