One await was costing me 4 seconds

One await in a for loop was costing 4 seconds.

When I ran time node src/bench.js yesterday, the network was the bottleneck for this super tiny codebase. Turns out the culprit was the await getSiteData(entry.url) within the for loop. Await was waiting for the network for every entry that was coming from my sitemap.xml, one at a time.

Here's the loop:

for (const entry of entries) {
    const html = await getSiteData(entry.url)
    htmlEnriched.push({ ...entry, html })
}

PR: https://github.com/bak-george/Pharos/pull/3

Solution: map the entries and gather up all the promises

The trick is moving await out of the loop and into a callback. The loop then runs to completion, launching all the requests at once.

export async function getHTMLContentFromUrl(rawData) {
  const entries = filterUrlAndModDate(rawData)
  const promises = entries.map(async (entry) => {
    const html = await getSiteData(entry.url)

    return { ...entry, html }
  })

  return await Promise.all(promises)
}

The map method doesn't wait for any network call. It gathers the promises up in the new array, and then await Promise.all(promises) waits for the whole batch to finish and hands back the results in the original order.

One await for the whole batch instead of one await per entry.

Never await inside a for loop when the requests don't depend on each other.