Node js script doesn't run in another folder

[peterchan75]

Hi davidktw,
I have two sets of code. The first is 5X faster than the latter. The latter code is able to catch the completion of all https events. Why is there a speed difference ?

while (tickers.length > 0) {
    var ticker_list = [];
    while (ticker_list.length <= ticker_list_num && tickers.length > 0) {
        ticker_list.push(tickers[0]);
        tickers.splice(0,1)
    }
    var my_url = url + ticker_list.join(',');
    const download_eod = async () => {
        try {
            const response = await axios.get(my_url);
            const body = response.data;
            array = body.quoteResponse.result;
            return array;
        }
        catch (error) {
            throw error;
        }
    };
   
    download_eod().then(array => process_json(array));
   
}

const download_eod = async () => {
    while (tickers.length > 0) {
        var ticker_list = [];
        while (ticker_list.length <= ticker_list_num && tickers.length > 0) {
            ticker_list.push(tickers[0]);
            tickers.splice(0,1)
        }
        var my_url = url + ticker_list.join(',');      
        try {
            const response = await axios.get(my_url);
            const body = response.data;
            array = body.quoteResponse.result;
            eod_array = process_json(array,eod_array);
        }
        catch (error) {
            throw error;
        }
    }
    return (eod_array);
}
download_eod().then(eod_array => output_eod_to_file(eod_array));

 

[davidktw]

Hi davidktw,
I have two sets of code. The first is 5X faster than the latter. The latter code is able to catch the completion of all https events. Why is there a speed difference ?

Of course they are different. In fact so much different. Look at my explanation below

while (tickers.length > 0) {
    var ticker_list = [];
    while (ticker_list.length <= ticker_list_num && tickers.length > 0) {
        ticker_list.push(tickers[0]);
        tickers.splice(0,1)
    }
    var my_url = url + ticker_list.join(',');
    const download_eod = async () => {
        try {
            const response = await axios.get(my_url);
            const body = response.data;
            array = body.quoteResponse.result;
            return array;
        }
        catch (error) {
            throw error;
        }
    };
 
    download_eod().then(array => process_json(array));
 
}

In the outer while loop, you are merely scheduling work to the event queue. The actual waiting is the line const response = await axios.get(my_url);. But this waiting is not in sequence as you think your code is doing. Invocation of async function simply just enqueuing it which happens when you wrote download_eod(). .then(array => process_json(array)); is only invoked after your "promise" returned by the async download_eod function is resolved. Basically your outer loop didn't even wait for anything. It just keep creating a new download_eod function, then invoke it by enqueuing it to the event queue and also all those async operations happening in this async function is also doing some async operations. If your outer while loop runs for 1000 times, there will be at least 2x1000 enqueued async events. 1000 of them is the async function, the other 1000 that I can count is the axios.get(my_url); statement. However for axios.get(my_url);, you have an await keyword used, so execution flow will be suspended there until the promise is resolved or rejected.

In the code above, your consolidation point is the process_json(array) function where if there is anymore codes after the while loop, you need to in some way wait for all queued events to call the process_json(array) callback function. If you did not in wait, then basically your execution will exit before anything is completely done.

const download_eod = async () => {
    while (tickers.length > 0) {
        var ticker_list = [];
        while (ticker_list.length <= ticker_list_num && tickers.length > 0) {
            ticker_list.push(tickers[0]);
            tickers.splice(0,1)
        }
        var my_url = url + ticker_list.join(',');  
        try {
            const response = await axios.get(my_url);
            const body = response.data;
            array = body.quoteResponse.result;
            eod_array = process_json(array,eod_array);
        }
        catch (error) {
            throw error;
        }
    }
    return (eod_array);
}
download_eod().then(eod_array => output_eod_to_file(eod_array));

In this example above, you only invoked the download_eod async function once, but you have assigned a callback to invoke output_eod_to_file(eod_array) when the promise is resolved. Hence your callback may be invoked.

Within the outer while loop, all async operations are waited as accordingly in sequence. The only one I see is the const response = await axios.get(my_url); statement, which despite is async, you suspended further execution unless axios.get function resolve or reject. So within the outer while loop, there enqueuing and resolution all happened in sequence. Thus the entire execution must have been completed by the time the callback function at the bottom is invoked.

Hope you get what I have described.

 

[peterchan75]

Hi davidktw,
This explain why in the results in the first code are all jumbled up whereas the latter code, the results are all in order. Anyway, the first code complete in less than 1 seconds for about 400 https request. But need to execute another code to verify no data. The second code takes about 6 seconds with download and verify no data. From the execution time standpoint, the first code is more efficient but the program flow is more complicated.

Thanks.

 

[davidktw]

Hi davidktw,
This explain why in the results in the first code are all jumbled up whereas the latter code, the results are all in order. Anyway, the first code complete in less than 1 seconds for about 400 https request. But need to execute another code to verify no data. The second code takes about 6 seconds with download and verify no data. From the execution time standpoint, the first code is more efficient but the program flow is more complicated.

Thanks.

Take a look at the following and see if it fits your needs.
I purposely simulate the possibility of http requests failures as well as random delays.
Main idea is schedule all your requests as soon as possible and let the I/O delays create the concurrency of having multiple requests all accessing at the same time.
Using Promise.allSettled to create a barrier so that you can then process after the responses are all collected.
This is still not the most compact approach because I believe it may be possible to even have processing on-going while waiting for some slower requests to get back the complete response.
That will depends if maintaining the order matters to you.

const allfetches = [];

// to satisfy your fragment of codes and logics
const ticker_list_num = 10;
const tickers = [];
for (let i = 0; i < 50; i++) {
  tickers.push(`T${i}`);
}
const url = 'https://www.somewhere.com?tickers=';

// axios mock enough to simulate a http request & response
const axios = {
  get: (url) => {
    return new Promise((resolve,reject)=> {
      if (Math.random() <= 0.6) {
        setTimeout(()=> {
          resolve({
            data: {
              quoteResponse: {
                result: [url]
              }
            }
          });
        }, 1000 + Math.floor(Math.random() * 5000));
      } else {
        setTimeout(()=> {
          reject(`Failed for ${url}`);
        }, 1000 + Math.floor(Math.random() * 5000));
      }
    });
  }
};

// mock of your processing function
function process_json(arr) {
  console.log("Processing...");
  console.log(arr);
};

console.log(`[${new Date().toLocaleTimeString()}] BEFORE START OF WHILE`);
while (tickers.length > 0) {
  // var ticker_list = [];
  // "let" is a more modern JS scoping technique
  // in fact in this case, even "const" will work
  const ticker_list = [];
  
  while (ticker_list.length <= ticker_list_num && 
         tickers.length > 0) {
      ticker_list.push(tickers[0]);
      tickers.splice(0,1)
  }
  
  const my_url = url + ticker_list.join(',');
  
  // instead of performing the work here
  // just simply queue it and let event driven
  // works for you
  //const download_eod = async () => {
  //    try {
  //        const response = await axios.get(my_url);
  //        const body = response.data;
  //        array = body.quoteResponse.result;
  //        return array;
  //    }
  //    catch (error) {
  //        throw error;
  //    }
  //};
  //download_eod().then(array => process_json(array));

  allfetches.push(axios.get(my_url));
}
console.log(`[${new Date().toLocaleTimeString()}] DONE WITH ALL REQUESTS ISSUANCE`);

console.log(`[${new Date().toLocaleTimeString()}] START OF PROMISE SETUP`);
Promise.allSettled(allfetches).then((results) => {
  results.forEach((result) => {
    console.log(`[${new Date().toLocaleTimeString()}] PROCESS RESPONSE`);
    if (result.status == "fulfilled") {
      const response = result.value;
      const body = response.data;
      const array = body.quoteResponse.result;
      process_json(array);
    } else { //if (result.status == "reject")
      console.log(result);
    }
  })
});
console.log(`[${new Date().toLocaleTimeString()}] END OF PROMISE SETUP`);

Here is a possible output

[2:35:17 PM] BEFORE START OF WHILE
[2:35:17 PM] DONE WITH ALL REQUESTS ISSUANCE
[2:35:17 PM] START OF PROMISE SETUP
[2:35:17 PM] END OF PROMISE SETUP
Hint: hit control+c anytime to enter REPL.
[2:35:23 PM] PROCESS RESPONSE
Processing...
[
  'https://www.somewhere.com?tickers=T0,T1,T2,T3,T4,T5,T6,T7,T8,T9,T10'
]
[2:35:23 PM] PROCESS RESPONSE
{
  status: 'rejected',
  reason: 'Failed for https://www.somewhere.com?tickers=T11,T12,T13,T14,T15,T16,T17,T18,T19,T20,T21'
}
[2:35:23 PM] PROCESS RESPONSE
{
  status: 'rejected',
  reason: 'Failed for https://www.somewhere.com?tickers=T22,T23,T24,T25,T26,T27,T28,T29,T30,T31,T32'
}
[2:35:23 PM] PROCESS RESPONSE
{
  status: 'rejected',
  reason: 'Failed for https://www.somewhere.com?tickers=T33,T34,T35,T36,T37,T38,T39,T40,T41,T42,T43'
}
[2:35:23 PM] PROCESS RESPONSE
Processing...
[ 'https://www.somewhere.com?tickers=T44,T45,T46,T47,T48,T49' ]

Feel free to run the codes at https://replit.com/languages/nodejs

 

[davidktw]

Then if you don't care about the order, you can simply process as soon as the response for any requests comes in. Like the earlier codes, just issue all together. Let the callback handle the processing when the response for each request is received.

const allfetches = [];

// to satisfy your fragment of codes and logics
const ticker_list_num = 10;
const tickers = [];
for (let i = 0; i < 50; i++) {
  tickers.push(`T${i}`);
}
const url = 'https://www.somewhere.com?tickers=';

// axios mock enough to simulate a http request & response
const axios = {
  get: (url) => {
    return new Promise((resolve,reject)=> {
      if (Math.random() <= 0.6) {
        setTimeout(()=> {
          resolve({
            data: {
              quoteResponse: {
                result: [url]
              }
            }
          });
        }, 1000 + Math.floor(Math.random() * 5000));
      } else {
        setTimeout(()=> {
          reject(`Failed for ${url}`);
        }, 1000 + Math.floor(Math.random() * 5000));
      }
    });
  }
};

// mock of your processing function
function process_json(arr) {
  console.log(`[${new Date().toLocaleTimeString()}] Processing ...`);
  console.log(arr);
};

console.log(`[${new Date().toLocaleTimeString()}] BEFORE START OF WHILE`);
while (tickers.length > 0) {
  // var ticker_list = [];
  // "let" is a more modern JS scoping technique
  // in fact in this case, even "const" will work
  const ticker_list = [];
  
  while (ticker_list.length <= ticker_list_num && 
         tickers.length > 0) {
      ticker_list.push(tickers[0]);
      tickers.splice(0,1)
  }
  
  const my_url = url + ticker_list.join(',');

  axios.get(my_url).then((result) => {
    const response = result;
    const body = response.data;
    const array = body.quoteResponse.result;
    process_json(array);
  }).catch((result) => {
    console.log(`[${new Date().toLocaleTimeString()}] Received failure ...`);
    console.log(result);
  });
}
console.log(`[${new Date().toLocaleTimeString()}] DONE WITH ALL REQUESTS ISSUANCE`);

Possible output:

[2:51:41 PM] BEFORE START OF WHILE
[2:51:41 PM] DONE WITH ALL REQUESTS ISSUANCE
Hint: hit control+c anytime to enter REPL.
[2:51:42 PM] Received failure ...
Failed for https://www.somewhere.com?tickers=T44,T45,T46,T47,T48,T49
[2:51:43 PM] Received failure ...
Failed for https://www.somewhere.com?tickers=T11,T12,T13,T14,T15,T16,T17,T18,T19,T20,T21
[2:51:44 PM] Processing ...
[
  'https://www.somewhere.com?tickers=T33,T34,T35,T36,T37,T38,T39,T40,T41,T42,T43'
]
[2:51:45 PM] Processing ...
[
  'https://www.somewhere.com?tickers=T0,T1,T2,T3,T4,T5,T6,T7,T8,T9,T10'
]
[2:51:45 PM] Received failure ...
Failed for https://www.somewhere.com?tickers=T22,T23,T24,T25,T26,T27,T28,T29,T30,T31,T32

Notice how the processing order jumble up, but the benefit is you process them as and when it comes back. This is the more compact approach to have request/response process as soon as possible.

 

[peterchan75]

Then if you don't care about the order, you can simply process as soon as the response for any requests comes in. Like the earlier codes, just issue all together. Let the callback handle the processing when the response for each request is received.

const allfetches = [];

// to satisfy your fragment of codes and logics
const ticker_list_num = 10;
const tickers = [];
for (let i = 0; i < 50; i++) {
  tickers.push(`T${i}`);
}
const url = 'https://www.somewhere.com?tickers=';

// axios mock enough to simulate a http request & response
const axios = {
  get: (url) => {
    return new Promise((resolve,reject)=> {
      if (Math.random() <= 0.6) {
        setTimeout(()=> {
          resolve({
            data: {
              quoteResponse: {
                result: [url]
              }
            }
          });
        }, 1000 + Math.floor(Math.random() * 5000));
      } else {
        setTimeout(()=> {
          reject(`Failed for ${url}`);
        }, 1000 + Math.floor(Math.random() * 5000));
      }
    });
  }
};

// mock of your processing function
function process_json(arr) {
  console.log(`[${new Date().toLocaleTimeString()}] Processing ...`);
  console.log(arr);
};

console.log(`[${new Date().toLocaleTimeString()}] BEFORE START OF WHILE`);
while (tickers.length > 0) {
  // var ticker_list = [];
  // "let" is a more modern JS scoping technique
  // in fact in this case, even "const" will work
  const ticker_list = [];
 
  while (ticker_list.length <= ticker_list_num &&
         tickers.length > 0) {
      ticker_list.push(tickers[0]);
      tickers.splice(0,1)
  }
 
  const my_url = url + ticker_list.join(',');

  axios.get(my_url).then((result) => {
    const response = result;
    const body = response.data;
    const array = body.quoteResponse.result;
    process_json(array);
  }).catch((result) => {
    console.log(`[${new Date().toLocaleTimeString()}] Received failure ...`);
    console.log(result);
  });
}
console.log(`[${new Date().toLocaleTimeString()}] DONE WITH ALL REQUESTS ISSUANCE`);

Possible output:

[2:51:41 PM] BEFORE START OF WHILE
[2:51:41 PM] DONE WITH ALL REQUESTS ISSUANCE
Hint: hit control+c anytime to enter REPL.
[2:51:42 PM] Received failure ...
Failed for https://www.somewhere.com?tickers=T44,T45,T46,T47,T48,T49
[2:51:43 PM] Received failure ...
Failed for https://www.somewhere.com?tickers=T11,T12,T13,T14,T15,T16,T17,T18,T19,T20,T21
[2:51:44 PM] Processing ...
[
  'https://www.somewhere.com?tickers=T33,T34,T35,T36,T37,T38,T39,T40,T41,T42,T43'
]
[2:51:45 PM] Processing ...
[
  'https://www.somewhere.com?tickers=T0,T1,T2,T3,T4,T5,T6,T7,T8,T9,T10'
]
[2:51:45 PM] Received failure ...
Failed for https://www.somewhere.com?tickers=T22,T23,T24,T25,T26,T27,T28,T29,T30,T31,T32

Notice how the processing order jumble up, but the benefit is you process them as and when it comes back. This is the more compact approach to have request/response process as soon as possible.

I think I have fixed the problem using Promise.all with return array from async function.
Thanks for all your help.

let promises = [];
while (tickers.length > 0) {
        var ticker_list = [];
        while (ticker_list.length <= ticker_list_num && tickers.length > 0) {
            ticker_list.push(tickers[0]);
            tickers.splice(0,1)
        }
        var my_url = url + ticker_list.join(',');
        promises.push(eod_download(my_url));
}
Promise.all(promises)
    .then((results) => {
        output_eod_to_file(results)
    })
    .catch((err) => console.log(err));

async function eod_download(my_url) {
    try {
        var eod_array = [];
        const response = await axios.get(my_url);
        const body = response.data;
        array = body.quoteResponse.result;
        eod_array = process_json(array);
        return (eod_array);
    } 
    catch (error) {
        throw error;
    }

}

 

[davidktw]

I think I have fixed the problem using Promise.all with return array from async function.
Thanks for all your help.

let promises = [];
while (tickers.length > 0) {
        var ticker_list = [];
        while (ticker_list.length <= ticker_list_num && tickers.length > 0) {
            ticker_list.push(tickers[0]);
            tickers.splice(0,1)
        }
        var my_url = url + ticker_list.join(',');
        promises.push(eod_download(my_url));
}
Promise.all(promises)
    .then((results) => {
        output_eod_to_file(results)
    })
    .catch((err) => console.log(err));

async function eod_download(my_url) {
    try {
        var eod_array = [];
        const response = await axios.get(my_url);
        const body = response.data;
        array = body.quoteResponse.result;
        eod_array = process_json(array);
        return (eod_array);
    }
    catch (error) {
        throw error;
    }

}

You may want to find out the differences between Promise.all() and Promise.allSettled(). If you know the differences and you find Promise.all() is what you want, do go ahead.

Also you will want to read up on the differences between let and var, the way you are mixing them up is out for scoping problems unless you know EXACTLY what you are doing. Please read up on the history of "var" scoping in Javascript and find out why "let" is encouraged and how "const" is later used.

https://replit.com/@davidktw/UnwieldySalmonLinuxkernel

var i;

// section A
console.log("=".repeat(30));
console.log("SECTION A");
console.log("=".repeat(30));

try { console.log("Ax=" + A); }
catch (e) { console.log(e); }

i = 0;
while (i <= 0) {
  i++;
  var A = 1;
  try { console.log("Ay=" + A); }
  catch (e) { console.log(e); }
}

try { console.log("Az=" + A); }
catch (e) { console.log(e); }

// section B
console.log("=".repeat(30));
console.log("SECTION B");
console.log("=".repeat(30));

try { console.log("Bx=" + B); }
catch (e) { console.log(e); }

i = 0;
while (i <= 0) {
  i++;
  let B = 1;
  try { console.log("By=" + B); }
  catch (e) { console.log(e); }
}

try { console.log("Bz=" + B); }
catch (e) { console.log(e); }

// section C & D
console.log("=".repeat(30));
console.log("SECTION C&D");
console.log("=".repeat(30));

try { console.log("Cx=" + C); }
catch (e) { console.log(e); }
try { console.log("Dx=" + D); }
catch (e) { console.log(e); }

if (1 > 0) {
  if (1 > 0) {
 
    var C = 123;
    let D = 345;
  
    try { console.log("Cy=" + C); }
    catch (e) { console.log(e); }
    try { console.log("Dy=" + D); }
    catch (e) { console.log(e); }
  }
}

try { console.log("Cz=" + C); }
catch (e) { console.log(e); }
try { console.log("Dz=" + D); }
catch (e) { console.log(e); }

// section E & F
console.log("=".repeat(30));
console.log("SECTION E&F");
console.log("=".repeat(30));

try { console.log("Ex=" + E); }
catch (e) { console.log(e); }
try { console.log("Fx=" + F); }
catch (e) { console.log(e); }

(() => {
  if (1 > 0) {
 
    var E = 246;
    let F = 468;
  
  try { console.log("Ey=" + E); }
  catch (e) { console.log(e); }
  try { console.log("Fy=" + F); }
  catch (e) { console.log(e); }
  }
})();

try { console.log("Ez=" + E); }
catch (e) { console.log(e); }
try { console.log("Fz=" + F); }
catch (e) { console.log(e); }

Output:

==============================
SECTION A
==============================
Ax=undefined
Ay=1
Az=1
==============================
SECTION B
==============================
ReferenceError: B is not defined
    at Object.<anonymous> (/home/runner/v8ssdbhik4o/index.js:27:27)
    at Module._compile (internal/modules/cjs/loader.js:999:30)
By=1
ReferenceError: B is not defined
    at Object.<anonymous> (/home/runner/v8ssdbhik4o/index.js:38:27)
    at Module._compile (internal/modules/cjs/loader.js:999:30)
==============================
SECTION C&D
==============================
Cx=undefined
ReferenceError: D is not defined
    at Object.<anonymous> (/home/runner/v8ssdbhik4o/index.js:48:27)
    at Module._compile (internal/modules/cjs/loader.js:999:30)
Cy=123
Dy=345
Cz=123
ReferenceError: D is not defined
    at Object.<anonymous> (/home/runner/v8ssdbhik4o/index.js:66:27)
    at Module._compile (internal/modules/cjs/loader.js:999:30)
==============================
SECTION E&F
==============================
ReferenceError: E is not defined
    at Object.<anonymous> (/home/runner/v8ssdbhik4o/index.js:74:27)
    at Module._compile (internal/modules/cjs/loader.js:999:30)
ReferenceError: F is not defined
    at Object.<anonymous> (/home/runner/v8ssdbhik4o/index.js:76:27)
    at Module._compile (internal/modules/cjs/loader.js:999:30)
Ey=246
Fy=468
ReferenceError: E is not defined
    at Object.<anonymous> (/home/runner/v8ssdbhik4o/index.js:92:27)
    at Module._compile (internal/modules/cjs/loader.js:999:30)
ReferenceError: F is not defined
    at Object.<anonymous> (/home/runner/v8ssdbhik4o/index.js:94:27)
    at Module._compile (internal/modules/cjs/loader.js:999:30)

 

[peterchan75]

@davidktw,
It looks like I have butchered it.

 

[davidktw]

@davidktw,
It looks like I have butchered it.

It may be alright for you now, but if it gets more complicated, it will be harder to debug at times. Hence it will be good to understand the different scoping in Javascript.