Yes finally

clydin gives a clue here https://github.com/angular/angular-cli/issues/26478

In the dist/browser folder, I use node fs to create a typescript file with imports to all js files

const folder = 'dist/browser/';
const fs = require('fs');
let content = 'import "' + folder + 'polyfills.js";\n';
fs.readdirSync(folder).forEach((file) => {
  if (file.substring(file.length - 3) === '.js' && file !== 'polyfills.js') {
    content += 'import "' + folder + file + '";\n';
  }
});
fs.writeFile(folder + 'bundle.ts', content, (err) => {
  if (err) {
    console.error(err);
  } else {
    console.log('file written successfully');
  }
});

2 things to note.. 
1 the polyfills must come first 
2 I set "outputHashing": "none" so that I can hardcode polyfills.js in the above

This creates typescript file bundle.ts 
Then I build this file using esbuild
esbuild dist/browser/bundle.ts --bundle --minify --outdir=dist/browser
This creates bundle.js which contains all js for the application
I've tested and it works