Proposal: Set `--experimental-default-type` mode by detecting ESM syntax in entry point

[GeoffreyBooth]

After landing #49869, I opened nodejs/TSC#1445 to discuss when to flip the default value of --experimental-default-type from commonjs to module, which would make Node itself default to ESM whenever the user didn’t make it explicit that they wanted to work in CommonJS by using "type": "commonjs" in package.json, or a .cjs extension, etc. There were concerns raised on that thread around breaking the “loose” files case, where there is no package.json present, and breaking tutorials such as https://nodejs.org/en/docs/guides/getting-started-guide that assume a CommonJS default.

I propose we create --experimental-default-type=detect-module that would function as follows:

• As with --experimental-default-type=commonjs and --experimental-default-type=module, it would only apply to ambiguous cases: a file with no explicit .mjs or .cjs extension, either no package.json or one that lacks a type field, no --input-type or --experimental-default-type=module flags passed, not under node_modules.
• The entry point would be parsed (not evaluated) and we would look for import or export statements. (Not import() expressions that are allowed in CommonJS, not the CommonJS wrapper variables which could be set as globals by user code.) If the file cannot be parsed, error.
• If an import or export statement is found, run the entry point (and the rest of the app) as if --experimental-default-type=module had been passed. Else run as if --experimental-default-type=commonjs had been passed.

This would solve the “loose files” and tutorials cases: ambiguous files would continue to be run as CommonJS. Only files with import or export statements, which currently error if you try to run them, would start to run as ESM. This would solve the goal of flipping, which is to let people start using ESM syntax by default without first opting in somehow.

I implemented big parts of this in 2019 in nodejs/ecmascript-modules#55. It’s very similar to a feature request from 2021, #39353. The main difference between then and now is the existence of --experimental-default-type. I can respond to some of the concerns raised on those earlier issues:

• People won’t understand the distinction between disambiguating CommonJS and ESM, and what this is doing. This is why I called the value detect-module, not auto, and I think “it runs as ESM if import or export statements are found, or as CommonJS otherwise” is simple enough that users will get it.
• CommonJS and ES modules cannot be disambiguated. This is true, but I’m not trying to disambiguate them. There are three sets of files, basically: files that can only run as CommonJS modules, files that can only run as ES modules, and files that can run as either (ambiguous syntax, like console.log(3)). I can’t tell apart the “ambiguous syntax” files from either of the other two, but I don’t have to: those run as CommonJS today, and they can continue to run as CommonJS. That would be the status quo, avoiding a breaking change. I’m only changing how the “runs only as ESM” files are interpreted, which turns a guaranteed error (Unexpected token import) into running code.
• How would files imported by the entry point be interpreted? This is where the existence of --experimental-default-type today helps us out, because now I have a framework to use that defines how all files in various scopes are interpreted. The detection triggers either --experimental-default-type=module or leaves us in the default --experimental-default-type=commonjs, and that’s it.
• What about future syntax, like whatever comes after import attributes? In my PR in 2019, I used Acorn since it’s already a dependency within Node. Acorn throws on any syntax that it doesn’t know, and I think we would likewise error. The error would be informative, instructing users to try again with an explicit marker (type field, file extension).
• What about the performance impact? The entry point is parsed twice, yes, but only the entry point; and only when no explicit marker is present.

@nodejs/loaders @nodejs/tsc

[GeoffreyBooth]

In the TSC meeting today we agreed that at some point, Node needs to run ESM code by default. So currently the options are:

1. Flip the default, to --experimental-default-type=module, and users need to take a minute to update their project to add a type field or explicit file extension or flag to continue running their non-explicit CommonJS code as before.
2. Ship --experimental-default-type=detect-module and set that as the new default, and most (all?) current CommonJS users are fine without needing to make any changes, but ESM syntax works by default without first opting in.

I’m fine with either. In nodejs/TSC#1445 there were people hesitant about the “require some people to make a one-minute change” first option, so I’m curious if they prefer this instead. I’m unaware of other solutions yet.

[JakobJingleheimer]

This sounds like a great feature to me. And one that would have a lot of community support. As a user, it is very frustrating for node to say "I know exactly what is wrong, but too bad".

There may be some technical challenges to facilitate, but I think we can accomplish it.

The only downside I can think of is a performance cost paid only by those who would otherwise hit a brick wall (those who don't need this feature pay no cost); and that seems a better alternative.

[guybedford]

I agree it could well be time to explore this direction, so far as it can fit within the narrow space we've created for it. When previously discussed this was always in the context of being the primary determiner, but having it restricted to the top-level and as an option sounds very interesting to explore further experimentally. I would be weary unflagging something though until all the edge cases have been worked through including things like workspace symlink consistency and the potential publishing footgun.

[ljharb]

Detection isn't reliable, though, because the two parse goals are ambiguous.

[isaacs]

I think this is a good idea. It's kind of rage inducing that node doesn't do this already, tbh.

However, the issue of ambiguous commonjs deps I think needs a subtle change (unless I'm misreading the proposal, and this is already the intent, in which case, just a clarification).

If I set --experimental-default-module-type=detect-module (and, ideally, eventually that's the default), and it sees import, export, or import.meta in the entry point and thus sets "type": "module" effectively, it should only affect modules:

1. Within the current working directory
2. Not within ./node_modules

Ie, it should act as if a file at ./package.json contains "type": "module". But, like in the case where ./package.json contains "type": "module", it should not default the type for any other packages lacking an explicit type that are encountered along the way by virtue of the entry point being detected as type:module. If those dependency packages are detected to be ESM, then yes, treat them as ESM as well. Otherwise, having a file that does import blah from 'old-dep' will try to load old-dep in ESM mode, and blow up when it uses __dirname, require(), and so on.

With this, the ways to trigger ESM are:

1. a package.json file containing "type": "module"
2. an entry point containing code that triggers a SyntaxError outside of a Module context
3. naming the file .mjs
4. explicitly setting --default-type=module

The issue of parsing being fundamentally ambiguous is not really a big issue, imo, because it doesn't have to be 100% accurate in all cases; all it does is add another way to trigger ESM mode, by just using ESM, which is as it should be.

[joyeecheung]

What about the performance impact? The entry point is parsed twice, yes, but only the entry point; and only when no explicit marker is present.

I am fairly certain that it is possible to let V8 return this information after parsing & hit the compilation cache when we parse it for the second time (which is essentially no-op and we pay no extra cost)

[GeoffreyBooth]

Not within ./node_modules

Yes like the other default-type modes, this new one only applies outside of node_modules.

[joyeecheung]

Actually for the use case alone I don't think even Acorn is needed, we can just compile the script via ComepileFunction/ContextifyScript and if there is an syntax error, we check the error message to see if it's caused by import/export (even though that's not quite nice, but we already do something like that for JSON cycle detection anyway). It's likely that a script that uses import uses this very early on and V8 won't parse more than a few lines before it throws, so the overhead should be manageble.

[GeoffreyBooth]

ctually for the use case alone I don’t think even Acorn is needed, we can just compile the script via ComepileFunction/ContextifyScript and if there is an syntax error, we check the error message to see if it’s caused by import/export

That’s clever, and it eliminates the “future syntax” problem. V8 would be able to parse any syntax that V8 can evaluate 😄

[targos]

I really like this idea. Are you already working on an implementation?

[benjamingr]

+1 for this and +1 for letting V8 do the heavy lifting I think that was the reservation last time.

[benjamingr]

To be clear - there are some caveats:

• The goals are ambiguous AND
• modules run in strict mode and scripts run in loose mode by default

This means that if someone has code that works in loose mode but not strict mode like:

with(someObject) {
  // do stuff
}

and they rely on it running in loose mode - running it as a module (because e.g no require) or doing the opposite (defaulting to CJS again, and someone writes code that only works in strict mode) is going to break their code.

I went over the 20 first Google results for Node.js tutorial right now and none of them are affected. Personally I'm not concerned with this issue and think this is a great solution overall to 'how can we migrate to esm without breaking anything' so I'm still heavily +1 on the issue but I just wanted to raise this point.

[targos]

With this proposal, it would only run unambiguous module code as ESM, so with(someObject) {} wouldn't be affected, would it?