Parser plugins

[natew]

I'm enjoying playing with Babel plugins so far and have been looking at sweet.js as well. Unfortunately using the two together (as I've seen talked in threads here before) isn't great.

How could one implement a similar macro system for JS but using Babel/Acorn rather than Sweet/Esprima? Are there plans for custom syntax support, I would imaging this issue really falls more on the AST generation side, but then I realize Babel already has custom syntax like decorators and JSX.

I guess I'm a little green on how it all works, but curious on plans for this or ways to implement this today. As a minimal example I'd like to do some of the following transformations:

#x = 1

to:

State.set('x', 1)

And also

component XYZ {
  <sortaJSX></sortaJSX>
}

into:

class XYZ extends Some.Component {
  render() {
    return { name: 'sortaJSX' };
  }
}

Just throwing out a couple examples of things that would be really awesome and enabled by macro support. For now I'm hacking together stuff with webpack loaders with regex replace, checking into PEG, and also reading through Babel, but would like to see if there's any interest in this, and would appreciate pointers in the right direction.

[sebmck]

Why do you want macros?

[natew]

To make a DSL for things I'm currently writing out at length. Just to play with adding features to the language. I want to be able to write something that transforms my code like shown in those two examples.

[natew]

Wouldn't macros make your life easier? In that really all a macro is is what you're using to make Babel in the first place, so to open that power up to people would allow anyone with a new spec the ability to implement it themselves, let people try it, and eventually have it submitted back to the core.

I find it very exciting. But from your reply I'm guessing you are against them in some sense. Care to explain? I'm sure there are many angles to this, so I'm curious what your thoughts are.

[sebmck]

How could one implement a similar macro system for JS but using Babel/Acorn rather than Sweet/Esprima?

There's no way at the moment.

I guess I'm a little green on how it all works, but curious on plans for this or ways to implement this today.

There are plans to add something like this, although a macro interface isn't the direction it'll be in. More like just some way to hook into the parser and code gen logic.

Wouldn't macros make your life easier?

Not really. I don't think throwing syntax at something is the solution.

In that really all a macro is is what you're using to make Babel in the first place, so to open that power up to people would allow anyone with a new spec the ability to implement it themselves, let people try it, and eventually have it submitted back to the core.

Are you referring to simply allowing pluggable syntax into the parser? That's different to macros.

I find it very exciting. But from your reply I'm guessing you are against them in some sense. Care to explain? I'm sure there are many angles to this, so I'm curious what your thoughts are.

I think macros are misguided. You're trying to reduce repetition but add an extremely large and convuluted layer of abstraction on top instead. The two examples you provided are pretty minimal. I find that 99% of macros are basically find-and-replace like things where it's a 1:1 mapping, not really powerful or necessary IMO.

Whenever you try and do anything complicated you get a massive clusterfuck of syntax productions which makes things infinitely more complicated than dealing with an AST. Take rest parameters for example, you have to support all the following forms:

function foo(..rest) {}
var foo = function (...rest) {};
var foo = (...rest) => i * i;
var foo = (...rest) => { return i * i };
var foo = { foo(...rest) {} };
class Foo { foo(...rest) {} }

I'm unsure what you're asking for specifically? Just way to define custom syntax or an actual macro interface? Because they're both very different things.

[btholt]

@natew I have a couple of suggestions:

• Try ES6 Macros. This will allow you to bypass dealing with Babel and just deal with Sweet.
• Wait for Babel to settle a bit more. @sebmck has stated that a plugin system is likely in the future.
• If you're feeling adventurous, I imagine it may be possible to rip the parser out of one, replace it with another parser, and then write a glue layer between the two, and then resolve the mess that you'll be left with. That's the only way I can see to actually do what you're asking for.
• Perhaps you could even just write a simple macro layer on top of Acorn. That would seem much simpler than trying to port Sweet/Babel.

All of that said, I'm inclined to agree with @sebmck. While at reddit, we toyed around using some Sweet.js macros with React after Sweet.js got readtables. We wrote a little app in it but we found it to be an absolute rat's nest to debug; not only are you debugging your code but you're debugging your tools too; this added level of complexity made the proposition of going forward with Sweet.js a total nonstarter. We promptly rewrote the little experiment we did in plain JSX.

[natew]

@sebmck I think I didn't clarify really, for me macros vs plugging into parser is all the same, I just was wondering about the ability to do the transforms (result rather than method). I appreciate your reply. In an ideal world I'd be able to design a new language myself, and not have to worry about optimization and all the other things that come with that. But for now there are certainly a few things that I could do with JS today by extending syntax that would vastly simplify what I'm writing and even simplify things in many ways. In fact, I'd want to combine these syntax extensions with restrictions (think: soundscript, remove the bad parts, but today, with Babel plugins).

Either way, it's great to hear that custom parser plugins are on the table. Count this as my 👍 for them.

@btholt Awesome recommendations. I hesitate to go with sweet.js/esprima, given the amazing work @sebmck is doing and support generally around Babel. Though sweet.js pretty much solves my problems today, so perhaps it's the practical solution to go with for now. I hasn't seen the es6 extensions. Your other suggestion of writing a macro layer on Acorn to me would be the next best option.

What I think is this: if you create macros not to ease common tasks but with the intent of drastically simplifying the interface a programmer needs to use (ie: restricting the surface area of JS while moving very few new ideas out there into more declarative interfaces, which could be achieved with a combination of macros and babel plugins), the double edged sword of macros could be balanced well.

To me, React is getting closer to what we need. But there's a lot of boilerplate still. Especially for beginners. JS will move from a scripting language, to an interface for building UI's, and it would be cool to even prototype a system where you can think that way and avoid writing so much boilerplate. Even if it's brittle/hacky, it would help show new ideas and eventually push JS forward much in the way CoffeeScript did.

Anyway, I'm rambling. Appreciate both responses. Let me know what would be the most productive way I can help move things forward.

[btholt]

@natew: Totally on board with what you said. JS is definitely permeating into every device and venue. It's also frustrating to have to remember and deal with a lot of boilerplate. The way I've chosen to deal with it are twofold:

• Sublime snippets. Especially helpful with React. It's nice to just say cwm-Tab and have it autocomplete out to a componentWillMount method. While the boilerplate code is still there, you're not stuck typing it every time.
• A living and enforced code style guide. While something like macros could make it so you couldn't use some parts of the language, a style guide just makes it so you don't. Couple your style guide to the pluggable babel-eslint and git hooks, and you accomplish the same thing: a subset of your choosing of JavaScript that's enforced.

I know this may not be what you're looking for but I wanted to share that I have the same problems/feelings and I feel like those two things helped me solve it for myself satisfactorily. In any case, I'm interested to hear how you solve it and hope it works out. 😄

[monsanto]

@natew Is there anything actionable to do here, or can we close this topic & resume discussion when Babel has better support for your use case?

[natew]

However you run the repo is fine with me, I think I've got a better feel for things. I tweeted to the ticket here to for anyone else to chime in.

[sebmck]

I'll keep this open as a tracking issue for parser support in Babel.