unified

Interface for parsing, inspecting, transforming, and serializing content through syntax trees

Downloads in past

Stats

StarsIssuesVersionUpdatedCreatedSize
unified
3,727410.1.2a year ago8 years agoMinified + gzip package size for unified in KB

Popular Searches

Readme

!Buildbuild-badgebuild !Coveragecoverage-badgecoverage !Downloadsdownloads-badgedownloads !Sizesize-badgesize !Sponsorssponsors-badgecollective !Backersbackers-badgecollective !Chatchat-badgechat
unified is an interface for processing text using syntax trees. It’s what powers remarkremark (Markdown), retextretext (natural language), and rehyperehype (HTML), and allows for processing between formats.

Intro

unified enables new exciting projects like Gatsby to pull in Markdown, MDX to embed JSX, and Prettier to format it. It’s used in about 700k projects on GitHub and has about 35m downloads each month on npm: you’re probably using it. Some notable users are Node.js, Vercel, Netlify, GitHub, Mozilla, WordPress, Adobe, Facebook, Google, and many more.
  • To read about what we are up to, follow us Twitter
  • For a less technical and more practical introduction to unified, visit
[`unifiedjs.com`][site] and peruse its [Learn][] section
Get help on [Discussions][chat]!
  • Check out Contribute below to find out how to help out, or become a
backer or sponsor on [OpenCollective][collective]

Sponsors

Support this effort and give back by sponsoring on OpenCollectivecollective!

Vercel

 Motif

 HashiCorp

 American Express

 GitBook
Gatsby

 Netlify

 Coinbase

 ThemeIsle

 Expo

 Boost Note

 Holloway

You?

Install

This package is ESM only: Node 12+ is needed to use it and it must be imported instead of required.
npm install unified 
## Use

```js
import {unified} from 'unified'
import remarkParse from 'remark-parse'
import remarkRehype from 'remark-rehype'
import rehypeDocument from 'rehype-document'
import rehypeFormat from 'rehype-format'
import rehypeStringify from 'rehype-stringify'
import {reporter} from 'vfile-reporter'

unified()
  .use(remarkParse)
  .use(remarkRehype)
  .use(rehypeDocument, {title: '👋🌍'})
  .use(rehypeFormat)
  .use(rehypeStringify)
  .process('# Hello world!')
  .then(
    (file) => {
      console.error(reporter(file))
      console.log(String(file))
    },
    (error) => {
      // Handle your error here!
      throw error
    }
  )

Yields:
no issues found

<!doctype html>
<html lang="en">
  <head>
    <meta charset="utf-8">
    <title>👋🌍</title>
    <meta name="viewport" content="width=device-width, initial-scale=1">
  </head>
  <body>
    <h1>Hello world!</h1>
  </body>
</html>

Contents

*   [`processor()`](#processor)
*   [`processor.use(plugin[, options])`](#processoruseplugin-options)
*   [`processor.parse(file)`](#processorparsefile)
*   [`processor.stringify(node[, file])`](#processorstringifynode-file)
*   [`processor.run(node[, file][, done])`](#processorrunnode-file-done)
*   [`processor.runSync(node[, file])`](#processorrunsyncnode-file)
*   [`processor.process(file[, done])`](#processorprocessfile-done)
*   [`processor.processSync(file|value)`](#processorprocesssyncfilevalue)
*   [`processor.data([key[, value]])`](#processordatakey-value)
*   [`processor.freeze()`](#processorfreeze)
*   [`function attacher([options])`](#function-attacheroptions)
*   [`function transformer(node, file[, next])`](#function-transformernode-file-next)

Description

unified is an interface for processing text using syntax trees. Syntax trees are a representation of text understandable to programs. Those programs, called pluginplugins, take these trees and inspect and modify them. To get to the syntax tree from text, there is a parserparser. To get from that back to text, there is a compilercompiler. This is the processprocess of a processor.
| ........................ process ........................... |
| .......... parse ... | ... run ... | ... stringify ..........|

          +--------+                     +----------+
Input ->- | Parser | ->- Syntax Tree ->- | Compiler | ->- Output
          +--------+          |          +----------+
                              X
                              |
                       +--------------+
                       | Transformers |
                       +--------------+
Processors
Every processor implements another processor. To create a processor, call another processor. The new processor is configured to work the same as its ancestor. But when the descendant processor is configured in the future it does not affect the ancestral processor.
When processors are exposed from a module (for example, unified itself) they should not be configured directly, as that would change their behavior for all module users. Those processors are frozenfreeze and they should be called to create a new processor before they are used.
Syntax trees
The syntax trees used in unified are unistunist nodes. A nodenode is a plain JavaScript objects with a type field. The semantics of nodes and format of syntax trees is defined by other projects.
There are several utilitiesunist-utilities for working with nodes.
List of processors
The following projects process different syntax treesyntax-tree formats. They parse text to a syntax tree and compile that back to text. These processors can be used as is, or their parser and compiler can be mixed and matched with unified and plugins to process between different syntaxes.
List of plugins
The below pluginsplugin work with unified, on all syntax treesyntax-tree formats:
— Ignore messages for unchanged lines in Travis
— Enable, disable, and ignore messages
See remarkremark-plugins, rehyperehype-plugins, and retextretext-plugins for their lists of plugins.
File
When processing a document, metadata is often gathered about that document. vfilevfile is a virtual file format that stores data, metadata, and messages about files for unified and its plugins.
There are several utilitiesvfile-utilities for working with these files.
Configuration
Processorsprocessors are configured with pluginplugins or with the datadata method.
Integrations
unified can integrate with the file system with unified-engineengine. CLI apps can be created with unified-argsargs, Gulp plugins with unified-engine-gulpgulp, and Atom Linters with unified-engine-atomatom.
unified-streamstream provides a streaming interface.
Programming interface
The API provided by unified allows multiple files to be processed and gives access to metadata (such as lint messages):
import {unified} from 'unified'
import remarkParse from 'remark-parse'
import remarkPresetLintMarkdownStyleGuide from 'remark-preset-lint-markdown-style-guide'
import remarkRetext from 'remark-retext'
import retextEnglish from 'retext-english'
import retextEquality from 'retext-equality'
import remarkRehype from 'remark-rehype'
import rehypeStringify from 'rehype-stringify'
import {reporter} from 'vfile-reporter'

unified()
  .use(remarkParse)
  .use(remarkPresetLintMarkdownStyleGuide)
  .use(remarkRetext, unified().use(retextEnglish).use(retextEquality))
  .use(remarkRehype)
  .use(rehypeStringify)
  .process('*Emphasis* and _stress_, you guys!')
  .then(
    (file) => {
      console.error(reporter(file))
      console.log(String(file))
    },
    (error) => {
      // Handle your error here!
      throw error
    }
  )

Yields:
1:16-1:24  warning  Emphasis should use `*` as a marker                                  emphasis-marker  remark-lint
1:30-1:34  warning  `guys` may be insensitive, use `people`, `persons`, `folks` instead  gals-man         retext-equality

⚠ 2 warnings

<p><em>Emphasis</em> and <em>stress</em>, you guys!</p>
Processing between syntaxes
Processorsprocessors can be combined in two modes.
Bridge mode transforms the syntax treesyntax-tree from one format (origin) to another (destination). Another processor runs on the destination tree. Finally, the original processor continues transforming the origin tree.
Mutate mode also transforms the syntax tree from one format to another. But the original processor continues transforming the destination tree.
In the previous example (“Programming interface”), remark-retext is used in bridge mode: the origin syntax tree is kept after retextretext is done; whereas remark-rehype is used in mutate mode: it sets a new syntax tree and discards the origin tree.

API

This package exports the following identifiers: unified. There is no default export.

processor()

Processorprocessors describing how to process text.
Returns
Function — New unfrozenfreeze processor that is configured to work the same as its ancestor. When the descendant processor is configured in the future it does not affect the ancestral processor.
Example
The following example shows how a new processor can be created (from the remark processor) and linked to stdin(4) and stdout(4).
import {remark} from 'remark'
import concatStream from 'concat-stream'

process.stdin.pipe(
  concatStream((buf) => {
    process.stdout.write(remark().processSync(buf).toString())
  })
)

processor.use(plugin[, options])

Configureconfiguration the processor to use a pluginplugin and optionally configure that plugin with options.
If the processor is already using this plugin, the previous plugin configuration is changed based on the options that are passed in. The plugin is not added a second time.
Signatures
  • processor.use(plugin[, options])
  • processor.use(preset)
  • processor.use(list)
Parameters
  • plugin (Attacherplugin)
  • options (*, optional) — Configuration for plugin
  • preset (Object) — Object with an optional plugins (set to list),
and/or an optional `settings` object
  • list (Array) — List of plugins, presets, and pairs (plugin and
`options` in an array)
Returns
processor — The processor that use was called on.
Note
use cannot be called on frozenfreeze processors. Call the processor first to create a new unfrozen processor.
Example
There are many ways to pass plugins to .use(). The below example gives an overview.
import {unified} from 'unified'

unified()
  // Plugin with options:
  .use(pluginA, {x: true, y: true})
  // Passing the same plugin again merges configuration (to `{x: true, y: false, z: true}`):
  .use(pluginA, {y: false, z: true})
  // Plugins:
  .use([pluginB, pluginC])
  // Two plugins, the second with options:
  .use([pluginD, [pluginE, {}]])
  // Preset with plugins and settings:
  .use({plugins: [pluginF, [pluginG, {}]], settings: {position: false}})
  // Settings only:
  .use({settings: {position: false}})

processor.parse(file)

Parse text to a syntax treesyntax-tree.
Parameters
Returns
Nodenode — Parsed syntax treesyntax-tree representing file.
Note
parse freezes the processor if not already frozenfreeze.
parse performs the parse phasedescription, not the run phase or other phases.
Example
The below example shows how parse can be used to create a syntax tree from a file.
import {unified} from 'unified'
import remarkParse from 'remark-parse'

const tree = unified().use(remarkParse).parse('# Hello world!')

console.log(tree)

Yields:
{
  type: 'root',
  children: [
    {type: 'heading', depth: 1, children: [Array], position: [Position]}
  ],
  position: {
    start: {line: 1, column: 1, offset: 0},
    end: {line: 1, column: 15, offset: 14}
  }
}

processor.Parser

A parser handles the parsing of text to a syntax treesyntax-tree. Used in the parse phasedescription and called with a string and VFilevfile representation of the text to parse.
Parser can be a function, in which case it must return a Nodenode: the syntax tree representation of the given file.
Parser can also be a constructor function (a function with a parse field, or other fields, in its prototype), in which case it’s constructed with new. Instances must have a parse method that is called without arguments and must return a Nodenode.

processor.stringify(node[, file])

Compile a syntax treesyntax-tree.
Parameters
`vfile()`
Returns
string or Buffer (see notes) — Textual representation of the syntax treesyntax-tree
Note
stringify freezes the processor if not already frozenfreeze.
stringify performs the stringify phasedescription, not the run phase or other phases.
unified typically compiles by serializing: most compilercompilers return string (or Buffer). Some compilers, such as the one configured with rehype-reactrehype-react, return other values (in this case, a React tree). If you’re using a compiler doesn’t serialize, expect different result values. When using TypeScript, cast the type on your side.
Example
The below example shows how stringify can be used to serialize a syntax tree.
import {unified} from 'unified'
import rehypeStringify from 'rehype-stringify'
import {h} from 'hastscript'

const tree = h('h1', 'Hello world!')

const doc = unified().use(rehypeStringify).stringify(tree)

console.log(doc)

Yields:
<h1>Hello world!</h1>

processor.Compiler

A compiler handles the compiling of a syntax treesyntax-tree to text. Used in the stringify phasedescription and called with a Nodenode and VFilefile representation of syntax tree to compile.
Compiler can be a function, in which case it should return a string: the textual representation of the syntax tree.
Compiler can also be a constructor function (a function with a compile field, or other fields, in its prototype), in which case it’s constructed with new. Instances must have a compile method that is called without arguments and should return a string.

processor.run(node[, file][, done])

Run transformerstransformer on a syntax treesyntax-tree.
Parameters
`vfile()`
Returns
Promisepromise if done is not given. The returned promise is rejected with a fatal error, or resolved with the transformed syntax treesyntax-tree.
Note
run freezes the processor if not already frozenfreeze.
run performs the run phasedescription, not other phases.

function done(err[, node, file])

Callback called when transformerstransformer are done. Called with either an error or results.
Parameters
Example
The below example shows how run can be used to transform a syntax tree.
import {unified} from 'unified'
import remarkReferenceLinks from 'remark-reference-links'
import {u} from 'unist-builder'

const tree = u('root', [
  u('paragraph', [
    u('link', {href: 'https://example.com'}, [u('text', 'Example Domain')])
  ])
])

unified()
  .use(remarkReferenceLinks)
  .run(tree)
  .then(
    (changedTree) => console.log(changedTree),
    (error) => {
      // Handle your error here!
      throw error
    }
  )

Yields:
{
  type: 'root',
  children: [
    {type: 'paragraph', children: [Array]},
    {type: 'definition', identifier: '1', title: undefined, url: undefined}
  ]
}

processor.runSync(node[, file])

Run transformerstransformer on a syntax treesyntax-tree.
An error is thrown if asynchronous pluginplugins are configured.
Parameters
`vfile()`
Returns
Nodenode — Transformed syntax treesyntax-tree.
Note
runSync freezes the processor if not already frozenfreeze.
runSync performs the run phasedescription, not other phases.

processor.process(file[, done])

Processdescription the given filefile as configured on the processor.
Parameters
Returns
Promisepromise if done is not given. The returned promise is rejected with a fatal error, or resolved with the processed filefile.
The parsed, transformed, and compiled value is exposed on file.valuevfile-value or file.result (see notes).
Note
process freezes the processor if not already frozenfreeze.
process performs the parse, run, and stringify phasesdescription.
unified typically compiles by serializing: most compilercompilers return string (or Buffer). Some compilers, such as the one configured with rehype-reactrehype-react, return other values (in this case, a React tree). If you’re using a compiler that serializes, the result is available at file.value. Otherwise, the result is available at file.result.
Example
The below example shows how process can be used to process a file, whether transformers are asynchronous or not, with promises.
import {unified} from 'unified'
import remarkParse from 'remark-parse'
import remarkRehype from 'remark-rehype'
import rehypeDocument from 'rehype-document'
import rehypeFormat from 'rehype-format'
import rehypeStringify from 'rehype-stringify'

unified()
  .use(remarkParse)
  .use(remarkRehype)
  .use(rehypeDocument, {title: '👋🌍'})
  .use(rehypeFormat)
  .use(rehypeStringify)
  .process('# Hello world!')
  .then(
    (file) => console.log(String(file)),
    (error) => {
      // Handle your error here!
      throw error
    }
  )

Yields:
<!doctype html>
<html lang="en">
  <head>
    <meta charset="utf-8">
    <title>👋🌍</title>
    <meta name="viewport" content="width=device-width, initial-scale=1">
  </head>
  <body>
    <h1>Hello world!</h1>
  </body>
</html>

function done(err, file)

Callback called when the processdescription is done. Called with a fatal error, if any, and a filefile.
Parameters
Example
The below example shows how process can be used to process a file, whether transformers are asynchronous or not, with a callback.
import {unified} from 'unified'
import remarkParse from 'remark-parse'
import remarkGithub from 'remark-github'
import remarkStringify from 'remark-stringify'
import {reporter} from 'vfile-reporter'

unified()
  .use(remarkParse)
  .use(remarkGithub)
  .use(remarkStringify)
  .process('@unifiedjs')
  .then(
    (file) => {
      console.error(reporter(file))
      console.log(String(file))
    },
    (error) => {
      // Handle your error here!
      throw error
    }
  )

Yields:
no issues found

[**@unifiedjs**](https://github.com/unifiedjs)

processor.processSync(file|value)

Processdescription the given filefile as configured on the processor.
An error is thrown if asynchronous pluginplugins are configured.
Parameters
Returns
(VFilevfile) — Processed filefile
The parsed, transformed, and compiled value is exposed on file.valuevfile-value or file.result (see notes).
Note
processSync freezes the processor if not already frozenfreeze.
processSync performs the parse, run, and stringify phasesdescription.
unified typically compiles by serializing: most compilercompilers return string (or Buffer). Some compilers, such as the one configured with rehype-reactrehype-react, return other values (in this case, a React tree). If you’re using a compiler that serializes, the result is available at file.value. Otherwise, the result is available at file.result.
Example
The below example shows how processSync can be used to process a file, if all transformers are synchronous.
import {unified} from 'unified'
import remarkParse from 'remark-parse'
import remarkRehype from 'remark-rehype'
import rehypeDocument from 'rehype-document'
import rehypeFormat from 'rehype-format'
import rehypeStringify from 'rehype-stringify'

const processor = unified()
  .use(remarkParse)
  .use(remarkRehype)
  .use(rehypeDocument, {title: '👋🌍'})
  .use(rehypeFormat)
  .use(rehypeStringify)

console.log(processor.processSync('# Hello world!').toString())

Yields:
<!doctype html>
<html lang="en">
  <head>
    <meta charset="utf-8">
    <title>👋🌍</title>
    <meta name="viewport" content="width=device-width, initial-scale=1">
  </head>
  <body>
    <h1>Hello world!</h1>
  </body>
</html>

processor.data([key[, value]])

Configureconfiguration the processor with information available to all pluginplugins. Information is stored in an in-memory key-value store.
Typically, options can be given to a specific plugin, but sometimes it makes sense to have information shared with several plugins. For example, a list of HTML elements that are self-closing, which is needed during all phasesdescription of the process.
Signatures
  • processor = processor.data(key, value)
  • processor = processor.data(values)
  • value = processor.data(key)
  • info = processor.data()
Parameters
  • key (string, optional) — Identifier
  • value (*, optional) — Value to set
  • values (Object, optional) — Values to set
Returns
  • processor — If setting, the processor that data is called on
  • value (*) — If getting, the value at key
  • info (Object) — Without arguments, the key-value store
Note
Setting information cannot occur on frozenfreeze processors. Call the processor first to create a new unfrozen processor.
Example
The following example show how to get and set information:
import {unified} from 'unified'

const processor = unified().data('alpha', 'bravo')

processor.data('alpha') // => 'bravo'

processor.data() // => {alpha: 'bravo'}

processor.data({charlie: 'delta'})

processor.data() // => {charlie: 'delta'}

processor.freeze()

Freeze a processor. Frozen processors are meant to be extended and not to be configured directly.
Once a processor is frozen it cannot be unfrozen. New processors working the same way can be created by calling the processor.
It’s possible to freeze processors explicitly by calling .freeze(). Processors freeze implicitly when .parse()parse, .run()run, .runSync()run-sync, .stringify()stringify, .process()process, or .processSync()process-sync are called.
Returns
processor — The processor that freeze was called on.
Example
The following example, index.js, shows how rehype prevents extensions to itself:
import {unified} from 'unified'
import remarkParse from 'rehype-parse'
import remarkStringify from 'rehype-stringify'

export const rehype = unified().use(remarkParse).use(remarkStringify).freeze()

The below example, a.js, shows how that processor can be used and configured.
import {rehype} from 'rehype'
import rehypeFormat from 'rehype-format'
// …

rehype()
  .use(rehypeFormat)
  // …

The below example, b.js, shows a similar looking example that operates on the frozen rehype interface because it does not call rehype. If this behavior was allowed it would result in unexpected behavior so an error is thrown. This is invalid:
import {rehype} from 'rehype'
import rehypeFormat from 'rehype-format'
// …

rehype
  .use(rehypeFormat)
  // …

Yields:
~/node_modules/unified/index.js:426
    throw new Error(
    ^

Error: Cannot call `use` on a frozen processor.
Create a new processor first, by calling it: use `processor()` instead of `processor`.
    at assertUnfrozen (~/node_modules/unified/index.js:426:11)
    at Function.use (~/node_modules/unified/index.js:165:5)
    at ~/b.js:6:4

Plugin

Plugins configureconfiguration the processors they are applied on in the following ways:
[*compiler*][compiler], or configuring [*data*][data]

Plugins are a concept. They materialize as attacherattachers.
Example
move.js:
export function move(options = {}) {
  const {extname} = options

  if (!extname) {
    throw new Error('Missing `extname` in options')
  }

  return transformer

  function transformer(tree, file) {
    if (file.extname && file.extname !== extname) {
      file.extname = extname
    }
  }
}

index.md:
# Hello, world!

index.js:
import {unified} from 'unified'
import remarkParse from 'remark-parse'
import remarkRehype from 'remark-rehype'
import rehypeStringify from 'rehype-stringify'
import {toVFile} from 'to-vfile'
import {reporter} from 'vfile-reporter'
import {move} from './move.js'

unified()
  .use(remarkParse)
  .use(remarkRehype)
  .use(move, {extname: '.html'})
  .use(rehypeStringify)
  .process(toVFile.readSync('index.md'))
  .then(
    (file) => {
      console.error(reporter(file))
      toVFile.writeSync(file) // Written to `index.html`.
    },
    (error) => {
      // Handle your error here!
      throw error
    }
  )

Yields:
index.md: no issues found

…and in index.html:
<h1>Hello, world!</h1>

function attacher([options])

Attachers are materialized pluginplugins. An attacher is a function that can receive options and configuresconfiguration the processor.
Attachers change the processor, such as the parserparser, the compilercompiler, configuring datadata, or by specifying how the syntax treesyntax-tree or filefile are handled.
Context
The context object (this) is set to the processor the attacher is applied on.
Parameters
  • options (*, optional) — Configuration
Returns
transformertransformer — Optional.
Note
Attachers are called when the processor is frozenfreeze, not when they are applied.

function transformer(node, file[, next])

Transformers handle syntax treesyntax-trees and filefiles. A transformer is a function that is called each time a syntax tree and file are passed through the run phasedescription. If an error occurs (either because it’s thrown, returned, rejected, or passed to nextnext), the process stops.
The run phase is handled by troughtrough, see its documentation for the exact semantics of these functions.
Parameters
Returns
  • void — If nothing is returned, the next transformer keeps using same tree.
  • Error — Fatal error to stop the process
  • node (Nodenode) — New syntax treesyntax-tree.
If returned, the next transformer is given this new tree
  • Promise — Returned to perform an asynchronous operation.
The promise **must** be resolved (optionally with a [`Node`][node]) or
rejected (optionally with an `Error`)

function next(err[, tree[, file]])

If the signature of a transformertransformer includes next (the third argument), the transformer may perform asynchronous operations, and must call next().
Parameters
If given, the next transformer is given this new tree
If given, the next transformer is given this new file

Preset

Presets are sharable configurationconfiguration. They can contain pluginsplugin and settings.
Example
preset.js:
import remarkPresetLintRecommended from 'remark-preset-lint-recommended'
import remarkPresetLintConsistent from 'remark-preset-lint-consistent'
import remarkCommentConfig from 'remark-comment-config'
import remarkToc from 'remark-toc'
import remarkLicense from 'remark-license'

export const preset = {
  settings: {bullet: '*', emphasis: '*', fences: true},
  plugins: [
    remarkPresetLintRecommended,
    remarkPresetLintConsistent,
    remarkCommentConfig,
    [remarkToc, {maxDepth: 3, tight: true}],
    remarkLicense
  ]
}

example.md:
# Hello, world!

_Emphasis_ and **importance**.

## Table of contents

## API

## License

index.js:
import {remark} from 'remark'
import {toVFile} from 'to-vfile'
import {reporter} from 'vfile-reporter'
import {preset} from './preset.js'

remark()
  .use(preset)
  .process(toVFile.readSync('example.md'))
  .then(
    (file) => {
      console.error(reporter(file))
      toVFile.writeSync(file)
    },
    (error) => {
      // Handle your error here!
      throw error
    }
  )

Yields:
example.md: no issues found

example.md now contains:
# Hello, world!

*Emphasis* and **importance**.

## Table of contents

*   [API](#api)
*   [License](#license)

## API

## License

[MIT](license) © [Titus Wormer](https://wooorm.com)

Contribute

See contributing.mdcontributing in unifiedjs/.githubhealth for ways to get started. See support.mdsupport for ways to get help. Ideas for new plugins and tools can be posted in unifiedjs/ideasideas.
A curated list of awesome unified resources can be found in awesome unifiedawesome.
This project has a code of conductcoc. By interacting with this repository, organization, or community you agree to abide by its terms.

Acknowledgments

Preliminary work for unified was done in 2014preliminary for retextretext and inspired by wareware. Further incubation happened in remarkremark. The project was finally externalised in 2015 and published as unified. The project was authored by @wooorm.
Although unified since moved its plugin architecture to troughtrough, thanks to @calvinfo, @ianstormtaylor, and others for their work on wareware, as it was a huge initial inspiration.

License

MITlicense © Titus Wormerauthor