{%html:
%}{{!"quick_intro"}< Introduction}{%html:
%}{{!"writing-ppxs"}Writing PPXs >}{%html:
%} {0 How It Works} {1 General Concepts} {2 The Driver} [ppxlib] sits in between the PPXs authors and the compiler toolchain. For the PPX author, it provides an API to define the transformation and register it to [ppxlib]. Then, all registered transformations can be turned into a single executable, called the {e driver}, that is responsible for applying all the transformations. The driver will be called by the compiler. The PPX authors register their transformations using the {{!Ppxlib.Driver.register_transformation}[Driver.register_transformation]} function, as explained in the {{!"writing-ppxs"}Writing PPXs} section. The different arguments of this function corresponds to the {{!"derivers-and-extenders"}different kinds} of PPXs supported by [ppxlib] or the {{!driver_execution}phase}, at which time they will be executed. The driver is created by calling either {{!Ppxlib.Driver.standalone}[Driver.standalone]} or {{!Ppxlib.Driver.run_as_ppx_rewriter}[Driver.run_as_ppx_rewriter]}. Note that when used through Dune, none of these functions will need to be called by the PPX author. As we will see, Dune will be responsible for generating the driver after all required PPXs from different libraries have been registered. These functions will interpret the command line arguments and start the rewriting accordingly. The {{!Ppxlib.Driver.standalone}[Driver.standalone]} function creates an executable that parses an OCaml file, transforms it according to the registered transformations, and outputs the transformed file. This makes it suitable for use with the [-pp] {{:https://v2.ocaml.org/releases/5.0/htmlman/comp.html#s:comp-options}option} of the OCaml compiler. It is a preprocessor for sources and is standalone in the sense that it can be called independently from the OCaml compiler (e.g., it includes an OCaml parser). On the other hand, the {{!Ppxlib.Driver.run_as_ppx_rewriter}[Driver.run_as_ppx_rewriter]}-generated driver is a proper PPX, as it will read and output a {{!Ppxlib.Parsetree}[Parsetree]} marshalled value directly. This version is suitable for use with the [-ppx] {{:https://v2.ocaml.org/releases/5.0/htmlman/comp.html#s:comp-options}option} of the OCaml compiler, as well as any tool that requires control of parsing the file. For instance, {{:https://ocaml.github.io/merlin/}Merlin} includes an OCaml parser that tries hard to recover from errors in order to generate a valid AST most of the time. Several arguments can be passed to the driver when executing it. Those arguments can also be easily passed using Dune, as explained in its {{:https://dune.readthedocs.io/en/stable/concepts.html#preprocessing-with-ppx-rewriters}manual}. PPX authors can add arguments to their generated drivers using {{!Ppxlib.Driver.add_arg}[Driver.add_arg]}. Here are the default arguments for respectively {{!Ppxlib.Driver.standalone}[standalone]} and {{!Ppxlib.Driver.run_as_ppx_rewriter}[run_as_ppx_rewriter]} generated drivers: {%html:
Standalone driver%} {v driver.exe [extra_args] [] -as-ppx Run as a -ppx rewriter (must be the first argument) --as-ppx Same as -as-ppx -as-pp Shorthand for: -dump-ast -embed-errors --as-pp Same as -as-pp -o Output file (use '-' for stdout) - Read input from stdin -dump-ast Dump the marshaled ast to the output file instead of pretty-printing it --dump-ast Same as -dump-ast -dparsetree Print the parsetree (same as ocamlc -dparsetree) -embed-errors Embed errors in the output AST (default: true when -dump-ast, false otherwise) -null Produce no output, except for errors -impl Treat the input as a .ml file --impl Same as -impl -intf Treat the input as a .mli file --intf Same as -intf -debug-attribute-drop Debug attribute dropping -print-transformations Print linked-in code transformations, in the order they are applied -print-passes Print the actual passes over the whole AST in the order they are applied -ite-check (no effect -- kept for compatibility) -pp Pipe sources through preprocessor (incompatible with -as-ppx) -reconcile (WIP) Pretty print the output using a mix of the input source and the generated code -reconcile-with-comments (WIP) same as -reconcile but uses comments to enclose the generated code -no-color Don't use colors when printing errors -diff-cmd Diff command when using code expectations (use - to disable diffing) -pretty Instruct code generators to improve the prettiness of the generated code -styler Code styler -output-metadata FILE Where to store the output metadata -corrected-suffix SUFFIX Suffix to append to corrected files -loc-filename File name to use in locations -reserve-namespace Mark the given namespace as reserved -no-check Disable checks (unsafe) -check Enable checks -no-check-on-extensions Disable checks on extension point only -check-on-extensions Enable checks on extension point only -no-locations-check Disable locations check only -locations-check Enable locations check only -apply Apply these transformations in order (comma-separated list) -dont-apply Exclude these transformations -no-merge Do not merge context free transformations (better for debugging rewriters). As a result, the context-free transformations are not all applied before all impl and intf. -cookie NAME=EXPR Set the cookie NAME to EXPR --cookie Same as -cookie -deriving-keep-w32 {impl|intf|both} Do not try to disable warning 32 for the generated code -deriving-disable-w32-method {code|attribute} How to disable warning 32 for the generated code -type-conv-keep-w32 {impl|intf|both} Deprecated, use -deriving-keep-w32 -type-conv-w32 {code|attribute} Deprecated, use -deriving-disable-w32-method -deriving-keep-w60 {impl|intf|both} Do not try to disable warning 60 for the generated code -unused-code-warnings {true|false|force} Allow ppx derivers to enable unused code warnings (default: false) -unused-type-warnings {true|false|force} Allow unused type warnings for types with [@@deriving ...] (default: false) -help Display this list of options --help Display this list of options v} {%html:
%} and {%html:
Ppx rewriter driver%} {v driver.exe [extra_args] -loc-filename File name to use in locations -reserve-namespace Mark the given namespace as reserved -no-check Disable checks (unsafe) -check Enable checks -no-check-on-extensions Disable checks on extension point only -check-on-extensions Enable checks on extension point only -no-locations-check Disable locations check only -locations-check Enable locations check only -apply Apply these transformations in order (comma-separated list) -dont-apply Exclude these transformations -no-merge Do not merge context free transformations (better for debugging rewriters). As a result, the context-free transformations are not all applied before all impl and intf. -cookie NAME=EXPR Set the cookie NAME to EXPR --cookie Same as -cookie -help Display this list of options --help Display this list of options v} {%html:
%} {3:exception_handling Exception handling} In general, raising an exception in a registered transformation will make the ppxlib driver crash with an uncaught exception error. However, when spawned with the [-embed-errors] or [-as-ppx] flags (that's the case when Merlin calls the driver) the ppxlib driver still handles a specific kind of exception: Located exceptions. They have type {{!Ppxlib.Location.exception-Error}[Location.Error]} and contain enough information to display a located error message. During its {{!page-driver.driver_execution}execution}, the driver will run many different rewriters. In the case described above, it will catch any located exception thrown by a rewriter. When catching an exception, it will collect the error in a list, take the last valid AST (the one that was given to the raising rewriter) and continue its execution from there. At the end of the rewriting process, the driver will prepend all collected errors to the beginning of the AST, in the order in which they appeared. The same mechanism applies for the {{!"derivers-and-extenders"}context-free rewriters}: if any of them raises, the error is collected, the part of the AST that the rewriter was responsible to rewrite remains unmodified, and the {{!"context-free-phase"}context-free phase} continues. {2 Cookies} Cookies are values that are passed to the driver via the command line, or set as side effects of transformations, which can be accessed by the transformations. They have a name to identify them and a value consisting of an OCaml expression. The module to access cookies is {{!Ppxlib.Driver.Cookies}[Driver.Cookies]}. {2 Integration With Dune} The {{:https://dune.build}Dune} build system is well integrated with the [ppxlib] mechanism of registering transformations. In every [dune] file, Dune will read the set of PPXs that are to be used (i.e. the PPXs in `(preprocess (pps ))`). For a given set of rewriters, it will generate a driver using {{!Ppxlib.Driver.run_as_ppx_rewriter}[Driver.run_as_ppx_rewriter]} that contains all registered transformations. Using a single driver for multiple transformations from multiple PPXs ensures better composition semantics and improves the speed of the combined transformations. Moreover, [ppxlib] communicates with Dune through [.corrected] files to allow for promotion, for instance when using {{!"writing-ppxs"."inlining-transformations"}[[@@deriving_inline]]}. A PPX author can also generate its own promotion suggestion using the {{!Ppxlib.Driver.register_correction}[Driver.register_correction]} function. {1:compat_mult_ver Compatibility With Multiple OCaml Versions} One of the important issues with working with the {{!Ppxlib.Parsetree}[Parsetree]} is that the API is not stable. For instance, in the {{:https://ocaml.org/releases/4.13.0}OCaml 4.13 release}, the following {{:https://github.com/ocaml/ocaml/pull/9584/files#diff-ebecf307cba2d756cc28f0ec614dfc57d3adc6946eb4faa9825eb25a92b2596d}two} {{:https://github.com/ocaml/ocaml/pull/10133/files#diff-ebecf307cba2d756cc28f0ec614dfc57d3adc6946eb4faa9825eb25a92b2596d}changes} were made to the {{!Ppxlib.Parsetree}[Parsetree]} type. Although they are small changes, they may break any PPX that is written to directly manipulate the (evolving) type. This instability causes an issue with maintenance. PPX authors wish to maintain a single version of their PPX, not one per OCaml version, and ideally not have to update their code when an irrelevant (for them) field is changed in the {{!Ppxlib.Parsetree}[Parsetree]}. [ppxlib] helps to solve both issues. The first one, having to maintain a single PPX version working for every OCaml version, is done by migrating the {{!Ppxlib.Parsetree}[Parsetree]}. The PPX author only maintains a version working with the latest version, and the [ppxlib] driver will convert the values from one version to another. For example, say a deriver is applied in the context of OCaml 4.08. After the 4.08 {{!Ppxlib.Parsetree}[Parsetree]} has been given to it, the [ppxlib] driver will migrate this value into the latest {{!Ppxlib.Parsetree}[Parsetree]} version, using the {!Astlib} module. The "latest" here depends on the version of [ppxlib], but at any given time, the latest released version of [ppxlib] will always use the latest released version of the {{!Ppxlib.Parsetree}[Parsetree]}. After the migration to the latest {{!Ppxlib.Parsetree}[Parsetree]}, the driver runs all transformations on it, which ends with a rewritten {{!Ppxlib.Parsetree}[Parsetree]} of the latest version. However, since the context of rewriting is OCaml 4.08 (in this example), the driver needs to migrate back the rewritten {{!Ppxlib.Parsetree}[Parsetree]} to an OCaml 4.08 version. Again, [ppxlib] uses the {!Astlib} module for this migration. Once the driver has rewritten the AST for OCaml 4.08, the compilation can continue as usual. {1:derivers-and-extenders Context-Free Transformations} [ppxlib] defines several kinds of transformations whose core property is that they can only read and modify the code locally. The parts of the AST given to the transformation are only portions of the whole AST. In this regard, they are usually called {e context-free} transformations. While being not as general-purpose as plain AST transformations, they are more than often sufficient and have many nice properties such as a well-defined semantics for composition. The two most important context-free transformations are {e derivers} and {e extenders}. {2:def_derivers Derivers} A {e deriver} is a context-free transformation that, given a certain structure or signature item, will generate code {e to append after} this item. The given code is never modified. A deriver can be very useful to generate values depending on the structure of a user-defined type, for instance a converter for a type to and from a JSON value. A deriver is triggered by adding an {{:https://v2.ocaml.org/manual/attributes.html}attribute} to a structure or signature item. For instance, the folowing code: {@ocaml[ type t = Int of int | Float of float [@@deriving yojson] let x = ... ]} would be rewritten to: {@ocaml[ type ty = Int of int | Float of float [@@deriving yojson] let ty_of_yojson = ... let ty_to_yojson = ... let x = ... ]} {2:def_extenders Extenders} An {e extender} is a context-free transformation that is triggered on {{:https://v2.ocaml.org/manual/extensionnodes.html}extension nodes}, and that will replace the extension node by some code generated from the extension node's payload. This can be very useful to generate values of a DSL using a more user-friendly syntax, e.g., to generate OCaml values from the JSON syntax. For instance, the following code: {@ocaml[ let json = [%yojson [ { name = "Anne"; grades = ["A"; "B-"; "B+"] } ; { name = "Bernard"; grades = ["B+"; "A"; "B-"] } ] ] ]} could be rewritten into: {@ocaml[ let json = `List [ `Assoc [ ("name", `String "Anne") ; ("grades", `List [`String "A"; `String "B-"; `String "B+"]) ] ; `Assoc [ ("name", `String "Bernard") ; ("grades", `List [`String "B+"; `String "A"; `String "B-"]) ] ] ]} {2 Advantages} There are multiple advantages of using context-free transformations. First, they provide the PPX user a much clearer understanding of the AST parts that will be rewritten, rather than a fully general AST rewriting. Secondly, they provide a much better composition semantic, which does not depend on the order. Finally, context-free transformations are applied in a single phase factorising the work for all transformations, resulting in a much faster driver than when combining multiple, whole AST transformations. More details on the execution of this phase are given in its {{!"context-free-phase"}dedicated section.} See the {{!"writing-ppxs"}Writing PPXs} section for how to define derivers and extenders. {1:driver_execution The Execution of the Driver} The actual rewriting of the AST is done in multiple phases: {ol {- The linting phase} {- The preprocessing phase} {- The first instrumentation phase} {- The context-free phase} {- The global transformation phase} {- The last instrumentation phase} } When registering a transformation through the {{!Ppxlib.Driver.register_transformation}[Driver.register_transformation]} function, the phase in which the transformation has to be applied is specified. The multiplicity of phases is mostly to account for potential constraints on the execution order. However, most of the time there are no such constraints, and in this case, either the {{!"context-free-phase"}context-free} or the {{!"global-transfo-phase"}global transformation phase} should be used. (Note that whenever possible, which should be almost always, context-free transformations are possible and better.) If you register in another phase, be sure to know what you are doing. {2 The Linter Phase} Linters are preprocessors that take as input the whole AST and output a list of "lint" errors. Such an error is of type {{!Ppxlib.Driver.Lint_error.t}[Driver.Lint_error.t]} and includes a string (the error message) and the location of the error. The errors will be reported as preprocessors warnings. This is the first phase, so linting errors can only be reported for code handwritten by the user. An example of a PPX registered in this phase is {{:https://github.com/janestreet/ppx_js_style}ppx_js_style}. {2 The Preprocessing Phase} The preprocessing phase is the first transformation that actually alters the AST. In fact, the property of being the "first transformation applied" is what defines this phase, and [ppxlib] will thus ensure that only one transformation is registered in this phase; otherwise, it will generate an error. You should only register a transformation in this phase if it is really strongly necessary, and you know what you are doing. Your PPX will not be usable at the same time as another one registering a transformation in this phase. An example of a PPX registered in this phase is {{:https://github.com/thierry-martinez/metapp}metapp}. {2 The First Instrumentation Phase} This phase is for transformations that {e need} to be run before the context-free phase. Historically, it was meant for {{:https://en.wikipedia.org/wiki/Instrumentation_(computer_programming)}instrumentation}-related PPXs, hence the name. Unlike the {{!"the-preprocessing-phase"}preprocessing phase}, registering to this phase provides no guarantee that the transformation is run early in the rewriting, as there is no limit in the number of transformations registered in this phase, which are then applied in the alphabetical order by their name. If it is not crucial for a transformation to run before the context-free phase, it should be registered to the {{!"global-transfo-phase"}global transformation phase}. {2:context-free-phase The Context-Free Phase} The execution of all registered context-free rules is done in a single top-down pass through the AST. Whenever the top-down pass encounters a situation that triggers rewriting, the corresponding transformation is called. For instance, when encountering an extension point corresponding to a rewriting rule, the extension point is replaced by the rule's execution, and the top-down pass continues inside the generated code. Similarly, when a deriving attribute is found attached to a structure or signature item, the result of the deriving rule’s application is appended to the AST, and the top-down pass continues in the generated code. Note that the code generation for derivers is applied when "leaving" the AST node, that is when all rewriters have been run. Indeed, a deriver like this: {[ type t = [%my_type] [@@deriving deriver_from_type] ]} would need the information generated by the [my_type] extender to match on the structure of [t]. Also note that in this phase, the execution of the context-free rules are intertwined altogether, and it would not make sense to speak about the order of application, contrary to the next phase. {2:global-transfo-phase The Global Transformation Phase} The global transformation phase is the phase where registered transformations, seen as function from and to the {{!Ppxlib.Parsetree}[Parsetree]}, are run. The applied order might matter and change the outcome, but since [ppxlib] knows nothing about the transformations, the order applied is alphabetical by the transformation's name. {2 The Last Instrumentation Phase} This phase is for global transformation to escape the alphabetical order and be executed as a last phase. For instance, {{:https://github.com/aantron/bisect_ppx}[bisect_ppx]} needs to be executed after all rewriting has occurred. Note that only one global transformation can be executed last. If several transformations rely on being the last transformation, it will be true for only one of them. Thus, only register your transformation in this phase if it is absolutely vital to be the last transformation, as your PPX will become incompatible with any other that registers a transformation during this phase. {%html:
%}{{!"quick_intro"}< Introduction}{%html:
%}{{!"writing-ppxs"}Writing PPXs >}{%html:
%}