qapi: add qapi2texi script

QAPI parser does not).  At present, there is no place where a QAPI

schema requires the use of JSON numbers or null.

=== Comments ===

Comments are allowed; anything between an unquoted # and the following

newline is ignored.  Although there is not yet a documentation

generator, a form of stylized comments has developed for consistently

documenting details about an expression and when it was added to the

schema.  The documentation is delimited between two lines of ##, then

the first line names the expression, an optional overview is provided,

then individual documentation about each member of 'data' is provided,

and finally, a 'Since: x.y.z' tag lists the release that introduced

the expression.  Optional members are tagged with the phrase

'#optional', often with their default value; and extensions added

after the expression was first released are also given a '(since

x.y.z)' comment.  For example:

newline is ignored.

A multi-line comment that starts and ends with a '##' line is a

documentation comment.  These are parsed by the documentation

generator, which recognizes certain markup detailed below.

==== Documentation markup ====

Comment text starting with '=' is a section title:

    # = Section title

Double the '=' for a subsection title:

    # == Subection title

'|' denotes examples:

    # | Text of the example, may span

    # | multiple lines

'*' starts an itemized list:

    # * First item, may span

    #   multiple lines

    # * Second item

You can also use '-' instead of '*'.

A decimal number followed by '.' starts a numbered list:

    # 1. First item, may span

    #    multiple lines

    # 2. Second item

The actual number doesn't matter.  You could even use '*' instead of

'2.' for the second item.

Lists can't be nested.  Blank lines are currently not supported within

lists.

Additional whitespace between the initial '#' and the comment text is

permitted.

*foo* and _foo_ are for strong and emphasis styles respectively (they

do not work over multiple lines). @foo is used to reference a name in

the schema.

Example:

##

# = Section

# == Subsection

#

# Some text foo with *strong* and _emphasis_

# 1. with a list

# 2. like that

#

# And some code:

# | $ echo foo

# | -> do this

# | <- get that

#

##

==== Expression documentation ====

Each expression that isn't an include directive must be preceded by a

documentation block.  Such blocks are called expression documentation

blocks.

The documentation block consists of a first line naming the

expression, an optional overview, a description of each argument (for

commands and events) or member (for structs, unions and alternates),

and optional tagged sections.

FIXME: the parser accepts these things in almost any order.

Optional arguments / members are tagged with the phrase '#optional',

often with their default value; and extensions added after the

expression was first released are also given a '(since x.y.z)'

comment.

A tagged section starts with one of the following words:

"Note:"/"Notes:", "Since:", "Example"/"Examples", "Returns:", "TODO:".

The section ends with the start of a new section.

A 'Since: x.y.z' tagged section lists the release that introduced the

expression.

For example:

##

# @BlockStats:

# @device: #optional If the stats are for a virtual block device, the name

#          corresponding to the virtual block device.

#

    # @stats:  A @BlockDeviceStats for the device.

# @node-name: #optional The node name of the device. (since 2.3)

#

    # @parent: #optional This describes the file block device if it has one.

    #

    # @backing: #optional This describes the backing block device if it has one.

    #           (Since 2.0)

# ... more members ...

#

# Since: 0.14.0

##

{ 'struct': 'BlockStats',

      'data': {'*device': 'str', 'stats': 'BlockDeviceStats',

               '*parent': 'BlockStats',

               '*backing': 'BlockStats'} }

  'data': {'*device': 'str', '*node-name': 'str',

           ... more members ... } }

##

# @query-blockstats:

#

# Query the @BlockStats for all virtual block devices.

#

# @query-nodes: #optional If true, the command will query all the

#               block nodes ... explain, explain ...  (since 2.3)

#

# Returns: A list of @BlockStats for each virtual block devices.

#

# Since: 0.14.0

#

# Example:

#

# -> { "execute": "query-blockstats" }

# <- {

#      ... lots of output ...

#    }

#

##

{ 'command': 'query-blockstats',

  'data': { '*query-nodes': 'bool' },

  'returns': ['BlockStats'] }

==== Free-form documentation ====

A documentation block that isn't an expression documentation block is

a free-form documentation block.  These may be used to provide

additional text and structuring content.

=== Schema overview ===

The schema sets up a series of types, as well as commands and events

that will use those types.  Forward references are allowed: the parser

                           info['parent'], msg)

class QAPIDoc(object):

    class Section(object):

        def __init__(self, name=None):

            # optional section name (argument/member or section name)

            self.name = name

            # the list of lines for this section

            self.content = []

        def append(self, line):

            self.content.append(line)

        def __repr__(self):

            return "\n".join(self.content).strip()

    class ArgSection(Section):

        pass

    def __init__(self, parser, info):

        # self.parser is used to report errors with QAPIParseError.  The

        # resulting error position depends on the state of the parser.

        # It happens to be the beginning of the comment.  More or less

        # servicable, but action at a distance.

        self.parser = parser

        self.info = info

        self.symbol = None

        self.body = QAPIDoc.Section()

        # dict mapping parameter name to ArgSection

        self.args = OrderedDict()

        # a list of Section

        self.sections = []

        # the current section

        self.section = self.body

        # associated expression (to be set by expression parser)

        self.expr = None

    def has_section(self, name):

        """Return True if we have a section with this name."""

        for i in self.sections:

            if i.name == name:

                return True

        return False

    def append(self, line):

        """Parse a comment line and add it to the documentation."""

        line = line[1:]

        if not line:

            self._append_freeform(line)

            return

        if line[0] != ' ':

            raise QAPIParseError(self.parser, "Missing space after #")

        line = line[1:]

        # FIXME not nice: things like '#  @foo:' and '# @foo: ' aren't

        # recognized, and get silently treated as ordinary text

        if self.symbol:

            self._append_symbol_line(line)

        elif not self.body.content and line.startswith("@"):

            if not line.endswith(":"):

                raise QAPIParseError(self.parser, "Line should end with :")

            self.symbol = line[1:-1]

            # FIXME invalid names other than the empty string aren't flagged

            if not self.symbol:

                raise QAPIParseError(self.parser, "Invalid name")

        else:

            self._append_freeform(line)

    def _append_symbol_line(self, line):

        name = line.split(' ', 1)[0]

        if name.startswith("@") and name.endswith(":"):

            line = line[len(name)+1:]

            self._start_args_section(name[1:-1])

        elif name in ("Returns:", "Since:",

                      # those are often singular or plural

                      "Note:", "Notes:",

                      "Example:", "Examples:",

                      "TODO:"):

            line = line[len(name)+1:]

            self._start_section(name[:-1])

        self._append_freeform(line)

    def _start_args_section(self, name):

        # FIXME invalid names other than the empty string aren't flagged

        if not name:

            raise QAPIParseError(self.parser, "Invalid parameter name")

        if name in self.args:

            raise QAPIParseError(self.parser,

                                 "'%s' parameter name duplicated" % name)

        if self.sections:

            raise QAPIParseError(self.parser,

                                 "'@%s:' can't follow '%s' section"

                                 % (name, self.sections[0].name))

        self.section = QAPIDoc.ArgSection(name)

        self.args[name] = self.section

    def _start_section(self, name=""):

        if name in ("Returns", "Since") and self.has_section(name):

            raise QAPIParseError(self.parser,

                                 "Duplicated '%s' section" % name)

        self.section = QAPIDoc.Section(name)

        self.sections.append(self.section)

    def _append_freeform(self, line):

        in_arg = isinstance(self.section, QAPIDoc.ArgSection)

        if (in_arg and self.section.content

                and not self.section.content[-1]

                and line and not line[0].isspace()):

            self._start_section()

        if (in_arg or not self.section.name

                or not self.section.name.startswith("Example")):

            line = line.strip()

        self.section.append(line)

class QAPISchemaParser(object):

    def __init__(self, fp, previously_included=[], incl_info=None):

        self.line = 1

        self.line_pos = 0

        self.exprs = []

        self.docs = []

        self.accept()

        while self.tok is not None:

            info = {'file': fname, 'line': self.line,

                    'parent': self.incl_info}

            if self.tok == '#':

                doc = self.get_doc(info)

                self.docs.append(doc)

                continue

            expr = self.get_expr(False)

            if isinstance(expr, dict) and "include" in expr:

                if len(expr) != 1:

                        raise QAPISemError(info, "Inclusion loop for %s"

                                           % include)

                    inf = inf['parent']

                # skip multiple include of the same file

                if incl_abs_fname in previously_included:

                    continue

                exprs_include = QAPISchemaParser(fobj, previously_included,

                                                 info)

                self.exprs.extend(exprs_include.exprs)

                self.docs.extend(exprs_include.docs)

            else:

                expr_elem = {'expr': expr,

                             'info': info}

                if (self.docs

                        and self.docs[-1].info['file'] == fname

                        and not self.docs[-1].expr):

                    self.docs[-1].expr = expr

                    expr_elem['doc'] = self.docs[-1]

                self.exprs.append(expr_elem)

    def accept(self):

    def accept(self, skip_comment=True):

        while True:

            self.tok = self.src[self.cursor]

            self.pos = self.cursor

            self.val = None

            if self.tok == '#':

                if self.src[self.cursor] == '#':

                    # Start of doc comment

                    skip_comment = False

                self.cursor = self.src.find('\n', self.cursor)

                if not skip_comment:

                    self.val = self.src[self.pos:self.cursor]

                    return

            elif self.tok in "{}:,[]":

                return

            elif self.tok == "'":

            raise QAPIParseError(self, 'Expected "{", "[" or string')

        return expr

    def get_doc(self, info):

        if self.val != '##':

            raise QAPIParseError(self, "Junk after '##' at start of "

                                 "documentation comment")

        doc = QAPIDoc(self, info)

        self.accept(False)

        while self.tok == '#':

            if self.val.startswith('##'):

                # End of doc comment

                if self.val != '##':

                    raise QAPIParseError(self, "Junk after '##' at end of "

                                         "documentation comment")

                self.accept()

                return doc

            else:

                doc.append(self.val)

            self.accept(False)

        raise QAPIParseError(self, "Documentation comment must end with '##'")

#

# Semantic analysis of schema expressions

# TODO fold into QAPISchema

    for expr_elem in exprs:

        expr = expr_elem['expr']

        info = expr_elem['info']

        if 'doc' not in expr_elem:

            raise QAPISemError(info,

                               "Expression missing documentation comment")

        if 'enum' in expr:

            check_keys(expr_elem, 'enum', ['data'], ['prefix'])

            add_enum(expr['enum'], info, expr['data'])

    return exprs

def check_freeform_doc(doc):

    if doc.symbol:

        raise QAPISemError(doc.info,

                           "Documention for '%s' is not followed"

                           " by the definition" % doc.symbol)

    body = str(doc.body)

    if re.search(r'@\S+:', body, re.MULTILINE):

        raise QAPISemError(doc.info,

                           "Free-form documentation block must not contain"

                           " @NAME: sections")

def check_definition_doc(doc, expr, info):

    for i in ('enum', 'union', 'alternate', 'struct', 'command', 'event'):

        if i in expr:

            meta = i

            break

    name = expr[meta]

    if doc.symbol != name:

        raise QAPISemError(info, "Definition of '%s' follows documentation"

                           " for '%s'" % (name, doc.symbol))

    if doc.has_section('Returns') and 'command' not in expr:

        raise QAPISemError(info, "'Returns:' is only valid for commands")

    if meta == 'union':

        args = expr.get('base', [])

    else:

        args = expr.get('data', [])

    if isinstance(args, str):

        return

    if isinstance(args, dict):

        args = args.keys()

    assert isinstance(args, list)

    if (meta == 'alternate'

            or (meta == 'union' and not expr.get('discriminator'))):

        args.append('type')

    for arg in args:

        if arg[0] == '*':

            opt = True

            desc = doc.args.get(arg[1:])

        else:

            opt = False

            desc = doc.args.get(arg)

        if not desc:

            continue

        desc_opt = "#optional" in str(desc)

        if desc_opt and not opt:

            raise QAPISemError(info, "Description has #optional, "

                               "but the declaration doesn't")

        if not desc_opt and opt:

            # silently fix the doc

            # TODO either fix the schema and make this an error,

            # or drop #optional entirely

            desc.append("#optional")

    doc_args = set(doc.args.keys())

    args = set([name.strip('*') for name in args])

    if not doc_args.issubset(args):

        raise QAPISemError(info, "The following documented members are not in "

                           "the declaration: %s" % ", ".join(doc_args - args))

def check_docs(docs):

    for doc in docs:

        for section in doc.args.values() + doc.sections:

            content = str(section)

            if not content or content.isspace():

                raise QAPISemError(doc.info,

                                   "Empty doc section '%s'" % section.name)

        if not doc.expr:

            check_freeform_doc(doc)

        else:

            check_definition_doc(doc, doc.expr, doc.info)

    return docs

#

# Schema compiler frontend

#

class QAPISchema(object):

    def __init__(self, fname):

        try:

            self.exprs = check_exprs(QAPISchemaParser(open(fname, "r")).exprs)

            parser = QAPISchemaParser(open(fname, "r"))

            self.exprs = check_exprs(parser.exprs)

            self.docs = check_docs(parser.docs)

            self._entity_dict = {}

            self._predefining = True

            self._def_predefineds()

#!/usr/bin/env python

# QAPI texi generator

#

# This work is licensed under the terms of the GNU LGPL, version 2+.

# See the COPYING file in the top-level directory.

"""This script produces the documentation of a qapi schema in texinfo format"""

import re

import sys

import qapi

COMMAND_FMT = """

@deftypefn {type} {{}} {name}

{body}

@end deftypefn

""".format

ENUM_FMT = """

@deftp Enum {name}

{body}

@end deftp

""".format

STRUCT_FMT = """

@deftp {{{type}}} {name}

{body}

@end deftp

""".format

EXAMPLE_FMT = """@example

{code}

@end example

""".format

def subst_strong(doc):

    """Replaces *foo* by @strong{foo}"""

    return re.sub(r'\*([^*\n]+)\*', r'@emph{\1}', doc)

def subst_emph(doc):

    """Replaces _foo_ by @emph{foo}"""

    return re.sub(r'\b_([^_\n]+)_\b', r' @emph{\1} ', doc)

def subst_vars(doc):

    """Replaces @var by @code{var}"""

    return re.sub(r'@([\w-]+)', r'@code{\1}', doc)

def subst_braces(doc):

    """Replaces {} with @{ @}"""

    return doc.replace("{", "@{").replace("}", "@}")

def texi_example(doc):

    """Format @example"""

    # TODO: Neglects to escape @ characters.

    # We should probably escape them in subst_braces(), and rename the

    # function to subst_special() or subs_texi_special().  If we do that, we

    # need to delay it until after subst_vars() in texi_format().

    doc = subst_braces(doc).strip('\n')

    return EXAMPLE_FMT(code=doc)

def texi_format(doc):

    """

    Format documentation

    Lines starting with:

    - |: generates an @example

    - =: generates @section

    - ==: generates @subsection

    - 1. or 1): generates an @enumerate @item

    - */-: generates an @itemize list

    """

    lines = []

    doc = subst_braces(doc)

    doc = subst_vars(doc)

    doc = subst_emph(doc)

    doc = subst_strong(doc)

    inlist = ""

    lastempty = False

    for line in doc.split('\n'):

        empty = line == ""

        # FIXME: Doing this in a single if / elif chain is

        # problematic.  For instance, a line without markup terminates

        # a list if it follows a blank line (reaches the final elif),

        # but a line with some *other* markup, such as a = title

        # doesn't.

        #

        # Make sure to update section "Documentation markup" in

        # docs/qapi-code-gen.txt when fixing this.

        if line.startswith("| "):

            line = EXAMPLE_FMT(code=line[2:])

        elif line.startswith("= "):

            line = "@section " + line[2:]

        elif line.startswith("== "):

            line = "@subsection " + line[3:]

        elif re.match(r'^([0-9]*\.) ', line):

            if not inlist:

                lines.append("@enumerate")

                inlist = "enumerate"

            line = line[line.find(" ")+1:]

            lines.append("@item")

        elif re.match(r'^[*-] ', line):

            if not inlist:

                lines.append("@itemize %s" % {'*': "@bullet",

                                              '-': "@minus"}[line[0]])

                inlist = "itemize"

            lines.append("@item")

            line = line[2:]

        elif lastempty and inlist:

            lines.append("@end %s\n" % inlist)

            inlist = ""

        lastempty = empty

        lines.append(line)

    if inlist:

        lines.append("@end %s\n" % inlist)

    return "\n".join(lines)

def texi_body(doc):

    """

    Format the body of a symbol documentation:

    - main body

    - table of arguments

    - followed by "Returns/Notes/Since/Example" sections

    """

    body = texi_format(str(doc.body)) + "\n"

    if doc.args:

        body += "@table @asis\n"

        for arg, section in doc.args.iteritems():

            desc = str(section)

            opt = ''

            if "#optional" in desc:

                desc = desc.replace("#optional", "")

                opt = ' (optional)'

            body += "@item @code{'%s'}%s\n%s\n" % (arg, opt,

                                                   texi_format(desc))

        body += "@end table\n"

    for section in doc.sections:

        name, doc = (section.name, str(section))

        func = texi_format

        if name.startswith("Example"):

            func = texi_example

        if name:

            # FIXME the indentation produced by @quotation in .txt and

            # .html output is confusing

            body += "\n@quotation %s\n%s\n@end quotation" % \

                    (name, func(doc))

        else:

            body += func(doc)

    return body

def texi_alternate(expr, doc):

    """Format an alternate to texi"""

    body = texi_body(doc)

    return STRUCT_FMT(type="Alternate",

                      name=doc.symbol,

                      body=body)

def texi_union(expr, doc):

    """Format a union to texi"""

    discriminator = expr.get("discriminator")

    if discriminator:

        union = "Flat Union"

    else:

        union = "Simple Union"

    body = texi_body(doc)

    return STRUCT_FMT(type=union,

                      name=doc.symbol,

                      body=body)

def texi_enum(expr, doc):

    """Format an enum to texi"""

    for i in expr['data']:

        if i not in doc.args:

            doc.args[i] = ''

    body = texi_body(doc)

    return ENUM_FMT(name=doc.symbol,

                    body=body)