2.1   Document classes¶

The classes of DM documentation are:

Requirements and interface control documentation (LPM, LSE)

These documents specify functionality that the Data Management System must deliver to the Project. Similarly, interface control documents are agreements between subsystems on functionality that cross subsystem boundaries. The production of this documentation is described in LPM-19: Change Control Process and its management is described in LPM-51: Document Management Plan.

Change-controlled DM design documentation (LDM)

These documents define the scope and budget of Data Management work that meet the requirements and interface control documentation. DM work is expected to be consistent with these design documents. LDM document authors are responsible for updating design documentation whenever necessitated by revised project requirements, or by evolving implementation choices. DM design documents must be approved by the DM Technical Control Team before becoming the new baseline. These documents are also governed by LPM-51: Document Management Plan. See 7   Change-Controlled Design Documents for how change-controlled documents are handled by the Data Management Documentation Architecture.

Technical notes (DMTN, SQR)

Technical notes are a standardized, durable, document series allowing useful information to be published on demand by any Data Management team member wishing to capture and share knowledge. Common uses of technical notes are descriptions of experiments with code or data, reviews of literature or technologies, usage notes, or technical documents that are too limited to trigger the change-control process. The primary audiences of technical notes are DM team members who build, maintain, or interface with a system and the science community. See 8   Technical Notes for details.

User guides (DMG)

These documentation products describe usage of DM software, platforms, and data products to end-users. Typically, end-users are astronomers in the scientific community. Some user guides may instead be considered internal, such as operational guides or developer guides. The most important aspect of user guides is that they are written intentionally for their intended audience. See 9   User Guides for details.

Validation documents

These documents report the Data Management System’s performance, particularly with respect to key performance metrics defined in LPM-17: Science Requirements Document. Validation documents are primarily generated by automated Quality Control (QC) services, which are part of the SDQA system (see §6 of LDM-151: Data Management Application Design). LSST Systems Engineering is the primary consumer of DM validation documentation in construction and commissioning, though these reports are also valuable to the DM team and the LSST science community.

Publications

Publications—namely journal articles, and conference presentations and proceedings—speak directly to the science community. With respect to the Data Management Documentation Architecture, publications should be a synthesis of DM documentation products (Design Documents, Implementation Technical Notes, Technical Notes, and User Guides). Publications are described in LPM-162: Project Publication Policy.

2.2   Information flow down¶

DM’s documentation taxonomy facilitates a flow of information from research and design, to implementation, and finally to operations.

Figure 1 Idealized information flow across documentation classes.

As Figure 1 illustrates, the scope and functionality of the Data Management System is specified by Requirements Documents. Design documents translate requirements into actionable designs and documentation of system implementations. Designs are reflected in change controlled design documents (LDM), though details can be deferred to technical notes (DMTN, SQR). User guides are written for end users using a combination of information from the design documentation and the implemented software itself. Verification documentation is written as a consequence of testing activities. Finally, scientific publications are written as a holistic synthesis of the entire Data Management System for the community.

Note that this is an idealized linear information flow. Software development work will spur new technical notes that in turn create revise design documentation. However, Figure 1 shows the role of each document class in supporting the Data Management System in reporting research, documenting designs, and documenting for users.

3.1   Role of the DM Documentation Engineer¶

DM members are not alone when documenting their work. The DM Documentation Engineer is responsible for leading the culture, establishing the practices, and building the infrastructure for technical documentation. The Documentation Engineer is part of the Data Management Science Quality and Reliability Engineering (SQuaRE) team under WBS 02C10.5.3.

While the Documentation Engineer writes documentation (for their own work, and when needed for important documentation on behalf of other DM teams), the Documentation Engineer does not by default write all DM documentation content. Instead, the default is for DM subject matter experts—engineers, managers and scientists—to document within their areas of expertise. This policy acknowledges DM as an agile organization where a technical writing team simply cannot scale with the engineering effort. It is more efficient for an engineer to write about what they make while they make it, and for developers to be allowed time to nurture the important skill of producing well-documented software. Since documentation is continuously delivered (see 5   Publishing Platforms), there is opportunity to refine the quality of documentation content. The role of the Documentation Engineer is to empower the DM team to produce quality documentation and efficiently as possible.

The Documentation Engineer fosters DM documentation best practices in four ways:

• By creating and maintaining platforms for publishing and discovering documentation, and ensuring that DM documentation is citeable by the scientific community (see 5   Publishing Platforms).
• By creating and maintaining the tools that convert content into consistently presented websites and artifacts (see 6   Documentation Formats and Generators).
• By helping authors create content that fits into an effective DM-wide information architecture through templates and content strategy.
• By helping authors create documentation with a consistent voice through a style guide and developer education.

3.2   Documentation Working Groups¶

Some documents, and document types, have special working groups that oversee their production. The organization of these working groups is discussed in the individual sections that describe document classes in the DM Documentation Architecture. Whether explicitly stated or not, the Documentation Engineer is available to support all DM documentation projects.

5.1   LSST the Docs¶

LSST the Docs (LTD) is a platform for continuously publishing versioned documentation to the web. LTD is more fully described in SQR-006; please see that technote for details beyond the scope of this summary.

All DM documentation projects maintained in a version control system (specifically, Git with GitHub) are published with LTD. Other documents are published with DocuShare, see below.

5.2   LSST Project DocuShare¶

DocuShare is the LSST Project’s official document repository (see LPM-51: Document Management Plan).

All change controlled design documents (with LDM handles) are deposited in DocuShare once approved by the Change Control Board (CCB; see LPM-19: Change Control Process), even if they are otherwise published through LSST the Docs. The LSST Project considers the version in DocuShare as the official version of a document that reflects a technical, schedule and budget baseline. LSST DocHub links to a document published in both LSST the Docs and DocuShare, with DocuShare being denoted as the baselined version.

Although DocuShare will be used in conjunction with LSST the Docs, some documentation formats are not managed in a version control system (Git) and thus are not publishable through LSST the Docs. These include documents from ‘office’ suites, such as Apple Pages, Google Docs, Dropbox Paper, Microsoft Word and Excel. For such documents, DocuShare is the only publishing platform: a document only in Google Docs, for example, is not yet considered delivered.

5.3   LSST DocHub¶

LSST DocHub is a platform for discovering LSST documentation and other digital resources. DocHub is both a metadata database and API, and also a set of user-facing web applications for searching and browsing LSST documentation. All DM documentation is available through DocHub.

5.4   Citeable documentation¶

LSST Data Management documentation is considered on par with scientific literature. Being close to the implementation, continuously tested, and written by the collective team, DM’s technical documentation is the most accurate and scientifically useful reference for detailed aspects of the Data Management System. To integrate with scientific literature, DM technical documentation is citeable according to the expectations of the astronomy community. This section describes how DM documentation is made citeable through Digital Object Identifiers and registration with the NASA Astrophysics Data System.

6.1   Sphinx¶

Sphinx is the first-class documentation generator for LSST Data Management. All user guides are produced with Sphinx. Although not required, other documentation classes should be preferably produced with Sphinx as well.

Some of the reasons Sphinx was chosen as DM’s documentation generator include:

• Sphinx and reStructuredText are implemented in Python, which matches the DM technology stack.
• Since reStructuredText is plain text, Sphinx projects integrate well with Data Management’s Git-based development workflow.
• Through Python APIs, both Sphinx and the reStructuredText markup language are thoroughly extensible, giving the Documentation Engineer opportunity to build solutions that both improve developer efficiency, and improve the quality of documentation. For example, Sphinx is able to introspect Python code to build API reference documentation.

The DM Documentation Architecture uses two types of Sphinx projects, depending on the document class: single-page projects for narrative documents and multi-page projects for user guides.

6.2   Landing Pages for Alternative Formats¶

Although Sphinx is the preferred DM documentation format, not all Git-backed documentation is produced as a Sphinx project. Some documents are written in LaTeX for legacy reasons or to be compatible with scientific publishers. Jupyter notebooks are also being used for producing documents that are tightly integrated with code and data.

Since they are managed in Git, these document formats are eligible for being published as static websites with LSST the Docs. However, LaTeX documents, Jupyter notebooks, and similar formats, do not necessarily create polished websites that have the look and feel of LSST documentation. Thus the DM Documentation Architecture shims these formats through a landing page framework.

6.3   Formats not Managed in Git¶

All formats previously in this section are published with LSST the Docs, as they are managed in a version control system (specifically, Git with GitHub). This section describes policies for formats not publishable with LSST the Docs.

7.1   Background¶

Change-controlled design documentation defines the scope and budget of the Data Management System. DM’s change-controlled design documents carry “LDM” handles. New and revised LDM documents are submitted to the DM Technical Control Team (TCT) for approval, as described in LDM-294: Data Management Organization and Charter.

7.2   Maintenance¶

Design documents must always be consistent with the system’s implementation since they formally define the Data Management System’s scope and budget. If an implementation exceeds the envelope of the design document, a ticket should be raised for the project management team to review and reconcile the design and implementation.

7.3   Technical Implementation¶

Per LPM-51, all change-controlled documentation is deposited in DocuShare upon acceptance by the TCT.

Some change-controlled documents are only available through DocuShare, such as word-processing files and spreadsheets, and all classified (non-public) documents. In these cases, DocHub indexes and links to the document from DocuShare. These documents are not mirrored on LSST the Docs.

DM members are encouraged to author documents in formats that can be version-controlled in Git and published by LSST the Docs. Single-page Sphinx projects are preferred, and LaTeX documents published with the Landing Page Framework are allowed secondarily. The status of these Git-hosted documents in the change-control process is reflected in its Git branches (and hence editions on LSST the Docs). The master branch (and main LSST the Docs edition) is reserved for change-controlled versions of a document. Ticket and integration branches allow intermediate drafts of the document to be shared. When a document is accepted by the TCT, that version is simultaneously merged to master and archived as a PDF to DocuShare.

To facilitate citation in scientific literature, the accepted versions of un-classified documents are also deposited in a science data archive (such as Zenodo) to be granted a DOI.

LSST DocHub is aware of each version of a document (as published on LSST the Docs, on DocuShare, and Zenodo) and usefully directs a user to the relevant version and citation information.

8.1   Role¶

Technotes are containers for ideas, rather than documentation for products. This distinguishes technotes from user guides.

Technotes can be used to describe results from an experiment (like a scientific paper). They can also be used to investigate technologies or design decisions and suggest a plan. Technotes are primarily a product for DM, not a product by DM. Through technotes, DM captures its research effort to make better implementation choices. In Figure 1, technotes are shown as inputs to design documentation in DM’s idealized information flow.

Another use for technotes is to provide high level overviews of shipped products, pitched primarily to external audiences. These narratives are like the blog posts that tech companies write about their products, and are opportunities to explain the philosophy and design reasoning of DM products. Often the tone of such narratives does not fit in user documentation, but works well in a technote article.

Overall, technotes are catch-all containers for narratives that are not design documentation nor user guides.

Technotes are created and published on-demand by DM team members. There is no approval process for technotes other than the standard DM code review for ticketed work. Through this lack of administrative process, and infrastructural automations (see below), the bar for creating a technote is intentionally low. This is important since capturing more information in a standardized technote format increases the proportion of DM knowledge that is accessible through DocHub.

8.2   Technical Implementation¶

Single-page Sphinx projects are the recommended format for technotes. This format publishes natively to the web, making these documents highly useable in browsing and information hunting contexts. The single-page format is also straightforward to reduce into static, archived documents (like PDFs) that can be cited in scientific literature. The single page format also emulates scientific papers, and is appropriate for a narrative format.

Where it is more convenient to the author, technotes can also be published through alternative formats (such as LaTeX articles and Jupyter Notebook collections) using the Landing Page Framework. Nonetheless, Sphinx projects are recommended over LaTeX projects for a better online reading experience.

8.3   Authorship¶

Technotes have explicit author lists, similar to academic papers. This property raises the profile of individual intellectual contributions, which is important for DM team members who are writing fewer academic papers, yet want to maintain a CV. Associating individuals with technote authorship also builds trust in the content: technote authors are subject matter experts sharing knowledge with DM.

Because technote authors are intellectually responsible for their content, DM team members should always involve the original authors when making a pull request against a technote that is not theirs.

8.4   Maintenance¶

Technotes do not need to be updated. Akin to a traditional paper publication, a technote implicitly represents a state of knowledge at the time of publication. Creating a technote does not oblige the authors to continually re-assess and update the document.

When significant changes occur, authors can add metadata and other notices. For example, if content from a technote has been migrated into a different document (upstream to a design document, or downstream to a user guide), the technote can be marked as deprecated.

A technote can be updated freely by the authors. If new information needs to be documented that fits an existing technotes’ scope, that existing technote can be updated rather than create an entirely new document. This approach reduces the amount of legacy documentation. Archival tools and the LSST the Docs publishing platform maintain older versions of technotes for historical interest.

9.1   Role¶

User guides empower the people who use DM’s software, platforms, and data products. A user guide can contain reference information, conceptual narratives, task guidance, tutorials, and even marketing information. In many cases, a user guide can be the primary web-facing manifestation of a DM product.

User guides can be aimed at internal or external audiences, or both simultaneously. The DM Developer Guide, and back-end operations guides, are examples of internal guides. User guides, of course can also be public facing: documentation of software products, web platforms, and data products. Public documentation also has tremendous value to the team itself. A good example of this is documentation for open source software where the information needs of a consumer are nearly the same as a developer or contributor.

Unlike design documentation, user guides are tightly integrated with a DM product’s implementation and delivery. Consequently, user guides are the most definitive references for what DM products are and how they work. Their content will be used by scientists to create science. These aspects make DM guides ideal primary references for scientific literature. We enable this usage by making individual pages of guides citeable entities (see 6.1.2.4   Citeable content).

9.2   Technical Implementation¶

All DM user guides are created as multi-page Sphinx projects. This format facilitates a topic-based information architecture needed for comprehensive documentation projects.

Given that astronomers will use multiple DM products simultaneously (for example: Pipeline software, DataSpace, and data release), mandating the multi-page Sphinx format unifies the user experience across LSST’s astronomer-facing documentation.

9.3   Organization¶

Every DM software, platform and data product has a corresponding user guide.

Rather than a monolithic DM documentation site, user documentation is partitioned according to the codebase or data product that each guide documents. This approach keeps documentation close to the source so that documentation is versioned and delivered in step with the product itself.

The organization of user guides reflects, and even shapes, the perception of a DM product. A project may have multiple user guides if the user audience is highly disjoint (for example a project may have an internal operations guide that is separate from a guide for consumers). Multiple audiences can share a common user guide if their interests are similar. This is common for open source APIs, like pipelines.lsst.io. Finally, multiple source repositories may share a common user guide project. For example, the Science Pipelines consist of many Git repositories, yet are commonly documented at pipelines.lsst.io since we want to portray the Pipelines as a coherent entity.

Because user guides are websites, links allow independently produced user guides to still act like a consistent, monolithic resource. Instead of duplicating or rephrasing content from a different guide, user guides always link to the most authoritative page.

9.4   User Guide Working Groups¶

User guides are DM products, much like the software, platforms and datasets they accompany. To oversee their delivery, each user guide project is managed by a corresponding working group.

User guide working groups are a mixture of subject matter experts (SMEs; usually the engineers, scientists, or managers directly involved in building a product) with the Documentation Engineer who provides expertise in infrastructure, technical writing, and information architecture. These working groups are structured to solve key management challenges:

• User guides need strong cohesion and editorial curation, especially in their early development. A working group can provide consistent vision to a user guide project.
• DM products are highly technical, and their guides cannot be effectively produced by a separate technical writing team (see 3   Who Writes the Docs). Subject matter experts (SMEs) must be involved in producing documentation for the products they build. SMEs in the working group ensure that the user guide is technically appropriate and complete. SMEs in the working group can also liaison with the rest of the engineering team to manage documentation contributions.
• Individual user guides exist within a larger environment of LSST documentation. The Documentation Engineer embedded in the working group develops the common infrastructure necessary for producing and publishing content. The Documentation Engineer also contributes knowledge in areas of information architecture and technical writing. Being shared across user guide working groups, the Documentation Engineer provides consistency and quality to the LSST documentation experience.

Each user guide working group’s composition will be unique and tuned to the project’s needs. However, the following roles should be filled, possibly by the same or multiple people:

• Subject matter expert who leads the curriculum development of the user guide.
• T/CAM who is able to schedule effort for all engineers that may need to contribute documentation content.
• The Documentation Engineer who provides documentation infrastructure and provides advice on content (technical writing) and organization (information architecture).

Again, user guide working groups play a leadership role in documentation delivery. The engineering and scientific teams who build a project will be responsible for producing most of a user guide’s content, especially reference content. The Documentation Engineer will also contribute critical (highly used) and complex content pieces.

9.5   Maintenance¶

User guides are continuously delivered in step with product development.

As APIs change or are added, software developers must update the corresponding reference documentation. This process is convenient for developers since reference documentation is typically extracted from source code itself. Reference documentation writing is expected to be part of all software development tickets.

Tutorial and conceptual documentation is more expensive to produce than reference documentation, and is typically written in tickets separate from software development. API changes may break conceptual or tutorial documentation. Where possible, the software development ticket’s scope should including fixing incompatibilities in the documentation. Where the changes are too numerous, the outdated documentation should still be identified and excluded from documentation builds, and a follow-up documentation ticket should created and scheduled.