A New Task For An Old Server¶

conf.py¶

Next thing is that we switch to a completely new version of the Sphinx configuration conf.py. Note: This file is invariant and exactly the same for ALL projects, no matter whether the rendering is done on the server or locally at home.

conf-2015-10.py is specifically tailored to the new theme and provides all the necessary defaults. It only needs to be changed for bugfixes or further developments. To work locally at home add a MAKEDIR/Overrides.cfg with setting [html_theme_options]docstypo3org to False. Example: Overrides.cfg of the TypoScript Reference.

Party!

• So we are NOT using the old conf.py any more.
• We don’t use t3sphinx any more.
• We don’t use RestTools.git with the exception of the Pygment’s TypoScript Lexer (one file).
• We don’t use Yaml any more. So we don’t need to install the Python PyYAML module. Instead we make use of standard Python modules only.

Saying goodbye to that old t3sphinx module means that we are getting rid of the buggy t3html builder which had caused quite some headaches. So: drop the t3html in the Makefile and use the standard html builder instead. This is exactly what has changed in Makefile-2015-10.

INI-files¶

New: PROJECT/Settings.cfg, MAKEDIR/Defaults.cfg, MAKEDIR/Overrides.cfg

Gone: PROJECT/Settings.yml

The reason that we have configuration files is that we don’t want to have different versions of conf-2015-10.py.

In the documentation project itself we now use an INI-style textfile Settings.cfg instead of the Yaml file Settings.yml.

Compiling the effective settings for Sphinx rendering works like this:

1. conf-2015-10.py comes with sensible defaults
2. The user settings from Settings.cfg are added.
3. IF Defaults.cfg is present then settings from there are added but ONLY IF they don’t exist.
4. In the next step some hardcoded overrides in conf-2015-10.py are applied.
5. Finally Overrides.cfg is applied without further checks and “has the last word”. Note that Overrides.cfg is under control of the administrator.

Tip: The effective settings are logged to LOGDIR/Settings.pprinted.txt

All of the three files Settings.cfg, Defaults.cfg, Overrides.cfg are in INI-style and could have the same contents. INI-file format doesn’t allow all Python constructs that are possible in conf.py. There is a reference file conf-2015-10.Reference.Settings that should visualize what the current conf-2015-10.py file understands.

Other than before array data is now ADDED to what is already existing. This means that an Intersphinx mapping or an extra Sphinx extension can be specified by the user and ADDS to whatelse is defined already.

Sphinx extensions will only be used if they are member of a white-list in conf-2015-10.py.

New: Automatic Defaults for other formats¶

Another new feature is that conf-2015-10.py now automatically defines default settings for the output formats latex_documents, man_pages, texinfo_documents and epub. The required values are derived from the existing ones. This means:

The defaults can of course be overridden by the user in Settings.cfg or by the administrator in Overrides.cfg.

New: WEBROOT/services/ajaxdownloads.php¶

ajaxdownloads.php replaces pdfchoices.php. In the end pdfchoices.php will not be needed any more.

New: WEBROOT/services/ajaxversions.php¶

ajaxversions.php replaces versionchoices.php. In the end versionchoices.php will not be needed any more.

New: USERROOT/scripts/bin/conf-2015-10.py¶

Now we have a single, “one and only” conf.py file that is going to be used for ALL of our Sphinx projects that are using the t3SphinxThemeRtd theme. It doesn’t matter if you’re using the file at home or on the server: Don’t make any modifications, just use it. Customizations are done in the *.cfg INI-files.

And here it is: conf-2015-10.py Preferrably you will want to create a symlink to it: ln -s conf-2015-10.py conf.py.

MAKEDIR/buildsettings.sh  # New! Needed! Is sourced into Bash AND read as INI-file
MAKEDIR/conf.py           # Make it a symlink to *conf-2015-10.py*!
MAKEDIR/Makefile          # In general: just what 'sphinx-quickstart' proposes.
MAKEDIR/Defaults.cfg      # Optional! Not used at the moment.
MAKEDIR/Overrides.cfg     # Optional!

# buildsetting.sh

# absolute path, or relative to conf.py, without suffix (.rst)
MASTERDOC=../Index

# absolute path, or relative to conf.py
BUILDDIR=./build

# absolute path, or relative to conf.py
LOGDIR=./_not_versioned

New: USERROOT/scripts/bin/cron_rebuild-2015-10.sh¶

cron_rebuild-2015-10.sh replaces cron_rebuild.sh.

Please compare the new file with the old one to find out about the differences. .. note:

- Please keep BINDIRECTORY a plain assignment to facilitate debugging of the script

- The Latex target_name is now set to the constant 'PROJECT', which always
  gives *PROJECT.tex*

New: USERROOT/scripts/config/Makefile-2015-10¶

Makefile-2015-10 replaces Makefile. In the end the older Makefile will not be needed any more.

Here we only have a tiny difference: As html builder we have reverted to the standard html builder and dropped the buggy t3html builder.

New: PROJECT/Documentation/Settings.cfg¶

Example: Settings.cfg

Settings.cfg is a text file in INI-style. It is easy to use and secure. Details about the syntax are given here.. Settings.cfg replaces the Yaml file Settings.yml we have been using up to now. Yaml isn’t used any in the build chain.

Relevant rules for Settings.cfg in TYPO3 context:

1. There needs to be a section [general] in Settings.cfg.

2. It is expected that the documentation project owner supplies these settings in the [genera] section:

   • project: A line with the title of your project
   • version: Should be a version number like 1.2
   • release: Should be a release number like 1.2.3
   • t3author: As line with author information
   • copyright: Notes about the copyright
   • description: A possibly multiline but still short description of you project.

   The user is not restricted to these settings. All simple definitions that would have otherwise gone into conf.py can be given here.

Please refer to this REFERENCE OF Settings.cfg to find out what’s possible.

In the end Settings.yml will not be needed any more.

New: MAKEDIR/buildsettings.sh¶

MAKEDIR/buildsettings.sh replaces MAKEDIR/cron_rebuild.conf.

Example:

# buildsettings.sh

# absolute path, or relative to conf.py, without suffix (.rst)
MASTERDOC=../ExtbaseFluid.git/Documentation/Index

# absolute path, or relative to conf.py
LOGDIR=.

PROJECT=t3extbasebook
VERSION=latest

# Where to publish documentation
BUILDDIR=/home/user/public_html/typo3cms/ExtbaseFluidBook/latest

# If GITURL is empty then GITDIR is expected to be "ready" to be processed
GITURL=https://github.com/TYPO3-Documentation/TYPO3CMS-Book-ExtbaseFluid.git
GITDIR=/home/user/HTDOCS/github.com/TYPO3-Documentation/TYPO3/Book/ExtbaseFluid.git
GITBRANCH=latest

# Path to the documentation within the Git repository
T3DOCDIR=$GITDIR/Documentation

# Packaging information
PACKAGE_ZIP=0
PACKAGE_KEY=typo3cms.books.t3extbasebook
PACKAGE_LANGUAGE=default

The insider will notice that nothing much has changed. Only two settings are new: MASTERDOC and LOGDIR. Note that both can be in absolute form or relative to the location of conf.py. For local rendering only MASTERDOC, BUILDDIR and LOGDIR are needed.

In the end cron_rebuild.conf will not be needed any more.

New: MAKEDIR/Overrides.cfg¶

Please read In words: What’s happening?.

1 Add PROJECT/Settings.cfg¶

You may take the Settings.cfg file of the TypoScript Reference as a template and example.

2 Update the MAKEDIR¶

Example:

1. Copy this codeblock as a whole into your clipboard:

   # cd /path/to/MAKEDIR
   
   # make backup
   mv  conf.py            OLD-conf.py
   mv  cron_rebuild.conf  OLD-cron_rebuild.conf
   mv  cron_rebuild.sh    OLD-cron_rebuild.sh
   mv  Makefile           OLD-Makefile
   
   # find old MASTERDOC value
   LINE=$(cat OLD-conf.py | grep relpath_to_master_doc\']\ \=)
   MASTERDOC=$(echo $LINE | awk '{print $(NF)}' | sed s/[\"\']//g)
   
   # find old LOGDIR value
   LINE=$(cat OLD-conf.py | grep relpath_to_logdir\']\ \=)
   LOGDIR=$(echo $LINE | awk '{print $(NF)}' | sed s/[\"\']//g)
   
   # make buildsettings.sh
   # Verify MASTERDOC and LOGDIR afterwards!
   echo "# buildsettings.sh"         >  buildsettings.sh
   echo ""                           >> buildsettings.sh
   echo "# absolute path, or relative to conf.py, without suffix (.rst)" >> buildsettings.sh
   echo "MASTERDOC=$MASTERDOC/Index" >> buildsettings.sh
   echo ""                           >> buildsettings.sh
   echo "# absolute path, or relative to conf.py" >> buildsettings.sh
   echo "LOGDIR=$LOGDIR"             >> buildsettings.sh
   echo ""                           >> buildsettings.sh
   cat  OLD-cron_rebuild.conf        >> buildsettings.sh
   
   # Switch to version 2015-10
   ln -s /home/mbless/scripts/bin/cron_rebuild-2015-10.sh cron_rebuild.sh
   ln -s /home/mbless/scripts/bin/conf-2015-10.py         conf.py
   ln -s /home/mbless/scripts/config/Makefile-2015-10     Makefile
   
   # schedule a rebuild
   rm build.checksum; php request_rebuild.php
   
   # Be happy!
   
   # clean up more?!:
   rm 10* 20* 30* warnings.txt
   rm dirs-of-last-build.txt
   rm included-files-check.log.txt
   
   ls -la
   cat buildsettings.sh | grep MASTERDOC
   
   #
   

2. Go to a MAKEDIR in old style on server srv123 and insert your clipboard at the commandline.

3. You may want to have a look at your files with ls -la and cat buildsettings.sh.

4. Don’t forget to add the Settings.cfg file to each branch of your project! As an example you can look at the Settings.cfg file of the TypoScript Reference

5. Wait for the next rendering!

3 Supply a _make folder¶

As a service for users that want to render the documentation project locally at home provide a PROJECT/Documentation/_make folder. For this to do simply copy the complete \_make folder of the TypoScript reference <>.

It is a clever idea to check whether the conf-2015-10.py in there is still the latest version of conf-2015-10.py. Otherwise download the latest version of *conf-2015-10.py*.

Reasons¶

Easiness: INI-files are human-friendly and easy to use. You can insert comments much better. The format is widely spread and much older. In general it’s just about writing key value pairs as assignment. Compare with Rfc-822 (1982). It is a lot less frightning to non programmers than Yaml files. It is the same format that Sphinx itself uses for the configuration of themes. A PHP parser is available. A Python parser is available.

Security: INI-files are secure, Yaml-files can be dangerous. See:

• Don’t use pickle. Don’t use pickle. Don’t use pickle.

• Security explanation Sphinx docs

  [1] It is not an executable Python file, as opposed to conf.py, because that would pose an unnecessary security risk if themes are shared.

Maintenance costs To make Yaml secure an extra Python module is needed which we would have to create and maintain ourselves. It needs to be documented and people have to install it.

Aside: It was me myself who proposed the Yaml format back then in 2012. It was a decision I regretted soon. But until recently I didn’t have a better idea.

I had taken security measures in the Yaml script in t3sphinx. But that code has never been approved by somebody else and I wouldn’t stand in for it with my life.

Migration Issues¶

For new documentation it is easy to start with a PROJECT/Settings.cfg right away.

For existing documentation there are two ways that come into my mind. Maybe there are more.

1. We could implement an automatic creation of the PROJECT/Settings.cfg file on the server and read from PROJECT/Settings.yml. This means that we at least have to have our own MakeYamlSecure module on the server. Or can we use PHP for it without running into security issues?
2. Writing an INI-file if you have a Yaml file already is trivial. And how many developers are actually affected? So another solution would be: Yes, DO render the documentation even if Settings.cfg is missing. And insert a <a href="link/to/SettingsCfgHowTo">((missing))</a> where otherwise title, author and copyright appear.

I would prefer solution two as

• I’m sure documentation maintainers are happier with that
• it will soon lead to a transition everywhere
• it requires the least effort on all sides.

At the moment rendering stops when Settings.cfg is missing with the new build chain. So I still would have to change conf-2015-10.py for that.

More thoughts: When people switch to Settings.cfg its a good moment to have them check their entries, add a description setting and update the copyright setting.