.. Documentation of matholymp static site generation.
Copyright 2014-2020 Joseph Samuel Myers.
This program is free software; you can redistribute it and/or
modify it under the terms of the GNU General Public License as
published by the Free Software Foundation; either version 3 of the
License, or (at your option) any later version.
This program is distributed in the hope that it will be useful, but
WITHOUT ANY WARRANTY; without even the implied warranty of
MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the GNU
General Public License for more details.
You should have received a copy of the GNU General Public License
along with this program. If not, see
.
Additional permission under GNU GPL version 3 section 7:
If you modify this program, or any covered work, by linking or
combining it with the OpenSSL project's OpenSSL library (or a
modified version of that library), containing parts covered by the
terms of the OpenSSL or SSLeay licenses, the licensors of this
program grant you additional permission to convey the resulting
work. Corresponding Source for a non-source form of such a
combination shall include the source code for the parts of OpenSSL
used as well as that of the covered work.
.. _static-site:
Static site generation
======================
Matholymp supports generating a static site with information about
past competitions. This is done with the
:command:`mo-static-generate` script. Before using this script, you
need to set up the directory that will contain the site. This will
contain files :file:`staticsite.cfg` and :file:`page-template`, as
well as a directory :file:`data/` with the information about past
events. Some parts of the site are not generated by this script and
need to be set up manually.
:command:`mo-static-generate` currently expects there to have been at
least one past event. Thus, if setting up a web presence for a new
competition, you should defer running :command:`mo-static-generate`
until after the first event (although other parts of the site can
still be set up).
:command:`mo-static-generate` should be run from the directory
containing the site. It reads the files listed above and generates
other files (overwriting them if they already exist). It does not
delete files even if they were generated by a previous run of
:command:`mo-static-generate` and are no longer generated because of
changed data (in such cases, e.g., if the record for a person was
removed because they were not in fact at the event they were listed
at, the files must be removed manually).
Example files
-------------
The :file:`examples/static-site/` directory in the matholymp source
distribution includes versions of :download:`staticsite.cfg
<../examples/static-site/staticsite.cfg>` and :download:`page-template
<../examples/static-site/page-template>` that may be used as a basis
for configuring the generation of your site.
The example shows a site accessed over HTTPS, since this is a good
idea for the registration system, although all the data on the static
site itself is public. It is expected that if both the static site
and the registration system are used, the registration system uses
URLs of the form :samp:`registration/{year}/` relative to the top of
the static site.
Site design
-----------
It is up to you to determine the visual design of your site and to set
up an appropriate HTML template, using appropriate CSS styles, that
implements this visual design.
If you wish, you can generate :file:`.html` files directly with
:command:`mo-static-generate` and put your complete HTML template
directly in the :file:`page-template` file. Alternatively, you may
wish to use some external templating system to apply your design both
to the files generated by :command:`mo-static-generate` and to the
manually maintained parts of the site. You could, for example,
configure :command:`mo-static-generate` to create :file:`.php` files
(using ``page_suffix = .php`` in :file:`staticsite.cfg`), with
:file:`page-template` containing the PHP boilerplate to include the
bulk of the page template from other files. Or you could post-process
the files generated by :command:`mo-static-generate` rather than using
them directly on the website at all.
You may wish to put extra manually-maintained content on event pages,
such as links to regulations or solutions. To support this,
:file:`staticsite.cfg` specifies ``page_include_extra`` as text that
goes on event pages to include such extra content; ``%(dir)s`` in that
text is replaced by the name of the directory for that event. For
this to be useful, you need some templating system (possibly as
minimal as server-side includes, as shown in the example file) that
can include such extra content when encountering such text.
Similarly, ``scoreboard_include_extra`` is text that goes on
scoreboard pages; ``%(dir)s`` in that text is replaced by the name of
the directory for the event (not for the scoreboard itself).
Your page template should reference a stylesheet which provides styles
whose names are specified in :file:`staticsite.cfg` for various
constructs in the generated pages. The `corresponding styles used for
EGMO `_ may give a useful idea of some
possibilities, although of course you will need to make your styles
fit in with your overall site design.
Your page template, or a manually maintained page, should contain
links to the overall ``countries/``, :samp:`{event}s/` and ``people/``
pages on the site. You also need to create a front page to the site
manually.
Miscellaneous generated files
-----------------------------
:command:`mo-static-generate` generates some files that you may wish
to use in your site, but are not used automatically. These are
generated in the :file:`{event}/auto/` directory.
* :file:`{event}-contact-list{suffix}` is an HTML fragment with
contact details for each event, which you may wish to include in
some manually-maintained page.
* :file:`sidebar-{event}-list{suffix}` is an HTML fragment listing
events in a form that may be suitable to include in a sidebar in
your page template.
* :file:`redirects-{n}` is a file listing redirection rules to
redirect accesses to the registration system for event *n* to the
corresponding pages on the static site, so that public URLs for
registration system pages continue to work after the registration
system is shut down. These rules are in the format used by `Apache
mod_rewrite
`_,
which must thus be enabled in your Apache configuration if you wish
to use such redirects. You will also need to set up Apache to use
these rules.
Data files
----------
:command:`mo-static-generate` generates the site using data files in
the :file:`data/` directory. These are UTF-8 CSV files starting with
a byte order mark. The files for EGMO (`egmos.csv
`_, `papers.csv
`_, `countries.csv
`_, `people.csv
`_) may be useful in
illustrating the format. The first file has a name that depends on
``short_name_url_plural`` in :file:`staticsite.cfg`; the names of the
other files do not depend on the event. The names of some columns in
these files depend on the settings of ``num_key`` and
``official_desc``.
The file :file:`{event}s.csv` has to be maintained manually; this
means entering details of past events, entering details of new events
when announced and putting in medal boundaries after each event. The
other files can be updated during and after each event by the scripts
:command:`mo-static-papers-import` and :command:`mo-static-import`, as
described below.
Some columns in :file:`{event}s.csv` are optional, and are not present
in the example for EGMO but may be added if certain information varies
from event to event so cannot go in :file:`staticsite.cfg`. A column
``Honourable Mentions Available`` may be used to specify whether the
rules of a particular event allowed for Honourable Mentions to be
awarded; a column ``Distinguish Official Countries`` may be used to
indicate whether there was a distinction between official and
unofficial countries that should be reflected in the lists of
countries and results. The ``Age Day Description`` column, present
for EGMO, is also optional.
Creating the other files initially, with details of past events, is
more complicated. You may start with files simply containing the CSV
header row (with BOM). That will suffice to add data for new events
using :command:`mo-static-papers-import` and
:command:`mo-static-import`.
To add data for past events, you need to put it into the correct
format for these CSV files (which includes identifying when the same
person or country was at more than one event and using that to
determine whether to assign new Person Number or Country Number
values); you also need to copy papers and any photos or flags into the
locations given in the relevant CSV columns. Because this depends on
the form in which you have the data for past events, there is no
general solution to this. You might have spreadsheets with data, in
which case it may be appropriate to convert those to CSV format and
then write your own Python script to convert that CSV file into
exactly the right format. You might have web pages that need screen
scraping to extract the data from tables on those pages. For older
events you might have data on paper that needs scanning and careful
proof-reading (in such a case, it would be a good idea to make the
scans available on the site as well, in case any questions arise about
possible mistakes).
It may well be convenient, when adding data for past events, to put it
in the format expected as input to :command:`mo-static-import` and
then use that script to deal with assigning person or country numbers.
This input format is almost the same as the format of the
:file:`countries.csv` and :file:`people.csv` files on the static site;
the script (:download:`matholymp/scripts/mo_static_import.py
<../matholymp/scripts/mo_static_import.py>`) should be examined for
the details of how the formats differ (in particular, the different
meanings of Person Number and Country Number in the input files).
Special prizes are represented in :file:`people.csv` by putting
``Special Prize`` in the ``Extra Awards`` column. If different types
of special prizes are distinguished, this column may give a name or
description of the prize (in general, it can contain a comma-separated
list of extra awards, formatted as if the single row of a CSV file).
.. note::
Matholymp does not currently support results of past events with
partial information (such as total scores without scores on
individual problems, or names not matched to scores). If you would
like to put up such partial results, please contact me so we can
work out how best to handle them. There is no problem, however,
with putting up information where results of contestants are
present but details of leaders and staff are missing, and then
adding the details of those other people later when available.
Static site maintenance tasks
-----------------------------
The following describes how to carry out various maintenance on the
static site. In all cases, the directory where you run the listed
commands should be the top-level directory for the static site,
containing :file:`staticsite.cfg`.
Regeneration
^^^^^^^^^^^^
To regenerate the site (needed after any change to the configuration
or data, including after following any of the instructions below for
other tasks)::
mo-static-generate
When registration opens
^^^^^^^^^^^^^^^^^^^^^^^
Ensure that the relevant event is described in :file:`{event}s.csv`,
and list the number of the event in ``event_active_number`` in
:file:`staticsite.cfg`.
Adding papers
^^^^^^^^^^^^^
To add a particular day's papers (day 1 in this example) to the site,
where :command:`mo-document-generate` was used to produce papers with
a logo on them, rather than papers being printed on paper with a
pre-printed background design:
.. parsed-literal::
mo-static-papers-import --day 1 *input-directory*
where *input-directory* is the directory in which
:command:`mo-document-generate` was run to generate the papers, with
the :file:`data/` subdirectory containing its input CSV files and
:file:`out/` containing the output PDF files. See
:ref:`document-generation` for details of the use of
:command:`mo-document-generate` to generate those files.
To add a particular day's papers (day 2 in this example) to the site,
where papers were printed on paper with a pre-printed background
design, and both versions with and without that design are to be added
to the site:
.. parsed-literal::
mo-static-papers-import --background --day 2 *input-directory*
In both cases, you may omit the :samp:`--day {day}` option if you are
adding all papers at the same time, and must omit it if there is only
one day at the competition.
If the papers being added to the website have corrections relative to
the versions used in the exam, you may wish to edit the Description
column in :file:`papers.csv` to note this.
Adding data after an event
^^^^^^^^^^^^^^^^^^^^^^^^^^
This should be done once the final scores and medal boundaries have
been approved (and once any known corrections to participant names or
other public details have been entered in the registration system).
Ensure that the data for the relevant event in :file:`{event}s.csv` is
complete, including the number of problems, the maximum number of
marks for each problem and the medal boundaries. Choose a directory
(*input-directory* below) outside the website for
:command:`mo-static-import` to download files into. Make sure that
``event_active_number`` is set in :file:`staticsite.cfg`; that is
required for :command:`mo-static-import` to download files
automatically. Then:
.. parsed-literal::
mo-static-import *input-directory*
If this is the first event, at this point you should fill in Country
Number for the host country in :file:`{event}s.csv` before
regenerating the site (the number not having been allocated before
then; this is the only reason it is not currently possible to use
:command:`mo-static-generate` before the first event).
If you wish to use the redirections from the registration system to
the static site, you may add them to your Apache configuration after
regenerating the site. Before doing this, make sure that anyone using
the registration data administratively (e.g., to plan airport
connections for departures) has all the data they need from the site.
It is also possible to run :command:`mo-static-import` with files
downloaded manually from the registration system rather than having it
download them automatically. To do so, download the following files
from the registration system (the lists of countries and people must
be downloaded while not logged in administratively), and put them in
the directory *input-directory*: the list of countries
(:file:`countries.csv`), the list of people (:file:`people.csv`), the
ZIP file of flags (:file:`flags.zip`), the ZIP file of participant
photos (:file:`photos.zip`), the RSS feed of scores (save it as
:file:`scores-rss.xml`). (The ZIP files do not need to be unpacked;
if they are, the directories :file:`flags/` and :file:`photos/` will
be used instead of the ZIP files.) Any files already present in that
directory will be used by :command:`mo-static-import` in preference to
downloading files automatically.
Removing a photo
^^^^^^^^^^^^^^^^
If someone wants their photo removed from the static site, remove the
files from the relevant directory (note that there may be multiple
copies of it, for someone at multiple events, and thumbnail versions
that need to be removed as well) and remove the entry or
entries from the Photo URL column for that person in
:file:`people.csv`, before regenerating the site.
Removing a person or merging two people
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
If it turns out that a person's record on the site (generally, or for
a particular event) should be removed because that person was not
present at the event they are listed at, or if there are multiple
person numbers assigned to a single person at different events, then
the relevant data needs editing directly.
To remove a person, delete the problem rows from :file:`people.csv`.
Remove any corresponding photos for the events that person was in fact
not at. If the person was not at any event, remove their directory
under :file:`people/`. Then regenerate the site.
To merge two (or more) records for a person, decide which person
number is to be canonical for that person. Change any rows in
:file:`people.csv` with the noncanonical person numbers to use the
canonical number. Move any photos from the directories with the
noncanonical numbers to the directory with the canonical number,
update the Photo URL columns accordingly and delete the noncanonical
directory. Regenerate the site. If you wish, also add redirects in
the web server configuration from the URLs of the removed directories
to the canonical directory.
.. note::
If the highest allocated person number is removed, a future use of
:command:`mo-static-import` will reallocate that number to another
person. You can avoid this by temporarily creating a dummy entry
with that number in :file:`people.csv` before running
:command:`mo-static-import`, then removing that entry afterwards.
A future version of matholymp may provide a more automated way of
handling removing and merging people.