Helping web2py: Bugs, enhancements and documentation

web2py is very welcoming towards bug reports, documentation improvements and enhancements.

Google Group

The main forum for discussing bugs and new features is: web2py-users (URL is https://groups.google.com/forum/#!forum/web2py)

Filing a bug report

Until January 2015, issues were tracked on Google Code, and many historical references in the newsgroup and in old documents will point you there. As part of the migration to GitHub, the project now uses GitHub issues. Only open issues were transferred to GitHub.

If you have found a bug and discussed it on the group, you may be requested to file a bug report by creating an Issue on GitHub.

Please visit the project’s GitHub page and follow the link to Issues.

https://github.com/web2py/web2py

(or directly visit https://github.com/web2py/web2py/issues) .

As usual, good bug reporting means we need to know your operating system, the version of Python and web2py, what you expected to happen and what happened instead. Bonus points for posting a complete code example that points out the issue: a little bit of code is better than a thousand words.

Reporting Security Bugs

When reporting security bugs, be careful about disclosing information which would assist a malicious exploitation. After posting on the Google group or raising an issue on GitHub, you will be contacted by a developer to share information privately.

Contributing code and documentation changes

How the project code is managed

The web2py code-base is managed in a GitHub git repository. The original version control system was Mercurial at Google Code, but active development is on GitHub.

PyDAL: The DAL is now an independent project

An important part of web2py is the Database Abstraction Layer (DAL). In early 2015 this was decoupled into a separate code-base (PyDAL). In terms of git, it is a sub-module of the main repository.

The use of a sub-module requires a one-time use of the --recursive flag for git clone if you are cloning web2py from scratch.

git clone --recursive https://github.com/web2py/web2py.git

If you have an existing repository, the commands below need to be executed at least once:

git submodule update --init --recursive

If you have a folder gluon/dal you should delete it:

rm -r gluon/dal

PyDAL uses a separate stable release cycle to the rest of web2py. PyDAL releases will use a date-naming scheme similar to Ubuntu. Issues related to PyDAL should be reported to its separate repository.

GitHub repositories

The GitHub repositories are:

https://github.com/web2py/web2py

https://github.com/web2py/pydal

web2py has two important versions: the current stable version, and the development snapshot. In git the development snapshot is usually referred to as “master”, but the web2py project tends to say “trunk”, which is the equivalent term from Mercurial and SVN. Please note that “trunk” is USUALLY perfectly fine to use on production, but it’s not a guarantee. Please use tagged releases if you need a stable codebase.

Bug fixes and enhancements go into trunk, so to take quick advantage of a bug fix you will need to use trunk. The best way to do this is to clone the git repository. This is the first step when using code stored on GitHub, and is well documented there. Because releases are tagged, you can switch between the trunk branch (which is ‘master’) and the current release (or any release). This makes the git repository very practical, even in production.

The core git skills you will need are:

1. cloning a repository hosted on GitHub
2. checkout out the master branch
3. checking out a branch based on a tag

These are well covered by GitHub tutorials: see below.

Discussing proposed changes

If the change relates to a bug, the discussion will probably occur on the Issue on GitHub (see above). If the change relates to an enhancement, the developers’ group is the best place initially: web2py-developers group (URL is https://groups.google.com/forum/#!forum/web2py-developers)

Because web2py promises backwards compatibility for non-experimental features, accepting a change into the code base is a promise of maintenance to future users, and this is not a promise than can be made lightly. Less experienced users may discover via discussion that there is an existing method to achieve what they want; unfortunately, the documentation of power features sometimes lags (see improving the documentation below).

When you’re ready to submit a code change, discussion will probably move to the comments attached to the Pull Request (discussed below).

Coding style

Generally, follow PEP8 coding style. http://www.python.org/dev/peps/pep-0008/

There is specific guidance here: web2py style

Documentation style

Documentation style is specified below

Tips on setting up a development environment

Chapter 14 has some tips on using various IDEs with web2py.

Preparation: using GitHub

Since web2py uses GitHub, the overall flow of changes goes like this:

1. You create a clone of web2py on GitHub, which GitHub calls “forking”.
2. You create a clone of your web2py ‘fork’ on your local machine (or wherever you want to make changes)
3. You create a new branch for your changes
4. You work on your new branch and commit
5. You push your changes back to your GitHub fork (git push orgin name_of_new_branch)
6. You make a pull request via GitHub
7. If your pull request is accepted, your commit will be in the master branch of web2py.

At the time of writing, the core git/GitHub techniques are covered here: https://help.GitHub.com/articles/fork-a-repo

However, you need to follow the instructions below on how to create patches which can be cleanly merged.

Please note: the web2py git repository contains as submodule for DAL. This means the initial checkout should use —recursive to initialise the submodule.

Preparation: using travis with your GitHub fork

web2py works with travis, a continuous integration testing service. This means that a commit will trigger all the integrated unit tests.

You can easily add travis to your own GitHub repository to avoid sending patches which will fail travis testing upstream.

Follow the instructions here: http://about.travis-ci.org/docs/user/getting-started/

The four parts of a good quality enhancement

A high-quality enhancement has four parts:

1. The code change
2. Documentating API changes (see API documentation below)
3. Testing for the new functionality
4. Updating the book (which is also maintained at GitHub)

Ensuring a clean patch: using the correct git branch technique

Using git properly is a little more complex in a shared project.

Here are the specific guidelines to make sure your pull request can be applied cleanly.

Firstly, make sure you have a remote repository setup which points to the main web2py GitHub repository. Your remote repository should be called upstream. You only need to do this once.

Adding an upstream repository is covered by the introductory GitHub article linked above, but in case you missed it, you can add upstream like this:

git remote add upstream https://github.com/web2py/web2py/

Next you need to create a branch name for your changes. git encourages lots of branch names, as specific as possible. For web2py, we recommend names like this:

• every bug-fixing commit should come from a branch named “issue/number_of_the_issue_on_github” (like issue/1684)
• every enhancement commit should come in a branch named “enhancement/title_of_the_enhancement” (like enhancement/trapped_links)

The trick to ensure sending a clean PR is to start your branch always from the most recent web2py code, which is on the master branch. If you NEVER work on your repo’s master branch, you can keep it in sync with web2py’s master, and be sure to “play safe” only in your newly-created branches. You can keep in sync your master with web2py’s with these commands:

git checkout master
git fetch upstream
git merge upstream/master

Now, in your local environment, checkout the branch for your changes (while you’re on master)…substitute CHANGE1 for your branch name.

git checkout -b CHANGE1

… Make changes or cherry pick from another local branch. commit if necessary. When you are ready to send your local changes to your web2py fork:

git push origin CHANGE1

Now your branch in on GitHub… go the GitHub website, change to the new branch and do a pull request. GitHub has a “delete branch” button after your pull request is merged or closed. There can be no guarantees, but PRs are usually reviewed within a few days. Most people submitting patches as PRs also update either the issue report or enhancement thread. Now, remember that if you want to start another PR, you MUST return to the master branch on your repository, re-sync, and create a new branch. Let’s suppose you need to create another branch, named CHANGE2

git checkout master
git fetch upstream
git merge upstream/master
# now master is in sync
git checkout -b CHANGE2
...

When you do several commits on the same “feature” branch, the resulting merge from the Pull Request would include every commit you did on that branch. In order to keep the “merge tree” clean, you should rebase your PR in order to contain - ideally - only one commit. In this case, the process of rebasing “rewrites” the history retaining the same changes in code, while “collapses” all those changes in a single commit.

To start the rebase process, while in your branch, do something like

git rebase --interactive HEAD~10

This code will bring up the latest 10 commits: what you want to do is changing “pick” to “squash” in every commit you want to “collapse”, and keep only one of them

Adding tests

Unit tests should be added when an enhancement changes or adds functionality. Tests are python scripts contained in gluon/tests Copy the approach of existing tests. You will notice that tests often need to create something such as table, perform a test of the functionality and check the outcome, and then return the state to before the test (which in this case would mean dropping the table). There is a good reference to testing here: Web2pyslice 1691

http://www.web2pyslices.com/slice/show/1691/help-developers-adding-tests-to-web2py

Running tests

You can run all tests:

python web2py.py --run_system_tests

See the slice referenced above for more information, including running individual tests.

Example unit test

Writing tests is easy. This is an example from gluon/tests/test_dal.py You can see how the test extends a TestCase class. Note how a sufficiently complex table is added, the test performed and output validated, and then the database changes are torn down, leaving the campsite as clean as when you arrived.

class TestVirtualFields(unittest.TestCase):
    def testRun(self):
        db = DAL(DEFAULT_URI, check_reserved=['all'])
        db.define_table('tt', Field('aa'))
        db.commit()
        db.tt.insert(aa="test")
        class Compute:
            def a_upper(row): return row.tt.aa.upper()
        db.tt.virtualfields.append(Compute())
        assert db(db.tt.id>0).select().first().a_upper == 'TEST'
        db.tt.drop()
        db.commit()

Documentation: Updating the book

The book is also in GitHub and the same git workflow can be used. The book source contains sources in various languages, under the sources directory. The content is written in markmin .

A tip: The markmin content of the book uses anchors for links.

To refer to an anchor in another book chapter, write a link like so:

[[markmin ../05#markmin_syntax]]

The book is a web2py app. You may find it convenient to make a GitHub fork of the book, and clone your local repository in the applications directory of a web2py installation. This makes it easy to see your changes rendered in a browser.

Documentation: web2pyslices.com

web2pyslices.com is a resource for recipes, code snippets and samples.

Documentation: API, in-code and docstring documentation

The web2py API documentation is here: http://web2py.readthedocs.org/en/latest/

It uses sphinx, which means it is generated based on documentation included in the source files.

API (i.e. docstrings) and in-code documentation should follow the Google Python Style Guide - Comments

Here is an example from the web2py code-base:

def getcfs(key, filename, filter=None):
    """
Caches the *filtered* file `filename` with `key` until the file is
modified.
Args:
key(str): the cache key
filename: the file to cache
filter: is the function used for filtering. Normally `filename` is a
.py file and `filter` is a function that bytecode compiles the file.
In this way the bytecode compiled file is cached. (Default = None)
This is used on Google App Engine since pyc files cannot be saved.
"""

and an example from Google’s style-guide:

def fetch_bigtable_rows(big_table, keys, other_silly_variable=None):
    """Fetches rows from a Bigtable.
    Retrieves rows pertaining to the given keys from the Table instance
    represented by big_table.  Silly things may happen if
    other_silly_variable is not None.
    Args:
        big_table: An open Bigtable Table instance.
        keys: A sequence of strings representing the key of each table row
            to fetch.
        other_silly_variable: Another optional variable, that has a much
            longer name than the other args, and which does nothing.
    Returns:
        A dict mapping keys to the corresponding table row data
        fetched. Each row is represented as a tuple of strings. For
        example:
        {'Serak': ('Rigel VII', 'Preparer'),
         'Zim': ('Irk', 'Invader'),
         'Lrrr': ('Omicron Persei 8', 'Emperor')}
        If a key from the keys argument is missing from the dictionary,
        then that row was not found in the table.
    Raises:
        IOError: An error occurred accessing the bigtable.Table object.
    """
    pass

Some further web2py-specific notes re API documentation via docstrings:

• “experimental” features should be clearly marked
• sphinx was chosen because it’s largely supported, maintained AND has the added bonus of readthedocs.org
• new features should be clearly marked (i.e. “starting from 2.4.6…”)
• new features should be closely related to the official changelog
• readthedocs.org allows us to build “statically linked” version of the docs for future tagged releases (good for “feature comparison”)
• The web2py developers adopt the Google style guide as it offers the best of sphinx without all the hassle of ReST “decorations”
• until sphinx 1.3 is out, sphinx-napoleon is a requirement for building the docs