Setting up a new python project with Cookiecutter
=================================================

.. post:: 2020-12-05
    :tags: code-snippets, python
    :author: Adriaan Rol


.. image:: /images/example_project.png
   :width: 1000
   :align: center
   :target: https://gitlab.com/AdriaanRol/example-gitlab-python-project



Every so often, I need to set up a new repository for a python project including a lot of `boilerplate <https://en.wikipedia.org/wiki/Boilerplate_code>`_.
Because you only set it up once I always spend a lot of time googling best practices and reinventing the wheel.
To resolve this once and for all I decided to make a template tailored to my own preferences.

You can find it on GitLab: `gitlab.com/AdriaanRol/cookiecutter-pypackage-gitlab <https://gitlab.com/AdriaanRol/cookiecutter-pypackage-gitlab>`_ along with an `example project <https://gitlab.com/AdriaanRol/example-gitlab-python-project>`_.

Here I documented my best practices and how to use this template.

.. TEASER_END

I host my projects on Gitlab because I find it to be slightly better than Github in pretty much all aspects.
The only thing for which I find Github better is for getting public exposure.

The things I want for any project before I get started are the following.


.. |check_| raw:: html

    <input checked=""  disabled="" type="checkbox">

.. |uncheck_| raw:: html

    <input disabled="" type="checkbox">


Requirements:

* |check_| Hosted on GitLab
* |check_| :code:`setup.py` that is compatible with pypi.
* |check_| Automated test build that runs tests and reports coverage.
* |check_| Sphinx documentation that builds and is hosted on readthedocs.
* |check_| Fancy buttons that show project status.
* |check_| Comes with requirements set up that I like.
* |uncheck_| Coverage reports integrated with GitLab.
* |uncheck_| Set up notification bots for slack.



Step 1. Set up a project using Cookiecutter
-------------------------------------------------------------

Start by `creating a "blank" project <https://gitlab.com/projects/new#blank_project>`_ on GitLab.
Be sure to note down the *name* and *project_slug*, you'll be needing that later.
*Do not initialize the repository with a readme*.
After setting up the project using cookiecutter locally (see below), we'll be following the instructions for pushing an existing folder shown by GitLab.

The next step is to set up the folder structure of the project.
This can be quite a tedious project so we use `Cookiecutter <https://cookiecutter.readthedocs.io/>`_ to do this for us.
Cookiecutter is a framework to generate a folder structure based on a template.
Here, we will not focus on how to use Cookiecutter to create a template but just use an existing template.
I have created a template for this purpose, you can find it `here <https://gitlab.com/AdriaanRol/cookiecutter-pypackage-gitlab/>`_.
Alternatively, you can use the `pypackage template <https://github.com/audreyfeldroy/cookiecutter-pypackage>`_  by `Audrey Feldroy <https://github.com/audreyfeldroy>`_ the creator of Cookiecutter, or the `scientific python cookiecutter <https://nsls-ii.github.io/scientific-python-cookiecutter/preliminaries.html>`_ mostly because of its extensive documentation.
The docs of this project also go into some best practices

.. code-block:: sh

    python3 -m pip install --upgrade cookiecutter
    cookiecutter https://gitlab.com/AdriaanRol/cookiecutter-pypackage-gitlab

The command line interface will ask you to provide the following information:

* :code:`full_name` : Your name
* :code:`email` : Your email
* :code:`gitlab_username` : Your GitLab username
* :code:`project_name` : The project name
* :code:`project_slug` : The `slug <https://docs.gitlab.com/ee/gitlab-basics/create-project.html#blank-projects>`_ of the python project. This is auto populated based on the project_name where spaces and dashes are replaced to get a valid name.
* :code:`project_url` : The url of the GitLab project. This is auto populated based on the project_slug and username but if you are adding the project to an organization or group this can be different.
* :code:`project_short_description` : A short description of the project.
* :code:`pypi_username` : Your PyPI username
* :code:`copyright_holder` : The name of the copyright holder. Defaults to "full_name", but can be different if setting up a commercial project.
* :code:`version` : The current version, defaults to 0.1.0.
* :code:`create_author_file` : If True, creates an "authors" file
* :code:`open_source_license` : Select a predefined license


You can test if the cookiecutter worked correctly by running the following commands


.. code-block:: sh

    pip install -e .                       # install the package locally
    pip install -r requirements_dev.txt    # install development requirements
    py.test                                # run a dummy test using the local environment
    tox                                    # run the tests using the tox environment. Takes some time as it also installs the requirements.
    sphinx-autobuild docs docs/_build/html # build the docs and host at localhost:8000

If it is working correctly you can now push to the repository you created using the following commands.

.. code-block:: sh

    cd project_slug
    git init
    git remote add origin git@gitlab.com:gitlab_username/project_slug.git
    git add .
    git commit -m "Initial commit"
    git push -u origin master


The presence of the :code:`.gitlab-ci.yml` file should ensure that a test build is triggered upon the first push.



Step 2. Set up documentation (manual)
-------------------------------------------------------------

Host documentation on readthedocs
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~

The repository comes with a :code:`.readthedocs.yml` file which ensures the project can be imported.
If you repository is public you can host it for free on `readthedocs.org <https://readthedocs.org/>`_.
Sign in using your GitLab account and import your package.

Show build status and coverage
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~

Badges need to be created manually on GitLab.


We will be adding two badges, the pipeline status and coverage badges.
You will need to manually add them in the GitLab interface in order for the project badges to work.
To do this go to your project, :code:`settings/general/badges`, select "Add Badge" and enter the following information:

* Name: Pipeline Status
* Link: :code:`https://gitlab.com/%{project_path}/-/pipelines/%{default_branch}`
* Badge Image URL: :code:`https://gitlab.com/%{project_path}/badges/%{default_branch}/pipeline.svg`


* Name: Coverage
* Link: :code:`https://gitlab.com/%{project_path}/-/pipelines/%{default_branch}`
* Badge Image URL: :code:`https://gitlab.com/%{project_path}/badges/%{default_branch}/coverage.svg`

The CI/CD is set up to automatically report the coverage of each pipeline.
The flake8 and coverage reports can be downloaded from the test pipelines as artifacts.

It is also possible to `integrate the coverage.xml reports with merge requests <https://docs.gitlab.com/ee/user/project/merge_requests/test_coverage_visualization.html>`_.
According to the GitLab documentation, the current version should already work.
However, this doesn't seem to be the case.
I'll update this post (and the template) when I resolve `this issue <https://gitlab.com/AdriaanRol/cookiecutter-pypackage-gitlab/-/issues/1>`_.


Upload to PyPI
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~

Once you have made some commits and are ready to release your package on PyPI upload the package to PyPI.
The setup.py should already be compatible with PyPI.
You can check out my `previous post <https://adriaanrol.com/posts/pypi/>`_ on uploading a package to PyPI.