Contibute to Funkwhale development

First of all, thank you for your interest in the project! We really appreciate the fact that you’re about to take some time to read this and hack on the project.

This document will guide you through common operations such as:

  • Setup your development environment
  • Working on your first issue
  • Writing unit tests to validate your work
  • Submit your work

Setup your development environment

If you want to fix a bug or implement a feature, you’ll need to run a local, development copy of funkwhale.

We provide a docker based development environment, which should be both easy to setup and work similarly regardless of your development machine setup.

Instructions for bare-metal setup will come in the future (Merge requests are welcome).

Installing docker and docker-compose

This is already cover in the relevant documentations:

Cloning the project

Visit https://code.eliotberriot.com/funkwhale/funkwhale and clone the repository using SSH or HTTPS. Exemple using SSH:

git clone ssh://git@code.eliotberriot.com:2222/funkwhale/funkwhale.git
cd funkwhale

A note about branches

Next release development occurs on the “develop” branch, and releases are made on the “master” branch. Therefor, when submitting Merge Requests, ensure you are merging on the develop branch.

Working with docker

In developpement, we use the docker-compose file named dev.yml, and this is why all our docker-compose commands will look like this:

docker-compose -f dev.yml logs

If you do not want to add the -f dev.yml snippet everytime, you can run this command before starting your work:

export COMPOSE_FILE=dev.yml

Building the containers

On your initial clone, or if there have been some changes in the app dependencies, you will have to rebuild your containers. This is done via the following command:

docker-compose -f dev.yml build

Creating your env file

We provide a working .env.dev configuration file that is suitable for development. However, to enable customization on your machine, you should also create a .env file that will hold your personal environment variables (those will not be commited to the project).

Create it like this:

touch .env

Database management

To setup funkwhale’s database schema, run this:

docker-compose -f dev.yml run --rm api python manage.py migrate

This will create all the tables needed for the API to run proprely. You will also need to run this whenever changes are made on the database schema.

It is safe to run this command multiple times, so you can run it whenever you fetch develop.

Development data

You’ll need at least an admin user and some artists/tracks/albums to work locally.

Create an admin user with the following command:

docker-compose -f dev.yml run --rm api python manage.py createsuperuser

Injecting fake data is done by running the fllowing script:

artists=25
command="from funkwhale_api.music import fake_data; fake_data.create_data($artists)"
echo $command | docker-compose -f dev.yml run --rm api python manage.py shell -i python

The previous command will create 25 artists with random albums, tracks and metadata.

Launch all services

Then you can run everything with:

docker-compose -f dev.yml up

This will launch all services, and output the logs in your current terminal window. If you prefer to launch them in the background instead, use the -d flag, and access the logs when you need it via docker-compose -f dev.yml logs --tail=50 --follow.

Once everything is up, you can access the various funkwhale’s components:

Stopping everything

Once you’re down with your work, you can stop running containers, if any, with:

docker-compose -f dev.yml stop

Removing everything

If you want to wipe your development environment completely (e.g. if you want to start over from scratch), just run:

docker-compose -f dev.yml down -v

This will wipe your containers and data, so please be careful before running it.

You can keep your data by removing the -v flag.

Working with federation locally

This is not needed unless you need to work on federation-related features.

To achieve that, you’ll need:

  1. to update your dns resolver to resolve all your .dev hostnames locally
  2. a reverse proxy (such as traefik) to catch those .dev requests and and with https certificate
  3. two instances (or more) running locally, following the regular dev setup

Resolve .dev names locally

If you use dnsmasq, this is as simple as doing:

echo "address=/test/172.17.0.1" | sudo tee /etc/dnsmasq.d/test.conf
sudo systemctl restart dnsmasq

If you use NetworkManager with dnsmasq integration, use this instead:

echo "address=/test/172.17.0.1" | sudo tee /etc/NetworkManager/dnsmasq.d/test.conf
sudo systemctl restart NetworkManager

Add wildcard certificate to the trusted certificates

Simply copy bundled certificates:

sudo cp docker/ssl/test.crt /usr/local/share/ca-certificates/
sudo update-ca-certificates

This certificate is a wildcard for *.funkwhale.test

Run a reverse proxy for your instances

Create docker network

Create the federation network:

docker network create federation

Launch everything

Launch the traefik proxy:

docker-compose -f docker/traefik.yml up -d

Then, in separate terminals, you can setup as many different instances as you need:

export COMPOSE_PROJECT_NAME=node2
docker-compose -f dev.yml run --rm api python manage.py migrate
docker-compose -f dev.yml run --rm api python manage.py createsuperuser
docker-compose -f dev.yml up nginx api front nginx api celeryworker

Note that by default, if you don’t export the COMPOSE_PROJECT_NAME, we will default to node1 as the name of your instance.

Assuming your project name is node1, your server will be reachable at https://node1.funkwhale.test/. Not that you’ll have to trust the SSL Certificate as it’s self signed.

When working on federation with traefik, ensure you have this in your env:

# This will ensure we don't bind any port on the host, and thus enable
# multiple instances of funkwhale to be spawned concurrently.
WEBPACK_DEVSERVER_PORT_BINDING=
# This disable certificate verification
EXTERNAL_REQUESTS_VERIFY_SSL=false
# this ensure you don't have incorrect urls pointing to http resources
FUNKWHALE_PROTOCOL=https

Typical workflow for a contribution

  1. Fork the project if you did not already or if you do not have access to the main repository
  2. Checkout the development branch and pull most recent changes: git checkout develop && git pull
  3. If working on an issue, assign yourself to the issue. Otherwise, consider open an issue before starting to work on something, especially for new features.
  4. Create a dedicated branch for your work 42-awesome-fix. It is good practice to prefix your branch name with the ID of the issue you are solving.
  5. Work on your stuff
  6. Commit small, atomic changes to make it easier to review your contribution
  7. Add a changelog fragment to summarize your changes: echo "Implemented awesome stuff (#42)" > changes/changelog.d/42.feature"
  8. Push your branch
  9. Create your merge request
  10. Take a step back and enjoy, we’re really grateful you did all of this and took the time to contribute!

Internationalization

When working on the front-end, any end-user string should be translated using either <i18next path="yourstring"> or the $t('yourstring') function.

Extraction is done by calling yarn run i18n-extract, which will pull all the strings from source files and put them in a PO file.

Contributing to the API

Project structure

tree api -L 2 -d
api
├── config          # configuration directory (settings, urls, wsgi server)
│   └── settings    # Django settings files
├── funkwhale_api   # project directory, all funkwhale logic is here
├── requirements    # python requirements files
└── tests           # test files, matches the structure of the funkwhale_api directory

Note

Unless trivial, API contributions must include unittests to ensure your fix or feature is working as expected and won’t break in the future

Running tests

To run the pytest test suite, use the following command:

docker-compose -f dev.yml run --rm api pytest

This is regular pytest, so you can use any arguments/options that pytest usually accept:

# get some help
docker-compose -f dev.yml run --rm api pytest -h
# Stop on first failure
docker-compose -f dev.yml run --rm api pytest -x
# Run a specific test file
docker-compose -f dev.yml run --rm api pytest tests/test_acoustid.py

Writing tests

Although teaching you how to write unit tests is outside of the scope of this document, you’ll find below a collection of tips, snippets and resources you can use if you want to learn on that subject.

Useful links:

Recommendations:

  • Test files must target a module and mimic funkwhale_api directory structure: if you’re writing tests for funkwhale_api/myapp/views.py, you should put thoses tests in tests/myapp/test_views.py
  • Tests should be small and test one thing. If you need to test multiple things, write multiple tests.

We provide a lot of utils and fixtures to make the process of writing tests as painless as possible. You’ll find some usage examples below.

Use factories to create arbitrary objects:

# funkwhale_api/myapp/users.py

def downgrade_user(user):
    """
    A simple function that remove superuser status from users
    and return True if user was actually downgraded
    """
    downgraded = user.is_superuser
    user.is_superuser = False
    user.save()
    return downgraded

# tests/myapp/test_users.py
from funkwhale_api.myapp import users

def test_downgrade_superuser(factories):
    user = factories['users.User'](is_superuser=True)
    downgraded = users.downgrade_user(user)

    assert downgraded is True
    assert user.is_superuser is False

def test_downgrade_normal_user_does_nothing(factories):
    user = factories['users.User'](is_superuser=False)
    downgraded = something.downgrade_user(user)

    assert downgraded is False
    assert user.is_superuser is False

Note

We offer factories for almost if not all models. Factories are located in a factories.py file inside each app.

Mocking: mocking is the process of faking some logic in our code. This is useful when testing components that depend on each other:

# funkwhale_api/myapp/notifications.py

def notify(email, message):
    """
    A function that sends an email to the given recipient
    with the given message
    """

    # our email sending logic here
    # ...

# funkwhale_api/myapp/users.py
from . import notifications

def downgrade_user(user):
    """
    A simple function that remove superuser status from users
    and return True if user was actually downgraded
    """
    downgraded = user.is_superuser
    user.is_superuser = False
    user.save()
    if downgraded:
        notifications.notify(user.email, 'You have been downgraded!')
    return downgraded

# tests/myapp/test_users.py
def test_downgrade_superuser_sends_email(factories, mocker):
    """
    Your downgrade logic is already tested, however, we want to ensure
    an email is sent when user is downgraded, but we don't have any email
    server available in our testing environment. Thus, we need to mock
    the email sending process.
    """
    mocked_notify = mocker.patch('funkwhale_api.myapp.notifications.notify')
    user = factories['users.User'](is_superuser=True)
    users.downgrade_user(user)

    # here, we ensure our notify function was called with proper arguments
    mocked_notify.assert_called_once_with(user.email, 'You have been downgraded')


def test_downgrade_not_superuser_skips_email(factories, mocker):
    mocked_notify = mocker.patch('funkwhale_api.myapp.notifications.notify')
    user = factories['users.User'](is_superuser=True)
    users.downgrade_user(user)

    # here, we ensure no email was sent
    mocked_notify.assert_not_called()

Views: you can find some readable views tests in tests/users/test_views.py

Note

A complete list of available-fixtures is available by running docker-compose -f dev.yml run --rm api pytest --fixtures

Contributing to the front-end

Running tests

To run the front-end test suite, use the following command:

docker-compose -f dev.yml run --rm front yarn run unit

We also support a “watch and test” mode were we continually relaunch tests when changes are recorded on the file system:

docker-compose -f dev.yml run --rm front yarn run unit-watch

The latter is especially useful when you are debugging failing tests.

Note

The front-end test suite coverage is still pretty low