+ An example configuration can be found here - https://github.com/CentOS/duffy/blob/dev/etc/duffy-example-config.yaml

+ 

+ # `app` section

+ 

+ This section has all the configuration settings for the `duffy` application.

+ 

+ ## Key/value pairs

+ 

+ 1. `loglevel` - Set the default loglevel for the `duffy` application.

+ 2. `host` - Set the default host address for the `duffy` application to listen to.

+ 3. `port` - Set the default port number for the `duffy` application to listen to.

+ 4. `logging` - Set the global logging configuration for the parts of the `duffy` application.

+ 

+ # `metaclient` section

+ 

+ This section has all the configuration settings for the `duffy` metaclient application.

+ 

+ ## Key/value pairs

+ 

+ 1. `loglevel` - Set the default loglevel for the `duffy` metaclient application.

+ 2. `host` - Set the default host address for the `duffy` metaclient application to listen to.

+ 3. `port` - Set the default port number for the `duffy` metaclient application to listen to.

+ 4. `dest` - Set the destination address and port number for the deployed `duffy` application.

+ 5. `usermap` - Set the mapping of UUIDs with legacy `duffy` usernames.

+ 6. `poolmap` - Set the mapping of poolname and hardware configuration (`version` and `arch`).

+ 

+ # `tasks` section

+ 

+ This section has all the configuration settings required for the `duffy` worker backend.

+ 

+ ## Key/value pairs

+ 1. `celery` - Configure the Celery backend.

+    1. `broker_url` - Set the address to a reachable Redis store.

+    2. `result_backend` - Set the address to a reachable Redis store.

+    3. `worker_redirect_stdouts_level` - Set the loglevel for worker redirect operations.

+ 2. `locking` - Configure the Celery backend.

+    1. `url` - Set the address to a reachable Redis store.

+ 3. `periodic` - Configure recurring operations.

+    1. `fill-pools` - Configure checking for new devices for the pools.

+       1. `interval` - Set the interval in seconds for the period.

+    1. `expire-sessions` - Configure checking for provisioned nodes reaching expiry.

+       1. `interval` - Set the interval in seconds for the period.

+ 

+ # `database` section

+ 

+ This section has all the configuration settings required for the `duffy` database.

+ 

+ ## Key/value pairs

+ 1. `sqlalchemy` - Configure SQLAlchemy settings.

+    1. `sync_url` - Set the reachable database location for synchronous database operations.

+    2. `async_url` - Set the reachable database location for asynchronous database operations.

+ 

+ # `misc` section

+ 

+ This section has all the configuration settings required for the session allowed time.

+ 

+ ## Key/value pairs

+ 1. `session-lifetime` - Set the allowed time for a normal session.

+ 2. `session-lifetime-max` - Set the allowed time for an extended session.

+ 

+ # `nodepools` section

+ 

+ This section has all the configuration settings required for describing node pools.

+ 

+ ## Key/value pairs

+ 1. `abstract` - Set the Ansible paths and settings for physical and virtual nodes.

+ 2. `concrete` - Set the settings for physical and virtual nodes by extending configs. 

+ To start with, thank you for taking the time to contribute! ❤️

+ 

+ ## How to contribute to the project?

+ There are a lot of ways with which one can contribute to this project.

+ 1. As a user of the project, if one comes across a certain discrepancy or if one wants a certain feature to be implemented, they can open up an issue ticket at https://github.com/CentOS/duffy/issues.

+ 2. As a developer of the project, one can help with providing the code that helps fix a certain issue or helps attain a certain new feature by opening up a pull request against the primary branch.

+ 3. As a user or developer of the project, one can help with documenting the ins and outs of the project with respect to its usage, development, contribution, maintenance, deployment etc.

+ 

+ ## How to set up the development environment?

+ 1. Fork the repository to your own GitHub namespace.

+ 2. Clone the forked repository and navigate into the project directory.

+    ```

+    git clone https://github.com/<namespace>/duffy.git

+    cd duffy

+    ```

+ 3. Set up and activate a virtual environment.

+    * Using native virtual environment

+      ```

+      python3 -m venv duffyenv

+      source duffyenv/bin/activate

+      ```

+    Or

+    * Using virtualenv wrapper

+      ```

+      virtualenv duffyenv

+      source duffyenv/bin/activate

+      ```

+    Or

+    * Using Poetry virtual environment shell

+      ```

+      poetry shell

+      ```

+ 4. Install using Poetry

+    ```

+    poetry install

+    ```

+ 5. Checkout to a new branch with a clear descriptive name.

+    ```

+    git checkout -b <some-branch-name>

+    ```

+ 6. Open up the project in an IDE or a code-editor to start adding your contributions.

+ 7. Test your changes by running the server using 

+    ```

+    duffy -p 8000 -4 -l trace

+    ```

+    1. Please choose a port number that you have the permissions for and is not already in use.

+    2. Please make use of the variety of log levels in order to better facilitate the debugging process

+       1. `critical` - Exhibit extremely severe error events, which may result in the application's termination

+       2. `error` - Exhibit significant error events that will halt normal programme execution but may still allow the application to execute

+       3. `warning` - Exhibit potentially dangerous circumstances that may be of interest to end users or system administrators and identify potential issues

+       4. `info` - Exhibit informational messages that may be useful to end users and system administrators, as well as the application's progress

+       5. `debug` - Exhibit application developers' usage of relatively detailed debugging

+       6. `trace` - Exhibits all related messages.

+ 8. Once done making the changes, be sure to add tests for the code and see if your code changes comply with them by running

+    ```

+    pytest

+    ```

+ 10. Please commit with a precise commit message and signature.

+     ```

+     git commit -sm "<some-commit-message-which-actually-makes-sense>"

+     ```

+ 11. Push your local commits to the remote branch of your fork.

+     ```

+     git push origin <some-branch-name>

+     ```

+ 

+ ## How do I contribute in the right way?

+ 1. Please follow the following standard for your commit messages,

+    1. Limit the subject line of a commit to 50 characters and the body of a commit to 72 characters.

+    2. Use the imperative sense of a verb in the subject line (eg. Use `Update ...` and not `Updated ...`).

+    3. Capitalize the subject line and do not use periods at the end of the sentence.

+    4. Use the body to justify and describe the changes and start it after leaving a blank line under the subject.

+    5. Be sure to sign your commits before pushing them to the remote branch of your fork.

+ 2. For every code addition made to the project, 

+    1. Add inline comments to the parts of the code which require additional context and add to the documentation as well.

+    2. Ensure that the code is semantic and the names provided to variables, functions and classes describe their purpose.

+    3. Tests must be added in the same pull request to ensure that a good coverage and great overall code quality.

+    4. If there are parts in the code that do not require testing or cannot be tested, be sure to exclude them in the config.

+    5. Use `black .` to format the code and `isort .` to automatically sort the imports before pushing the changes.

+ 

+ ## Where do I reach out if I wish to discuss the project?

+ The current maintainers of the project are available at the #centos-ci IRC channel of libera.chat. Please feel free to reach out to them. As the team members hail from a various countries across the world, patience in waiting for a reply back is greatly appreciated.  

+ # Deployment

+ 

+ ## Environment requirement

+ * Python 3.8

+ * Pip3

+ 

+ ## Installation

+ Use `<Ansible playbook link>` to pull and setup duffy. This doesn't configure

+ duffy but pulls all needed packages, sets environment and ensures things are at

+ the right place.

+ 

+ ## Configuration

+ Once installed, make a config file that you will use to guide Duffy(see [example config file](https://github.com/CentOS/duffy/blob/dev/etc/duffy-example-config.yaml)). [Configurations Page](https://github.com/CentOS/duffy/wiki/Configuration) covers different details about the options that you can learn about.

+ 

+ ## Usage

+ Check [usage page](https://github.com/CentOS/duffy/wiki/Usage) to learn how to run/serve the app

+ To install/develop Duffy:

+ 

+ 1. Clone the repository and navigate into the project directory.

+    ```

+    git clone https://github.com/CentOS/duffy.git

+    cd duffy

+    ```

+ 

+ 2. Set up and activate a virtual environment.

+    * Using native virtual environment

+      ```

+      python3 -m venv duffyenv

+      source duffyenv/bin/activate

+      ```

+    Or

+    * Using virtualenv wrapper

+      ```

+      virtualenv duffyenv

+      source duffyenv/bin/activate

+      ```

+    Or

+    * Using Poetry virtual environment shell

+      ```

+      poetry shell

+      ```

+ 

+ 3. Install using Poetry

+    The following command would install all the dependencies for the project including the ones required for the development and testing of the project.

+    ```

+    poetry install

+    ```

+    Execute the following command if you wish to installing only the dependencies which are required to run the project but not those required for its development and testing.

+    ```

+    poetry install --no-dev

+    ```

+ 

+ 4. Add changes and then test those by running Tox

+    ```

+    tox

+    ```

+ 

+ 5. Please be descriptive about the changes introduced in your pull request while making one. 

+ The project can be either installed via the included Poetry project configuration or via the PyPI package which can be found [here](https://pypi.org/project/duffy/). 

+ 

+ The package can be easily installed by executing the following command in an activated virtual environment.

+ 

+ ```

+ pip install duffy

+ ```

+ 

+ Be sure to append the command with a version number if you are looking for setting up the project at its state on a specific tag or release. For example, the following command would help specifying the version `3.0.0` for installation.

+ 

+ ```

+ pip install duffy==3.0.0

+ ```

+ 

+ Do note that the pre-release versions of the project are usually not installed implicitly and in order to install them, the version number need to explicitly specified in the manner stated above. This is not true if the pre-release version is the only version available.

+ 

+ ```

+ pip install duffy=3.0.0a0

+ ```

+ 

+ Installing the PyPI package is greatly recommended as it allows you to conveniently upgrade to the newer version of the project without having to keep up with the updates made to the repository. 

+ 

+ As of now, the PyPI package has the following four maintainers.

+ 

+ - [Akashdeep Dhar](https://pypi.org/user/t0xic0der/)

+ - [Fabian Arrotin](https://pypi.org/user/arrfab/)

+ - [Nils Philippsen](https://pypi.org/user/nilsph/)

+ - [Vipul Siddharth](https://pypi.org/user/siddharthvipul/)

+ 

+ The aforesaid maintainers can publish new versions of the project by first, creating a tag of a distinct version number following the semantic versioning scheme and then, using the following command to build and publish packages in succession. Please make sure that you are at the root directory of the project where the `pyproject.toml` and that the project configuration is valid.

+ 

+ ```

+ poetry publish --build

+ ```

+ 

+ Please do note that the mentioned command requires authentication with the use of username and password so as to ensure that only the authorized maintainers can push new releases. It is also greatly recommended to first push a finished release with an alpha tag (say, a finished release `3.0.1` can be pushed to PyPI as a pre-release package `3.0.1a0`) to allow for testing before pushing the release again with the actual version number to not use up the version number if the release requires revisions. 

+ ## CLI usage overview

+ 

+ ```

+ duffy --help

+ ```

+ 

+ The rewritten version of Duffy includes a lot of functionality under one umbrella, making it convenient for the maintainers to operate upon the following things from a single command.

+ 

+ 1. [Configuring the variables](https://github.com/CentOS/duffy/wiki/Usage#the-config-command)

+ 2. [Migrating the database](https://github.com/CentOS/duffy/wiki/Usage#the-migration-command)

+ 3. [Serving the primary web application](https://github.com/CentOS/duffy/wiki/Usage#the-serve-command)

+ 4. [Serving the legacy web application](https://github.com/CentOS/duffy/wiki/Usage#the-serve-legacy-command)

+ 5. [Setting up the database](https://github.com/CentOS/duffy/wiki/Usage#the-setup-db-command)

+ 6. [Interact with a running instance](https://github.com/CentOS/duffy/wiki/Usage#the-shell-command)

+ 7. Starting a backend processor 

+ 

+ ```

+ Usage: duffy [OPTIONS] COMMAND [ARGS]...

+ 

+ Options:

+   -c, --config FILE  Read option defaults from the specified YAML file.

+                      [default: /etc/duffy.yaml]

+   --version          Show the version and exit.

+   --help             Show this message and exit.

+ 

+ Commands:

+   config        Check and dump configuration.

+   migration     Handle database migrations.

+   serve         Run the Duffy web application server.

+   serve-legacy  Serve the Duffy Metaclient for Legacy Support app.

+   setup-db      Create tables from the database model.

+   shell         Run an interactive shell.

+   worker        Start a Celery worker to process backend tasks.

+ ```

+ 

+ We will go in detail about each of the aforementioned functionalities in the sections below.

+ 

+ ### The `config` command

+ 

+ To understand how the config command makes use of a YAML-based configuration, please refer to [the configuration documentation](https://github.com/CentOS/duffy/wiki/Configuration).

+ 

+ ```

+ duffy config --help

+ ```

+ 

+ Duffy is written to be flexible enough to allow for smaller configurations (like setting log levels for an operation, selecting port number for publishing etc.) to be quickly appended to the command as options as well as for bigger configurations (like setting up the worker environment, pointing to Ansible Playbook locations etc.) to be descriptively specified in a YAML-based configuration file. The subcommands included in the CLI allows for verifying the correctness of and, dumping merged configurations of the latter kind.

+ 

+ ```

+ Usage: duffy config [OPTIONS] COMMAND [ARGS]...

+ 

+   Check and dump configuration.

+ 

+ Options:

+   --help  Show this message and exit.

+ 

+ Commands:

+   check  Validate configuration structure.

+   dump   Dump merged configuration.

+ ```

+ 

+ ### The `migration` command

+ 

+ ```

+ duffy migration --help

+ ```

+ 

+ With the use of SQLAlchemy, the interactions with the database are abstracted enough such that, Duffy becomes agnostic of the kind of database that it makes use of. It does allow for migrating schemas and managing changes with the use of the said command. The command requires URL of the database for both synchronous and asynchronous operations with it. As Duffy is agnostic of what database it makes use of, any kind of relational database can be used but we strongly recommend the use of Postgres.

+ 

+ ```

+ Usage: duffy migration [OPTIONS] COMMAND [ARGS]...

+ 

+   Handle database migrations.

+ 

+ Options:

+   --help  Show this message and exit.

+ 

+ Commands:

+   create      Create a new migration.

+   db-version

+   downgrade

+   upgrade

+ ```

+ 

+ ### The `serve` command

+ 

+ ```

+ duffy serve --help

+ ```

+ 

+ The primary web server is the main interface with which clients can connect to Duffy and perform tasks like create a session, request baremetal or virtual nodes etc. The server can be started up using the aforementioned command by providing parameters like host address, port number, log level etc. These variables can either be passed as options appended to a command or be listed under the `app` section of the YAML-based configuration file. 

+ 

+ ```

+ Usage: duffy serve [OPTIONS]

+ 

+   Run the Duffy web application server.

+ 

+   Duffy is the middle layer running ci.centos.org that manages the

+   provisioning, maintenance, teardown and rebuild of the Nodes (physical

+   hardware and virtual machines) that are used to run the tests in the CI

+   Cluster.

+ 

+ Options:

+   --reload / --no-reload          Automatically reload if the code is changed.

+   -H, --host TEXT                 Set the host address to listen on.

+   -p, --port INTEGER RANGE        Set the port value.  [1<=x<=65535]

+   -l, --loglevel [critical|error|warning|info|debug|trace]

+                                   Set the log level.

+   --help                          Show this message and exit.

+ ```

+ 

+ ### The `serve-legacy` command

+ 

+ ```

+ duffy serve-legacy --help

+ ```

+ 

+ The legacy web server is the secondary interface for existing clients of legacy Duffy that provides for a 1:1 compatibility with the older endpoints. Clients can connect here to do the same tasks that they could in the primary interface with minimal changes to the workflow. As like the primary interface, this command can be customized with the use of appended options or as variables under the `metaclient` section of the YAML-based configuration file.

+ 

+ ```

+ Usage: duffy serve-legacy [OPTIONS]

+ 

+   Serve the Duffy Metaclient for Legacy Support app.

+ 

+   Duffy is the middle layer running ci.centos.org that manages the

+   provisioning, maintenance and teardown / rebuild of the Nodes (physical

+   hardware for now, VMs coming soon) that are used to run the tests in the CI

+   Cluster.

+ 

+   This metaclient exposes older endpoints for legacy support and connects them

+   to the path operations introduced by the newer version of the Duffy

+   endpoint, until the support for the older endpoints is deprecated.

+ 

+ Options:

+   --reload / --no-reload          Automatically reload if the code is changed.

+   -H, --host TEXT                 Set the host address to listen on.

+   -p, --port INTEGER RANGE        Set the port value.  [1<=x<=65535]

+   -D, --dest TEXT                 Set the destination address of Duffy

+                                   deployment.

+   -l, --loglevel [critical|error|warning|info|debug|trace]

+                                   Set the log level.

+   --help                          Show this message and exit.

+ ```

+ 

+ ### The `setup-db` command

+ 

+ ```

+ duffy setup-db --help

+ ```

+ 

+ According to the database of the maintainer's choice, Duffy can set up tables from the included database model. To facilitate for a development and testing environment, contributors can also populate the created database with test data - although the same database must not be used in a production environment. The command requires URL of the database for both synchronous and asynchronous operations with it. As Duffy is agnostic of what database it makes use of, any kind of relational database can be used but we strongly recommend the use of Postgres.

+ 

+ ```

+ Usage: duffy setup-db [OPTIONS]

+ 

+   Create tables from the database model.

+ 

+ Options:

+   --test-data / --no-test-data  Initialized database with test data.

+   --help                        Show this message and exit.

+ ```

+ 

+ ### The `shell` command

+ 

+ ```

+ duffy shell --help

+ ```

+ 

+ Performing management tasks on a currently running instance of Duffy is made possible with the inclusion of an interactive shell - that can be run on the same environment as the one where the application is deployed. The shell can be managed either by a Python interpreter or an IPython interpreter - with IPython providing additional enrichment to a normal REPL-based interface. Maintainers can use the shell to perform CRUD operations on the attached database with respect to sessions, tenants, nodes, projects etc.

+ 

+ ```

+ Usage: duffy shell [OPTIONS]

+ 

+   Run an interactive shell.

+ 

+ Options:

+   -t, --shell-type [python|ipython]

+                                   Type of interactive shell to use.

+   --help                          Show this message and exit.

+ ``` 

+ 1. The entire codebase of the project has been re-implemented in Python 3 using a curated set of libraries and dependencies which are actively developed and maintained.

+ 2. The project has been structured to facilitate simpler installation for both consumers and maintainers, with possible easy distribution as a PyPI package and an RPM package.

+ 3. Our test suite ensures that the codebase conforms to the relevant code quality standards and style guidelines, and all of the code is covered in tests.

+ 4. The entire codebase of the project complies with the relevant code quality standards and the included integration and unit tests help attain a 100% code coverage.

+ 5. This version of Duffy introduces CLI tools to set up the database structure as well as create and perform database migrations (for potential future upgrades).

+ 6. The project is now capable of provisioning both baremetal nodes as well as virtual machine nodes using a pool-based allocation, agnostic of the specifications of the nodes.

+ 7. The rewritten API path operations are now more semantic in CRUD operations, more verbose with varied status responses and are documented according to the OpenAPI standards.

+ 8. The included metaclient bridges the compatibility for existing users of Duffy to migrate to a limited release of the newer version without much changes to the automation scripts.

+ 9. The continuous integration pipelines allow for automatic updates made to the dependency lockfiles to enrich contributor convenience by ensuring a common set of dependencies.

+ 10. The backend-agnostic approach of the API makes the project pluggable and paves way for the usage of EC2 resources with a possible future integration with the AWS API.

+ 11. The newly introduced task worker unifies the operations for provisioning, fetching and returning nodes in a single unit, backed by a Redis queue mediated by Celery.

+ 12. The project can now be easily managed as a service with the use of the included Systemd scripts and the CLI can interact with an existing running service easily.

+ 13. The project makes use of a convenient yet powerful CLI flag based and YAML based configuration that admins can derive theirs from by using the included example.

+ 14. The logging system added in the project is appropriately verbose and specifically descriptive to mark important events and can be configured to exhibit different levels.

+ 15. The security scheme marks a departure from a simple API key based authentication to a more secure and standardized username/password-based simple HTTP authentication. 

+ # Duffy

+ 

+ 

+ ## Information

+ 

+ Community Platform Engineering team (of Red Hat) is working on revamping this project and thus, have cleaned this repository by marking other branches stale and creating new branches for development. In order to see the current deployed version of Duffy in [CentOS CI](https://github.com/centosci## ) Infra, please check [`stale/master`](https://github.com/CentOS/duffy/tree/stale/master) branch.

+ 

+         - Deployment: apps/duffy/Deployment.md

+         - Configurations: apps/duffy/Configuration.md

+         - Usage: apps/duffy/Usage.md

+         - Contribution: apps/duffy/Contributing.md

+         - Development: apps/duffy/Development.md

+         - PyPI: apps/duffy/PyPI.md