Deploying a git server using ansible with notifications via Bugout.dev

Hello, today we will set up our private git server and organize status notification delivery using Bugout.dev. Here is the link to the repository with ansible playbooks, services and scripts. If you want to understand the details, please check out the instruction below.

The distinctive feature of my proposed approach is trying to avoid writing our own heavyweight solutions. The majority of such assumes an open port with an admin panel to the outer world, or a docker-compose with elasticsearch maintaining our ten small-scale repositories on the git server. In turn, Bugout.dev provides us the required minimum for displaying and searching in a readable manner, and endless ways to manipulate and process data, which we will want to redirect to our journal.

We will need three tools:

  • git
  • bugout
  • ansible

Git will be installed according to the official documentation. If you want to do so manually, here is the link: https://git-scm.com/book/en/v2/Git-on-the-Server-Setting-Up-the-Server

Bugout.dev (https://bugout.dev/) is a service for collecting and processing software usage statistics and error reports in libraries supported by the developers. We’ll be using it for transmitting and displaying data about the number of the repositories on our server and the number of branches in these repositories.

We will use Ansible playbook for quickly deploying the server on the new instance, which will let us happily forget everything we coded yesterday.

Preparing the server

Our server configuration will assume a newly created AWS EC2 instance, and this entails the following key points:

  • Firstly, allow the “ubuntu” user to run sudo commands without a password
  • Secondly, make sure that you have access to the server via a private ssh key (I will be using a key called id_rsa_dev)

In order to work with git, generate a new pair of a private and a public key that will be used for accessing our server via the git commands (mine are called id_rsa_git and id_rsa_git.pub respectively). If you want to facilitate multiple users working on one git server, you will have to change the ansible script to allow for adding multiple keys. Then, you will have to modify the following code lines in gitmonitor.yml:


...
vars:
  git_ssh_public_key: ""
...
  - name: Add git SSH authorized public key
    become: yes
    become_user: git
    authorized_key:
      user: git
      key: "{{ lookup('file', git_ssh_public_key) }}"
      state: present
...

 

Bugout.dev registration

Accordingly, we will need a Bugout.dev account to store and receive our data from the git server. We will create a journal and a token at https://bugout.dev. To do so visit https://bugout.dev/account/tokens and generate a new token (do not publish it or show it to anyone, as it provides full access to the account).

Create a new journal for our git server at
https://bugout.dev/app/personal

After creating the journal, we will need its id, which we can extract from the url:
https://bugout.dev/app/personal/361c297b-b61d-4d7c-877f-2ca2e7fc138d/entrieswhere 361c297b-b61d-4d7c-877f-2ca2e7fc138dis our journal_id.

Building with ansible

Create a python environment and install ansible.


python3 -m venv .venv
source .venv/bin/activate
pip install -r requirements.txt

Navigate to the directory playbooks/gitmonitor, create a file called machines.ini, and add our future git server’s IP into it.


[machines]
127.0.0.1

Our ansible playbook consists of two sections:

  • Git configuration
  • Bugout, script and service installation

The full version of gitmonitor.yml is available at github. When you launch playbook you’ll be asked to provide:

  • The git user password for our server
  • Bugout token (from the previous step, where we created a Bugout.dev account)
  • Journal Id (from the previous step)

Once you have everything ready run the command:


ansible-playbook \
  --ssh-common-args "-o UserKnownHostsFile=/dev/null 
    -o StrictHostKeyChecking=no 
    -o IdentityFile=~/.ssh/id_rsa_dev" \
  -e "git_ssh_public_key=~/.ssh/id_rsa_git.pub" \
  -i machines.ini gitmonitor.yml

Where IdentityFile is our server access key, git_ssh_public_key is the public key to access the git repository. Only the developer should have the private key.

Once we correctly run ansible playbook, we should verify that there is a file with environment variables and a bash script on our git server.


root@dev:~# ls -la /etc/gitmonitor/
total 16
drwxr-xr-x 2 root root 4096 Apr 28 18:51 .
drwxr-xr-x 94 root root 4096 Apr 28 16:55 ..
-rw-r----- 1 root root 156 Apr 28 18:51 .env
-rwxr----- 1 root root 2072 Apr 28 16:58 gitmonitor.bash

Create test repositories

To test that our config is working we’ll create a couple test repositories on the server. To do so use the script playbooks/gitmonitor/files/createrepo.bash from our local terminal:


ubuntu@home:~# ./createrepo.bash 127.0.0.1 ~/.ssh/id_rsa_dev
How to call new repository: scraper
Initialized empty Git repository in /srv/git/scraper.git/

The first argument in the script is our git server’s IPaddress, the second is our “ubuntu” user’s server access key. Let’s repeat this a couple more times for the other repositories backend, logger.

service, timer, path and bash scripts 

Great, now if we open the journal at Bugout.dev we’ll see that we have an entry called Git Server Dashboard, and also (possibly within a minute) three additional entries called:

  • Repository – backend
  • Repository – logger
  • Repository – scraper

The entries have been generated for every repository we created, these entries will be tracked and updated depending on new branches in the repositories. The three systemd service files responsible for this are: gitmonitor.service, gitmonitor.path, gitmonitor.timer

gitmonitor.service will start the /etc/gitmonitor/gitmonitor.bash script and in turn will be carried out by the path and timer services. In its config we’ll specify WorkingDirectory=/etc/gitmonitor – this is our working directory that our gitmonitor.bash script file has been copied to, and the .env file with our environment variables respectively specified as EnvironmentFile=/etc/gitmonitor/.env

gitmonitor.path tracks changes on the path PathModified=/srv/git, and triggers our gitmonitor.service every time there is a change.

gitmonitor.timer will launch gitmonitor.service at an interval of OnUnitActiveSec=1m

That leaves us with the following: a service publishing the changes in our repository storage with an additional trigger every minute, since path does not track changes in the inner directories. The gitmonitor.bash script works with Bugout.dev, i.e., executes the three commands:

  • bugout entries search– searches for an existing entry in our journal
  • bugout entries create – creates a new entry if it has not been created before and it is a new repository on our server
  • bugout entries update – updates an existing repository, if we have already published it to our journal

Result

By slightly changing the bash script we will be able to track and publish the latest commits in our repository, the accounts the changes were published from, et cetera. You can also describe all the logic in python, since Bugout.dev has an API published at https://github.com/bugout-dev/bugout-python. Or you can apply an analogous approach for managing any of your services on the server!

If you have any questions about the article, please leave a comment. Also Bugout.dev has a friendly developer community at Slack. You can join it by following this link.

Tags:
Tags :