Spring Boot MySQL Nginx Gitlab-CI Docker on VPS

In this Spring Boot and MySQL guide, we will implement a part of DevOps in the Spring Boot application to continuous integration using Gitlab-CI and Docker, and deploy to VPS such as Digital Ocean Droplet, AWS EC2, Microsoft Azure, Vultr, etc. 

The goal of this continuous integration is very simple. It just tests, builds, and deploys the Spring Boot and MySQL app automatically to the VPS (Digital Ocean Droplet, AWS EC2, Microsoft Azure, Vultr, etc) after being pushed to the Gitlab Repository. So, the user can use this application via a browser that is served by the Nginx HTTP server while the development is in progress.

We assume you already have Docker HUB and Gitlab accounts. So, we are not covering how to register or sign up for Docker HUB and Gitlab.

You can watch the video tutorial on our YouTube channel here.

Let's get started with the main steps!

Step #1: Setup Docker HUB Repository

To set up the Docker HUB repository, go to hub.docker.com then log in using your account. 

Click a Create Repository button on the Docker Dashboard page. Fill the repository name and make it private then click a Create Button.

Now, the Docker HUB repository is ready to use. We will use the docker command later in the Gitlab and VPS.

Step #2: Prepare a VPS with Docker, Docker Compose, and Gitlab Runner

We assume that you have a fresh VPS from your provider (Digital Ocean Droplet, AWS EC2, Microsoft Azure, Vultr, etc) and are able to connect to it via SSH. Ours is using Digital Ocean Droplet, so the just need to add my SSH keys to the VPS. First, create a new SSH key using this command.

cd ~/.ssh/
ssh-keygen -f myvps-id -t rsa

Leave the passphrase blank by clicking on enter a few times and you will see this console for successful SSH key generation.

Your identification has been saved in myvps-id.
Your public key has been saved in myvps-id.pub.
The key fingerprint is:
SHA256:GVQ2ag/pUcqwCqeaXhEwJQ4aAnS/pKvmCOvFPel0pYA [email protected]
The key's randomart image is:
+---[RSA 3072]----+
|B=.o  . ..=      |
|=.= .  = * .     |
|.....o. X        |
|   +=..o *       |
|  .E.o  S..      |
| o. + o o        |
|+  = = o         |
|+o+ o o          |
|*=   .           |
+----[SHA256]-----+

In Digital Ocean, we can access the VPS through the console. Just start the Droplets console in the Digital Ocean dashboard then log in using the provided root username and password.

Next, edit the SSH authorized_keys file.

nano ~/.ssh/authorized_keys

Back to your terminal then show the previously created SSH public key.

cat ~/.ssh/myvps-id.pub

Copy the content of ~/.ssh/myvps-id.pub then pastes it to the authorized_keys in the Digital Ocean Droplets console. Save the authorized_keys file then close the console. Next, try to connect to the VPS using SSH from your terminal.

ssh [email protected]

Now, the VPS is ready to use for Gitlab CI/CD deployment.

Next, we need to install Docker, Docker Compose, and Gitlab Runner. For that, type this command to update the existing default list of Ubuntu packages

apt update

Install package to let APT transport over HTTPS.

apt install apt-transport-https ca-certificates curl software-properties-common

Add GPG Key.

curl -fsSL https://download.docker.com/linux/ubuntu/gpg | sudo apt-key add -

Add Docker repository to the APT source.

add-apt-repository "deb [arch=amd64] https://download.docker.com/linux/ubuntu bionic stable"

Update again the list of packages.

apt update

Install Docker-CE from the Docker repository.

apt install docker-ce

Check the status of installed and running Docker CE.

systemctl status docker

The successful Docker installation will show the output like this.

● docker.service - Docker Application Container Engine
   Loaded: loaded (/lib/systemd/system/docker.service; enabled; vendor preset: enabled)
   Active: active (running) since Wed 2020-12-23 03:38:49 UTC; 22min ago
     Docs: https://docs.docker.com
 Main PID: 11281 (dockerd)
    Tasks: 8
   CGroup: /system.slice/docker.service
           └─11281 /usr/bin/dockerd -H fd:// --containerd=/run/containerd/containerd.sock

Dec 23 03:38:49 ubuntu-s-1vcpu-1gb-sgp1-01 dockerd[11281]: time="2020-12-23T03:38:49.030947689Z" level=info msg="ClientConn switching balancer to \"pick_first\"" module=grpc
Dec 23 03:38:49 ubuntu-s-1vcpu-1gb-sgp1-01 dockerd[11281]: time="2020-12-23T03:38:49.089856532Z" level=warning msg="Your kernel does not support swap memory limit"
Dec 23 03:38:49 ubuntu-s-1vcpu-1gb-sgp1-01 dockerd[11281]: time="2020-12-23T03:38:49.090228665Z" level=warning msg="Your kernel does not support CPU realtime scheduler"
Dec 23 03:38:49 ubuntu-s-1vcpu-1gb-sgp1-01 dockerd[11281]: time="2020-12-23T03:38:49.090615892Z" level=info msg="Loading containers: start."
Dec 23 03:38:49 ubuntu-s-1vcpu-1gb-sgp1-01 dockerd[11281]: time="2020-12-23T03:38:49.237818403Z" level=info msg="Default bridge (docker0) is assigned with an IP address 172.17.0.0
Dec 23 03:38:49 ubuntu-s-1vcpu-1gb-sgp1-01 dockerd[11281]: time="2020-12-23T03:38:49.321685622Z" level=info msg="Loading containers: done."
Dec 23 03:38:49 ubuntu-s-1vcpu-1gb-sgp1-01 dockerd[11281]: time="2020-12-23T03:38:49.352831979Z" level=info msg="Docker daemon" commit=f001486 graphdriver(s)=overlay2 version=20.1
Dec 23 03:38:49 ubuntu-s-1vcpu-1gb-sgp1-01 dockerd[11281]: time="2020-12-23T03:38:49.353406920Z" level=info msg="Daemon has completed initialization"
Dec 23 03:38:49 ubuntu-s-1vcpu-1gb-sgp1-01 systemd[1]: Started Docker Application Container Engine.
Dec 23 03:38:49 ubuntu-s-1vcpu-1gb-sgp1-01 dockerd[11281]: time="2020-12-23T03:38:49.399081326Z" level=info msg="API listen on /var/run/docker.sock"

Next, try to log in to Docker using your Docker username and password.

docker login -u didinj

If you get this error.

Error saving credentials: error storing credentials - err: exit status 1, out: `Cannot autolaunch D-Bus without X11 $DISPLAY`

The solution for that error is storing Docker credentials using docker-credentials-pass. First, download the docker-credentials-pass.

wget https://github.com/docker/docker-credential-helpers/releases/download/v0.6.3/docker-credential-pass-v0.6.3-amd64.tar.gz

Extract that downloaded Tar file.

tar -xvf docker-credential-pass-v0.6.3-amd64.tar.gz

Change the permission for this file.

chmod u+x docker-credential-pass

Move that file to /usr/bin.

mv docker-credential-pass /usr/bin

Run the GPG command to generate the GPG key.

gpg --generate-key

Fill out all of the questions then enter your passphrase. If the key generation hangs or freezes in this part.

We need to generate a lot of random bytes. It is a good idea to perform
some other action (type on the keyboard, move the mouse, utilize the
disks) during the prime generation; this gives the random number
generator a better chance to gain enough entropy.

SSH again in another terminal tab or shell then run this command.

dd if=/dev/vda1 of=/dev/zero

Change "/dev/vda1" to your VPS main volume then wait for a few minutes. You will see this in the previous terminal.

gpg: /root/.gnupg/trustdb.gpg: trustdb created
gpg: key F49BA7D33A1D4B78 marked as ultimately trusted
gpg: directory '/root/.gnupg/openpgp-revocs.d' created
gpg: revocation certificate stored as '/root/.gnupg/openpgp-revocs.d/40901CB6F51D9069FB4CDC8DF49BA7D33A1D4B78.rev'
public and secret key created and signed.

pub   rsa3072 2020-12-23 [SC] [expires: 2022-12-23]
      40901CB6F51D9069FB4CDC8DF49BA7D33A1D4B78
uid                      Didin Jamaludin <[email protected]>
sub   rsa3072 2020-12-23 [E] [expires: 2022-12-23]

Copy the characters under pub the run-pass init command with those characters (install pass package if it does not exist).

pass init 40901CB6F51D9069FB4CDC8DF49BA7D33A1D4B78

Run again this command then set the password "pass is initialized".

pass insert docker-credential-helpers/docker-pass-initialized-check

To check it, type this command then enter the previously entered passphrase.

pass show docker-credential-helpers/docker-pass-initialized-check

Now, it should show this line.

pass is initialized

To see the Docker credentials, type this command.

docker-credential-pass list

You will see the blank object.

{}

Next, create a new folder with the JSON file.

mkdir ~/.docker
nano ~/.docker/config.json

Fill that file with this.

{
"credsStore": "pass"
}

Now, the Docker login should work.

docker login -u didinj
Password: 
Login Succeeded

If you get the error "pass store is uninitialized" in a future run, run the below command.

pass show docker-credential-helpers/docker-pass-initialized-check

Next, to install the Docker Compose, simply run this command.

apt install docker-compose

Now, Docker and docker-compose are ready to work with Gitlab CI.

Step #3: Create a New Spring Boot and MySQL App

As usual, we will create a new Spring Boot and MySQL app by generating the project using start.spring.io.

Fill in the required fields and radio button and select the required library as described in the picture then click the Generate button. The initial Spring Boot project was downloaded. Extract then move to your Projects folder. Open this Spring Boot project with your IDE or Text Editor. Some IDE will automatically resolve and download dependencies when the project opens.

Before configuring the MySQL connection, first, we need to create a new database and user for this tutorial. We assume you have installed the MySQL or MariaDB server and client. Run this command from the terminal to enter the MySQL console.

mysql -u root

Next, create a database and its user using these MySQL commands.

create database MySpringBootApp;
create user 'MyUser'@'%' identified by 'MyPassword';
grant all on MySpringBootApp.* to 'MyUser'@'%';
flush privileges;

Next, configure the MySQL connection by opening and editing "src/main/resources/application.properties" and then add these lines of MySQL connection parameter.

spring.jpa.database-platform=org.hibernate.dialect.MySQL5Dialect
spring.jpa.hibernate.ddl-auto=update
spring.datasource.url=jdbc:mysql://${MYSQL_HOST:localhost}:3306/MySpringBootApp?useUnicode=true&useJDBCCompliantTimezoneShift=true&useLegacyDatetimeCode=false&serverTimezone=Asia/Jakarta
spring.datasource.username=MyUser
spring.datasource.password=MyPassword

Also, we need to configure the embedded database for the test because there's no MySQL server in the Gitlab CI jobs. For that, create a folder and a property file in the test folder.

mkdir src/test/resources
touch src/test/resources/application.properties

Fill that file with this H2 Database connection parameters.

spring.datasource.driver-class-name=org.h2.Driver
spring.datasource.url=jdbc:h2:mem:db;DB_CLOSE_DELAY=-1
spring.datasource.username=sa
spring.datasource.password=sa

Also, we need to install the embedded package for the H2 database. Open and edit pom.xml then add this dependency.

<dependencies>
...
    <dependency>
        <groupId>com.h2database</groupId>
        <artifactId>h2</artifactId>
    </dependency>
</dependencies>     

Next, create a package and a simple model/domain/entity class "src/main/java/com/djamware/myspringbootapp/entity/MyProduct.java" then replace all Java code with this.

package com.djamware.myspringbootapp.entity;

import lombok.*;
import org.springframework.data.annotation.Id;

import javax.persistence.Entity;
import javax.persistence.GeneratedValue;
import javax.persistence.GenerationType;
import java.io.Serializable;

@Getter
@Setter
@Builder
@NoArgsConstructor
@AllArgsConstructor
@Entity
public class MyProduct implements Serializable {
    @javax.persistence.Id
    @Id
    @GeneratedValue(strategy= GenerationType.AUTO)
    private Integer id;

    private String prodName;

    private String prodDesc;

    private Double prodPrice;
}

Next, create a package and a simple repository class "src/main/java/com/djamware/myspringbootapp/repository/MyProductRepository.java" then replace all Java code with this.

package com.djamware.myspringbootapp.repository;

import com.djamware.myspringbootapp.entity.MyProduct;
import org.springframework.data.repository.CrudRepository;

public interface MyProductRepository extends CrudRepository<MyProduct, Integer> {
}

Next, create a package and a simple REST controller class "src/main/java/com/djamware/myspringbootapp/repository/MyProductController.java" then replace all Java code with this.

package com.djamware.myspringbootapp.controller;

import com.djamware.myspringbootapp.entity.MyProduct;
import com.djamware.myspringbootapp.repository.MyProductRepository;
import org.springframework.stereotype.Controller;
import org.springframework.web.bind.annotation.*;

@Controller
@RequestMapping(path="/product")
public class MyProductController {

    private final MyProductRepository myProductRepository;

    public MyProductController(MyProductRepository myProductRepository) {
        this.myProductRepository = myProductRepository;
    }

    @GetMapping()
    public @ResponseBody
    Iterable<MyProduct> getAllUsers() {
        return myProductRepository.findAll();
    }

    @PostMapping()
    public @ResponseBody String addNewUser (@RequestBody MyProduct myProduct) {
        MyProduct newProduct = MyProduct.builder()
                .prodName(myProduct.getProdName())
                .prodDesc(myProduct.getProdDesc())
                .prodPrice(myProduct.getProdPrice())
                .build();
        myProductRepository.save(newProduct);
        return "Saved";
    }
}

Next, run the application by clicking the play button in the IDE or type this command in the terminal.

mvn spring-boot:run

If there's no error found, that means the Spring Boot MySQL app is ready to deploy.

Step #4: Dockerize Spring Boot and MySQL App

While your IDE still opens the Spring Boot MySQL project. Add this file to the root of the project folder.

touch Dockerfile

Add these lines of Docker commands.

FROM openjdk:11
LABEL maintainer="[email protected]"
VOLUME /tmp
EXPOSE 8080
ARG JAR_FILE=target/*.jar
COPY ${JAR_FILE} myspringbootapp-0.0.1-SNAPSHOT.jar
ENTRYPOINT ["java","-Dspring.profiles.active=dev","-jar","myspringbootapp-0.0.1-SNAPSHOT.jar"]

We are using Docker OpenJDK-11 image same JDK version as the previously generated Spring Boot app. The volume to save the Docker image is set to /tmp directory of the Host. A port using standard Java Web app port 8080. ARG JAR_FILE means the built jar file save to the target directory with the JAR extension. COPY and ENTRYPOINT should point to the Spring Boot jar file name exactly as the artifactId and version in the pom.xml.

Next, to simplify and Dockerize every server to run the Spring Boot application on VPS, we will use Docker Compose. For that, SSH again to the VPS then create a docker-compose.yml file in the home or default ROOT directory.

nano docker-compose.yml

Then fill that file with these Docker Compose parameters.

version: '3.3'
services:
  nginx:
    container_name: nginx
    image: nginx:1.13
    restart: always
    ports:
      - 80:80
      - 443:443
    network_mode: host
    hostname: localhost
    volumes:
      - ./nginx/conf.d:/etc/nginx/conf.d
    depends_on:
      - springboot
    environment:
      TZ: Asia/Jakarta
    healthcheck:
      test: "exit 0"
  springboot:
    restart: always
    image: didinj/myspringbootapp
    container_name: springboot
    ports:
      - 8080:8080
    network_mode: host
    hostname: localhost
    healthcheck:
      test: "exit 0"

We use Nginx as an HTTP reverse proxy for the Spring Boot app. The Nginx container use image from the Docker repository. So, the configuration will use a separate file. For that, create a new folder and Nginx configuration file.

mkdir nginx
mkdir conf.d
nano nginx/conf.d/app.conf

Then fill that configuration file with these lines of Nginx configuration.

server {
    listen 80;
    location / {
        proxy_pass http://127.0.0.1:8080/;
    }
}

As you see in the docker-compose file, that configuration file will copy to the Nginx container to this directory /etc/nginx/conf.d. The Spring boot container uses a Docker image from the docker hub repository. For MySQL, we will run it using the Docker run command because we have an error using Docker Compose for MySQL.

[ERROR] [Entrypoint]: Database is uninitialized and password option is not specified
    You need to specify one of MYSQL_ROOT_PASSWORD, MYSQL_ALLOW_EMPTY_PASSWORD and MYSQL_RANDOM_ROOT_PASSWORD

So, we will create an environment variables file to load the MySQL root password, database, user, and password. Type this command to create it.

nano mysql.env

Add these lines of MYSQL parameters.

MYSQL_ROOT_PASSWORD=my-secret-pw
MYSQL_DATABASE=MySpringBootApp
MYSQL_USER=MyUser
MYSQL_PASSWORD=MyPassword

To store the MySQL data in the host volume, create a new directory.

mkdir mysql-data

Next, run the Docker for MySQL by typing this command.

docker run --name mysql --network host --env-file=mysql.env --volume=mysql-data:/var/lib/mysql -d mysql:latest

To check the running MySQL Docker container, type this command.

docker ps -a

Step #5: Setup Gitlab and Gitlab-CI

To set up a Gitlab repository and Gitlab CI/CD, login to Gitlab.com using your own account or your organization account. Click the New Project button then click create a blank project.

Fill the project slug name with the Spring boot docker image name as in the Docker compose file. Leave other options as default then click the Create Project button. Next, back to the terminal and Spring boot project folder. Type these commands to add the Gitlab repository.

git init
git remote add origin [email protected]:didinj/myspringbootapp.git
git add .
git commit -m "Initial commit"
git push -u origin master

Next, back to the Gitlab project dashboard and click the settings -> CI/CD menu on the left navigation.

Expand Variables then add Variable. Enter all required variables that will use in the Gitlab-CI.yml file (SSH_PRIVATE_KEY, DOCKER_USER, DOCKER_PASSWORD, DOCKER_REPO, APP_NAME, SERVER).

Which is SSH_PRIVATE_KEY value is the SSH key that was previously generated before SSH to the VPS. DOCKER_USER and DOCKER_PASSWORD are the Docker HUB username and password. DOCKER_REPO is the Docker HUB repository that was previously created. APP_NAME is the Spring boot app name. SERVER is the VPS IP address.

Next, we need to install the Gitlab Runner in the VPS. SSH again to the VPS then type this command to add the official Gitlab repository.

curl -L "https://packages.gitlab.com/install/repositories/runner/gitlab-runner/script.deb.sh" | sudo bash

Install the latest version of Gitlab runner.

export GITLAB_RUNNER_DISABLE_SKEL=true; sudo -E apt-get install gitlab-runner

Next, back to the Gitlab Settings CI/CD and expand the Runners.

Copy the Registration token in the Specific Runners then back to the VPS and type this command to set up the runners.

gitlab-runner register

Enter the Gitlab instance URL.

https://gitlab.com/

Enter the registration token by pasting the previously copied token then pressing enter. Enter a description for the runner.

myspringbootapp-runner

Enter tags for the runner.

test,myspringbootapp,master,dev

Enter an executor: virtualbox, docker-ssh+machine, kubernetes, parallels, shell, ssh, docker+machine, custom, docker, docker-ssh.

docker

Enter the default Docker image.

openjdk:11

If you refresh the Gitlab Settings CI/CD Runners, you will see the runners activated for this project.

Next, back to the IDE or Text editor then create a new file in the root of the project folder.

touch .gitlab-ci.yml

Fill the .gitlab-ci.yml with this.

stages:
  - test
  - build jar
  - build and push docker image
  - deploy
test_job:
  image: maven:3.6.2-jdk-11-slim
  stage: test
  script:
    - pwd
    - mvn clean
    - mvn compile
    - mvn test
  tags:
    - test
  only:
    - master
build:
  image: maven:3.6.2-jdk-11-slim
  stage: build jar
  script:
    - mvn clean install -Dspring.profiles.active=dev && mvn package -B -e -Dspring.profiles.active=dev
  artifacts:
    paths:
      - target/*.jar
  only:
    - master
docker build:
  services:
    - docker:19.03.13-dind
  image: docker:19.03.13-dind
  stage: build and push docker image
  script:
    - docker build --build-arg SPRING_ACTIVE_PROFILE=dev -t $DOCKER_REPO:latest .
    - docker login -u $DOCKER_USER -p $DOCKER_PASSWORD docker.io
    - docker push $DOCKER_REPO:latest
  only:
    - master
deploy:
  image: ubuntu:latest
  stage: deploy
  before_script:
    - 'which ssh-agent || ( apt-get update -y && apt-get install openssh-client -y )'
    - eval $(ssh-agent -s)
    - echo "$SSH_PRIVATE_KEY" | tr -d '\r' | ssh-add -
    - mkdir -p ~/.ssh
    - chmod 700 ~/.ssh
    - echo -e "Host *\n\tStrictHostKeyChecking no\n\n" > ~/.ssh/config
  script:
    - pwd
    - ssh root@$SERVER "docker login -u $DOCKER_USER -p $DOCKER_PASSWORD docker.io; docker stop $APP_NAME; docker system prune -a -f; docker run --name mysql --network host --env-file=mysql.env --volume=mysql-data:/var/lib/mysql -d mysql:latest; docker pull $DOCKER_REPO:latest; docker-compose up -d; docker logout"
  only:
    - master

We are creating 4 stages of Gitlab CI/CD. First test the Spring Boot app, build the Spring Boot app to JAR, build the Docker images then push to the Docker repository, and SSH to VPS then pull the Docker image from the Docker repository then run the Docker compose.

Next, commit and push the updated Spring Boot app with .gitlab-ci.yml file.

git add .
git commit -m "Add .gitlab-ci.yml"
git push

If the Gitlab CI jobs working, you should see this at the checkout for all stages in the Gitlab CI/CD pipelines.

To check the deployed Spring Boot application, back to the VPS by SSH then type this command.

docker ps -a

You should see this Docker container running up and healthy.

Finally, you can check the application by accessing it through the VPS IP address. In the terminal type this CURL POST command after exiting the VPS SSH.

curl -i -X POST -H "Content-Type: application/json" -d '{"prodName":"Dummy Product 1","prodDesc":"The Fresh Dummy Product in The world part 1","prodPrice":100}' http://139.59.243.46/product

The successful response should be like this.

HTTP/1.1 200 
Server: nginx/1.13.12
Date: Thu, 31 Dec 2020 23:02:17 GMT
Content-Type: text/plain;charset=UTF-8
Content-Length: 5
Connection: keep-alive

And GET request to the API.

curl http://139.59.243.46/product

The successful response should be like this.

[{"id":1,"prodName":"Dummy Product 1","prodDesc":"The Fresh Dummy Product in The world part 1","prodPrice":100.0},{"id":2,"prodName":"Dummy Product 1","prodDesc":"The Fresh Dummy Product in The world part 1","prodPrice":100.0}]

That it's, The Spring Boot MySQL Nginx Gitlab-CI Docker on VPS tutorial. You can get the full source code from our GitHub.

That is just the basics. If you need more deep learning about Java and Spring Framework you can take the following cheap course:

• Java basics, Java in Use //Complete course for beginners
• Java Programming: Master Basic Java Concepts
• Master Java Web Services and REST API with Spring Boot
• JDBC Servlets and JSP - Java Web Development Fundamentals
• The Complete Java Web Development Course
• Spring MVC For Beginners: Build Java Web App in 25 Steps
• Practical RESTful Web Services with Java EE 8 (JAX-RS 2.1)

Thanks!