Self Hosted Gitlab Pipeline End to End

Gitlab Set Up on Ec2 Instance/GCP VM

Prerequisites

• A running EC2 instance or GCP VM with Ubuntu 20.04 or later.
• SSH access to the instance.
• Security group/Firewall rules allowing HTTP (80, 8088 or your-port) and HTTPS (443) traffic.
• Basic knowledge of GitLab and CI/CD concepts.
• Domain name (optional, for accessing GitLab via a custom domain).

Installing Docker and Docker Compose on your EC2 instance or GCP VM is essential for running GitLab. Here’s how to do it:

# Add Docker's official GPG key:
sudo apt-get update
sudo apt-get install ca-certificates curl
sudo install -m 0755 -d /etc/apt/keyrings
sudo curl -fsSL https://download.docker.com/linux/ubuntu/gpg -o /etc/apt/keyrings/docker.asc
sudo chmod a+r /etc/apt/keyrings/docker.asc

# Add the repository to Apt sources:
echo \
  "deb [arch=$(dpkg --print-architecture) signed-by=/etc/apt/keyrings/docker.asc] https://download.docker.com/linux/ubuntu \
  $(. /etc/os-release && echo "${UBUNTU_CODENAME:-$VERSION_CODENAME}") stable" | \
  sudo tee /etc/apt/sources.list.d/docker.list > /dev/null
sudo apt-get update

Other linux distributions can follow the official Docker installation guide.

Installing docker-compose

sudo apt-get update
sudo apt-get install docker-compose-plugin
docker-compose version

Next Create a Docker Compose file for GitLab

mkdir gitlab cd gitlab vi docker-compose.yml

version: '3.8'
services:   
  gitlab:
    image: gitlab/gitlab-ce:latest
    restart: always
    hostname: gitlab.${EXTERNAL_IP}.nip.io # Replace with your domain or IP
    environment:
      GITLAB_OMNIBUS_CONFIG: |
        external_url 'http://{EXTERNAL_IP}:8088'
        gitlab_rails['initial_root_password'] = 'password' # Change this to a secure password
    ports:
      - '8088:8088'
    volumes:
      - gitlab-config:/etc/gitlab
      - gitlab-logs:/var/log/gitlab
      - gitlab-data:/var/opt/gitlab
    restart: unless-stopped

Start GitLab

docker-compose up -d

Access GitLab

• Open your web browser and navigate to http://EXTERNAL_IP: 8088
• Log in with the username root and the password you set in the docker-compose.yml file.
• If you did not set a password in docker-compose.yml, or want to retrive auto-generated password, run following command:

docker exec -it gitlab grep 'Password:' /etc/gitlab/initial_root_password

Configure GitLab

After logging in, you can configure GitLab settings, create projects, and manage users as needed

Create a New repository

1. Click on the New Project button.
2. Choose Create blank project.
3. Fill in the project details:
   • Project name: Enter a name for your project.
   • Can keep other settings as default.
4. Click on Create project.

Create a New Pipeline

1. Navigate to your project.
2. You can clone repository to your local machine using the command:

   git clone http://<EXTERNAL_IP>:8088/<your-username>/<your-project-name>.git

3. Create a new file named .gitlab-ci.yml in the root of your project directory.
4. Before making changes we need runner to execute the pipeline. Can click here to jump how it is set up.

Set Up GitLab Runner

1. Install GitLab Runner on your EC2 instance or GCP VM:

    sudo apt-get install gitlab-runner

2. Create runner in GitLab:

• Go to your GitLab project.
• Navigate to Settings > CI/CD > Runners.
• Click on Expand Create project runner.
• Provide tags, description, and timeout settings as needed.
• Choose the executor type (e.g., shell, docker, etc.).
• Copy the registration token.

3. Register the runner:

sudo gitlab-runner register

• Enter the GitLab instance URL (e.g., http://EXTERNAL_IP: 8088).
• Enter the registration token you copied earlier.
• provide shell as executor.
• if promted for name, you can leave it blank or provide a name for the runner like ec2-runner.

4. Add the following content to the .gitlab-ci.yml file:

stages:
  - build
  - test
  - deploy
build_job:
  stage: build
  script:
    - echo "Building the application..."
    - sleep 10 # Simulating build time
test_job:
  stage: test 
  script:
    - echo "Running tests..."
    - sleep 5 # Simulating test time
deploy_job:
  stage: deploy
  script:
    - echo "Deploying the application..."
    - sleep 5 # Simulating deployment time

5. Commit and push the changes to your GitLab repository:

   git add .gitlab-ci.yml
   git commit -m "Add GitLab CI/CD pipeline"
   git push origin main

check the pipeline status in GitLab:

• Navigate to your project in GitLab.
• click on pipelines in left sidebar.
• You should see the pipeline running with the stages you defined in .gitlab-ci.yml.

Create Child Pipelines

Child pipelines allow you to break down complex CI/CD processes into smaller, manageable parts. Here’s how to create child pipelines in GitLab:

1. Create a Parent Pipeline: In your .gitlab-ci.yml, define a parent pipeline that triggers child pipelines.

frontend:
  trigger:
    include: frontend/.gitlab-ci.yml
    strategy: depend
  rules:
    - changes: [frontend/*]
backend:
  trigger:
    include: backend/.gitlab-ci.yml
    strategy: depend
  rules:
    - changes: [backend/*]

2. Create Child Pipelines: In the frontend and backend directories, create separate .gitlab-ci.yml files for each child pipeline.
3. Example Child Pipeline: Here’s an example of a child pipeline in frontend/.gitlab-ci.yml:

stages:
  - build
  - test
build_frontend:
  stage: build  
  script:
    - echo "Building frontend..."
    - sleep 5 # Simulating build time
test_frontend:
  stage: test 
  script:
    - echo "Testing frontend..."
    - sleep 5 # Simulating test time

4. Example Child Pipeline for Backend: Similarly, create a backend/.gitlab-ci.yml file:

stages:
  - build
  - test
build_backend:
  stage: build
  script:
    - echo "Building backend..."
    - sleep 5 # Simulating build time
test_backend:
  stage: test 
  script:
    - echo "Testing backend..."
    - sleep 5 # Simulating test time

5. Commit and Push Changes: Commit and push the changes to your GitLab repository.

   git add frontend/.gitlab-ci.yml backend/.gitlab-ci.yml
   git commit -m "Add child pipelines for frontend and backend"
   git push origin main

6. Check Child Pipelines: Navigate to your project in GitLab and check the pipelines section. You should see the parent pipeline triggering the child pipelines for frontend and backend.

Conclusion

In this guide, we covered how to set up a self-hosted GitLab instance on an EC2 instance or GCP VM, create a new repository, set up a basic CI/CD pipeline, and create child pipelines. This setup allows you to manage your code and automate your development workflows effectively. Feel free to customize the .gitlab-ci.yml files and child pipelines according to your project’s requirements. GitLab provides extensive documentation and community support to help you explore more advanced features and configurations. For more information, refer to the GitLab CI/CD documentation.

Additional Resources

• GitLab Runner Documentation
• GitLab CI/CD Pipelines Documentation

Troubleshooting

• If you encounter issues with the GitLab Runner, check the logs using:

  sudo gitlab-runner status

• Ensure that the GitLab instance is accessible from your browser.
• Verify that the ports are correctly configured in your security group/firewall settings.