Deploy and Secure Airbyte with Nginx Reverse Proxy, Basic Authentication & Let’s Encrypt SSL Certificates

Index

Introduction

Introduction

In this post we will deploy Airbyte, one of the most exciting Open source ELT tools in modern data engineering. This is an ongoing series of posts on deploying and using Airbyte for data engineering use-cases. There is already a deployment guide available for Airbyte on OCI. This setup is a production grade setup build using components on Oracle Cloud Infrastructure (OCI), with minimum cost and by using the Always Free tier available on OCI you can build it for almost $0.

Architecture

The stack has the below important components, it is a mix of Network and IaaS components on Oracle Cloud Infrastructure which will host Nginx and Airbyte. For the Nginx deployment we will use OCI ARM based A1 instances and for Airbyte we will use a AMD E4 Flex instance. Both these instances are available in the always free tier

  1. OCI AMD E4 Flex Instance — Airbyte on Docker
  2. OCI DNS Public Zone which has Domain Management and the A-records are added here
  3. VCN (Virtual Cloud Network) — 2 Subnets , 1 Public subnet hosting the Nginx VM, 1 Private Subnet running the Airbyte Docker container

Solution Deployment

A. Deploy the Virtual Machines for Nginx and Airbyte in Public and Private Subnet Respectively

  1. Deploy OCI ARM Instance in Public Subnet and install nginx on it. Ensure port 80 is allowed in security list of Public subnet as stateless rule

sudo setsebool -P httpd_can_network_connect 1
[nginx]
name=nginx repo
baseurl=https://nginx.org/packages/rhel/$releasever/$basearch/
gpgcheck=0
enabled=1
## Install nginxsudo yum install nginx
sudo systemctl start nginx
sudo systemctl status nginx
sudo systemctl enable nginx
## Whitelist HTTP Port 80 on the Instance for External Acessssudo firewall-cmd -zone=public -permanent -add-port=80/tcp
sudo firewall-cmd -zone=public -permanent -add-service=http
sudo firewall-cmd -reload
sudo firewall-cmd -zone=public -permanent -list-ports

B. Configure HTTP Basic Authentication and Install Nginx

1. Configure nginx to act as reverse proxy for Airbyte with basic http authentication

sudo mkdir -p /etc/apache2/
sudo htpasswd -c /etc/apache2/.htpasswd admin
sudo vim /etc/nginx/nginx.conf
user root;events {
worker_connections 4096; ## Default: 1024
}
http {
server {
listen 80;
listen [::]:80;
server_name 10.10.1.138;location / {
proxy_pass http://10.10.1.147:8000;
proxy_set_header X-Forwarded-User $http_authorization;
auth_basic “Administrator’s Area”;
auth_basic_user_file /etc/apache2/.htpasswd;
proxy_set_header X-Real-IP $remote_addr;
proxy_set_header Host $host;
proxy_set_header X-Forward-For $proxy_add_x_forwarded_for;
proxy_pass_header Accept;
proxy_pass_header Server;
proxy_http_version 1.1;
proxy_set_header Authorization $http_authorization;
proxy_pass_header Authorization;
proxy_set_header ns_server-ui yes;
}
}
}
sudo nginx -t
sudo systemctl restart nginx
http://<public-ip>/
sudo tail -30f /var/log/nginx/error.log
sudo yum install -y yum-utils
sudo yum-config-manager — enable ol7_optional_latest
sudo yum-config-manager — enable ol7_developer_EPEL
cd /tmp
wget https://dl.fedoraproject.org/pub/epel/epel-release-latest-7.noarch.rpm
sudo rpm -Uvh /tmp/epel-release-latest-7.noarch.rpm
sudo yum install certbot
sudo yum install python-certbot-nginx
sudo vi /etc/nginx/nginx.conf
user root;events {
worker_connections 4096; ## Default: 1024
}
http {
server {
listen 80;
listen [::]:80;
server_name airbyte.yourdomain.com www.airbyte.yourdomain.com;location / {
proxy_pass http://10.10.1.147:8000;
proxy_set_header X-Forwarded-User $http_authorization;
auth_basic “Administrators Area”;
auth_basic_user_file /etc/apache2/.htpasswd;
proxy_set_header X-Real-IP $remote_addr;
proxy_set_header Host $host;
proxy_set_header X-Forward-For $proxy_add_x_forwarded_for;
proxy_pass_header Accept;
proxy_pass_header Server;
proxy_http_version 1.1;
proxy_set_header Authorization $http_authorization;
proxy_pass_header Authorization;
proxy_set_header ns_server-ui yes;
}
}
}
sudo nginx -s reload
sudo certbot — nginx -d airbyte.yourdomain.com -d www.airbyte.yourdomain.com
sudo grep -r -P ‘[^\x00-\x7f]’ /etc/apache2 /etc/letsencrypt /etc/nginx
sudo certbot — nginx -d airbyte.yourdomain.com -d www.airbyte.yourdomain.com
sudo firewall-cmd -zone=public -permanent -add-port=443/tcp
sudo firewall-cmd -zone=public -permanent -add-service=https
sudo firewall-cmd -reload
sudo firewall-cmd -zone=public -permanent -list-ports

Summary

We have seen how easy it is to create a production grade Airbyte deployment on Oracle Cloud Infrastructure using Lets Encrypt, Nginx and Http Authentication. This stack can be build with almost zero cost using the Oracle Cloud Always Free tier (barring the DNS domain cost)

References:

[1] Oracle Cloud Compute E4 platform — https://blogs.oracle.com/cloud-infrastructure/post/announcing-oracle-cloud-compute-e4-platform-on-third-gen-amd-epyc-processors

Principal Cloud Solutions Architect at Oracle