In the “Cloud First” era, we have become accustomed to the convenience of provisioning databases or Kubernetes clusters with just a single click. However, the reality is that many organizations still need to maintain On-Premise Data Centers due to security, compliance, or cost management reasons.
The challenge for our Platform Infrastructure team was clear:
"How can we make our existing hardware (VMware vSphere & Enterprise Storage) deliver the same fast, automated, and flexible Developer Experience (DX) as a Public Cloud like AWS or Google Cloud?"
This article takes you behind the scenes of the Tech Stack and Design Choices we selected to transform bare metal “Hardware” into a fully functional “Internal Developer Platform (IDP).” To provide a clear picture of how these components work together, we have categorized the architecture into the following key sections:
1. The Foundation: Infrastructure as Code (IaC)
The first step in building a platform is eliminating manual operations. We shifted everything to “Code” for precision and speed.
Compute Provisioning (Terraform): We said goodbye to manual VM cloning and embraced Terraform as our primary tool for vSphere IaC. This allows us to version control our infrastructure and provision new VMs in minutes, virtually eliminating human error.
DNS as a Service (PowerDNS): A classic on-prem bottleneck is the long wait for IP/DNS assignment. We solved this with PowerDNS due to its API-First approach. As soon as Terraform finishes creating a VM, it triggers an API call to update the DNS records immediately—no more waiting for admins to map IPs.
2. Kubernetes Automation: The “10-Minute” Cluster
Kubernetes is the heart of modern applications, but managing clusters on-premise is notoriously difficult. We chose a purely Cloud-Native approach.
Cluster Management (Cluster API): Instead of relying on imperative Ansible scripts, we utilize Cluster API (CAPI) to manage Kubernetes declaratively.
The Workflow: We define the cluster spec using Helm Templates and apply it to a Management Cluster.
The Result: We can spin up a production-ready Kubernetes cluster, complete with hardening scripts, in under 10 minutes. This also simplifies Day-2 operations like version upgrades or node scaling, making it feel just like a managed cloud service.
Storage Integration (CSI Driver): To ensure seamless communication between Kubernetes and our Enterprise Storage, we use the vendor-native CSI Driver. This enables automatic Persistent Volume Claim (PVC) provisioning and leverages enterprise storage features directly.
3. Application Services: Cloud-Native Experience
With a solid foundation in place, we built a “Service Catalog” that the Application Team can consume without worrying about the underlying infrastructure.
Networking (Kube-VIP & Nginx Ingress):
L4 Load Balance: We use Kube-VIP to simulate a LoadBalancer Service type for Kubernetes (acting like an ELB on the cloud), providing IP addresses for services exposed externally.
L7 Load Balance: Nginx Ingress handles HTTP Routing and SSL Termination—standard tools that developers are already familiar with.
S3-Compatible Storage (MinIO): Modern applications require Object Storage. We deployed MinIO on Kubernetes to provide an S3-compatible API. This empowers developers to manage buckets themselves, removing dependency on legacy SAN/NAS requests.
4. The Bridge: Modernizing Legacy Workflows
One of the most interesting parts of our architecture is bridging the gap between the “Old World” (Legacy) and the “New World” (Cloud-Native).
Smart SFTP Service (SFTPGo + MinIO): Many legacy systems still rely on SFTP for file transfer, but we didn’t want to store files on traditional disks. We chose SFTPGo running on Kubernetes.
The Magic: We configured SFTPGo to use MinIO (Object Storage) as its backend.
The Benefit: The moment a legacy system uploads a file via SFTP, it instantly appears in an S3 Bucket. Modern applications can then trigger workflows or process these files via API seamlessly.
5. Operations & Observability
To ensure the platform is self-sustaining and trustworthy, we provide a comprehensive observability stack.
Logging & Tracing (ECK): We utilize Elastic Cloud on Kubernetes (ECK) to deploy the ELK Stack, enabling the App Team to search logs and investigate issues via a self-service model.
Monitoring (Prometheus & Grafana): We provide centralized dashboards visualizing both Application Metrics and Infrastructure Health, facilitating accurate capacity planning.
Job Scheduler (Airflow): For batch jobs and data pipelines, we host Airflow on Kubernetes to manage complex schedules and workflows systematically.
Conclusion
Transforming On-Premise into a Cloud-like environment isn’t just about installing trendy tools; it’s about designing a Workflow that eliminates blocking processes and converts everything into Self-Service via APIs.
Most importantly, this architecture provides a true “Cloud Agnostic” advantage. Since all selected tools (Terraform, Cluster API, MinIO, etc.) are open standards and not tied to specific hardware or a single cloud provider, we are future-proof. If the business decides to expand to Public Cloud or adopt a Hybrid Cloud strategy tomorrow, we can scale out immediately without the Application Team needing to change their way of working.
This stack allows our Platform Team to work smarter—managing larger scale with the same resources—while enabling the Application Team to deliver business value faster, regardless of where the servers physically reside.
Looking for a DevOps solution that automates your workflow and reduces business costs? SCB TechX helps you modernize your delivery pipeline and bring high-quality products to market faster, building a foundation for long-term growth.
For service inquiries, please contact us at https://bit.ly/4etA8Ym
Learn more: https://bit.ly/3H7W9zm
