Building Scalable Backend & Distributed Systems

Architecting high-throughput backend systems that scale.

Backend engineer with 4+ years of production experience across Go and Python. I currently engineer solar forecasting pipelines at Open Climate Fix — building the mission-critical systems that predict how much power the grid generates from the sun, and keeping predictions flowing seamlessly even when satellite feeds cut out.

GitHub (68+ PRs)LinkedIn
Solar Forecasting Curve
Open Climate Fix
06:0008:0010:0012:0014:0016:0018:0020:00
what grid produced
what we said
4+Years

Backend engineering across Go, Python, and distributed systems.

30%Cost Cut

Reduced Airflow DAG runtimes via gRPC Data Platform migration.

98%Reliability

Drop in payout errors via Go microservices with AWS SQS retries.

68+Merged PRs

Public contributions across 10 Open Climate Fix repositories.

Career Journey

Production Experience

Delivering resilient distributed systems, ML pipelines, and high-concurrency microservices.

Backend Development Engineer

Active
Open Climate Fix·London, UK (Remote)
Jan 2026 — Present

Lead backend architecture for multi-national solar forecast blending pipelines, ingesting satellite feeds and ML predictions to calculate solar generation across UK, India, and EU grids.

  • Architect and maintain the solar forecast blending service end-to-end across three country deployments, blending day-ahead and 6-hour models with automated failover during satellite telemetry dropouts.
  • Refactored the blending engine to be dynamically driven by a centralized BlendConfig rather than hardcoded per-country forks, enabling instantaneous multi-region rollouts via config commits.
  • Built Hexatron, an internal operational management portal in Go that triggers Airflow DAGs and surfaces live pipeline execution logs to engineers in real time.
  • Migrated 8 forecasting applications onto the internal gRPC Data Platform, cutting data-saving costs 30% by shortening Airflow DAG runtimes.
  • Overhauled API authorization to dp-sdk v0.32.0, moving permissions from per-site access to multi-tenant organization scope using HubSpot company metadata in JWT claims.
  • Engineered an interactive Streamlit data-science console for machine learning researchers to visualize multi-model forecast divergence.
  • Authored 68+ merged pull requests across 10 open-source repositories in the Open Climate Fix org (solar-consumer, quartz-api, pv-site-api, india-forecast-app, uk-pv-forecast-blend, pv-site-production).
GoPythongRPCAirflowStreamlitPostgreSQLJWT AuthDockerAWSDistributed Pipelines
Verify 68+ PRs on GitHub

Software Engineer (Backend)

Orange Health Labs·Bengaluru, India
Feb 2025 — Jan 2026

Architected core diagnostic logistics and financial transaction pipelines using Go and Python, ensuring high reliability for time-critical medical diagnostics.

  • Engineered a decoupled payout architecture in Go backed by AWS SQS and background worker pools, slashing partner payout failure rates by 98% with automated idempotent retries.
  • Implemented fine-grained JWT RBAC for partner invoice systems, securing sensitive healthcare billing records across laboratory franchises.
  • Built an automated monthly invoicing engine with pre-settlement 'dry run' validation in Go, cutting invoice errors 95%.
  • Migrated high-throughput order and sample lookup services from AWS RDS to Elasticsearch, dropping search query latency by 96% while removing database CPU spikes.
  • Designed asynchronous task queues and messaging topologies using RabbitMQ, Redis, and AWS SQS for sample collection routing.
GoPythonGinDjangoElasticsearchAWS SQSRabbitMQRedisPostgreSQLJWT RBAC

Software Engineer Professional

Kaleyra (now Tata Communications)·Bengaluru, India
Jul 2022 — Jan 2025

Built high-throughput communication microservices and fault-tolerant conversational bot engines processing thousands of requests per second.

  • Built the FastAPI chatbot service to handle thousands of requests per second, with session state cached in Redis ElastiCache, cutting latency 33% from 1.5s to under 1s.
  • Engineered real-time log ingestion microservices with RabbitMQ streaming to MongoDB, moving writes off the request path to cut request latency 20%.
  • Optimized MongoDB data schemas and implemented database sharding, improving complex document lookup speeds by 18%.
  • Owned the chatbot availability redesign, choosing an active-active multi-region AWS topology with cross-region replication and failover to end single-region downtime.
  • Built an intelligent request retry layer that rescued 5% of peak-traffic dropped webhook payloads.
PythonFastAPIRedis ElastiCacheMongoDB (Sharding)RabbitMQActive-Active AWSFault Tolerance

Software Engineer Intern

Kaleyra·Bengaluru, India
Apr 2022 — Jul 2022
  • Built 4 full-stack applications utilizing the MERN stack (MongoDB, Express.js, React, Node.js) for internal customer support automation.
  • Optimized relational and document database query execution plans, delivering a 20% latency reduction.
  • Integrated third-party REST APIs and automated cloud deployments, improving test pipeline reliability by 30%.
MERN StackNode.jsExpress.jsMongoDBRESTful APIsCloud Infrastructure
Case Studies

Engineering Deep Dives

The work I am best at: root-causing production bottlenecks, fixing subtle distributed systems edge cases, and hardening data pipelines.

Data Ingestion & Postgres

Generation Data Vanishing on Pipeline Re-runs

100% Data Integrity Restored
Root Cause / Issue

Deduplication filtered on timestamp alone. Once national-level aggregate solar data was saved for a given timestamp, every regional GSP (Grid Supply Point) sharing that timestamp was silently dropped — zero errors thrown, but missing regional data downstream.

Resolution & Impact

Corrected the deduplication filter so regional GSPs are tracked independently of the national aggregate at the same timestamp, and pinned the fix with a regression test.

Key Takeaways
  • Single-field deduplication is a silent data loss trap — always use composite keys when records share natural dimensions.
  • Absence of errors doesn't mean correctness; added downstream data-count regression tests to catch shape mismatches.
  • Pinned the fix with an automated regression suite that validates row counts per location after every pipeline run.
Stack:PythonPostgreSQLAirflowPytest
Capabilities

Stack & Production Tooling

Languages, frameworks, databases, and cloud infrastructure I have shipped to production.

Go

Go (Golang)High-throughput services
Gin & ChiREST APIs & middleware
Goroutines & ChannelsConcurrent pipelines
Worker PoolsAsync jobs & SQS workers
GORM & SQLTransactional DB ops
pprof & BenchmarksMemory & CPU profiling

Python

FastAPIAsync microservices
Django & FlaskEnterprise web backends
Celery & AsyncIOEvent-driven background jobs
StreamlitData science tools & portals
Pytest & MockRegression testing suites

Data & Messaging

PostgreSQL & MySQLRelational query tuning
MongoDB (Sharding)Document store scaling
ElasticsearchSub-second full text & log search
Redis / ElastiCacheLow-latency distributed cache
RabbitMQ & AWS SQSReliable decoupled queues
Airflow & gRPCDAG orchestration & streaming

Cloud & Infrastructure

AWS (EC2, S3, RDS, SQS)Production cloud infra
Docker & ContainersReproducible builds
Active-Active ArchitectureCross-region resilience
GitHub Actions CI/CDAutomated test & deploy
JWT & OAuth AuthMulti-tenant RBAC security
Let's Connect

Get in Touch

Open to backend engineering roles, distributed systems consulting, and technical conversations.

Send a Message

Pavan Kulkarni — Backend Engineer

Go · Python · Distributed Systems · Production Solar Forecasting