Heartin Jacob Kanikathottu

API Gateway ≠ AI Gateway Both sit in front of services, but they solve different classes of problems. API Gateway -Protocol-level: HTTP, REST, GraphQL, gRPC -Stateless request/response routing -Focus on auth, throttling, caching, logging -Examples: Kong, NGINX, AWS API Gateway AI Gateway -Model-level: LLMs, embeddings, multi-model orchestration -Stateful: tracks context, conversation, memory -Handles token accounting, latency metrics, model routing -AI-specific observability (usage, cost per token, provider SLAs) -Examples: OpenRouter, LangChain Gateway, Hugging Face LLM Router Think of it this way: API Gateway optimizes API traffic AI Gateway optimizes model traffic As AI stacks evolve, I expect AI Gateways to become as foundational as API Gateways — especially in enterprises juggling multiple LLMs across vendors.

780

58 Comments

𝐈𝐧𝐬𝐭𝐚𝐠𝐫𝐚𝐦 𝐒𝐲𝐬𝐭𝐞𝐦 𝐃𝐞𝐬𝐢𝐠𝐧 Designing a system like Instagram involves a complex network of components and services. Below is the overview of the system of Instagram: Client Interaction: Users interact through mobile or web apps. Load Balancer: Helps in providing a balance of requests for API gateways. API Gateways: Gateway to the micro services. Write Operations: Uploading comments and uploads sent to the App Server, which further validates the request and writes data along with video processing. Feed Generation Service: Feeds generation and update service for users. Read Operations: Routes read operations (e.g., view feed) to appropriate service(s). Metadata Database: Stores user profiles and post metadata. Caching: Utilises Redis or Memcached for caching, thereby reducing response times and database load. Search Service (Elastic search): Indexes users and content for quickly returning relevant results. Blob Storage: Stores user-uploaded media files (e.g., images, videos). CDN: Caches and serves static content with low latency. Video Processing: Responsible for transcoding, resizing, and creating thumbnails for all user-uploaded videos. Activity Feeds: It lets users know about the likes, comments, and interactions. 𝐉𝐨𝐢𝐧 𝐦𝐲 𝐲𝐨𝐮𝐭𝐮𝐛𝐞 𝐜𝐡𝐚𝐧𝐧𝐞𝐥 𝐭𝐨 𝐀𝐜𝐞 𝐒𝐲𝐬𝐭𝐞𝐦 𝐃𝐞𝐬𝐢𝐠𝐧 https://lnkd.in/dASFyabu Did I miss anything? Comment below if there are other areas or different system design aspects you would like me to cover.

800

33 Comments

Can you answer these questions about your cloud infra deployments if your manager asks you today? 🙋 ↳ “Who changed the prod database config three weeks ago?” ↳ “Do we have any configuration drift?” ↳ “Can you show me exactly who in the org can access what?” ↳ “Do we have any hardcoded secrets in our IaC repos?” If you can’t answer all of these confidently, don’t freak out, you’re probably the majority and not the exception. But let’s change that. I just finished recording a deep dive where I walk through the most common IaC security gaps teams struggle with during audits, and how to fix them. This video has practical advice you can immediately turn into action plans. ➡️ Watch it here → https://lnkd.in/gnPuxshu What IaC security issues have you commonly seen? LMK in the comments 👇 ♻️ Share with your colleagues who need this before the next audit And follow Christophe Limpalair for more.

54

4 Comments

When does it pay to ditch Kafka’s Traditional Replication? I built a tool to help you visualize just that. The tool compares the cost of Kafka's replication in all 3 major clouds (AWS, GCP, Azure) with the alternative design of writing directly to the object store that vendors like Warpstream and Bufstream have adopted. By all accounts, this new design is going to be the future of Kafka. The timeline of adoption was roughly the following: • August 2023 - Warpstream announced out of stealth • May 2024 - Confluent Freight announced • May 2024 - Ursa released in StreamNative cloud • July 2024 - Bufstream 0.0.1 released • September 2024 - Confluent buys Warpstream • September 2024 - RedPanda cloud topics anounced • February 2025 - I posted a napkin design on how to achieve this in Kafka lol. just kidding on the last one 😂 With these direct-to-S3 designs, you have to be efficient at batching the S3 writes, otherwise it can end up being more expensive. In AWS, the cost of 10 S3 PUTs/s is equal to the cost of 1.28 MB/s of produce throughput with a replication factor of 3. 💡 The number of PUTs/s depends massively on your configuration, and it heavily influences your expected latency. The end to end latency of these direct-to-S3 system is roughly: • the batch interval • one S3 PUT latency • one S3 GET latency • the centralized metadata store latency (e.g DynamoDB, Spanner) If you increase the batch interval, you reduce your PUT calls but increase your latency. And vice-versa. My tool shows you the expected e2e latency per batch interval config. It then shows you the break even producer throughput, after which it becomes financially worth it to deploy the new model. 🤑 Note that it doesn't take into account the vendor's costs. Today, every direct-to-S3 solution is proprietary and requires some sort of license. 🔒 From what I can tell, Bufstream is by far the cheapest solution so far. But that's neither here nor there. 🤷‍♂️ The takeaway is that this new model is INCREDIBLY cost-effective. If your workload can relax its latency requirements ever-so-slightly, you're eligible to reduce your cost MASSIVELY. Check the tool out now and let me know what you think: ✅ https://lnkd.in/dFzt9ev6

67

15 Comments

Stuck keeping #Spring apps alive on #EC2 (or herding #Kubernetes) just to ship a few endpoints? ⚙️ What if #Java on #AWS #Lambda could meet your latency targets and cut the ops tax too? ⏱️💸 Good news! Serverless can be your escape hatch... Yes, Java runs great on AWS Lambda. You can hit realistic p95s, keep costs sensible, and still have a genuinely pleasant developer experience. ☕️ In this fresh episode of AWS Bites, I sit down with Mark Sailes (ex-AWS, Java-on-Lambda expert). We get practical: when Java is the right call for Lambda and when it isn’t, how SnapStart and provisioned concurrency actually help, which frameworks behave at startup (vanilla, #Micronaut, #Quarkus, Spring Boot), and how to test without drowning in brittle mocks (#Testcontainers, #LocalStack). 🧪🔧 If your stack feels like a cage, this episode points to the exits. 🧭🚪 🎧 Watch/listen: https://awsbites.com/148 Time to migrate your Java services to Lambda, let's get to work! 🚀 PS: Hope you enjoy the slightly hallucinated cover... no substances involved, unless you count the AI’s triple espresso. ☕️🤖 #Serverless #AWSBites #SnapStart #ProvisionedConcurrency #GraalVM #CloudEngineering

39

7 Comments

𝑻𝑶𝑷 10 𝑪𝑳𝑶𝑼𝑫 𝑨𝑼𝑻𝑶𝑴𝑨𝑻𝑰𝑶𝑵 𝑻𝑶𝑶𝑳𝑺 - AWS CloudFormation: Utilizes JSON/YAML templates to define and manage AWS infrastructure. - Terraform: A declarative Infrastructure as Code (IaC) tool for managing and provisioning resources across various cloud providers. - Ansible: Streamlines automation tasks using an easy-to-understand, agentless language. - Chef: A Ruby-based IaC tool for automating system configuration and management. - Azure Automation: Automates processes within Microsoft Azure using PowerShell scripts. - Jenkins: A popular open-source automation server for continuous integration and continuous delivery (CI/CD). - SaltStack: Provides configuration management with a master-minion architecture for scalable deployments. - Puppet: Uses a declarative language for automating and managing configuration at scale. - Google Cloud Deployment Manager: Allows resource management on Google Cloud using YAML/Python templates. - VMware vRealize Automation: Automates the provisioning of virtual resources in VMware environments. I help technical professionals build impactful career brands on LinkedIn. 👉 { https://lnkd.in/g7Gp68cV }

139

21 Comments