DevOps and Backend Development Services for Swedish E-Commerce SaaS
- Company name: Leeroy Group AB
- Company size: 50
- Services provided: Staff Augmentation (Dedicated Teams & Extensions)
- Duration of project: 12 Months
- Engineers (FTE): 8
- Technology AWS, Kubernetes, Java, Typescript, Microservice
Customer Profile
Leeroy Group is a software development company based in Stockholm, Sweden that provides solutions like online food ordering , mobile apps, business analytics, restaurant management systems, integrations with external vendors (like Uber Eats) and custom devices for collecting orders in restaurants (Points of Sale). Clients of Leeroy Group are mainly large restaurant chains based in the Nordics.
Leeroy provided an all in-one software and hardware solutions for restaurant chains to become fully digitalized and process food online and offline. They had an established customer base and more than 5 years of experience on the market, with approximately 50 employees based in Sweden.
Technologies
- Cloud : AWS (RDS, EKS, ECS, EC2, Elasticache, Elasticsearch, SNS, SQS, MQ)
- Platform : Containers (Docker ECS + Kubernetes) and Serverless (Lambda)
- Backend: Java (SpringBoot) + NodeJS (Typescript)
- Frontend : React + React Native (PWA)
- CI/CD : CircleCi, Github Actions, ArgoCD
- Architecture: Monolith + Microservices (approx 50)
- Observability : Datadog, Prometheus, Grafana, AWS
- Automation : Terraform, Helm
Project & Assignment
At the end of 2021 Leeroy got in touch with Relout and requested a typical team-extension service, to backfilling stepping down DevOps engineers and to increase competence level in an already established platform team. We started with providing 2 talented Senior DevOps engineers who’s primary role was to support and accelerate an ongoing Kubernetes migration and reduce AWS billing and costs. Over time the scope of assignment has changed significantly due to adapting to business realities (increasing inflation levels due to War in Ukraine and its effects on restaurants) but also thanks to a very promising and fruitful cooperation between Leeroy and Relout. Over time the staff from Relout has been increased to 4 DevOps/SRE Engineers including promotion of one of members to team leader and a total of 4 Backend (Java) Engineers including team leaders and managers that were primarily focused on redesigning the monolithical core of the app and developing new microservices for the global platform (called Leeroy Cloud).
During H2 of 2022 Relout was supporting Leeroy in 2 big rounds of hiring and outstaffing for Backend Engineers roles which resulted in recruiting and verifying nearly 90 candidates out of which 20 were presented and 11 were approved by the Client
Over time Relout has been adjusting to the requirements and requests from Leeroy to backfill their staff and increase competence levels of the teams while ensuring flexibility and performance levels of entire cooperation. At the end of 2022 Relout became a strategic technology partner for Leeroy not only due to number of staff and also a wide range of services provided (like organizing meetups in Poland) and general trust and solid partnership between Relout and Leeroy’s executives.
Challenges
- Increasing and out-of-control cost levels of AWS infrastructure and tools without strong coorelation with new clients and business growth
- Lack of visibility and observability within the system and product – no E2E monitoring, no SLO’s defined and good business dashboards and escalations
- Lack of reliability – no change or incident management in place, no SLO’s defined, no clear E2E flow monitored and defined
- Performance issues and ongoing incidents – due to monolithical design of the app, legacy code and technical debts and unmaintained areas of the system and its underlying infrastructure
- Poor technical knowledge and understading of the infrastructure in in-house teams
- Legacy code and infrastructure with plenty of technical debt, started but unfinished initiatives and no Infrastructure as Code defined
Goals
- Migrate from legacy container infrastructure to AWS EKS (Kubernetes) for greater flexibility and insights into the application
- Optimize AWS Infrastructure Costs and stabilize the trend of cost increasing
- Optimize tooling costs (mainly Datadog and CircleCI)
- Maintain the old (legacy) monolithical core of the application and develop new microservices
- Migrate from Circle CI to Github Actions
- Train on-site teams on SRE principles and using and implementing Kubernetes and AWS Cloud Native approach
Our Approach
From beginning we ensured that all the work we’re providing as Relout is properly tracked and documented, which, apparently, was not the common standard at Leeroy back in 2021. Our teams also delivered and implemented good practices including
- Ensure Blameless culture – during many incidents caused by legacy code, communication issues within the teams, monolithical core of the app as well as lack of observability and reliability we showed how to deal with incidents, how to run postmortems and resolve issues effectively and proactively
- Apply DevOps Mindset – through trainings, automation, documentation and demos we enabled multiple on-site engineers to start interacting with infrastructure more and more, helping them understand the underlying principles and resolve the issues together
- Introduce Change Management – this helped to ensure the changes done on production and well tracked and can be rolled back with ease in case of unsuccessfull deployment
- Introduce Incident Management – due to multiple issues with application this was necessary to address the customer’s needs to keep the system stable and be proactive about resolving and communicating the incidents
- Perform AWS Infrastructure Review and Cleanup – this allowed us to identify forgotten and unneded resources, rightscale the infratructure, plan the roadmap and ultimatelly cut AWS spend by 45% from original levels of 27 000 USD to 15 000 USD (per month)
Result and Outcome
After 12 months of cooperation and becoming a strategic technology partner for Leeroy we significantly modernized the platform, limited amount of legacy code and technical depts, cut the AWS and tooling costs by 45% and increased the level of competences of Leeroy engineers significantly by applying DevOps Mindset and SRE principles. Many business challenges were still to be faced and new technologies to be implemented. In 2023 the platform and core development was officially moved to new owner of Leeroy – Caspeco who aquired the Leeroy Group AB company in february 2023 and officially took over the product and platform development.
Services provided
Team Extension
3 FTE Engineers that joined and transitioned between existing teams responsible for maintaining and developing core product functionality
Dedicated Teams
2 dedicated teams - Evolution and Platform with 6 DevOps/SRE and Backend FTE Engineers involved
Consulting & Advisory
We ensured process and procedure improvements are made, applied Infrastructure Audit & Reviews, provided recruitment and HR support and organized off-site company meetup
Achievements
- Save AWS Billing Costs by 45% (from 27 000 USD to 15 000 USD per month)
- Reduce Datadog costs by 70% (from 5500 to 1500 USD)
- Implement DevOps Culture and SRE Practices among in-house teams and engineers
- Extend and redefine Platform & Evolution teams managing core parts of the product
- Organize company-wide on-site event in Poland for engineering and C-Level execs
- Improve security through SSO, minimum-access policy, access review and cleanup, environment & priviledge separation and governance
- Migrate core microservices to Kubernetes
- Migrate CICD to Github Actions & ArgoCD
- Implement Change Management & Incident Management