The energy business of the future is green, decentralized, international and highly flexible. The heat, electricity and mobility sectors will be inseparably linked. With KiwiOS, we have developed a software platform that enables companies to integrate, monitor and optimize the multitude of plants, systems and partners of the decentralized energy world.
Flexibility and modularity are KiwiOS' greatest strengths, but also pose the greatest technological challenges. Customers either want to develop applications on the platform independently or use Kiwigrid's turnkey software products. Therefore, data and functions (called services) must be available via standardized interfaces depending on the use case and customer requirements – at Kiwigrid, we call this Platform-as-a-Service.
The requirements for platform-based services, such as the provision of data, KPIs, analysis, time series, forecasts or optimization functions, are diverse. The services must be flexible, scalable, easy to maintain and secure. In addition, their delivery must be near real-time and reasonably priced. Leaving behind our hybrid data storage and migrating all our services and data models to the Google Cloud means we are now a big step closer to this.
What have we achieved?
Enhanced scalability
To optimize the flow of energy for energy management systems (EMS), the KiwiOS platform collects data such as the outgoing power or voltage in each grid phase or the state of charge of stationary storage devices. As each EMS is connected to multiple physical devices – including electricity meters, PV systems and heat pumps – a powerful system for data management and analysis is required. Based on tens of thousands of data points, EMS provide valuable insights for the best possible control of the energy system, including local self-consumption optimization and the dynamic regulation of an inverter.
Energy management is highly complex. It encompasses sector coupling, device connectivity, control and physics. If the underlying cloud architecture is also complex, it makes scalability difficult. Switching from a hybrid system to the Google Cloud reduces IT complexity. It is therefore the best option for international scalability and directly serves our customers’ interests.
Highest security
Power grids form part of our critical infrastructure. A secure power supply is the basis of our everyday lives. As EMS are part of the energy system, security is one of the most important requirements for the cloud infrastructure they operate on. The Google Cloud delivers the highest security standards. Its services conform to the C5 standard issued by the German Federal Office for Information Security (BSI). This guarantees that highly secure and sensitive information such as meter reading data from a smart meter gateway is stored and processed in compliance with the law. Our customers can be confident that their data is stored securely and that it is transferred and processed anonymously.
Minimal downtime and improved stability
Kiwigrid's microservice architecture and data storage are based on Google Cloud technologies such as Dataproc, Kubernetes and CloudSQL. These guarantee high service-level agreements (SLAs), reducing unplanned downtime to almost zero. The availability of our platform is further increased.
Accelerated time to market
With the help of Google technologies, Kiwigrid is able to bring new energy products to market much more flexibly and quickly. We are also enabling customers and clients to access apps and services much faster. The barrier to developing mobile apps, creating data analytics and applying the latest artificial intelligence solutions is significantly reduced.
How did we approach the challenge?
The following figures show that a migration of this level is extremely demanding and time-consuming. With a preparation time of six months and subsequent planned downtime of only two hours, 40 terabytes (TB) of data were migrated, 20 platform services migrated and 915 Jira issues resolved. The overall system consists of 770 running Kubernetes pods, 55 nodes, 480 CPU cores and 1 TB of RAM. It handles approximately 5,000 API requests per second. Development, staging and production environments were updated, while the backup concept was improved and extended.
Challenges
From the beginning, we were convinced by the advantages of the Google Cloud and its services. At the same time, the migration was necessary because the existing Hortonworks Data Platform (HDP) as the basis of the Apache Hadoop installation was discontinued due to the acquisition of Cloudera. Therefore, the biggest challenge was to migrate and transfer data from the Hbase database, which backs up all data points from the connected energy devices. Dataproc was used as the target technology in the Google Cloud Platform.
40 terabytes of data transferred in two hours
As a first approach to data transfer, we chose multiple snapshots or restore cycles, where each subsequent run is faster than the previous run. This was meant to reduce the amount of written data so much that the rest could be transferred within two hours of downtime. Unfortunately, this was not successful, as contrary to our expectation the algorithm turned out to be unreliable. It was also not possible to increase the transmission bandwidth within the specified time.
David Rabe, Director Platform at Kiwigrid, recalls:
"The final breakthrough came with a method based on several functions and tools available in the HBase ecosystem. This involved transferring a single snapshot of the database to Google-Dataproc over an extended period of time and then using Hbase cluster replication to complete it afterwards via a VPN connection to the on-premise machines."
Data transfer was also a challenge when transferring MySQL data, where the capacity of associated CPUs and load balancers reached its limits. Deploying a Google VPN failed due to the unworkable IPSec implementation on-premise at the previous server center. The transfer of 750 gigabytes was eventually successful thanks to the use of VPN and an extended network routing.
We continue to implement improvements
Since the successful migration we focus on minor additions regarding the scalability of services and traffic shaping. A hybrid system requires concepts that need an increased employment of load balancers and the use of the public cloud. These implementations can now be optimized. Additionally, we look at adjustments to the cloud DNS for Google Kubernetes, the removal of public IPs, extensions to monitoring as well as improvements to auto-scaling based on specific metrics.
The team behind this success
The migration of all services to the Google Cloud was a substantial undertaking that we could only accomplish thanks to the excellent work of an outstanding team. We would like to thank our team at Kiwigrid, including David Rabe, Christian Geie, Christian Buder, Christian Laußat, Tomasz Rarok, Manuel Hoffmann, Jörg Eichhorn, Robert Bach, Holger Mauermann, Erdi Papavangjeli, Steffen Rietz and Norman Leyda, our colleagues at Jaconi, especially Jan Krüger, Jan Fiedler, Gerrit Schlüter and Julian Nodorp, as well as the two Google colleagues Daniel Klessing and Sascha Schnorbach.
If you enjoyed reading this report on the migration to the Google Cloud and you’d like to find out more about working at Kiwigrid, take a look at our career page and join our team! Also check out Google's case study.