Abstract:In the field of software engineering, the environmental dependency conflict is a significant problem facing software engineers. Containerization (Pahl 2015 ) was proposed to resolve environmental dependency conflicts, Currently widely used in cloud computing and distributed systems. Simultaneously, in the process of large-scale application development and deployment, the microservice (Nadareishvili et al. 2016 ) architecture has the advantages of robust scalability and low coupling. Therefore, it is becoming increasingly favored by software developers. For example, Google is one of the few companies that need to manage the deployment and development of a large number of service components on hundreds of thousands of servers. With the concept of containerization and microservices at the core, an open-source distributed container management system called Kubernetes was developed. Kubernetes can not only maintain complete independence of applications but also improve the utilization of hardware resources, so it is affected by Internet companies and widely used by many institutions. In recent years, the demand for computing resources for machine learning-related applications is increasing, and the stand-alone computing for machine learning tasks is often unsustainable. Many data scientists will rely on distributed systems to provide sufficient computing resources for machine learning tasks. Kubernetes can be used for machine learning related applications and provides support for deployment on distributed systems. Meanwhile, it has many excellent features such as containerization and microservices. Therefore, the idea of developing and deploying machine-learning applications based on Kubernetes is favored by data scientists. Google has developed Kubeflow, a machine learning tool suite based on the Kubernetes system. Kubeflow can help data scientists run machine learning workloads on distributed clusters. For historical reasons, Kubeflow’s support for Tensorflow is quite complete, but the support of the framework is not perfect for Pytorch. Besides, although the pipeline component included in Kubeflow can build an entire machine learning workflow, this component is still dependent on the Google Cloud Platform. Therefore, Kubeflow pipeline is not friendly enough for developers who have no conditions to lease Google Cloud Service. This paper designs a complete solution to the problem of Kubeflow pipeline, and verifies the feasibility of the solution through an example, so that Kubernetes no longer depends on Google Cloud Service and better supports machine learning applications based on Pytorch. This solution ensures that data scientists can accomplish Pytorch-based deep learning applications on Kubernetes-based distributed systems development, demonstration, construction workflow, deployment services, operation, maintenance, and other functions. In principle, any data scientist who wants to use Pytorch to develop a machine-learning project can deploy their machine “learning applications on distributed systems by following such a set of solutions. The proposed approach makes the application stably and continuously running and reasonably scheduled on the cluster containing heterogeneous computing resources.

APPLICATION OF THE KUBEFLOW TOOL FOR THE INTEGRATION OF MACHINE LEARNING AND ARTIFICIAL INTELLIGENCE IN UNMANNED AERIAL VEHICLE

Deployment and Verification of Machine Learning Tool-Chain Based on Kubernetes Distributed Clusters

MLOps: Automatic, Zero-Touch and Reusable Machine Learning Training and Serving Pipelines

Machine Learning Operations: A Mapping Study

KModels: Unlocking AI for Business Applications

Machine Learning-Aided Operations and Communications of Unmanned Aerial Vehicles: A Contemporary Survey

KFIML: Kubernetes-Based Fog Computing IoT Platform for Online Machine Learning

An Efficient Artificial Intelligence Framework for UAV Systems

Automating the Training and Deployment of Models in MLOps by Integrating Systems with Machine Learning

FlockAI: A Testing Suite for ML-Driven Drone Applications

Human-in-the-loop extension to stream classification for labeling of low altitude drone imagery

Machine Learning Based Fluid-Transportation Monitoring and Controlling

On Continuous Integration / Continuous Delivery for Automated Deployment of Machine Learning Models using MLOps

MLOps: A Step Forward to Enterprise Machine Learning

STREAMLINE: A Simple, Transparent, End-To-End Automated Machine Learning Pipeline Facilitating Data Analysis and Algorithm Comparison

An Efficient UAV-based Artificial Intelligence Framework for Real-Time Visual Tasks

Machine Learning Applications in Internet-of-Drones: Systematic Review, Recent Deployments, and Open Issues

Continual learning on deployment pipelines for Machine Learning Systems

Accelerating the Machine Learning Model Deployment using MLOps

Decentralized Federated Learning for UAV Networks: Architecture, Challenges, and Opportunities

Blockchain-enabled Clustered and Scalable Federated Learning (BCS-FL) Framework in UAV Networks