MetaFFI -- Multilingual Indirect Interoperability System

Tsvi Cherny-Shahar,Amiram Yehudai

2024-08-26

Abstract:The development of software applications using multiple programming languages has increased in recent years, as it allows the selection of the most suitable language and runtime for each component of the system and the integration of third-party libraries. However, this practice involves complexity and error proneness, due to the absence of an adequate system for the interoperability of multiple programming languages. Developers are compelled to resort to workarounds, such as library reimplementation or language-specific wrappers, which are often dependent on C as the common denominator for interoperability. These challenges render the use of multiple programming languages a burdensome and demanding task that necessitates highly skilled developers for implementation, debugging, and maintenance, and raise doubts about the benefits of interoperability. To overcome these challenges, we propose MetaFFI, a pluggable in-process indirect-interoperability system that allows the loading and utilization of entities from multiple programming languages. This is achieved by exploiting the less restrictive shallow binding mechanisms (e.g., Foreign Function Interface) to offer deep binding features (e.g., object creation, methods, fields). MetaFFI provides a runtime-independent framework to load and \emph{xcall} (Cross-Call) foreign entities (e.g., functions, objects). MetaFFI uses Common Data Types (CDTs) to pass parameters and return values, including objects and complex types, and even cross-language callbacks. The indirect interoperability approach of MetaFFI has the significant advantage of requiring only $2n$ mechanisms to support $n$ languages, as opposed to the direct interoperability approaches that need $n^2$ mechanisms. We have successfully tested the binding between Go, Python3.11, and Java in a proof-of-concept on Windows and Ubuntu.

Programming Languages

What problem does this paper attempt to address?

This paper attempts to solve the problem of multi - programming - language interoperability. As the demand for using multiple programming languages in software application development increases, developers can choose the languages and runtime environments that are most suitable for each system component and integrate third - party libraries. However, this practice brings complexity and error - proneness because of the lack of an effective multi - programming - language interoperability system. Current solutions usually rely on C as a common interoperability benchmark and require re - implementing libraries or writing language - specific wrappers. These methods often rely on highly - skilled developers for implementation, debugging and maintenance.

To solve these problems, the author proposes MetaFFI, which is a pluggable in - process indirect interoperability system that allows loading and using entities from multiple programming languages. The main contributions of MetaFFI include:

1. **Indirect Interoperability**: Provide deep - binding functions (such as object creation, method and field access) by using less - restrictive shallow - binding mechanisms (such as the External Function Interface, FFI). This enables MetaFFI to achieve cross - language calls without directly connecting languages.
2. **Reduce the Number of Interoperability Mechanisms**: Traditional direct interoperability methods require \(O(n^2)\) interoperability mechanisms to support \(n\) languages, while MetaFFI only requires \(O(2n)\) mechanisms.
3. **Modular Plug - in Design**: Developers can extend support for new programming languages by implementing at most three C interfaces, and the complexity of adding a new language does not depend on the number of already - supported languages.
4. **Unified API**: MetaFFI provides a unified set of APIs, allowing modules in different languages to be loaded and called in a similar way, simplifying the developer's experience.

Through these features, MetaFFI aims to reduce the complexity and burden of multi - language development, enabling developers to more easily integrate and reuse code written in different languages in a single application.

### Summary of the Core Problems in the Paper
- **Problem Background**: The demand for multi - programming - language development is increasing, but existing interoperability solutions are complex and error - prone.
- **Main Challenges**: Existing solutions rely on C as an intermediary and require a great deal of manual work and highly - skilled developers.
- **Solution**: Propose MetaFFI to reduce the number of mechanisms through indirect interoperability and simplify multi - language integration.

### Formula Explanation
The formulas mentioned in the paper are expressed as follows:
- Direct interoperability methods require \(O(n^2)\) mechanisms.
- MetaFFI only requires \(O(2n)\) mechanisms.

These formulas indicate that as the number of supported languages increases, the number of MetaFFI's interoperability mechanisms grows linearly rather than quadratically, significantly reducing the number of required interoperability mechanisms.