Derandomization with Pseudorandomness

Emin Karayel
2024-04-25
Abstract:Derandomization techniques are often used within advanced randomized algorithms. In particular, pseudorandom objects, such as hash families and expander graphs, are key components of such algorithms, but their verification presents a challenge. This work shows how such algorithms can be expressed and verified in Isabelle and presents a pseudorandom objects library that abstracts away the involved deep algebraic/analytic results. Moreover, it presents examples that show how the library eases and enables the verification of advanced randomized algorithms. Highlighting the value of this framework is that it was recently used to verify the optimal-space distinct elements algorithm by Blasiok from 2018, which relies on the combination of many derandomization techniques to achieve its optimality.
Logic in Computer Science
What problem does this paper attempt to address?
The problem that this paper attempts to solve is: **How to express and verify derandomization techniques in advanced random algorithms in the formal verification framework Isabelle, especially the use of pseudo - random objects (such as k - independent hash function families and expander graphs)**. ### Specific problem decomposition 1. **Expression and verification of pseudo - random objects**: - Pseudo - random objects (such as k - independent hash function families and expander graphs) are key components of many advanced random algorithms. However, their verification is challenging. - The paper shows how to express and verify these pseudo - random objects in Isabelle and provides a pseudo - random object library, which abstracts complex algebraic and analytical results. 2. **Optimizing space complexity**: - By using pseudo - random objects, the number of random bits required by the algorithm can be reduced, thereby optimizing the space complexity. - Taking a simple random stream algorithm as an example, the paper shows how to reduce the space complexity from \(O(n+\log m)\) to \(O(\log n+\log m)\) through pseudo - random objects. 3. **Specific applications of formal verification**: - The paper introduces how to use the pseudo - random object library to verify the Distinct Elements Algorithm with optimal space, which depends on multiple derandomization techniques. - The application of these techniques is not limited to the distinct elements problem, but also includes other fields such as Heavy Hitters, Frequency Moments, Graph Matching, etc. ### Main contributions 1. **Construction method of pseudo - random objects**: - A new method for constructing pseudo - random objects based on simple primitives is proposed, and a concise notation is introduced, making these objects easy to use in algorithms. 2. **Formal verification of non - primitive finite field calculation library**: - A verifiable executable library for non - primitive finite field (such as \(GF(p^n)\)) calculations is verified, which is the basis for constructing k - independent hash function families. 3. **Formal verification of expander graphs**: - The expander graphs and their construction methods are verified, and the tail bound of random walks is proved, which is the basis for constructing expander graph pseudo - random objects. 4. **Combined application of multiple pseudo - random objects**: - It is shown how to combine multiple types of pseudo - random objects (such as expander graphs and k - independent hash function families) to improve the previous simple examples and further optimize algorithm performance. In summary, this paper aims to solve how to effectively express, verify and apply pseudo - random objects in the formal verification tool Isabelle, thereby optimizing the space complexity and other performance indicators of random algorithms.