Abstract:Pre-training model on large-scale unlabeled web videos followed by task-specific fine-tuning is a canonical approach to learning video and language representations. However, the accompanying Automatic Speech Recognition (ASR) transcripts in these videos are directly transcribed from audio, which may be inconsistent with visual information and would impair the language modeling ability of the model. Meanwhile, previous V-L models fuse visual and language modality features using single- or dual-stream architectures, which are not suitable for the current situation. Besides, traditional V-L research focuses mainly on the interaction between vision and language modalities and leaves the modeling of relationships within modalities untouched. To address these issues andmaintain a smallmanual labor cost, we add automatically extracted dense captions as a supplementary text and propose a new trilinear video-language interaction framework TEVL (Trilinear Encoder for Video-Language representation learning). TEVL contains three unimodal encoders, a TRIlinear encOder (TRIO) block, and a temporal Transformer. TRIO is specially designed to support effective text-vision-text interaction, which encourages inter-modal cooperation while maintaining intra-modal dependencies. We pre-train TEVL on the HowTo100M and TV datasets with four task objectives. Experimental results demonstrate that TEVL can learn powerful video-text representation and achieve competitive performance on three downstream tasks, including multimodal video captioning, video Question Answering (QA), as well as video and language inference. Implementation code is available at https://github.com/Gufrannn/TEVL.

Perceptron Implementation of Triple-Valued Logic Operations

The Realization Of Parallel Subtraction Using Lclv In Optical Neural Network

Ten-Valued Multiplication Based on 2-5 Mixed-Valued Coding

A SYMMETRIC TERNARY LOGIC BASED MULTI-VALUED LEARNING NETWORK

Realization of Neural Networks for Basic Operation on Multi-Valued Logic

Threshold Logic in a Flash

Multi-thresholded neuron and it's application in multi-valued logic

Perceptron Training Algorithms Designed Using Discrete-Time Control Liapunov Functions

Spiking Perceptrons

On the Equivalence of the Weighted Tsetlin Machine and the Perceptron

Training DNA Perceptrons via Fractional Coding

Perceptron Linear Function Design with CMOS-Memristive Circuits

Neuromorphic Implementation of Logic Functions Based on Parallel Dual-Gate Thin-Film Transistors

Multi-valued Logic Design Methodology with Double Negative Differential Resistance Transistors

Reconfigurable electro-optical logic gates using a 2-layer multilayer perceptron

TEVL: Trilinear Encoder for Video-language Representation Learning

A Mixed Parallel Perceptron Classifier and Several Application Problems

Expanding the Set of Three-Input Logic Functions in Inverted T-Shaped TFETs

Gray-Tone Image-Processing by Threshold Superposition in an Optical Cellular-Logic Binary-Image Processor

Implementing An Artificial Quantum Perceptron

Optical implementation of programmable binary logic processor by using one electronic trapping device