Juan A. Rodriguez

ServiceNow Research

Mila, Quebec AI Institute

École de Technologie Superieure, University of Quebec

Montreal, Quebec, Canada

Hi, I am Juan Rodriguez, you can also call me Joan (in Catalan).

I am a Researcher at ServiceNow Research and a PhD student at Mila and École de Technologie Superieure, University of Quebec. I am based in Montreal, Canada, but I am from Barcelona, Spain. I am advised by Prof. Marco Pedersoli, Prof. Chris Pal, and Dr. David Vazquez

My research interests are in the intersection of Computer Vision and Natural Language Processing, with a focus on multimodal generative models. I am interested in learning how to leverage information from different modalities to generate more accurate and controlable outputs in all modalities. Recently I have been working with large language and vision models using transformers and diffusion. I have also been exploring the paradigm of generating code as an alternative for generating images (e.g. scalable vector graphics).

I obtained a M.Sc. in Computer Vision from Universitat Autònoma de Barcelona (UAB) with honors for my master thesis Text to Scientific Figure Generation performed at ServiceNow Research and advised by Dr. David Vazquez and Dr. Pau Rodríguez. Previously, I obtained a B.Sc. in Telecommunication Networks Engineering from Universitat Pompeu Fabra (UPF) and carried out my bachelor thesis on Handwritten Text Recognition advised by Prof. Xavier Binefa. I did research internships at UPF advised by Prof. Xavier Binefa and Prof. Miquel Oliver, and at CVC-UAB advised by Prof. Joost van de Weijer.

Publications

OCR-VQGAN: Taming text-within-image generation

Juan A. Rodriguez, David Vazquez, Issam Laradji, Marco Pedersoli, and Pau Rodriguez

In WACV 2023 (Oral), 2023

Abs arXiv Code

Synthetic image generation has recently experienced significant improvements in domains such as natural image or art generation. However, the problem of figure and diagram generation remains unexplored. A challenging aspect of generating figures and diagrams is effectively rendering readable texts within the images. To alleviate this problem, we present OCR-VQGAN, an image encoder, and decoder that leverages OCR pre-trained features to optimize a text perceptual loss, encouraging the architecture to preserve high-fidelity text and diagram structure. To explore our approach, we introduce the Paper2Fig100k dataset, with over 100k images of figures and texts from research papers. The figures show architecture diagrams and methodologies of articles available at arXiv.org from fields like artificial intelligence and computer vision. Figures usually include text and discrete objects, e.g., boxes in a diagram, with lines and arrows that connect them. We demonstrate the effectiveness of OCR-VQGAN by conducting several experiments on the task of figure reconstruction. Additionally, we explore the qualitative and quantitative impact of weighting different perceptual metrics in the overall loss function. We release code, models, and dataset at this https URL.
FigGen: Text to Scientific Figure Generation

Juan A. Rodriguez, David Vazquez, Issam Laradji, Marco Pedersoli, and Pau Rodriguez

In ICLR 2023 (Tiny paper track), 2023

Abs arXiv Code

The generative modeling landscape has experienced tremendous growth in recent years, particularly in generating natural images and art. Recent techniques have shown impressive potential in creating complex visual compositions while delivering impressive realism and quality. However, state-of-the-art methods have been focusing on the narrow domain of natural images, while other distributions remain unexplored. In this paper, we introduce the problem of text-to-figure generation, that is creating scientific figures of papers from text descriptions. We present FigGen, a diffusion-based approach for text-to-figure as well as the main challenges of the proposed task. Code and models are available at this https URL
Affective State-Based Framework for e-Learning Systems

Juan A. Rodriguez, Joaquim Comas, and Xavier Binefa

In CCIA 2021 (Oral), 2021

PDF
StarVector: Generating Scalable Vector Graphics Code from Images

Juan A. Rodriguez, Shubham Agarwal, Issam H. Laradji, Pau Rodriguez, David Vazquez, Christopher Pal, and Marco Pedersoli

In preprint, 2023

Abs arXiv Code

Scalable Vector Graphics (SVGs) have become integral in modern image rendering applications due to their infinite scalability in resolution, versatile usability, and editing capabilities. SVGs are particularly popular in the fields of web development and graphic design. Existing approaches for SVG modeling using deep learning often struggle with generating complex SVGs and are restricted to simpler ones that require extensive processing and simplification. This paper introduces StarVector, a multimodal SVG generation model that effectively integrates Code Generation Large Language Models (CodeLLMs) and vision models. Our approach utilizes a CLIP image encoder to extract visual representations from pixel-based images, which are then transformed into visual tokens via an adapter module. These visual tokens are pre-pended to the SVG token embeddings, and the sequence is modeled by the StarCoder model using next-token prediction, effectively learning to align the visual and code tokens. This enables StarVector to generate unrestricted SVGs that accurately represent pixel images. To evaluate StarVector’s performance, we present SVG-Bench, a comprehensive benchmark for evaluating SVG methods across multiple datasets and relevant metrics. Within this benchmark, we introduce novel datasets including SVG-Stack, a large-scale dataset of real-world SVG examples, and use it to pre-train StarVector as a large foundation model for SVGs. Our results demonstrate significant enhancements in visual quality and complexity handling over current methods, marking a notable advancement in SVG generation technology