Jiang Liu

I am an Applied Research Scientist in the Generative AI team at AMD. I received my Ph.D degree from Department of Electrical and Computer Engineering, Johns Hopkins University in 2024, advised by Prof. Rama Chellappa. I was awarded the Amazon AI2AI Fellowship in 2023. I received my BSE degree from Department of Automation, Tsinghua University in 2019 advised by Prof. Jianjiang Feng and Prof. Jie Zhou.

In summer 2023, I interned at Microsoft Azure AI with Dr. Jianfeng Wang working on multi-modal large language models. In summer 2022, I worked as an Applied Scientist Intern at Amazon AWS AI working on vision-language models mentored by Dr. Hui Ding, Dr. Zhaowei Cai, and Dr. Yuting Zhang. I've also worked as a Deep Learning Research Scientist Intern at Subtle Medical developing novel Transformer-based magnetic resonance imaging (MRI) algorithms.

Email  /  Google Scholar  /  Github

My current research interests include large language models, vision-language models, and trustworthy AI.

We are hiring research interns in all areas of generative AI! Feel free to drop me an email with your CV if interested.

We propose Instruct2Attack (I2A), a language-guided semantic attack that generates semantically meaningful perturbations according to free-form language instructions. We show that I2A can successfully break state-of-the-art deep neural networks even under strong adversarial defenses, and demonstrate great transferability among a variety of network architectures.

We propose DiffProtect, which utilizes a diffusion autoencoder to generate semantically meaningful perturbations on FR systems. Extensive experiments demonstrate that DiffProtect produces more natural-looking encrypted images than state-of-the-art methods while achieving significantly higher attack success rates.

We propose a novel threat model called Joint Space Threat Model (JSTM), which exploit the underlying manifold information with Normalizing Flow, ensuring that exact manifold assumption holds. Under JSTM, we develop novel adversarial attacks and defenses. Furthermore, we propose the Robust Mixup strategy in which we maximize the adversity of the interpolated images and gain robustness and prevent overfitting.

In this paper, we formulate missing data imputation as a sequence-to-sequence learning problem and propose a multi-contrast multi-scale Transformer (MMT), which can take any subset of input contrasts and synthesize those that are missing. It can efficiently capture intra- and inter-contrast dependencies for accurate image synthesis. Moreover, MMT is inherently interpretable. It allows us to understand the importance of each input contrast in different regions by analyzing the in-built attention maps of MMT decoder.

In this work, instead of directly predicting the pixel-level segmentation masks, the problem of referring image segmentation is formulated as sequential polygon generation, and the predicted polygons can be later converted into segmentation masks. This is enabled by a new sequence-to-sequence framework, Polygon Transformer (PolyFormer), which takes a sequence of image patches and text query tokens as input, and outputs a sequence of polygon vertices autoregressively.

In this paper, we propose Segment and Complete defense (SAC), a general framework for defending object detectors against patch attacks through detection and removal of adversarial patches. SAC achieves superior robustness even under strong adaptive attacks with no reduction in performance on clean images, and generalizes well to unseen patch shapes, attack budgets, and unseen attack methods.

In this paper, we propose mutual adversarial training (MAT), in which multiple models are trained together and share the knowledge of adversarial examples to achieve improved robustness. MAT allows robust models to explore a larger space of adversarial samples, and find more robust feature spaces and decision boundaries. We show that MAT can improve model robustness for both single and multiple perturbations.

Source code credit to Dr. Jon Barron.