Muhammad Jehanzeb Mirza

MIT, USA.

CV | Linkedin | Google Scholar | Github | Email

Hi, I am Jehanzeb Mirza. I am a Post doctoral Researcher at MIT CSAIL, in the Spoken Langugage Systems Group, led by Dr. James Glass. I received my PhD. in Computer Science (Computer Vision) from TU Graz, Austria, where I was advised by Professor Horst Bischof, and Professor Serge Belongie served as an external referee. I did my Masters from KIT, Germany in Electrical Engineering and Information Technology (ETIT) and received my Bachelors in Electrical Engineering (EE) from NUST, Pakistan.

My PhD. research mainly focused on learning from unlabeled data. During my Phd., I have worked with various data modalities including images, point clouds, videos, radar signals and most recently natural language. I am particularly interested in Self-Supervised learning for uni-modal models and also Multi-Modal Learning for vision-language models.

Contact

jmirza [at] mit.edu
Boston, USA.

Education

Ph.D. in Computer Vision (2021 - 2024)

TU Graz, Austria.

MS in ETIT (2017 - 2020)

KIT, Germany.

BS in EE (2013 - 2017)

NUST, Pakistan

Recent News

09/24: 1 paper accepted at NeurIPS, 2024.

07/24: 1 paper accepted at BMVC, 2024.

07/24: 2 paper accepted at ECCV, 2024.

04/24: I successfully defended my Ph.D. thesis.

12/23: Our workshop "What's Next in Multi-Modal Foundation Models" got accepted at CVPR 2024.

10/23: Invited talk at Cohere.

10/23: Invited talk at VIS Lab, University of Amsterdam.

9/23: 1 paper accepted at NeurIPS, 2023.

9/23: Invited talk at Center for Robotics, Paris Tech.

7/23: 1 paper accepted at ICCV, 2023.

4/23: I will be attending International Computer Vision Summer School.

3/23: 2 papers accepted at CVPR, 2023.

2/23: Reviewing for CVPR, ICCV and TPAMI.

3/22: 2 papers accepted at CVPR, 2022.

Experience

Postdoctoral Researcher - MIT (Boston, USA): Multi-modal Learning with Speech/Audio, Vision, and Language. (11.24 - Present).
Research Assistant - TU Graz (Graz, Austria): Self-supervised learning and vision-language understanding (01.21 - 10.24).
Research Scientist Internship - Sony AI (Tokyo, Japan): Multimodal vision-language understanding (05.24 - 8.24)
Internship - Intel (Karlsruhe, Germany): Evaluating robustness of object detectors in degrading weather (03.19 - 08.20).

Selected Publications

ConMe: Rethinking Evaluation of Compositional Reasoning for Modern VLMs
^*Irene Huang, ^*Wei Lin, ^*M. Jehanzeb Mirza, Jacob Hansen, Sivan Doveh, Victor Ion Butoi, Roei Herzig, Assaf Arbelle,
Hilde Kuehne, Trevor Darrel, Chuang Gan, Aude Oliva, Rogerio Feris, Leonid Karlinsky (*Equal Contribution),
NeurIPS 2024

[ Paper | Dataset | Code ]
Meta-Prompting for Automating Zero-shot Visual Recognition with LLMs

M. Jehanzeb Mirza, Leonid Karlinsky, Wei Lin, Sivan Doveh, Jakub Micorek, Mateusz Kozinski, Hilde Kuhene, Horst Possegger
ECCV 2024

[ Paper | Project Page | Code ]
Towards Multimodal In-Context Learning for Vision & Language Models

Sivan Doveh, Shaked Perek, M. Jehanzeb Mirza, Amit Alfassy, Assaf Arbelle, Shimon Ullman, Leonid Karlinsky
ECCVW 2024

[ Paper ]
Are Vision Language Models Texture or Shape Biased and Can We Steer Them?

Paul Gavrikov, Jovita Lukasik, Steffen Jung, Robert Geirhos, Bianca Lamm,
M. Jehanzeb Mirza, Margret Keuper, Janis Keuper
ArXiv Preprint 2024

[ Paper ]
TTT-KD: Test-Time Training for 3D Semantic Segmentation through Knowledge Distillation from Foundation Models

Lisa Weijler, M. Jehanzeb Mirza, Leon Sick, Can Ekkazan, Pedro Hermosilla
ArXiv Preprint 2024

[ Paper ]
LaFTer: Label-Free Tuning of Zero-shot Classifier using Language and Unlabeled Image Collections

M. Jehanzeb Mirza, Leonid Karlinsky, Wei Lin, Mateusz Kozinski, Horst Possegger, Rogerio Feris, Horst Bischof

NeurIPS 2023

[ Paper | Project Page | Code ]
MATE: Masked Autoencoders are Online 3D Test-Time Learners

^*M. Jehanzeb Mirza,^*Inkyu Shin, ^*Wei Lin, Andreas Schriebl, Kunyang Sun, Jaesung Choe,
Mateusz Kozinski, Horst Possegger, In So Kweon, Kun-Jin Yoon, Horst Bischof (*Equal Contribution)

ICCV 2023

[ Paper | Project Page | Code ]
ActMAD: Activation Matching to Align Distributions for Test-Time-Training

M. Jehanzeb Mirza, Pol Jane Soneira, Wei Lin, Mateusz Kozinski, Horst Possegger, Horst Bischof

CVPR 2023

[ Paper | Project Page | Code ]
Video Test-Time Adaptation for Action Recognition

^*Wei Lin, ^*M. Jehanzeb Mirza, Mateusz Kozinski, Horst Possegger, Hilde Kuehne, Horst Bischof (*Equal Contribution)

CVPR 2023

[ Paper | Code ]
The Norm Must Go On: Dynamic Unsupervised Domain Adaptation by Normalization

M. Jehanzeb Mirza, Jakub Micorek, Horst Possegger, Horst Bischof

CVPR 2022

[ Paper | Code ]
An Efficient Domain-Incremental Learning Approach to Drive in All Weather Conditions

M. Jehanzeb Mirza, Marc Masana, Horst Possegger, Horst Bischof

CVPRW 2022

[ Paper ]

Supervised Student Works (Selected)

Bachelor Thesis: Test-Time Adaptation for Multi-Modal Vision-Language Models (Ongoing).

Master Thesis: Online Test-Time Training for 3D point clouds with Masked Autoencoders (Completed) [ paper @ ICCV ]

Bachelor Thesis: Online Domain Incremental Learning for Driving in Adverse Weather Conditions (Completed) [ paper @ IEEE IV (oral) ]

Master Thesis: How Much are Data Augmentations Worth for Representation Learning in 3D Point Clouds (Completed).