Share this page!

Enter URL or ID to Unroll

You can paste full URL like: https://x.com/threadreaderapp/status/1644127596119195649
or just the ID like: 1644127596119195649

How to get URL link on X (Twitter) App

  1. On the Twitter thread, click on or icon on the bottom
  2. Click again on or Share Via icon
  3. Click on Copy Link to Tweet
  4. Paste it above and click "Unroll Thread"!
  5. More info at Twitter Help
Wasi Ahmad Profile picture
Wasi Ahmad
@ahmadwasi
Applied Scientist, AWS AI Labs
May 4, 2023 • 4 tweets • 2 min read
#ACL2023NLP
Do causal language models (CLMs) yield good representations with good isotropy and discrimination?

The answer is not always! To address the issue, our ACL2023 paper (arxiv.org/pdf/2210.01185…) proposes ContraCLM.

Joint work with @DejiaoZhang @nihalj_ We show that CodeGen (350M to 16B) pretrained on source code, and text-based CLMs (smaller than GPT2-Large) generated representations suffer from anisotropy and poor discrimination. Image