專題演講 主講人：花文妤博士  (美國 Apple 西雅圖分部資深機器學習工程師)

  • 事件日期: 2023-12-07
  • 演講者:  /  主持人:
This is an image


題　目：LACoS-BLOOM

Low-rank Adaptation with Contrastive objective on 8-bit Siamese-BLOOM

主講人：花文妤博士 
(美國 Apple 西雅圖分部資深機器學習工程師)

時　間：112年12月7日(星期四)12:10-13:00
　　　　
地　點：綜合一館427室
 
使用Google Meet線上直播，

演講開始前20分鐘可進入會議，請點選下列連結後按下「加入」即可

https://meet.google.com/sqy-fzus-oog

 
摘要

Text embeddings are useful features for several NLP applications, such as sentence similarity, text clustering, and semantic search. In this paper, we present a Low-rank Adaptation with a Contrastive objective on top of 8-bit Siamese-BLOOM, a multilingual large lan- guage model optimized to produce semantically meaningful word embeddings. The innovation is threefold. First, we cast BLOOM weights to 8-bit values.
Second, we fine-tune BLOOM with a scalable adapter (LoRA) and 8-bit Adam optimizer for sentence similarity classification. Third, we apply a Siamese architecture on BLOOM model with a contrastive objective to ease the multi-lingual labeled data scarcity. The experiment results show the quality of learned embeddings from LACoS-BLOOM is proportional to the number of model parameters and the amount of unlabeled training data. With the parameter efficient fine-tuning design, we are able to run BLOOM 7.1 billion parameters end-to-end on a single GPU ma- chine with 32GB memory. Compared to previous solution
Sentence- BERT, we achieve significant improvement on both English and multi-lingual STS tasks.
 

1/0