Spike No More: Stabilizing the Pre-training of Large Language Models
Sho Takase, Shun Kiyono, Sosuke Kobayashi, Jun Suzuki
Preprint, Dec. 2023. [arxiv]
Sosuke Kobayashi
I’m a researcher at Preferred Networks, Inc. and a Specially Appointed Associate Professor (Visiting) at the Center for Language AI Research, Tohoku University (history). I hold a PhD in Information Science, and have worked in areas like machine learning, natural language processing, and 2D/3D/4D computer vision. I'm broadly interested in exploring surprising and useful findings and applications across various fields.
- neural scene representations with semantic or language features (NeurIPS 2022) and real-time volumetric videos (video: 1, 2, 3, 4)
- analyzing influence of training by deterministic dropout (SustaiNLP 2020, JNLP 2021)
- subnetworks for building diverse models at once (BigScience 2022, SustaiNLP 2020, JNLP 2021)
- random parameters in neural networks (SustaiNLP 2020, JNLP 2021, Takase and me. NeurIPS 2020)
- controlling robots/agents with language or other interactions (ICRA 2018 [Best Paper in HRI], Fu et al. ViGIL 2019, Arakawa et al. RT-DUNE 2019, GTC 2019)
- language-model-based conditional data augmentation (NAACL 2018)
- entity-centric representations in NLP (NAACL 2016, IJCNLP 2017)
improving deep Transformers and large language models (Takase et al. ACL Findings 2023, Takase et al. Arxiv 2023), machine translation for language learning (Arakawa et al. CHI 2022), shift invariance in Transformers (Kiyono et al. EMNLP 2021), instance-based learning (Ouchi et al. ACL 2020, Ouchi et al. TACL 2021), video generation (Saito et al. IJCV 2020), test-time data augmentation (Shimada et al. LLD 2019), co-occurrence estimation (Yokoi et al. EMNLP 2018), controlling text styles (Akama et al. IJCNLP 2017, Akama et al. ACL 2018).
Refereed Publications and Preprints
(or see Google Scholaroogle Scholar)B2T Connection: Serving Stability and Performance in Deep Transformers
Sho Takase, Shun Kiyono, Sosuke Kobayashi, Jun Suzuki
In Proceedings of Findings of ACL 2023, July 2023. [paper] [arxiv] [code]
Decomposing NeRF for Editing via Feature Field Distillation
Sosuke Kobayashi, Eiichi Matsumoto, Vincent Sitzmann
In Proceedings of NeurIPS 2022, Nov. 2022. [paper] [arxiv] [code] [project]
Diverse Lottery Tickets Boost Ensemble from a Single Pretrained Model
Sosuke Kobayashi, Shun Kiyono, Jun Suzuki, Kentaro Inui
In Proceedings of Workshop on Challenges & Perspectives in Creating Large Language Models (BigScience), May 2022. [paper] [arxiv] [bib]
VocabEncounter: NMT-powered Vocabulary Learning by Presenting Computer-Generated Usages of Foreign Words into Users' Daily Lives
Riku Arakawa, Hiromu Yakura, Sosuke Kobayashi
Instance-Based Neural Dependency Parsing
Hiroki Ouchi, Jun Suzuki, Sosuke Kobayashi, Sho Yokoi, Tatsuki Kuribayashi, Masashi Yoshikawa, Kentaro Inui
TACL (Transactions of the Association for Computational Linguistics), Vol. 9, Dec. 2021. [paper] [arxiv] [bib]
SHAPE: Shifted Absolute Position Embedding for Transformers
Shun Kiyono, Sosuke Kobayashi, Jun Suzuki, Kentaro Inui
In Proceedings of EMNLP 2021, Nov. 2021. [paper] [arxiv] [bib]
*Efficient Estimation of Influence of a Training Instance
Sosuke Kobayashi, Sho Yokoi, Jun Suzuki, Kentaro Inui
Journal of Natural Language Processing (*written in Japanese, extended version of the paper at SustaiNLP 2020), Vol. 28-2, June 2021. (論文賞; Best Paper Award) [paper] [arxiv] [code] [bib]
Efficient Estimation of Influence of a Training Instance
Sosuke Kobayashi, Sho Yokoi, Jun Suzuki, Kentaro Inui
In Proceedings of First Workshop on Simple and Efficient Natural Language Processing (SustaiNLP 2020), Nov. 2020. [paper] [arxiv] [code] [bib]
All Word Embeddings from One Embedding
Sho Takase, Sosuke Kobayashi
In Proceedings of NeurIPS 2020, Dec. 2020. [paper] [arxiv] [code] [bib]
Instance-based Learning of Span Representations: A Case Study through Named Entity Recognition
Hiroki Ouchi, Jun Suzuki, Sosuke Kobayashi, Sho Yokoi, Tatsuki Kuribayashi, Ryuto Konno, Kentaro Inui
In Proceedings of ACL 2020, July 2020. [paper] [arxiv] [bib]
Train Sparsely, Generate Densely: Memory-Efficient Unsupervised Training of High-Resolution Temporal GAN
Masaki Saito, Shunta Saito, Masanori Koyama, Sosuke Kobayashi
International Journal of Computer Vision, May 2020. [paper] [arxiv] [code]
Learning from Observation-Only Demonstration for Task-Oriented Language Grounding via Self-Examination
Tsu-Jui Fu, Yuta Tsuboi, Sosuke Kobayashi, Yuta Kikuchi
In Proceedings of 3rd Workshop on Visually Grounded Interaction and Language (ViGIL), Dec. 2019. [paper]
Data Interpolating Prediction: Alternative Interpretation of Mixup
Takuya Shimada, Shoichiro Yamaguchi, Kohei Hayashi, Sosuke Kobayashi
In Proceedings of 2nd Workshop on Learning with Limited Labeled Data: Weak Supervision and Beyond (LLD), May 2019. [paper] [arxiv]
DQN-TAMER: Human-in-the-Loop Reinforcement Learning with Intractable Feedback
Riku Arakawa, Sosuke Kobayashi, Yuya Unno, Yuta Tsuboi, Shin-ichi Maeda
In Proceedings of 2nd Workshop on Human-Robot Teaming Beyond Human Operational Speeds and Robot Teammates Operating in Dynamic, Unstructured Environments (RT-DUNE), May 2019. [arxiv]
Pointwise HSIC: A Linear-Time Kernelized Co-occurrence Norm for Sparse Linguistic Expressions
Sho Yokoi, Sosuke Kobayashi, Kenji Fukumizu, Jun Suzuki, Kentaro Inui
In Proceedings of EMNLP 2018, Oct. 2018. [paper] [arxiv] [slide] [bib]
Unsupervised Learning of Style-sensitive Word Vectors
Reina Akama, Kento Watanabe, Sho Yokoi, Sosuke Kobayashi, Kentaro Inui
In Proceedings of ACL 2018, July 2018. [paper] [arxiv] [code] [bib] [project]
Contextual Augmentation: Data Augmentation by Words with Paradigmatic Relations
Sosuke Kobayashi
In Proceedings of NAACL HLT 2018, June 2018. [paper] [arxiv] [code] [bib]
Interactively Picking Real-World Objects with Unconstrained Spoken Language Instructions
Jun Hatori*, Yuta Kikuchi*, Sosuke Kobayashi*, Kuniyuki Takahashi*, Yuta Tsuboi*, Yuya Unno*, Wilson Ko, Jethro Tan (* equally contributed)
In Proceedings of International Conference on Robotics and Automation (ICRA) 2018, June 2018. (Best Paper Award on Human-Robot Interaction (HRI)) [paper] [arxiv] [dataset] [bib] [project] [video]
A subsequent work of this is on an autonomous robot system demo (WSJ, BBC, NVIDIA GTC 2019)
A Neural Language Model for Dynamically Representing the Meanings of Unknown Words and Entities in a Discourse
Sosuke Kobayashi, Naoaki Okazaki, Kentaro Inui
In Proceedings of IJCNLP 2017, Nov. 2017. [paper] [arxiv] [slide] [bib] [video]
Generating Stylistically Consistent Dialog Responses with Transfer Learning
Reina Akama, Kazuaki Inada, Naoya Inoue, Sosuke Kobayashi, Kentaro Inui
An RNN-based Binary Classifier for the Story Cloze Test
Melissa Roemmele, Sosuke Kobayashi, Naoya Inoue, Andrew Gordon
In Proceedings of 2nd Workshop on Linking Models of Lexical, Sentential and Discourse-level Semantics (LSDSem), Apr. 2017. [paper] [bib]
Explaining Potential Risks in Traffic Scenes by Combining Logical Inference and Physics Simulation
Ryo Takahashi, Naoya Inoue, Yasutaka Kuriya, Sosuke Kobayashi, Kentaro Inui
International Journal of Machine Learning and Computing (IJMLC), Oct. 2016. [paper]
Tohoku at SemEval-2016 Task 6: Feature-based Model versus Convolutional Neural Network for Stance Detection
Yuki Igarashi, Hiroya Komatsu, Sosuke Kobayashi, Naoaki Okazaki, Kentaro Inui
In Proceedings of 10th International Workshop on Semantic Evaluation (SemEval 2016), June 2016. [paper] [bib]
Dynamic Entity Representation with Max-pooling Improves Machine Reading
Sosuke Kobayashi, Ran Tian, Naoaki Okazaki, Kentaro Inui
In Proceedings of NAACL HLT 2016, June 2016. [paper] [code] [bib]
Recognizing Potential Traffic Risks through Logic-based Deep Scene Understanding
Naoya Inoue, Yasutaka Kuriya, Sosuke Kobayashi, Kentaro Inui
In Proceedings of 22nd ITS World Congress, Oct. 2015. (Best of the Rest) [paper]
Other Publications (mainly in Japanese)
大規模言語モデル事前学習の安定化
高瀬翔, 清野舜, 小林颯介, 鈴木潤
In Proceedings of 言語処理学会第30回年次大会, Mar. 2024. (委員特別賞; Special Committee Award) [paper]
Transformer を多層にする際の勾配消失問題と解決法について
高瀬翔, 清野舜, 小林颯介, 鈴木潤
In Proceedings of 言語処理学会第28回年次大会, Mar. 2022. (優秀賞; Outstanding Paper Award) [paper]
シフト付き絶対位置埋め込み
清野舜, 小林颯介, 鈴木潤, 乾健太郎
In Proceedings of 言語処理学会第28回年次大会, Mar. 2022. (若手奨励賞; Young Researcher Award) [paper]
事例ベース依存構造解析のための依存関係表現学習
大内啓樹, 鈴木潤, 小林颯介, 横井祥, 栗林樹生, 吉川将司, 乾健太郎
In Proceedings of 言語処理学会第27回年次大会, Mar. 2021. [paper]
単一事例エキスパートの統合によるドメイン適応
清野舜, 小林颯介, 鈴木潤, 乾健太郎
In Proceedings of 言語処理学会第27回年次大会, Mar. 2021. [paper]
スパン間の類似性に基づく事例ベース構造予測
大内啓樹, 鈴木潤, 小林颯介, 横井祥, 栗林樹生, 乾健太郎
In Proceedings of 言語処理学会第26回年次大会, Mar. 2020. [paper]
実世界での話し言葉指示による物体移動：深層学習による画像・言語理解
高橋城志*, 羽鳥潤*, 菊池悠太*, 小林颯介*, 坪井祐太*, 海野裕也*, 中島統太郎, 福田昌昭, Wilson Ko, Jethro Tan (* equally contributed)
In Proceedings of 第36回日本ロボット学会学術講演会, Sep. 2018. (研究奨励賞; Young Investigation Excellence Award) [paper]
カーネル法に基づく疎な言語表現のための高速計算可能な共起尺度
横井祥, 小林颯介, 福水健次, 乾健太郎
In Proceedings of 第32回人工知能学会全国大会, June 2018. (全国大会優秀賞; JSAI Annual Conference Award) [paper]
スタイルの類似性を捉えた単語ベクトルの教師なし学習
赤間怜奈, 渡邉研斗, 横井祥, 小林颯介, 乾健太郎
In Proceedings of 第32回人工知能学会全国大会, June 2018. (全国大会優秀賞; JSAI Annual Conference Award) [paper]
実世界におけるインタラクティブな物体指示
羽鳥潤*, 菊池悠太*, 小林颯介*, 高橋城志*, 坪井祐太*, 海野裕也*, Wilson Ko, Jethro Tan (* equally contributed)
In Proceedings of 言語処理学会第24回年次大会, Mar. 2018. [paper]
カーネル法に基づく疎な言語表現のための共起尺度
横井祥, 小林颯介, 福水健次, 乾健太郎
In Proceedings of 言語処理学会第24回年次大会, Mar. 2018. [paper]
サンプリング戦略に基づく単語ベクトルの意味成分とスタイル成分の分離
赤間怜奈, 横井祥, 渡邉研斗, 小林颯介, 田然, 乾健太郎
In Proceedings of 言語処理学会第24回年次大会, Mar. 2018. [paper]
カーネル法に基づく共起尺度
横井祥, 福水健次, 小林颯介, 乾健太郎
In Proceedings of 第20回情報論的学習理論ワークショップ (IBIS), Nov. 2017. [slide]
対話返答生成における個性の追加反映
濱田晃一, 藤川和樹, 小林颯介, 菊池悠太, 海野裕也, 土田正明
In Proceedings of 第232回 情報処理学会 自然言語処理研究会, July 2017. [paper] [slide] [bib]
転移学習を用いた対話応答のスタイル制御
赤間怜奈, 稲田和明, 小林颯介, 佐藤祥多, 乾健太郎
In Proceedings of 言語処理学会第23回年次大会, Mar. 2017. [paper]
Recognizing Potential Traffic Risks through Logic-based Deep Scene Understanding
Shunya Maruta, Yasutaka Kuriya, Naoya Inoue, Sosuke Kobayashi, Kentaro Inui
In Proceedings of Denso Technical Review, 2016. (Modified version of the paper of 22nd ITSWC) [paper]
共参照関係に基づく分散表現の共有と動的更新
小林颯介, 岡崎直観, 乾健太郎
In Proceedings of NLP若手の会 (YANS) 第11回シンポジウム, Aug. 2016. (奨励賞; Encouragement Award)
談話内における局所文脈の動的分散表現
小林颯介, 田然, 岡崎直観, 乾健太郎
In Proceedings of 言語処理学会第22回年次大会, Mar. 2016. (優秀賞; Outstanding Paper Award) [paper]
再帰型ニューラルネットワークを用いた対話破綻検出と言語モデルのマルチタスク学習
小林颯介, 海野裕也, 福田昌昭
In Proceedings of 第75回 人工知能学会 言語・音声理解と対話処理研究会, Oct. 2015. [paper]
物理モデルと論理推論の統合による運転シーンの潜在的危険の予測
小林颯介, 井之上直也, 栗谷康隆, 近藤敏之, 安部克則, 奥野英一, 乾健太郎
In Proceedings of 自動車技術会 2015年春季大会学術講演会講演予稿集, May 2015.
Other Activities
Education and Experience
- 2024.10 - present: Specially Appointed Associate Professor (Visiting) at the Center for Language AI Research, Tohoku University, Sendai.
- 2021.11 - 2024.10: Researcher (part-time) at FaiLab in Tohoku University, Sendai.
- 2018.10 - 2021.9: Doctor of Information Science at Inui-Suzuki Lab in Graduate School of Information Sciences, Tohoku University, Sendai.
- 2016.10 - present: Researcher at Preferred Networks, Inc., Tokyo. Research and development on machine learning, 2D/3D computer vision, NLP, robotics, etc.
- 2014.10 - 2016.9: Master of Information Science at Inui-Okazaki Lab in Graduate School of Information Sciences, Tohoku University, Sendai.
- 2011.4 - 2014.9: Department of Information and Intelligent Systems, Tohoku University, Sendai. (Early graduation)
- 2008.4 - 2011.3: Suzaka High School, Nagano.
Internship and Part-time Work
- Part-time Work at Preferred Infrastructure, Inc., Neural Dialogue Response Generation and Multimodal NLP, Oct. 2015, Feb. and Mar. 2016.
- Internship at Preferred Infrastructure, Inc., Neural Dialogue Response Generation, Aug. and Sep. 2015.
- Part-time Work at Recruit Jobs Co., Ltd., Data Analysis and Development for Web Services, 2015.
- Internship at Recruit Holdings Co., Ltd., Data Analysis and Development for Web Services, Feb. 2015. (最優秀賞; Best Team Award)
- Collaborative Research with DENSO Corporation, Recognizing Potential Risks in Traffic Scenes, 2014--2015
Competition
- 20th/946, solo silver medal, at chaii - Hindi and Tamil Question Answering in Kaggle, Nov. 2021. (First time participating)
Review
- Paper review of NLP Conferences/Journals: COLM (2024), ACL Rolling Review (2021-2022, 2024), ACL (2018-2021), EMNLP (2018-2020), NAACL (2019), COLING [Area Chair] (2022), 論文誌 自然言語処理 (2017, 2020-2022), [Editor] (2023-2024), 情報処理学会NL研 [Award Committee] (2023-2024)
- Paper review of ML Conferences: ICLR (2024), NeurIPS (2022-2023), ICML (2022, 2024)
- Paper review of Robotics Conference: ICRA (2019)
- Review and revision of Deep Learning Tutorial with Chainer, 2019.
- Review of Kaggleに挑む深層学習プログラミングの極意 (Essentials of Deep Learning Programming for Kaggle Challenge), 小嵜耕平, 秋葉拓哉, 林孝紀, 石原祥太郎, 講談社, 2023. [amazon]
- Review of 深層学習による自然言語処理 (機械学習プロフェッショナルシリーズ) (Natural Language Processing by Deep Learning, A Series of Machine Learning Professionals), 坪井祐太, 海野裕也, 鈴木潤, 講談社, 2017. [amazon]
- Review of ウェブデータの機械学習 (機械学習プロフェッショナルシリーズ) (Web Data and Machine Learning, A Series of Machine Learning Professionals), ダヌシカボレガラ, 岡﨑直観, 前原貴憲, 講談社, 2016. [amazon]
- Review of 深層学習 Deep Learning, 神嶌敏弘 (編), 麻生英樹, 安田宗樹, 前田新一, 岡野原大輔, 岡谷貴之, 久保陽太郎, ボレガラダヌシカ, 人工知能学会監修, 近代科学社, 2015. [amazon]
Research/Teaching Assistant
- Research Assistant of JST, CREST, 構造理解に基づく大規模文献情報からの知識発見, Dec. 2015--Sep. 2016.
- Teaching Assistant (Research Mentor) of Team-based Engineering Design Course (for underguraduates) of Tohoku University, Vector Representations of Emoji (Emoticons) on Microblogs, Autumn--Winter 2015--2016.
- Teaching Assistant of Information Communication Theory and Natural Language Processing (for guraduates) of Tohoku University, Autumn--Winter 2015--2016.
- Teaching Assistant of Programming Practice A (for undergraduates) of Tohoku University, Spring--Summer 2015.
- Teaching Assistant (Research Mentor) of Team-based Engineering Design Course (for underguraduates) of Tohoku University, Knowledge Extraction from Web for Question Answering, Autumn--Winter 2014--2015.
Talk
- Talk about Machine Learning for Language and the World at NLP若手の会 (YANS) 第18回シンポジウム (18th Symposium of Young Researcher Association for NLP Studies), Aug. 2023.
- Talk about Efficient Estimation of Influence of a Training Instance (best paper invited talk of JNLP) at the 28th Annual Meeting of the Association for Natural Language Processing (remote), Mar. 2022.
- Talk about an efficient estimation of machine learning models trained without an instance at the 22nd Doctoral Student Presentations of GSIS, Tohoku University, Dec. 2019. (Best Presentation Award)
- Talk about ビジョン＆ランゲージによる意図理解と曖昧性解消 (Vision and Language Help Intent Understanding and its Disambiguation) at 第25回 画像センシングシンポジウム (SSII2019) オーガナイズドセッション『ビジョン & ランゲージ 「意図」をどのようにモデリングするか？』, Jun. 2019
- Talk about Learning Communication with Neural Networks at PFN Seminar, Mar. 2017.
- Talk about NAACL-HLT 2016 at NLP若手の会 (YANS) 第11回シンポジウム (11th Symposium of Young Researcher Association for NLP Studies), Aug. 2016.
- Talk about Recent Progress in RNN and NLP at NLP-DL Study Group, Jun. 2016.
Organizer
- Editorial Comittee of 論文誌 自然言語処理 (Journal of Natural Language Processing), 2023-2024.
- Comittee of 情報処理学会自然言語処理(NL)研究会 (Special Interest Group on Natural Language Processing in IPSJ), 2023-2024.
- Organizer of 言語処理技術セミナー2021 (NLP Technology Seminar 2021), Oct. 2021.
- Organizer of 第11回 最先端NLP勉強会 (11th NLP Paper Reading Circle), Sep. 2019.
- Organizer of 第10回 最先端NLP勉強会 (10th NLP Paper Reading Circle), Aug. 2018.
- Organizer of 第9回 最先端NLP勉強会 (9th NLP Paper Reading Circle), Sep. 2017.
- Organizer of 第8回 最先端NLP勉強会 (8th NLP Paper Reading Circle), Sep. 2016.
Github
- Research: Reproduction
- Interval Bound Propagation - "Scalable Verified Training for Provably Robust Image Classification" (Goyal et al., 2019)
- Meta-learning LSTM Optimizer - "Learning to learn by gradient descent by gradient descent" (Andrychowicz et al., 2016)
- BERT - "BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding" (Devlin et al., 2018)
- Crawler of BookCorpus - "Aligning Books and Movies: Towards Story-like Visual Explanations by Watching Movies and Reading Books" (Zhu et al., 2015)
- Finetuned Transformer Language Model - "Improving Language Understanding by Generative Pre-Training" (Radford et al., 2018)
- ELMo - "Deep contextualized word representations" (Peters et al., 2018)
- CapsNet - "Dynamic Routing Between Capsules" (Sabour et al., 2017)
- Transformer - "Attention is All You Need" (Vaswani et al., 2017)
- Convolutional seq2seq Model - "Convolutional Sequence to Sequence Learning" (Gehring et al., 2017)
- Neural Image Caption - "Show and Tell: A Neural Image Caption Generator" (Vinyals et al., 2015)
- Skip-Thought Vectors and Sentence-level RNN Language Model - "Skip-Thought Vectors" (Kiros et al., 2015)
- Document-level RNN Language Model - "Recurrent Neural Network Regularization" (Zaremba et al., 2014)
- SRU (Simple Recurrent Unit) Language Model and Proposed Variants - "Training RNNs as Fast as CNNs." (Lei and Zhang, 2017)
- Efficient Softmax Approximation: Adaptive Softmax and BlackOut - "Efficient softmax approximation for GPUs" (Grave et al., 2017) and "BlackOut: Speeding up Recurrent Neural Network Language Models With Very Large Vocabularies" (Ji et al., 2016)
- Weight Normalization - "Weight Normalization: A Simple Reparameterization to Accelerate Training of Deep Neural Networks" (Salimans and Kingma, 2016)
- Variational Dropout - "Variational Dropout Sparsifies Deep Neural Networks" (Molchanov et al., 2017)
- Research: Original
- Turn-over Dropout - (My work at SustaiNLP 2020)
- Contextual Augmentation - (My work at NAACL 2018)
- PFN Picking Instructions for Commodities Dataset (PFN-PIC) - (Our work at ICRA 2018)
- Emergence of Language Using Discrete Sequences with Autoencoder - (My work not published, in 2017)
- Next Event Predictor - (Related to our work at LSDSem 2017)
- Sentence Generater Using Deep Convolutional Generative Adversarial Network (DCGAN) - (My work not published, in 2016)
- Dynamic Entity Representation Network - (Our work at NAACL 2016)
- Dynamic Neural Text Model - (Our work at IJCNLP 2017)
- Others
- ArxivLeaks - Read secret comment of TeX in arXiv papers
- Contributions for Chainer, a python-based Deep Learning framework. e.g., Layer normalization, Tree LSTM, seq2seq example and text classification example.