Using Transfer Learning for Code-Related Tasks
release_jszf3dckjfejdjndyxn3xkqviq
by
Antonio Mastropaolo, Nathan Cooper, David Nader Palacio, Simone Scalabrino, Denys Poshyvanyk, Rocco Oliveto, Gabriele Bavota
2022
Abstract
Deep learning (DL) techniques have been used to support several code-related
tasks such as code summarization and bug-fixing. In particular, pre-trained
transformer models are on the rise, also thanks to the excellent results they
achieved in Natural Language Processing (NLP) tasks. The basic idea behind
these models is to first pre-train them on a generic dataset using a
self-supervised task (e.g, filling masked words in sentences). Then, these
models are fine-tuned to support specific tasks of interest (e.g, language
translation). A single model can be fine-tuned to support multiple tasks,
possibly exploiting the benefits of transfer learning. This means that
knowledge acquired to solve a specific task (e.g, language translation) can be
useful to boost performance on another task (e.g, sentiment classification).
While the benefits of transfer learning have been widely studied in NLP,
limited empirical evidence is available when it comes to code-related tasks. In
this paper, we assess the performance of the Text-To-Text Transfer Transformer
(T5) model in supporting four different code-related tasks: (i) automatic
bug-fixing, (ii) injection of code mutants, (iii) generation of assert
statements, and (iv) code summarization. We pay particular attention in
studying the role played by pre-training and multi-task fine-tuning on the
model's performance. We show that (i) the T5 can achieve better performance as
compared to state-of-the-art baselines; and (ii) while pre-training helps the
model, not all tasks benefit from a multi-task fine-tuning.
In text/plain
format
Archived Files and Locations
application/pdf 12.9 MB
file_swpjumtzy5a2pie36yzwpdqrym
|
arxiv.org (repository) web.archive.org (webarchive) |
2206.08574v1
access all versions, variants, and formats of this works (eg, pre-prints)