GPT-2, a language model recently published by OpenAI researchers has attracted a lot of attention for a number of reasons. The model has been trained on a giant, somewhat curated and minimally processed text scraped off the Internet. It is based on the Transformer architecture that has been highly successful on a variety of NLP tasks such as machine translation, text comprehension, question answering and many more.
The model indeed demonstrates some impressive abilities, especially when considering that it has not been trained on any specific NLP task. Instead, it only learned to predict the next word given a sequence of words, a task known as language modelling. For example, the model will complete the sentence “Thank you for the meal, it was absolutely” with the word “delicious”.
Its ability to complete sentences, or invent paragraphs of text is particularly interesting (or funny/creepy, depending on whom you ask). For example, given the following text excerpt:
Context (human-written): In a shocking finding, scientist discovered a herd of unicorns living in a remote, previously unexplored valley, in the Andes Mountains. Even more surprising to the researchers was the fact that the unicorns spoke perfect English
The model continues:
The scientist named the population, after their distinctive horn, Ovid’s Unicorn. These four-horned, silver-white unicorns were previously unknown to science. Now, after almost two centuries, the mystery of what sparked this odd phenomenon is finally solved. Dr. Jorge Pérez, an evolutionary biologist from the University of La Paz, and several companions, were exploring the Andes Mountains when they found a small valley, with no other animals or humans. Pérez noticed that the valley had what appeared to be a natural fountain, surrounded by two peaks of rock and silver snow.
The fully completed text is available in the manuscript (more nice examples are at pages 19 and 20). The text is very convincing at the first reading, the model comes up with reasonable names given the location in South America (Dr. Jorge Pérez, University of La Paz), and it also gets some important details right (evolutionary scientists, prepends the “Dr” title). It might take some time to notice inconsistencies, such as a unicorn having four horns or two centuries timeline.
The authors imply that texts generated by the model are of such quality that they can be perceived as written by a human, and thus misused, which is the reason why they decided not to make the model available to the public (yet). Instead, they released a smaller version of the model, which is not as powerful, but still does a decent job on many language problems.
I was wondering how “good” would the small model be at coming up with alternate lyrics for some famous songs. So I selected a few well-known songs, based on my personal preferences, as well as the popularity (the number of YouTube views). In some cases, the model was able to complete the original lyrics correctly, but I was more interested in less conventional options. There were many possible completions, and I selected ones which I thought were particularly… refreshing :). Some of the results are quite entertaining. The last verse is the produced by the model.
Metallica - Nothing Else Matters
Revealing the dark side of AI…
So close, no matter how far Couldn't be much more from the heart Forever trust in who we are And nothing else matters Never opened myself this way Life is ours, we live it our way All these words, I don't just say And nothing else matters Trust I seek and I find in you Every day for us something new Open mind for a different view And nothing else matters Not long ago I've been telling you That if you don't believe in God's Will I will kill you, but if I tell you so much It will be true Do it as you've done, if you're a liar
Police - Every Breath You Take
Every breath you take and every move you make Every bond you break, every step you take, I'll be watching you Every single day, every word you say Every game you play, every night you stay, I'll be watching you Oh can't you see, you belong to me How my poor heart aches with every step you take Every move you make and every vow you break Every smile you fake, every claim you stake, I'll be watching you Oh and the world moves with me every tiny moment I hear, you belong to me How my heart aches with every bite you make, every kiss you make, I'll be watching you No matter what you say - I'll be watching you
Led Zeppelin - Stairway to Heaven
There's a lady who's sure all that glitters is gold And she's buying a stairway to heaven. When she gets there she knows, if the stores are all closed With a word she can get what she came for. Ooh, ooh, and she's buying a stairway to heaven. There's a sign on the wall but she wants to be sure 'Cause you know sometimes words have two meanings. In a tree by the brook, there's a songbird who sings, Sometimes all of our thoughts are misgiven. But all of our thoughts are just misunderstood. So you don't know which words are true! So she thinks she's heard all the things her father taught her. And then she tries to tell her she could read those things for herself.
Rihanna - Umbrella
An unexpected, but consistent, plot twist.
You had my heart, and we'll never be worlds apart Maybe in magazines, but you'll still be my star Baby 'cause in the dark, you can't see shiny cars And that's when you need me there With you, I'll always share Because when the sun shines, we'll shine together Told you I'll be here forever Said I'll always be your friend Took an oath, I'ma stick it out to the end Now that it's raining more than ever Know that we'll still have each other You can stand under my umbrella And call me 'Lil Mama' 'Cause Mama's right here And I'm always here to help you break your fast, Because Mama's right there And I'm always here to help you be safe 'Cause Mama's right there 'Cause there's my baby And Mama's right there And we'll be your friends And we'll always be there.
Ed Sheeran - Shape of You
This one has a rather unexpected ending… But it has good intentions.
The club isn't the best place to find a lover So the bar is where I go Me and my friends at the table doing shots Drinking fast, and then we talk slow And you come over and start up a conversation with just me And trust me, I'll give it a chance now Take my hand, stop Put "Van the Man" on the jukebox And then we start to dance And now I'm singing like You know what I mean That's the way it is So please, I know it's weird But I know it's right Because I just tell it And I say "Hi, nice to meet you You better get dressed"
Justin Bieber - Sorry
It flows nicely… until it doesn’t.
You gotta go and get angry at all of my honesty You know I try but I don't do too well with apologies I hope I don't run out of time, could someone call a referee? Cause I just need one more shot at forgiveness I know you know that I made those mistakes maybe once or twice By once or twice I mean maybe a couple a hundred times So let me, oh let me redeem, oh redeem, oh myself tonight Cause I just need one more shot at second chances Yeah, is it too late now to say sorry? Cause I'm missing more than just your body Is it too late now to say sorry? Yeah I know that I let you down Is it too late to say I'm sorry now? Can't you just say something, can you just say something ? That's it So let me, oh let me redeem, oh redeem I know that I let you down Is it too late now to say I'm sorry? I think it hurts to think that you're just another one of those young boys who just tried to get a job like a lawyer and then couldn't Is it too late now to say I'm sorry?