【重新定义matlab强大系列十八】Matlab深度学习长短期记忆 (LSTM) 网络生成文本

🔗 运行环境：Matlab

🚩 撰写作者：左手の明天

🥇 精选专栏：《python》

🔥 推荐专栏：《算法研究》

#### 防伪水印——左手の明天 ####

💗 大家好🤗🤗🤗，我是左手の明天！好久不见💗

💗今天更新系列——重新定义matlab强大系列💗

📆 最近更新：2024 年 03 月 17 日，左手の明天的第 320 篇原创博客

📚 更新于专栏：matlab

#### 防伪水印——左手の明天 ####

此示例说明如何训练深度学习长短期记忆 (LSTM) 网络以生成文本。要训练深度学习网络以生成文本，先训练“序列到序列”的 LSTM 网络，以预测字符序列中的下一个字符。要训练网络以预测下一个字符，请将移位一个时间步的输入序列指定为响应。

一、前言

二、加载训练数据

三、数据展示（词云）

四、将文本数据转换为序列

五、创建和训练 LSTM 网络

5.1 创建LSTM网络

5.2 训练LSTM网络

六、生成新文本

6.1 文本生成函数generateText

6.2 经过训练的网络生成文本

一、前言

此示例从网站读取文本。提取相关文本，然后使用自定义小批量数据存储，将文档作为小批量序列数据输入到网络。数据存储将文档转换为数字单词索引序列。深度学习网络是包含词嵌入层的 LSTM 网络。

要将字符序列输入到 LSTM 网络中，将每个训练观测值转换为由向量 $x\in \mathbb{R}^{^{D}}$ 表示的字符序列，其中 D 是词汇表中唯一字符的数量。对于每个向量，如果 x 对应于给定词汇表中索引为 i 的字符，则 $x_{i}=1$ ；如果 j≠i，则 $x_{j}=0$ 。

二、加载训练数据

从文本文件 sonnets.txt 中提取文本数据。

filename = "sonnets.txt";
textData = fileread(filename);

十四行诗缩进两个空白字符，并用两个换行符分隔。使用 replace 删除缩进，并使用 split 将文本拆分为单独的十四行诗。删除前三个元素中的主标题和每一首十四行诗之前出现的十四行诗标题。

textData = replace(textData,"  ","");
textData = split(textData,[newline newline]);
textData = textData(5:2:end);

三、数据展示（词云）

查看前几个观测值。

textData(1:10)

ans = 10×1 cell array
    {'From fairest creatures we desire increase,↵That thereby beauty's rose might never die,↵But as the riper should by time decease,↵His tender heir might bear his memory:↵But thou, contracted to thine own bright eyes,↵Feed'st thy light's flame with self-substantial fuel,↵Making a famine where abundance lies,↵Thy self thy foe, to thy sweet self too cruel:↵Thou that art now the world's fresh ornament,↵And only herald to the gaudy spring,↵Within thine own bud buriest thy content,↵And tender churl mak'st waste in niggarding:↵Pity the world, or else this glutton be,↵To eat the world's due, by the grave and thee.'                                 }
    {'When forty winters shall besiege thy brow,↵And dig deep trenches in thy beauty's field,↵Thy youth's proud livery so gazed on now,↵Will be a tatter'd weed of small worth held:↵Then being asked, where all thy beauty lies,↵Where all the treasure of thy lusty days;↵To say, within thine own deep sunken eyes,↵Were an all-eating shame, and thriftless praise.↵How much more praise deserv'd thy beauty's use,↵If thou couldst answer 'This fair child of mine↵Shall sum my count, and make my old excuse,'↵Proving his beauty by succession thine!↵This were to be new made when thou art old,↵And see thy blood warm when thou feel'st it cold.'               }
    {'Look in thy glass and tell the face thou viewest↵Now is the time that face should form another;↵Whose fresh repair if now thou not renewest,↵Thou dost beguile the world, unbless some mother.↵For where is she so fair whose unear'd womb↵Disdains the tillage of thy husbandry?↵Or who is he so fond will be the tomb,↵Of his self-love to stop posterity?↵Thou art thy mother's glass and she in thee↵Calls back the lovely April of her prime;↵So thou through windows of thine age shalt see,↵Despite of wrinkles this thy golden time.↵But if thou live, remember'd not to be,↵Die single and thine image dies with thee.'                                    }
    {'Unthrifty loveliness, why dost thou spend↵Upon thy self thy beauty's legacy?↵Nature's bequest gives nothing, but doth lend,↵And being frank she lends to those are free:↵Then, beauteous niggard, why dost thou abuse↵The bounteous largess given thee to give?↵Profitless usurer, why dost thou use↵So great a sum of sums, yet canst not live?↵For having traffic with thy self alone,↵Thou of thy self thy sweet self dost deceive:↵Then how when nature calls thee to be gone,↵What acceptable audit canst thou leave?↵Thy unused beauty must be tombed with thee,↵Which, used, lives th' executor to be.'                                                      }
    {'Those hours, that with gentle work did frame↵The lovely gaze where every eye doth dwell,↵Will play the tyrants to the very same↵And that unfair which fairly doth excel;↵For never-resting time leads summer on↵To hideous winter, and confounds him there;↵Sap checked with frost, and lusty leaves quite gone,↵Beauty o'er-snowed and bareness every where:↵Then were not summer's distillation left,↵A liquid prisoner pent in walls of glass,↵Beauty's effect with beauty were bereft,↵Nor it, nor no remembrance what it was:↵But flowers distill'd, though they with winter meet,↵Leese but their show; their substance still lives sweet.'                   }
    {'Then let not winter's ragged hand deface,↵In thee thy summer, ere thou be distill'd:↵Make sweet some vial; treasure thou some place↵With beauty's treasure ere it be self-kill'd.↵That use is not forbidden usury,↵Which happies those that pay the willing loan;↵That's for thy self to breed another thee,↵Or ten times happier, be it ten for one;↵Ten times thy self were happier than thou art,↵If ten of thine ten times refigur'd thee:↵Then what could death do if thou shouldst depart,↵Leaving thee living in posterity?↵Be not self-will'd, for thou art much too fair↵To be death's conquest and make worms thine heir.'                                }
    {'Lo! in the orient when the gracious light↵Lifts up his burning head, each under eye↵Doth homage to his new-appearing sight,↵Serving with looks his sacred majesty;↵And having climb'd the steep-up heavenly hill,↵Resembling strong youth in his middle age,↵Yet mortal looks adore his beauty still,↵Attending on his golden pilgrimage:↵But when from highmost pitch, with weary car,↵Like feeble age, he reeleth from the day,↵The eyes, 'fore duteous, now converted are↵From his low tract, and look another way:↵So thou, thyself outgoing in thy noon:↵Unlook'd, on diest unless thou get a son.'                                                            }
    {'Music to hear, why hear'st thou music sadly?↵Sweets with sweets war not, joy delights in joy:↵Why lov'st thou that which thou receiv'st not gladly,↵Or else receiv'st with pleasure thine annoy?↵If the true concord of well-tuned sounds,↵By unions married, do offend thine ear,↵They do but sweetly chide thee, who confounds↵In singleness the parts that thou shouldst bear.↵Mark how one string, sweet husband to another,↵Strikes each in each by mutual ordering;↵Resembling sire and child and happy mother,↵Who, all in one, one pleasing note do sing:↵Whose speechless song being many, seeming one,↵Sings this to thee: 'Thou single wilt prove none.''}
    {'Is it for fear to wet a widow's eye,↵That thou consum'st thy self in single life?↵Ah! if thou issueless shalt hap to die,↵The world will wail thee like a makeless wife;↵The world will be thy widow and still weep↵That thou no form of thee hast left behind,↵When every private widow well may keep↵By children's eyes, her husband's shape in mind:↵Look! what an unthrift in the world doth spend↵Shifts but his place, for still the world enjoys it;↵But beauty's waste hath in the world an end,↵And kept unused the user so destroys it.↵No love toward others in that bosom sits↵That on himself such murd'rous shame commits.'                           }
    {'For shame! deny that thou bear'st love to any,↵Who for thy self art so unprovident.↵Grant, if thou wilt, thou art belov'd of many,↵But that thou none lov'st is most evident:↵For thou art so possess'd with murderous hate,↵That 'gainst thy self thou stick'st not to conspire,↵Seeking that beauteous roof to ruinate↵Which to repair should be thy chief desire.↵O! change thy thought, that I may change my mind:↵Shall hate be fairer lodg'd than gentle love?↵Be, as thy presence is, gracious and kind,↵Or to thyself at least kind-hearted prove:↵Make thee another self for love of me,↵That beauty still may live in thine or thee.'                     }

在词云中可视化文本数据。

figure
wordcloud(textData);
title("LSTM test")

四、将文本数据转换为序列

将文本数据转换为预测变量的向量序列和响应向量的分类序列。

创建特殊字符来表示“文本开始”、“空白”、“文本结束”和“换行符”。分别使用特殊字符 "\x0002"（文本开始）、"\x00B7"（“·”，间隔点）、"\x2403"（“␃”，文本结束）和 "\x00B6"（“¶”，段落符号）。为防止出现歧义，必须选择文本中未出现的特殊字符。由于这些字符未出现在训练数据中，因此可用于此目的。

startOfTextCharacter = compose("\x0002");
whitespaceCharacter = compose("\x00B7");
endOfTextCharacter = compose("\x2403");
newlineCharacter = compose("\x00B6");

对于每个观测值，在开头插入文本开始字符，并用对应的字符替换空白和换行符。

textData = startOfTextCharacter + textData;
textData = replace(textData,[" " newline],[whitespaceCharacter newlineCharacter]);

创建文本中唯一字符的词汇表。

uniqueCharacters = unique([textData{:}]);
numUniqueCharacters = numel(uniqueCharacters);

循环处理文本数据，并创建表示每个观测值的字符的向量序列以及响应的字符分类序列。要表示每个观测值的结束，请包含文本结束字符。

numDocuments = numel(textData);
XTrain = cell(1,numDocuments);
YTrain = cell(1,numDocuments);
for i = 1:numel(textData)
    characters = textData{i};
    sequenceLength = numel(characters);
    
    % Get indices of characters.
    [~,idx] = ismember(characters,uniqueCharacters);
    
    % Convert characters to vectors.
    X = zeros(numUniqueCharacters,sequenceLength);
    for j = 1:sequenceLength
        X(idx(j),j) = 1;
    end
    
    % Create vector of categorical responses with end of text character.
    charactersShifted = [cellstr(characters(2:end)')' endOfTextCharacter];
    Y = categorical(charactersShifted);
    
    XTrain{i} = X;
    YTrain{i} = Y;
end

查看第一个观测值和相应序列的大小。该序列是一个 D×S 矩阵，其中 D 是特征数（唯一字符的数量），S 是序列长度（文本中的字符数量）。

textData{1}

ans = 
'From·fairest·creatures·we·desire·increase,¶That·thereby·beauty's·rose·might·never·die,¶But·as·the·riper·should·by·time·decease,¶His·tender·heir·might·bear·his·memory:¶But·thou,·contracted·to·thine·own·bright·eyes,¶Feed'st·thy·light's·flame·with·self-substantial·fuel,¶Making·a·famine·where·abundance·lies,¶Thy·self·thy·foe,·to·thy·sweet·self·too·cruel:¶Thou·that·art·now·the·world's·fresh·ornament,¶And·only·herald·to·the·gaudy·spring,¶Within·thine·own·bud·buriest·thy·content,¶And·tender·churl·mak'st·waste·in·niggarding:¶Pity·the·world,·or·else·this·glutton·be,¶To·eat·the·world's·due,·by·the·grave·and·thee.'

size(XTrain{1})

ans = 1×2

    62   611

查看相应的响应序列。该序列是由响应组成的 1×S 分类向量。

YTrain{1}

ans = 1×611 categorical array
     F      r      o      m      ·      f      a      i      r      e      s      t      ·      c      r      e      a      t      u      r      e      s      ·      w      e      ·      d      e      s      i      r      e      ·      i      n      c      r      e      a      s      e      ,      ¶      T      h      a      t      ·      t      h      e      r      e      b      y      ·      b      e      a      u      t      y      '      s      ·      r      o      s      e      ·      m      i      g      h      t      ·      n      e      v      e      r      ·      d      i      e      ,      ¶      B      u      t      ·      a      s      ·      t      h      e      ·      r      i      p      e      r      ·      s      h      o      u      l      d      ·      b      y      ·      t      i      m      e      ·      d      e      c      e      a      s      e      ,      ¶      H      i      s      ·      t      e      n      d      e      r      ·      h      e      i      r      ·      m      i      g      h      t      ·      b      e      a      r      ·      h      i      s      ·      m      e      m      o      r      y      :      ¶      B      u      t      ·      t      h      o      u      ,      ·      c      o      n      t      r      a      c      t      e      d      ·      t      o      ·      t      h      i      n      e      ·      o      w      n      ·      b      r      i      g      h      t      ·      e      y      e      s      ,      ¶      F      e      e      d      '      s      t      ·      t      h      y      ·      l      i      g      h      t      '      s      ·      f      l      a      m      e      ·      w      i      t      h      ·      s      e      l      f      -      s      u      b      s      t      a      n      t      i      a      l      ·      f      u      e      l      ,      ¶      M      a      k      i      n      g      ·      a      ·      f      a      m      i      n      e      ·      w      h      e      r      e      ·      a      b      u      n      d      a      n      c      e      ·      l      i      e      s      ,      ¶      T      h      y      ·      s      e      l      f      ·      t      h      y      ·      f      o      e      ,      ·      t      o      ·      t      h      y      ·      s      w      e      e      t      ·      s      e      l      f      ·      t      o      o      ·      c      r      u      e      l      :      ¶      T      h      o      u      ·      t      h      a      t      ·      a      r      t      ·      n      o      w      ·      t      h      e      ·      w      o      r      l      d      '      s      ·      f      r      e      s      h      ·      o      r      n      a      m      e      n      t      ,      ¶      A      n      d      ·      o      n      l      y      ·      h      e      r      a      l      d      ·      t      o      ·      t      h      e      ·      g      a      u      d      y      ·      s      p      r      i      n      g      ,      ¶      W      i      t      h      i      n      ·      t      h      i      n      e      ·      o      w      n      ·      b      u      d      ·      b      u      r      i      e      s      t      ·      t      h      y      ·      c      o      n      t      e      n      t      ,      ¶      A      n      d      ·      t      e      n      d      e      r      ·      c      h      u      r      l      ·      m      a      k      '      s      t      ·      w      a      s      t      e      ·      i      n      ·      n      i      g      g      a      r      d      i      n      g      :      ¶      P      i      t      y      ·      t      h      e      ·      w      o      r      l      d      ,      ·      o      r      ·      e      l      s      e      ·      t      h      i      s      ·      g      l      u      t      t      o      n      ·      b      e      ,      ¶      T      o      ·      e      a      t      ·      t      h      e      ·      w      o      r      l      d      '      s      ·      d      u      e      ,      ·      b      y      ·      t      h      e      ·      g      r      a      v      e      ·      a      n      d      ·      t      h      e      e      .      ␃

五、创建和训练 LSTM 网络

5.1 创建LSTM网络

定义 LSTM 架构。指定一个“序列到序列”LSTM 分类网络，其中包含 200 个隐含单元。将训练数据的特征维度（唯一字符的数量）设置为输入大小，将响应中的类别数量设置为全连接层的输出大小。

inputSize = size(XTrain{1},1);
numHiddenUnits = 200;
numClasses = numel(categories([YTrain{:}]));

layers = [
    sequenceInputLayer(inputSize)
    lstmLayer(numHiddenUnits,'OutputMode','sequence')
    fullyConnectedLayer(numClasses)
    softmaxLayer
    classificationLayer];

使用 trainingOptions 函数指定训练选项。将训练轮数指定为 500，将初始学习率指定为 0.01。要防止梯度爆炸，将梯度阈值设置为 2。通过将 'Shuffle' 选项设置为 'every-epoch'，指定在每轮对数据进行乱序处理。

要监控训练进度，将 'Plots' 选项设置为 'training-progress'。
要隐藏详尽输出，将 'Verbose' 设置为 false。

5.2 训练LSTM网络

小批量大小选项指定一次迭代要处理的观测值数量。指定能够均分数据的小批量大小，以确保函数使用全部观测值进行训练。否则，函数将忽略不能完成一个小批量的观测值。将小批量大小设置为 77。

options = trainingOptions('adam', ...
    'MaxEpochs',500, ...
    'InitialLearnRate',0.01, ...
    'GradientThreshold',2, ...
    'MiniBatchSize',77,...
    'Shuffle','every-epoch', ...
    'Plots','training-progress', ...
    'Verbose',false);

使用trainNetwork训练网络。

net = trainNetwork(XTrain,YTrain,layers,options);

六、生成新文本

6.1 文本生成函数`generateText`

generateText 函数逐字符生成文本，从文本开始字符开始，并使用特殊字符重新构造文本。该函数使用输出预测分数对每个字符进行采样。当网络预测到文本结束字符或生成的文本长度为 500 个字符时，该函数停止预测。

function generatedText = generateText(net,uniqueCharacters,startOfTextCharacter,newlineCharacter,whitespaceCharacter,endOfTextCharacter)

6.2 经过训练的网络生成文本

使用 generateText 函数，使用经过训练的网络生成文本。

generatedText = generateText(net,uniqueCharacters,startOfTextCharacter,newlineCharacter,whitespaceCharacter,endOfTextCharacter)

generatedText = 
    "Look, that your lepperites of such soous toor men,
     Where than proud on your sweetest but lever ill lie.
     One of Death a deal doth teal hearts come,
     And that which gives did mistress one learn
     Made mens of tongue that hands hear,
     And all they with me, do I fortune to brief;
     And every peinted could with this right ampontion sorend
     By genilir'd lime thau hours, and wonder sposing,
     And night by day you waster'd then new;
     For ailling thuse borrowest vein fulse were of here spent,
     Since my heart morey "

通过查找文本开始字符的索引来创建其向量。

numUniqueCharacters = numel(uniqueCharacters);
X = zeros(numUniqueCharacters,1);
idx = strfind(uniqueCharacters,startOfTextCharacter);
X(idx) = 1;

使用经过训练的 LSTM 网络，使用 predictAndUpdateState 和 datasample 逐字符生成文本。当网络预测到文本结束字符或生成的文本长度为 500 个字符时，停止预测。

generatedText = "";
vocabulary = string(net.Layers(end).Classes);

maxLength = 500;
while strlength(generatedText) < maxLength
    % Predict the next character scores.
    [net,characterScores] = predictAndUpdateState(net,X,'ExecutionEnvironment','cpu');
    
    % Sample the next character.
    newCharacter = datasample(vocabulary,1,'Weights',characterScores);
    
    % Stop predicting at the end of text.
    if newCharacter == endOfTextCharacter
        break
    end
    
    % Add the character to the generated text.
    generatedText = generatedText + newCharacter;
    
    % Create a new vector for the next input.
    X(:) = 0;
    idx = strfind(uniqueCharacters,newCharacter);
    X(idx) = 1;
end

对于大型数据集合、长序列或大型网络，在 GPU 上进行预测计算通常比在 CPU 上快。其他情况下，在 CPU 上进行预测计算通常更快。对于单时间步预测，请使用 CPU。要使用 CPU 进行预测，将 predictAndUpdateState 的 'ExecutionEnvironment' 选项设置为 'cpu'。

通过将特殊字符替换为对应的空白字符和换行符，重新构造生成的文本。

generatedText = replace(generatedText,[newlineCharacter whitespaceCharacter],[newline " "]);

end