3

I have recently gone about and made a simple AI, one that gives responses to an input (albeit completely irrelevant and nonsensical ones), using Synaptic.js. Unfortunately, this is not the type of text generation I am looking for. What I am looking for would be a way to get connections between words and generate text from that. (What would be preferable would be to also generate at least semi-sensible answers also.)

This is part of project Raphiel, and can be checked out in the room associated with this site. What I want to know is what layer combination would I use for text generation?

I have been told to avoid retrieval-based bots.

I have the method to send and receive messages, I just need to figure out what combination of layers would be the best.

Unless I have the numbers wrong, this will be SE's second NN chatbot.

FreezePhoenix
  • 442
  • 3
  • 20

1 Answers1

2

This seems like a problem for the use of an encoder-decoder pair such as those seen in text summarization (see this paper by Rush et al.: https://arxiv.org/pdf/1509.00685.pdfï%C2%BC).

You would need the following layers:

  • LSTM layer to encode the given input text into an embedding

  • LSTM layer the looks over the currently generated output to encode the text into an embedding

  • A dense soft-max layer for generating words probabilistically based on the output of the two contextual LSTM encoders

Please see the following blog post by Jason Brownlee that outlines this approach and others, while giving implementation details and snippets (https://machinelearningmastery.com/encoder-decoder-models-text-summarization-keras/)!

Also note that this would require a large set of training examples of input text and reasonable responses. You might be able to scrape Reddit post responses and comments off of those for a start? Let me know if I misunderstood the question.

JMed
  • 76
  • 3