Google Home Development Tutorial with Ruby, PaizaCloud

(Japanese article is here)

Hello, I'm Tsuneo(@yoshiokatsuneo).

Are you using smart speaker?

I just started using Google Home. At first, I thought I don't need such a device. But, once start using Google Home, I cannot go back the life without the Google Home.

With Google home, I don't need to pick up my mobile phone but I can just speak to use it without using hands. As the result of evolving machine learning and AI technology, I feel the speech recognition is practical level.

On a research, 18% of the adults in the US have a smart speaker, and 65% of that wouldn't go back to life without one. (Ref: National Public Media The Smart Audio Report from NPR and Edison Research).

And, we can write programs for the Google Home to extend and add features!

Google Home itself is quite useful, but by adding your own feature, it becomes more fun and useful!

As platform handle speech recognition, you can easily write code just by handling simple string.

Everyone can use the smart speaker just by voice. It is fun that your family, friends, or colleague can use your voice application right away.

But, creating and running the Google Home application requires installing and setting up the development environment, and a server to run the Google Home application. Those can be annoying when we just want to create a simple Google Home program.

So, here comes PaizaCloud Cloud IDE, a browser-based online web and application development environment.

As PaizaCloud have Google Home application development environment, you can just start coding for the Google Home application program in your browser.

And, as you can develop in the cloud, you can just run the Google Home application on the same machine without setting up another server and deploying to it.

Here, we develop a Google Home application reading tweets on Twitter using Ruby on the PaizaCloud Cloud IDE.

Following the instruction below, you'll create and run the Google Home application just in 10 minutes.

Getting started with PaizaCloud Cloud IDE

Let's start!

Here is the website of PaizaCloud Cloud IDE.

Just sign up with email and click a link in the confirmation email. You can also sign up with GitHub or Google.

Create new server

Let's create a new server for the development workspace.

Click "new server" to open a dialog to set up the server. Just click "New Server" button in the dialog without any settings.

Just in 3 seconds, you'll get a browser-based online development environment for creating Google Home application using Ruby.

Create Google Assistant application

We can add features to Google Home by creating a Google Assistant application.

The Google Assistant application can be built on a web site called Actions on Google.

Open the Actions on Google page below:

https://console.actions.google.com/

Click "Add/import project" button.

As "Add project" dialog box is shown, input your project name and the country name. Here, we input "MyTweet" to the project name, and set country to "Japan". Then, click "CREATE PROJECT" button.

A project is created on the Actions on Google. To build speech application for the project, we use Dialogflow.

Click "BUILD" button on "Dialogflow".

Click "CREATE ACTIONS ON DIALOGFLOW" button to create a Dialogflow action.

If you are required to login or authenticate, login to the service.

Switch "DEFAULT LANGUAGE" to your language(Ex: Japanese), and click "CREATE" button.

An agent for the conversation application is created on the Dialogflow.

The first conversation

At first, let the Google assistant application say something!

On Dialogflow menu, click "Intents".

Click "Default Welcome Intent", which is the place to write action just after the application starts.

Go "Response" section. "Text response" is the place to set what the application says. Click trash icon to remove the default message. Click "ADD MESSAGE CONTENT" button and choose "Text response" to add a message. Here, we set the message to "Welcome to My Twitter".

Then, click "SAVE" button on the top of the page to save the configuration.

Next, we set up Google Assistant application on Actions on Google.

Open https://console.actions.google.com/ , and click the project we created(My Twitter).

We set up the application on "Overview" page. While we need to set up correctly when we publish the application. For now, we just need to put something on the all required fields.

Go "App information" and click "EDIT".

To change the language, you can click "Add Language" to choose the language(Ex: "Japanese").

Set the application name and the pronunciation for that. Here, we can set "My Twitter" to both application name and pronunciation field, and click "NEXT" button.

As we launch the application by talking "OK Google, talk to My Twitter", you need to set the name easy to pronounce.

On "Details" section, set "Assistant app introduction"、"Short description"、"Full description". We can set like "Read tweets on Twitter". Then, click "NEXT" button.

On "Images" section, set the application images. For now, we can set any image files as placeholders.

On "Contact details" section, set your e-mail address.

On "Privacy and consent" section, set the application's privacy policy("Privacy Policy"). you can set something like "https://privaci.policy/" as a placeholder for now.

On "Additional Information", choose the category(Category) for your application. Set "Social & comminication", and click "Save" button.

After setting up everything for the application, click "TEST DRAFT" button. The actions Simulator launches.

Let's type "Talk to My Twitter".

A message "Welcome to My Twitter" we set was written, and was speeched.

Now, we can use the application from your Google Home. Let's talk to your Google Home like "OK Google, talk to My Twitter".

Now, Google Home says the response message!

Call conversation program

Next, let's the application read tweets from Twitter.

Open Dialogflow console( https://console.dialogflow.com/ ).

To call external service from Dialogflow, we use Fulfillment.

On menu, choose Fulfillment.

Now, we set the URL of the external service to the URL field. The service runs on PaizaCloud have hostname "[SERVER-NAME".paiza-user.cloud".

On PaizaCloud, the server name is "[SERVER NAME].paiza-user.cloud". The port number of "Sinatra" application we'll create is "4567". As you can automatically use HTTPS instead of HTTP on PaizaCloud, the protocol of the URL is HTTP. And, we'll use path name '/webhook' .

So, the URL is " https://[SERVER-NAME].paiza-user.cloud:4567/webhook "(Replace "[SERVER-NAME]" to your server name on PaizaCloud).

Click "SAVE" button to save the settings.

Next, on the menu, choose "Intents", and "Default Welcome Intent". Now, we can see "Fulfillment" section, so click the "Fullfillment" to open, and click "User webhook". Now, when the application launch, the Google Home application connect to the server running on the PaizaCloud.

Create server program

Now, we create a program on PaizaCloud. Here, we use Ruby and Sinatra to create the server.

On PaizaCloud, you can create and edit the file in the browser. On PaizaCloud, click "New File" button at the left-side of the page.

As a dialog box to put filename is shown, type a filename "server.rb" and click "Create" button.

The file is created!

Let's write a simple application just return the fixed message. Write a Ruby program like below.

server.rb:

require 'sinatra'
require 'sinatra/reloader'
require 'sinatra/json'

post '/webhook' do
  return json({
      speech: "Here is the server program."
  });
end

get '/' do
  return "Hello World"
end

Then, click "Save" button to save the file.

Let's see the code. On "post /webhook'" block, we write the action for POST "/webhook" request. The return value for the block will be the HTTP response.

As Dialogflow's Fulfillment Webhook requires the response message on JSON format, we use json() method here.

"speech" field of the return object is the message the application speech. Here, we set the fixed message "Here is the server program.".

Also, to test the program, we write the action for GET '/' request to "get '/'" block. Here, we just return a text "Hello World".

Then, let's run the Slack bot program we created. To run the program, run a command "ruby ./server.rb -o 0". As Sinatra listen only on localhost and does not accept connection from out of the server by default, we add "-o 0"(or, "-o 0.0.0.0") option to listen on the global address to accept connection from the clients.

On PaizaCloud, you can use a Terminal application in the browser to run the command.

Click "Terminal" button at the left side of the PaizaCloud page.

The Terminal launched. Type the command "ruby ./server.rb -o 0", and type enter key.

$ ruby ./server.rb -o 0

The command started. The text "tcp://0:4567" means that the server is running on port 4567.

Now, you see a button with a label "4567" at the left-side of the PaizaCloud page.

Sinatra listens on port 4567. PaizaCloud detects the port number 4567, and add the button for the port 4567 to launch browser automatically.

By clicking the button, browser(a browser on PaizaCloud) launch and you see the message from the server "Hello World".

Now, let's test the program.

On Dialogflow, see the right side of the page and click "See how it works in Google Assistant." to run the action Simulator. (Or, on "Actions on Google" project menu, click "Simulator")

Type "talk to My Twitter". The message "Here is the server program." was written. We got the response message from the Ruby server program we just created!

Now, let's talk to Google Home.

Say "talk to My Twitter" to the Google Home.

The Google Home says "Here is the server program"! Yeah, it just works!

Retrieving tweets on Twitter

Then, let's the application read tweets on Twitter.

To read tweets from Twitter on the program, we need Twitter API key. Create the Twitter API key following the article below.

http://paiza.hatenablog.com/entry/paizacloud_twitter_bot_ruby/2018/01/10#twitter_apikey

Then, create a configuration file(twitter-config.rb) on PaizaCloud, and set API keys like below.

Here, replace YOUR_CONSUMER_KEY / YOUR_CONSUMER_SECRET / YOUR_ACCESS_TOKEN / YOUR_ACCESS_SECRET to your application's API key you just created.

twitter-config.rb:

require 'twitter'

config = {
    consumer_key: "YOUR_CONSUMER_KEY",
    consumer_secret: "YOUR_CONSUMER_SECRET",
    access_token: "YOUR_ACCESS_TOKEN",
    access_token_secret: "YOUR_ACCESS_SECRET",
}

$twitterRestClient = Twitter::REST::Client.new(config)
$twitterStreamingClient = Twitter::Streaming::Client.new(config)

Then, change the server program to read tweets.

server.rb:

require 'sinatra'
require 'sinatra/reloader'
require 'sinatra/json'
require './config'

post '/webhook' do
  tweets = $restClient.home_timeline
  text = tweets[0].text
  return json({
      speech: "Reading the latest tweets using API." + text
  });
end

get '/' do
  return "Hello World"
end

Let's test the application.

On Dialogflow, see the right side of the page and click "See how it works in Google Assistant." to run the action Simulator. (Or, on "Actions on Google" project menu, click "Simulator")

Let's type "talk to My Twitter".

You got the latest tweet message!

Then, let's talk to Google Home.

Say "OK Google, talk to My Twitter".

Google Home read the latest tweet on Twitter!

Search tweet on Twitter

Next, let's add a feature to search a tweet from Twitter.

Let's make the application response to the message "search xxx".

Open Dialogflow( https://console.dialogflow.com/ ). On the menu, choose "Intents" and click "CREATE INTENT" button to show the page to create the Intent.

Set expected user's voice to "User says" field.

At first, type "Search foo".

As the part of "foo" is not fixed, set the part as a parameter. Choose the area "foo" and click it. As the list is shown, choose "@sys.any".

Set "keyword" to "PARAMETER NAME". Now, the part of "foo" can be anything and can be referred as "keyword" parameter of the message.

As the parameter is required, check "REQUIRED" checkbox. To set the message for the case user's message doesn't containers the parameter, click "Define prompts...".

Here, we set "Set search keyword like: search whatever".

As we use the server program on PaizaCloud, click "Fulfillment", and check "Use Fulfillment", and check "Use webhook".

Click "SAVE" button to save the settings.

Change the Ruby server program("server.rb") like below:

server.rb:

require 'sinatra'
require 'sinatra/reloader'
require 'sinatra/json'
require './config'

post '/webhook' do
    obj = JSON.parse(request.body.read)
    keyword = obj['result']['parameters']['keyword']
    puts "keyword=#{keyword}"
    if ! keyword
        tweets = $twitterRestClient.home_timeline
        text = tweets[0].text
        return json({
            speech: "I'm reading the latest tweet on Twitter. " + text
        });
    else
        tweets = $twitterRestClient.search(keyword).take(5)
        if tweets[0]
            text = tweets[0].text
            return json({
                speech: "I'm reading a tweet of the search result for keyword #{keyword}." + text
            });
        else
            return json({
                speech: "I cannot find any tweet for keyword #{keyword}."
            });
        end
            
    end
end

get '/' do
  return "Hello World"
end

Let's see the code. For the user's message "Search for xxx", the keyword part "xxx" is passed to the program as HTTP request body as JSON format. So, convert it to the object using JSON.parse(). As the parameters are passed as "result", "parameters" field, we get the "keyword" parameter from it.

Let's test the program.

On Dialogflow, see the right side of the page, and click "See how it works in Google Assistant." to start action Simulator(Or, on "Actions on Google" project menu, click "Simulator".)

Type "talk to My Twitter". Then input "search by ...".

We got the search result!

Next, let's talks to the Google Home.

Say "OK Google, talk to My Twitter", the say "search by ...".

Google Home says the Twitter search result! It's succeeded!

Now, we created the Google Home application to speech tweets on Twitter!

Note that on PaizaCloud free plan, the server will be suspended. To run the bot continuously, please upgrade to the BASIC plan.

Summary

We created a Google Home application on PaizaCloud using Ruby, and run it on the same environment.

We can create Google Home application without implementing speech recognition. It is fascinating that everyone can easily use the voice application. It is quite fun to create the application thinking what to response to the user's message. Now, it is the time to create your own application!

---

With「PaizaCloud Cloud IDE」, you can flexibly and easily develop your web application or server application, and publish it, just in your browser.

---