Background processing with .NET

Many apps need to do background processing outside of the context of a web request. This tutorial creates a web app that lets users input text to translate, and then displays a list of previous translations. The translation is done in a background process to avoid blocking the user's request.

The following diagram illustrates the translation request process.

Diagram of architecture.

Here is the sequence of events for how the tutorial app works:

  1. Visit the web page to see a list of previous translations, stored in Firestore.
  2. Request a translation of text by entering an HTML form.
  3. The translation request is published to Pub/Sub.
  4. A Cloud Run service subscribed to that Pub/Sub topic is triggered.
  5. The Cloud Run service uses Cloud Translation to translate the text.
  6. The Cloud Run service stores the result in Firestore.

This tutorial is intended for anyone who is interested in learning about background processing with Google Cloud. No prior experience is required with Pub/Sub, Firestore, App Engine, or Cloud Run functions. However, to understand all of the code, some experience with .NET, JavaScript, and HTML is helpful.

Objectives

  • Understand and deploy Cloud Run services.
  • Try the app.

Costs

In this document, you use the following billable components of Google Cloud:

To generate a cost estimate based on your projected usage, use the pricing calculator.

New Google Cloud users might be eligible for a free trial.

When you finish the tasks that are described in this document, you can avoid continued billing by deleting the resources that you created. For more information, see Clean up.

Before you begin

  1. Sign in to your Google Cloud account. If you're new to Google Cloud, create an account to evaluate how our products perform in real-world scenarios. New customers also get $300 in free credits to run, test, and deploy workloads.

  2. In the Google Cloud console, on the project selector page, select or create a Google Cloud project.

    Roles required to select or create a project

    • Select a project: Selecting a project doesn't require a specific IAM role—you can select any project that you've been granted a role on.
    • Create a project: To create a project, you need the Project Creator role (roles/resourcemanager.projectCreator), which contains the resourcemanager.projects.create permission. Learn how to grant roles.

    Go to project selector

  3. Verify that billing is enabled for your Google Cloud project.

  4. Enable the Firestore, Cloud Run, Pub/Sub, and Cloud Translation APIs.

    Roles required to enable APIs

    To enable APIs, you need the serviceusage.services.enable permission. If you created the project, then you likely already have this permission through the Owner role (roles/owner). Otherwise, you can get this permission through the Service Usage Admin role (roles/serviceusage.serviceUsageAdmin). Learn how to grant roles.

    Enable the APIs

  5. Install the Google Cloud CLI.

  6. If you're using an external identity provider (IdP), you must first sign in to the gcloud CLI with your federated identity.

  7. To initialize the gcloud CLI, run the following command: 

    gcloud init

  8. In the Google Cloud console, on the project selector page, select or create a Google Cloud project.

    Roles required to select or create a project

    • Select a project: Selecting a project doesn't require a specific IAM role—you can select any project that you've been granted a role on.
    • Create a project: To create a project, you need the Project Creator role (roles/resourcemanager.projectCreator), which contains the resourcemanager.projects.create permission. Learn how to grant roles.

    Go to project selector

  9. Verify that billing is enabled for your Google Cloud project.

  10. Enable the Firestore, Cloud Run, Pub/Sub, and Cloud Translation APIs.

    Roles required to enable APIs

    To enable APIs, you need the serviceusage.services.enable permission. If you created the project, then you likely already have this permission through the Owner role (roles/owner). Otherwise, you can get this permission through the Service Usage Admin role (roles/serviceusage.serviceUsageAdmin). Learn how to grant roles.

    Enable the APIs

  11. Install the Google Cloud CLI.

  12. If you're using an external identity provider (IdP), you must first sign in to the gcloud CLI with your federated identity.

  13. To initialize the gcloud CLI, run the following command: 

    gcloud init
  14. Update gcloud components: 
    gcloud components update
  15. Prepare your development environment.

    Setting up a .NET development environment

Preparing the app

  1. In your terminal window, clone the sample app repository to your local machine:

    git clone https://github.com/GoogleCloudPlatform/getting-started-dotnet.git

    Alternatively, you can download the sample as a zip file and extract it.

  2. Change to the directory that contains the background task sample code:

    cd getting-started-dotnet/BackgroundProcessing

Understanding the TranslateWorker service

  • The service starts by importing several dependencies like Firestore and Translation.

    using Google.Cloud.Firestore;
using Google.Cloud.Translation.V2;

  • The Firestore and Translation clients are initialized so they can be reused between handler invocations. That way, you don't have to initialize new clients for every invocation, which would slow down execution.

    public void ConfigureServices(IServiceCollection services)
{
    services.AddSingleton<FirestoreDb>(provider =>
        FirestoreDb.Create(GetFirestoreProjectId()));
    services.AddSingleton<TranslationClient>(
        TranslationClient.Create());
    services.AddMvc().SetCompatibilityVersion(CompatibilityVersion.Version_2_1);
}

  • The Translation API translates the string to the language you selected.

    var result = await _translator.TranslateTextAsync(sourceText, "es");

  • The controller's constructor receives the Firestore and Pub/Sub clients.

    The Post method parses the Pub/Sub message to get the text to translate. It uses the message ID as a unique name for the translation request to make sure it doesn't store any duplicate translations.

    using Google.Cloud.Firestore;
using Google.Cloud.Translation.V2;
using Microsoft.AspNetCore.Mvc;
using Microsoft.Extensions.Logging;
using System;
using System.Collections.Generic;
using System.Text;
using System.Threading.Tasks;

namespace TranslateWorker.Controllers
{
    /// <summary>
    /// The message Pubsub posts to our controller.
    /// </summary>
    public class PostMessage
    {
        public PubsubMessage message { get; set; }
        public string subscription { get; set; }
    }

    /// <summary>
    /// Pubsub's inner message.
    /// </summary>
    public class PubsubMessage
    {
        public string data { get; set; }
        public string messageId { get; set; }
        public Dictionary<string, string> attributes { get; set; }
    }


    [Route("api/[controller]")]
    [ApiController]
    public class TranslateController : ControllerBase
    {
        private readonly ILogger<TranslateController> _logger;
        private readonly FirestoreDb _firestore;
        private readonly TranslationClient _translator;
        // The Firestore collection where we store translations.
        private readonly CollectionReference _translations;

        public TranslateController(ILogger<TranslateController> logger,
            FirestoreDb firestore,
            TranslationClient translator)
        {
            _logger = logger ?? throw new ArgumentNullException(nameof(logger));
            _firestore = firestore ?? throw new ArgumentNullException(
                nameof(firestore));
            _translator = translator ?? throw new ArgumentNullException(
                nameof(translator));
            _translations = _firestore.Collection("Translations");
        }

        /// <summary>
        /// Handle a posted message from Pubsub.
        /// </summary>
        /// <param name="request">The message Pubsub posts to this process.</param>
        /// <returns>NoContent on success.</returns>
        [HttpPost]
        public async Task<IActionResult> Post([FromBody] PostMessage request)
        {
            // Unpack the message from Pubsub.
            string sourceText;
            try
            {
                byte[] data = Convert.FromBase64String(request.message.data);
                sourceText = Encoding.UTF8.GetString(data);
            }
            catch (Exception e)
            {
                _logger.LogError(1, e, "Bad request");
                return BadRequest();
            }
            // Translate the source text.
            _logger.LogDebug(2, "Translating {0} to Spanish.", sourceText);
            var result = await _translator.TranslateTextAsync(sourceText, "es");
            // Store the result in Firestore.
            Translation translation = new Translation()
            {
                TimeStamp = DateTime.UtcNow,
                SourceText = sourceText,
                TranslatedText = result.TranslatedText
            };
            _logger.LogDebug(3, "Saving translation {0} to {1}.",
                translation.TranslatedText, _translations.Path);
            await _translations.Document(request.message.messageId)
                .SetAsync(translation);
            // Return a success code.
            return NoContent();
        }

        /// <summary>
        /// Serve a root page so Cloud Run knows this process is healthy.
        /// </summary>
        [Route("/")]
        public