<?xml version="1.0" encoding="utf-8"?><feed xmlns="http://www.w3.org/2005/Atom" ><generator uri="https://jekyllrb.com/" version="3.10.0">Jekyll</generator><link href="https://pal-paul.github.io/tutorials/feed.xml" rel="self" type="application/atom+xml" /><link href="https://pal-paul.github.io/tutorials/" rel="alternate" type="text/html" /><updated>2026-05-12T07:45:07+00:00</updated><id>https://pal-paul.github.io/tutorials/feed.xml</id><title type="html">Coding with Pal-Paul</title><subtitle>Tutorials and hands-on guides on cloud infrastructure, Terraform, Go, GitHub Actions, and Google Cloud Platform.</subtitle><entry><title type="html">Detecting Firestore Changes and Triggering Cloud Run via Eventarc</title><link href="https://pal-paul.github.io/tutorials/firestore-eventarc-cloudrun/" rel="alternate" type="text/html" title="Detecting Firestore Changes and Triggering Cloud Run via Eventarc" /><published>2026-05-12T00:00:00+00:00</published><updated>2026-05-12T00:00:00+00:00</updated><id>https://pal-paul.github.io/tutorials/firestore-eventarc-cloudrun</id><content type="html" xml:base="https://pal-paul.github.io/tutorials/firestore-eventarc-cloudrun/"><![CDATA[<h1 id="detecting-firestore-changes-and-triggering-cloud-run-via-eventarc">Detecting Firestore Changes and Triggering Cloud Run via Eventarc</h1>

<p><strong>Date:</strong> May 2026</p>

<hr />

<h2 id="overview">Overview</h2>

<p>This document explains how to detect changes in Firestore data and use them to trigger a Cloud Run service using <strong>Google Cloud Eventarc</strong>.</p>

<hr />

<h2 id="architecture">Architecture</h2>

<div class="language-plaintext highlighter-rouge"><div class="highlight"><pre class="highlight"><code>Firestore document change
        │
        ▼
  Eventarc Trigger
  (written/created/updated/deleted)
        │
        ▼
  Cloud Run Service
  (receives CloudEvents HTTP POST)
</code></pre></div></div>

<p>Alternatively, if fan-out or buffering is needed (matching the existing pattern in this repo):</p>

<div class="language-plaintext highlighter-rouge"><div class="highlight"><pre class="highlight"><code>Firestore document change
        │
        ▼
  Eventarc Trigger
        │
        ▼
  Pub/Sub Topic
        │
        ▼
  Cloud Run (Push Subscription)
</code></pre></div></div>

<hr />

<h2 id="option-1-eventarc--cloud-run-recommended">Option 1: Eventarc → Cloud Run (Recommended)</h2>

<p>Eventarc natively listens to Firestore document change events and routes them directly to a Cloud Run service.</p>

<h3 id="gcloud-cli-example">gcloud CLI example</h3>

<div class="language-bash highlighter-rouge"><div class="highlight"><pre class="highlight"><code>gcloud eventarc triggers create firestore-trigger <span class="se">\</span>
  <span class="nt">--location</span><span class="o">=</span>europe-west1 <span class="se">\</span>
  <span class="nt">--destination-run-service</span><span class="o">=</span>your-cloud-run-service <span class="se">\</span>
  <span class="nt">--destination-run-region</span><span class="o">=</span>europe-west1 <span class="se">\</span>
  <span class="nt">--event-filters</span><span class="o">=</span><span class="s2">"type=google.cloud.firestore.document.v1.written"</span> <span class="se">\</span>
  <span class="nt">--event-filters</span><span class="o">=</span><span class="s2">"database=(default)"</span> <span class="se">\</span>
  <span class="nt">--event-filters-path-pattern</span><span class="o">=</span><span class="s2">"document=your-collection/{document}"</span> <span class="se">\</span>
  <span class="nt">--service-account</span><span class="o">=</span>your-sa@project.iam.gserviceaccount.com
</code></pre></div></div>

<h3 id="terraform-example">Terraform example</h3>

<div class="language-hcl highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="c1"># Grant the service account the Eventarc receiver role</span>
<span class="nx">resource</span> <span class="s2">"google_project_iam_member"</span> <span class="s2">"sa_eventarc_receiver"</span> <span class="p">{</span>
  <span class="nx">project</span> <span class="p">=</span> <span class="nx">var</span><span class="err">.</span><span class="nx">project</span>
  <span class="nx">role</span>    <span class="p">=</span> <span class="s2">"roles/eventarc.eventReceiver"</span>
  <span class="nx">member</span>  <span class="p">=</span> <span class="s2">"serviceAccount:${google_service_account.service_account.email}"</span>
<span class="p">}</span>

<span class="c1"># Grant Firestore permission to publish to Eventarc</span>
<span class="nx">resource</span> <span class="s2">"google_project_iam_member"</span> <span class="s2">"firestore_eventarc_publisher"</span> <span class="p">{</span>
  <span class="nx">project</span> <span class="p">=</span> <span class="nx">var</span><span class="err">.</span><span class="nx">project</span>
  <span class="nx">role</span>    <span class="p">=</span> <span class="s2">"roles/eventarc.publisher"</span>
  <span class="nx">member</span>  <span class="p">=</span> <span class="s2">"serviceAccount:service-${data.google_project.project.number}@gcp-sa-firestore.iam.gserviceaccount.com"</span>
<span class="p">}</span>

<span class="c1"># Eventarc trigger: Firestore document write → Cloud Run</span>
<span class="nx">resource</span> <span class="s2">"google_eventarc_trigger"</span> <span class="s2">"firestore_change_trigger"</span> <span class="p">{</span>
  <span class="nx">name</span>     <span class="p">=</span> <span class="s2">"${var.component}-firestore-trigger"</span>
  <span class="nx">project</span>  <span class="p">=</span> <span class="nx">var</span><span class="err">.</span><span class="nx">project</span>
  <span class="nx">location</span> <span class="p">=</span> <span class="nx">var</span><span class="err">.</span><span class="nx">region</span>

  <span class="nx">matching_criteria</span> <span class="p">{</span>
    <span class="nx">attribute</span> <span class="p">=</span> <span class="s2">"type"</span>
    <span class="nx">value</span>     <span class="p">=</span> <span class="s2">"google.cloud.firestore.document.v1.written"</span>
  <span class="p">}</span>
  <span class="nx">matching_criteria</span> <span class="p">{</span>
    <span class="nx">attribute</span> <span class="p">=</span> <span class="s2">"database"</span>
    <span class="nx">value</span>     <span class="p">=</span> <span class="s2">"(default)"</span>
  <span class="p">}</span>
  <span class="nx">matching_criteria</span> <span class="p">{</span>
    <span class="nx">attribute</span> <span class="p">=</span> <span class="s2">"document"</span>
    <span class="nx">value</span>     <span class="p">=</span> <span class="s2">"your-collection/{document}"</span>
    <span class="nx">operator</span>  <span class="p">=</span> <span class="s2">"match-path-pattern"</span>   <span class="c1"># required for wildcard paths</span>
  <span class="p">}</span>

  <span class="nx">destination</span> <span class="p">{</span>
    <span class="nx">cloud_run_service</span> <span class="p">{</span>
      <span class="nx">service</span> <span class="p">=</span> <span class="nx">google_cloud_run_v2_service</span><span class="err">.</span><span class="nx">service_queuer</span><span class="err">.</span><span class="nx">name</span>
      <span class="nx">region</span>  <span class="p">=</span> <span class="nx">var</span><span class="err">.</span><span class="nx">region</span>
      <span class="nx">path</span>    <span class="p">=</span> <span class="s2">"/firestore-event"</span>     <span class="c1"># endpoint on your Cloud Run service</span>
    <span class="p">}</span>
  <span class="p">}</span>

  <span class="nx">service_account</span> <span class="p">=</span> <span class="nx">google_service_account</span><span class="err">.</span><span class="nx">service_account</span><span class="err">.</span><span class="nx">email</span>

  <span class="nx">labels</span> <span class="p">=</span> <span class="p">{</span>
    <span class="nx">env</span> <span class="p">=</span> <span class="nx">var</span><span class="err">.</span><span class="nx">env</span>
  <span class="p">}</span>
<span class="p">}</span>
</code></pre></div></div>

<hr />

<h2 id="option-2-eventarc--pubsub--cloud-run">Option 2: Eventarc → Pub/Sub → Cloud Run</h2>

<p>Use this when you need fan-out (multiple consumers) or want to reuse the existing Pub/Sub subscriber pattern already present in this repo.</p>

<div class="language-hcl highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="c1"># Eventarc trigger routes Firestore events to a Pub/Sub topic</span>
<span class="nx">resource</span> <span class="s2">"google_eventarc_trigger"</span> <span class="s2">"firestore_pubsub_trigger"</span> <span class="p">{</span>
  <span class="nx">name</span>     <span class="p">=</span> <span class="s2">"${var.component}-firestore-pubsub-trigger"</span>
  <span class="nx">project</span>  <span class="p">=</span> <span class="nx">var</span><span class="err">.</span><span class="nx">project</span>
  <span class="nx">location</span> <span class="p">=</span> <span class="nx">var</span><span class="err">.</span><span class="nx">region</span>

  <span class="nx">matching_criteria</span> <span class="p">{</span>
    <span class="nx">attribute</span> <span class="p">=</span> <span class="s2">"type"</span>
    <span class="nx">value</span>     <span class="p">=</span> <span class="s2">"google.cloud.firestore.document.v1.written"</span>
  <span class="p">}</span>
  <span class="nx">matching_criteria</span> <span class="p">{</span>
    <span class="nx">attribute</span> <span class="p">=</span> <span class="s2">"database"</span>
    <span class="nx">value</span>     <span class="p">=</span> <span class="s2">"(default)"</span>
  <span class="p">}</span>
  <span class="nx">matching_criteria</span> <span class="p">{</span>
    <span class="nx">attribute</span> <span class="p">=</span> <span class="s2">"document"</span>
    <span class="nx">value</span>     <span class="p">=</span> <span class="s2">"your-collection/{document}"</span>
    <span class="nx">operator</span>  <span class="p">=</span> <span class="s2">"match-path-pattern"</span>
  <span class="p">}</span>

  <span class="nx">destination</span> <span class="p">{</span>
    <span class="nx">cloud_run_service</span> <span class="p">{</span>
      <span class="nx">service</span> <span class="p">=</span> <span class="nx">google_cloud_run_v2_service</span><span class="err">.</span><span class="nx">service_queuer</span><span class="err">.</span><span class="nx">name</span>
      <span class="nx">region</span>  <span class="p">=</span> <span class="nx">var</span><span class="err">.</span><span class="nx">region</span>
      <span class="nx">path</span>    <span class="p">=</span> <span class="s2">"/enqueue"</span>
    <span class="p">}</span>
  <span class="p">}</span>

  <span class="nx">service_account</span> <span class="p">=</span> <span class="nx">google_service_account</span><span class="err">.</span><span class="nx">service_account</span><span class="err">.</span><span class="nx">email</span>
<span class="p">}</span>
</code></pre></div></div>

<hr />

<h2 id="firestore-event-types">Firestore Event Types</h2>

<table>
  <thead>
    <tr>
      <th>Event Type</th>
      <th>Fires When</th>
    </tr>
  </thead>
  <tbody>
    <tr>
      <td><code class="language-plaintext highlighter-rouge">google.cloud.firestore.document.v1.written</code></td>
      <td>Document created, updated, <strong>or</strong> deleted</td>
    </tr>
    <tr>
      <td><code class="language-plaintext highlighter-rouge">google.cloud.firestore.document.v1.created</code></td>
      <td>New document created only</td>
    </tr>
    <tr>
      <td><code class="language-plaintext highlighter-rouge">google.cloud.firestore.document.v1.updated</code></td>
      <td>Existing document changed</td>
    </tr>
    <tr>
      <td><code class="language-plaintext highlighter-rouge">google.cloud.firestore.document.v1.deleted</code></td>
      <td>Document deleted</td>
    </tr>
  </tbody>
</table>

<hr />

<h2 id="cloudevents-payload">CloudEvents Payload</h2>

<p>Your Cloud Run service receives an HTTP POST with a <code class="language-plaintext highlighter-rouge">Content-Type: application/cloudevents+json</code> header. The body:</p>

<div class="language-json highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="p">{</span><span class="w">
  </span><span class="nl">"specversion"</span><span class="p">:</span><span class="w"> </span><span class="s2">"1.0"</span><span class="p">,</span><span class="w">
  </span><span class="nl">"type"</span><span class="p">:</span><span class="w"> </span><span class="s2">"google.cloud.firestore.document.v1.written"</span><span class="p">,</span><span class="w">
  </span><span class="nl">"source"</span><span class="p">:</span><span class="w"> </span><span class="s2">"//firestore.googleapis.com/projects/my-project/databases/(default)"</span><span class="p">,</span><span class="w">
  </span><span class="nl">"subject"</span><span class="p">:</span><span class="w"> </span><span class="s2">"documents/your-collection/doc-id"</span><span class="p">,</span><span class="w">
  </span><span class="nl">"id"</span><span class="p">:</span><span class="w"> </span><span class="s2">"unique-event-id"</span><span class="p">,</span><span class="w">
  </span><span class="nl">"time"</span><span class="p">:</span><span class="w"> </span><span class="s2">"2026-05-12T10:00:00Z"</span><span class="p">,</span><span class="w">
  </span><span class="nl">"datacontenttype"</span><span class="p">:</span><span class="w"> </span><span class="s2">"application/json"</span><span class="p">,</span><span class="w">
  </span><span class="nl">"data"</span><span class="p">:</span><span class="w"> </span><span class="p">{</span><span class="w">
    </span><span class="nl">"oldValue"</span><span class="p">:</span><span class="w"> </span><span class="p">{</span><span class="w">
      </span><span class="nl">"name"</span><span class="p">:</span><span class="w"> </span><span class="s2">"projects/my-project/databases/(default)/documents/your-collection/doc-id"</span><span class="p">,</span><span class="w">
      </span><span class="nl">"fields"</span><span class="p">:</span><span class="w"> </span><span class="p">{</span><span class="w"> </span><span class="nl">"status"</span><span class="p">:</span><span class="w"> </span><span class="p">{</span><span class="w"> </span><span class="nl">"stringValue"</span><span class="p">:</span><span class="w"> </span><span class="s2">"pending"</span><span class="w"> </span><span class="p">}</span><span class="w"> </span><span class="p">}</span><span class="w">
    </span><span class="p">},</span><span class="w">
    </span><span class="nl">"value"</span><span class="p">:</span><span class="w"> </span><span class="p">{</span><span class="w">
      </span><span class="nl">"name"</span><span class="p">:</span><span class="w"> </span><span class="s2">"projects/my-project/databases/(default)/documents/your-collection/doc-id"</span><span class="p">,</span><span class="w">
      </span><span class="nl">"fields"</span><span class="p">:</span><span class="w"> </span><span class="p">{</span><span class="w"> </span><span class="nl">"status"</span><span class="p">:</span><span class="w"> </span><span class="p">{</span><span class="w"> </span><span class="nl">"stringValue"</span><span class="p">:</span><span class="w"> </span><span class="s2">"processed"</span><span class="w"> </span><span class="p">}</span><span class="w"> </span><span class="p">}</span><span class="w">
    </span><span class="p">},</span><span class="w">
    </span><span class="nl">"updateMask"</span><span class="p">:</span><span class="w"> </span><span class="p">{</span><span class="w">
      </span><span class="nl">"fieldPaths"</span><span class="p">:</span><span class="w"> </span><span class="p">[</span><span class="s2">"status"</span><span class="p">]</span><span class="w">
    </span><span class="p">}</span><span class="w">
  </span><span class="p">}</span><span class="w">
</span><span class="p">}</span><span class="w">
</span></code></pre></div></div>

<ul>
  <li><code class="language-plaintext highlighter-rouge">oldValue</code> — state before the change (<code class="language-plaintext highlighter-rouge">null</code> on create)</li>
  <li><code class="language-plaintext highlighter-rouge">value</code> — state after the change (<code class="language-plaintext highlighter-rouge">null</code> on delete)</li>
  <li><code class="language-plaintext highlighter-rouge">updateMask.fieldPaths</code> — list of fields that changed (only present on update)</li>
</ul>

<hr />

<h2 id="go-handler-example-cloud-run">Go Handler Example (Cloud Run)</h2>

<div class="language-go highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="k">package</span> <span class="n">main</span>

<span class="k">import</span> <span class="p">(</span>
    <span class="s">"encoding/json"</span>
    <span class="s">"io"</span>
    <span class="s">"log"</span>
    <span class="s">"net/http"</span>
<span class="p">)</span>

<span class="k">type</span> <span class="n">FirestoreEvent</span> <span class="k">struct</span> <span class="p">{</span>
    <span class="n">OldValue</span>   <span class="k">map</span><span class="p">[</span><span class="kt">string</span><span class="p">]</span><span class="k">interface</span><span class="p">{}</span> <span class="s">`json:"oldValue"`</span>
    <span class="n">Value</span>      <span class="k">map</span><span class="p">[</span><span class="kt">string</span><span class="p">]</span><span class="k">interface</span><span class="p">{}</span> <span class="s">`json:"value"`</span>
    <span class="n">UpdateMask</span> <span class="k">struct</span> <span class="p">{</span>
        <span class="n">FieldPaths</span> <span class="p">[]</span><span class="kt">string</span> <span class="s">`json:"fieldPaths"`</span>
    <span class="p">}</span> <span class="s">`json:"updateMask"`</span>
<span class="p">}</span>

<span class="k">func</span> <span class="n">firestoreEventHandler</span><span class="p">(</span><span class="n">w</span> <span class="n">http</span><span class="o">.</span><span class="n">ResponseWriter</span><span class="p">,</span> <span class="n">r</span> <span class="o">*</span><span class="n">http</span><span class="o">.</span><span class="n">Request</span><span class="p">)</span> <span class="p">{</span>
    <span class="n">body</span><span class="p">,</span> <span class="n">err</span> <span class="o">:=</span> <span class="n">io</span><span class="o">.</span><span class="n">ReadAll</span><span class="p">(</span><span class="n">r</span><span class="o">.</span><span class="n">Body</span><span class="p">)</span>
    <span class="k">if</span> <span class="n">err</span> <span class="o">!=</span> <span class="no">nil</span> <span class="p">{</span>
        <span class="n">http</span><span class="o">.</span><span class="n">Error</span><span class="p">(</span><span class="n">w</span><span class="p">,</span> <span class="s">"failed to read body"</span><span class="p">,</span> <span class="n">http</span><span class="o">.</span><span class="n">StatusBadRequest</span><span class="p">)</span>
        <span class="k">return</span>
    <span class="p">}</span>

    <span class="c">// CloudEvents wraps the Firestore payload in the "data" field</span>
    <span class="k">var</span> <span class="n">envelope</span> <span class="k">struct</span> <span class="p">{</span>
        <span class="n">Data</span> <span class="n">FirestoreEvent</span> <span class="s">`json:"data"`</span>
    <span class="p">}</span>
    <span class="k">if</span> <span class="n">err</span> <span class="o">:=</span> <span class="n">json</span><span class="o">.</span><span class="n">Unmarshal</span><span class="p">(</span><span class="n">body</span><span class="p">,</span> <span class="o">&amp;</span><span class="n">envelope</span><span class="p">);</span> <span class="n">err</span> <span class="o">!=</span> <span class="no">nil</span> <span class="p">{</span>
        <span class="n">http</span><span class="o">.</span><span class="n">Error</span><span class="p">(</span><span class="n">w</span><span class="p">,</span> <span class="s">"failed to parse event"</span><span class="p">,</span> <span class="n">http</span><span class="o">.</span><span class="n">StatusBadRequest</span><span class="p">)</span>
        <span class="k">return</span>
    <span class="p">}</span>

    <span class="n">event</span> <span class="o">:=</span> <span class="n">envelope</span><span class="o">.</span><span class="n">Data</span>
    <span class="n">log</span><span class="o">.</span><span class="n">Printf</span><span class="p">(</span><span class="s">"changed fields: %v"</span><span class="p">,</span> <span class="n">event</span><span class="o">.</span><span class="n">UpdateMask</span><span class="o">.</span><span class="n">FieldPaths</span><span class="p">)</span>
    <span class="n">log</span><span class="o">.</span><span class="n">Printf</span><span class="p">(</span><span class="s">"new value: %v"</span><span class="p">,</span> <span class="n">event</span><span class="o">.</span><span class="n">Value</span><span class="p">)</span>

    <span class="c">// Your business logic here</span>

    <span class="n">w</span><span class="o">.</span><span class="n">WriteHeader</span><span class="p">(</span><span class="n">http</span><span class="o">.</span><span class="n">StatusOK</span><span class="p">)</span>
<span class="p">}</span>
</code></pre></div></div>

<hr />

<h2 id="required-iam-roles">Required IAM Roles</h2>

<table>
  <thead>
    <tr>
      <th>Principal</th>
      <th>Role</th>
      <th>Purpose</th>
    </tr>
  </thead>
  <tbody>
    <tr>
      <td>Cloud Run service account</td>
      <td><code class="language-plaintext highlighter-rouge">roles/eventarc.eventReceiver</code></td>
      <td>Receive Eventarc events</td>
    </tr>
    <tr>
      <td>Firestore service account (<code class="language-plaintext highlighter-rouge">gcp-sa-firestore.iam.gserviceaccount.com</code>)</td>
      <td><code class="language-plaintext highlighter-rouge">roles/eventarc.publisher</code></td>
      <td>Publish change events to Eventarc</td>
    </tr>
    <tr>
      <td>Cloud Run service account</td>
      <td><code class="language-plaintext highlighter-rouge">roles/run.invoker</code></td>
      <td>Allow Eventarc to invoke the Cloud Run service</td>
    </tr>
  </tbody>
</table>

<hr />

<h2 id="prerequisites">Prerequisites</h2>

<ul>
  <li>Google provider <code class="language-plaintext highlighter-rouge">&gt;= 4.50</code> for <code class="language-plaintext highlighter-rouge">google_eventarc_trigger</code> with Firestore support</li>
  <li>Firestore must be in <strong>Native mode</strong> (not Datastore mode)</li>
  <li>Eventarc API must be enabled: <code class="language-plaintext highlighter-rouge">gcloud services enable eventarc.googleapis.com</code></li>
</ul>

<hr />

<h2 id="references">References</h2>

<ul>
  <li><a href="https://registry.terraform.io/providers/hashicorp/google/latest/docs/resources/eventarc_trigger">Eventarc Terraform resource</a></li>
  <li><a href="https://cloud.google.com/eventarc/docs/reference/supported-events#cloud-firestore">Firestore Eventarc event types</a></li>
  <li><a href="https://cloudevents.io/">CloudEvents spec</a></li>
</ul>]]></content><author><name></name></author><category term="Terraform" /><summary type="html"><![CDATA[Detecting Firestore Changes and Triggering Cloud Run via Eventarc]]></summary></entry><entry><title type="html">GitHub Action for authenticating to Google Cloud with GitHub Actions OIDC tokens and Workload Identity Federation</title><link href="https://pal-paul.github.io/tutorials/GitHub-Action-for-authenticating-to-Google-Cloud-with-GitHub-Actions-OIDC-tokens-and-workload-identity-federation/" rel="alternate" type="text/html" title="GitHub Action for authenticating to Google Cloud with GitHub Actions OIDC tokens and Workload Identity Federation" /><published>2022-01-10T11:44:45+00:00</published><updated>2022-01-10T11:44:45+00:00</updated><id>https://pal-paul.github.io/tutorials/GitHub-Action-for-authenticating-to-Google-Cloud-with-GitHub-Actions-OIDC-tokens-and-workload-identity-federation</id><content type="html" xml:base="https://pal-paul.github.io/tutorials/GitHub-Action-for-authenticating-to-Google-Cloud-with-GitHub-Actions-OIDC-tokens-and-workload-identity-federation/"><![CDATA[<h2 id="auth">Auth</h2>
<p>GitHub Action exchanges a GitHub Actions OIDC token into a Google Cloud access token using <a href="https://cloud.google.com/iam/docs/workload-identity-federation">Workload Identity Federation</a>. This obviates the need to export a long-lived Google Cloud service account key and establishes a trust delegation relationship between a particular GitHub Actions workflow invocation and permissions on Google Cloud</p>

<h3 id="previously">Previously</h3>
<ul>
  <li>Create a Google Cloud service account and grant IAM permissions</li>
  <li>Export the long-lived JSON service account key</li>
  <li>Upload the JSON service account key to a GitHub secret</li>
</ul>

<h3 id="with-workload-identity-federation">With Workload Identity Federation</h3>
<ul>
  <li>Create a Google Cloud service account and grant IAM permissions</li>
  <li>Create and configure a Workload Identity Provider for GitHub</li>
  <li>Exchange the GitHub Actions OIDC token for a short-lived Google Cloud access token</li>
</ul>

<h2 id="prerequisites">Prerequisites</h2>
<p>This action requires you to create and configure a Google Cloud Workload Identity Provider</p>

<h2 id="setup-workload-identity-provider">Setup Workload Identity Provider</h2>

<p>To exchange a GitHub Actions OIDC token for a Google Cloud access token, you must create and configure a Workload Identity Provider.</p>

<p>Here terraform example how to create/configure Workload Identity Provider for Github</p>

<div class="language-plaintext highlighter-rouge"><div class="highlight"><pre class="highlight"><code>data "google_project" "project" {
  project_id = var.project
}

resource "google_iam_workload_identity_pool" "ugc_wip_gh_pool" {
  project                   = var.project
  provider                  = google-beta
  workload_identity_pool_id = "wpi-github-pool"
}

resource "google_iam_workload_identity_pool_provider" "ugc_wip_provider" {
  provider                           = google-beta
  project                            = var.project
  workload_identity_pool_id          = google_iam_workload_identity_pool.ugc_wip_gh_pool.workload_identity_pool_id
  workload_identity_pool_provider_id = "wpi-github-provider"
  attribute_mapping = {
    "google.subject"       = "assertion.sub"
    "attribute.repository" = "assertion.repository"
  }
  oidc {
    allowed_audiences = [var.wip_allowed_audiences]
    issuer_uri        = "https://token.actions.githubusercontent.com"
  }
}
</code></pre></div></div>
<p>See <a href="https://registry.terraform.io/providers/hashicorp/google/latest/docs/resources/iam_workload_identity_pool_provider">documentation</a></p>

<p>project is your google project id
The audience is the key that gcp allows to connect from Github</p>

<p>The attribute mappings map claims in the GitHub Actions JWT to assertions you can make about the request (like the repository or GitHub username of the principal invoking the GitHub Action). These can be used to further restrict the authentication using attribute.repository flags.</p>

<p>For example, you can map the attribute repository values (which can be used later to restrict the authentication to specific repositories):</p>

<div class="language-plaintext highlighter-rouge"><div class="highlight"><pre class="highlight"><code>"google.subject"       = "assertion.sub"
"attribute.repository" = "assertion.repository"
</code></pre></div></div>

<p>The full ID for the Workload Identity Provider will be like below format:</p>

<div class="language-plaintext highlighter-rouge"><div class="highlight"><pre class="highlight"><code>projects/123456789/locations/global/workloadIdentityPools/&lt;${var.env}-wpi-github-pool&gt;/providers/&lt;${var.env}-wpi-github-provider&gt;
</code></pre></div></div>

<h2 id="setup-workload-identity-provider-to-impersonate-the-service-account">Setup Workload Identity Provider to impersonate the Service Account</h2>

<p>Allow authentications from the Workload Identity Provider to impersonate the Service Account created above
<em>Warning</em>: This grants access to any resource in the pool (all GitHub repos). It’s strongly recommended that you map to a specific attribute such as the owner/organization or repository name instead.</p>

<p>See mapping <a href="https://cloud.google.com/iam/docs/configuring-workload-identity-federation#oidc">external identities</a> for more information.</p>

<p>Here terraform example how to create/configure service account for the Github action</p>

<div class="language-plaintext highlighter-rouge"><div class="highlight"><pre class="highlight"><code># Create service account for ugc-support service
module "service_accounts_ugc_support" {
  source        = "terraform-google-modules/service-accounts/google"
  version       = "~&gt; 3.0"
  project_id    = var.project
  prefix        = var.env
  names         = ["tf-ugc-support"]
  generate_keys = false
  project_roles = [
    "${var.project}=&gt;roles/iam.serviceAccountTokenCreator",
    "${var.project}=&gt;roles/secretmanager.secretAccessor",
    "${var.project}=&gt;roles/secretmanager.viewer",
  ]
}

data "google_iam_policy" "ugc_support_wlif_gh_policy" {
  binding {
    role = "roles/iam.workloadIdentityUser"
    members = [
      "principalSet://iam.googleapis.com/projects/${data.google_project.project.number}/locations/global/workloadIdentityPools/${google_iam_workload_identity_pool.ugc_wip_gh_pool.workload_identity_pool_id}/attribute.repository/${var.github_owner}/${var.repo}",
    ]
  }
}
</code></pre></div></div>

<ul>
  <li><em>roles/iam.serviceAccountTokenCreator</em> role is must to create token to validate</li>
  <li><em>roles/iam.workloadIdentityUser</em> role is must to use as workload identity user</li>
  <li><em>policy</em> should add to connect from specific Github owner/organization or repository name</li>
</ul>

<p>The full ID of pilicy should look like</p>
<div class="language-plaintext highlighter-rouge"><div class="highlight"><pre class="highlight"><code>"principalSet://iam.googleapis.com/projects/123456789/locations/global/workloadIdentityPools/${var.env}-wpi-github-pool/attribute.repository/${var.github_owner}/${var.repo}"
</code></pre></div></div>

<h2 id="github-action-using-access-token-oauth-20">Github Action using Access Token (OAuth 2.0)</h2>
<p>Use this GitHub Action with the Workload Identity Provider ID and Service Account email. The GitHub Action will mint a GitHub OIDC token and exchange the GitHub token for a Google Cloud access token (assuming the authorization is correct). This all happens without exporting a Google Cloud service account key JSON!</p>

<div class="language-plaintext highlighter-rouge"><div class="highlight"><pre class="highlight"><code>name: Deploy - Dev
on:
  push:
    branches:
      - master

concurrency:
  group: dev-deploy-$

jobs:
  build-and-deploy:
    runs-on: ubuntu-latest
    permissions:
      contents: 'read'
      id-token: 'write'
    steps:
      - name: Checkout repo
        uses: actions/checkout@v2
        with:
          fetch-depth: 0

      - name: Authenticate to Google Cloud
        uses: 'google-github-actions/auth@v0.4.4'
        with:
          token_format: 'access_token'
          workload_identity_provider: $
          service_account: $
          audience: $ 

      - name: Build and deploy to dev
        run: /bin/bash ./build/package/build.bash
</code></pre></div></div>

<ul>
  <li>DEV_WORKLOAD_IDENTITY_PROVIDER=projects/123456789/locations/global/workloadIdentityPools/wpi-github-pool/providers/wpi-github-provider</li>
  <li>DEV_TF_GCP_SERVICE_ACCOUNT=dev-tf-ugc-support@<your-project-id>.iam.gserviceaccount.com</your-project-id></li>
  <li>DEV_AUDIENCES=var.wip_allowed_audiences</li>
</ul>

<p>See <a href="https://github.com/google-github-actions/auth">documentation</a> and more <a href="https://github.com/google-github-actions/auth#examples">example</a> of google-github-actions/auth</p>]]></content><author><name></name></author><category term="Terraform" /><summary type="html"><![CDATA[Auth GitHub Action exchanges a GitHub Actions OIDC token into a Google Cloud access token using Workload Identity Federation. This obviates the need to export a long-lived Google Cloud service account key and establishes a trust delegation relationship between a particular GitHub Actions workflow invocation and permissions on Google Cloud]]></summary></entry><entry><title type="html">How to print Github secret in Action</title><link href="https://pal-paul.github.io/tutorials/how-to-print-github-secret/" rel="alternate" type="text/html" title="How to print Github secret in Action" /><published>2021-07-16T10:12:12+00:00</published><updated>2021-07-16T10:12:12+00:00</updated><id>https://pal-paul.github.io/tutorials/how-to-print-github-secret</id><content type="html" xml:base="https://pal-paul.github.io/tutorials/how-to-print-github-secret/"><![CDATA[<p>In some scenario we may need to print or debug Github action secret, but the problem is that we can’t debug secrets. If we try to debug print or secrets we’ll get <code class="language-plaintext highlighter-rouge">***</code> in the log.</p>

<div class="language-plaintext highlighter-rouge"><div class="highlight"><pre class="highlight"><code>- name: print-secret
  run: echo $
</code></pre></div></div>

<p>From the above the <code class="language-plaintext highlighter-rouge">echo</code> command, the Github action secret will show as <code class="language-plaintext highlighter-rouge">***</code> in the log.</p>

<div class="language-plaintext highlighter-rouge"><div class="highlight"><pre class="highlight"><code>Run echo ***
</code></pre></div></div>

<p>This is makes sense because Github is trying to help us to keep the secrets secret as the action log is visible for all having access to the repository.</p>

<p>There is a way to show this secret if we really want to print or debug and show it log to verify.</p>

<p>We can separate the characters with a space using the following code. The secret will visible in the action log.</p>

<div class="language-plaintext highlighter-rouge"><div class="highlight"><pre class="highlight"><code>- name: print-secret
  run: echo $ | sed 's/./&amp; /g'
</code></pre></div></div>

<p>From the above the <code class="language-plaintext highlighter-rouge">echo</code> command, the Github action secret will show as actual secret in the log but seperated with space in each char.</p>

<div class="language-plaintext highlighter-rouge"><div class="highlight"><pre class="highlight"><code>Run echo *** | sed 's/./&amp; /g'
t e s t
</code></pre></div></div>

<p><em>Note: Its recommended that we should not print any secret in the github action log, once we debug or verified the secret in log we should remove the print option before merge.</em></p>]]></content><author><name></name></author><category term="Github" /><category term="Action" /><summary type="html"><![CDATA[In some scenario we may need to print or debug Github action secret, but the problem is that we can’t debug secrets. If we try to debug print or secrets we’ll get *** in the log.]]></summary></entry><entry><title type="html">Reusable infrastructure with Terraform modules</title><link href="https://pal-paul.github.io/tutorials/reusable-infrastructure-with-terraform-modules/" rel="alternate" type="text/html" title="Reusable infrastructure with Terraform modules" /><published>2021-06-29T10:12:12+00:00</published><updated>2021-06-29T10:12:12+00:00</updated><id>https://pal-paul.github.io/tutorials/reusable-infrastructure-with-terraform-modules</id><content type="html" xml:base="https://pal-paul.github.io/tutorials/reusable-infrastructure-with-terraform-modules/"><![CDATA[<h1 id="terraform-module">Terraform module</h1>

<p>A Terraform module is simple set of Terraform configuration files in a folder is called a module.
Modules are containers for multiple resources that are used together. A module consists of a collection of terraform files .tf
and/or .tf.json files kept together in a directory.</p>

<p>Modules are the main way to package and reuse resource configurations with Terraform.</p>

<p><a href="https://www.terraform.io/docs/language/modules/index.html">More Information</a></p>

<h2 id="how-to-define-module">How to define module?</h2>

<p>A module can be define using below syntax:</p>

<div class="language-plaintext highlighter-rouge"><div class="highlight"><pre class="highlight"><code>module "NAME" {
  source = "SOURCE"

  [CONFIG ...]
}
</code></pre></div></div>

<p>Within the module definition, the source parameter specifies the folder where the module’s code can be found.
For example, you can create a new file in <code class="language-plaintext highlighter-rouge">services/compute_instance/main.tf</code> and use the <code class="language-plaintext highlighter-rouge">compute_instance</code> module in it as follows:</p>

<div class="language-plaintext highlighter-rouge"><div class="highlight"><pre class="highlight"><code>provider "google" {
    credentials = file("gcp-account.json")
    project     = "gcp-terraform-307119"
    region      = "europe-west4"
    zone        = "europe-west4-a"
}

module "compute_instance" {
  source = "modules/services/compute_instance"
}
</code></pre></div></div>

<p>Now <code class="language-plaintext highlighter-rouge">compute_instance</code> module can be reuse the exact same way in the different environment by creating main.tf for each environment.</p>

<p>Every module should have its own <code class="language-plaintext highlighter-rouge">variables.tf</code> to use it inside it and it can be access out side of the module by <code class="language-plaintext highlighter-rouge">module.&lt;MODULE_NAME&gt;.&lt;OUTPUT_NAME&gt;</code></p>

<p>Lets over an example by creating a module <code class="language-plaintext highlighter-rouge">pubsub-topic</code></p>

<p>with files <code class="language-plaintext highlighter-rouge">main.tf</code> , <code class="language-plaintext highlighter-rouge">variables.tf</code> and <code class="language-plaintext highlighter-rouge">outputs.tf</code></p>

<p>main.tf will be as follows:</p>

<div class="language-plaintext highlighter-rouge"><div class="highlight"><pre class="highlight"><code>resource "google_pubsub_topic" "topic" {
  name = var.name
  labels = {
    env       = var.env
    component = var.component
  }
}
</code></pre></div></div>

<p>outputs.tf will be as follows:</p>

<div class="language-plaintext highlighter-rouge"><div class="highlight"><pre class="highlight"><code>variable "name" {
}
variable "env" {
}
variable "component" {
}
</code></pre></div></div>

<p>variables.tf will be as follows:</p>

<div class="language-plaintext highlighter-rouge"><div class="highlight"><pre class="highlight"><code>output "id" {
  value = google_pubsub_topic.topic.id
}
</code></pre></div></div>]]></content><author><name></name></author><category term="Terraform" /><summary type="html"><![CDATA[Terraform module]]></summary></entry><entry><title type="html">Golang Map</title><link href="https://pal-paul.github.io/tutorials/go-map/" rel="alternate" type="text/html" title="Golang Map" /><published>2021-06-23T11:44:45+00:00</published><updated>2021-06-23T11:44:45+00:00</updated><id>https://pal-paul.github.io/tutorials/go-map</id><content type="html" xml:base="https://pal-paul.github.io/tutorials/go-map/"><![CDATA[<h1 id="map-in-golang">map in Golang</h1>

<h2 id="what-is-map-in-golang">What is map in Golang?</h2>

<p>map is not a data type, map keyword to create a map with keys of a certain type and values of another type or the same type similar like hashes or dicts in other languages</p>

<p>To create an empty map, use the builtin make:</p>

<div class="language-plaintext highlighter-rouge"><div class="highlight"><pre class="highlight"><code>make(map[key-type]val-type)
</code></pre></div></div>

<p>Set key/value pairs using typical <code class="language-plaintext highlighter-rouge">name[key] = val</code></p>

<p>Get a value for a key with <code class="language-plaintext highlighter-rouge">name[key]</code>.</p>

<p>The <code class="language-plaintext highlighter-rouge">len</code> returns the number of key/value pairs when called on a map like <code class="language-plaintext highlighter-rouge">len(m)</code></p>

<h3 id="example-1">Example: 1</h3>

<div class="language-plaintext highlighter-rouge"><div class="highlight"><pre class="highlight"><code>package main

import (
    "fmt"
)

func main() {

    m := make(map[string]int)

    m["key1"] = 10
    m["key2"] = 15

    fmt.Println("map:", m)

    value1 := m["key1"]
    fmt.Println("value1: ", value1)

    fmt.Println("len:", len(m))

    delete(m, "key2")
    fmt.Println("map:", m)

    _, ok := m["key2"]
    fmt.Println("is key found:", ok)

    n := map[string]int{"foo": 1, "bar": 2}
    fmt.Println("map:", n)
}
</code></pre></div></div>

<p><a href="https://play.golang.org/p/SL9jpqt5LAU">go play</a></p>

<p><em>Note: The optional second return value when getting a value from a map indicates if the key was present in the map or not.</em></p>

<h3 id="example-2">Example: 2</h3>

<p>struct can be used as a value type in map</p>

<div class="language-plaintext highlighter-rouge"><div class="highlight"><pre class="highlight"><code>package main

import (
	"encoding/json"
	"fmt"
	"log"
)

type Option struct {
	Value          string `json:"value"`
	Order          int    `json:"order"`
	TranslationKey string `json:"translationKey"`
}
type Type struct {
	Options []Option `json:"options"`
}

type MapOptionType map[string]Type

func main() {
	data := []byte(`{
        "input": {
            "options": [
            {
                "value": "",
                "order": 1,
                "translationKey": null
            }
            ]
        },
        "yesno": {
            "options": [
            {
                "value": "yes",
                "order": 1,
                "translationKey": "YES",
                "preselected": true
            },
            {
                "value": "no",
                "order": 2,
                "translationKey": "NO",
                "preselected": false
            }
            ]
        }
    }`)

	var objmap MapOptionType
	err := json.Unmarshal(data, &amp;objmap)
	if err != nil {
		log.Fatal(err)
	}

	fmt.Printf("%#v", objmap["input"])

}

</code></pre></div></div>

<p><a href="https://play.golang.org/p/Nm05_cQgowM">go play</a></p>

<h3 id="example-3">Example: 3</h3>

<p>interface can be used as a value type in map like <code class="language-plaintext highlighter-rouge">map[string]interface{}</code></p>

<div class="language-plaintext highlighter-rouge"><div class="highlight"><pre class="highlight"><code>type Person struct {
    Name    string
    Age     int
    Hobbies []string
}

person:=map[string]interface{
   "name":"Johana",
   "age":30,
   "hobbies":[
      "arts",
      "foods",
      "piano"
   ]
}

</code></pre></div></div>

<p>When to use <code class="language-plaintext highlighter-rouge">map[string]interface{}</code>?</p>

<p>The <em>map of string to empty interface</em> type is very useful when we need to deal with JSON data of an unknown schema.</p>]]></content><author><name></name></author><category term="Go" /><summary type="html"><![CDATA[map in Golang]]></summary></entry><entry><title type="html">Terraform Basic</title><link href="https://pal-paul.github.io/tutorials/terraform-basic/" rel="alternate" type="text/html" title="Terraform Basic" /><published>2021-06-22T11:44:45+00:00</published><updated>2021-06-22T11:44:45+00:00</updated><id>https://pal-paul.github.io/tutorials/terraform-basic</id><content type="html" xml:base="https://pal-paul.github.io/tutorials/terraform-basic/"><![CDATA[<h2 id="what-is-terraform">What is Terraform?</h2>

<p>Terraform is an open source infrastructure as code (IaC) software tool that allows DevOps engineers to programmatically provision the physical resources an application requires to run.</p>

<h3 id="pre-requisite">Pre-requisite</h3>

<p>Before we start working with Terraform, here are the pre-requisites -</p>

<ul>
  <li>You must install terraform <a href="https://learn.hashicorp.com/tutorials/terraform/install-cli">click here on how to install terraform</a></li>
  <li>You must have either AWS or <a href="https://cloud.google.com/">Google Cloud</a> service account key</li>
</ul>

<h3 id="prepare-terraform-file">Prepare terraform file</h3>

<p>Now we are going to write our first Terrafrom script.
Create a file named <code class="language-plaintext highlighter-rouge">main.tf</code></p>

<h3 id="add-provider-details">Add provider details</h3>

<p>we are going to provision the VMInstance on Google, so we need to select the provider as google.</p>

<div class="language-plaintext highlighter-rouge"><div class="highlight"><pre class="highlight"><code>provider "google" {
     credentials = file("gcp-account.json")
     project     = "gcp-terraform-307119"
     region      = "europe-west4"
     zone        = "europe-west4-a"
}

</code></pre></div></div>

<p><strong>gcp-account.json</strong> - is the JSON key file of a sevice account that we going to use to provision the VMInstance on Google</p>

<p><strong>project</strong> - is the google project id</p>

<p><strong>region &amp;&amp; zone</strong> - is the region and zone which you need to mention to deploy the VMInstance</p>

<h3 id="add-resource">Add resource</h3>

<p>Since we aim to create a VMInstance on Google Cloud Platform, so we need to add <code class="language-plaintext highlighter-rouge">google_compute_instance</code> configuration.</p>

<div class="language-plaintext highlighter-rouge"><div class="highlight"><pre class="highlight"><code>resource "google_compute_instance" "default" {
  name         = var.instance_name
  machine_type = var.instance_type

  boot_disk {
    initialize_params {
      image = "debian-cloud/debian-9"
    }
  }

  network_interface {
    network = "default"

    access_config {
      // Ephemeral IP
    }
  }
}

</code></pre></div></div>

<p><strong>name</strong> - You can keep the name of your virtual instance as per your choice</p>

<p><strong>machine_type</strong> - I have chosen e2-micro but you can choose from - e2-small, e2-medium, e2-standard-2 etc.</p>

<p><strong>boot_disk</strong> - Here you need to mention the host OS and I have opted for - debian-cloud/debian-9</p>

<p><strong>network_interface</strong> - This configuration is needed for getting the IP address of the virtual machine.</p>

<p>Now create <code class="language-plaintext highlighter-rouge">variable.tf</code> but without any default values</p>

<div class="language-plaintext highlighter-rouge"><div class="highlight"><pre class="highlight"><code>variable "instance_type" {
}
variable "instance_name" {
}
</code></pre></div></div>

<p>There can be situation where you need create multiple tfvars files based on the environment like stage, production.</p>

<p>create your <code class="language-plaintext highlighter-rouge">stage.tfvars</code> for staging</p>

<div class="language-plaintext highlighter-rouge"><div class="highlight"><pre class="highlight"><code>instance_type="t2.micro"
instance_name ="stage_vm"
</code></pre></div></div>

<p>create your <code class="language-plaintext highlighter-rouge">prod.tfvars</code> for production</p>

<div class="language-plaintext highlighter-rouge"><div class="highlight"><pre class="highlight"><code>instance_type="t2.micro"
instance_name ="prod_vm"
</code></pre></div></div>

<p>Finally files structure will look like -</p>

<div class="language-plaintext highlighter-rouge"><div class="highlight"><pre class="highlight"><code>/root/
    main.tf
    variable.tf
    stage.tfvars
    prod.tfvars
</code></pre></div></div>

<h2 id="terraform">Terraform</h2>

<p>Now we have completed all the pre-requisites needed for provisioning the VMInstance on Google Cloud.</p>

<p>The first command which we are going to run is -</p>

<div class="language-plaintext highlighter-rouge"><div class="highlight"><pre class="highlight"><code>$ terraform init
</code></pre></div></div>

<p>This command is going to download all the required dependencies based on the provider name mentioned in the main.tf. In the current example the provider name is Google so it is going to install Google’s terraform dependencies onto your machine</p>

<p>The second command which we are going to run is -</p>

<div class="language-plaintext highlighter-rouge"><div class="highlight"><pre class="highlight"><code>$ terraform plan -var-file="stage.tfvars"
</code></pre></div></div>

<p>This command is tell us how many resources are going to be created/destroyed or change for a specific environment (tfvars)
<em>Note - terraform plan never creates the VM on google plan, it just tells you what it is going to perform.</em></p>

<p>The final command which we are going to run is terraform apply.</p>

<p>This command is going to install/setup the virtual machine on Google Cloud.</p>

<div class="language-plaintext highlighter-rouge"><div class="highlight"><pre class="highlight"><code>$ terraform apply -var-file="stage.tfvars"
</code></pre></div></div>

<p>After running the above command you should see as per the plan it will created/change/destroyed the resources for a specific environment (tfvars)</p>

<h2 id="terraform-setting-variable-using-command-line">Terraform setting variable using command line</h2>

<div class="language-plaintext highlighter-rouge"><div class="highlight"><pre class="highlight"><code>provider "google" {
     credentials = file("gcp-account.json")
     project     = "gcp-terraform-307119"
     region      = "europe-west4"
     zone        = "europe-west4-a"
}

resource "google_compute_instance" "default" {
  name         = var.instance_name
  machine_type = var.instance_type

  boot_disk {
    initialize_params {
      image = "debian-cloud/debian-9"
    }
  }

  network_interface {
    network = "default"

    access_config {
      // Ephemeral IP
    }
  }
}

variable "instance_type" {
}

variable "instance_name" {
}

</code></pre></div></div>

<p>To specify individual variables on the command line, use the -var option when running the terraform plan and terraform apply commands:</p>

<div class="language-plaintext highlighter-rouge"><div class="highlight"><pre class="highlight"><code>$ terraform init
$ terraform plan -var="instance_type=t2.micro" -var="instance_name:prod_vm"
$ terraform apply -var="instance_type=t2.micro" -var="instance_name:prod_vm"
</code></pre></div></div>

<p>If we need to destroy the environment using following command -</p>

<div class="language-plaintext highlighter-rouge"><div class="highlight"><pre class="highlight"><code>$ terraform destroy -var="instance_type=t2.micro" -var="instance_name:prod_vm"
</code></pre></div></div>]]></content><author><name></name></author><category term="Terraform" /><summary type="html"><![CDATA[What is Terraform?]]></summary></entry></feed>