<?xml version="1.0" encoding="utf-8" standalone="yes"?>
<rss version="2.0" xmlns:atom="http://www.w3.org/2005/Atom" xmlns:content="http://purl.org/rss/1.0/modules/content/">
  <channel>
    <title>Posts on Vladimir Kuznichenkov | Engineer</title>
    <link>https://kuzaxak.dev/posts/</link>
    <description>Recent content in Posts on Vladimir Kuznichenkov | Engineer</description>
    <generator>Hugo -- gohugo.io</generator>
    <language>en-us</language>
    <copyright>© Vladimir Kuznichenkov</copyright>
    <lastBuildDate>Tue, 26 May 2026 00:00:00 +0000</lastBuildDate><atom:link href="https://kuzaxak.dev/posts/index.xml" rel="self" type="application/rss+xml" />
    <item>
      <title>Sixty Seconds to Nothing: The Missing Daemons Behind Containerized UniFi Access</title>
      <link>https://kuzaxak.dev/posts/2026/05/26/unifi-access-unvr-daemons/</link>
      <pubDate>Tue, 26 May 2026 00:00:00 +0000</pubDate>
      
      <guid>https://kuzaxak.dev/posts/2026/05/26/unifi-access-unvr-daemons/</guid>
      <description>How a Dockerized UniFi Access controller booted, adopted doors, and still failed cloud activation because the image was missing the UniFi OS daemons that make a container behave like a console.</description>
      <content:encoded><![CDATA[<h2 id="preamble-access-on-top-of-a-protect-container">Preamble: Access on Top of a Protect Container</h2>
<p>This did not start as a clean-room UniFi OS container project.</p>
<p>The starting point was much more practical: take the already working <a href="https://github.com/dciancu/unifi-protect-unvr-docker-arm64">dciancu/unifi-protect-unvr-docker-arm64</a> image shape and add Access on top of it.</p>
<p>That upstream project had already solved the boring but painful parts:</p>
<ol>
<li>extract UNVR firmware;</li>
<li>repack the required UniFi OS <code>.deb</code> packages;</li>
<li>install them into an ARM64 Debian image;</li>
<li>run systemd as PID 1 inside a privileged container;</li>
<li>fake enough UNVR hardware identity and storage state for UniFi OS to boot;</li>
<li>run Protect from the firmware bundle.</li>
</ol>
<p>So the first Access attempt was intentionally small. Keep the Protect container shape, install the Access package and its obvious dependencies, add the launcher/nginx bits, and see how far it gets.</p>
<p>The annoying part was that it got very far.</p>
<p>The UniFi OS wizard completed. Access opened. After the network work from Vol 1, the Hub Mini and G3 reader adopted. Door unlock worked. Logs were ugly in places, but nothing looked fundamentally dead.</p>
<p>That made the project feel mostly finished.</p>
<p>Then we started adding capabilities.</p>
<p>Not just “can the local door open”, but:</p>
<ul>
<li>can the console sign in with a Ubiquiti account;</li>
<li>can ownership state sync back from UI cloud;</li>
<li>can Access and Identity agree on users;</li>
<li>can the log noise be reduced instead of hidden;</li>
<li>can the image carry Protect and Access in one UNVR-shaped userspace;</li>
<li>can the repo become something reproducible instead of a one-off patched container.</li>
</ul>
<p>That is where the first image stopped being enough.</p>
<p>The lesson of this post is not “how to start Access once”. The first image already did that. The lesson is what changed when the goal moved from an Access app running inside a Protect-style container to a fuller UniFi OS console shape.</p>
<hr>
<h2 id="it-booted-it-adopted-doors-then-it-tried-to-phone-home">It Booted. It Adopted Doors. Then It Tried to Phone Home.</h2>
<p><a href="/posts/2026/05/24/unifi-access-vxlan-over-ipsec/">Vol 1</a> was the network story.</p>
<p>Access adoption did not care that the controller was reachable over a normal routed tunnel. It cared that discovery looked local. The final shape was VXLAN-over-IPSec, a Linux bridge on the AWS side, and one Docker macvlan interface that made the controller appear as <code>10.255.255.200</code> on the office LAN.</p>
<p>That solved the physical part of the problem. The Hub Mini and G3 reader could discover the controller, adopt, receive config, and stay online.</p>
<p>The image looked solved too.</p>
<p>The controller booted. The UniFi OS wizard completed. Access opened. The door hardware adopted. Door unlocks worked.</p>
<p>Then I clicked <strong>Sign in with Ubiquiti Account</strong>.</p>
<p>The browser waited for one minute and returned:</p>
<div class="highlight"><pre tabindex="0" style="color:#f8f8f2;background-color:#272822;-moz-tab-size:4;-o-tab-size:4;tab-size:4;-webkit-text-size-adjust:none;"><code class="language-text" data-lang="text"><span style="display:flex;"><span>408 Request Timeout
</span></span></code></pre></div><p>Not five seconds. Not “somewhere around a minute”.</p>
<p>Almost exactly sixty seconds.</p>
<p>The obvious logs did not say “wrong package”, “missing daemon”, or “cloud rejected this fake console”. The browser SSO flow itself finished. The failure happened after that, when UniFi Core tried to bind the signed-in UI account to the local console.</p>
<p>That became the second half of the project.</p>
<p>Vol 1 made the network look like the office LAN. Vol 2 is about making Docker look enough like a UniFi console that UniFi Core, Access, Identity, and UI cloud could agree on what the console was.</p>
<p>Ubiquiti does not support this deployment model. Their self-hosting docs allow UniFi OS Server for Network and some UniFi OS features, but state that other applications such as Protect and Access must run on a compatible UniFi Console. Their Access docs also say setup requires a UniFi Console that supports the Access application. This post is about an unsupported lab build, not a production recommendation.</p>
<hr>
<h2 id="the-pattern-we-copied">The Pattern We Copied</h2>
<p>The upstream Protect image was the parent pattern, not a random reference.</p>
<p>It already showed that a UNVR-shaped userspace could run in Docker if the image kept enough of UniFi OS intact. It downloaded UNVR firmware, extracted UniFi-maintained packages, carried <code>unifi-core</code>, <code>ulp-go</code>, <code>uos*</code>, <code>unifi-directory</code>, <code>unifi-assets-unvr</code>, Node packages, and installed Protect from the firmware bundle.</p>
<p>My first mental model was simple:</p>
<div class="highlight"><pre tabindex="0" style="color:#f8f8f2;background-color:#272822;-moz-tab-size:4;-o-tab-size:4;tab-size:4;-webkit-text-size-adjust:none;"><code class="language-text" data-lang="text"><span style="display:flex;"><span>Protect-in-Docker
</span></span><span style="display:flex;"><span>    -&gt; add Access package
</span></span><span style="display:flex;"><span>    -&gt; add Access dependencies
</span></span><span style="display:flex;"><span>    -&gt; patch launcher / nginx
</span></span><span style="display:flex;"><span>    -&gt; keep the same UNVR identity
</span></span><span style="display:flex;"><span>    -&gt; done
</span></span></code></pre></div><p>That was a reasonable starting point because the local app path really did work.</p>
<p>It was also incomplete. Protect and Access share the same UniFi OS shell, but they do not exercise the same internal workflows. Protect could be useful with the smaller upstream daemon set. Access cloud ownership pushed deeper into the Identity and credential side of UniFi OS.</p>
<p><p>
  <img src="https://kuzaxak.dev/posts/2026/05/26/unifi-access-unvr-daemons/protect-pattern.png" alt="Firmware extraction pattern"  />
</p>
</p>
<details>
  <summary>mermaid</summary>
  <div class="highlight"><pre tabindex="0" style="color:#f8f8f2;background-color:#272822;-moz-tab-size:4;-o-tab-size:4;tab-size:4;-webkit-text-size-adjust:none;"><code class="language-go" data-lang="go"><span style="display:flex;"><span><span style="color:#a6e22e">flowchart</span> <span style="color:#a6e22e">LR</span>
</span></span><span style="display:flex;"><span>    <span style="color:#a6e22e">A</span>[<span style="color:#a6e22e">dciancu</span> <span style="color:#a6e22e">Protect</span> <span style="color:#a6e22e">image</span> <span style="color:#a6e22e">shape</span>] <span style="color:#f92672">--</span>&gt; <span style="color:#a6e22e">B</span>[<span style="color:#a6e22e">UNVR</span> <span style="color:#a6e22e">firmware</span> <span style="color:#a6e22e">extraction</span>]
</span></span><span style="display:flex;"><span>    <span style="color:#a6e22e">B</span> <span style="color:#f92672">--</span>&gt; <span style="color:#a6e22e">C</span>[<span style="color:#a6e22e">Debian</span> <span style="color:#ae81ff">11</span> <span style="color:#a6e22e">arm64</span> <span style="color:#f92672">+</span> <span style="color:#a6e22e">systemd</span>]
</span></span><span style="display:flex;"><span>    <span style="color:#a6e22e">C</span> <span style="color:#f92672">--</span>&gt; <span style="color:#a6e22e">D</span>[<span style="color:#a6e22e">UniFi</span> <span style="color:#a6e22e">Core</span> <span style="color:#f92672">+</span> <span style="color:#a6e22e">ULP</span>]
</span></span><span style="display:flex;"><span>    <span style="color:#a6e22e">D</span> <span style="color:#f92672">--</span>&gt; <span style="color:#a6e22e">E</span>[<span style="color:#a6e22e">Protect</span> <span style="color:#a6e22e">from</span> <span style="color:#a6e22e">firmware</span>]
</span></span><span style="display:flex;"><span>    <span style="color:#a6e22e">E</span> <span style="color:#f92672">--</span>&gt; <span style="color:#a6e22e">F</span>[<span style="color:#a6e22e">Storage</span> <span style="color:#f92672">/</span> <span style="color:#a6e22e">identity</span> <span style="color:#f92672">/</span> <span style="color:#a6e22e">nginx</span> <span style="color:#a6e22e">shims</span>]
</span></span><span style="display:flex;"><span>    <span style="color:#a6e22e">F</span> <span style="color:#f92672">--</span>&gt; <span style="color:#a6e22e">G</span>[<span style="color:#a6e22e">Access</span> <span style="color:#f92672">package</span> <span style="color:#a6e22e">added</span>]
</span></span><span style="display:flex;"><span>    <span style="color:#a6e22e">G</span> <span style="color:#f92672">--</span>&gt; <span style="color:#a6e22e">H</span>[<span style="color:#a6e22e">Local</span> <span style="color:#a6e22e">Access</span> <span style="color:#a6e22e">works</span>]
</span></span><span style="display:flex;"><span>    <span style="color:#a6e22e">H</span> <span style="color:#f92672">--</span>&gt; <span style="color:#a6e22e">I</span>[<span style="color:#a6e22e">Cloud</span> <span style="color:#a6e22e">bind</span> <span style="color:#a6e22e">exposes</span> <span style="color:#a6e22e">missing</span> <span style="color:#a6e22e">daemons</span>]
</span></span></code></pre></div>
</details>

<hr>
<h2 id="the-first-image-looked-better-than-it-was">The First Image Looked Better Than It Was</h2>
<p>The first Access-on-Protect image was good enough to be deceptive.</p>
<p>It had the usual console shims:</p>
<table>
  <thead>
      <tr>
          <th>Patch</th>
          <th>Why it existed</th>
      </tr>
  </thead>
  <tbody>
      <tr>
          <td><code>ustorage</code> sed patch in <code>service.js</code></td>
          <td>Push one storage path away from a missing local gRPC service and into a shell-command fallback.</td>
      </tr>
      <tr>
          <td><code>/usr/bin/ustorage</code> shim</td>
          <td>Return plausible disk and space JSON when UniFi Core shells out.</td>
      </tr>
      <tr>
          <td><code>/sbin/mdadm</code> and <code>/usr/sbin/smartctl</code> shims</td>
          <td>Fake RAID and SMART responses that real console hardware would normally provide.</td>
      </tr>
      <tr>
          <td><code>/sbin/ubnt-tools</code> identity shim</td>
          <td>Report a UNVR-shaped console identity instead of whatever container hardware actually looks like.</td>
      </tr>
      <tr>
          <td>nginx setup hostname patch</td>
          <td>Allow reverse-proxied hostnames instead of redirecting every non-<code>unifi</code> host to <code>https://unifi</code>.</td>
      </tr>
      <tr>
          <td>PostgreSQL relocation</td>
          <td>Persist UniFi Core and Access databases under mounted <code>/data</code>, not ephemeral image paths.</td>
      </tr>
      <tr>
          <td>timesync overrides</td>
          <td>Let systemd time sync run in a container. Cloud auth and bad clocks do not mix.</td>
      </tr>
  </tbody>
</table>
<p>None of that was wasted work. Those patches got the image far enough to pass setup, run Access, and operate a door.</p>
<p>But “the door opens” was only the first capability.</p>
<p>The next capability was cloud ownership. That required more than the Access app and more than the Protect-era shims. It required local UniFi OS daemons that could consume Identity and credential messages from UniFi Core.</p>
<p>The bug only appeared when the local console tried to become a cloud-managed console.</p>
<p>That is a nastier class of failure. Local services can look mostly fine while one internal request/reply chain is missing a subscriber.</p>
<hr>
<h2 id="the-symptom-exactly-sixty-seconds">The Symptom: Exactly Sixty Seconds</h2>
<p>The user-visible failure was boring:</p>
<div class="highlight"><pre tabindex="0" style="color:#f8f8f2;background-color:#272822;-moz-tab-size:4;-o-tab-size:4;tab-size:4;-webkit-text-size-adjust:none;"><code class="language-text" data-lang="text"><span style="display:flex;"><span>POST /api/cloud/register
</span></span><span style="display:flex;"><span>408 Request Timeout
</span></span></code></pre></div><p>The useful part was the timing.</p>
<p>The nginx-side request time was basically one minute:</p>
<div class="highlight"><pre tabindex="0" style="color:#f8f8f2;background-color:#272822;-moz-tab-size:4;-o-tab-size:4;tab-size:4;-webkit-text-size-adjust:none;"><code class="language-text" data-lang="text"><span style="display:flex;"><span>POST /api/cloud/register
</span></span><span style="display:flex;"><span>status=408
</span></span><span style="display:flex;"><span>request_time=60.001
</span></span><span style="display:flex;"><span>upstream_response_time=-
</span></span></code></pre></div><p>The <code>u_rt=-</code> part matters. This was not a normal upstream HTTP service returning an error. From nginx's point of view, there was no useful upstream response.</p>
<p>That sent me down two wrong paths first.</p>
<p>The wrong paths are worth keeping in the story because they teach the difference between loud logs and blocking logs.</p>
<hr>
<h2 id="dead-end-1-the-storage-grpc-error-was-loud-not-causal">Dead End 1: The Storage gRPC Error Was Loud, Not Causal</h2>
<p>The loudest error was this:</p>
<div class="highlight"><pre tabindex="0" style="color:#f8f8f2;background-color:#272822;-moz-tab-size:4;-o-tab-size:4;tab-size:4;-webkit-text-size-adjust:none;"><code class="language-text" data-lang="text"><span style="display:flex;"><span>[grpc] Failed to connect to the gRPC server:
</span></span><span style="display:flex;"><span>14 UNAVAILABLE: No connection established.
</span></span><span style="display:flex;"><span>Last error: Error: connect ECONNREFUSED 127.0.0.1:11052
</span></span></code></pre></div><p>It repeated every few seconds.</p>
<p>That looked guilty. A local gRPC service was missing. A cloud-register API was timing out. Easy conclusion: fix storage and cloud sign-in will work.</p>
<p>But the timing did not match.</p>
<p>The <code>11052</code> error happened before the cloud-register attempt, during the cloud-register attempt, and after the cloud-register attempt. It was background noise from an incomplete console image, not the exact thing holding <code>/api/cloud/register</code> open for sixty seconds.</p>
<p>The repo still carries the inherited storage workaround because it is needed for setup paths:</p>
<div class="highlight"><pre tabindex="0" style="color:#f8f8f2;background-color:#272822;-moz-tab-size:4;-o-tab-size:4;tab-size:4;-webkit-text-size-adjust:none;"><code class="language-bash" data-lang="bash"><span style="display:flex;"><span>grep -c <span style="color:#e6db74">&#39;return at(),!0&#39;</span> /usr/share/unifi-core/app/service.js
</span></span><span style="display:flex;"><span><span style="color:#75715e"># 1</span>
</span></span></code></pre></div><p>The sed patch forces one UniFi Core storage path into a shell-command fallback. The <code>/usr/bin/ustorage</code> shim then returns disk, space, and config JSON.</p>
<p>But that patch never meant “all storage/state calls are fixed”. Other state paths still tried to talk to a real console daemon on <code>127.0.0.1:11052</code>.</p>
<p>The later UNVR rebuild did fix this noise by including <code>ustate-exporter</code> and <code>ustd</code>. That was good, but it was not the root cause of the 408.</p>
<table>
  <thead>
      <tr>
          <th>Signal</th>
          <th>What it proved</th>
          <th>What it did not prove</th>
      </tr>
  </thead>
  <tbody>
      <tr>
          <td><code>ECONNREFUSED 127.0.0.1:11052</code></td>
          <td>A local UniFi OS storage/state daemon was missing.</td>
          <td>It did not explain the exact 60-second cloud-register timeout.</td>
      </tr>
      <tr>
          <td><code>ustorage</code> sed patch present</td>
          <td>One setup-breaking storage path was redirected to the shell shim.</td>
          <td>It did not cover every storage/state RPC.</td>
      </tr>
      <tr>
          <td>Wizard completed</td>
          <td>The storage failure was survivable for setup.</td>
          <td>It did not mean the console service graph was complete.</td>
      </tr>
  </tbody>
</table>
<hr>
<h2 id="dead-end-2-ulp-go-was-alive">Dead End 2: <code>ulp-go</code> Was Alive</h2>
<p>The next suspect was <code>ulp-go</code>, the Users/Login Platform service.</p>
<p>The request looked like a UI account bind operation, so <code>ulp-go</code> seemed like the obvious place to look.</p>
<p>But <code>ulp-go</code> was not dead.</p>
<p>It was running. It had local connections to UniFi Core's IPC port. Its status showed it was configured. Its log had the normal startup lines:</p>
<div class="highlight"><pre tabindex="0" style="color:#f8f8f2;background-color:#272822;-moz-tab-size:4;-o-tab-size:4;tab-size:4;-webkit-text-size-adjust:none;"><code class="language-text" data-lang="text"><span style="display:flex;"><span>init wsClient
</span></span><span style="display:flex;"><span>Connected to server
</span></span><span style="display:flex;"><span>Subscribe event success
</span></span><span style="display:flex;"><span>controllerStatus: READY
</span></span><span style="display:flex;"><span>isConfigured: true
</span></span></code></pre></div><p>During the exact sixty seconds where <code>/api/cloud/register</code> hung, <code>ulp-go</code> did not log a failed outbound request. It did not log a token rejection. It did not crash.</p>
<p>It was just quiet.</p>
<p>That broke my first model of the flow. I had assumed something like this:</p>
<div class="highlight"><pre tabindex="0" style="color:#f8f8f2;background-color:#272822;-moz-tab-size:4;-o-tab-size:4;tab-size:4;-webkit-text-size-adjust:none;"><code class="language-text" data-lang="text"><span style="display:flex;"><span>unifi-core
</span></span><span style="display:flex;"><span>    -&gt; HTTP request
</span></span><span style="display:flex;"><span>    -&gt; ulp-go
</span></span><span style="display:flex;"><span>    -&gt; UI cloud
</span></span></code></pre></div><p>The actual shape was closer to this:</p>
<div class="highlight"><pre tabindex="0" style="color:#f8f8f2;background-color:#272822;-moz-tab-size:4;-o-tab-size:4;tab-size:4;-webkit-text-size-adjust:none;"><code class="language-text" data-lang="text"><span style="display:flex;"><span>unifi-core handler
</span></span><span style="display:flex;"><span>    -&gt; publish internal message
</span></span><span style="display:flex;"><span>    -&gt; wait for local subscriber reply
</span></span><span style="display:flex;"><span>    -&gt; no subscriber replies
</span></span><span style="display:flex;"><span>    -&gt; timeout
</span></span></code></pre></div><p>A healthy <code>ulp-go</code> process did not prove that every <code>ulp.*</code> or cloud-binding topic had a consumer.</p>
<hr>
<h2 id="the-number-that-explained-the-timeout">The Number That Explained the Timeout</h2>
<p>The useful clue was a config value, not a stack trace.</p>
<p>Inside UniFi Core:</p>
<div class="highlight"><pre tabindex="0" style="color:#f8f8f2;background-color:#272822;-moz-tab-size:4;-o-tab-size:4;tab-size:4;-webkit-text-size-adjust:none;"><code class="language-yaml" data-lang="yaml"><span style="display:flex;"><span><span style="color:#f92672">messageBox</span>:
</span></span><span style="display:flex;"><span>  <span style="color:#f92672">maxSubscriptionBacklog</span>: <span style="color:#ae81ff">1000</span>
</span></span><span style="display:flex;"><span>  <span style="color:#f92672">subscriptionReaperInterval</span>: <span style="color:#ae81ff">30000</span>
</span></span><span style="display:flex;"><span>  <span style="color:#f92672">longPollTimeout</span>: <span style="color:#ae81ff">60000</span>
</span></span><span style="display:flex;"><span>  <span style="color:#f92672">keepAliveTimeout</span>: <span style="color:#ae81ff">60000</span>
</span></span></code></pre></div><p>The API call waited sixty seconds because the handler was waiting on a messageBox request/reply path.</p>
<p>The chain looked like this:</p>
<div class="highlight"><pre tabindex="0" style="color:#f8f8f2;background-color:#272822;-moz-tab-size:4;-o-tab-size:4;tab-size:4;-webkit-text-size-adjust:none;"><code class="language-text" data-lang="text"><span style="display:flex;"><span>POST /api/cloud/register
</span></span><span style="display:flex;"><span>    -&gt; unifi-core handler
</span></span><span style="display:flex;"><span>    -&gt; publish(&#34;ulp.bindSsoAccount&#34;, payload)
</span></span><span style="display:flex;"><span>    -&gt; wait for response
</span></span><span style="display:flex;"><span>    -&gt; no local subscriber answers
</span></span><span style="display:flex;"><span>    -&gt; messageBox.longPollTimeout
</span></span><span style="display:flex;"><span>    -&gt; 408
</span></span></code></pre></div><p>That was the important inversion.</p>
<p>The browser-visible failure did not mean “UI cloud rejected the console”. It meant UniFi Core published an internal message and nobody in the local container answered it.</p>
<p>Reverse-engineering this out of <code>service.js</code> was ugly. In this UniFi Core build, <code>service.js</code> is a large minified blob. Every diagnostic involves grepping for strings around anonymous variables and then trying to reconstruct intent from call shape.</p>
<p>But the timeout made the architecture visible.</p>
<p><p>
  <img src="https://kuzaxak.dev/posts/2026/05/26/unifi-access-unvr-daemons/messagebox-timeout.png" alt="MessageBox timeout path"  />
</p>
</p>
<details>
  <summary>mermaid</summary>
  <div class="highlight"><pre tabindex="0" style="color:#f8f8f2;background-color:#272822;-moz-tab-size:4;-o-tab-size:4;tab-size:4;-webkit-text-size-adjust:none;"><code class="language-gdscript3" data-lang="gdscript3"><span style="display:flex;"><span>sequenceDiagram
</span></span><span style="display:flex;"><span>    participant Browser
</span></span><span style="display:flex;"><span>    participant Nginx as nginx
</span></span><span style="display:flex;"><span>    participant Core as unifi<span style="color:#f92672">-</span>core
</span></span><span style="display:flex;"><span>    participant MB as messageBox <span style="color:#f92672">/</span> IPC broker<span style="color:#f92672">&lt;</span>br<span style="color:#f92672">/&gt;</span><span style="color:#ae81ff">127.0</span><span style="color:#f92672">.</span><span style="color:#ae81ff">0.1</span>:<span style="color:#ae81ff">11081</span>
</span></span><span style="display:flex;"><span>    participant Missing as missing local subscriber
</span></span><span style="display:flex;"><span>
</span></span><span style="display:flex;"><span>    Browser<span style="color:#f92672">-&gt;&gt;</span>Nginx: POST <span style="color:#f92672">/</span>api<span style="color:#f92672">/</span>cloud<span style="color:#f92672">/</span>register
</span></span><span style="display:flex;"><span>    Nginx<span style="color:#f92672">-&gt;&gt;</span>Core: proxy request
</span></span><span style="display:flex;"><span>    Core<span style="color:#f92672">-&gt;&gt;</span>MB: publish(<span style="color:#e6db74">&#34;ulp.bindSsoAccount&#34;</span>, payload)
</span></span><span style="display:flex;"><span>    MB<span style="color:#f92672">--</span>xMissing: no daemon subscribed to finish the request
</span></span><span style="display:flex;"><span>    Note over Core,MB: waits <span style="color:#66d9ef">for</span> messageBox<span style="color:#f92672">.</span>longPollTimeout<span style="color:#f92672">&lt;</span>br<span style="color:#f92672">/&gt;</span><span style="color:#ae81ff">60000</span> ms
</span></span><span style="display:flex;"><span>    Core<span style="color:#f92672">--&gt;&gt;</span>Nginx: <span style="color:#ae81ff">408</span>
</span></span><span style="display:flex;"><span>    Nginx<span style="color:#f92672">--&gt;&gt;</span>Browser: Request Timeout
</span></span></code></pre></div>
</details>

<p>The missing piece was not another nginx route. It was not a firewall rule. It was not the public cloud endpoint.</p>
<p>It was the local UniFi OS daemon graph.</p>
<hr>
<h2 id="the-package-set-was-the-real-bug">The Package Set Was the Real Bug</h2>
<p>The firmware family was not the problem anymore. We were already using the right family: UNVR.</p>
<p>The problem was that the first Access image treated the upstream Protect package selection as enough of UniFi OS.</p>
<p>For Protect, that selection is practical: core OS packages, <code>ulp-go</code>, directory, assets, UOS helpers, Node packages, and the Protect application from the firmware bundle. That is enough to make Protect useful inside the UNVR-shaped container.</p>
<p>Access raised the bar.</p>
<p>The 408 showed that local Access was not the same as a complete console identity flow. UniFi Core published an internal message for SSO binding, then waited for a local daemon to answer. The app was present. The web UI was present. The door device path was present. The local subscriber for the cloud-ownership path was not.</p>
<p>So the rebuild did not mean “switch away from the Protect pattern”. It meant extend the Protect pattern in the direction it was already pointing:</p>
<div class="highlight"><pre tabindex="0" style="color:#f8f8f2;background-color:#272822;-moz-tab-size:4;-o-tab-size:4;tab-size:4;-webkit-text-size-adjust:none;"><code class="language-text" data-lang="text"><span style="display:flex;"><span>UNVR firmware is still the source
</span></span><span style="display:flex;"><span>UNVR identity is still the runtime lie
</span></span><span style="display:flex;"><span>Protect still comes from the firmware bundle
</span></span><span style="display:flex;"><span>Access is installed on top
</span></span><span style="display:flex;"><span>but the extracted daemon set must be larger
</span></span></code></pre></div><p>That changed the target from:</p>
<div class="highlight"><pre tabindex="0" style="color:#f8f8f2;background-color:#272822;-moz-tab-size:4;-o-tab-size:4;tab-size:4;-webkit-text-size-adjust:none;"><code class="language-text" data-lang="text"><span style="display:flex;"><span>make Access start
</span></span></code></pre></div><p>to:</p>
<div class="highlight"><pre tabindex="0" style="color:#f8f8f2;background-color:#272822;-moz-tab-size:4;-o-tab-size:4;tab-size:4;-webkit-text-size-adjust:none;"><code class="language-text" data-lang="text"><span style="display:flex;"><span>make UniFi OS complete enough for Access, Identity, and cloud ownership
</span></span></code></pre></div><p>Those are not the same thing.</p>
<hr>
<h2 id="the-daemons-that-made-it-a-console">The Daemons That Made It a Console</h2>
<p>The rebuild stayed with UNVR firmware, but pulled in the missing UniFi OS support packages instead of only the smaller Protect-era selection.</p>
<p>The critical additions were not application packages. They were the daemons around the application.</p>
<table>
  <thead>
      <tr>
          <th>Package / daemon</th>
          <th>Role</th>
          <th>Why it mattered</th>
      </tr>
  </thead>
  <tbody>
      <tr>
          <td><code>ucs-agent</code></td>
          <td>UniFi Credential Server agent</td>
          <td>Handles credential and SSO ownership workflows that the cloud-bind path depends on.</td>
      </tr>
      <tr>
          <td><code>uid-agent</code></td>
          <td>UniFi Identity agent</td>
          <td>Connects local identity state, users, and UID flows.</td>
      </tr>
      <tr>
          <td><code>unifi-identity-update</code></td>
          <td>Identity update worker</td>
          <td>Keeps local identity state synchronized after setup and sign-in.</td>
      </tr>
      <tr>
          <td><code>ucore-setup-listener</code></td>
          <td>Post-setup hook listener</td>
          <td>Handles “console configured” transitions after the wizard.</td>
      </tr>
      <tr>
          <td><code>ustate-exporter</code></td>
          <td>Local state exporter</td>
          <td>Provides the gRPC endpoint UniFi Core was polling on <code>127.0.0.1:11052</code>.</td>
      </tr>
      <tr>
          <td><code>ustd</code></td>
          <td>Storage/status daemon</td>
          <td>Part of the local state and storage path.</td>
      </tr>
      <tr>
          <td><code>ubnt-rpsd</code></td>
          <td>RPC subprocess / hardware-related daemon</td>
          <td>Comes with the console graph; may be cosmetic or hardware-only in Docker.</td>
      </tr>
      <tr>
          <td><code>ubnt-ucp4cpp</code></td>
          <td>UCP4 protocol library</td>
          <td>Shared device protocol dependency used by UniFi apps.</td>
      </tr>
      <tr>
          <td><code>unifi-directory</code>, <code>unifi-hal</code>, <code>ubnt-common</code>, <code>libubnt*</code></td>
          <td>Shared platform libraries/services</td>
          <td>Support the daemon graph above.</td>
      </tr>
  </tbody>
</table>
<p>The current repo extracts the selected UNVR packages explicitly instead of copying every <code>.deb</code> blindly:</p>
<div class="highlight"><pre tabindex="0" style="color:#f8f8f2;background-color:#272822;-moz-tab-size:4;-o-tab-size:4;tab-size:4;-webkit-text-size-adjust:none;"><code class="language-dockerfile" data-lang="dockerfile"><span style="display:flex;"><span>cp ubnt-archive-keyring_* ubnt-tools_* ubnt-ucp4cpp_* ubnt-rpsd_* <span style="color:#ae81ff">\
</span></span></span><span style="display:flex;"><span>   ubnt-common_* ubnt-binmecpp_* ubnt-disk-smart-mon_* <span style="color:#ae81ff">\
</span></span></span><span style="display:flex;"><span>   unifi-core_* ulp-go_* unifi-assets-unvr_* unifi-directory_* unifi-hal_* <span style="color:#ae81ff">\
</span></span></span><span style="display:flex;"><span>   unifi-email-templates-all_* unifi-identity-update_* <span style="color:#ae81ff">\
</span></span></span><span style="display:flex;"><span>   uos_* uos-agent_* uos-discovery-client_* <span style="color:#ae81ff">\
</span></span></span><span style="display:flex;"><span>   ucs-agent_* uid-agent_* ucore-setup-listener_* uled-control_* <span style="color:#ae81ff">\
</span></span></span><span style="display:flex;"><span>   ustate-exporter_* ustd_* ubnd_* <span style="color:#ae81ff">\
</span></span></span><span style="display:flex;"><span>   c2lib_* libcurlpp_* libsodiumpp_* libubnt_* libuled_* simple-pid_* <span style="color:#ae81ff">\
</span></span></span><span style="display:flex;"><span>   analytic-report-go_* ble-http-transport_* <span style="color:#ae81ff">\
</span></span></span><span style="display:flex;"><span>   python3-unifi-console-protos_* <span style="color:#ae81ff">\
</span></span></span><span style="display:flex;"><span>   node* ../debs/<span style="color:#960050;background-color:#1e0010">
</span></span></span></code></pre></div><p>This manifest is intentionally explicit. If Ubiquiti renames or removes a package, I want the build to fail loudly.</p>
<p>There is one practical footnote: the line between “hardware-only” and “required dependency” is not always clean. Some packages look like they should be skipped in a container but still satisfy service dependencies. That is why the validation suite matters more than trying to reason about every package name in isolation.</p>
<hr>
<h2 id="why-the-protect-pattern-worked-anyway">Why the Protect Pattern Worked Anyway</h2>
<p>This is not a criticism of the Protect container pattern.</p>
<p>The parent project solved a different problem. It built a UNVR-shaped userspace for Protect, and Protect's normal paths were satisfied enough for that application.</p>
<p>Access made the gap visible because its cloud sign-in path exercised more of the Identity and credential side of UniFi OS.</p>
<p>So the lesson is not:</p>
<div class="highlight"><pre tabindex="0" style="color:#f8f8f2;background-color:#272822;-moz-tab-size:4;-o-tab-size:4;tab-size:4;-webkit-text-size-adjust:none;"><code class="language-text" data-lang="text"><span style="display:flex;"><span>Protect-in-Docker is incomplete.
</span></span></code></pre></div><p>The lesson is:</p>
<div class="highlight"><pre tabindex="0" style="color:#f8f8f2;background-color:#272822;-moz-tab-size:4;-o-tab-size:4;tab-size:4;-webkit-text-size-adjust:none;"><code class="language-text" data-lang="text"><span style="display:flex;"><span>Access uses more of the console identity stack than the first copied image included.
</span></span></code></pre></div><p>The first build could adopt doors because the local Access service worked. It could not finish cloud ownership because the local message-bus consumer chain was incomplete.</p>
<hr>
<h2 id="the-rebuild-protect-plus-access-with-the-missing-daemons">The Rebuild: Protect Plus Access, With the Missing Daemons</h2>
<p>The fix was not to abandon the upstream image shape.</p>
<p>The fix was to make the image more honest about what it wanted to be: a UNVR-shaped UniFi OS userspace with both Protect and Access, plus the support daemons that Access needs for Identity and cloud ownership.</p>
<p>The repo now builds in two stages:</p>
<ol>
<li><code>build-firmware.sh</code> downloads or uses a selected UNVR firmware image, extracts the firmware, and writes generated artifacts under <code>firmware/&lt;series&gt;/</code>.</li>
<li><code>build-os.sh</code> installs the extracted packages, plus Protect and Access, into a Debian 11 ARM64 systemd image.</li>
</ol>
<p>There is also a single-file multi-stage <code>Dockerfile</code> for users who want <code>docker build .</code>, but the split scripts are easier to debug.</p>
<p><p>
  <img src="https://kuzaxak.dev/posts/2026/05/26/unifi-access-unvr-daemons/unvr-rebuild.png" alt="Protect plus Access rebuild flow"  />
</p>
</p>
<details>
  <summary>mermaid</summary>
  <div class="highlight"><pre tabindex="0" style="color:#f8f8f2;background-color:#272822;-moz-tab-size:4;-o-tab-size:4;tab-size:4;-webkit-text-size-adjust:none;"><code class="language-go" data-lang="go"><span style="display:flex;"><span><span style="color:#a6e22e">flowchart</span> <span style="color:#a6e22e">LR</span>
</span></span><span style="display:flex;"><span>    <span style="color:#a6e22e">A</span>[<span style="color:#a6e22e">UNVR</span> <span style="color:#a6e22e">firmware</span>] <span style="color:#f92672">--</span>&gt; <span style="color:#a6e22e">B</span>[<span style="color:#a6e22e">build</span><span style="color:#f92672">-</span><span style="color:#a6e22e">firmware</span>.<span style="color:#a6e22e">sh</span>]
</span></span><span style="display:flex;"><span>    <span style="color:#a6e22e">B</span> <span style="color:#f92672">--</span>&gt; <span style="color:#a6e22e">C</span>[<span style="color:#a6e22e">binwalk</span> <span style="color:#a6e22e">extraction</span>]
</span></span><span style="display:flex;"><span>    <span style="color:#a6e22e">C</span> <span style="color:#f92672">--</span>&gt; <span style="color:#a6e22e">D</span>[<span style="color:#a6e22e">dpkg</span><span style="color:#f92672">-</span><span style="color:#a6e22e">repack</span> <span style="color:#a6e22e">selected</span> <span style="color:#a6e22e">UniFi</span> <span style="color:#a6e22e">OS</span> <span style="color:#a6e22e">packages</span>]
</span></span><span style="display:flex;"><span>    <span style="color:#a6e22e">D</span> <span style="color:#f92672">--</span>&gt; <span style="color:#a6e22e">E</span>[<span style="color:#a6e22e">base</span> <span style="color:#a6e22e">Protect</span><span style="color:#f92672">-</span><span style="color:#a6e22e">era</span> <span style="color:#f92672">package</span> <span style="color:#a6e22e">set</span>]
</span></span><span style="display:flex;"><span>    <span style="color:#a6e22e">D</span> <span style="color:#f92672">--</span>&gt; <span style="color:#a6e22e">F</span>[<span style="color:#a6e22e">extra</span> <span style="color:#a6e22e">Identity</span> <span style="color:#f92672">/</span> <span style="color:#a6e22e">state</span> <span style="color:#a6e22e">daemons</span>]
</span></span><span style="display:flex;"><span>    <span style="color:#a6e22e">D</span> <span style="color:#f92672">--</span>&gt; <span style="color:#a6e22e">G</span>[<span style="color:#a6e22e">firmware</span><span style="color:#f92672">-</span><span style="color:#a6e22e">bundled</span> <span style="color:#a6e22e">Protect</span> <span style="color:#a6e22e">debs</span>]
</span></span><span style="display:flex;"><span>
</span></span><span style="display:flex;"><span>    <span style="color:#a6e22e">H</span>[<span style="color:#a6e22e">build</span><span style="color:#f92672">-</span><span style="color:#a6e22e">os</span>.<span style="color:#a6e22e">sh</span>] <span style="color:#f92672">--</span>&gt; <span style="color:#a6e22e">I</span>[<span style="color:#a6e22e">Debian</span> <span style="color:#ae81ff">11</span> <span style="color:#a6e22e">arm64</span>]
</span></span><span style="display:flex;"><span>    <span style="color:#a6e22e">E</span> <span style="color:#f92672">--</span>&gt; <span style="color:#a6e22e">I</span>
</span></span><span style="display:flex;"><span>    <span style="color:#a6e22e">F</span> <span style="color:#f92672">--</span>&gt; <span style="color:#a6e22e">I</span>
</span></span><span style="display:flex;"><span>    <span style="color:#a6e22e">G</span> <span style="color:#f92672">--</span>&gt; <span style="color:#a6e22e">I</span>
</span></span><span style="display:flex;"><span>    <span style="color:#a6e22e">J</span>[<span style="color:#a6e22e">Access</span> <span style="color:#f92672">package</span> <span style="color:#a6e22e">from</span> <span style="color:#a6e22e">fw</span><span style="color:#f92672">-</span><span style="color:#a6e22e">update</span>] <span style="color:#f92672">--</span>&gt; <span style="color:#a6e22e">I</span>
</span></span><span style="display:flex;"><span>    <span style="color:#a6e22e">K</span>[<span style="color:#a6e22e">Access</span> <span style="color:#a6e22e">deps</span> <span style="color:#f92672">/</span> <span style="color:#a6e22e">user</span> <span style="color:#a6e22e">assets</span> <span style="color:#f92672">/</span> <span style="color:#a6e22e">media</span> <span style="color:#a6e22e">services</span>] <span style="color:#f92672">--</span>&gt; <span style="color:#a6e22e">I</span>
</span></span><span style="display:flex;"><span>    <span style="color:#a6e22e">I</span> <span style="color:#f92672">--</span>&gt; <span style="color:#a6e22e">L</span>[<span style="color:#a6e22e">unifi</span><span style="color:#f92672">-</span><span style="color:#a6e22e">os</span><span style="color:#f92672">-</span><span style="color:#a6e22e">docker</span><span style="color:#f92672">-</span><span style="color:#a6e22e">arm64</span>:<span style="color:#a6e22e">stable</span> <span style="color:#a6e22e">or</span> :<span style="color:#a6e22e">edge</span>]
</span></span></code></pre></div>
</details>

<p>The stable build uses pinned Access and AI-feature package URLs where needed, and installs Protect from the firmware bundle. The edge build resolves newer app packages from Ubiquiti's firmware update endpoints. That gives two useful modes:</p>
<table>
  <thead>
      <tr>
          <th>Build mode</th>
          <th>Use it when</th>
      </tr>
  </thead>
  <tbody>
      <tr>
          <td><code>BUILD_STABLE=1</code></td>
          <td>You want a known baseline and fewer moving parts.</td>
      </tr>
      <tr>
          <td><code>BUILD_EDGE=1</code></td>
          <td>You are testing newer Access/Protect packages or chasing a bug fixed upstream.</td>
      </tr>
  </tbody>
</table>
<p>Access is downloaded on top because UNVR firmware does not bundle it in the same way it bundles Protect. Protect comes from the UNVR firmware extraction, so the repo installs both apps into one image.</p>
<p>That gives the image a clearer identity:</p>
<div class="highlight"><pre tabindex="0" style="color:#f8f8f2;background-color:#272822;-moz-tab-size:4;-o-tab-size:4;tab-size:4;-webkit-text-size-adjust:none;"><code class="language-text" data-lang="text"><span style="display:flex;"><span>UNVR-shaped UniFi OS userspace
</span></span><span style="display:flex;"><span>    + Protect from UNVR firmware
</span></span><span style="display:flex;"><span>    + Access from fw-update package
</span></span><span style="display:flex;"><span>    + extra Identity and state daemons Access needs
</span></span></code></pre></div><p>The important design choice is that the repo keeps the upstream Protect inheritance visible. This is not “my own UniFi OS from scratch”. It is the Protect container pattern expanded until Access cloud ownership stopped timing out.</p>
<hr>
<h2 id="the-swap-that-proved-the-theory">The Swap That Proved the Theory</h2>
<p>I did not replace the old container blindly.</p>
<p>The old image stayed around as rollback:</p>
<div class="highlight"><pre tabindex="0" style="color:#f8f8f2;background-color:#272822;-moz-tab-size:4;-o-tab-size:4;tab-size:4;-webkit-text-size-adjust:none;"><code class="language-bash" data-lang="bash"><span style="display:flex;"><span>docker rename unifi-access unifi-access-v0.2.0
</span></span></code></pre></div><p>The persistent state stayed mounted:</p>
<div class="highlight"><pre tabindex="0" style="color:#f8f8f2;background-color:#272822;-moz-tab-size:4;-o-tab-size:4;tab-size:4;-webkit-text-size-adjust:none;"><code class="language-text" data-lang="text"><span style="display:flex;"><span>/srv/unifi-access/srv
</span></span><span style="display:flex;"><span>/srv/unifi-access/data
</span></span><span style="display:flex;"><span>/srv/unifi-access/persistent
</span></span></code></pre></div><p>The new image started with the same state and the same office-facing network identity from Vol 1.</p>
<p>The first surprise was that cloud activation was already true after restart:</p>
<div class="highlight"><pre tabindex="0" style="color:#f8f8f2;background-color:#272822;-moz-tab-size:4;-o-tab-size:4;tab-size:4;-webkit-text-size-adjust:none;"><code class="language-bash" data-lang="bash"><span style="display:flex;"><span>docker exec unifi-os systemctl show ulp-go -p StatusText | <span style="color:#ae81ff">\
</span></span></span><span style="display:flex;"><span>  grep -oE <span style="color:#e6db74">&#39;standardActivated&#34;:[a-z]+|ucsAgentVersion&#34;:&#34;[^&#34;]+&#34;&#39;</span>
</span></span></code></pre></div><p>Expected shape:</p>
<div class="highlight"><pre tabindex="0" style="color:#f8f8f2;background-color:#272822;-moz-tab-size:4;-o-tab-size:4;tab-size:4;-webkit-text-size-adjust:none;"><code class="language-text" data-lang="text"><span style="display:flex;"><span>standardActivated&#34;:true
</span></span><span style="display:flex;"><span>ucsAgentVersion&#34;:&#34;1.6.98+733&#34;
</span></span></code></pre></div><p>I had not clicked the cloud sign-in button again.</p>
<p>That changed the interpretation of the earlier 408.</p>
<p>The browser-visible <code>/api/cloud/register</code> request had timed out locally, but the server-side cloud claim had apparently completed. The missing part was local state convergence. The old image had no daemon to consume the internal message and flip the local state. The new one did.</p>
<p>The evidence table looked much better:</p>
<table>
  <thead>
      <tr>
          <th>Check</th>
          <th>Result</th>
      </tr>
  </thead>
  <tbody>
      <tr>
          <td><code>ulp-go</code> status</td>
          <td><code>standardActivated: true</code></td>
      </tr>
      <tr>
          <td><code>ucsAgentVersion</code></td>
          <td>Present</td>
      </tr>
      <tr>
          <td><code>ulp-go-app</code> outbound TLS</td>
          <td>Established to UI cloud</td>
      </tr>
      <tr>
          <td><code>uid-agent-app</code> outbound TLS</td>
          <td>Established to UI cloud</td>
      </tr>
      <tr>
          <td><code>unifi-protect</code> outbound TLS</td>
          <td>Established to UI cloud</td>
      </tr>
      <tr>
          <td><code>127.0.0.1:11052</code></td>
          <td>Listening</td>
      </tr>
      <tr>
          <td>repeated <code>ECONNREFUSED 127.0.0.1:11052</code></td>
          <td>Gone</td>
      </tr>
      <tr>
          <td>Access device sessions</td>
          <td>Still established on <code>12443</code> / <code>12812</code></td>
      </tr>
  </tbody>
</table>
<p>Example checks:</p>
<div class="highlight"><pre tabindex="0" style="color:#f8f8f2;background-color:#272822;-moz-tab-size:4;-o-tab-size:4;tab-size:4;-webkit-text-size-adjust:none;"><code class="language-bash" data-lang="bash"><span style="display:flex;"><span>docker exec unifi-os ss -tnp state established | grep <span style="color:#e6db74">&#39;:443&#39;</span>
</span></span><span style="display:flex;"><span>
</span></span><span style="display:flex;"><span>docker exec unifi-os ss -ltnp | grep <span style="color:#ae81ff">11052</span>
</span></span><span style="display:flex;"><span>
</span></span><span style="display:flex;"><span>docker exec unifi-os tail -n <span style="color:#ae81ff">20</span> /data/unifi-core/logs/grpc.log
</span></span></code></pre></div><p>The log line that mattered:</p>
<div class="highlight"><pre tabindex="0" style="color:#f8f8f2;background-color:#272822;-moz-tab-size:4;-o-tab-size:4;tab-size:4;-webkit-text-size-adjust:none;"><code class="language-text" data-lang="text"><span style="display:flex;"><span>call cloud api https://cell3.api.identity.ui.com/standard/api/v1/public/console/ownership/check [POST] success
</span></span></code></pre></div><p>That was the answer to the title.</p>
<p>The sixty-second timeout was not “UI cloud hates Docker”. It was “UniFi Core is waiting for a local console daemon that is missing”.</p>
<hr>
<h2 id="the-last-mile-registered-is-not-connected">The Last Mile: Registered Is Not Connected</h2>
<p>The daemon rebuild fixed the sixty-second bind timeout, but it exposed one more trap.</p>
<p>Cloud registration and cloud connection are different states.</p>
<p>After the Ubiquiti account flow completed, UniFi Core had clearly registered the console:</p>
<div class="highlight"><pre tabindex="0" style="color:#f8f8f2;background-color:#272822;-moz-tab-size:4;-o-tab-size:4;tab-size:4;-webkit-text-size-adjust:none;"><code class="language-text" data-lang="text"><span style="display:flex;"><span>2026-05-26T16:38:03+03: registering device for cloud access
</span></span><span style="display:flex;"><span>2026-05-26T16:38:04+03: registered device for cloud access: deviceId=&lt;device-id&gt;, ownerId=&lt;owner-id&gt;
</span></span></code></pre></div><p>But the console still did not appear in the cloud list, and cloud-backed operations failed locally:</p>
<div class="highlight"><pre tabindex="0" style="color:#f8f8f2;background-color:#272822;-moz-tab-size:4;-o-tab-size:4;tab-size:4;-webkit-text-size-adjust:none;"><code class="language-text" data-lang="text"><span style="display:flex;"><span>Can&#39;t publish to &#34;api/v1/create-auth-token/...&#34;, cloud connection not established
</span></span><span style="display:flex;"><span>Can&#39;t publish to &#34;api/v1/create-invitation/&lt;device-id&gt;&#34;, cloud connection not established
</span></span></code></pre></div><p>That made the state machine clearer:</p>
<table>
  <thead>
      <tr>
          <th>State</th>
          <th>What it means</th>
      </tr>
  </thead>
  <tbody>
      <tr>
          <td>Cloud account bind succeeded</td>
          <td>Ubiquiti accepted the ownership request.</td>
      </tr>
      <tr>
          <td><code>standardActivated: true</code></td>
          <td>Local ULP state knows the console is owned.</td>
      </tr>
      <tr>
          <td>Cloud connection established</td>
          <td>UniFi Core has a live cloud transport and can publish operations.</td>
      </tr>
  </tbody>
</table>
<p>The broken value was hiding in UniFi Core's settings:</p>
<div class="highlight"><pre tabindex="0" style="color:#f8f8f2;background-color:#272822;-moz-tab-size:4;-o-tab-size:4;tab-size:4;-webkit-text-size-adjust:none;"><code class="language-yaml" data-lang="yaml"><span style="display:flex;"><span><span style="color:#f92672">anonymous_device_id</span>: <span style="color:#e6db74">&#34;Ubiquiti system tools\nCopyright 2006-2024, Ubiquiti, Inc. &lt;support@ui.com&gt;\n...\thwaddr&#34;</span>
</span></span></code></pre></div><p>That value should have been a stable UUID. Instead, a failed identity-helper path had persisted <code>ubnt-tools</code> banner/help text as the anonymous console ID. <code>/data/uuid.txt</code> already had a valid UUID; UniFi Core was just not using it.</p>
<p>The durable fix had two parts:</p>
<ol>
<li><code>ubnt-tools id &lt;file&gt;</code> now writes the same generated board identity to the requested file that <code>ubnt-tools id</code> prints to stdout.</li>
<li><code>init_console.sh</code> now repairs missing, <code>null</code>, or non-UUID <code>anonymous_device_id</code> values back to the stable <code>/data/uuid.txt</code> value.</li>
</ol>
<p>After repairing the live settings file and restarting <code>uos-agent</code> plus <code>unifi-core</code>, the cloud log changed shape:</p>
<div class="highlight"><pre tabindex="0" style="color:#f8f8f2;background-color:#272822;-moz-tab-size:4;-o-tab-size:4;tab-size:4;-webkit-text-size-adjust:none;"><code class="language-text" data-lang="text"><span style="display:flex;"><span>2026-05-27T00:09:10+03: Device connected to cloud
</span></span><span style="display:flex;"><span>subscribe device/&lt;device-id&gt;
</span></span><span style="display:flex;"><span>subscribe $aws/things/&lt;device-id&gt;/shadow/name/+/update/delta
</span></span><span style="display:flex;"><span>subscribe device/&lt;device-id&gt;/client/#
</span></span><span style="display:flex;"><span>subscribe d/&lt;device-id&gt;/c/#
</span></span><span style="display:flex;"><span>subscribe device/&lt;device-id&gt;/ping
</span></span><span style="display:flex;"><span>subscribe device/&lt;device-id&gt;/application/+
</span></span><span style="display:flex;"><span>subscribe d/proxy/&lt;device-id&gt;/#
</span></span><span style="display:flex;"><span>Successful sync owner to cloud
</span></span></code></pre></div><p>At that point <code>unifi-core</code> also had an established MQTT/TLS connection to <code>:8883</code>, and the log stopped producing fresh <code>cloud connection not established</code> errors.</p>
<p>That is the distinction I wish I had checked earlier: a successful account bind proves ownership, not necessarily a usable cloud transport.</p>
<hr>
<h2 id="local-repo-defaults-vs-the-cloud-topology">Local Repo Defaults vs the Cloud Topology</h2>
<p>The public repo defaults to host networking:</p>
<div class="highlight"><pre tabindex="0" style="color:#f8f8f2;background-color:#272822;-moz-tab-size:4;-o-tab-size:4;tab-size:4;-webkit-text-size-adjust:none;"><code class="language-yaml" data-lang="yaml"><span style="display:flex;"><span><span style="color:#f92672">network_mode</span>: <span style="color:#ae81ff">host</span>
</span></span></code></pre></div><p>That is the right default for most Linux users. If the container runs on a local ARM64 host on the same LAN as the Access devices, host networking is the simplest way to make discovery and adoption work.</p>
<p>The cloud deployment from Vol 1 is different. There the container does not live on the office LAN, so host networking on the EC2 instance would only expose AWS interfaces. The cloud version still needs the macvlan/VXLAN shape:</p>
<div class="highlight"><pre tabindex="0" style="color:#f8f8f2;background-color:#272822;-moz-tab-size:4;-o-tab-size:4;tab-size:4;-webkit-text-size-adjust:none;"><code class="language-text" data-lang="text"><span style="display:flex;"><span>Office LAN
</span></span><span style="display:flex;"><span>    -&gt; VXLAN-over-IPSec
</span></span><span style="display:flex;"><span>    -&gt; br-office on EC2
</span></span><span style="display:flex;"><span>    -&gt; Docker macvlan parent=br-office
</span></span><span style="display:flex;"><span>    -&gt; unifi-os container IP 10.255.255.200
</span></span></code></pre></div><p>The repo README documents both ideas:</p>
<table>
  <thead>
      <tr>
          <th>Deployment</th>
          <th>Network mode</th>
      </tr>
  </thead>
  <tbody>
      <tr>
          <td>Local Linux host on the real LAN</td>
          <td><code>network_mode: host</code></td>
      </tr>
      <tr>
          <td>macOS Docker Desktop lab UI</td>
          <td>NATed VM, UI-only; no real hardware adoption</td>
      </tr>
      <tr>
          <td>Cloud or routed deployment</td>
          <td>Explicit L2 extension, bridge, and macvlan-only container</td>
      </tr>
  </tbody>
</table>
<p>The warning from Vol 1 still applies: do not attach both the default Docker bridge and the office macvlan. UniFi OS can see the bridge address, select it as controller identity, and generate device or invitation URLs pointing at an unreachable <code>172.x.x.x</code> address.</p>
<p>One identity is boring. Boring is good.</p>
<hr>
<h2 id="the-network-repair-timer-stayed">The Network Repair Timer Stayed</h2>
<p>The rebuilt image still carries the Access network repair timer.</p>
<p>This is a Vol 1 lesson that became repo code.</p>
<p>The timer runs once shortly after boot and then every minute:</p>
<div class="highlight"><pre tabindex="0" style="color:#f8f8f2;background-color:#272822;-moz-tab-size:4;-o-tab-size:4;tab-size:4;-webkit-text-size-adjust:none;"><code class="language-ini" data-lang="ini"><span style="display:flex;"><span><span style="color:#66d9ef">[Timer]</span>
</span></span><span style="display:flex;"><span><span style="color:#a6e22e">OnBootSec</span><span style="color:#f92672">=</span><span style="color:#e6db74">20s</span>
</span></span><span style="display:flex;"><span><span style="color:#a6e22e">OnUnitActiveSec</span><span style="color:#f92672">=</span><span style="color:#e6db74">1min</span>
</span></span><span style="display:flex;"><span><span style="color:#a6e22e">AccuracySec</span><span style="color:#f92672">=</span><span style="color:#e6db74">10s</span>
</span></span><span style="display:flex;"><span><span style="color:#a6e22e">Persistent</span><span style="color:#f92672">=</span><span style="color:#e6db74">true</span>
</span></span></code></pre></div><p>The service calls:</p>
<div class="highlight"><pre tabindex="0" style="color:#f8f8f2;background-color:#272822;-moz-tab-size:4;-o-tab-size:4;tab-size:4;-webkit-text-size-adjust:none;"><code class="language-text" data-lang="text"><span style="display:flex;"><span>/usr/sbin/configure_unifi_access_network.sh
</span></span></code></pre></div><p>The script resolves the Access management interface in this order:</p>
<ol>
<li>explicit <code>ACCESS_MNGT_NETWORK_ID</code>;</li>
<li>interface holding <code>ACCESS_MNGT_NETWORK_IP</code>;</li>
<li>default-route interface.</li>
</ol>
<p>Then it lowers the interface MTU if it is larger than <code>ACCESS_DEVICE_MTU</code>, defaulting to <code>1200</code>:</p>
<div class="highlight"><pre tabindex="0" style="color:#f8f8f2;background-color:#272822;-moz-tab-size:4;-o-tab-size:4;tab-size:4;-webkit-text-size-adjust:none;"><code class="language-bash" data-lang="bash"><span style="display:flex;"><span>device_mtu<span style="color:#f92672">=</span><span style="color:#e6db74">&#34;</span><span style="color:#e6db74">${</span>ACCESS_DEVICE_MTU<span style="color:#66d9ef">:-</span>1200<span style="color:#e6db74">}</span><span style="color:#e6db74">&#34;</span>
</span></span></code></pre></div><p>Finally, if the Access API is up, it checks <code>/api/v2/networks</code> and <code>/api/v2/settings</code>, then updates <code>mngt_network_id</code> when needed.</p>
<p>That timer exists because discovery success was not enough. Small UDP discovery packets worked even when larger post-adoption TLS/MQTT flows did not. Docker could recreate the macvlan interface with an unsafe MTU, and Access could drift back to the wrong management network selection.</p>
<p>So the image keeps repairing the two things that made the reader stable:</p>
<div class="highlight"><pre tabindex="0" style="color:#f8f8f2;background-color:#272822;-moz-tab-size:4;-o-tab-size:4;tab-size:4;-webkit-text-size-adjust:none;"><code class="language-text" data-lang="text"><span style="display:flex;"><span>correct device-facing interface
</span></span><span style="display:flex;"><span>safe device-facing MTU
</span></span></code></pre></div><hr>
<h2 id="the-ipc-map-i-wish-i-had-first">The IPC Map I Wish I Had First</h2>
<p>This is the model that made the failure understandable.</p>
<p>It is not a full UniFi OS spec. It is the map that was useful for debugging this container.</p>
<p><p>
  <img src="https://kuzaxak.dev/posts/2026/05/26/unifi-access-unvr-daemons/ipc-map.png" alt="UniFi OS IPC map"  />
</p>
</p>
<details>
  <summary>mermaid</summary>
  <div class="highlight"><pre tabindex="0" style="color:#f8f8f2;background-color:#272822;-moz-tab-size:4;-o-tab-size:4;tab-size:4;-webkit-text-size-adjust:none;"><code class="language-gdscript3" data-lang="gdscript3"><span style="display:flex;"><span>flowchart TB
</span></span><span style="display:flex;"><span>    Browser[Browser <span style="color:#f92672">/</span> setup UI]
</span></span><span style="display:flex;"><span>    Nginx[nginx<span style="color:#f92672">&lt;</span>br<span style="color:#f92672">/&gt;</span><span style="color:#ae81ff">443</span>]
</span></span><span style="display:flex;"><span>    Core[unifi<span style="color:#f92672">-</span>core<span style="color:#f92672">&lt;</span>br<span style="color:#f92672">/&gt;</span><span style="color:#a6e22e">Node</span><span style="color:#f92672">.</span>js API]
</span></span><span style="display:flex;"><span>    MB[messageBox <span style="color:#f92672">/</span> IPC broker<span style="color:#f92672">&lt;</span>br<span style="color:#f92672">/&gt;</span><span style="color:#ae81ff">127.0</span><span style="color:#f92672">.</span><span style="color:#ae81ff">0.1</span>:<span style="color:#ae81ff">11081</span>]
</span></span><span style="display:flex;"><span>    ULP[ulp<span style="color:#f92672">-</span>go<span style="color:#f92672">&lt;</span>br<span style="color:#f92672">/&gt;</span>users <span style="color:#f92672">/</span> login platform]
</span></span><span style="display:flex;"><span>    Sock[<span style="color:#e6db74">&#34;/run/ulp-go/jsonrpc.sock&#34;</span>]
</span></span><span style="display:flex;"><span>    UCS[ucs<span style="color:#f92672">-</span>agent]
</span></span><span style="display:flex;"><span>    UID[uid<span style="color:#f92672">-</span>agent]
</span></span><span style="display:flex;"><span>    Dir[unifi<span style="color:#f92672">-</span>directory]
</span></span><span style="display:flex;"><span>    Update[unifi<span style="color:#f92672">-</span>identity<span style="color:#f92672">-</span>update]
</span></span><span style="display:flex;"><span>    State[ustate<span style="color:#f92672">-</span>exporter <span style="color:#f92672">/</span> ustd<span style="color:#f92672">&lt;</span>br<span style="color:#f92672">/&gt;</span>gRPC <span style="color:#ae81ff">127.0</span><span style="color:#f92672">.</span><span style="color:#ae81ff">0.1</span>:<span style="color:#ae81ff">11052</span>]
</span></span><span style="display:flex;"><span>    Access[unifi<span style="color:#f92672">-</span>access<span style="color:#f92672">&lt;</span>br<span style="color:#f92672">/&gt;</span><span style="color:#ae81ff">12080</span> <span style="color:#f92672">/</span> <span style="color:#ae81ff">12443</span> <span style="color:#f92672">/</span> <span style="color:#ae81ff">12812</span>]
</span></span><span style="display:flex;"><span>    Protect[unifi<span style="color:#f92672">-</span>protect]
</span></span><span style="display:flex;"><span>    Cloud[UI cloud<span style="color:#f92672">&lt;</span>br<span style="color:#f92672">/&gt;*.</span>ui<span style="color:#f92672">.</span>com HTTPS]
</span></span><span style="display:flex;"><span>
</span></span><span style="display:flex;"><span>    Browser <span style="color:#f92672">--&gt;</span> Nginx
</span></span><span style="display:flex;"><span>    Nginx <span style="color:#f92672">--&gt;</span> Core
</span></span><span style="display:flex;"><span>
</span></span><span style="display:flex;"><span>    Core <span style="color:#f92672">&lt;--&gt;</span> MB
</span></span><span style="display:flex;"><span>    ULP <span style="color:#f92672">&lt;--&gt;</span> MB
</span></span><span style="display:flex;"><span>
</span></span><span style="display:flex;"><span>    ULP <span style="color:#f92672">&lt;--&gt;</span> Sock
</span></span><span style="display:flex;"><span>    UCS <span style="color:#f92672">&lt;--&gt;</span> Sock
</span></span><span style="display:flex;"><span>    UID <span style="color:#f92672">&lt;--&gt;</span> Sock
</span></span><span style="display:flex;"><span>    Dir <span style="color:#f92672">&lt;--&gt;</span> Sock
</span></span><span style="display:flex;"><span>    Update <span style="color:#f92672">&lt;--&gt;</span> Sock
</span></span><span style="display:flex;"><span>
</span></span><span style="display:flex;"><span>    Core <span style="color:#f92672">&lt;--&gt;</span> State
</span></span><span style="display:flex;"><span>
</span></span><span style="display:flex;"><span>    Core <span style="color:#f92672">--&gt;</span> Access
</span></span><span style="display:flex;"><span>    Core <span style="color:#f92672">--&gt;</span> Protect
</span></span><span style="display:flex;"><span>
</span></span><span style="display:flex;"><span>    ULP <span style="color:#f92672">--&gt;</span> Cloud
</span></span><span style="display:flex;"><span>    UCS <span style="color:#f92672">--&gt;</span> Cloud
</span></span><span style="display:flex;"><span>    UID <span style="color:#f92672">--&gt;</span> Cloud
</span></span><span style="display:flex;"><span>    Protect <span style="color:#f92672">--&gt;</span> Cloud
</span></span></code></pre></div>
</details>

<p>The split that matters:</p>
<table>
  <thead>
      <tr>
          <th>Path</th>
          <th>What it proves when healthy</th>
      </tr>
  </thead>
  <tbody>
      <tr>
          <td>Browser → nginx → <code>unifi-core</code></td>
          <td>The UI/API entrypoint works.</td>
      </tr>
      <tr>
          <td><code>unifi-core</code> → messageBox <code>:11081</code></td>
          <td>Core can publish internal requests.</td>
      </tr>
      <tr>
          <td><code>ulp-go</code> connected to <code>:11081</code></td>
          <td>ULP is alive, but not necessarily the handler for every topic.</td>
      </tr>
      <tr>
          <td><code>ulp-go</code> / agents → <code>/run/ulp-go/jsonrpc.sock</code></td>
          <td>Identity daemons can share local state.</td>
      </tr>
      <tr>
          <td><code>unifi-core</code> → <code>127.0.0.1:11052</code></td>
          <td>Storage/state exporter path works.</td>
      </tr>
      <tr>
          <td>agents → <code>*.ui.com:443</code></td>
          <td>Local console identity stack can reach UI cloud.</td>
      </tr>
  </tbody>
</table>
<p>The trap was assuming that one healthy service meant the whole chain was healthy.</p>
<p><code>ulp-go</code> was alive. It just was not enough.</p>
<hr>
<h2 id="current-patch-catalog">Current Patch Catalog</h2>
<p>The UNVR rebuild reduced the number of things that had to be faked, but it did not remove all patches.</p>
<p>This is still Docker pretending to be a console.</p>
<table>
  <thead>
      <tr>
          <th>Patch / shim</th>
          <th>Where</th>
          <th>Without it</th>
      </tr>
  </thead>
  <tbody>
      <tr>
          <td><code>ustorage</code> sed patch</td>
          <td><code>os.Dockerfile</code>, <code>/usr/share/unifi-core/app/service.js</code></td>
          <td>Setup can hit storage paths that expect real console hardware.</td>
      </tr>
      <tr>
          <td><code>/usr/bin/ustorage</code> shim</td>
          <td><code>files/usr/bin/ustorage</code></td>
          <td>Shell fallback has nothing useful to call.</td>
      </tr>
      <tr>
          <td><code>mdadm</code> shim</td>
          <td><code>files/sbin/mdadm</code></td>
          <td>RAID probes fail or return nonsense.</td>
      </tr>
      <tr>
          <td><code>smartctl</code> shim</td>
          <td><code>files/usr/sbin/smartctl</code></td>
          <td>SMART checks fail or block on missing hardware.</td>
      </tr>
      <tr>
          <td><code>ubnt-tools id</code> shim</td>
          <td><code>files/sbin/ubnt-tools</code></td>
          <td>UniFi OS sees the wrong board identity, and file-output callers can poison persistent identity state with non-UUID text.</td>
      </tr>
      <tr>
          <td>nginx setup hostname patch</td>
          <td><code>/usr/share/unifi-core/http/site-setup.conf</code></td>
          <td>Reverse-proxied setup hostnames redirect to <code>https://unifi</code>.</td>
      </tr>
      <tr>
          <td>PostgreSQL cluster handling</td>
          <td><code>patch_db.sh</code>, <code>dbpermissions.service</code></td>
          <td>Core/app state can land in the wrong place or wrong ownership after restart.</td>
      </tr>
      <tr>
          <td>systemd time sync overrides</td>
          <td>systemd drop-ins</td>
          <td>Time sync can refuse to run in a container.</td>
      </tr>
      <tr>
          <td><code>init_console</code> / <code>init_device</code></td>
          <td>oneshot services</td>
          <td>Device identity and env-driven console settings are not persisted cleanly; a bad <code>anonymous_device_id</code> can survive restarts and block cloud transport.</td>
      </tr>
      <tr>
          <td><code>fix_hosts.service</code></td>
          <td><code>fix_hosts.sh</code></td>
          <td>Docker rewrites <code>/etc/hosts</code>; local resolution can drift.</td>
      </tr>
      <tr>
          <td>apt source repair</td>
          <td><code>fix_apt_ubiquiti_sources.sh</code></td>
          <td>Ubiquiti apt metadata can lag package expectations during app installs or updates.</td>
      </tr>
      <tr>
          <td><code>pg-cluster-upgrade</code> <code>rm -f</code> → <code>rm -rf</code></td>
          <td>sed in <code>os.Dockerfile</code></td>
          <td>Cluster upgrade can leave directories that block re-init.</td>
      </tr>
      <tr>
          <td><code>uled-ctrl</code> stub</td>
          <td>touched executable in <code>os.Dockerfile</code></td>
          <td>LED-control calls can fail on hardware that does not exist.</td>
      </tr>
      <tr>
          <td>Access nginx route installer</td>
          <td><code>install_unifi_access_route.sh</code></td>
          <td>Access can run but fail to appear or proxy correctly through the UniFi OS launcher in some rollback/core combinations.</td>
      </tr>
      <tr>
          <td>Access network timer</td>
          <td><code>unifi-access-network.timer</code></td>
          <td>Wrong management interface or unsafe MTU can return after boot.</td>
      </tr>
  </tbody>
</table>
<p>Some of these patches are ugly.</p>
<p>That is fine.</p>
<p>The dangerous thing is not an ugly patch. The dangerous thing is a silent patch that keeps applying after upstream code moved.</p>
<p>Every sed should fail loudly when its anchor disappears:</p>
<div class="highlight"><pre tabindex="0" style="color:#f8f8f2;background-color:#272822;-moz-tab-size:4;-o-tab-size:4;tab-size:4;-webkit-text-size-adjust:none;"><code class="language-bash" data-lang="bash"><span style="display:flex;"><span><span style="color:#66d9ef">if</span> ! sed -i <span style="color:#e6db74">&#39;s|old-pattern|new-pattern|&#39;</span> /path/to/file; <span style="color:#66d9ef">then</span>
</span></span><span style="display:flex;"><span>    echo <span style="color:#e6db74">&#34;ERROR: expected patch pattern not found&#34;</span>
</span></span><span style="display:flex;"><span>    exit <span style="color:#ae81ff">1</span>
</span></span><span style="display:flex;"><span><span style="color:#66d9ef">fi</span>
</span></span></code></pre></div><p>The repo follows this pattern for the storage and nginx setup patches. That should stay non-negotiable.</p>
<hr>
<h2 id="validation-suite">Validation Suite</h2>
<p>A clean boot is not proof.</p>
<p>For this image, validation has to prove the app, the console service graph, cloud ownership, and device traffic.</p>
<h3 id="1-package-inventory">1. Package inventory</h3>
<div class="highlight"><pre tabindex="0" style="color:#f8f8f2;background-color:#272822;-moz-tab-size:4;-o-tab-size:4;tab-size:4;-webkit-text-size-adjust:none;"><code class="language-bash" data-lang="bash"><span style="display:flex;"><span>docker run --rm unifi-os-docker-arm64:stable dpkg -l | <span style="color:#ae81ff">\
</span></span></span><span style="display:flex;"><span>  grep -iE <span style="color:#e6db74">&#39;unifi|ulp|ucs|uid|uos|ubnt|ucore|ustd|ustate&#39;</span>
</span></span></code></pre></div><p>Expected shape:</p>
<div class="highlight"><pre tabindex="0" style="color:#f8f8f2;background-color:#272822;-moz-tab-size:4;-o-tab-size:4;tab-size:4;-webkit-text-size-adjust:none;"><code class="language-text" data-lang="text"><span style="display:flex;"><span>unifi-core
</span></span><span style="display:flex;"><span>ulp-go
</span></span><span style="display:flex;"><span>ucs-agent
</span></span><span style="display:flex;"><span>uid-agent
</span></span><span style="display:flex;"><span>unifi-identity-update
</span></span><span style="display:flex;"><span>ucore-setup-listener
</span></span><span style="display:flex;"><span>ustate-exporter
</span></span><span style="display:flex;"><span>ustd
</span></span><span style="display:flex;"><span>unifi-access
</span></span><span style="display:flex;"><span>unifi-protect
</span></span><span style="display:flex;"><span>unifi-assets-unvr
</span></span></code></pre></div><p>Do not pin every version in the article. Pin versions in the repo or release notes.</p>
<h3 id="2-systemd-health">2. Systemd health</h3>
<div class="highlight"><pre tabindex="0" style="color:#f8f8f2;background-color:#272822;-moz-tab-size:4;-o-tab-size:4;tab-size:4;-webkit-text-size-adjust:none;"><code class="language-bash" data-lang="bash"><span style="display:flex;"><span>docker exec unifi-os systemctl list-units <span style="color:#ae81ff">\
</span></span></span><span style="display:flex;"><span>  --type<span style="color:#f92672">=</span>service <span style="color:#ae81ff">\
</span></span></span><span style="display:flex;"><span>  --state<span style="color:#f92672">=</span>running <span style="color:#ae81ff">\
</span></span></span><span style="display:flex;"><span>  --no-pager | grep -iE <span style="color:#e6db74">&#39;ulp|ucs|uid|unifi|uos|ustate|ustd|ucore&#39;</span>
</span></span></code></pre></div><p>Expected: the identity and app daemons are running.</p>
<p>Not every hardware daemon matters in Docker. Treat hardware-only failures separately from identity/app failures.</p>
<h3 id="3-cloud-activation-state">3. Cloud activation state</h3>
<div class="highlight"><pre tabindex="0" style="color:#f8f8f2;background-color:#272822;-moz-tab-size:4;-o-tab-size:4;tab-size:4;-webkit-text-size-adjust:none;"><code class="language-bash" data-lang="bash"><span style="display:flex;"><span>docker exec unifi-os systemctl show ulp-go -p StatusText | <span style="color:#ae81ff">\
</span></span></span><span style="display:flex;"><span>  grep -oE <span style="color:#e6db74">&#39;standardActivated&#34;:[a-z]+|ucsAgentVersion&#34;:&#34;[^&#34;]+&#34;&#39;</span>
</span></span></code></pre></div><p>Expected:</p>
<div class="highlight"><pre tabindex="0" style="color:#f8f8f2;background-color:#272822;-moz-tab-size:4;-o-tab-size:4;tab-size:4;-webkit-text-size-adjust:none;"><code class="language-text" data-lang="text"><span style="display:flex;"><span>standardActivated&#34;:true
</span></span><span style="display:flex;"><span>ucsAgentVersion&#34;:&#34;...&#34;
</span></span></code></pre></div><p>If <code>standardActivated</code> stays false after a successful UI-account sign-in, the identity chain is still broken.</p>
<h3 id="4-cloud-connectivity">4. Cloud connectivity</h3>
<div class="highlight"><pre tabindex="0" style="color:#f8f8f2;background-color:#272822;-moz-tab-size:4;-o-tab-size:4;tab-size:4;-webkit-text-size-adjust:none;"><code class="language-bash" data-lang="bash"><span style="display:flex;"><span>docker exec unifi-os grep <span style="color:#e6db74">&#39;^anonymous_device_id:&#39;</span> <span style="color:#ae81ff">\
</span></span></span><span style="display:flex;"><span>  /data/unifi-core/config/settings.yaml
</span></span><span style="display:flex;"><span>
</span></span><span style="display:flex;"><span>docker exec unifi-os ss -tnp state established | grep -E <span style="color:#e6db74">&#39;:(443|8883)&#39;</span>
</span></span><span style="display:flex;"><span>
</span></span><span style="display:flex;"><span>docker exec unifi-os grep -E <span style="color:#ae81ff">\
</span></span></span><span style="display:flex;"><span>  <span style="color:#e6db74">&#39;Device connected to cloud|Successful sync owner|cloud connection not established&#39;</span> <span style="color:#ae81ff">\
</span></span></span><span style="display:flex;"><span>  /data/unifi-core/logs/cloud.log | tail -n <span style="color:#ae81ff">40</span>
</span></span></code></pre></div><p>Expected: <code>anonymous_device_id</code> is a UUID, identity-related processes have outbound TLS sessions to UI cloud endpoints, <code>unifi-core</code> has cloud MQTT/TLS on <code>:8883</code>, and there are no fresh <code>cloud connection not established</code> errors after connection.</p>
<p>This proves connectivity, not supportability.</p>
<h3 id="5-storagestate-grpc-path">5. Storage/state gRPC path</h3>
<div class="highlight"><pre tabindex="0" style="color:#f8f8f2;background-color:#272822;-moz-tab-size:4;-o-tab-size:4;tab-size:4;-webkit-text-size-adjust:none;"><code class="language-bash" data-lang="bash"><span style="display:flex;"><span>docker exec unifi-os ss -ltnp | grep <span style="color:#ae81ff">11052</span>
</span></span><span style="display:flex;"><span>
</span></span><span style="display:flex;"><span>docker exec unifi-os tail -n <span style="color:#ae81ff">20</span> /data/unifi-core/logs/grpc.log
</span></span></code></pre></div><p>Expected: <code>ustate-exporter</code> or the relevant state daemon is listening, and the endless <code>ECONNREFUSED 127.0.0.1:11052</code> loop is gone.</p>
<h3 id="6-access-api-and-device-ports">6. Access API and device ports</h3>
<p>Ubiquiti documents these Access ports:</p>
<table>
  <thead>
      <tr>
          <th style="text-align: right">Port</th>
          <th>Proto</th>
          <th>Purpose</th>
      </tr>
  </thead>
  <tbody>
      <tr>
          <td style="text-align: right"><code>10001</code></td>
          <td>UDP</td>
          <td>device discovery</td>
      </tr>
      <tr>
          <td style="text-align: right"><code>8080</code></td>
          <td>TCP/HTTPS</td>
          <td>device adoption</td>
      </tr>
      <tr>
          <td style="text-align: right"><code>12080</code></td>
          <td>HTTP</td>
          <td>localhost Access API</td>
      </tr>
      <tr>
          <td style="text-align: right"><code>12443</code></td>
          <td>HTTPS</td>
          <td>Access device communication</td>
      </tr>
      <tr>
          <td style="text-align: right"><code>12455</code></td>
          <td>HTTPS</td>
          <td>OpenAPI when enabled</td>
      </tr>
      <tr>
          <td style="text-align: right"><code>12812</code></td>
          <td>MQTT</td>
          <td>Access device messaging</td>
      </tr>
  </tbody>
</table>
<p>Check device sessions:</p>
<div class="highlight"><pre tabindex="0" style="color:#f8f8f2;background-color:#272822;-moz-tab-size:4;-o-tab-size:4;tab-size:4;-webkit-text-size-adjust:none;"><code class="language-bash" data-lang="bash"><span style="display:flex;"><span>docker exec unifi-os ss -tnp state established | grep -E <span style="color:#e6db74">&#39;:(12443|12812)&#39;</span>
</span></span></code></pre></div><p>Check local API:</p>
<div class="highlight"><pre tabindex="0" style="color:#f8f8f2;background-color:#272822;-moz-tab-size:4;-o-tab-size:4;tab-size:4;-webkit-text-size-adjust:none;"><code class="language-bash" data-lang="bash"><span style="display:flex;"><span>docker exec unifi-os curl -sS http://127.0.0.1:12080/api/v2/settings | jq .
</span></span></code></pre></div><h3 id="7-network-identity">7. Network identity</h3>
<p>For local host-network deployments, the expected address is the host LAN address.</p>
<p>For the Vol 1 cloud/macvlan deployment, the expected address is the office-LAN macvlan address:</p>
<div class="highlight"><pre tabindex="0" style="color:#f8f8f2;background-color:#272822;-moz-tab-size:4;-o-tab-size:4;tab-size:4;-webkit-text-size-adjust:none;"><code class="language-bash" data-lang="bash"><span style="display:flex;"><span>docker exec unifi-os ip -brief addr
</span></span><span style="display:flex;"><span>
</span></span><span style="display:flex;"><span>docker exec unifi-os curl -sS http://127.0.0.1:12080/api/v2/info | <span style="color:#ae81ff">\
</span></span></span><span style="display:flex;"><span>  jq -c <span style="color:#e6db74">&#39;{host:.data.host.ip, wan:.data.host.wan_ip}&#39;</span>
</span></span></code></pre></div><p>Expected cloud/macvlan shape:</p>
<div class="highlight"><pre tabindex="0" style="color:#f8f8f2;background-color:#272822;-moz-tab-size:4;-o-tab-size:4;tab-size:4;-webkit-text-size-adjust:none;"><code class="language-json" data-lang="json"><span style="display:flex;"><span>{<span style="color:#f92672">&#34;host&#34;</span>:<span style="color:#e6db74">&#34;10.255.255.200&#34;</span>,<span style="color:#f92672">&#34;wan&#34;</span>:<span style="color:#e6db74">&#34;10.255.255.200&#34;</span>}
</span></span></code></pre></div><p>There should be no Docker bridge <code>172.x.x.x</code> address for UniFi OS to advertise.</p>
<h3 id="8-mtu-repair">8. MTU repair</h3>
<div class="highlight"><pre tabindex="0" style="color:#f8f8f2;background-color:#272822;-moz-tab-size:4;-o-tab-size:4;tab-size:4;-webkit-text-size-adjust:none;"><code class="language-bash" data-lang="bash"><span style="display:flex;"><span>docker exec unifi-os sh -c <span style="color:#e6db74">&#39;
</span></span></span><span style="display:flex;"><span><span style="color:#e6db74">iface=$(ip -o -4 addr show | awk &#34;$4 ~ /^10\\.255\\.255\\.200\\// {print \$2; exit}&#34;)
</span></span></span><span style="display:flex;"><span><span style="color:#e6db74">cat /sys/class/net/$iface/mtu
</span></span></span><span style="display:flex;"><span><span style="color:#e6db74">&#39;</span>
</span></span></code></pre></div><p>Expected cloud/VXLAN shape:</p>
<div class="highlight"><pre tabindex="0" style="color:#f8f8f2;background-color:#272822;-moz-tab-size:4;-o-tab-size:4;tab-size:4;-webkit-text-size-adjust:none;"><code class="language-text" data-lang="text"><span style="display:flex;"><span>1200
</span></span></code></pre></div><p>Small discovery packets are not enough. The acceptance test is discovery, adoption, config push, firmware push, and long-lived MQTT.</p>
<hr>
<h2 id="what-this-design-commits-you-to">What This Design Commits You To</h2>
<p>This is not a small “Access container”.</p>
<p>It is closer to this:</p>
<div class="highlight"><pre tabindex="0" style="color:#f8f8f2;background-color:#272822;-moz-tab-size:4;-o-tab-size:4;tab-size:4;-webkit-text-size-adjust:none;"><code class="language-text" data-lang="text"><span style="display:flex;"><span>UNVR-shaped UniFi OS userspace inside Docker
</span></span></code></pre></div><p>That has consequences.</p>
<p>You now depend on a pinned UNVR firmware family. Every UniFi OS bump means extracting packages again, rebuilding, and checking whether package names or minified JavaScript anchors moved.</p>
<p>You inherit services you may not care about. Some are required by Identity or Access. Some exist because firmware packages depend on them. Some expect hardware that a container will never have.</p>
<p>You must disable UniFi OS and app auto-updates after setup. A normal console update can replace patched files, move dependencies, or make the sed anchors invalid.</p>
<p>You must keep backups of all persistent paths:</p>
<div class="highlight"><pre tabindex="0" style="color:#f8f8f2;background-color:#272822;-moz-tab-size:4;-o-tab-size:4;tab-size:4;-webkit-text-size-adjust:none;"><code class="language-text" data-lang="text"><span style="display:flex;"><span>/srv
</span></span><span style="display:flex;"><span>/data
</span></span><span style="display:flex;"><span>/persistent
</span></span></code></pre></div><p>Or in the repo's default local compose layout:</p>
<div class="highlight"><pre tabindex="0" style="color:#f8f8f2;background-color:#272822;-moz-tab-size:4;-o-tab-size:4;tab-size:4;-webkit-text-size-adjust:none;"><code class="language-text" data-lang="text"><span style="display:flex;"><span>./storage/srv
</span></span><span style="display:flex;"><span>./storage/data
</span></span><span style="display:flex;"><span>./storage/persistent
</span></span></code></pre></div><p>You must treat downgrades as unsafe. Access, Protect, PostgreSQL clusters, device firmware, and UniFi Core state do not all roll back cleanly.</p>
<p>The practical split:</p>
<table>
  <thead>
      <tr>
          <th>Failure</th>
          <th>Treat as</th>
      </tr>
  </thead>
  <tbody>
      <tr>
          <td><code>rpsd</code> failed because no RPS hardware exists</td>
          <td>Usually cosmetic; mask or ignore after confirming no dependency.</td>
      </tr>
      <tr>
          <td><code>ustate-exporter</code> missing on <code>127.0.0.1:11052</code></td>
          <td>Real image defect.</td>
      </tr>
      <tr>
          <td><code>ucs-agent</code> missing</td>
          <td>Real image defect.</td>
      </tr>
      <tr>
          <td><code>uid-agent</code> missing</td>
          <td>Real image defect.</td>
      </tr>
      <tr>
          <td><code>standardActivated: false</code> after cloud sign-in</td>
          <td>Real cloud/identity defect.</td>
      </tr>
      <tr>
          <td>Docker bridge IP appears as controller identity</td>
          <td>Real network/design defect.</td>
      </tr>
      <tr>
          <td>Access device sessions on <code>12443</code> / <code>12812</code> are missing</td>
          <td>Real device-management defect.</td>
      </tr>
      <tr>
          <td><code>ACCESS_DEVICE_MTU</code> not applied on VXLAN/IPSec path</td>
          <td>Real transport defect.</td>
      </tr>
  </tbody>
</table>
<p>The final lesson is simple:</p>
<blockquote>
<p>A UniFi application is not the same thing as a UniFi console.</p>
</blockquote>
<p>Vol 1 made the controller look local to the door hardware.</p>
<p>Vol 2 made the container look complete enough that UniFi Core, Identity, Access, and UI cloud could agree on the same console.</p>
<hr>
<h2 id="references">References</h2>
<ul>
<li><a href="/posts/2026/05/24/unifi-access-vxlan-over-ipsec/">Vol 1: Doors Don't Route</a></li>
<li><a href="https://github.com/kuzaxak/unifi-access-unvr-docker-arm64">Source repo: kuzaxak/unifi-access-unvr-docker-arm64</a></li>
<li><a href="https://github.com/dciancu/unifi-protect-unvr-docker-arm64">Parent repo: dciancu/unifi-protect-unvr-docker-arm64</a></li>
<li><a href="https://help.ui.com/hc/en-us/articles/34210126298775-Self-Hosting-UniFi">Ubiquiti: Self-Hosting UniFi</a></li>
<li><a href="https://help.ui.com/hc/en-us/articles/22230509487639-UniFi-Consoles-with-UniFi-Access-Support">Ubiquiti: UniFi Consoles with UniFi Access Support</a></li>
<li><a href="https://help.ui.com/hc/en-us/articles/17452334269975-Getting-Started-with-UniFi-Access">Ubiquiti: Getting Started with UniFi Access</a></li>
<li><a href="https://github.com/ConnorsApps/unifi-os-helm/blob/main/SERVICES.md">ConnorsApps/unifi-os-helm SERVICES.md</a></li>
</ul>
]]></content:encoded>
    </item>
    
    <item>
      <title>Doors Don&#39;t Route: Cloud-Hosting UniFi Access with VXLAN-over-IPSec</title>
      <link>https://kuzaxak.dev/posts/2026/05/24/unifi-access-vxlan-over-ipsec/</link>
      <pubDate>Sun, 24 May 2026 00:00:00 +0000</pubDate>
      
      <guid>https://kuzaxak.dev/posts/2026/05/24/unifi-access-vxlan-over-ipsec/</guid>
      <description>How UniFi Access adoption forced an AWS-hosted controller into the office L2 domain, and what broke along the way: DHCP option 43, VXLAN, MTU, macvlan, and Docker port publishing.</description>
      <content:encoded><![CDATA[<h2 id="preamble-the-key-was-the-first-problem">Preamble: The Key Was the First Problem</h2>
<p>This did not start as a VXLAN project.</p>
<p>We got a small office with one physical ABLOY key. One key is fine until more than one person needs to enter the office, open it for a delivery, or get in after hours. The obvious fix was to copy the key a few times.</p>
<p>Then we checked the price.</p>
<p>Making four proper copies of that ABLOY key was expensive enough that the UniFi Access G3 Starter Kit started to look reasonable. Instead of paying for more metal keys, we could get a door hub, a reader, NFC cards, mobile unlocks, logs, and something that was at least possible to manage like the rest of the office infrastructure.</p>
<p>So the plan was simple:</p>
<ol>
<li>install the G3 kit on the office door;</li>
<li>run the Access controller somewhere local;</li>
<li>add users;</li>
<li>stop thinking about keys.</li>
</ol>
<p>That plan lasted until setup.</p>
<p>The kit was not just a reader and a relay. UniFi Access expects an Access application running on a supported UniFi console. We did not have one on-site, and waiting for more hardware was not attractive. The door hardware was already on the desk, the office still had one key, and this was supposed to be a quick fix.</p>
<p>So the lab plan became: run UniFi Access in Docker.</p>
<p>At first this sounded like an application problem. Extract the right packages, make the services start, expose the web UI, and adopt the devices. The existing UniFi Protect Docker work suggested that this was at least possible.</p>
<p>It turned out the hard part was not starting the application.</p>
<p>The hard part was making a Docker container look, from the door controller’s point of view, like a real UniFi console sitting on the same office LAN.</p>
<p>That is where the story actually begins: one expensive key copy quote, one unsupported Docker controller, and a door system that cared much more about Layer 2 discovery than I expected.</p>
<h2 id="the-first-constraint-access-wanted-a-console">The First Constraint: Access Wanted a Console</h2>
<p>With that context, the first target was obvious: keep everything local. The office already had a Synology NAS, Docker, Portainer, Traefik, and the rest of the small-service stack. If Access could run there, the network would stay boring.</p>
<p>Ubiquiti officially does not support this. They sell consoles with Access pre-installed and list the compatible console models in their own documentation (<a href="https://help.ui.com/hc/en-us/articles/22230509487639-UniFi-Consoles-with-UniFi-Access-Support">docs</a>). The <a href="https://github.com/dciancu/unifi-protect-unvr-docker-arm64">dciancu/unifi-protect-unvr-docker-arm64</a> repository proves the application layer can be lifted off a console image; nobody has done it for Access, and the device adoption story is materially different from Protect.</p>
<p>This post is the path from that wrong assumption to the final shape: UniFi Access running on AWS arm64, attached to the office broadcast domain through VXLAN-over-IPSec and a single Docker macvlan interface.</p>
<p>The final design has three layers:</p>
<ol>
<li>Underlay: Mikrotik to AWS IPSec.</li>
<li>L2 extension: VXLAN over that IPSec tunnel.</li>
<li>Container identity: UniFi Access attached only to the office bridge through macvlan.</li>
</ol>
<p>Most of the debugging pain came from confusing those layers. A ping proved the underlay. UDP discovery proved L2. A stable reader proved MTU and management traffic. The web UI proved only the browser path.</p>
<p>The examples below use real values from the deployment, sanitized only where they would not help a reader reproduce the design.</p>
<hr>
<h2 id="what-adoption-actually-means-for-access">What &quot;Adoption&quot; Actually Means for Access</h2>
<p>UniFi Access is not purely L2 forever. Ubiquiti documents the device ports as <code>10001/udp</code> for discovery, <code>8080</code> for adoption, <code>12443</code> for HTTPS device communication, and <code>12812</code> for MQTT messaging. The same page says Access Control Hubs communicate with the Access application over IP, while readers and hubs must stay on the same L2 network (<a href="https://help.ui.com/hc/en-us/articles/17452334269975-Getting-Started-with-UniFi-Access">docs</a>).</p>
<p>The part that broke this unsupported Docker deployment was the first discovery/adoption path. In this setup, the Hub Mini only offered itself to a controller that answered from the same discovery domain.</p>
<p>That sounds the same as Network adoption, but it is not. UniFi Network firmware honors <strong>DHCP option 43</strong> as a fallback (a controller IP encoded as <code>01 04 &lt;ip-bytes&gt;</code>), and the devices have a <code>set-inform</code> CLI over SSH. Access devices have neither. A community-confirmed <a href="https://www.reddit.com/r/UNIFI/comments/16v6js7/is_dhcp_option_43_supported_by_unifi_door_access/">thread</a> documents this directly:</p>
<blockquote>
<p>&quot;Access needs to see the other device in the same VLAN (broadcast domain).&quot;</p>
</blockquote>
<p>I tested option 43 anyway. The MikroTik sent it, the lease included the encoded controller IP, the Hub Mini ignored it. No <code>set-inform</code>, no SSH, no useful fallback for this case. The device only adopted once the controller could participate in the discovery exchange.</p>
<p>That single fact controls every networking decision in this story.</p>
<p><p>
  <img src="https://kuzaxak.dev/posts/2026/05/24/unifi-access-vxlan-over-ipsec/access-adoption.png" alt="Access discovery adoption flow"  />
</p>
</p>
<details>
  <summary>mermaid</summary>
  <div class="highlight"><pre tabindex="0" style="color:#f8f8f2;background-color:#272822;-moz-tab-size:4;-o-tab-size:4;tab-size:4;-webkit-text-size-adjust:none;"><code class="language-fallback" data-lang="fallback"><span style="display:flex;"><span>sequenceDiagram
</span></span><span style="display:flex;"><span>    participant Device as G3 Door Hub Mini&lt;br/&gt;10.255.255.230
</span></span><span style="display:flex;"><span>    participant LAN as Office LAN&lt;br/&gt;broadcast domain
</span></span><span style="display:flex;"><span>    participant Controller as UniFi Access&lt;br/&gt;Controller
</span></span><span style="display:flex;"><span>
</span></span><span style="display:flex;"><span>    Note over Device,Controller: Boots fresh from box
</span></span><span style="display:flex;"><span>
</span></span><span style="display:flex;"><span>    Device-&gt;&gt;LAN: UDP broadcast&lt;br/&gt;10.255.255.255:10001&lt;br/&gt;&#34;I am MAC 84:78:48:...&#34;
</span></span><span style="display:flex;"><span>    LAN--&gt;&gt;Controller: ✓ delivered (same L2)
</span></span><span style="display:flex;"><span>    Controller-&gt;&gt;Device: UDP unicast&lt;br/&gt;:10001 &#34;I am your controller&#34;
</span></span><span style="display:flex;"><span>    Device-&gt;&gt;Controller: TCP 12443 + MQTT 12812&lt;br/&gt;establish session
</span></span><span style="display:flex;"><span>
</span></span><span style="display:flex;"><span>    Note over Device,Controller: Adoption complete
</span></span><span style="display:flex;"><span>
</span></span><span style="display:flex;"><span>    rect rgb(255, 230, 230)
</span></span><span style="display:flex;"><span>        Note over Device,LAN: ❌ Broadcast does NOT cross&lt;br/&gt;L3 routers / IPSec tunnels
</span></span><span style="display:flex;"><span>        Note over Device,LAN: ❌ No DHCP option 43 fallback&lt;br/&gt;(unlike UniFi Network)
</span></span><span style="display:flex;"><span>        Note over Device,LAN: ❌ No SSH / set-inform CLI
</span></span><span style="display:flex;"><span>    end
</span></span></code></pre></div>
</details>

<hr>
<h2 id="dead-end-1-the-local-nas-was-the-wrong-cpu">Dead End 1: The Local NAS Was the Wrong CPU</h2>
<p>The first idea was the obvious one. We had a DSM 7.2.2 box on the office LAN, with Portainer, Traefik, the existing service mesh, and plenty of CPU. Why deploy anywhere else?</p>
<p>Ubiquiti only publishes Access for arm64. Synology is x86_64. The expected workaround was QEMU user-mode emulation through <code>binfmt_misc</code>:</p>
<div class="highlight"><pre tabindex="0" style="color:#f8f8f2;background-color:#272822;-moz-tab-size:4;-o-tab-size:4;tab-size:4;-webkit-text-size-adjust:none;"><code class="language-bash" data-lang="bash"><span style="display:flex;"><span>docker run --privileged --rm tonistiigi/binfmt --install arm64
</span></span><span style="display:flex;"><span>docker run --platform<span style="color:#f92672">=</span>linux/arm64 alpine uname -m
</span></span><span style="display:flex;"><span><span style="color:#75715e"># expected: aarch64</span>
</span></span></code></pre></div><p>The first command failed:</p>
<div class="highlight"><pre tabindex="0" style="color:#f8f8f2;background-color:#272822;-moz-tab-size:4;-o-tab-size:4;tab-size:4;-webkit-text-size-adjust:none;"><code class="language-fallback" data-lang="fallback"><span style="display:flex;"><span>installing: arm64 cannot register &#34;/usr/bin/qemu-aarch64&#34; to
</span></span><span style="display:flex;"><span>/proc/sys/fs/binfmt_misc/register: write /proc/sys/fs/binfmt_misc/register:
</span></span><span style="display:flex;"><span>invalid argument
</span></span><span style="display:flex;"><span>
</span></span><span style="display:flex;"><span>supported:
</span></span><span style="display:flex;"><span>  - linux/amd64
</span></span><span style="display:flex;"><span>  - linux/amd64/v2
</span></span><span style="display:flex;"><span>  - linux/amd64/v3
</span></span><span style="display:flex;"><span>  - linux/386
</span></span></code></pre></div><p>Probing the kernel directly showed why:</p>
<div class="highlight"><pre tabindex="0" style="color:#f8f8f2;background-color:#272822;-moz-tab-size:4;-o-tab-size:4;tab-size:4;-webkit-text-size-adjust:none;"><code class="language-bash" data-lang="bash"><span style="display:flex;"><span>docker run --rm --privileged --platform<span style="color:#f92672">=</span>linux/amd64 alpine <span style="color:#ae81ff">\
</span></span></span><span style="display:flex;"><span>  sh -c <span style="color:#e6db74">&#39;ls -la /proc/sys/fs/binfmt_misc/; cat /proc/sys/fs/binfmt_misc/status&#39;</span>
</span></span><span style="display:flex;"><span>
</span></span><span style="display:flex;"><span><span style="color:#75715e"># dr-xr-xr-x    2 root     root             0 May 23 19:36 .</span>
</span></span><span style="display:flex;"><span><span style="color:#75715e"># (no &#39;register&#39; or &#39;status&#39; files)</span>
</span></span><span style="display:flex;"><span><span style="color:#75715e"># cat: can&#39;t open &#39;/proc/sys/fs/binfmt_misc/status&#39;: No such file or directory</span>
</span></span></code></pre></div><p>The <code>binfmt_misc</code> filesystem is mounted, but it is read-only and the <code>register</code>/<code>status</code> interface is gone. Synology's kernel (4.4.302+) is built with <code>binfmt_misc</code> deliberately locked down. No userspace can register a new format, with or without privileges, with or without IAM equivalents.</p>
<p>VMM was the next instinct: run a Linux ARM VM on the Synology. VMM uses KVM, which can only virtualize the host architecture. Without nested full-system QEMU (which DSM VMM does not expose), an arm64 VM on Intel Synology is not an option.</p>
<p>The Synology had to come out of the picture as the controller host.</p>
<hr>
<h2 id="dead-end-2-a-working-l3-tunnel-still-did-not-carry-discovery">Dead End 2: A Working L3 Tunnel Still Did Not Carry Discovery</h2>
<p>The next plan was cleaner architecturally. We have AWS, we have arm64 instances (<code>t4g.medium</code>), and we have a Mikrotik RB5009 with a static public IP that already terminates one IPSec tunnel.</p>
<p>The design looked like this:</p>
<p><p>
  <img src="https://kuzaxak.dev/posts/2026/05/24/unifi-access-vxlan-over-ipsec/ipsec-l3-design.png" alt="L3 IPsec design that did not solve adoption"  />
</p>
</p>
<details>
  <summary>mermaid</summary>
  <div class="highlight"><pre tabindex="0" style="color:#f8f8f2;background-color:#272822;-moz-tab-size:4;-o-tab-size:4;tab-size:4;-webkit-text-size-adjust:none;"><code class="language-fallback" data-lang="fallback"><span style="display:flex;"><span>flowchart LR
</span></span><span style="display:flex;"><span>    subgraph Office[&#34;Office LAN 10.255.255.0/24&#34;]
</span></span><span style="display:flex;"><span>        Hub[Door Hub Mini&lt;br/&gt;10.255.255.230]
</span></span><span style="display:flex;"><span>        Reader[G3 Reader&lt;br/&gt;10.255.255.229]
</span></span><span style="display:flex;"><span>        Mikrotik[Mikrotik RB5009&lt;br/&gt;10.255.255.1]
</span></span><span style="display:flex;"><span>    end
</span></span><span style="display:flex;"><span>
</span></span><span style="display:flex;"><span>    subgraph AWS[&#34;AWS eu-north-1&#34;]
</span></span><span style="display:flex;"><span>        Controller[unifi-access&lt;br/&gt;EC2 t4g.medium&lt;br/&gt;172.31.26.37]
</span></span><span style="display:flex;"><span>    end
</span></span><span style="display:flex;"><span>
</span></span><span style="display:flex;"><span>    Hub --&gt; Mikrotik
</span></span><span style="display:flex;"><span>    Reader --&gt; Mikrotik
</span></span><span style="display:flex;"><span>    Mikrotik -- &#34;IPSec IKEv2&lt;br/&gt;10.255.255.0/24 ↔ 172.31.26.37/32&#34; --&gt; Controller
</span></span></code></pre></div>
</details>

<p><p>
  <img src="https://kuzaxak.dev/posts/2026/05/24/unifi-access-vxlan-over-ipsec/option43-fails.png" alt="DHCP option 43 does not replace L2 Access discovery"  />
</p>
</p>
<details>
  <summary>mermaid</summary>
  <div class="highlight"><pre tabindex="0" style="color:#f8f8f2;background-color:#272822;-moz-tab-size:4;-o-tab-size:4;tab-size:4;-webkit-text-size-adjust:none;"><code class="language-fallback" data-lang="fallback"><span style="display:flex;"><span>sequenceDiagram
</span></span><span style="display:flex;"><span>    participant Hub as Door Hub Mini&lt;br/&gt;10.255.255.230
</span></span><span style="display:flex;"><span>    participant MT as Mikrotik&lt;br/&gt;10.255.255.1
</span></span><span style="display:flex;"><span>    participant Tunnel as IPSec L3 Tunnel
</span></span><span style="display:flex;"><span>    participant EC2 as Controller&lt;br/&gt;172.31.26.37
</span></span><span style="display:flex;"><span>
</span></span><span style="display:flex;"><span>    Note over Hub: DHCP lease arrives&lt;br/&gt;with option 43 = 172.31.26.37
</span></span><span style="display:flex;"><span>
</span></span><span style="display:flex;"><span>    rect rgb(255, 230, 230)
</span></span><span style="display:flex;"><span>        Hub-&gt;&gt;Hub: Ignore option 43&lt;br/&gt;(Access firmware does not honor it)
</span></span><span style="display:flex;"><span>    end
</span></span><span style="display:flex;"><span>
</span></span><span style="display:flex;"><span>    Hub-&gt;&gt;MT: Broadcast UDP 10001&lt;br/&gt;dst: 255.255.255.255
</span></span><span style="display:flex;"><span>    Note over MT: Broadcasts do not cross L3
</span></span><span style="display:flex;"><span>    MT--xTunnel: ❌ Dropped at routing
</span></span><span style="display:flex;"><span>
</span></span><span style="display:flex;"><span>    Note over Hub,EC2: Controller never learns about the Hub Mini.&lt;br/&gt;Tunnel works for ICMP/TCP unicast, but useless&lt;br/&gt;for L2 adoption broadcasts.
</span></span></code></pre></div>
</details>

<p>The IPSec tunnel came up cleanly. From Mikrotik with an explicit source address, pings reached the controller at 10ms:</p>
<div class="highlight"><pre tabindex="0" style="color:#f8f8f2;background-color:#272822;-moz-tab-size:4;-o-tab-size:4;tab-size:4;-webkit-text-size-adjust:none;"><code class="language-fallback" data-lang="fallback"><span style="display:flex;"><span>SEQ HOST                                     SIZE TTL TIME       STATUS
</span></span><span style="display:flex;"><span>  0 172.31.26.37                               56  64 10ms260us
</span></span><span style="display:flex;"><span>  1 172.31.26.37                               56  64 10ms404us
</span></span><span style="display:flex;"><span>sent=3 received=3 packet-loss=0%
</span></span></code></pre></div><p>I configured DHCP option 43 on the Mikrotik to hand out the controller IP to UniFi devices:</p>
<div class="highlight"><pre tabindex="0" style="color:#f8f8f2;background-color:#272822;-moz-tab-size:4;-o-tab-size:4;tab-size:4;-webkit-text-size-adjust:none;"><code class="language-fallback" data-lang="fallback"><span style="display:flex;"><span>/ip dhcp-server option add name=unifi-controller code=43 value=0x0104ac1f1a25
</span></span><span style="display:flex;"><span>/ip dhcp-server network set [find address=10.255.255.0/24] dhcp-option=unifi-controller
</span></span></code></pre></div><p><code>0x01 04 ac 1f 1a 25</code> decodes as sub-option 1, length 4, controller IP <code>172.31.26.37</code>. The lease confirmed the option was offered. The Hub Mini took the lease and ignored the option.</p>
<p>Several controller-side captures showed the same thing: the Hub Mini was reachable, the IPSec tunnel was healthy, and no device traffic ever reached the controller. The community thread was right.</p>
<p>The discovery model was the architectural constraint, not a knob to tune. Either the controller had to be on the same L2 broadcast domain as the door hardware, or the broadcast domain had to be extended to where the controller was.</p>
<hr>
<h2 id="the-shape-that-matched-the-protocol-extend-l2-do-not-fight-it">The Shape That Matched the Protocol: Extend L2, Do Not Fight It</h2>
<p>The fix had to bring the controller container into the office broadcast domain at L2, while keeping its lifecycle in AWS. The shape that worked:</p>
<ol>
<li>Keep the IPSec tunnel as the underlay - it already exists, it is encrypted, and it terminates at known endpoints.</li>
<li>Build a VXLAN tunnel between Mikrotik and the EC2 host, with VXLAN packets riding inside the IPSec tunnel.</li>
<li>Bridge the VXLAN endpoint into the office LAN on the Mikrotik side, and into a Linux bridge on the EC2 side.</li>
<li>Attach the docker container to the Linux bridge through a <code>macvlan</code> network, giving it a real <code>10.255.255.x</code> address.</li>
</ol>
<p><p>
  <img src="https://kuzaxak.dev/posts/2026/05/24/unifi-access-vxlan-over-ipsec/vxlan-over-ipsec.png" alt="VXLAN over IPsec architecture"  />
</p>
</p>
<details>
  <summary>mermaid</summary>
  <div class="highlight"><pre tabindex="0" style="color:#f8f8f2;background-color:#272822;-moz-tab-size:4;-o-tab-size:4;tab-size:4;-webkit-text-size-adjust:none;"><code class="language-fallback" data-lang="fallback"><span style="display:flex;"><span>flowchart TB
</span></span><span style="display:flex;"><span>    subgraph Office[&#34;Office LAN 10.255.255.0/24&#34;]
</span></span><span style="display:flex;"><span>        Hub[Door Hub Mini&lt;br/&gt;10.255.255.230]
</span></span><span style="display:flex;"><span>        Reader[G3 Reader&lt;br/&gt;10.255.255.229]
</span></span><span style="display:flex;"><span>        Synology[Synology Traefik&lt;br/&gt;10.255.255.245]
</span></span><span style="display:flex;"><span>        Mikrotik[Mikrotik bridge&lt;br/&gt;10.255.255.1]
</span></span><span style="display:flex;"><span>        VXLANmt[vxlan-aws&lt;br/&gt;VNI 100, UDP 4789]
</span></span><span style="display:flex;"><span>        Mikrotik --- VXLANmt
</span></span><span style="display:flex;"><span>    end
</span></span><span style="display:flex;"><span>
</span></span><span style="display:flex;"><span>    subgraph Tunnel[&#34;IPSec ESP / UDP 4500&#34;]
</span></span><span style="display:flex;"><span>        ipsec[encrypted underlay]
</span></span><span style="display:flex;"><span>    end
</span></span><span style="display:flex;"><span>
</span></span><span style="display:flex;"><span>    subgraph AWS[&#34;AWS eu-north-1&#34;]
</span></span><span style="display:flex;"><span>        ens5[EC2 ens5&lt;br/&gt;172.31.26.37]
</span></span><span style="display:flex;"><span>        vxlan0[vxlan0&lt;br/&gt;VNI 100, dev=ens5]
</span></span><span style="display:flex;"><span>        broffice[br-office&lt;br/&gt;10.255.255.50]
</span></span><span style="display:flex;"><span>        macvlan[docker macvlan&lt;br/&gt;parent=br-office]
</span></span><span style="display:flex;"><span>        Container[unifi-access container&lt;br/&gt;eth0=10.255.255.200]
</span></span><span style="display:flex;"><span>        ens5 --- vxlan0
</span></span><span style="display:flex;"><span>        vxlan0 --- broffice
</span></span><span style="display:flex;"><span>        broffice --- macvlan
</span></span><span style="display:flex;"><span>        macvlan --- Container
</span></span><span style="display:flex;"><span>    end
</span></span><span style="display:flex;"><span>
</span></span><span style="display:flex;"><span>    VXLANmt --&gt; ipsec
</span></span><span style="display:flex;"><span>    ipsec --&gt; ens5
</span></span><span style="display:flex;"><span>
</span></span><span style="display:flex;"><span>    Hub -. &#34;broadcast UDP 10001&#34; .-&gt; Mikrotik
</span></span><span style="display:flex;"><span>    Reader -. &#34;broadcast UDP 10001&#34; .-&gt; Mikrotik
</span></span><span style="display:flex;"><span>    Container -. &#34;discovery probes&lt;br/&gt;233.89.188.1:10001&lt;br/&gt;255.255.255.255:10001&#34; .-&gt; macvlan
</span></span></code></pre></div>
</details>

<p>The mental model that helped was this:</p>
<blockquote>
<p>The controller does not need to be in AWS. It needs to be unreachable from anywhere except the office LAN, and to look to the door hardware exactly like a host on <code>10.255.255.0/24</code>.</p>
</blockquote>
<p>The L2 bridge made that true. AWS became invisible from the protocol's point of view.</p>
<hr>
<h2 id="about-the-controller-image">About the Controller Image</h2>
<p>The controller image is not the main subject of this post. It deserves its own write-up.</p>
<p>For context, I built it from arm64 UniFi OS packages using the same broad approach as the existing UniFi Protect container work. Access needed several console-assumption patches: storage reporting, pre-setup hostname handling, and console identity. The final image runs UniFi Core, UniFi Access, and the required PostgreSQL services inside a Debian arm64 systemd container.</p>
<p>The important networking fact is this: the container could run, and it could be attached to a Docker macvlan network with <code>10.255.255.200/24</code>. Everything below is about making that address behave like a real office LAN host.</p>
<p>The image path is a separate article. That write-up is now <a href="/posts/2026/05/26/unifi-access-unvr-daemons/">Vol 2: Sixty Seconds to Nothing</a>. It covers making UniFi Access believe Docker is a console: firmware extraction, package repacking, <code>ustorage</code> and <code>mdadm</code> shims, the <code>UNVR</code> identity trick, nginx setup patches, PostgreSQL clusters, and the build-time guards that keep future firmware upgrades honest.</p>
<p><p>
  <img src="https://kuzaxak.dev/posts/2026/05/24/unifi-access-vxlan-over-ipsec/image-build.png" alt="Container image build flow"  />
</p>
</p>
<details>
  <summary>mermaid</summary>
  <div class="highlight"><pre tabindex="0" style="color:#f8f8f2;background-color:#272822;-moz-tab-size:4;-o-tab-size:4;tab-size:4;-webkit-text-size-adjust:none;"><code class="language-gdscript3" data-lang="gdscript3"><span style="display:flex;"><span>flowchart LR
</span></span><span style="display:flex;"><span>    A[fw<span style="color:#f92672">-</span>update<span style="color:#f92672">.</span>ubnt<span style="color:#f92672">.</span>com<span style="color:#f92672">&lt;</span>br<span style="color:#f92672">/&gt;</span>UCG<span style="color:#f92672">-</span>Max <span style="color:#f92672">.</span>bin] <span style="color:#f92672">--&gt;|</span>wget<span style="color:#f92672">|</span> B[binwalk <span style="color:#f92672">-</span>e]
</span></span><span style="display:flex;"><span>    B <span style="color:#f92672">--&gt;</span> C[squashfs<span style="color:#f92672">-</span>root]
</span></span><span style="display:flex;"><span>    C <span style="color:#f92672">--&gt;|</span>dpkg<span style="color:#f92672">-</span>repack<span style="color:#f92672">|</span> D[<span style="color:#f92672">*.</span>deb per package]
</span></span><span style="display:flex;"><span>    D <span style="color:#f92672">--&gt;</span> E[base layer<span style="color:#f92672">&lt;</span>br<span style="color:#f92672">/&gt;</span>unifi<span style="color:#f92672">-</span>core, ulp<span style="color:#f92672">-</span>go,<span style="color:#f92672">&lt;</span>br<span style="color:#f92672">/&gt;</span>uos<span style="color:#f92672">*</span>, ubnt<span style="color:#f92672">-</span>tools,<span style="color:#f92672">&lt;</span>br<span style="color:#f92672">/&gt;</span>unifi<span style="color:#f92672">-</span>assets<span style="color:#f92672">-</span>ucgmax,<span style="color:#f92672">&lt;</span>br<span style="color:#f92672">/&gt;</span>node<span style="color:#f92672">*</span>]
</span></span><span style="display:flex;"><span>    E <span style="color:#f92672">--&gt;</span> F[debian:<span style="color:#ae81ff">11</span> arm64<span style="color:#f92672">&lt;</span>br<span style="color:#f92672">/&gt;+</span> apt<span style="color:#f92672">-</span>get install]
</span></span><span style="display:flex;"><span>    F <span style="color:#f92672">--&gt;</span> G[unifi<span style="color:#f92672">-</span>access<span style="color:#f92672">.</span>deb<span style="color:#f92672">&lt;</span>br<span style="color:#f92672">/&gt;+</span> ms<span style="color:#f92672">.</span>deb<span style="color:#f92672">&lt;</span>br<span style="color:#f92672">/&gt;+</span> unifi<span style="color:#f92672">-</span>user<span style="color:#f92672">-</span>assets<span style="color:#f92672">.</span>deb<span style="color:#f92672">&lt;</span>br<span style="color:#f92672">/&gt;+</span> unifi<span style="color:#f92672">-</span>face<span style="color:#f92672">-</span>shared<span style="color:#f92672">-</span>lib<span style="color:#f92672">.</span>deb]
</span></span><span style="display:flex;"><span>    G <span style="color:#f92672">--&gt;</span> H[final image]
</span></span></code></pre></div>
</details>

<hr>
<h2 id="building-the-l2-extension">Building the L2 Extension</h2>
<p>The VXLAN tunnel uses UDP 4789. With both endpoints having explicit unicast addresses, the inner Ethernet frames travel inside outer UDP packets between <code>10.255.255.1</code> (Mikrotik LAN side) and <code>172.31.26.37</code> (EC2 private IP). Linux VXLAN is UDP-based, and the kernel documentation shows the IANA-assigned <code>4789</code> port in its example (<a href="https://docs.kernel.org/networking/vxlan.html">docs</a>).</p>
<p>The existing IPSec policy is <code>10.255.255.0/24 ↔ 172.31.26.37/32</code>. VXLAN outer packets between those two addresses match the policy, so they ride encrypted inside ESP. No new IPSec policy was needed; the L2 extension reuses the existing L3 tunnel as its transport.</p>
<h3 id="mikrotik-side">Mikrotik side</h3>
<div class="highlight"><pre tabindex="0" style="color:#f8f8f2;background-color:#272822;-moz-tab-size:4;-o-tab-size:4;tab-size:4;-webkit-text-size-adjust:none;"><code class="language-fallback" data-lang="fallback"><span style="display:flex;"><span>/interface vxlan
</span></span><span style="display:flex;"><span>add name=vxlan-aws vni=100 port=4789 mtu=1380 local-address=10.255.255.1
</span></span><span style="display:flex;"><span>
</span></span><span style="display:flex;"><span>/interface vxlan vteps
</span></span><span style="display:flex;"><span>add interface=vxlan-aws remote-ip=172.31.26.37
</span></span><span style="display:flex;"><span>
</span></span><span style="display:flex;"><span>/interface bridge port
</span></span><span style="display:flex;"><span>add bridge=bridge interface=vxlan-aws
</span></span><span style="display:flex;"><span>
</span></span><span style="display:flex;"><span>/ip firewall filter
</span></span><span style="display:flex;"><span>add chain=input action=accept protocol=udp dst-port=4789 \
</span></span><span style="display:flex;"><span>    src-address=172.31.26.37/32 place-before=0 \
</span></span><span style="display:flex;"><span>    comment=&#34;VXLAN from AWS via IPSec&#34;
</span></span></code></pre></div><p>The <code>local-address=10.255.255.1</code> is important. Without it, Mikrotik picks the source IP for VXLAN outer packets based on the egress interface (the WAN), and the resulting tuple does not match the IPSec policy. Forcing the LAN IP keeps everything inside the policy's transform set.</p>
<h3 id="ec2-side">EC2 side</h3>
<p>The first attempt looked like this:</p>
<div class="highlight"><pre tabindex="0" style="color:#f8f8f2;background-color:#272822;-moz-tab-size:4;-o-tab-size:4;tab-size:4;-webkit-text-size-adjust:none;"><code class="language-bash" data-lang="bash"><span style="display:flex;"><span>ip link add vxlan0 type vxlan id <span style="color:#ae81ff">100</span> <span style="color:#ae81ff">\
</span></span></span><span style="display:flex;"><span>    remote 10.255.255.1 local 172.31.26.37 dstport <span style="color:#ae81ff">4789</span> nolearning
</span></span><span style="display:flex;"><span>ip link set vxlan0 master br-office
</span></span></code></pre></div><p>VXLAN packets <em>arrived</em> from Mikrotik. Nothing went the other way. The kernel was silently dropping outbound encapsulation.</p>
<p>Two issues:</p>
<p><strong><code>nolearning</code> plus an empty FDB.</strong> With learning off and no manual entries, the kernel had no idea where to send broadcasts and unknown unicast. The fix is to add a default flood entry:</p>
<div class="highlight"><pre tabindex="0" style="color:#f8f8f2;background-color:#272822;-moz-tab-size:4;-o-tab-size:4;tab-size:4;-webkit-text-size-adjust:none;"><code class="language-bash" data-lang="bash"><span style="display:flex;"><span>bridge fdb append 00:00:00:00:00:00 dev vxlan0 dst 10.255.255.1
</span></span></code></pre></div><p><strong>No explicit <code>dev</code>.</strong> Without <code>dev ens5</code>, the kernel does a generic route lookup for the outer destination <code>10.255.255.1</code> and finds that <code>10.255.255.0/24</code> is reachable through <code>br-office</code>. The VXLAN outer packet routes back into the bridge, where it loops. The fix is to bind the underlay to a specific egress device:</p>
<div class="highlight"><pre tabindex="0" style="color:#f8f8f2;background-color:#272822;-moz-tab-size:4;-o-tab-size:4;tab-size:4;-webkit-text-size-adjust:none;"><code class="language-bash" data-lang="bash"><span style="display:flex;"><span>ip link add vxlan0 type vxlan id <span style="color:#ae81ff">100</span> <span style="color:#ae81ff">\
</span></span></span><span style="display:flex;"><span>    remote 10.255.255.1 local 172.31.26.37 dstport <span style="color:#ae81ff">4789</span> dev ens5
</span></span></code></pre></div><p>With both fixes:</p>
<div class="highlight"><pre tabindex="0" style="color:#f8f8f2;background-color:#272822;-moz-tab-size:4;-o-tab-size:4;tab-size:4;-webkit-text-size-adjust:none;"><code class="language-bash" data-lang="bash"><span style="display:flex;"><span>ip link add br-office type bridge
</span></span><span style="display:flex;"><span>ip link set br-office mtu <span style="color:#ae81ff">1380</span> up
</span></span><span style="display:flex;"><span>ip addr add 10.255.255.50/24 dev br-office
</span></span><span style="display:flex;"><span>
</span></span><span style="display:flex;"><span>ip link add vxlan0 type vxlan id <span style="color:#ae81ff">100</span> <span style="color:#ae81ff">\
</span></span></span><span style="display:flex;"><span>    remote 10.255.255.1 local 172.31.26.37 dstport <span style="color:#ae81ff">4789</span> dev ens5
</span></span><span style="display:flex;"><span>ip link set vxlan0 mtu <span style="color:#ae81ff">1380</span> master br-office up
</span></span><span style="display:flex;"><span>
</span></span><span style="display:flex;"><span>bridge fdb append 00:00:00:00:00:00 dev vxlan0 dst 10.255.255.1
</span></span></code></pre></div><p>A <code>tcpdump</code> on <code>ens5</code> after this is more honest than any documentation:</p>
<div class="highlight"><pre tabindex="0" style="color:#f8f8f2;background-color:#272822;-moz-tab-size:4;-o-tab-size:4;tab-size:4;-webkit-text-size-adjust:none;"><code class="language-fallback" data-lang="fallback"><span style="display:flex;"><span>22:24:52.110815 IP 172.31.26.37.4500 &gt; 194.126.118.207.4500:
</span></span><span style="display:flex;"><span>    UDP-encap: ESP(spi=0x00607012,seq=0x80), length 120
</span></span><span style="display:flex;"><span>22:24:52.252695 IP 194.126.118.207.4500 &gt; 172.31.26.37.4500:
</span></span><span style="display:flex;"><span>    UDP-encap: ESP(spi=0xc291d279,seq=0x186), length 136
</span></span><span style="display:flex;"><span>22:24:52.252695 IP 10.255.255.1.50999 &gt; 172.31.26.37.4789:
</span></span><span style="display:flex;"><span>    VXLAN, flags [I] (0x08), vni 100
</span></span><span style="display:flex;"><span>    STP 802.1w, Rapid STP, ...
</span></span><span style="display:flex;"><span>22:24:55.861464 IP 10.255.255.1.48863 &gt; 172.31.26.37.4789:
</span></span><span style="display:flex;"><span>    VXLAN, flags [I] (0x08), vni 100
</span></span><span style="display:flex;"><span>    ARP, Request who-has 10.255.255.230 tell 10.255.255.230
</span></span></code></pre></div><p>STP frames and the Hub Mini's gratuitous ARP both arrive over the encrypted VXLAN tunnel. The L2 extension is alive.</p>
<p>A useful red herring during this debugging: the IPSec tunnel uses NAT-T because the EC2 instance lives behind AWS's network address translation. So encrypted traffic shows up as <code>udp port 4500</code>, not raw ESP. A <code>tcpdump</code> filter that only matches <code>esp</code> returns nothing.</p>
<h3 id="persistence">Persistence</h3>
<p>VXLAN created with <code>ip link</code> does not survive a reboot. A small systemd unit owns the lifecycle:</p>
<div class="highlight"><pre tabindex="0" style="color:#f8f8f2;background-color:#272822;-moz-tab-size:4;-o-tab-size:4;tab-size:4;-webkit-text-size-adjust:none;"><code class="language-ini" data-lang="ini"><span style="display:flex;"><span><span style="color:#75715e"># /etc/systemd/system/vxlan-office.service</span>
</span></span><span style="display:flex;"><span><span style="color:#66d9ef">[Unit]</span>
</span></span><span style="display:flex;"><span><span style="color:#a6e22e">Description</span><span style="color:#f92672">=</span><span style="color:#e6db74">VXLAN office-bridge to Mikrotik (L2 extension over IPSec)</span>
</span></span><span style="display:flex;"><span><span style="color:#a6e22e">After</span><span style="color:#f92672">=</span><span style="color:#e6db74">network.target strongswan-starter.service</span>
</span></span><span style="display:flex;"><span><span style="color:#a6e22e">Wants</span><span style="color:#f92672">=</span><span style="color:#e6db74">strongswan-starter.service</span>
</span></span><span style="display:flex;"><span>
</span></span><span style="display:flex;"><span><span style="color:#66d9ef">[Service]</span>
</span></span><span style="display:flex;"><span><span style="color:#a6e22e">Type</span><span style="color:#f92672">=</span><span style="color:#e6db74">oneshot</span>
</span></span><span style="display:flex;"><span><span style="color:#a6e22e">RemainAfterExit</span><span style="color:#f92672">=</span><span style="color:#e6db74">yes</span>
</span></span><span style="display:flex;"><span><span style="color:#a6e22e">ExecStart</span><span style="color:#f92672">=</span><span style="color:#e6db74">/usr/local/sbin/vxlan-office-up.sh</span>
</span></span><span style="display:flex;"><span><span style="color:#a6e22e">ExecStop</span><span style="color:#f92672">=</span><span style="color:#e6db74">/usr/local/sbin/vxlan-office-down.sh</span>
</span></span><span style="display:flex;"><span>
</span></span><span style="display:flex;"><span><span style="color:#66d9ef">[Install]</span>
</span></span><span style="display:flex;"><span><span style="color:#a6e22e">WantedBy</span><span style="color:#f92672">=</span><span style="color:#e6db74">multi-user.target</span>
</span></span></code></pre></div><div class="highlight"><pre tabindex="0" style="color:#f8f8f2;background-color:#272822;-moz-tab-size:4;-o-tab-size:4;tab-size:4;-webkit-text-size-adjust:none;"><code class="language-bash" data-lang="bash"><span style="display:flex;"><span><span style="color:#75715e"># /usr/local/sbin/vxlan-office-up.sh</span>
</span></span><span style="display:flex;"><span><span style="color:#75715e">#!/bin/bash</span>
</span></span><span style="display:flex;"><span>set -e
</span></span><span style="display:flex;"><span>ip link show br-office &gt;/dev/null 2&gt;&amp;<span style="color:#ae81ff">1</span> <span style="color:#f92672">||</span> ip link add br-office type bridge
</span></span><span style="display:flex;"><span>ip link set br-office mtu <span style="color:#ae81ff">1380</span> up
</span></span><span style="display:flex;"><span>ip addr add 10.255.255.50/24 dev br-office 2&gt;/dev/null <span style="color:#f92672">||</span> true
</span></span><span style="display:flex;"><span>
</span></span><span style="display:flex;"><span>ip link del vxlan0 2&gt;/dev/null <span style="color:#f92672">||</span> true
</span></span><span style="display:flex;"><span>ip link add vxlan0 type vxlan id <span style="color:#ae81ff">100</span> <span style="color:#ae81ff">\
</span></span></span><span style="display:flex;"><span>    remote 10.255.255.1 local 172.31.26.37 dstport <span style="color:#ae81ff">4789</span> dev ens5
</span></span><span style="display:flex;"><span>ip link set vxlan0 mtu <span style="color:#ae81ff">1380</span> master br-office up
</span></span><span style="display:flex;"><span>
</span></span><span style="display:flex;"><span>bridge fdb append 00:00:00:00:00:00 dev vxlan0 dst 10.255.255.1
</span></span></code></pre></div><p><code>Wants=strongswan-starter.service</code> ensures the IPSec tunnel comes up first. Without that ordering, the VXLAN socket is open before the underlay can carry packets, and Mikrotik silently drops the first batch.</p>
<hr>
<h2 id="mtu-the-calculation-was-right-the-fix-was-in-the-wrong-place">MTU: The Calculation Was Right, the Fix Was in the Wrong Place</h2>
<p>The first VXLAN version made discovery work. That was misleading.</p>
<p>I had already reduced <code>br-office</code> and <code>vxlan0</code> to 1380, so I assumed the encapsulation problem was handled. The reader proved otherwise. It adopted, showed &quot;Getting Ready&quot;, then went offline when Access started pushing real config over the management channel.</p>
<p>The first version of my MTU math was also too confident. Ethernet headers are on the wire, but they are not part of the <code>1500</code> byte IP MTU. VXLAN is different because it carries an inner Ethernet frame inside UDP. IPSec NAT-T then wraps the VXLAN packet again; strongSwan documents ESP-in-UDP NAT-T as inserting an eight-byte UDP header on port <code>4500</code> (<a href="https://docs.strongswan.org/docs/latest/features/natTraversal.html">docs</a>).</p>
<p>The useful way to count it is from the narrowest public IP path, assuming a normal 1500-byte WAN MTU:</p>
<table>
  <thead>
      <tr>
          <th>Component</th>
          <th style="text-align: right">Bytes</th>
          <th>Counted where</th>
      </tr>
  </thead>
  <tbody>
      <tr>
          <td>Public outer IPv4</td>
          <td style="text-align: right">20</td>
          <td>The real WAN packet</td>
      </tr>
      <tr>
          <td>UDP 4500 NAT-T</td>
          <td style="text-align: right">8</td>
          <td>The real WAN packet</td>
      </tr>
      <tr>
          <td>ESP header, IV, trailer, auth tag, padding</td>
          <td style="text-align: right">variable</td>
          <td>The real WAN packet; depends on the IPSec proposal</td>
      </tr>
      <tr>
          <td>VXLAN carrier IPv4</td>
          <td style="text-align: right">20</td>
          <td>The protected packet inside IPSec</td>
      </tr>
      <tr>
          <td>UDP 4789</td>
          <td style="text-align: right">8</td>
          <td>The protected packet inside IPSec</td>
      </tr>
      <tr>
          <td>VXLAN header</td>
          <td style="text-align: right">8</td>
          <td>The protected packet inside IPSec</td>
      </tr>
      <tr>
          <td>Inner Ethernet header</td>
          <td style="text-align: right">14</td>
          <td>The bridged frame carried by VXLAN</td>
      </tr>
      <tr>
          <td>Inner IP packet</td>
          <td style="text-align: right"><code>M</code></td>
          <td>What the container interface MTU controls</td>
      </tr>
  </tbody>
</table>
<p>So the rough ceiling for the container-facing MTU is:</p>
<div class="highlight"><pre tabindex="0" style="color:#f8f8f2;background-color:#272822;-moz-tab-size:4;-o-tab-size:4;tab-size:4;-webkit-text-size-adjust:none;"><code class="language-text" data-lang="text"><span style="display:flex;"><span>M &lt;= 1500 - (20 + 8 + ESP_OVERHEAD + 20 + 8 + 8 + 14)
</span></span><span style="display:flex;"><span>M &lt;= 1422 - ESP_OVERHEAD
</span></span></code></pre></div><p><code>ESP_OVERHEAD</code> is not a single constant. It depends on cipher mode, IV size, integrity tag, padding, and NAT-T details. The exact number mattered less than the lesson: VXLAN-over-IPSec made the safe payload smaller, and path-MTU discovery through this stack was not something I wanted the door controller to depend on.</p>
<p>First I lowered <code>br-office</code>, <code>vxlan0</code>, and Mikrotik's <code>vxlan-aws</code> to 1380. Discovery packets crossed the tunnel, so it looked fixed. Then the G3 reader went offline during post-adoption management. ARP, UDP discovery, SYN/SYN-ACK, and small TLS records survived. Larger controller pushes did not.</p>
<p>This was the point where detection and debugging had to separate:</p>
<table>
  <thead>
      <tr>
          <th>Signal</th>
          <th>What it proved</th>
      </tr>
  </thead>
  <tbody>
      <tr>
          <td>UDP <code>10001</code> replies from <code>.229</code> and <code>.230</code> on <code>vxlan0</code></td>
          <td>L2 discovery was working</td>
      </tr>
      <tr>
          <td>TCP connect and TLS handshake to the reader</td>
          <td>Firewalling and basic routing were working</td>
      </tr>
      <tr>
          <td>Hub stayed online while G3 fell offline</td>
          <td>The controller process was not globally dead</td>
      </tr>
      <tr>
          <td>Access logs showed repeated config / broker update failures</td>
          <td>The failure was after adoption, during device management</td>
      </tr>
      <tr>
          <td>Large TLS application data stalled or reset</td>
          <td>The path was still not safe for full-size TCP payloads</td>
      </tr>
  </tbody>
</table>
<p>The capture point that mattered was the EC2 side of the L2 extension:</p>
<div class="highlight"><pre tabindex="0" style="color:#f8f8f2;background-color:#272822;-moz-tab-size:4;-o-tab-size:4;tab-size:4;-webkit-text-size-adjust:none;"><code class="language-bash" data-lang="bash"><span style="display:flex;"><span>tcpdump -ni vxlan0 host 10.255.255.229 and <span style="color:#e6db74">&#39;(udp port 10001 or tcp)&#39;</span>
</span></span></code></pre></div><p>Capturing on <code>vxlan0</code> shows the inner office traffic after VXLAN decapsulation. That made it possible to see both facts at once: discovery packets were crossing the bridge correctly, but the post-adoption TCP flow was not surviving once real payloads started.</p>
<p>The missing interface was inside the container:</p>
<div class="highlight"><pre tabindex="0" style="color:#f8f8f2;background-color:#272822;-moz-tab-size:4;-o-tab-size:4;tab-size:4;-webkit-text-size-adjust:none;"><code class="language-bash" data-lang="bash"><span style="display:flex;"><span>docker exec unifi-access ip -brief link show eth1
</span></span><span style="display:flex;"><span><span style="color:#75715e"># eth1@if15  UP  ...  mtu 1500</span>
</span></span></code></pre></div><p>So the topology looked like this:</p>
<div class="highlight"><pre tabindex="0" style="color:#f8f8f2;background-color:#272822;-moz-tab-size:4;-o-tab-size:4;tab-size:4;-webkit-text-size-adjust:none;"><code class="language-text" data-lang="text"><span style="display:flex;"><span>container eth1 MTU 1500
</span></span><span style="display:flex;"><span>    -&gt; macvlan
</span></span><span style="display:flex;"><span>    -&gt; br-office MTU 1380
</span></span><span style="display:flex;"><span>    -&gt; vxlan0 MTU 1380
</span></span><span style="display:flex;"><span>    -&gt; IPSec NAT-T underlay
</span></span></code></pre></div><p>That is enough to pass ARP, UDP discovery, SYN/SYN-ACK, and small TLS records. It is not enough to make every post-adoption controller push reliable. TCP peers can negotiate an MSS that is too large for the real encapsulated path, then the larger segments or fragments disappear in the VXLAN-over-IPSec path. The user-facing result is not &quot;MTU error&quot;; it is &quot;reader offline&quot;.</p>
<p>The fix was not another tunnel knob. It was lowering the container-facing NIC:</p>
<div class="highlight"><pre tabindex="0" style="color:#f8f8f2;background-color:#272822;-moz-tab-size:4;-o-tab-size:4;tab-size:4;-webkit-text-size-adjust:none;"><code class="language-bash" data-lang="bash"><span style="display:flex;"><span>docker exec unifi-access ip link set dev eth1 mtu <span style="color:#ae81ff">1200</span>
</span></span></code></pre></div><p>That forced a safer TCP MSS on the actual device-management path:</p>
<div class="highlight"><pre tabindex="0" style="color:#f8f8f2;background-color:#272822;-moz-tab-size:4;-o-tab-size:4;tab-size:4;-webkit-text-size-adjust:none;"><code class="language-bash" data-lang="bash"><span style="display:flex;"><span>docker exec unifi-access ss -tnpi dst 10.255.255.229
</span></span><span style="display:flex;"><span><span style="color:#75715e"># ... mss:1160 ...</span>
</span></span></code></pre></div><p>After that, the reader stayed connected long enough for Access to push config and firmware. The G3 reader upgraded to <code>v3.18.11.0</code>; the Hub Mini upgraded to <code>v1.5.14.0</code>.</p>
<p>The image still carries a small systemd timer because Docker can recreate the macvlan interface and lose the manual MTU. The timer detects the interface by the expected office IP instead of assuming Docker will call it <code>eth0</code> or <code>eth1</code>:</p>
<div class="highlight"><pre tabindex="0" style="color:#f8f8f2;background-color:#272822;-moz-tab-size:4;-o-tab-size:4;tab-size:4;-webkit-text-size-adjust:none;"><code class="language-bash" data-lang="bash"><span style="display:flex;"><span>network_ip<span style="color:#f92672">=</span><span style="color:#e6db74">&#34;</span><span style="color:#e6db74">${</span>ACCESS_MNGT_NETWORK_IP<span style="color:#66d9ef">:-</span>10.255.255.200<span style="color:#e6db74">}</span><span style="color:#e6db74">&#34;</span>
</span></span><span style="display:flex;"><span>device_mtu<span style="color:#f92672">=</span><span style="color:#e6db74">&#34;</span><span style="color:#e6db74">${</span>ACCESS_DEVICE_MTU<span style="color:#66d9ef">:-</span>1200<span style="color:#e6db74">}</span><span style="color:#e6db74">&#34;</span>
</span></span><span style="display:flex;"><span>
</span></span><span style="display:flex;"><span>network_id<span style="color:#f92672">=</span><span style="color:#e6db74">&#34;</span><span style="color:#66d9ef">$(</span>ip -o -4 addr show | awk -v ip<span style="color:#f92672">=</span><span style="color:#e6db74">&#34;</span>$network_ip<span style="color:#e6db74">&#34;</span> <span style="color:#e6db74">&#39;
</span></span></span><span style="display:flex;"><span><span style="color:#e6db74">    {
</span></span></span><span style="display:flex;"><span><span style="color:#e6db74">        split($4, address, &#34;/&#34;)
</span></span></span><span style="display:flex;"><span><span style="color:#e6db74">        if (address[1] == ip) {
</span></span></span><span style="display:flex;"><span><span style="color:#e6db74">            print $2
</span></span></span><span style="display:flex;"><span><span style="color:#e6db74">            exit
</span></span></span><span style="display:flex;"><span><span style="color:#e6db74">        }
</span></span></span><span style="display:flex;"><span><span style="color:#e6db74">    }
</span></span></span><span style="display:flex;"><span><span style="color:#e6db74">&#39;</span><span style="color:#66d9ef">)</span><span style="color:#e6db74">&#34;</span>
</span></span><span style="display:flex;"><span>
</span></span><span style="display:flex;"><span><span style="color:#66d9ef">if</span> ip link show <span style="color:#e6db74">&#34;</span>$network_id<span style="color:#e6db74">&#34;</span> &gt;/dev/null 2&gt;&amp;1; <span style="color:#66d9ef">then</span>
</span></span><span style="display:flex;"><span>    current_mtu<span style="color:#f92672">=</span><span style="color:#e6db74">&#34;</span><span style="color:#66d9ef">$(</span>cat <span style="color:#e6db74">&#34;/sys/class/net/</span>$network_id<span style="color:#e6db74">/mtu&#34;</span><span style="color:#66d9ef">)</span><span style="color:#e6db74">&#34;</span>
</span></span><span style="display:flex;"><span>    <span style="color:#66d9ef">if</span> <span style="color:#f92672">[</span> <span style="color:#e6db74">&#34;</span>$current_mtu<span style="color:#e6db74">&#34;</span> -gt <span style="color:#e6db74">&#34;</span>$device_mtu<span style="color:#e6db74">&#34;</span> <span style="color:#f92672">]</span>; <span style="color:#66d9ef">then</span>
</span></span><span style="display:flex;"><span>        ip link set dev <span style="color:#e6db74">&#34;</span>$network_id<span style="color:#e6db74">&#34;</span> mtu <span style="color:#e6db74">&#34;</span>$device_mtu<span style="color:#e6db74">&#34;</span>
</span></span><span style="display:flex;"><span>    <span style="color:#66d9ef">fi</span>
</span></span><span style="display:flex;"><span><span style="color:#66d9ef">fi</span>
</span></span></code></pre></div><p>The reliable proof was this sequence:</p>
<ol>
<li>L2 discovery frames visible on <code>vxlan0</code>.</li>
<li>Adoption started but G3 went offline during post-adoption management.</li>
<li>The container macvlan MTU was still too high while the VXLAN/IPSec path was smaller.</li>
<li>Lowering the macvlan interface to 1200 changed TCP MSS and the broker/config push survived.</li>
<li>Access upgraded the G3 firmware, after which the same database could be rolled forward to the newer 5.x UniFi OS image without readopting devices.</li>
</ol>
<p>The lesson is that &quot;device visible&quot; is only a discovery test. For Access, the real acceptance test is stricter: discovery, adoption, config push, firmware push, and long-lived MQTT all have to work over the same encapsulated path.</p>
<hr>
<h2 id="the-last-trap-the-container-had-two-identities">The Last Trap: The Container Had Two Identities</h2>
<p>At this point I thought two networks were needed:</p>
<ul>
<li>A <strong>default bridge</strong> for outbound traffic and Internet access.</li>
<li>The <strong>office-lan macvlan</strong> with parent=<code>br-office</code>, so the container gets a <code>10.255.255.x</code> address on the bridged L2.</li>
</ul>
<p>The mental model was understandable. Docker bridge networks use port publishing to make container ports reachable outside the host (<a href="https://docs.docker.com/engine/network/port-publishing/">docs</a>). Docker macvlan bridge mode makes the container appear physically attached to the network (<a href="https://docs.docker.com/engine/network/drivers/macvlan/">docs</a>). Access seemed to need both: bridge for public web ingress, macvlan for door hardware.</p>
<p>The natural Compose definition was:</p>
<div class="highlight"><pre tabindex="0" style="color:#f8f8f2;background-color:#272822;-moz-tab-size:4;-o-tab-size:4;tab-size:4;-webkit-text-size-adjust:none;"><code class="language-yaml" data-lang="yaml"><span style="display:flex;"><span><span style="color:#f92672">services</span>:
</span></span><span style="display:flex;"><span>  <span style="color:#f92672">unifi-access</span>:
</span></span><span style="display:flex;"><span>    <span style="color:#f92672">image</span>: <span style="color:#ae81ff">unifi-access-docker-arm64:stable</span>
</span></span><span style="display:flex;"><span>    <span style="color:#f92672">networks</span>:
</span></span><span style="display:flex;"><span>      <span style="color:#f92672">default</span>: {}
</span></span><span style="display:flex;"><span>      <span style="color:#f92672">office-lan</span>:
</span></span><span style="display:flex;"><span>        <span style="color:#f92672">ipv4_address</span>: <span style="color:#ae81ff">10.255.255.200</span>
</span></span><span style="display:flex;"><span>    <span style="color:#f92672">ports</span>:
</span></span><span style="display:flex;"><span>      - <span style="color:#e6db74">&#34;443:443/tcp&#34;</span>
</span></span><span style="display:flex;"><span>      - <span style="color:#e6db74">&#34;80:80/tcp&#34;</span>
</span></span><span style="display:flex;"><span>      - <span style="color:#e6db74">&#34;8080:8080/tcp&#34;</span>
</span></span><span style="display:flex;"><span>      - <span style="color:#e6db74">&#34;12442-12443:12442-12443/tcp&#34;</span>
</span></span><span style="display:flex;"><span>
</span></span><span style="display:flex;"><span><span style="color:#f92672">networks</span>:
</span></span><span style="display:flex;"><span>  <span style="color:#f92672">default</span>: {}
</span></span><span style="display:flex;"><span>  <span style="color:#f92672">office-lan</span>:
</span></span><span style="display:flex;"><span>    <span style="color:#f92672">external</span>: <span style="color:#66d9ef">true</span>
</span></span></code></pre></div><p>This deployed. The container started, got both NICs, and listened on the right ports inside the container. But from <code>docker ps</code>:</p>
<div class="highlight"><pre tabindex="0" style="color:#f8f8f2;background-color:#272822;-moz-tab-size:4;-o-tab-size:4;tab-size:4;-webkit-text-size-adjust:none;"><code class="language-fallback" data-lang="fallback"><span style="display:flex;"><span>NAMES          PORTS
</span></span><span style="display:flex;"><span>unifi-access   80/tcp, 443/tcp, 8080/tcp, 12442-12443/tcp
</span></span></code></pre></div><p>There was no <code>0.0.0.0:443-&gt;443/tcp</code>. The ports were exposed, not published. On the host, <code>ss -tlnp</code> showed no docker-proxy listening anywhere.</p>
<p>In my Compose/Engine version, this combination behaved like a bug: the service had both networks, but the published ports were only exposed, not bound on the host. I worked around it by doing the deploy in two phases: Compose declared only the bridge network with ports, then I attached the macvlan after the container was running.</p>
<p><p>
  <img src="https://kuzaxak.dev/posts/2026/05/24/unifi-access-vxlan-over-ipsec/compose-two-phase.png" alt="Two-phase Docker network attach"  />
</p>
</p>
<details>
  <summary>mermaid</summary>
  <div class="highlight"><pre tabindex="0" style="color:#f8f8f2;background-color:#272822;-moz-tab-size:4;-o-tab-size:4;tab-size:4;-webkit-text-size-adjust:none;"><code class="language-fallback" data-lang="fallback"><span style="display:flex;"><span>flowchart LR
</span></span><span style="display:flex;"><span>    A[Compose file&lt;br/&gt;default bridge + ports] --&gt; B[docker compose up -d]
</span></span><span style="display:flex;"><span>    B --&gt; C[Ports published&lt;br/&gt;through docker-proxy]
</span></span><span style="display:flex;"><span>    C --&gt; D[docker network connect&lt;br/&gt;--ip 10.255.255.200 office-lan]
</span></span><span style="display:flex;"><span>    D --&gt; E[Container has two NICs&lt;br/&gt;eth0 bridge + eth1 macvlan]
</span></span></code></pre></div>
</details>

<div class="highlight"><pre tabindex="0" style="color:#f8f8f2;background-color:#272822;-moz-tab-size:4;-o-tab-size:4;tab-size:4;-webkit-text-size-adjust:none;"><code class="language-bash" data-lang="bash"><span style="display:flex;"><span>docker compose -f docker-compose.cloud.yml up -d
</span></span><span style="display:flex;"><span>docker network connect --ip 10.255.255.200 office-lan unifi-access
</span></span></code></pre></div><p>After this:</p>
<div class="highlight"><pre tabindex="0" style="color:#f8f8f2;background-color:#272822;-moz-tab-size:4;-o-tab-size:4;tab-size:4;-webkit-text-size-adjust:none;"><code class="language-fallback" data-lang="fallback"><span style="display:flex;"><span>NAMES          PORTS
</span></span><span style="display:flex;"><span>unifi-access   0.0.0.0:443-&gt;443/tcp, 0.0.0.0:80-&gt;80/tcp, ...
</span></span></code></pre></div><div class="highlight"><pre tabindex="0" style="color:#f8f8f2;background-color:#272822;-moz-tab-size:4;-o-tab-size:4;tab-size:4;-webkit-text-size-adjust:none;"><code class="language-fallback" data-lang="fallback"><span style="display:flex;"><span>$ docker exec unifi-access ip -brief addr
</span></span><span style="display:flex;"><span>eth0@if31  UP  172.17.0.2/16
</span></span><span style="display:flex;"><span>eth1@if15  UP  10.255.255.200/24
</span></span></code></pre></div><p>That got the web UI working and the devices adopting, but it gave UniFi OS two local identities. UniFi Core's nginx listened on <code>0.0.0.0:443</code> inside the container. Cloudflare could reach the EIP through the bridge NIC. The Hub Mini could reach <code>10.255.255.200</code> through macvlan. Access could also see <code>172.17.x.x</code> / <code>172.18.x.x</code> and sometimes use that address in controller identity, management network selection, and invite-link generation.</p>
<p>The controller API exposed the symptom:</p>
<div class="highlight"><pre tabindex="0" style="color:#f8f8f2;background-color:#272822;-moz-tab-size:4;-o-tab-size:4;tab-size:4;-webkit-text-size-adjust:none;"><code class="language-bash" data-lang="bash"><span style="display:flex;"><span>curl -sS http://127.0.0.1:12080/api/v2/settings | jq .
</span></span><span style="display:flex;"><span>curl -sS http://127.0.0.1:12080/api/v2/info <span style="color:#ae81ff">\
</span></span></span><span style="display:flex;"><span>    | jq -c <span style="color:#e6db74">&#39;{host:.data.host.ip, wan:.data.host.wan_ip}&#39;</span>
</span></span></code></pre></div><p>The other problem was packet path. Docker documents that it creates firewall rules for bridge networks and port publishing, while no such rules are created for <code>macvlan</code> networks (<a href="https://docs.docker.com/engine/network/packet-filtering-firewalls/">docs</a>). In my two-NIC workaround, those worlds overlapped.</p>
<p><p>
  <img src="https://kuzaxak.dev/posts/2026/05/24/unifi-access-vxlan-over-ipsec/docker-proxy-dnat.png" alt="Docker port publish DNAT conflict"  />
</p>
</p>
<details>
  <summary>mermaid</summary>
  <div class="highlight"><pre tabindex="0" style="color:#f8f8f2;background-color:#272822;-moz-tab-size:4;-o-tab-size:4;tab-size:4;-webkit-text-size-adjust:none;"><code class="language-fallback" data-lang="fallback"><span style="display:flex;"><span>flowchart TB
</span></span><span style="display:flex;"><span>    subgraph Internet
</span></span><span style="display:flex;"><span>        CF[Cloudflare&lt;br/&gt;proxying access.example.com]
</span></span><span style="display:flex;"><span>    end
</span></span><span style="display:flex;"><span>    subgraph EC2[&#34;EC2 host network&#34;]
</span></span><span style="display:flex;"><span>        ens5[ens5&lt;br/&gt;172.31.26.37&lt;br/&gt;EIP-mapped]
</span></span><span style="display:flex;"><span>        iptables[iptables PREROUTING&lt;br/&gt;DNAT tcp/443 → 172.17.0.2:443]
</span></span><span style="display:flex;"><span>        docker0[docker0&lt;br/&gt;172.17.0.1]
</span></span><span style="display:flex;"><span>        broffice[br-office&lt;br/&gt;10.255.255.50]
</span></span><span style="display:flex;"><span>    end
</span></span><span style="display:flex;"><span>    subgraph Container[&#34;unifi-access container&#34;]
</span></span><span style="display:flex;"><span>        eth0[eth0&lt;br/&gt;172.17.0.2]
</span></span><span style="display:flex;"><span>        eth1[eth1&lt;br/&gt;10.255.255.200]
</span></span><span style="display:flex;"><span>        nginx[nginx :443&lt;br/&gt;0.0.0.0]
</span></span><span style="display:flex;"><span>    end
</span></span><span style="display:flex;"><span>    subgraph Office
</span></span><span style="display:flex;"><span>        LAN[Office host&lt;br/&gt;e.g. Synology]
</span></span><span style="display:flex;"><span>    end
</span></span><span style="display:flex;"><span>
</span></span><span style="display:flex;"><span>    CF -- &#34;TLS to EIP&#34; --&gt; ens5
</span></span><span style="display:flex;"><span>    ens5 --&gt; iptables
</span></span><span style="display:flex;"><span>    iptables --&gt; docker0
</span></span><span style="display:flex;"><span>    docker0 --&gt; eth0
</span></span><span style="display:flex;"><span>    eth0 --&gt; nginx
</span></span><span style="display:flex;"><span>
</span></span><span style="display:flex;"><span>    LAN -- &#34;via VXLAN-over-IPSec&#34; --&gt; broffice
</span></span><span style="display:flex;"><span>    broffice -- &#34;expected path&#34; --&gt; eth1
</span></span><span style="display:flex;"><span>    broffice -. &#34;iptables PREROUTING wins:&lt;br/&gt;also DNATs to docker0!&#34; .-&gt; iptables
</span></span><span style="display:flex;"><span>
</span></span><span style="display:flex;"><span>    style iptables fill:#f66
</span></span></code></pre></div>
</details>

<p>The observed DNAT rule was broad enough to match traffic by destination port:</p>
<div class="highlight"><pre tabindex="0" style="color:#f8f8f2;background-color:#272822;-moz-tab-size:4;-o-tab-size:4;tab-size:4;-webkit-text-size-adjust:none;"><code class="language-fallback" data-lang="fallback"><span style="display:flex;"><span>DNAT  tcp  *  *  0.0.0.0/0  0.0.0.0/0  tcp dpt:443  to:172.17.0.2:443
</span></span></code></pre></div><p>That matched traffic destined for the EIP and also matched traffic destined for the macvlan IP <code>10.255.255.200</code>. From the office, a request to <code>https://10.255.255.200</code> arrived on <code>br-office</code>, got DNATed to <code>172.17.0.2:443</code>, and the kernel forwarded it to the docker bridge instead of delivering it to the macvlan NIC that the LAN client expected.</p>
<p>The final fix was to remove the second identity:</p>
<ul>
<li>no docker bridge network</li>
<li>no <code>ports:</code> block</li>
<li>one macvlan interface</li>
<li>default route through the office gateway</li>
<li>web ingress through the existing Synology Traefik on the LAN</li>
</ul>
<p><p>
  <img src="https://kuzaxak.dev/posts/2026/05/24/unifi-access-vxlan-over-ipsec/traefik-access-path.png" alt="Traefik reaches Access through the office LAN"  />
</p>
</p>
<details>
  <summary>mermaid</summary>
  <div class="highlight"><pre tabindex="0" style="color:#f8f8f2;background-color:#272822;-moz-tab-size:4;-o-tab-size:4;tab-size:4;-webkit-text-size-adjust:none;"><code class="language-fallback" data-lang="fallback"><span style="display:flex;"><span>flowchart LR
</span></span><span style="display:flex;"><span>    Browser --&gt; CF[Cloudflare&lt;br/&gt;access.example.com]
</span></span><span style="display:flex;"><span>    CF --&gt; Synology[Synology Traefik&lt;br/&gt;10.255.255.245]
</span></span><span style="display:flex;"><span>    Synology -. &#34;via office LAN&#34; .-&gt; Container[unifi-access&lt;br/&gt;10.255.255.200&lt;br/&gt;via macvlan/VXLAN]
</span></span></code></pre></div>
</details>

<p>The durable Compose shape is boring:</p>
<div class="highlight"><pre tabindex="0" style="color:#f8f8f2;background-color:#272822;-moz-tab-size:4;-o-tab-size:4;tab-size:4;-webkit-text-size-adjust:none;"><code class="language-yaml" data-lang="yaml"><span style="display:flex;"><span><span style="color:#f92672">services</span>:
</span></span><span style="display:flex;"><span>  <span style="color:#f92672">unifi-os</span>:
</span></span><span style="display:flex;"><span>    <span style="color:#f92672">image</span>: <span style="color:#ae81ff">unifi-os-docker-arm64:stable</span>
</span></span><span style="display:flex;"><span>    <span style="color:#f92672">networks</span>:
</span></span><span style="display:flex;"><span>      <span style="color:#f92672">office-lan</span>:
</span></span><span style="display:flex;"><span>        <span style="color:#f92672">ipv4_address</span>: <span style="color:#ae81ff">10.255.255.200</span>
</span></span><span style="display:flex;"><span>
</span></span><span style="display:flex;"><span><span style="color:#f92672">networks</span>:
</span></span><span style="display:flex;"><span>  <span style="color:#f92672">office-lan</span>:
</span></span><span style="display:flex;"><span>    <span style="color:#f92672">external</span>: <span style="color:#66d9ef">true</span>
</span></span></code></pre></div><p>The final container has one network interface:</p>
<table>
  <thead>
      <tr>
          <th>Interface</th>
          <th>Subnet</th>
          <th>Used by</th>
          <th>Why it exists</th>
      </tr>
  </thead>
  <tbody>
      <tr>
          <td><code>eth0</code></td>
          <td>10.255.255.0/24 (macvlan / VXLAN bridge)</td>
          <td>UDP 10001 discovery, MQTT 12812, Access UI 12443, outbound HTTPS, DNS, firmware checks</td>
          <td>The only L2 identity UniFi OS and the door hardware should see</td>
      </tr>
  </tbody>
</table>
<p>The clean post-restart check looks like this:</p>
<div class="highlight"><pre tabindex="0" style="color:#f8f8f2;background-color:#272822;-moz-tab-size:4;-o-tab-size:4;tab-size:4;-webkit-text-size-adjust:none;"><code class="language-bash" data-lang="bash"><span style="display:flex;"><span>$ docker exec unifi-os ip -brief addr
</span></span><span style="display:flex;"><span>lo      UNKNOWN  127.0.0.1/8 ::1/128
</span></span><span style="display:flex;"><span>eth0    UP       10.255.255.200/24
</span></span><span style="display:flex;"><span>
</span></span><span style="display:flex;"><span>$ docker exec unifi-os ip route
</span></span><span style="display:flex;"><span>default via 10.255.255.1 dev eth0
</span></span><span style="display:flex;"><span>10.255.255.0/24 dev eth0 proto kernel scope link src 10.255.255.200
</span></span><span style="display:flex;"><span>
</span></span><span style="display:flex;"><span>$ docker exec unifi-os curl -sS http://127.0.0.1:12080/api/v2/info <span style="color:#ae81ff">\
</span></span></span><span style="display:flex;"><span>    | jq -c <span style="color:#e6db74">&#39;{host:.data.host.ip, wan:.data.host.wan_ip}&#39;</span>
</span></span><span style="display:flex;"><span><span style="color:#f92672">{</span><span style="color:#e6db74">&#34;host&#34;</span>:<span style="color:#e6db74">&#34;10.255.255.200&#34;</span>,<span style="color:#e6db74">&#34;wan&#34;</span>:<span style="color:#e6db74">&#34;10.255.255.200&#34;</span><span style="color:#f92672">}</span>
</span></span></code></pre></div><p>The Security Group on the EC2 also got much smaller. Web access does not enter through the EIP anymore, so 80/443 from the Internet are not needed. Access ports do not need direct allowlists from the office subnet either, because that traffic now arrives encapsulated as VXLAN inside IPSec - the SG only sees the outer IPSec/UDP-4500 packets.</p>
<p>Final inbound rules:</p>
<table>
  <thead>
      <tr>
          <th>Proto</th>
          <th>Port</th>
          <th>Source</th>
          <th>Purpose</th>
      </tr>
  </thead>
  <tbody>
      <tr>
          <td>UDP</td>
          <td>500</td>
          <td>Mikrotik public IP</td>
          <td>IKE</td>
      </tr>
      <tr>
          <td>UDP</td>
          <td>4500</td>
          <td>Mikrotik public IP</td>
          <td>IPSec NAT-T</td>
      </tr>
      <tr>
          <td>ESP (50)</td>
          <td>-</td>
          <td>Mikrotik public IP</td>
          <td>ESP</td>
      </tr>
      <tr>
          <td>ICMP</td>
          <td>-</td>
          <td>10.255.255.0/24</td>
          <td>Diagnostics over L2</td>
      </tr>
  </tbody>
</table>
<p>Everything else is denied. The controller has no Internet-exposed application surface. Its public identity is mediated by Cloudflare and Synology Traefik, not by an Internet-facing socket on the controller itself.</p>
<p><p>
  <img src="https://kuzaxak.dev/posts/2026/05/24/unifi-access-vxlan-over-ipsec/container-nics.png" alt="Final single-interface container shape"  />
</p>
</p>
<details>
  <summary>mermaid</summary>
  <div class="highlight"><pre tabindex="0" style="color:#f8f8f2;background-color:#272822;-moz-tab-size:4;-o-tab-size:4;tab-size:4;-webkit-text-size-adjust:none;"><code class="language-fallback" data-lang="fallback"><span style="display:flex;"><span>flowchart LR
</span></span><span style="display:flex;"><span>    subgraph Container[&#34;unifi-access container&#34;]
</span></span><span style="display:flex;"><span>        direction TB
</span></span><span style="display:flex;"><span>        Apps[unifi-core + unifi-access&lt;br/&gt;bind 0.0.0.0]
</span></span><span style="display:flex;"><span>        eth0[eth0&lt;br/&gt;10.255.255.200/24&lt;br/&gt;macvlan office-lan&lt;br/&gt;MTU 1200]
</span></span><span style="display:flex;"><span>        Apps --&gt; eth0
</span></span><span style="display:flex;"><span>    end
</span></span><span style="display:flex;"><span>
</span></span><span style="display:flex;"><span>    subgraph Internet[&#34;Outbound flows&#34;]
</span></span><span style="display:flex;"><span>        GW[Office gateway&lt;br/&gt;10.255.255.1]
</span></span><span style="display:flex;"><span>        FW[fw-update.ubnt.com&lt;br/&gt;NTP, DNS]
</span></span><span style="display:flex;"><span>    end
</span></span><span style="display:flex;"><span>
</span></span><span style="display:flex;"><span>    subgraph Office[&#34;Office L2 via VXLAN/IPSec&#34;]
</span></span><span style="display:flex;"><span>        Hub[Door Hub Mini&lt;br/&gt;10.255.255.230]
</span></span><span style="display:flex;"><span>        Reader[G3 Reader&lt;br/&gt;10.255.255.229]
</span></span><span style="display:flex;"><span>        Synology[Synology Traefik&lt;br/&gt;10.255.255.245]
</span></span><span style="display:flex;"><span>    end
</span></span><span style="display:flex;"><span>
</span></span><span style="display:flex;"><span>    eth0 --&gt; GW --&gt; FW
</span></span><span style="display:flex;"><span>    eth0 --&gt; Hub
</span></span><span style="display:flex;"><span>    eth0 --&gt; Reader
</span></span><span style="display:flex;"><span>    Synology --&gt; eth0
</span></span><span style="display:flex;"><span>
</span></span><span style="display:flex;"><span>    linkStyle 1 stroke:#999,stroke-dasharray:4 3
</span></span><span style="display:flex;"><span>    linkStyle 2 stroke:#999,stroke-dasharray:4 3
</span></span><span style="display:flex;"><span>    linkStyle 3 stroke:#999,stroke-dasharray:4 3
</span></span></code></pre></div>
</details>

<hr>
<h2 id="the-discovery-handshake-live">The Discovery Handshake, Live</h2>
<p>After the L2 bridge and macvlan attachment, <code>tcpdump</code> on <code>vxlan0</code> from the EC2 side captured the full UniFi discovery dance:</p>
<div class="highlight"><pre tabindex="0" style="color:#f8f8f2;background-color:#272822;-moz-tab-size:4;-o-tab-size:4;tab-size:4;-webkit-text-size-adjust:none;"><code class="language-fallback" data-lang="fallback"><span style="display:flex;"><span>22:30:58.954333 IP 10.255.255.200.35510 &gt; 233.89.188.1.10001:
</span></span><span style="display:flex;"><span>    UDP, length 4
</span></span><span style="display:flex;"><span>22:30:58.954387 IP 10.255.255.200.35510 &gt; 10.255.255.255.10001:
</span></span><span style="display:flex;"><span>    UDP, length 4
</span></span><span style="display:flex;"><span>
</span></span><span style="display:flex;"><span>22:30:58.965433 IP 10.255.255.230.10001 &gt; 10.255.255.200.35510:
</span></span><span style="display:flex;"><span>    UDP, length 241
</span></span><span style="display:flex;"><span>22:30:58.976442 IP 10.255.255.229.10001 &gt; 10.255.255.200.35510:
</span></span><span style="display:flex;"><span>    UDP, length 212
</span></span><span style="display:flex;"><span>22:30:58.977665 IP 10.255.255.229.44119 &gt; 255.255.255.255.10001:
</span></span><span style="display:flex;"><span>    UDP, length 212
</span></span></code></pre></div><p>The first two lines are the controller (10.255.255.200) sending its discovery probes to UniFi's multicast group <code>233.89.188.1:10001</code> and the LAN broadcast <code>255.255.255.255:10001</code>. The next three are the Hub Mini at <code>.230</code> and the G3 Reader at <code>.229</code> responding directly with their identity payloads.</p>
<p>This is the protocol working as Ubiquiti designed it. Nothing about the controller's actual location in AWS is visible from this trace; the L2 extension makes the topology look local.</p>
<p>Sessions established cleanly after that:</p>
<div class="highlight"><pre tabindex="0" style="color:#f8f8f2;background-color:#272822;-moz-tab-size:4;-o-tab-size:4;tab-size:4;-webkit-text-size-adjust:none;"><code class="language-fallback" data-lang="fallback"><span style="display:flex;"><span>$ docker exec unifi-access ss -tnp state established | grep 10.255
</span></span><span style="display:flex;"><span>tcp 0  26400  10.255.255.200:12443  10.255.255.230:51802  unifi-access-ap
</span></span><span style="display:flex;"><span>tcp 0  0      10.255.255.200:12812  10.255.255.230:51799  unifi-access-ap
</span></span></code></pre></div><p>The Hub Mini holds two TCP sessions to the controller: 12443 for the Access UI/API channel and 12812 for MQTT control commands. Real-time door operations flow over 12812.</p>
<hr>
<p><p>
  <img src="https://kuzaxak.dev/posts/2026/05/24/unifi-access-vxlan-over-ipsec/discovery-handshake.png" alt="Live UniFi Access discovery handshake"  />
</p>
</p>
<details>
  <summary>mermaid</summary>
  <div class="highlight"><pre tabindex="0" style="color:#f8f8f2;background-color:#272822;-moz-tab-size:4;-o-tab-size:4;tab-size:4;-webkit-text-size-adjust:none;"><code class="language-gdscript3" data-lang="gdscript3"><span style="display:flex;"><span>sequenceDiagram
</span></span><span style="display:flex;"><span>    participant <span style="color:#a6e22e">Container</span> as unifi<span style="color:#f92672">-</span>access<span style="color:#f92672">&lt;</span>br<span style="color:#f92672">/&gt;</span><span style="color:#ae81ff">10.255</span><span style="color:#f92672">.</span><span style="color:#ae81ff">255.200</span><span style="color:#f92672">&lt;</span>br<span style="color:#f92672">/&gt;</span>(macvlan)
</span></span><span style="display:flex;"><span>    participant Bridge as br<span style="color:#f92672">-</span>office bridge
</span></span><span style="display:flex;"><span>    participant VXLAN as vxlan0 <span style="color:#f92672">/</span> IPSec
</span></span><span style="display:flex;"><span>    participant Hub as Door Hub Mini<span style="color:#f92672">&lt;</span>br<span style="color:#f92672">/&gt;</span><span style="color:#ae81ff">10.255</span><span style="color:#f92672">.</span><span style="color:#ae81ff">255.230</span>
</span></span><span style="display:flex;"><span>
</span></span><span style="display:flex;"><span>    Note over <span style="color:#a6e22e">Container</span>,Hub: Controller side periodic discovery
</span></span><span style="display:flex;"><span>
</span></span><span style="display:flex;"><span>    <span style="color:#a6e22e">Container</span><span style="color:#f92672">-&gt;&gt;</span>Bridge: UDP src<span style="color:#f92672">=</span><span style="color:#ae81ff">10.255</span><span style="color:#f92672">.</span><span style="color:#ae81ff">255.200</span>:<span style="color:#ae81ff">35510</span><span style="color:#f92672">&lt;</span>br<span style="color:#f92672">/&gt;</span>dst<span style="color:#f92672">=</span><span style="color:#ae81ff">233.89</span><span style="color:#f92672">.</span><span style="color:#ae81ff">188.1</span>:<span style="color:#ae81ff">10001</span> (multicast)
</span></span><span style="display:flex;"><span>    <span style="color:#a6e22e">Container</span><span style="color:#f92672">-&gt;&gt;</span>Bridge: UDP src<span style="color:#f92672">=</span><span style="color:#ae81ff">10.255</span><span style="color:#f92672">.</span><span style="color:#ae81ff">255.200</span>:<span style="color:#ae81ff">35510</span><span style="color:#f92672">&lt;</span>br<span style="color:#f92672">/&gt;</span>dst<span style="color:#f92672">=</span><span style="color:#ae81ff">255.255</span><span style="color:#f92672">.</span><span style="color:#ae81ff">255.255</span>:<span style="color:#ae81ff">10001</span> (broadcast)
</span></span><span style="display:flex;"><span>    Bridge<span style="color:#f92672">-&gt;&gt;</span>VXLAN: encap, encrypt, send
</span></span><span style="display:flex;"><span>    VXLAN<span style="color:#f92672">-&gt;&gt;</span>Hub: arrives on office L2
</span></span><span style="display:flex;"><span>
</span></span><span style="display:flex;"><span>    Note over Hub: Hub Mini recognizes UniFi probe
</span></span><span style="display:flex;"><span>
</span></span><span style="display:flex;"><span>    Hub<span style="color:#f92672">-&gt;&gt;</span>VXLAN: UDP src<span style="color:#f92672">=</span><span style="color:#ae81ff">10.255</span><span style="color:#f92672">.</span><span style="color:#ae81ff">255.230</span>:<span style="color:#ae81ff">10001</span><span style="color:#f92672">&lt;</span>br<span style="color:#f92672">/&gt;</span>dst<span style="color:#f92672">=</span><span style="color:#ae81ff">10.255</span><span style="color:#f92672">.</span><span style="color:#ae81ff">255.200</span>:<span style="color:#ae81ff">35510</span><span style="color:#f92672">&lt;</span>br<span style="color:#f92672">/&gt;</span>length <span style="color:#ae81ff">241</span> (identity payload)
</span></span><span style="display:flex;"><span>    VXLAN<span style="color:#f92672">-&gt;&gt;</span>Bridge: decap on AWS
</span></span><span style="display:flex;"><span>    Bridge<span style="color:#f92672">-&gt;&gt;</span><span style="color:#a6e22e">Container</span>: delivered to eth1
</span></span><span style="display:flex;"><span>
</span></span><span style="display:flex;"><span>    Note over <span style="color:#a6e22e">Container</span>,Hub: Three more replies follow (Hub Mini retries)<span style="color:#f92672">&lt;</span>br<span style="color:#f92672">/&gt;</span>then G3 Reader at <span style="color:#f92672">.</span><span style="color:#ae81ff">229</span> also responds
</span></span><span style="display:flex;"><span>
</span></span><span style="display:flex;"><span>    <span style="color:#a6e22e">Container</span><span style="color:#f92672">-&gt;&gt;</span>Hub: TCP <span style="color:#ae81ff">12443</span> SYN (Access UI<span style="color:#f92672">/</span>API channel)
</span></span><span style="display:flex;"><span>    <span style="color:#a6e22e">Container</span><span style="color:#f92672">-&gt;&gt;</span>Hub: MQTT <span style="color:#ae81ff">12812</span> SYN (control plane)
</span></span><span style="display:flex;"><span>    Hub<span style="color:#f92672">--&gt;&gt;</span><span style="color:#a6e22e">Container</span>: <span style="color:#960050;background-color:#1e0010">✓</span> both sessions established
</span></span><span style="display:flex;"><span>
</span></span><span style="display:flex;"><span>    rect rgb(<span style="color:#ae81ff">230</span>, <span style="color:#ae81ff">255</span>, <span style="color:#ae81ff">230</span>)
</span></span><span style="display:flex;"><span>        Note over <span style="color:#a6e22e">Container</span>,Hub: Device adopted<span style="color:#f92672">.</span> Doors operate<span style="color:#f92672">.</span>
</span></span><span style="display:flex;"><span>    end
</span></span></code></pre></div>
</details>

<h2 id="mikrotik-bridge-counters-lie">Mikrotik Bridge Counters Lie</h2>
<p>A late-night debugging detour cost a real hour:</p>
<div class="highlight"><pre tabindex="0" style="color:#f8f8f2;background-color:#272822;-moz-tab-size:4;-o-tab-size:4;tab-size:4;-webkit-text-size-adjust:none;"><code class="language-fallback" data-lang="fallback"><span style="display:flex;"><span>/interface print stats where name=vxlan-aws
</span></span><span style="display:flex;"><span>0 RS vxlan-aws   RX-BYTE=0   TX-BYTE=28894   TX-PACKET=410
</span></span></code></pre></div><p>The RX counter on vxlan-aws was zero. The TX counter was incrementing rapidly. By all visible evidence, traffic was leaving Mikrotik for AWS but nothing was coming back.</p>
<p>The bridge host table told a different story:</p>
<div class="highlight"><pre tabindex="0" style="color:#f8f8f2;background-color:#272822;-moz-tab-size:4;-o-tab-size:4;tab-size:4;-webkit-text-size-adjust:none;"><code class="language-fallback" data-lang="fallback"><span style="display:flex;"><span>/interface bridge host print where on-interface=vxlan-aws
</span></span><span style="display:flex;"><span>#  MAC-ADDRESS         BRIDGE  REMOTE-IP
</span></span><span style="display:flex;"><span>0  22:0C:29:0C:2F:11   bridge  (local)
</span></span><span style="display:flex;"><span>1  4A:51:CF:46:03:91   bridge  172.31.26.37
</span></span><span style="display:flex;"><span>2  62:F0:60:4C:FB:10   bridge  172.31.26.37
</span></span></code></pre></div><p>The bridge had clearly learned the AWS-side MAC addresses, including the docker container's macvlan MAC, with REMOTE-IP correctly pointing at the EC2 private address. That can only happen from incoming VXLAN packets.</p>
<p>On RouterOS 7.19.1 in this setup, <code>/interface print stats</code> lied for the VXLAN interface. The bridge-level statistics were correct. Trust <code>/interface bridge host</code> and <code>/interface bridge port print stats</code>, not <code>/interface print stats</code> when troubleshooting VXLAN.</p>
<h2 id="what-i-would-validate-before-trusting-it">What I Would Validate Before Trusting It</h2>
<p>Working in this layered topology means there are several places where a single configuration error gives the wrong-but-plausible answer. I would not trust the design until all of these pass on a clean restart:</p>
<p><strong>IPSec underlay:</strong></p>
<div class="highlight"><pre tabindex="0" style="color:#f8f8f2;background-color:#272822;-moz-tab-size:4;-o-tab-size:4;tab-size:4;-webkit-text-size-adjust:none;"><code class="language-bash" data-lang="bash"><span style="display:flex;"><span>ip xfrm state | grep -A2 reqid
</span></span><span style="display:flex;"><span>ipsec statusall | grep -E <span style="color:#e6db74">&#34;ESTABLISHED|INSTALLED&#34;</span>
</span></span></code></pre></div><p><strong>VXLAN encapsulation:</strong></p>
<div class="highlight"><pre tabindex="0" style="color:#f8f8f2;background-color:#272822;-moz-tab-size:4;-o-tab-size:4;tab-size:4;-webkit-text-size-adjust:none;"><code class="language-bash" data-lang="bash"><span style="display:flex;"><span><span style="color:#75715e"># On EC2: outer encrypted packets should be flowing both directions</span>
</span></span><span style="display:flex;"><span>tcpdump -ni ens5 <span style="color:#e6db74">&#34;udp port 4500 or udp port 4789&#34;</span>
</span></span><span style="display:flex;"><span>
</span></span><span style="display:flex;"><span><span style="color:#75715e"># Inside the tunnel: bridge-relevant frames</span>
</span></span><span style="display:flex;"><span>tcpdump -ni vxlan0 <span style="color:#e6db74">&#34;udp port 10001 or arp&#34;</span>
</span></span></code></pre></div><p><strong>Bridge state, both ends:</strong></p>
<div class="highlight"><pre tabindex="0" style="color:#f8f8f2;background-color:#272822;-moz-tab-size:4;-o-tab-size:4;tab-size:4;-webkit-text-size-adjust:none;"><code class="language-bash" data-lang="bash"><span style="display:flex;"><span><span style="color:#75715e"># AWS side</span>
</span></span><span style="display:flex;"><span>bridge fdb show dev vxlan0
</span></span><span style="display:flex;"><span>ip neigh show dev br-office
</span></span><span style="display:flex;"><span>
</span></span><span style="display:flex;"><span><span style="color:#75715e"># Mikrotik side</span>
</span></span><span style="display:flex;"><span>/interface bridge host print where on-interface<span style="color:#f92672">=</span>vxlan-aws
</span></span></code></pre></div><p><strong>End-to-end L2 reachability:</strong></p>
<div class="highlight"><pre tabindex="0" style="color:#f8f8f2;background-color:#272822;-moz-tab-size:4;-o-tab-size:4;tab-size:4;-webkit-text-size-adjust:none;"><code class="language-bash" data-lang="bash"><span style="display:flex;"><span><span style="color:#75715e"># From AWS host</span>
</span></span><span style="display:flex;"><span>ping -I br-office 10.255.255.230   <span style="color:#75715e"># the Hub Mini</span>
</span></span><span style="display:flex;"><span>
</span></span><span style="display:flex;"><span><span style="color:#75715e"># From Mikrotik</span>
</span></span><span style="display:flex;"><span>/ping address<span style="color:#f92672">=</span>10.255.255.200 src-address<span style="color:#f92672">=</span>10.255.255.1
</span></span></code></pre></div><p><strong>UniFi-level adoption:</strong></p>
<div class="highlight"><pre tabindex="0" style="color:#f8f8f2;background-color:#272822;-moz-tab-size:4;-o-tab-size:4;tab-size:4;-webkit-text-size-adjust:none;"><code class="language-bash" data-lang="bash"><span style="display:flex;"><span>docker exec unifi-access ss -tnp state established | grep -E <span style="color:#e6db74">&#34;:(12443|12812)&#34;</span>
</span></span><span style="display:flex;"><span>docker exec unifi-access journalctl -u unifi-access --since <span style="color:#e6db74">&#34;5 minutes ago&#34;</span> <span style="color:#ae81ff">\
</span></span></span><span style="display:flex;"><span>    | grep -iE <span style="color:#e6db74">&#34;adopt|inform|84:78&#34;</span>
</span></span></code></pre></div><p><strong>Container management path:</strong></p>
<div class="highlight"><pre tabindex="0" style="color:#f8f8f2;background-color:#272822;-moz-tab-size:4;-o-tab-size:4;tab-size:4;-webkit-text-size-adjust:none;"><code class="language-bash" data-lang="bash"><span style="display:flex;"><span>docker exec unifi-access ip -brief addr
</span></span><span style="display:flex;"><span>docker exec unifi-access sh -c <span style="color:#ae81ff">\
</span></span></span><span style="display:flex;"><span>    <span style="color:#e6db74">&#39;iface=$(ip -o -4 addr show | awk &#39;</span><span style="color:#e6db74">&#34;&#39;&#34;</span><span style="color:#e6db74">&#39;$4 ~ /^10\\.255\\.255\\.200\\// {print $2; exit}&#39;</span><span style="color:#e6db74">&#34;&#39;&#34;</span><span style="color:#e6db74">&#39;); cat /sys/class/net/$iface/mtu&#39;</span>
</span></span><span style="display:flex;"><span>docker exec unifi-access curl -sS http://127.0.0.1:12080/api/v2/settings <span style="color:#ae81ff">\
</span></span></span><span style="display:flex;"><span>    | jq -c <span style="color:#e6db74">&#39;{controller_ip:.data.controller_ip,
</span></span></span><span style="display:flex;"><span><span style="color:#e6db74">              mngt_network_id:.data.mngt_network_id,
</span></span></span><span style="display:flex;"><span><span style="color:#e6db74">              discovery_broadcast_ips:.data.discovery_broadcast_ips}&#39;</span>
</span></span></code></pre></div><p>If any of these regress, the layer that broke is usually obvious from where the trace stops.</p>
<hr>
<h2 id="what-this-design-commits-you-to">What This Design Commits You To</h2>
<p>This design gives a cloud-hosted Access controller adoption parity with a local one, at the cost of explicit network plumbing.</p>
<p>The useful properties:</p>
<ul>
<li>The G3 Door Hub Mini adopts and operates exactly as it would on a Ubiquiti console - no fork of its firmware, no SSH, no manual <code>set-inform</code>.</li>
<li>The controller container is unreachable from the Internet directly. The only public surface is what Cloudflare and Synology Traefik expose.</li>
<li>Door operations continue for already-synced credentials if AWS becomes unavailable. The controller is needed for new changes and sync, not for every local unlock decision.</li>
<li>The architecture is reproducible. The VXLAN/bridge/macvlan bits are vanilla Linux primitives; the Mikrotik bits are stock RouterOS.</li>
</ul>
<p>The explicit constraints:</p>
<ul>
<li>MTU has to be set consistently at three layers (bridge, vxlan, container interface). Forget one and TCP flows hang under load.</li>
<li>The IPSec tunnel is on the critical path. Lose IPSec, the VXLAN encapsulation has nowhere to go, all adoption traffic stops within DPD timeout.</li>
<li>In my Compose/Engine version, <code>macvlan</code> plus <code>ports:</code> exposed ports without publishing them, and the bridge interface later leaked into UniFi OS controller identity. For this design, use a single macvlan network and no published EC2 ports.</li>
<li>Docker's port-publish DNAT competes with macvlan direct access. You can have one or the other; combining them through the same iptables PREROUTING chain leaves at least one path broken.</li>
<li>The container's outbound Internet path now depends on the office gateway accepting <code>10.255.255.200</code> as a normal LAN host. Verify firmware-update URLs from inside the container after every network change.</li>
<li>In my RouterOS 7.19.1 run, VXLAN interface counters lied. Use bridge-level stats.</li>
<li>The Synology kernel does not let you run <code>binfmt_misc</code>-registered foreign-arch binaries, regardless of privileges. arm64 workloads need a real arm64 host.</li>
</ul>
<p>For a homelab access control system that should not depend on Ubiquiti consoles or on bringing the controller indoors, this is the simplest design I could find that respects the actual protocol constraints instead of fighting them.</p>
<hr>
<h2 id="references">References</h2>
<ul>
<li><a href="https://github.com/dciancu/unifi-protect-unvr-docker-arm64">dciancu/unifi-protect-unvr-docker-arm64</a> - the foundation pattern this work extends</li>
<li><a href="/posts/2026/05/26/unifi-access-unvr-daemons/">Vol 2: Sixty Seconds to Nothing</a></li>
<li><a href="https://help.ui.com/hc/en-us/articles/22230509487639-UniFi-Consoles-with-UniFi-Access-Support">UniFi Access on UniFi Consoles - support article</a></li>
<li><a href="https://help.ui.com/hc/en-us/articles/17452334269975-Getting-Started-with-UniFi-Access">Getting Started with UniFi Access - ports and communication model</a></li>
<li><a href="https://www.reddit.com/r/UNIFI/comments/16v6js7/is_dhcp_option_43_supported_by_unifi_door_access/">DHCP Option 43 for UniFi Access devices - r/UNIFI thread</a></li>
<li><a href="https://docs.kernel.org/networking/vxlan.html">Linux VXLAN kernel documentation</a></li>
<li><a href="https://www.man7.org/linux/man-pages/man8/ip-link.8.html">Linux VXLAN man page</a></li>
<li><a href="https://help.mikrotik.com/docs/spaces/ROS/pages/100007937/VXLAN">RouterOS 7 VXLAN documentation</a></li>
<li><a href="https://docs.docker.com/engine/network/drivers/macvlan/">Docker macvlan driver</a></li>
<li><a href="https://docs.docker.com/engine/network/port-publishing/">Docker port publishing</a></li>
<li><a href="https://docs.docker.com/engine/network/packet-filtering-firewalls/">Docker packet filtering and firewalls</a></li>
<li><a href="https://docs.strongswan.org/docs/latest/features/natTraversal.html">strongSwan NAT traversal</a></li>
<li><a href="https://www.rfc-editor.org/rfc/rfc7348">VXLAN: RFC 7348</a></li>
</ul>
]]></content:encoded>
    </item>
    
    <item>
      <title>To be or NAT to be? Multus CNI for Telco Workloads on Kubernetes</title>
      <link>https://kuzaxak.dev/posts/2025/11/28/multus-cni-nat-or-not-to-nat/</link>
      <pubDate>Fri, 28 Nov 2025 00:00:00 +0000</pubDate>
      
      <guid>https://kuzaxak.dev/posts/2025/11/28/multus-cni-nat-or-not-to-nat/</guid>
      <description>A practical story about running SIP and RTP workloads on EKS with Multus CNI, per-pod public identity, and explicit routing.</description>
      <content:encoded><![CDATA[<h2 id="the-constraint">The Constraint</h2>
<p>SIP and RTP care about address ownership.</p>
<p>With SIP, the address in the packet is not enough. The application also writes addresses into SIP headers and SDP. RTP then uses those addresses for media. If the application advertises one <code>IP:port</code> tuple and the peer sees packets arriving from another one, the call may still signal successfully while media goes nowhere.</p>
<p>That was the constraint I had to design around: a Kubernetes pod needed a stable public identity, and the UDP source port had to remain the port the application advertised.</p>
<p>The examples below are sanitized. Names are generic, addresses are placeholders, and the code is intentionally written as pseudocode rather than something to copy into production.</p>
<hr>
<h2 id="the-packet-that-broke-the-assumption">The Packet That Broke the Assumption</h2>
<p>The first failure was not subtle. A pod sent SIP from a private address, Kubernetes or the VPC rewrote the packet on the way out, and the remote side tried to send media to the address found in SDP.</p>
<p>Signaling could still limp along because replies followed the NAT mapping. RTP did not have that luxury.</p>
<p><p>
  <img src="https://kuzaxak.dev/posts/2025/11/28/multus-cni-nat-or-not-to-nat/nat-sip.png" alt="nat-sip"  />
</p>
</p>
<details>
  <summary>mermaid</summary>
  <div class="highlight"><pre tabindex="0" style="color:#f8f8f2;background-color:#272822;-moz-tab-size:4;-o-tab-size:4;tab-size:4;-webkit-text-size-adjust:none;"><code class="language-fallback" data-lang="fallback"><span style="display:flex;"><span>sequenceDiagram
</span></span><span style="display:flex;"><span>    participant P as Pod (10.0.1.5:5060)
</span></span><span style="display:flex;"><span>    participant N as K8s Node / NAT
</span></span><span style="display:flex;"><span>    participant I as Server on Internet
</span></span><span style="display:flex;"><span>
</span></span><span style="display:flex;"><span>    Note over P: App binds UDP 5060&lt;br/&gt;SDP says: &#34;Reach me at 10.0.1.5:5060&#34;
</span></span><span style="display:flex;"><span>
</span></span><span style="display:flex;"><span>    P-&gt;&gt;N: SIP INVITE (Src: 10.0.1.5:5060)
</span></span><span style="display:flex;"><span>    Note over N: SNAT to Node IP + Random Port
</span></span><span style="display:flex;"><span>    N-&gt;&gt;I: SIP INVITE (Src: 1.2.3.4:34982)
</span></span><span style="display:flex;"><span>
</span></span><span style="display:flex;"><span>    Note over I: Server reads SDP body.&lt;br/&gt;Tries to send media to 10.0.1.5:5060&lt;br/&gt;(Private IP - Unreachable!)
</span></span><span style="display:flex;"><span>    I--xP: RTP Media Dropped ❌
</span></span><span style="display:flex;"><span>
</span></span><span style="display:flex;"><span>    Note over I: Server reads packet header.&lt;br/&gt;Replies to 1.2.3.4:34982
</span></span><span style="display:flex;"><span>    I-&gt;&gt;N: SIP 200 OK
</span></span><span style="display:flex;"><span>    N-&gt;&gt;P: SIP 200 OK (mapped back)
</span></span><span style="display:flex;"><span>
</span></span><span style="display:flex;"><span>    Note over P,I: Signaling works, but media fails.&lt;br/&gt;Result: &#34;1-way audio&#34; or complete silence.
</span></span></code></pre></div>
</details>

<p>The important part is not SIP syntax. The important part is that a middlebox changed the network tuple after the application had already told the peer what tuple to use.</p>
<p>Once that happens, there are only three broad options:</p>
<ol>
<li>Make the application aware of the translated address.</li>
<li>Relay the media through something that owns a reachable address.</li>
<li>Stop translating the media path.</li>
</ol>
<p>For this workload, I wanted option 3.</p>
<hr>
<h2 id="why-nat-keeps-showing-up">Why NAT Keeps Showing Up</h2>
<p>UDP does not establish a connection in the way TCP does. A NAT device can create a temporary mapping after an outbound packet, but that mapping is usually scoped to the peer that was contacted. With restrictive or symmetric NAT, the translated port may change per destination.</p>
<p>When both sides are behind NAT, direct communication needs help.</p>
<p><p>
  <img src="https://kuzaxak.dev/posts/2025/11/28/multus-cni-nat-or-not-to-nat/double-nat.png" alt="double-nat"  />
</p>
</p>
<details>
  <summary>mermaid</summary>
  <div class="highlight"><pre tabindex="0" style="color:#f8f8f2;background-color:#272822;-moz-tab-size:4;-o-tab-size:4;tab-size:4;-webkit-text-size-adjust:none;"><code class="language-fallback" data-lang="fallback"><span style="display:flex;"><span>sequenceDiagram
</span></span><span style="display:flex;"><span>    participant ClientA as Client A&lt;br/&gt;(Behind NAT-A)
</span></span><span style="display:flex;"><span>    participant NAT_A as NAT-A
</span></span><span style="display:flex;"><span>    participant Internet as Internet
</span></span><span style="display:flex;"><span>    participant NAT_B as NAT-B
</span></span><span style="display:flex;"><span>    participant ClientB as Client B&lt;br/&gt;(Behind NAT-B)
</span></span><span style="display:flex;"><span>
</span></span><span style="display:flex;"><span>    Note over ClientA,ClientB: Both clients have private IPs.&lt;br/&gt;Neither can receive unsolicited inbound traffic.
</span></span><span style="display:flex;"><span>
</span></span><span style="display:flex;"><span>    ClientA-&gt;&gt;NAT_A: UDP from 10.0.0.5:5000
</span></span><span style="display:flex;"><span>    NAT_A-&gt;&gt;Internet: UDP from 203.0.113.10:53801 (mapped)
</span></span><span style="display:flex;"><span>    Note over Internet: NAT-A opened port 53801,&lt;br/&gt;but NAT-B doesn&#39;t know about it
</span></span><span style="display:flex;"><span>
</span></span><span style="display:flex;"><span>    ClientB-&gt;&gt;NAT_B: UDP from 192.168.1.5:5000
</span></span><span style="display:flex;"><span>    NAT_B-&gt;&gt;Internet: UDP from 198.51.100.20:49152 (mapped)
</span></span><span style="display:flex;"><span>    Note over Internet: NAT-B opened port 49152,&lt;br/&gt;but NAT-A doesn&#39;t know about it
</span></span><span style="display:flex;"><span>
</span></span><span style="display:flex;"><span>    Note over ClientA,ClientB: ❌ Neither NAT will accept&lt;br/&gt;unsolicited inbound packets!
</span></span></code></pre></div>
</details>

<p>WebRTC deals with this using <a href="https://www.rfc-editor.org/rfc/rfc8445">ICE</a>. <a href="https://www.rfc-editor.org/rfc/rfc8489">STUN</a> tells a client what address it appears to have from outside. <a href="https://www.rfc-editor.org/rfc/rfc8656">TURN</a> gives up on direct media and relays it.</p>
<p><p>
  <img src="https://kuzaxak.dev/posts/2025/11/28/multus-cni-nat-or-not-to-nat/webrtc-turn.png" alt="webrtc-turn"  />
</p>
</p>
<details>
  <summary>mermaid</summary>
  <div class="highlight"><pre tabindex="0" style="color:#f8f8f2;background-color:#272822;-moz-tab-size:4;-o-tab-size:4;tab-size:4;-webkit-text-size-adjust:none;"><code class="language-fallback" data-lang="fallback"><span style="display:flex;"><span>sequenceDiagram
</span></span><span style="display:flex;"><span>    participant CA as Client A
</span></span><span style="display:flex;"><span>    participant STUN as STUN Server
</span></span><span style="display:flex;"><span>    participant TURN as TURN Server
</span></span><span style="display:flex;"><span>    participant CB as Client B
</span></span><span style="display:flex;"><span>
</span></span><span style="display:flex;"><span>    Note over CA,CB: Step 1: STUN Discovery
</span></span><span style="display:flex;"><span>
</span></span><span style="display:flex;"><span>    CA-&gt;&gt;STUN: What&#39;s my public IP:port?
</span></span><span style="display:flex;"><span>    STUN--&gt;&gt;CA: You appear as 203.0.113.10:53801
</span></span><span style="display:flex;"><span>
</span></span><span style="display:flex;"><span>    CB-&gt;&gt;STUN: What&#39;s my public IP:port?
</span></span><span style="display:flex;"><span>    STUN--&gt;&gt;CB: You appear as 198.51.100.5:49152
</span></span><span style="display:flex;"><span>
</span></span><span style="display:flex;"><span>    Note over CA,CB: Step 2: Direct Connection Attempt
</span></span><span style="display:flex;"><span>
</span></span><span style="display:flex;"><span>    CA-&gt;&gt;CB: Try P2P using discovered addresses
</span></span><span style="display:flex;"><span>    CB-&gt;&gt;CA: Try P2P using discovered addresses
</span></span><span style="display:flex;"><span>
</span></span><span style="display:flex;"><span>    Note over CA,CB: ❌ Often fails with Symmetric NAT&lt;br/&gt;(port changes per destination)
</span></span><span style="display:flex;"><span>
</span></span><span style="display:flex;"><span>    Note over CA,CB: Step 3: TURN Relay (Fallback)
</span></span><span style="display:flex;"><span>
</span></span><span style="display:flex;"><span>    CA-&gt;&gt;TURN: Allocate relay address
</span></span><span style="display:flex;"><span>    TURN--&gt;&gt;CA: Your relay: 54.1.2.3:12345
</span></span><span style="display:flex;"><span>
</span></span><span style="display:flex;"><span>    CB-&gt;&gt;TURN: Allocate relay address
</span></span><span style="display:flex;"><span>    TURN--&gt;&gt;CB: Your relay: 54.1.2.3:12346
</span></span><span style="display:flex;"><span>
</span></span><span style="display:flex;"><span>    rect rgb(255, 230, 230)
</span></span><span style="display:flex;"><span>        Note over CA,CB: All media now relayed through TURN
</span></span><span style="display:flex;"><span>        CA-&gt;&gt;TURN: Media traffic
</span></span><span style="display:flex;"><span>        TURN-&gt;&gt;CB: Relayed media
</span></span><span style="display:flex;"><span>        CB-&gt;&gt;TURN: Media traffic
</span></span><span style="display:flex;"><span>        TURN-&gt;&gt;CA: Relayed media
</span></span><span style="display:flex;"><span>    end
</span></span></code></pre></div>
</details>

<p>TURN is the correct fallback for many browser-to-browser cases, but it is not free. It adds relay latency, consumes relay bandwidth, and moves media capacity into a shared service.</p>
<p>Core <a href="https://www.rfc-editor.org/rfc/rfc3261">SIP</a> does not solve NAT by itself. There are SIP-related NAT traversal tools, including symmetric response routing with <code>rport</code> (<a href="https://www.rfc-editor.org/rfc/rfc3581">RFC 3581</a>), SIP Outbound (<a href="https://www.rfc-editor.org/rfc/rfc5626">RFC 5626</a>), and ICE usage in SDP/SIP flows (<a href="https://www.rfc-editor.org/rfc/rfc8839">RFC 8839</a>, <a href="https://www.rfc-editor.org/rfc/rfc8840">RFC 8840</a>). Session Border Controllers can also hide a lot of complexity, but that still means adding a media-aware intermediary. For a server-side SIP/RTP workload that should own its media path, I needed the pod to own the reachable address.</p>
<hr>
<h2 id="eks-added-more-translation">EKS Added More Translation</h2>
<p>The default EKS shape made the problem worse. A pod in a private subnet could cross kube-proxy rules, the node stack, and a NAT Gateway before the packet reached the Internet.</p>
<p><p>
  <img src="https://kuzaxak.dev/posts/2025/11/28/multus-cni-nat-or-not-to-nat/eks-flow.png" alt="eks-flow"  />
</p>
</p>
<details>
  <summary>mermaid</summary>
  <div class="highlight"><pre tabindex="0" style="color:#f8f8f2;background-color:#272822;-moz-tab-size:4;-o-tab-size:4;tab-size:4;-webkit-text-size-adjust:none;"><code class="language-fallback" data-lang="fallback"><span style="display:flex;"><span>sequenceDiagram
</span></span><span style="display:flex;"><span>    participant Pod as Pod&lt;br/&gt;10.0.1.50
</span></span><span style="display:flex;"><span>    participant KP as kube-proxy&lt;br/&gt;(MASQUERADE)
</span></span><span style="display:flex;"><span>    participant Node as Node&lt;br/&gt;10.0.1.10
</span></span><span style="display:flex;"><span>    participant NAT as NAT Gateway&lt;br/&gt;54.1.2.3
</span></span><span style="display:flex;"><span>    participant IGW as Internet Gateway
</span></span><span style="display:flex;"><span>    participant Peer as Remote Peer
</span></span><span style="display:flex;"><span>
</span></span><span style="display:flex;"><span>    Note over Pod,Peer: Outbound Connection from EKS Pod (Private Subnet)
</span></span><span style="display:flex;"><span>
</span></span><span style="display:flex;"><span>    Pod-&gt;&gt;KP: src: 10.0.1.50:40000&lt;br/&gt;dst: Remote Peer
</span></span><span style="display:flex;"><span>
</span></span><span style="display:flex;"><span>    Note over KP: SNAT (MASQUERADE)&lt;br/&gt;Port randomized!
</span></span><span style="display:flex;"><span>
</span></span><span style="display:flex;"><span>    KP-&gt;&gt;Node: src: 10.0.1.10:53801&lt;br/&gt;dst: Remote Peer
</span></span><span style="display:flex;"><span>
</span></span><span style="display:flex;"><span>    Node-&gt;&gt;NAT: src: 10.0.1.10:53801&lt;br/&gt;dst: Remote Peer
</span></span><span style="display:flex;"><span>
</span></span><span style="display:flex;"><span>    Note over NAT: SNAT&lt;br/&gt;Port randomized again!
</span></span><span style="display:flex;"><span>
</span></span><span style="display:flex;"><span>    NAT-&gt;&gt;IGW: src: 54.1.2.3:49152&lt;br/&gt;dst: Remote Peer
</span></span><span style="display:flex;"><span>
</span></span><span style="display:flex;"><span>    IGW-&gt;&gt;Peer: src: 54.1.2.3:49152&lt;br/&gt;dst: Remote Peer
</span></span><span style="display:flex;"><span>
</span></span><span style="display:flex;"><span>    Note over Pod,Peer: ⚠️ Original port 40000 is lost after double NAT
</span></span></code></pre></div>
</details>

<p>That flow is fine for traffic where only the remote service needs to be reachable. It is bad for a media server that has already advertised <code>203.0.113.50:40000</code> and expects the peer to send RTP there.</p>
<p>Bare EC2 instances in public subnets avoid most of this. That is why telecom stacks often end up there. The tradeoff is operational: slower rollout, less bin-packing, more custom lifecycle work, and less of the Kubernetes machinery that teams already use.</p>
<p>So the question became narrower: can a pod keep Kubernetes for lifecycle, but bypass Kubernetes for the media address?</p>
<hr>
<h2 id="wrong-assumption-1-public-nodes-and-hostnetwork">Wrong Assumption 1: Public Nodes and hostNetwork</h2>
<p>The first idea was to put a node group in public subnets and run the workload with <code>hostNetwork: true</code>.</p>
<p>The assumption was simple:</p>
<ol>
<li>The pod binds UDP <code>40000</code>.</li>
<li>The node has a public IP.</li>
<li>The peer sends packets to <code>&lt;node-public-ip&gt;:40000</code>.</li>
</ol>
<p>The measurement said otherwise.</p>
<div class="highlight"><pre tabindex="0" style="color:#f8f8f2;background-color:#272822;-moz-tab-size:4;-o-tab-size:4;tab-size:4;-webkit-text-size-adjust:none;"><code class="language-bash" data-lang="bash"><span style="display:flex;"><span><span style="color:#75715e"># Test pod using the host network namespace</span>
</span></span><span style="display:flex;"><span>kubectl run udp-test --image<span style="color:#f92672">=</span>alpine --restart<span style="color:#f92672">=</span>Never <span style="color:#ae81ff">\
</span></span></span><span style="display:flex;"><span>  --overrides<span style="color:#f92672">=</span><span style="color:#e6db74">&#39;{&#34;spec&#34;:{&#34;hostNetwork&#34;:true}}&#39;</span> <span style="color:#ae81ff">\
</span></span></span><span style="display:flex;"><span>  -- sh -c <span style="color:#e6db74">&#34;apk add socat &amp;&amp; socat -v UDP-LISTEN:40000,fork EXEC:&#39;/bin/cat&#39;&#34;</span>
</span></span><span style="display:flex;"><span>
</span></span><span style="display:flex;"><span><span style="color:#75715e"># Look at conntrack state</span>
</span></span><span style="display:flex;"><span>sudo conntrack -L -p udp | grep <span style="color:#ae81ff">40000</span>
</span></span><span style="display:flex;"><span>
</span></span><span style="display:flex;"><span><span style="color:#75715e"># Look for source NAT rules</span>
</span></span><span style="display:flex;"><span>sudo iptables-save | grep -E <span style="color:#e6db74">&#39;MASQUERADE|random-fully&#39;</span>
</span></span></code></pre></div><p>The useful observation was the <code>--random-fully</code> MASQUERADE behavior in the path. Even when the packet originated from the host network namespace, the source port could still be rewritten by rules installed for Kubernetes service traffic and hairpin behavior.</p>
<p><code>hostNetwork</code> also had a design problem independent of that measurement:</p>
<table>
  <thead>
      <tr>
          <th>Problem</th>
          <th>Why it mattered</th>
      </tr>
  </thead>
  <tbody>
      <tr>
          <td>Port collisions</td>
          <td>Multiple pods cannot all bind <code>5060</code> on the same node</td>
      </tr>
      <tr>
          <td>Shared identity</td>
          <td>The node IP is not a per-pod identity</td>
      </tr>
      <tr>
          <td>Scheduling coupling</td>
          <td>Placement decides external identity</td>
      </tr>
      <tr>
          <td>Security boundary</td>
          <td>The pod shares the host network namespace</td>
      </tr>
  </tbody>
</table>
<p>The correction was to stop trying to make the node IP behave like a pod IP.</p>
<hr>
<h2 id="wrong-assumption-2-put-an-nlb-in-front">Wrong Assumption 2: Put an NLB in Front</h2>
<p>The second idea was a Network Load Balancer with UDP listeners.</p>
<p>That solved reachability. It did not solve identity.</p>
<p>The main issue was not simply that UDP stickiness can be awkward. The real problem was that the application still did not own the address it needed to advertise.</p>
<p>NLBs do support UDP listeners. The difference is what the protocol gives the load balancer to work with. For TCP, a connection has a clear lifetime, and the load balancer can keep that connection associated with one target. For UDP, there is no connection, but it is still not round-robin per packet: the NLB hashes the flow tuple: protocol, source IP, source port, destination IP, and destination port (<a href="https://docs.aws.amazon.com/elasticloadbalancing/latest/network/introduction.html">AWS docs</a>).</p>
<p>Packets with the same UDP tuple should keep landing on the same target while that flow is active. If the client changes source port, the packet reaches a different NLB IP, or the UDP flow sits idle past the NLB timeout, the next packet is a new flow and can be hashed to a different target. AWS documents the UDP flow idle timeout as 120 seconds and not configurable (<a href="https://docs.aws.amazon.com/elasticloadbalancing/latest/network/network-load-balancers.html#connection-idle-timeout">AWS docs</a>).</p>
<p>That can work for a simple UDP request/response service where each flow is self-contained. SIP/RTP is not that shape. SIP signaling may be one flow, while RTP media uses separate ports negotiated in SDP. Those media flows can hash independently from signaling.</p>
<p>There is also a practical listener problem: RTP usually needs a port range. An NLB listener is configured for a specific protocol and port, not &quot;whatever UDP port this pod negotiated in SDP&quot;. You can create many listeners, but that turns the RTP range into load balancer configuration, and Network Load Balancers have finite listener quotas (<a href="https://docs.aws.amazon.com/elasticloadbalancing/latest/network/load-balancer-limits.html">AWS quotas</a>). A media range of thousands of ports does not map cleanly to NLB listeners and target groups.</p>
<p>The workload still has to advertise an address and port that the peer can send media to directly. The NLB address is reachable, but it is not the pod's own identity, and it is not a good representation of the pod's RTP port range.</p>
<p>With an NLB:</p>
<ol>
<li>The peer sends traffic to the NLB address.</li>
<li>The pod receives traffic through a load-balanced path.</li>
<li>The pod still has to decide what IP to put into SIP headers and SDP.</li>
<li>Replies must return through a path that keeps the peer seeing the advertised address.</li>
</ol>
<p>If the pod replies directly, the peer sees a different source address. If the pod always replies through the NLB, the media path is no longer owned by the pod and the design becomes tightly coupled to load balancer behavior.</p>
<p>That can be fine when the load balancer is the service identity. For SIP/RTP, where the workload needs to advertise and source the same tuple, it is the wrong abstraction.</p>
<hr>
<h2 id="the-shape-that-matched-the-protocol">The Shape That Matched the Protocol</h2>
<p>The design that held up was to treat the pod like a small VM for media traffic:</p>
<ol>
<li>Keep <code>eth0</code> for normal Kubernetes traffic.</li>
<li>Add <code>net1</code> with Multus for SIP/RTP.</li>
<li>Put <code>net1</code> in the public-subnet path.</li>
<li>Associate an Elastic IP with the private IP used by <code>net1</code>.</li>
<li>Force packets sourced from that identity to leave via <code>net1</code>.</li>
</ol>
<p><p>
  <img src="https://kuzaxak.dev/posts/2025/11/28/multus-cni-nat-or-not-to-nat/arch.png" alt="arch.png"  />
</p>
</p>
<details>
  <summary>mermaid</summary>
  <div class="highlight"><pre tabindex="0" style="color:#f8f8f2;background-color:#272822;-moz-tab-size:4;-o-tab-size:4;tab-size:4;-webkit-text-size-adjust:none;"><code class="language-fallback" data-lang="fallback"><span style="display:flex;"><span>flowchart LR
</span></span><span style="display:flex;"><span>    subgraph &#34;VPC&#34;
</span></span><span style="display:flex;"><span>        subgraph &#34;Private Subnet&#34;
</span></span><span style="display:flex;"><span>            Node[Worker Node&lt;br/&gt;Primary ENI]
</span></span><span style="display:flex;"><span>        end
</span></span><span style="display:flex;"><span>        subgraph &#34;Public Subnet&#34;
</span></span><span style="display:flex;"><span>            SecENI[Secondary ENI]
</span></span><span style="display:flex;"><span>        end
</span></span><span style="display:flex;"><span>        IGW[Internet Gateway]
</span></span><span style="display:flex;"><span>        RT[Route Table&lt;br/&gt;0.0.0.0/0 → IGW]
</span></span><span style="display:flex;"><span>    end
</span></span><span style="display:flex;"><span>
</span></span><span style="display:flex;"><span>    Node -- &#34;eth0&lt;br/&gt;(K8s traffic)&#34; --&gt; PrivateRT[Private Route Table]
</span></span><span style="display:flex;"><span>    SecENI -- &#34;net1&lt;br/&gt;(Telecom traffic)&#34; --&gt; RT
</span></span><span style="display:flex;"><span>    RT --&gt; IGW
</span></span><span style="display:flex;"><span>
</span></span><span style="display:flex;"><span>    Pod[Pod] -- &#34;eth0&#34; --&gt; Node
</span></span><span style="display:flex;"><span>    Pod -- &#34;net1&#34; --&gt; SecENI
</span></span><span style="display:flex;"><span>
</span></span><span style="display:flex;"><span>    SecENI &lt;--&gt; EIP[Elastic IP]
</span></span><span style="display:flex;"><span>    EIP &lt;--&gt; IGW
</span></span><span style="display:flex;"><span>    IGW &lt;--&gt; Internet[Internet]
</span></span></code></pre></div>
</details>

<p>This keeps the control plane boring. Kubernetes API traffic, probes, logs, and metrics stay on <code>eth0</code>. Media traffic gets a separate interface, route table, and public identity.</p>
<hr>
<h2 id="multus-gave-me-the-second-interface">Multus Gave Me the Second Interface</h2>
<p><a href="https://docs.aws.amazon.com/eks/latest/userguide/pod-multus.html">Multus</a> is a CNI meta-plugin. It does not replace the default pod network; it asks another CNI plugin to attach extra interfaces.</p>
<p><p>
  <img src="https://kuzaxak.dev/posts/2025/11/28/multus-cni-nat-or-not-to-nat/multus-cni.png" alt="multus-cni.png"  />
</p>
</p>
<details>
  <summary>mermaid</summary>
  <div class="highlight"><pre tabindex="0" style="color:#f8f8f2;background-color:#272822;-moz-tab-size:4;-o-tab-size:4;tab-size:4;-webkit-text-size-adjust:none;"><code class="language-fallback" data-lang="fallback"><span style="display:flex;"><span>flowchart TB
</span></span><span style="display:flex;"><span>    subgraph &#34;CNI Plugin Chain&#34;
</span></span><span style="display:flex;"><span>        Kubelet[Kubelet] --&gt; Multus[Multus&lt;br/&gt;Meta-Plugin]
</span></span><span style="display:flex;"><span>        Multus --&gt;|&#34;Primary: eth0&#34;| VPCCNI[Amazon VPC CNI]
</span></span><span style="display:flex;"><span>        Multus --&gt;|&#34;Secondary: net1&#34;| IPVLAN[ipvlan Plugin]
</span></span><span style="display:flex;"><span>        IPVLAN --&gt; Whereabouts[Whereabouts IPAM]
</span></span><span style="display:flex;"><span>    end
</span></span><span style="display:flex;"><span>
</span></span><span style="display:flex;"><span>    subgraph &#34;Pod Network Namespace&#34;
</span></span><span style="display:flex;"><span>        eth0_result[eth0&lt;br/&gt;K8s control plane]
</span></span><span style="display:flex;"><span>        net1_result[net1&lt;br/&gt;Telecom data plane]
</span></span><span style="display:flex;"><span>    end
</span></span><span style="display:flex;"><span>
</span></span><span style="display:flex;"><span>    VPCCNI --&gt; eth0_result
</span></span><span style="display:flex;"><span>    IPVLAN --&gt; net1_result
</span></span></code></pre></div>
</details>

<p>For the secondary interface I evaluated two practical modes:</p>
<table>
  <thead>
      <tr>
          <th>Mode</th>
          <th>What it gives you</th>
          <th>Cost</th>
      </tr>
  </thead>
  <tbody>
      <tr>
          <td><code>ipvlan</code></td>
          <td>Multiple pod interfaces over the host ENI</td>
          <td>Needs careful IPAM and routing</td>
      </tr>
      <tr>
          <td><code>host-device</code></td>
          <td>One physical or virtual device moved into the pod</td>
          <td>Lower density; scheduling is harder</td>
      </tr>
  </tbody>
</table>
<p><code>ipvlan</code> was the better default. It gave pod density and kept scheduling practical. <code>host-device</code> still makes sense when the network device itself is the unit of isolation, for example with SR-IOV-style setups.</p>
<p>The NetworkAttachmentDefinition is small, but every value matters:</p>
<div class="highlight"><pre tabindex="0" style="color:#f8f8f2;background-color:#272822;-moz-tab-size:4;-o-tab-size:4;tab-size:4;-webkit-text-size-adjust:none;"><code class="language-yaml" data-lang="yaml"><span style="display:flex;"><span><span style="color:#f92672">apiVersion</span>: <span style="color:#e6db74">&#34;k8s.cni.cncf.io/v1&#34;</span>
</span></span><span style="display:flex;"><span><span style="color:#f92672">kind</span>: <span style="color:#ae81ff">NetworkAttachmentDefinition</span>
</span></span><span style="display:flex;"><span><span style="color:#f92672">metadata</span>:
</span></span><span style="display:flex;"><span>  <span style="color:#f92672">name</span>: <span style="color:#ae81ff">telecom-net-az1</span>
</span></span><span style="display:flex;"><span>  <span style="color:#f92672">namespace</span>: <span style="color:#ae81ff">kube-system</span>
</span></span><span style="display:flex;"><span><span style="color:#f92672">spec</span>:
</span></span><span style="display:flex;"><span>  <span style="color:#f92672">config</span>: |<span style="color:#e6db74">
</span></span></span><span style="display:flex;"><span><span style="color:#e6db74">    {
</span></span></span><span style="display:flex;"><span><span style="color:#e6db74">      &#34;cniVersion&#34;: &#34;0.3.1&#34;,
</span></span></span><span style="display:flex;"><span><span style="color:#e6db74">      &#34;type&#34;: &#34;ipvlan&#34;,
</span></span></span><span style="display:flex;"><span><span style="color:#e6db74">      &#34;master&#34;: &#34;eth1&#34;,
</span></span></span><span style="display:flex;"><span><span style="color:#e6db74">      &#34;mode&#34;: &#34;l2&#34;,
</span></span></span><span style="display:flex;"><span><span style="color:#e6db74">      &#34;ipam&#34;: {
</span></span></span><span style="display:flex;"><span><span style="color:#e6db74">        &#34;type&#34;: &#34;whereabouts&#34;,
</span></span></span><span style="display:flex;"><span><span style="color:#e6db74">        &#34;range&#34;: &#34;10.0.2.128/26&#34;,
</span></span></span><span style="display:flex;"><span><span style="color:#e6db74">        &#34;gateway&#34;: &#34;10.0.2.1&#34;
</span></span></span><span style="display:flex;"><span><span style="color:#e6db74">      }
</span></span></span><span style="display:flex;"><span><span style="color:#e6db74">    }</span>
</span></span></code></pre></div><p>And a pod opts in explicitly:</p>
<div class="highlight"><pre tabindex="0" style="color:#f8f8f2;background-color:#272822;-moz-tab-size:4;-o-tab-size:4;tab-size:4;-webkit-text-size-adjust:none;"><code class="language-yaml" data-lang="yaml"><span style="display:flex;"><span><span style="color:#f92672">metadata</span>:
</span></span><span style="display:flex;"><span>  <span style="color:#f92672">annotations</span>:
</span></span><span style="display:flex;"><span>    <span style="color:#f92672">k8s.v1.cni.cncf.io/networks</span>: <span style="color:#ae81ff">telecom-net-az1</span>
</span></span></code></pre></div><hr>
<h2 id="ipam-was-a-real-part-of-the-design">IPAM Was a Real Part of the Design</h2>
<p>Whereabouts can assign an address to <code>net1</code>, but AWS VPC DHCP also assigns addresses from the subnet. If both systems believe they own the same range, sooner or later they can pick the same address.</p>
<p><p>
  <img src="https://kuzaxak.dev/posts/2025/11/28/multus-cni-nat-or-not-to-nat/conflict_dhcp.png" alt="conflict_dhcp.png"  />
</p>
</p>
<details>
  <summary>mermaid</summary>
  <div class="highlight"><pre tabindex="0" style="color:#f8f8f2;background-color:#272822;-moz-tab-size:4;-o-tab-size:4;tab-size:4;-webkit-text-size-adjust:none;"><code class="language-fallback" data-lang="fallback"><span style="display:flex;"><span>flowchart TB
</span></span><span style="display:flex;"><span>    subgraph &#34;Same Subnet: 10.0.2.0/24&#34;
</span></span><span style="display:flex;"><span>        DHCP[VPC DHCP&lt;br/&gt;assigns to ENIs]
</span></span><span style="display:flex;"><span>        WA[Whereabouts&lt;br/&gt;assigns to Pods]
</span></span><span style="display:flex;"><span>
</span></span><span style="display:flex;"><span>        DHCP --&gt;|&#34;Could assign&#34;| IP1[10.0.2.50]
</span></span><span style="display:flex;"><span>        WA --&gt;|&#34;Could assign&#34;| IP1
</span></span><span style="display:flex;"><span>
</span></span><span style="display:flex;"><span>        Conflict[💥 IP Conflict!&lt;br/&gt;Two entities, same IP]
</span></span><span style="display:flex;"><span>    end
</span></span></code></pre></div>
</details>

<p>The fix was to reserve the pod range at the subnet level with a <a href="https://docs.aws.amazon.com/vpc/latest/userguide/subnet-cidr-reservation.html">subnet CIDR reservation</a> and make Whereabouts use exactly that range.</p>
<p><p>
  <img src="https://kuzaxak.dev/posts/2025/11/28/multus-cni-nat-or-not-to-nat/dhcp_res.png" alt="dhcp_res.png"  />
</p>
</p>
<details>
  <summary>mermaid</summary>
  <div class="highlight"><pre tabindex="0" style="color:#f8f8f2;background-color:#272822;-moz-tab-size:4;-o-tab-size:4;tab-size:4;-webkit-text-size-adjust:none;"><code class="language-fallback" data-lang="fallback"><span style="display:flex;"><span>flowchart LR
</span></span><span style="display:flex;"><span>    subgraph &#34;Public Subnet: 10.0.2.0/24&#34;
</span></span><span style="display:flex;"><span>        subgraph &#34;DHCP Managed: .1 - .127&#34;
</span></span><span style="display:flex;"><span>            ENI1[Node ENI&lt;br/&gt;10.0.2.10]
</span></span><span style="display:flex;"><span>            ENI2[Node ENI&lt;br/&gt;10.0.2.11]
</span></span><span style="display:flex;"><span>        end
</span></span><span style="display:flex;"><span>
</span></span><span style="display:flex;"><span>        subgraph &#34;Reserved for Whereabouts: .128 - .191&#34;
</span></span><span style="display:flex;"><span>            Pod1[Pod net1&lt;br/&gt;10.0.2.128]
</span></span><span style="display:flex;"><span>            Pod2[Pod net1&lt;br/&gt;10.0.2.129]
</span></span><span style="display:flex;"><span>            Pod3[Pod net1&lt;br/&gt;10.0.2.130]
</span></span><span style="display:flex;"><span>        end
</span></span><span style="display:flex;"><span>
</span></span><span style="display:flex;"><span>        subgraph &#34;Future Use: .192 - .254&#34;
</span></span><span style="display:flex;"><span>            Reserved[Available]
</span></span><span style="display:flex;"><span>        end
</span></span><span style="display:flex;"><span>    end
</span></span></code></pre></div>
</details>

<p>In Terraform, the important bit is the matching pair:</p>
<div class="highlight"><pre tabindex="0" style="color:#f8f8f2;background-color:#272822;-moz-tab-size:4;-o-tab-size:4;tab-size:4;-webkit-text-size-adjust:none;"><code class="language-hcl" data-lang="hcl"><span style="display:flex;"><span><span style="color:#66d9ef">resource</span> <span style="color:#e6db74">&#34;aws_ec2_subnet_cidr_reservation&#34; &#34;whereabouts_az1&#34;</span> {
</span></span><span style="display:flex;"><span>  cidr_block       <span style="color:#f92672">=</span> <span style="color:#e6db74">&#34;10.0.2.128/26&#34;</span>
</span></span><span style="display:flex;"><span>  reservation_type <span style="color:#f92672">=</span> <span style="color:#e6db74">&#34;explicit&#34;</span>
</span></span><span style="display:flex;"><span>  subnet_id        <span style="color:#f92672">=</span> <span style="color:#66d9ef">aws_subnet</span>.<span style="color:#66d9ef">public_az1</span>.<span style="color:#66d9ef">id</span>
</span></span><span style="display:flex;"><span>  description      <span style="color:#f92672">=</span> <span style="color:#e6db74">&#34;Reserved for Multus/Whereabouts pod IPs&#34;</span>
</span></span><span style="display:flex;"><span>}
</span></span></code></pre></div><div class="highlight"><pre tabindex="0" style="color:#f8f8f2;background-color:#272822;-moz-tab-size:4;-o-tab-size:4;tab-size:4;-webkit-text-size-adjust:none;"><code class="language-yaml" data-lang="yaml"><span style="display:flex;"><span><span style="color:#f92672">ipam</span>:
</span></span><span style="display:flex;"><span>  <span style="color:#f92672">type</span>: <span style="color:#ae81ff">whereabouts</span>
</span></span><span style="display:flex;"><span>  <span style="color:#f92672">range</span>: <span style="color:#e6db74">&#34;10.0.2.128/26&#34;</span>
</span></span><span style="display:flex;"><span>  <span style="color:#f92672">exclude</span>:
</span></span><span style="display:flex;"><span>    - <span style="color:#e6db74">&#34;10.0.2.128/30&#34;</span>
</span></span></code></pre></div><p>One easy trap: Whereabouts assigning <code>10.0.2.128</code> inside the pod is not enough. AWS must also know that this private IP belongs to the ENI before an Elastic IP can be associated with it; EC2 treats these as <a href="https://docs.aws.amazon.com/AWSEC2/latest/UserGuide/instance-secondary-ip-addresses.html">secondary private IPs on the network interface</a>. In practice that means the controller or sidecar must reconcile both worlds:</p>
<ol>
<li>Observe the <code>net1</code> address assigned by Whereabouts.</li>
<li>Ensure the private IP is registered on the right ENI.</li>
<li>Only then associate the Elastic IP with that private IP.</li>
</ol>
<p>If step 2 is missing, the design looks right from inside Linux and wrong from the VPC control plane.</p>
<hr>
<h2 id="an-elastic-ip-is-only-half-the-path">An Elastic IP Is Only Half the Path</h2>
<p>AWS Elastic IPs are a 1:1 NAT mapping at the Internet Gateway. An EIP can be <a href="https://docs.aws.amazon.com/AWSEC2/latest/APIReference/API_AssociateAddress.html">associated with a private IP on a network interface</a>, and incoming Internet traffic to the public IP is translated to that ENI private IP.</p>
<p><p>
  <img src="https://kuzaxak.dev/posts/2025/11/28/multus-cni-nat-or-not-to-nat/eip_nat.png" alt="eip_nat.png"  />
</p>
</p>
<details>
  <summary>mermaid</summary>
  <div class="highlight"><pre tabindex="0" style="color:#f8f8f2;background-color:#272822;-moz-tab-size:4;-o-tab-size:4;tab-size:4;-webkit-text-size-adjust:none;"><code class="language-fallback" data-lang="fallback"><span style="display:flex;"><span>sequenceDiagram
</span></span><span style="display:flex;"><span>    participant Peer as Remote Peer
</span></span><span style="display:flex;"><span>    participant IGW as Internet Gateway
</span></span><span style="display:flex;"><span>    participant ENI as Secondary ENI
</span></span><span style="display:flex;"><span>    participant Pod as Pod (net1)
</span></span><span style="display:flex;"><span>
</span></span><span style="display:flex;"><span>    Peer-&gt;&gt;IGW: UDP to 203.0.113.50:5060
</span></span><span style="display:flex;"><span>    Note over IGW: DNAT: 203.0.113.50 → 10.0.2.128
</span></span><span style="display:flex;"><span>    IGW-&gt;&gt;ENI: UDP to 10.0.2.128:5060
</span></span><span style="display:flex;"><span>    ENI-&gt;&gt;Pod: Delivered to net1
</span></span><span style="display:flex;"><span>    Note over Pod: App sees dst=10.0.2.128:5060&lt;br/&gt;(Private IP after DNAT)
</span></span></code></pre></div>
</details>

<p>The return packet has to leave through the same public path. If Linux sends it through <code>eth0</code>, the packet is either dropped or exits through a path that does not match the advertised address.</p>
<p><p>
  <img src="https://kuzaxak.dev/posts/2025/11/28/multus-cni-nat-or-not-to-nat/eip_dnat.png" alt="eip_dnat.png"  />
</p>
</p>
<details>
  <summary>mermaid</summary>
  <div class="highlight"><pre tabindex="0" style="color:#f8f8f2;background-color:#272822;-moz-tab-size:4;-o-tab-size:4;tab-size:4;-webkit-text-size-adjust:none;"><code class="language-fallback" data-lang="fallback"><span style="display:flex;"><span>sequenceDiagram
</span></span><span style="display:flex;"><span>    participant Pod as Pod (net1)
</span></span><span style="display:flex;"><span>    participant ENI as Secondary ENI
</span></span><span style="display:flex;"><span>    participant IGW as Internet Gateway
</span></span><span style="display:flex;"><span>    participant Peer as Remote Peer
</span></span><span style="display:flex;"><span>
</span></span><span style="display:flex;"><span>    Pod-&gt;&gt;ENI: UDP from 10.0.2.128:5060
</span></span><span style="display:flex;"><span>    Note over ENI: Source-based routing&lt;br/&gt;directs to secondary ENI
</span></span><span style="display:flex;"><span>    ENI-&gt;&gt;IGW: UDP from 10.0.2.128:5060
</span></span><span style="display:flex;"><span>    Note over IGW: SNAT: 10.0.2.128 → 203.0.113.50
</span></span><span style="display:flex;"><span>    IGW-&gt;&gt;Peer: UDP from 203.0.113.50:5060
</span></span><span style="display:flex;"><span>    Note over Peer: Sees src=203.0.113.50:5060 ✓&lt;br/&gt;Matches advertised address!
</span></span></code></pre></div>
</details>

<p>The sidecar/controller logic is therefore not &quot;allocate an EIP&quot; and stop. It has to reconcile address ownership, association, routing, and cleanup.</p>
<p>I prefer to describe that logic like this:</p>
<div class="highlight"><pre tabindex="0" style="color:#f8f8f2;background-color:#272822;-moz-tab-size:4;-o-tab-size:4;tab-size:4;-webkit-text-size-adjust:none;"><code class="language-go" data-lang="go"><span style="display:flex;"><span><span style="color:#66d9ef">func</span> <span style="color:#a6e22e">reconcilePublicIdentity</span>(<span style="color:#a6e22e">ctx</span> <span style="color:#a6e22e">context</span>.<span style="color:#a6e22e">Context</span>, <span style="color:#a6e22e">pod</span> <span style="color:#a6e22e">Pod</span>) <span style="color:#66d9ef">error</span> {
</span></span><span style="display:flex;"><span>    <span style="color:#a6e22e">privateIP</span> <span style="color:#f92672">:=</span> <span style="color:#a6e22e">ipv4Address</span>(<span style="color:#e6db74">&#34;net1&#34;</span>)
</span></span><span style="display:flex;"><span>
</span></span><span style="display:flex;"><span>    <span style="color:#a6e22e">eniID</span> <span style="color:#f92672">:=</span> <span style="color:#a6e22e">ensurePrivateIPOnSecondaryENI</span>(<span style="color:#a6e22e">ctx</span>, <span style="color:#a6e22e">privateIP</span>)
</span></span><span style="display:flex;"><span>    <span style="color:#a6e22e">publicIP</span> <span style="color:#f92672">:=</span> <span style="color:#a6e22e">claimElasticIP</span>(<span style="color:#a6e22e">ctx</span>, <span style="color:#e6db74">&#34;telecom-pool&#34;</span>)
</span></span><span style="display:flex;"><span>
</span></span><span style="display:flex;"><span>    <span style="color:#a6e22e">associateElasticIP</span>(<span style="color:#a6e22e">ctx</span>, <span style="color:#a6e22e">publicIP</span>, <span style="color:#a6e22e">eniID</span>, <span style="color:#a6e22e">privateIP</span>)
</span></span><span style="display:flex;"><span>    <span style="color:#a6e22e">replaceVPCRoutes</span>(<span style="color:#a6e22e">ctx</span>, <span style="color:#a6e22e">publicIP</span><span style="color:#f92672">+</span><span style="color:#e6db74">&#34;/32&#34;</span>, <span style="color:#a6e22e">eniID</span>)
</span></span><span style="display:flex;"><span>
</span></span><span style="display:flex;"><span>    <span style="color:#a6e22e">addLocalAddress</span>(<span style="color:#e6db74">&#34;net1&#34;</span>, <span style="color:#a6e22e">publicIP</span><span style="color:#f92672">+</span><span style="color:#e6db74">&#34;/32&#34;</span>)
</span></span><span style="display:flex;"><span>    <span style="color:#a6e22e">addPolicyRoute</span>(<span style="color:#e6db74">&#34;from &#34;</span><span style="color:#f92672">+</span><span style="color:#a6e22e">privateIP</span><span style="color:#f92672">+</span><span style="color:#e6db74">&#34;/32&#34;</span>, <span style="color:#e6db74">&#34;telecom&#34;</span>)
</span></span><span style="display:flex;"><span>    <span style="color:#a6e22e">addPolicyRoute</span>(<span style="color:#e6db74">&#34;from &#34;</span><span style="color:#f92672">+</span><span style="color:#a6e22e">publicIP</span><span style="color:#f92672">+</span><span style="color:#e6db74">&#34;/32&#34;</span>, <span style="color:#e6db74">&#34;telecom&#34;</span>)
</span></span><span style="display:flex;"><span>
</span></span><span style="display:flex;"><span>    <span style="color:#a6e22e">writeFile</span>(<span style="color:#e6db74">&#34;/shared/public-ip&#34;</span>, <span style="color:#a6e22e">publicIP</span>)
</span></span><span style="display:flex;"><span>    <span style="color:#66d9ef">return</span> <span style="color:#66d9ef">nil</span>
</span></span><span style="display:flex;"><span>}
</span></span></code></pre></div><p>That is deliberately incomplete. A real implementation needs idempotency, retries, tag-based ownership, stale route cleanup, race handling, and a safe answer for &quot;what if the pod dies before cleanup runs?&quot;</p>
<hr>
<h2 id="replies-must-leave-through-net1">Replies Must Leave Through net1</h2>
<p>Linux chooses routes primarily by destination. A packet received on <code>net1</code> does not automatically reply through <code>net1</code>.</p>
<p><p>
  <img src="https://kuzaxak.dev/posts/2025/11/28/multus-cni-nat-or-not-to-nat/source_routing_d.png" alt="source_routing_d.png"  />
</p>
</p>
<details>
  <summary>mermaid</summary>
  <div class="highlight"><pre tabindex="0" style="color:#f8f8f2;background-color:#272822;-moz-tab-size:4;-o-tab-size:4;tab-size:4;-webkit-text-size-adjust:none;"><code class="language-fallback" data-lang="fallback"><span style="display:flex;"><span>flowchart TB
</span></span><span style="display:flex;"><span>    subgraph &#34;Pod with Two Interfaces&#34;
</span></span><span style="display:flex;"><span>        App[App receives packet&lt;br/&gt;on net1, sends reply]
</span></span><span style="display:flex;"><span>        eth0[eth0&lt;br/&gt;10.0.1.50&lt;br/&gt;default route]
</span></span><span style="display:flex;"><span>        net1[net1&lt;br/&gt;10.0.2.128]
</span></span><span style="display:flex;"><span>    end
</span></span><span style="display:flex;"><span>
</span></span><span style="display:flex;"><span>    subgraph &#34;Without Source Routing&#34;
</span></span><span style="display:flex;"><span>        App --&gt;|&#34;Reply from 10.0.2.128&#34;| Decision{Routing&lt;br/&gt;Decision}
</span></span><span style="display:flex;"><span>        Decision --&gt;|&#34;Default route&lt;br/&gt;says use eth0&#34;| eth0
</span></span><span style="display:flex;"><span>        eth0 --&gt;|&#34;Packet exits eth0&lt;br/&gt;with src=10.0.2.128&#34;| Wrong[❌ Wrong interface!&lt;br/&gt;Packet dropped or&lt;br/&gt;asymmetric routing breaks NAT]
</span></span><span style="display:flex;"><span>    end
</span></span></code></pre></div>
</details>

<p>Policy routing is the correction:</p>
<div class="highlight"><pre tabindex="0" style="color:#f8f8f2;background-color:#272822;-moz-tab-size:4;-o-tab-size:4;tab-size:4;-webkit-text-size-adjust:none;"><code class="language-bash" data-lang="bash"><span style="display:flex;"><span>echo <span style="color:#e6db74">&#34;200 telecom&#34;</span> &gt;&gt; /etc/iproute2/rt_tables
</span></span><span style="display:flex;"><span>
</span></span><span style="display:flex;"><span>ip route add 10.0.2.0/24 dev net1 src 10.0.2.128 table telecom
</span></span><span style="display:flex;"><span>ip route add default via 10.0.2.1 dev net1 table telecom
</span></span><span style="display:flex;"><span>
</span></span><span style="display:flex;"><span>ip rule add from 10.0.2.128/32 table telecom priority <span style="color:#ae81ff">100</span>
</span></span></code></pre></div><p><p>
  <img src="https://kuzaxak.dev/posts/2025/11/28/multus-cni-nat-or-not-to-nat/source_based_routing.png" alt="source_based_routing.png"  />
</p>
</p>
<details>
  <summary>mermaid</summary>
  <div class="highlight"><pre tabindex="0" style="color:#f8f8f2;background-color:#272822;-moz-tab-size:4;-o-tab-size:4;tab-size:4;-webkit-text-size-adjust:none;"><code class="language-fallback" data-lang="fallback"><span style="display:flex;"><span>flowchart TB
</span></span><span style="display:flex;"><span>    Packet[Outbound Packet&lt;br/&gt;src: 10.0.2.128]
</span></span><span style="display:flex;"><span>
</span></span><span style="display:flex;"><span>    Rule1{ip rule check:&lt;br/&gt;from 10.0.2.128?}
</span></span><span style="display:flex;"><span>    Packet --&gt; Rule1
</span></span><span style="display:flex;"><span>
</span></span><span style="display:flex;"><span>    Rule1 --&gt;|&#34;Yes&#34;| Table200[Use table &#39;telecom&#39;]
</span></span><span style="display:flex;"><span>    Rule1 --&gt;|&#34;No&#34;| MainTable[Use table &#39;main&#39;]
</span></span><span style="display:flex;"><span>
</span></span><span style="display:flex;"><span>    Table200 --&gt; Route200[default via 10.0.2.1&lt;br/&gt;dev net1]
</span></span><span style="display:flex;"><span>    Route200 --&gt; net1_out[net1 → Secondary ENI]
</span></span><span style="display:flex;"><span>    net1_out --&gt; Correct[✓ Correct path!&lt;br/&gt;IGW applies EIP NAT]
</span></span><span style="display:flex;"><span>
</span></span><span style="display:flex;"><span>    MainTable --&gt; RouteMain[default via 10.0.1.1&lt;br/&gt;dev eth0]
</span></span><span style="display:flex;"><span>    RouteMain --&gt; eth0_out[eth0 → Primary ENI]
</span></span><span style="display:flex;"><span>    eth0_out --&gt; Wrong[❌ Internal network]
</span></span><span style="display:flex;"><span>
</span></span><span style="display:flex;"><span>    style Table200 fill:#9f9
</span></span><span style="display:flex;"><span>    style Correct fill:#9f9
</span></span><span style="display:flex;"><span>    style Wrong fill:#f66
</span></span></code></pre></div>
</details>

<p>Reverse path filtering is another quiet failure mode. Strict <code>rp_filter</code> can drop packets in setups where the kernel's best return path does not match the ingress interface during startup or failover.</p>
<div class="highlight"><pre tabindex="0" style="color:#f8f8f2;background-color:#272822;-moz-tab-size:4;-o-tab-size:4;tab-size:4;-webkit-text-size-adjust:none;"><code class="language-bash" data-lang="bash"><span style="display:flex;"><span>sysctl -w net.ipv4.conf.all.rp_filter<span style="color:#f92672">=</span><span style="color:#ae81ff">2</span>
</span></span><span style="display:flex;"><span>sysctl -w net.ipv4.conf.net1.rp_filter<span style="color:#f92672">=</span><span style="color:#ae81ff">2</span>
</span></span></code></pre></div><p>And on the secondary ENI, source/destination checks have to be disabled:</p>
<div class="highlight"><pre tabindex="0" style="color:#f8f8f2;background-color:#272822;-moz-tab-size:4;-o-tab-size:4;tab-size:4;-webkit-text-size-adjust:none;"><code class="language-bash" data-lang="bash"><span style="display:flex;"><span>aws ec2 modify-network-interface-attribute <span style="color:#ae81ff">\
</span></span></span><span style="display:flex;"><span>  --network-interface-id eni-xxx <span style="color:#ae81ff">\
</span></span></span><span style="display:flex;"><span>  --no-source-dest-check
</span></span></code></pre></div><p>The rule I ended up using was this:</p>
<blockquote>
<p>The pod may have two interfaces, but the advertised media identity gets exactly one return path.</p>
</blockquote>
<hr>
<h2 id="the-tcpdump-detail-i-did-not-expect">The tcpdump Detail I Did Not Expect</h2>
<p>After the public path worked, the next surprise came from internal traffic.</p>
<p>Internet-originated traffic to an EIP is DNATed by the Internet Gateway. By the time it reaches the pod, the destination is the private IP:</p>
<div class="highlight"><pre tabindex="0" style="color:#f8f8f2;background-color:#272822;-moz-tab-size:4;-o-tab-size:4;tab-size:4;-webkit-text-size-adjust:none;"><code class="language-bash" data-lang="bash"><span style="display:flex;"><span>tcpdump -i net1 udp port <span style="color:#ae81ff">5060</span>
</span></span><span style="display:flex;"><span><span style="color:#75715e"># 198.51.100.50.5060 &gt; 10.0.2.128.5060: UDP</span>
</span></span><span style="display:flex;"><span><span style="color:#75715e">#                       private IP after IGW DNAT</span>
</span></span></code></pre></div><p>But VPC-internal traffic sent through a <code>/32</code> route to the public IP can arrive with the public IP still as the destination:</p>
<div class="highlight"><pre tabindex="0" style="color:#f8f8f2;background-color:#272822;-moz-tab-size:4;-o-tab-size:4;tab-size:4;-webkit-text-size-adjust:none;"><code class="language-bash" data-lang="bash"><span style="display:flex;"><span>tcpdump -i net1 udp port <span style="color:#ae81ff">5060</span>
</span></span><span style="display:flex;"><span><span style="color:#75715e"># 10.0.1.100.5060 &gt; 203.0.113.50.5060: UDP</span>
</span></span><span style="display:flex;"><span><span style="color:#75715e">#                   public IP, no IGW DNAT</span>
</span></span></code></pre></div><p>Without an explicit VPC route, the internal path is not reliable:</p>
<p><p>
  <img src="https://kuzaxak.dev/posts/2025/11/28/multus-cni-nat-or-not-to-nat/floating_1.png" alt="floating_1.png"  />
</p>
</p>
<details>
  <summary>mermaid</summary>
  <div class="highlight"><pre tabindex="0" style="color:#f8f8f2;background-color:#272822;-moz-tab-size:4;-o-tab-size:4;tab-size:4;-webkit-text-size-adjust:none;"><code class="language-fallback" data-lang="fallback"><span style="display:flex;"><span>flowchart TB
</span></span><span style="display:flex;"><span>    subgraph &#34;Without Floating IP&#34;
</span></span><span style="display:flex;"><span>        InternalClient[Internal Service&lt;br/&gt;10.0.1.100]
</span></span><span style="display:flex;"><span>        Pod[Pod&lt;br/&gt;net1: 10.0.2.128&lt;br/&gt;EIP: 203.0.113.50]
</span></span><span style="display:flex;"><span>
</span></span><span style="display:flex;"><span>        InternalClient --&gt;|&#34;Connect to&lt;br/&gt;203.0.113.50:5060&#34;| VPCRouter{VPC Router}
</span></span><span style="display:flex;"><span>        VPCRouter --&gt;|&#34;No route to&lt;br/&gt;203.0.113.50/32&#34;| Dropped[❌ Dropped or&lt;br/&gt;sent to IGW unnecessarily]
</span></span><span style="display:flex;"><span>    end
</span></span></code></pre></div>
</details>

<p>With a floating <code>/32</code> route, the VPC can deliver traffic directly to the ENI:</p>
<p><p>
  <img src="https://kuzaxak.dev/posts/2025/11/28/multus-cni-nat-or-not-to-nat/floating_2.png" alt="floating_2.png"  />
</p>
</p>
<details>
  <summary>mermaid</summary>
  <div class="highlight"><pre tabindex="0" style="color:#f8f8f2;background-color:#272822;-moz-tab-size:4;-o-tab-size:4;tab-size:4;-webkit-text-size-adjust:none;"><code class="language-fallback" data-lang="fallback"><span style="display:flex;"><span>flowchart TB
</span></span><span style="display:flex;"><span>    subgraph &#34;With Floating IP Route&#34;
</span></span><span style="display:flex;"><span>        InternalClient[Internal Service&lt;br/&gt;10.0.1.100]
</span></span><span style="display:flex;"><span>        Pod[Pod&lt;br/&gt;net1: 10.0.2.128&lt;br/&gt;+ 203.0.113.50/32 local]
</span></span><span style="display:flex;"><span>
</span></span><span style="display:flex;"><span>        InternalClient --&gt;|&#34;Connect to&lt;br/&gt;203.0.113.50:5060&#34;| VPCRouter{VPC Router}
</span></span><span style="display:flex;"><span>        VPCRouter --&gt;|&#34;/32 route → ENI&#34;| ENI[Secondary ENI]
</span></span><span style="display:flex;"><span>        ENI --&gt;|&#34;Direct delivery&#34;| Pod
</span></span><span style="display:flex;"><span>
</span></span><span style="display:flex;"><span>        Note[✓ Pod receives packet&lt;br/&gt;dst still = 203.0.113.50]
</span></span><span style="display:flex;"><span>    end
</span></span><span style="display:flex;"><span>
</span></span><span style="display:flex;"><span>    style Pod fill:#9f9
</span></span></code></pre></div>
</details>

<p>That leaves an application decision:</p>
<table>
  <thead>
      <tr>
          <th>Option</th>
          <th>Behavior</th>
          <th>Tradeoff</th>
      </tr>
  </thead>
  <tbody>
      <tr>
          <td>Bind <code>0.0.0.0</code></td>
          <td>Accept either destination address</td>
          <td>Simple, broad listener</td>
      </tr>
      <tr>
          <td>Bind both IPs</td>
          <td>Explicit sockets for private and public IPs</td>
          <td>More application logic</td>
      </tr>
      <tr>
          <td>Local DNAT</td>
          <td>Normalize public destination to private IP</td>
          <td>More iptables state</td>
      </tr>
  </tbody>
</table>
<p>For a service that only expects traffic on <code>net1</code>, local DNAT can be acceptable:</p>
<div class="highlight"><pre tabindex="0" style="color:#f8f8f2;background-color:#272822;-moz-tab-size:4;-o-tab-size:4;tab-size:4;-webkit-text-size-adjust:none;"><code class="language-bash" data-lang="bash"><span style="display:flex;"><span>iptables -t nat -A PREROUTING -i net1 <span style="color:#ae81ff">\
</span></span></span><span style="display:flex;"><span>  -d 203.0.113.50 -j DNAT --to-destination 10.0.2.128
</span></span></code></pre></div><p>The main point is to notice the difference. Otherwise you debug the wrong layer.</p>
<hr>
<h2 id="lifecycle-is-part-of-the-design">Lifecycle Is Part of the Design</h2>
<p>Creating routes and EIP associations at pod startup means cleanup has to be a first-class path, not a best-effort shell trap.</p>
<p><a href="https://kubernetes.io/docs/concepts/workloads/pods/sidecar-containers/">Kubernetes native sidecars</a> help because a restartable init container can start before the main container and terminate after it. That makes it a reasonable place to hold the public identity for the lifetime of the pod.</p>
<p><p>
  <img src="https://kuzaxak.dev/posts/2025/11/28/multus-cni-nat-or-not-to-nat/sidecar_life.png" alt="sidecar_life.png"  />
</p>
</p>
<details>
  <summary>mermaid</summary>
  <div class="highlight"><pre tabindex="0" style="color:#f8f8f2;background-color:#272822;-moz-tab-size:4;-o-tab-size:4;tab-size:4;-webkit-text-size-adjust:none;"><code class="language-fallback" data-lang="fallback"><span style="display:flex;"><span>sequenceDiagram
</span></span><span style="display:flex;"><span>    participant K as Kubelet
</span></span><span style="display:flex;"><span>    participant S as Floating IP Sidecar
</span></span><span style="display:flex;"><span>    participant M as Main Container (SIP)
</span></span><span style="display:flex;"><span>
</span></span><span style="display:flex;"><span>    K-&gt;&gt;S: Start sidecar
</span></span><span style="display:flex;"><span>    S-&gt;&gt;S: setup() - EIP + routes
</span></span><span style="display:flex;"><span>    S--&gt;&gt;K: Running (stays running)
</span></span><span style="display:flex;"><span>
</span></span><span style="display:flex;"><span>    K-&gt;&gt;M: Start main container
</span></span><span style="display:flex;"><span>    M-&gt;&gt;M: SIP server running...
</span></span><span style="display:flex;"><span>
</span></span><span style="display:flex;"><span>    Note over M: Pod termination triggered
</span></span><span style="display:flex;"><span>
</span></span><span style="display:flex;"><span>    K-&gt;&gt;M: SIGTERM
</span></span><span style="display:flex;"><span>    M-&gt;&gt;M: Graceful shutdown
</span></span><span style="display:flex;"><span>    M--&gt;&gt;K: Exited
</span></span><span style="display:flex;"><span>
</span></span><span style="display:flex;"><span>    K-&gt;&gt;S: SIGTERM (after main exits)
</span></span><span style="display:flex;"><span>    S-&gt;&gt;S: cleanup() - remove routes
</span></span><span style="display:flex;"><span>    S--&gt;&gt;K: Exited
</span></span><span style="display:flex;"><span>
</span></span><span style="display:flex;"><span>    Note over K: Pod fully terminated
</span></span></code></pre></div>
</details>

<p>Version detail matters here. Native sidecars were introduced behind a feature gate before becoming a stable API. On current Kubernetes, the pattern is:</p>
<div class="highlight"><pre tabindex="0" style="color:#f8f8f2;background-color:#272822;-moz-tab-size:4;-o-tab-size:4;tab-size:4;-webkit-text-size-adjust:none;"><code class="language-yaml" data-lang="yaml"><span style="display:flex;"><span><span style="color:#f92672">spec</span>:
</span></span><span style="display:flex;"><span>  <span style="color:#f92672">initContainers</span>:
</span></span><span style="display:flex;"><span>  - <span style="color:#f92672">name</span>: <span style="color:#ae81ff">floating-ip-manager</span>
</span></span><span style="display:flex;"><span>    <span style="color:#f92672">image</span>: <span style="color:#ae81ff">my-registry/floating-ip-manager:latest</span>
</span></span><span style="display:flex;"><span>    <span style="color:#f92672">restartPolicy</span>: <span style="color:#ae81ff">Always</span>
</span></span><span style="display:flex;"><span>    <span style="color:#f92672">securityContext</span>:
</span></span><span style="display:flex;"><span>      <span style="color:#f92672">capabilities</span>:
</span></span><span style="display:flex;"><span>        <span style="color:#f92672">add</span>: [<span style="color:#e6db74">&#34;NET_ADMIN&#34;</span>]
</span></span></code></pre></div><p>I would still not rely only on pod shutdown. Node termination, forced deletion, control-plane interruptions, and cloud API timeouts can all leave stale routes or associated EIPs. A separate reconciler should be able to find resources by tag and clean up anything whose owning pod no longer exists.</p>
<p>The IAM policy should also be scoped as far as the implementation allows. A tutorial often shows <code>Resource: &quot;*&quot;</code>, but the real policy should use tag conditions and only the route tables, ENIs, and EIPs this controller is allowed to manage.</p>
<hr>
<h2 id="node-bootstrap-still-matters">Node Bootstrap Still Matters</h2>
<p>The secondary ENI has to be attached, ignored by the VPC CNI, and ready before Multus starts scheduling workloads that depend on it.</p>
<p>The node-level work was roughly:</p>
<div class="highlight"><pre tabindex="0" style="color:#f8f8f2;background-color:#272822;-moz-tab-size:4;-o-tab-size:4;tab-size:4;-webkit-text-size-adjust:none;"><code class="language-bash" data-lang="bash"><span style="display:flex;"><span><span style="color:#75715e"># Discover instance identity using IMDSv2.</span>
</span></span><span style="display:flex;"><span>TOKEN<span style="color:#f92672">=</span><span style="color:#e6db74">&#34;</span><span style="color:#66d9ef">$(</span>curl -s -X PUT http://169.254.169.254/latest/api/token <span style="color:#ae81ff">\
</span></span></span><span style="display:flex;"><span>  -H <span style="color:#e6db74">&#39;X-aws-ec2-metadata-token-ttl-seconds: 21600&#39;</span><span style="color:#66d9ef">)</span><span style="color:#e6db74">&#34;</span>
</span></span><span style="display:flex;"><span>
</span></span><span style="display:flex;"><span>INSTANCE_ID<span style="color:#f92672">=</span><span style="color:#e6db74">&#34;</span><span style="color:#66d9ef">$(</span>curl -s -H <span style="color:#e6db74">&#34;X-aws-ec2-metadata-token: </span>$TOKEN<span style="color:#e6db74">&#34;</span> <span style="color:#ae81ff">\
</span></span></span><span style="display:flex;"><span>  http://169.254.169.254/latest/meta-data/instance-id<span style="color:#66d9ef">)</span><span style="color:#e6db74">&#34;</span>
</span></span><span style="display:flex;"><span>REGION<span style="color:#f92672">=</span><span style="color:#e6db74">&#34;</span><span style="color:#66d9ef">$(</span>curl -s -H <span style="color:#e6db74">&#34;X-aws-ec2-metadata-token: </span>$TOKEN<span style="color:#e6db74">&#34;</span> <span style="color:#ae81ff">\
</span></span></span><span style="display:flex;"><span>  http://169.254.169.254/latest/meta-data/placement/region<span style="color:#66d9ef">)</span><span style="color:#e6db74">&#34;</span>
</span></span><span style="display:flex;"><span>
</span></span><span style="display:flex;"><span><span style="color:#75715e"># Find the secondary ENI attached for the media path.</span>
</span></span><span style="display:flex;"><span>SECONDARY_ENI<span style="color:#f92672">=</span><span style="color:#e6db74">&#34;</span><span style="color:#66d9ef">$(</span>aws ec2 describe-network-interfaces <span style="color:#ae81ff">\
</span></span></span><span style="display:flex;"><span>  --region <span style="color:#e6db74">&#34;</span>$REGION<span style="color:#e6db74">&#34;</span> <span style="color:#ae81ff">\
</span></span></span><span style="display:flex;"><span>  --filters <span style="color:#e6db74">&#34;Name=attachment.instance-id,Values=</span>$INSTANCE_ID<span style="color:#e6db74">&#34;</span> <span style="color:#ae81ff">\
</span></span></span><span style="display:flex;"><span>            <span style="color:#e6db74">&#34;Name=tag:Purpose,Values=multus-telecom&#34;</span> <span style="color:#ae81ff">\
</span></span></span><span style="display:flex;"><span>  --query <span style="color:#e6db74">&#39;NetworkInterfaces[0].NetworkInterfaceId&#39;</span> <span style="color:#ae81ff">\
</span></span></span><span style="display:flex;"><span>  --output text<span style="color:#66d9ef">)</span><span style="color:#e6db74">&#34;</span>
</span></span><span style="display:flex;"><span>
</span></span><span style="display:flex;"><span>aws ec2 modify-network-interface-attribute <span style="color:#ae81ff">\
</span></span></span><span style="display:flex;"><span>  --region <span style="color:#e6db74">&#34;</span>$REGION<span style="color:#e6db74">&#34;</span> <span style="color:#ae81ff">\
</span></span></span><span style="display:flex;"><span>  --network-interface-id <span style="color:#e6db74">&#34;</span>$SECONDARY_ENI<span style="color:#e6db74">&#34;</span> <span style="color:#ae81ff">\
</span></span></span><span style="display:flex;"><span>  --no-source-dest-check
</span></span><span style="display:flex;"><span>
</span></span><span style="display:flex;"><span>aws ec2 create-tags <span style="color:#ae81ff">\
</span></span></span><span style="display:flex;"><span>  --region <span style="color:#e6db74">&#34;</span>$REGION<span style="color:#e6db74">&#34;</span> <span style="color:#ae81ff">\
</span></span></span><span style="display:flex;"><span>  --resources <span style="color:#e6db74">&#34;</span>$SECONDARY_ENI<span style="color:#e6db74">&#34;</span> <span style="color:#ae81ff">\
</span></span></span><span style="display:flex;"><span>  --tags Key<span style="color:#f92672">=</span>node.k8s.amazonaws.com/no_manage,Value<span style="color:#f92672">=</span>true
</span></span></code></pre></div><p>I also prefer to taint these nodes until the Multus config is present. Otherwise a pod can land on a node that looks schedulable but cannot attach <code>net1</code> yet.</p>
<div class="highlight"><pre tabindex="0" style="color:#f8f8f2;background-color:#272822;-moz-tab-size:4;-o-tab-size:4;tab-size:4;-webkit-text-size-adjust:none;"><code class="language-yaml" data-lang="yaml"><span style="display:flex;"><span><span style="color:#f92672">taints</span>:
</span></span><span style="display:flex;"><span>- <span style="color:#f92672">key</span>: <span style="color:#ae81ff">multus-not-ready</span>
</span></span><span style="display:flex;"><span>  <span style="color:#f92672">effect</span>: <span style="color:#ae81ff">NoSchedule</span>
</span></span></code></pre></div><p>A tiny untainter DaemonSet can remove the taint after <code>/etc/cni/net.d/00-multus.conf</code> exists. The exact mechanism is less important than the invariant: do not schedule media pods before the second network path exists.</p>
<hr>
<h2 id="what-i-would-validate-before-trusting-it">What I Would Validate Before Trusting It</h2>
<p>I would not call this design working until these checks pass on a fresh pod:</p>
<div class="highlight"><pre tabindex="0" style="color:#f8f8f2;background-color:#272822;-moz-tab-size:4;-o-tab-size:4;tab-size:4;-webkit-text-size-adjust:none;"><code class="language-bash" data-lang="bash"><span style="display:flex;"><span>ip addr show net1
</span></span><span style="display:flex;"><span>ip rule
</span></span><span style="display:flex;"><span>ip route show table telecom
</span></span><span style="display:flex;"><span>ip route get 198.51.100.50 from 10.0.2.128
</span></span></code></pre></div><p>From AWS:</p>
<div class="highlight"><pre tabindex="0" style="color:#f8f8f2;background-color:#272822;-moz-tab-size:4;-o-tab-size:4;tab-size:4;-webkit-text-size-adjust:none;"><code class="language-bash" data-lang="bash"><span style="display:flex;"><span>aws ec2 describe-network-interfaces <span style="color:#ae81ff">\
</span></span></span><span style="display:flex;"><span>  --filters <span style="color:#e6db74">&#34;Name=addresses.private-ip-address,Values=10.0.2.128&#34;</span>
</span></span><span style="display:flex;"><span>
</span></span><span style="display:flex;"><span>aws ec2 describe-addresses <span style="color:#ae81ff">\
</span></span></span><span style="display:flex;"><span>  --filters <span style="color:#e6db74">&#34;Name=public-ip,Values=203.0.113.50&#34;</span>
</span></span><span style="display:flex;"><span>
</span></span><span style="display:flex;"><span>aws ec2 describe-route-tables <span style="color:#ae81ff">\
</span></span></span><span style="display:flex;"><span>  --filters <span style="color:#e6db74">&#34;Name=route.destination-cidr-block,Values=203.0.113.50/32&#34;</span>
</span></span></code></pre></div><p>From the wire:</p>
<div class="highlight"><pre tabindex="0" style="color:#f8f8f2;background-color:#272822;-moz-tab-size:4;-o-tab-size:4;tab-size:4;-webkit-text-size-adjust:none;"><code class="language-bash" data-lang="bash"><span style="display:flex;"><span>tcpdump -ni net1 udp port <span style="color:#ae81ff">5060</span>
</span></span><span style="display:flex;"><span>conntrack -L -p udp | grep <span style="color:#ae81ff">5060</span>
</span></span></code></pre></div><p>And after deletion:</p>
<div class="highlight"><pre tabindex="0" style="color:#f8f8f2;background-color:#272822;-moz-tab-size:4;-o-tab-size:4;tab-size:4;-webkit-text-size-adjust:none;"><code class="language-bash" data-lang="bash"><span style="display:flex;"><span>kubectl delete pod sip-server
</span></span><span style="display:flex;"><span>
</span></span><span style="display:flex;"><span><span style="color:#75715e"># Then verify:</span>
</span></span><span style="display:flex;"><span><span style="color:#75715e"># - the EIP is disassociated or returned to the expected pool</span>
</span></span><span style="display:flex;"><span><span style="color:#75715e"># - /32 routes are gone or point to the new owner</span>
</span></span><span style="display:flex;"><span><span style="color:#75715e"># - no stale private IP remains assigned to the ENI</span>
</span></span></code></pre></div><p>If cleanup only works in the happy path, it is not cleanup. It is a demo.</p>
<hr>
<h2 id="what-this-design-commits-you-to">What This Design Commits You To</h2>
<p>This design gives a pod a stable public media identity without moving the whole workload back to bare EC2. It also makes network identity part of workload lifecycle.</p>
<p>The useful properties are:</p>
<ul>
<li>SIP/RTP can advertise an address that matches the packet path.</li>
<li>Multiple workloads can run in the same cluster without sharing a node IP.</li>
<li>Kubernetes still owns rollout, placement, and normal service operations.</li>
<li>Media traffic is separated from normal pod traffic.</li>
</ul>
<p>The constraints are now explicit:</p>
<ul>
<li>Capacity is bounded by EIPs, public IPv4 cost, subnet reservations, and ENI private IP quota before CPU or memory may become relevant.</li>
<li>The controller needs cloud API permissions, careful tagging, and ownership checks.</li>
<li>Stale routes, EIP associations, and secondary private IP assignments need reconciliation.</li>
<li>Public UDP exposure needs security groups, monitoring, and abuse handling.</li>
<li>Multi-AZ placement must match the selected NAD, subnet, ENI, and route tables.</li>
</ul>
<p>For SIP/RTP workloads that need stable public tuples, this is the Kubernetes design that matched the protocol constraint instead of hiding it behind another translation layer.</p>
<hr>
<h2 id="references">References</h2>
<ul>
<li><a href="https://aws.amazon.com/blogs/containers/amazon-eks-now-supports-multus-cni/">Amazon EKS Multus Support</a></li>
<li><a href="https://github.com/aws-samples/eks-install-guide-for-multus">EKS Multus Installation Guide</a></li>
<li><a href="https://docs.aws.amazon.com/eks/latest/userguide/pod-multus.html">Amazon EKS Multus User Guide</a></li>
<li><a href="https://aws.amazon.com/blogs/industries/automated-ip-address-management-for-multus-workers-and-pods/">Automated IP Management for Multus Pods</a></li>
<li><a href="https://github.com/k8snetworkplumbingwg/whereabouts">Whereabouts IPAM</a></li>
<li><a href="https://aws.amazon.com/blogs/industries/automated-application-failover-across-availability-zones-with-floating-virtual-ip-on-amazon-eks/">Floating Virtual IP on EKS</a></li>
<li><a href="https://cloud.google.com/kubernetes-engine/docs/concepts/multus-ipvlan-whereabouts">GCP Multus with ipvlan</a></li>
<li><a href="https://www.rfc-editor.org/rfc/rfc8445">ICE: RFC 8445</a></li>
<li><a href="https://www.rfc-editor.org/rfc/rfc8489">STUN: RFC 8489</a></li>
<li><a href="https://www.rfc-editor.org/rfc/rfc8656">TURN: RFC 8656</a></li>
<li><a href="https://www.rfc-editor.org/rfc/rfc3261">SIP: RFC 3261</a></li>
<li><a href="https://www.rfc-editor.org/rfc/rfc3581">SIP rport: RFC 3581</a></li>
<li><a href="https://www.rfc-editor.org/rfc/rfc5626">SIP Outbound: RFC 5626</a></li>
<li><a href="https://www.rfc-editor.org/rfc/rfc8839">ICE SDP usage: RFC 8839</a></li>
<li><a href="https://www.rfc-editor.org/rfc/rfc8840">Trickle ICE for SIP: RFC 8840</a></li>
<li><a href="https://kubernetes.io/docs/concepts/workloads/pods/sidecar-containers/">Kubernetes Sidecar Containers</a></li>
<li><a href="https://docs.aws.amazon.com/vpc/latest/userguide/subnet-cidr-reservation.html">AWS Subnet CIDR Reservations</a></li>
<li><a href="https://docs.aws.amazon.com/AWSEC2/latest/UserGuide/instance-secondary-ip-addresses.html">AWS EC2 Secondary Private IPs</a></li>
<li><a href="https://docs.aws.amazon.com/AWSEC2/latest/APIReference/API_AssociateAddress.html">AWS EC2 AssociateAddress</a></li>
<li><a href="https://docs.aws.amazon.com/elasticloadbalancing/latest/network/introduction.html">AWS Network Load Balancer routing</a></li>
<li><a href="https://docs.aws.amazon.com/elasticloadbalancing/latest/network/network-load-balancers.html#connection-idle-timeout">AWS Network Load Balancer idle timeout</a></li>
<li><a href="https://docs.aws.amazon.com/elasticloadbalancing/latest/network/load-balancer-limits.html">AWS Network Load Balancer quotas</a></li>
</ul>
]]></content:encoded>
    </item>
    
    <item>
      <title>Use AWS SSO as ABAC IAM for granular secure access</title>
      <link>https://kuzaxak.dev/posts/2025/06/30/aws-iam-sso-abac/</link>
      <pubDate>Mon, 30 Jun 2025 00:00:00 +0000</pubDate>
      
      <guid>https://kuzaxak.dev/posts/2025/06/30/aws-iam-sso-abac/</guid>
      <description>Learn how to implement ABAC with AWS SSO and your Identity Provider</description>
      <content:encoded><![CDATA[<h2 id="intro">Intro</h2>
<p>Most security teams want to limit developers' access to only their own resources in AWS. This is especially true for databases.
Previously, we solved this by creating multiple IAM roles, where users would choose the appropriate role to access specific cloud resources.
This works fine on a small scale, but doesn't scale beyond 3+ roles or teams.</p>
<p>Instead, we can use attribute-based access control (ABAC) to compare user properties with resource tags and determine access permissions.</p>
<p>I decided to write this article because I couldn't find a guide on how to do it. The <a href="https://docs.aws.amazon.com/singlesignon/latest/userguide/attributesforaccesscontrol.html">AWS Doc</a> is quite limited.</p>
<h2 id="aws-saml-vs-aws-sso">AWS SAML vs AWS SSO</h2>
<p>The AWS SAML auth provider linked with an IAM role has many more capabilities for SAML attributes and is <a href="https://docs.aws.amazon.com/IAM/latest/UserGuide/id_roles_providers_create_saml_assertions.html">well-documented</a>.</p>
<p>AWS SSO has limited functionality and does not support:</p>
<ul>
<li>Multi-valued SAML attributes</li>
<li><code>TransitiveTagKeys</code></li>
</ul>
<p>To pass attributes from an IdP via SAML to an assumed session's <code>PrincipalTag</code>, you can use only <code>https://aws.amazon.com/SAML/Attributes/AccessControl:&lt;name&gt;</code> with string-typed values up to 460 bytes long.</p>
<p>We use <code>::</code> as a separator for concatenating multiple values and <code>:</code> as a logical separator within individual entities.</p>
<h2 id="schema">Schema</h2>
<p>Authentication and authorization flow:
<p>
  <img src="https://kuzaxak.dev/posts/2025/06/30/aws-iam-sso-abac/diag-1.svg" alt="diag"  />
</p>
</p>
<details>
  <summary>Diagram source</summary>
  <div class="highlight"><pre tabindex="0" style="color:#f8f8f2;background-color:#272822;-moz-tab-size:4;-o-tab-size:4;tab-size:4;-webkit-text-size-adjust:none;"><code class="language-fallback" data-lang="fallback"><span style="display:flex;"><span>sequenceDiagram
</span></span><span style="display:flex;"><span>    participant User
</span></span><span style="display:flex;"><span>    participant AWS_SSO as AWS SSO
</span></span><span style="display:flex;"><span>    participant IDP
</span></span><span style="display:flex;"><span>    participant Target_AWS_Account as Target AWS Account
</span></span><span style="display:flex;"><span>
</span></span><span style="display:flex;"><span>    User-&gt;&gt;AWS_SSO: Initiates Login
</span></span><span style="display:flex;"><span>    AWS_SSO-&gt;&gt;IDP: Redirects user for authentication (SAML Request)
</span></span><span style="display:flex;"><span>    IDP--&gt;&gt;User: Login Page
</span></span><span style="display:flex;"><span>    User-&gt;&gt;IDP: Enters Credentials
</span></span><span style="display:flex;"><span>    IDP-&gt;&gt;AWS_SSO: SAML Assertion with&lt;br/&gt;AccessControl attributes
</span></span><span style="display:flex;"><span>    AWS_SSO-&gt;&gt;Target_AWS_Account: AssumeRoleWithSAML
</span></span><span style="display:flex;"><span>    Target_AWS_Account--&gt;&gt;AWS_SSO: Assumed Role Session&lt;br/&gt;with principal tags
</span></span><span style="display:flex;"><span>    AWS_SSO--&gt;&gt;User: Grants access to AWS Account
</span></span></code></pre></div>
</details>

<h2 id="idp">IdP</h2>
<p>To configure role attributes, we'll use Authentik (our SAML identity provider) that's already configured with AWS SSO.
The first step is to create the <code>Property Mappings</code>.</p>
<p><p>
  <img src="https://kuzaxak.dev/posts/2025/06/30/aws-iam-sso-abac/mappings.png" alt="mappings"  />
</p>
</p>
<div class="highlight"><pre tabindex="0" style="color:#f8f8f2;background-color:#272822;-moz-tab-size:4;-o-tab-size:4;tab-size:4;-webkit-text-size-adjust:none;"><code class="language-python" data-lang="python"><span style="display:flex;"><span>attributes_list <span style="color:#f92672">=</span> request<span style="color:#f92672">.</span>user<span style="color:#f92672">.</span>group_attributes()<span style="color:#f92672">.</span>get(<span style="color:#e6db74">&#34;security:authz:groups&#34;</span>, [])
</span></span><span style="display:flex;"><span>unique_attributes_set <span style="color:#f92672">=</span> set(attributes_list)
</span></span><span style="display:flex;"><span>
</span></span><span style="display:flex;"><span><span style="color:#66d9ef">return</span> <span style="color:#e6db74">&#34;::&#34;</span><span style="color:#f92672">.</span>join(unique_attributes_set)
</span></span></code></pre></div><p>This code collects properties from the user's assigned groups, deduplicates them, and concatenates the results using <code>::</code>.</p>
<p>After configuring the property mappings, log out of AWS SSO and log back in to test the configuration.</p>
 
 
<img
    src="/posts/2025/06/30/aws-iam-sso-abac/logout.png"
    alt="logout"
    style="
        display: inline-block;
        vertical-align: middle;
        height: 8em;
        margin: 0 0.2em;
    "
/>


<p>You should see a new record for <code>AssumeRoleWithSAML</code> in your CloudTrail events. The <code>principalTags</code> from this event can be used in IAM conditions.</p>
<div class="highlight"><pre tabindex="0" style="color:#f8f8f2;background-color:#272822;-moz-tab-size:4;-o-tab-size:4;tab-size:4;-webkit-text-size-adjust:none;"><code class="language-json" data-lang="json"><span style="display:flex;"><span>{
</span></span><span style="display:flex;"><span>  <span style="color:#f92672">&#34;requestParameters&#34;</span>: {
</span></span><span style="display:flex;"><span>      <span style="color:#f92672">&#34;sAMLAssertionID&#34;</span>: <span style="color:#e6db74">&#34;_550580ed-e988-46da-bebd-38d9bcd1f156&#34;</span>,
</span></span><span style="display:flex;"><span>      <span style="color:#f92672">&#34;roleSessionName&#34;</span>: <span style="color:#e6db74">&#34;XXXXX@qdo.ee&#34;</span>,
</span></span><span style="display:flex;"><span>      <span style="color:#f92672">&#34;principalTags&#34;</span>: {
</span></span><span style="display:flex;"><span>          <span style="color:#f92672">&#34;security:authz:dbusers&#34;</span>: <span style="color:#e6db74">&#34;beta:developer-iam::prod-us:developer-iam&#34;</span>,
</span></span><span style="display:flex;"><span>          <span style="color:#f92672">&#34;security:authz:groups&#34;</span>: <span style="color:#e6db74">&#34;beta:pfr:project::any:rpsql:project&#34;</span>
</span></span><span style="display:flex;"><span>      }
</span></span><span style="display:flex;"><span>}
</span></span></code></pre></div><h3 id="iam">IAM</h3>
<p>As mentioned in the beginning, to limit developers to certain resources in their domain of responsibility, we'll match the passed <code>aws:PrincipalTag/security:authz:groups</code> with a regexp formed from the <code>ResourceTag</code> assigned to the database:</p>
<div class="highlight"><pre tabindex="0" style="color:#f8f8f2;background-color:#272822;-moz-tab-size:4;-o-tab-size:4;tab-size:4;-webkit-text-size-adjust:none;"><code class="language-gdscript3" data-lang="gdscript3"><span style="display:flex;"><span>    condition {
</span></span><span style="display:flex;"><span>      test <span style="color:#f92672">=</span> <span style="color:#e6db74">&#34;StringLike&#34;</span>
</span></span><span style="display:flex;"><span>      values <span style="color:#f92672">=</span> flatten([
</span></span><span style="display:flex;"><span>        [<span style="color:#66d9ef">for</span> env <span style="color:#f92672">in</span> <span style="color:#66d9ef">var</span><span style="color:#f92672">.</span>allowed_environment : <span style="color:#e6db74">&#34;*${env}:$${ssm:ResourceTag/security:access:st}:$${ssm:ResourceTag/security:access:group}*&#34;</span>],
</span></span><span style="display:flex;"><span>        [<span style="color:#e6db74">&#34;*any:$${ssm:ResourceTag/security:access:st}:$${ssm:ResourceTag/security:access:group}*&#34;</span>]
</span></span><span style="display:flex;"><span>      ])
</span></span><span style="display:flex;"><span>
</span></span><span style="display:flex;"><span>      variable <span style="color:#f92672">=</span> <span style="color:#e6db74">&#34;aws:PrincipalTag/security:authz:groups&#34;</span>
</span></span><span style="display:flex;"><span>    }
</span></span></code></pre></div>]]></content:encoded>
    </item>
    
    <item>
      <title>CentOS: Fix CVE-2024-6387 by Updating OpenSSH</title>
      <link>https://kuzaxak.dev/posts/2024/07/01/centos-upgrade-ssh-cve/</link>
      <pubDate>Mon, 01 Jul 2024 00:00:00 +0000</pubDate>
      
      <guid>https://kuzaxak.dev/posts/2024/07/01/centos-upgrade-ssh-cve/</guid>
      <description>A step-by-step guide to upgrading OpenSSH on CentOS to mitigate CVE-2024-6387</description>
      <content:encoded><![CDATA[<h1 id="introduction">Introduction</h1>
<p>CVE-2024-6387, a recently discovered vulnerability, allows for remote code execution (RCE) on affected servers running OpenSSH versions. Given CentOS is deprecated, this guide details how to upgrade to the latest versions of OpenSSL and OpenSSH to mitigate this risk.</p>
<h2 id="affected-openssh-versions">Affected OpenSSH versions</h2>
<ul>
<li>OpenSSH versions earlier than 4.4p1 are vulnerable to this signal handler race condition unless they are patched for CVE-2006-5051 and CVE-2008-4109.</li>
<li>Versions from 4.4p1 up to, but not including, 8.5p1 are not vulnerable due to a transformative patch for CVE-2006-5051, which made a previously unsafe function secure.</li>
<li>The vulnerability resurfaces in versions from 8.5p1 up to, but not including, 9.8p1 due to the accidental removal of a critical component in a function.</li>
</ul>
<h1 id="installation-steps">Installation Steps</h1>
<h2 id="install-dependencies">Install Dependencies</h2>
<p>First, install the necessary development tools and libraries:</p>
<div class="highlight"><pre tabindex="0" style="color:#f8f8f2;background-color:#272822;-moz-tab-size:4;-o-tab-size:4;tab-size:4;-webkit-text-size-adjust:none;"><code class="language-shell" data-lang="shell"><span style="display:flex;"><span>yum groupinstall -y <span style="color:#e6db74">&#34;Development Tools&#34;</span>
</span></span><span style="display:flex;"><span>yum install -y wget perl coreutils perl-IPC-Cmd perl-Data-Dumper pam-devel
</span></span></code></pre></div><h2 id="backup-existing-openssl">Backup Existing OpenSSL</h2>
<p>Backup the current OpenSSL to avoid conflicts:</p>
<div class="highlight"><pre tabindex="0" style="color:#f8f8f2;background-color:#272822;-moz-tab-size:4;-o-tab-size:4;tab-size:4;-webkit-text-size-adjust:none;"><code class="language-shell" data-lang="shell"><span style="display:flex;"><span>mv /usr/bin/openssl /usr/bin/openssl_backup
</span></span></code></pre></div><h2 id="install-latest-openssl">Install Latest OpenSSL</h2>
<p>Check for the latest version on the <a href="https://www.openssl.org/source/">OpenSSL website</a>. For this guide, we use version <code>3.3.1</code>:</p>
<div class="highlight"><pre tabindex="0" style="color:#f8f8f2;background-color:#272822;-moz-tab-size:4;-o-tab-size:4;tab-size:4;-webkit-text-size-adjust:none;"><code class="language-shell" data-lang="shell"><span style="display:flex;"><span>OPENSSL_VERSION<span style="color:#f92672">=</span><span style="color:#e6db74">&#34;3.3.1&#34;</span>
</span></span><span style="display:flex;"><span>
</span></span><span style="display:flex;"><span>wget https://www.openssl.org/source/openssl-$OPENSSL_VERSION.tar.gz
</span></span><span style="display:flex;"><span>tar -zxvf openssl-$OPENSSL_VERSION.tar.gz
</span></span><span style="display:flex;"><span>cd openssl-$OPENSSL_VERSION
</span></span><span style="display:flex;"><span>./config --prefix<span style="color:#f92672">=</span>/usr/local/openssl --openssldir<span style="color:#f92672">=</span>/usr/local/openssl shared zlib
</span></span><span style="display:flex;"><span>make
</span></span><span style="display:flex;"><span>make install
</span></span><span style="display:flex;"><span>
</span></span><span style="display:flex;"><span><span style="color:#75715e"># Update the system OpenSSL to the new version</span>
</span></span><span style="display:flex;"><span>echo <span style="color:#e6db74">&#34;/usr/local/openssl/lib&#34;</span> &gt; /etc/ld.so.conf.d/openssl.conf
</span></span><span style="display:flex;"><span>ldconfig
</span></span><span style="display:flex;"><span>ln -sf /usr/local/openssl/bin/openssl /usr/bin/openssl
</span></span><span style="display:flex;"><span>ln -s /usr/local/openssl/lib64/libssl.so.3 /usr/lib64/
</span></span><span style="display:flex;"><span>ln -s /usr/local/openssl/lib64/libcrypto.so.3 /usr/lib64/
</span></span><span style="display:flex;"><span>openssl version -a
</span></span></code></pre></div><h2 id="install-latest-openssh">Install Latest OpenSSH</h2>
<p>First, backup the existing OpenSSH configuration:</p>
<div class="highlight"><pre tabindex="0" style="color:#f8f8f2;background-color:#272822;-moz-tab-size:4;-o-tab-size:4;tab-size:4;-webkit-text-size-adjust:none;"><code class="language-shell" data-lang="shell"><span style="display:flex;"><span>mkdir /etc/ssh_old
</span></span><span style="display:flex;"><span>mv /etc/ssh/* /etc/ssh_old/
</span></span></code></pre></div><p>Then, download and install the <a href="https://github.com/openssh/openssh-portable/tags">latest OpenSSH</a> version (<code>9.8p1</code> for this guide):</p>
<div class="highlight"><pre tabindex="0" style="color:#f8f8f2;background-color:#272822;-moz-tab-size:4;-o-tab-size:4;tab-size:4;-webkit-text-size-adjust:none;"><code class="language-shell" data-lang="shell"><span style="display:flex;"><span>OPENSSH_VERSION<span style="color:#f92672">=</span><span style="color:#e6db74">&#34;9.8p1&#34;</span>
</span></span><span style="display:flex;"><span>wget https://ftp.openbsd.org/pub/OpenBSD/OpenSSH/portable/openssh-$OPENSSH_VERSION.tar.gz
</span></span><span style="display:flex;"><span>tar -zxvf openssh-$OPENSSH_VERSION.tar.gz
</span></span><span style="display:flex;"><span>cd openssh-$OPENSSH_VERSION
</span></span><span style="display:flex;"><span>
</span></span><span style="display:flex;"><span><span style="color:#75715e"># Configure OpenSSH with the new OpenSSL</span>
</span></span><span style="display:flex;"><span>./configure --prefix<span style="color:#f92672">=</span>/usr --sysconfdir<span style="color:#f92672">=</span>/etc/ssh --with-ssl-dir<span style="color:#f92672">=</span>/usr/local/openssl --with-zlib --with-ssl-engine --with-pam
</span></span><span style="display:flex;"><span>make
</span></span><span style="display:flex;"><span>make install
</span></span></code></pre></div><p>Verify the installation:</p>
<div class="highlight"><pre tabindex="0" style="color:#f8f8f2;background-color:#272822;-moz-tab-size:4;-o-tab-size:4;tab-size:4;-webkit-text-size-adjust:none;"><code class="language-fallback" data-lang="fallback"><span style="display:flex;"><span>ssh -V
</span></span><span style="display:flex;"><span># OpenSSH_9.8p1, OpenSSL 3.3.1 4 Jun 2024
</span></span></code></pre></div><h2 id="update-configuration">Update Configuration</h2>
<p>Compare the old and new SSH configuration files:</p>
<div class="highlight"><pre tabindex="0" style="color:#f8f8f2;background-color:#272822;-moz-tab-size:4;-o-tab-size:4;tab-size:4;-webkit-text-size-adjust:none;"><code class="language-shell" data-lang="shell"><span style="display:flex;"><span>diff /etc/ssh/sshd_config /etc/ssh_old/sshd_config
</span></span><span style="display:flex;"><span><span style="color:#75715e"># Edit the new one if needed</span>
</span></span><span style="display:flex;"><span>vim /etc/ssh/sshd_config
</span></span></code></pre></div><p>Replace the SSH daemon service files:</p>
<div class="highlight"><pre tabindex="0" style="color:#f8f8f2;background-color:#272822;-moz-tab-size:4;-o-tab-size:4;tab-size:4;-webkit-text-size-adjust:none;"><code class="language-shell" data-lang="shell"><span style="display:flex;"><span>mv /usr/lib/systemd/system/sshd.service /etc/ssh_old/sshd.service
</span></span><span style="display:flex;"><span>mv /usr/lib/systemd/system/sshd.socket /etc/ssh_old/sshd.socket
</span></span><span style="display:flex;"><span>cp -a contrib/redhat/sshd.init /etc/init.d/sshd
</span></span><span style="display:flex;"><span>
</span></span><span style="display:flex;"><span>systemctl daemon-reload
</span></span><span style="display:flex;"><span>systemctl restart sshd
</span></span><span style="display:flex;"><span>
</span></span><span style="display:flex;"><span>chkconfig --add sshd
</span></span><span style="display:flex;"><span>chkconfig sshd on
</span></span></code></pre></div><h2 id="testing-the-new-configuration">Testing the New Configuration</h2>
<p>Finally, test the new SSH setup by connecting to the server using the <code>-vv</code> flag to verify the protocol version in use.</p>
<div class="highlight"><pre tabindex="0" style="color:#f8f8f2;background-color:#272822;-moz-tab-size:4;-o-tab-size:4;tab-size:4;-webkit-text-size-adjust:none;"><code class="language-shell" data-lang="shell"><span style="display:flex;"><span>ssh -vv user@your-server.com
</span></span></code></pre></div><p>Stay secure!</p>
]]></content:encoded>
    </item>
    
    <item>
      <title>Debugging PgLogical Replication</title>
      <link>https://kuzaxak.dev/posts/2024/03/23/pg-pglogical-debugging/</link>
      <pubDate>Sat, 23 Mar 2024 00:00:00 +0000</pubDate>
      
      <guid>https://kuzaxak.dev/posts/2024/03/23/pg-pglogical-debugging/</guid>
      <description>Learn how to debug PgLogical replication issues in PostgreSQL, covering both provider and receiver sides.</description>
      <content:encoded><![CDATA[<p>PgLogical is a powerful logical replication system for PostgreSQL that allows you to replicate data between databases. However, like any complex system, it can sometimes encounter issues. In this article, we'll explore how to debug PgLogical replication problems on both the provider and receiver sides.</p>
<h2 id="provider-side-debugging">Provider Side Debugging</h2>
<p>On the provider side, you can check the delay in data and the current linked WAL file by running the following query:</p>
<div class="highlight"><pre tabindex="0" style="color:#f8f8f2;background-color:#272822;-moz-tab-size:4;-o-tab-size:4;tab-size:4;-webkit-text-size-adjust:none;"><code class="language-sql" data-lang="sql"><span style="display:flex;"><span><span style="color:#66d9ef">SELECT</span> slot_name, <span style="color:#66d9ef">database</span>,
</span></span><span style="display:flex;"><span>       lpad((pg_control_checkpoint()).timeline_id::text, <span style="color:#ae81ff">8</span>, <span style="color:#e6db74">&#39;0&#39;</span>) <span style="color:#f92672">||</span>
</span></span><span style="display:flex;"><span>       lpad(split_part(restart_lsn::text, <span style="color:#e6db74">&#39;/&#39;</span>, <span style="color:#ae81ff">1</span>), <span style="color:#ae81ff">8</span>, <span style="color:#e6db74">&#39;0&#39;</span>) <span style="color:#f92672">||</span>
</span></span><span style="display:flex;"><span>       lpad(substr(split_part(restart_lsn::text, <span style="color:#e6db74">&#39;/&#39;</span>, <span style="color:#ae81ff">2</span>), <span style="color:#ae81ff">1</span>, <span style="color:#ae81ff">2</span>), <span style="color:#ae81ff">8</span>, <span style="color:#e6db74">&#39;0&#39;</span>) <span style="color:#66d9ef">AS</span> wal_file,
</span></span><span style="display:flex;"><span>	   pg_size_pretty(pg_wal_lsn_diff(pg_current_wal_lsn(), confirmed_flush_lsn)) <span style="color:#66d9ef">AS</span> replication_delay
</span></span><span style="display:flex;"><span><span style="color:#66d9ef">FROM</span> pg_replication_slots <span style="color:#66d9ef">WHERE</span> plugin <span style="color:#66d9ef">like</span> <span style="color:#e6db74">&#39;%pglogical%&#39;</span>;
</span></span></code></pre></div><p>In my case, I had a delay of 170GB for one particular database where I had PgLogical enabled. To clarify the status of the replication slot, you can run:</p>
<div class="highlight"><pre tabindex="0" style="color:#f8f8f2;background-color:#272822;-moz-tab-size:4;-o-tab-size:4;tab-size:4;-webkit-text-size-adjust:none;"><code class="language-sql" data-lang="sql"><span style="display:flex;"><span><span style="color:#66d9ef">select</span>   pid, client_addr, application_name, <span style="color:#66d9ef">state</span>, sync_state,
</span></span><span style="display:flex;"><span>         pg_wal_lsn_diff(pg_current_wal_lsn(),sent_lsn) <span style="color:#66d9ef">AS</span> sent_lag,
</span></span><span style="display:flex;"><span>         pg_wal_lsn_diff(sent_lsn,flush_lsn) <span style="color:#66d9ef">AS</span> receiving_lag,
</span></span><span style="display:flex;"><span>         pg_wal_lsn_diff(flush_lsn,replay_lsn) <span style="color:#66d9ef">AS</span> replay_lag,
</span></span><span style="display:flex;"><span>         pg_wal_lsn_diff(pg_current_wal_lsn(),replay_lsn) <span style="color:#66d9ef">AS</span> total_lag,
</span></span><span style="display:flex;"><span>         now()<span style="color:#f92672">-</span>reply_time <span style="color:#66d9ef">AS</span> reply_delay
</span></span><span style="display:flex;"><span><span style="color:#66d9ef">from</span> pg_stat_replication;
</span></span></code></pre></div><p>Pay attention to the <code>reply_delay</code> column, which indicates the time when the provider last received an update from the subscriber database.</p>
<p>If you see <code>catchup</code> in some slots, it means that your receiver is lagging behind, and you need to switch to it to investigate the reason.</p>
<h2 id="receiver-side-debugging">Receiver Side Debugging</h2>
<p>On the receiver side, you can start the investigation from the replication status. Repeat the following query for each logical database that you have configured with PgLogical:</p>
<div class="highlight"><pre tabindex="0" style="color:#f8f8f2;background-color:#272822;-moz-tab-size:4;-o-tab-size:4;tab-size:4;-webkit-text-size-adjust:none;"><code class="language-sql" data-lang="sql"><span style="display:flex;"><span><span style="color:#66d9ef">SELECT</span> subscription_name, status <span style="color:#66d9ef">FROM</span> pglogical.show_subscription_status();
</span></span></code></pre></div><p>If you see <code>down</code> status, you can check logs and grep for <code>pglogical apply</code> and then <code>ERROR</code>. You may have some conflicts in schema.</p>
<p>Log messages may look like:</p>
<div class="highlight"><pre tabindex="0" style="color:#f8f8f2;background-color:#272822;-moz-tab-size:4;-o-tab-size:4;tab-size:4;-webkit-text-size-adjust:none;"><code class="language-zed" data-lang="zed"><span style="display:flex;"><span>[<span style="color:#960050;background-color:#1e0010">2641263</span>]<span style="color:#f92672">:</span> [<span style="color:#960050;background-color:#1e0010">5</span><span style="color:#f92672">-</span><span style="color:#960050;background-color:#1e0010">1</span>] <span style="color:#960050;background-color:#1e0010">65705192</span>.<span style="color:#960050;background-color:#1e0010">284</span>d6f <span style="color:#960050;background-color:#1e0010">1848030187</span> databse_name [unknown] pglogical apply <span style="color:#960050;background-color:#1e0010">16794</span><span style="color:#f92672">:</span><span style="color:#960050;background-color:#1e0010">4247138421</span> ERROR<span style="color:#f92672">:</span> null value in column <span style="color:#960050;background-color:#1e0010">&#34;</span>ccy<span style="color:#960050;background-color:#1e0010">&#34;</span> of <span style="color:#f92672">relation</span> <span style="color:#960050;background-color:#1e0010">&#34;</span>report<span style="color:#960050;background-color:#1e0010">&#34;</span> violates not<span style="color:#f92672">-</span>null constraint
</span></span></code></pre></div><p>If the replication status is <code>replicating</code>, you can investigate further by running the following query to get data per table for the synced database:</p>
<div class="highlight"><pre tabindex="0" style="color:#f8f8f2;background-color:#272822;-moz-tab-size:4;-o-tab-size:4;tab-size:4;-webkit-text-size-adjust:none;"><code class="language-sql" data-lang="sql"><span style="display:flex;"><span><span style="color:#66d9ef">SELECT</span> 
</span></span><span style="display:flex;"><span>	<span style="color:#66d9ef">CASE</span> sync_kind
</span></span><span style="display:flex;"><span>		<span style="color:#66d9ef">WHEN</span> <span style="color:#e6db74">&#39;i&#39;</span> <span style="color:#66d9ef">THEN</span> <span style="color:#e6db74">&#39;Init&#39;</span>
</span></span><span style="display:flex;"><span>		<span style="color:#66d9ef">WHEN</span> <span style="color:#e6db74">&#39;f&#39;</span> <span style="color:#66d9ef">THEN</span> <span style="color:#e6db74">&#39;Full&#39;</span>
</span></span><span style="display:flex;"><span>		<span style="color:#66d9ef">WHEN</span> <span style="color:#e6db74">&#39;s&#39;</span> <span style="color:#66d9ef">THEN</span> <span style="color:#e6db74">&#39;Structure&#39;</span>
</span></span><span style="display:flex;"><span>		<span style="color:#66d9ef">WHEN</span> <span style="color:#e6db74">&#39;d&#39;</span> <span style="color:#66d9ef">THEN</span> <span style="color:#e6db74">&#39;Data&#39;</span>
</span></span><span style="display:flex;"><span>		<span style="color:#66d9ef">ELSE</span> <span style="color:#e6db74">&#39;Unknown kind&#39;</span>
</span></span><span style="display:flex;"><span>    <span style="color:#66d9ef">END</span> <span style="color:#66d9ef">AS</span> sync_kind,
</span></span><span style="display:flex;"><span>	sync_subid,
</span></span><span style="display:flex;"><span>	sync_nspname <span style="color:#f92672">||</span> <span style="color:#e6db74">&#39;.&#39;</span> <span style="color:#f92672">||</span> sync_relname <span style="color:#66d9ef">as</span> relation,
</span></span><span style="display:flex;"><span>    <span style="color:#66d9ef">CASE</span> sync_status
</span></span><span style="display:flex;"><span>        <span style="color:#66d9ef">WHEN</span> <span style="color:#e6db74">&#39;\0&#39;</span> <span style="color:#66d9ef">THEN</span> <span style="color:#e6db74">&#39;No sync&#39;</span>
</span></span><span style="display:flex;"><span>        <span style="color:#66d9ef">WHEN</span> <span style="color:#e6db74">&#39;i&#39;</span> <span style="color:#66d9ef">THEN</span> <span style="color:#e6db74">&#39;Ask for sync&#39;</span>
</span></span><span style="display:flex;"><span>        <span style="color:#66d9ef">WHEN</span> <span style="color:#e6db74">&#39;s&#39;</span> <span style="color:#66d9ef">THEN</span> <span style="color:#e6db74">&#39;Sync structure&#39;</span>
</span></span><span style="display:flex;"><span>        <span style="color:#66d9ef">WHEN</span> <span style="color:#e6db74">&#39;d&#39;</span> <span style="color:#66d9ef">THEN</span> <span style="color:#e6db74">&#39;Data sync&#39;</span>
</span></span><span style="display:flex;"><span>        <span style="color:#66d9ef">WHEN</span> <span style="color:#e6db74">&#39;c&#39;</span> <span style="color:#66d9ef">THEN</span> <span style="color:#e6db74">&#39;Constraint sync (post-data structure)&#39;</span>
</span></span><span style="display:flex;"><span>        <span style="color:#66d9ef">WHEN</span> <span style="color:#e6db74">&#39;w&#39;</span> <span style="color:#66d9ef">THEN</span> <span style="color:#e6db74">&#39;Table sync is waiting to get OK from main thread&#39;</span>
</span></span><span style="display:flex;"><span>        <span style="color:#66d9ef">WHEN</span> <span style="color:#e6db74">&#39;u&#39;</span> <span style="color:#66d9ef">THEN</span> <span style="color:#e6db74">&#39;Catching up&#39;</span>
</span></span><span style="display:flex;"><span>        <span style="color:#66d9ef">WHEN</span> <span style="color:#e6db74">&#39;y&#39;</span> <span style="color:#66d9ef">THEN</span> <span style="color:#e6db74">&#39;Synchronization finished (at lsn)&#39;</span>
</span></span><span style="display:flex;"><span>        <span style="color:#66d9ef">WHEN</span> <span style="color:#e6db74">&#39;r&#39;</span> <span style="color:#66d9ef">THEN</span> <span style="color:#e6db74">&#39;Done&#39;</span>
</span></span><span style="display:flex;"><span>        <span style="color:#66d9ef">ELSE</span> <span style="color:#e6db74">&#39;Unknown status&#39;</span>
</span></span><span style="display:flex;"><span>    <span style="color:#66d9ef">END</span> <span style="color:#66d9ef">AS</span> sync_status,
</span></span><span style="display:flex;"><span>	sync_statuslsn
</span></span><span style="display:flex;"><span><span style="color:#66d9ef">FROM</span> 
</span></span><span style="display:flex;"><span>    pglogical.local_sync_status;
</span></span></code></pre></div><p>If you see <code>Done</code> or <code>Synchronization finished</code> for some tables but the data is missing, you may need to re-sync tables or restart the replication, you may read more in the <a href="https://github.com/2ndQuadrant/pglogical/issues/204">issue</a>. Before doing so, save the results from the <code>local_sync_status</code> table, as they will be lost after restarting the replication.</p>
<p>You can attempt to resync specific tables before recreating the entire replication:</p>
<div class="highlight"><pre tabindex="0" style="color:#f8f8f2;background-color:#272822;-moz-tab-size:4;-o-tab-size:4;tab-size:4;-webkit-text-size-adjust:none;"><code class="language-sql" data-lang="sql"><span style="display:flex;"><span><span style="color:#66d9ef">SELECT</span> pglogical.alter_subscription_resynchronize_table(
</span></span><span style="display:flex;"><span>  subscription_name :<span style="color:#f92672">=</span> <span style="color:#e6db74">&#39;subscription_name&#39;</span>,
</span></span><span style="display:flex;"><span>  relation :<span style="color:#f92672">=</span> <span style="color:#e6db74">&#39;public.table_name&#39;</span>
</span></span><span style="display:flex;"><span>);
</span></span></code></pre></div><p><strong>NB!:</strong> Remember it will truncate table first. Do it outside of normal operations hours!</p>
<p>If nothing helps, you can restart the replication using the following helper query to build a command with the same parameters as before. If your database is big enough (&gt;100GB), you can turn off init sync for the data.</p>
<div class="highlight"><pre tabindex="0" style="color:#f8f8f2;background-color:#272822;-moz-tab-size:4;-o-tab-size:4;tab-size:4;-webkit-text-size-adjust:none;"><code class="language-sql" data-lang="sql"><span style="display:flex;"><span><span style="color:#66d9ef">SELECT</span> <span style="color:#e6db74">&#39;SELECT pglogical.drop_subscription(&#39;&#39;&#39;</span> <span style="color:#f92672">||</span> sub_name <span style="color:#f92672">||</span> <span style="color:#e6db74">&#39;&#39;&#39;);&#39;</span> 
</span></span><span style="display:flex;"><span>	   <span style="color:#f92672">||</span> <span style="color:#e6db74">&#39;SELECT pglogical.create_subscription(&#39;</span>
</span></span><span style="display:flex;"><span>       <span style="color:#f92672">||</span> <span style="color:#e6db74">&#39;subscription_name := &#39;&#39;&#39;</span> <span style="color:#f92672">||</span> sub_name <span style="color:#f92672">||</span> <span style="color:#e6db74">&#39;&#39;&#39;, &#39;</span>
</span></span><span style="display:flex;"><span>       <span style="color:#f92672">||</span> <span style="color:#e6db74">&#39;provider_dsn := &#39;&#39;&#39;</span> <span style="color:#f92672">||</span> provider_dsn <span style="color:#f92672">||</span> <span style="color:#e6db74">&#39;&#39;&#39;, &#39;</span>
</span></span><span style="display:flex;"><span>	   <span style="color:#f92672">||</span> <span style="color:#e6db74">&#39;replication_sets := &#39;&#39;{&#39;</span> <span style="color:#f92672">||</span> array_to_string(replication_sets, <span style="color:#e6db74">&#39;,&#39;</span>) <span style="color:#f92672">||</span> <span style="color:#e6db74">&#39;}&#39;&#39;, &#39;</span>
</span></span><span style="display:flex;"><span>       <span style="color:#f92672">||</span> <span style="color:#e6db74">&#39;synchronize_data := false);&#39;</span> <span style="color:#66d9ef">AS</span> create_subscription_cmd
</span></span><span style="display:flex;"><span><span style="color:#66d9ef">FROM</span> (
</span></span><span style="display:flex;"><span>  <span style="color:#66d9ef">SELECT</span> subscription_name <span style="color:#66d9ef">as</span> sub_name,
</span></span><span style="display:flex;"><span>         provider_dsn,
</span></span><span style="display:flex;"><span>		 replication_sets
</span></span><span style="display:flex;"><span>  <span style="color:#66d9ef">FROM</span> pglogical.show_subscription_status()
</span></span><span style="display:flex;"><span>) t;
</span></span></code></pre></div><p>This query will template a query to restart the subscription in a database by recreating it. It will reduce the lag but will repoint the slot to the latest state, so the data lag won't be synced.</p>
<p>After restarting the subscription, the records from <code>local_sync_status</code> will be lost. To overcome this, you can request a full resync (truncate, copy, sync) per table using the data from the <code>local_sync_status</code> table:</p>
<div class="highlight"><pre tabindex="0" style="color:#f8f8f2;background-color:#272822;-moz-tab-size:4;-o-tab-size:4;tab-size:4;-webkit-text-size-adjust:none;"><code class="language-sql" data-lang="sql"><span style="display:flex;"><span><span style="color:#66d9ef">SELECT</span> pglogical.alter_subscription_resynchronize_table(
</span></span><span style="display:flex;"><span>  subscription_name :<span style="color:#f92672">=</span> <span style="color:#e6db74">&#39;subscription_name&#39;</span>,
</span></span><span style="display:flex;"><span>  relation :<span style="color:#f92672">=</span> <span style="color:#e6db74">&#39;public.table_name&#39;</span>
</span></span><span style="display:flex;"><span>);
</span></span></code></pre></div><p><strong>NB!:</strong> Remember it will truncate table first. Do it outside of normal operations hours!</p>
<p>After triggering resync you will see that table got added to the <code>local_sync_status</code> with a status <code>Data sync</code> or <code>Ask for sync</code>.</p>
]]></content:encoded>
    </item>
    
    <item>
      <title>Upgrading PostgreSQL from 14 to 15 using Patroni in Kubernetes</title>
      <link>https://kuzaxak.dev/posts/2023/08/28/pg-upgrade-15/</link>
      <pubDate>Mon, 28 Aug 2023 00:00:00 +0000</pubDate>
      
      <guid>https://kuzaxak.dev/posts/2023/08/28/pg-upgrade-15/</guid>
      <description>This article provides a detailed guide on how to upgrade PostgreSQL from version 
14 to 15 using Patroni in Kubernetes. It covers the necessary preparations, including 
enabling in-place upgrades and failsafe mode, starting the upgrade, and fixing broken 
pglogical replication. It also addresses a specific issue related to pglogical 
replication and provides a solution for it.
</description>
      <content:encoded><![CDATA[<p><p>
  <img src="https://kuzaxak.dev/posts/2023/08/28/pg-upgrade-15/art-cover.webp" alt="image"  />
</p>
</p>
<p>We recently undertook the task of upgrading PostgreSQL (PG) from version 14 to 15.
For this, we used Patroni to control the PG deployment in Kubernetes (k8s).</p>
<h2 id="enabling-in-place-upgrade">Enabling In-Place Upgrade</h2>
<p>Initially, to start a deployment, we needed to enable the in-place upgrade option in the
PostgreSQL operator:</p>
<div class="highlight"><pre tabindex="0" style="color:#f8f8f2;background-color:#272822;-moz-tab-size:4;-o-tab-size:4;tab-size:4;-webkit-text-size-adjust:none;"><code class="language-yaml" data-lang="yaml"><span style="display:flex;"><span><span style="color:#f92672">configMajorVersionUpgrade</span>:
</span></span><span style="display:flex;"><span>  <span style="color:#75715e"># operator will run the upgrade script after the manifest is updated and pods are rotated</span>
</span></span><span style="display:flex;"><span>  <span style="color:#f92672">major_version_upgrade_mode</span>: <span style="color:#ae81ff">manual</span>
</span></span></code></pre></div><h2 id="enabling-failsafe-mode">Enabling Failsafe Mode</h2>
<p>Next, we decided to enable the <a href="https://patroni.readthedocs.io/en/master/dcs_failsafe_mode.html">failsafe mode</a> to reduce the number of failovers in a cluster.</p>
<div class="highlight"><pre tabindex="0" style="color:#f8f8f2;background-color:#272822;-moz-tab-size:4;-o-tab-size:4;tab-size:4;-webkit-text-size-adjust:none;"><code class="language-yaml" data-lang="yaml"><span style="display:flex;"><span><span style="color:#f92672">configPatroni</span>:
</span></span><span style="display:flex;"><span>  <span style="color:#75715e"># enable Patroni DCS failsafe_mode feature</span>
</span></span><span style="display:flex;"><span>  <span style="color:#f92672">enable_patroni_failsafe_mode</span>: <span style="color:#66d9ef">true</span>
</span></span></code></pre></div><p>We experienced quite a lot of failovers due to temporary k8s API unavailability.</p>
<h2 id="starting-the-upgrade">Starting the Upgrade</h2>
<p>With all preparations done, we started the upgrade by updating the PostgresCluster manifests:</p>
<div class="highlight"><pre tabindex="0" style="color:#f8f8f2;background-color:#272822;-moz-tab-size:4;-o-tab-size:4;tab-size:4;-webkit-text-size-adjust:none;"><code class="language-yaml" data-lang="yaml"><span style="display:flex;"><span><span style="color:#f92672">apiVersion</span>: <span style="color:#ae81ff">acid.zalan.do/v1</span>
</span></span><span style="display:flex;"><span><span style="color:#f92672">kind</span>: <span style="color:#ae81ff">postgresql</span>
</span></span><span style="display:flex;"><span><span style="color:#f92672">spec</span>:
</span></span><span style="display:flex;"><span>  <span style="color:#f92672">allowedSourceRanges</span>: <span style="color:#66d9ef">null</span>
</span></span><span style="display:flex;"><span>  <span style="color:#f92672">postgresql</span>:
</span></span><span style="display:flex;"><span>    <span style="color:#f92672">version</span>: <span style="color:#ae81ff">14</span> -<span style="color:#ae81ff">&gt; 15</span>
</span></span></code></pre></div><p>The upgrade itself went smoothly, but in a production cluster, pglogical replication was broken.</p>
<h2 id="fixing-broken-pglogical-replication">Fixing Broken pglogical Replication</h2>
<p>In the logs, we found a suspicious record regarding a missing origin:</p>
<div class="highlight"><pre tabindex="0" style="color:#f8f8f2;background-color:#272822;-moz-tab-size:4;-o-tab-size:4;tab-size:4;-webkit-text-size-adjust:none;"><code class="language-fallback" data-lang="fallback"><span style="display:flex;"><span>ERROR:  replication origin &#34;pgl_market_data_prod_0c0d721_sit_m6d604ee&#34; does not exist
</span></span></code></pre></div><p>Fortunately, we were not the first ones to experience this problem. The pglogical repository
has an <a href="https://github.com/2ndQuadrant/pglogical/issues/285">issue</a> with exactly the same problem. There, we found a solution: we needed to use
the <code>pg_replication_origin_create</code> function to restore the missing origin.</p>
<p>However, the documentation and issue did not describe what an origin is and how you need to call
this function. It took me a while to figure out how we could restore them.</p>
<p>To do that, I created a new replication and checked existing origins. It turns out that the origin name
for pglogical is equal to the replication slot name.</p>
<div class="highlight"><pre tabindex="0" style="color:#f8f8f2;background-color:#272822;-moz-tab-size:4;-o-tab-size:4;tab-size:4;-webkit-text-size-adjust:none;"><code class="language-sql" data-lang="sql"><span style="display:flex;"><span><span style="color:#66d9ef">SELECT</span> <span style="color:#f92672">*</span> <span style="color:#66d9ef">FROM</span> pg_replication_origin;
</span></span></code></pre></div><p>The problem got fixed by creating origins for every replication slot of pglogical:</p>
<div class="highlight"><pre tabindex="0" style="color:#f8f8f2;background-color:#272822;-moz-tab-size:4;-o-tab-size:4;tab-size:4;-webkit-text-size-adjust:none;"><code class="language-sql" data-lang="sql"><span style="display:flex;"><span><span style="color:#66d9ef">SELECT</span> <span style="color:#f92672">*</span> <span style="color:#66d9ef">FROM</span> pg_stat_replication_slots;
</span></span><span style="display:flex;"><span><span style="color:#66d9ef">SELECT</span> pg_replication_origin_create(<span style="color:#e6db74">&#39;pgl_market_data_prod_0c0d721_sit_m6d604ee&#39;</span>);
</span></span></code></pre></div><p>After that, the replication got restored.</p>
]]></content:encoded>
    </item>
    
    <item>
      <title>Rapid WAL Growth in Postgres PG15 on Kubernetes</title>
      <link>https://kuzaxak.dev/posts/2023/08/17/pg-wal-pilling-up/</link>
      <pubDate>Thu, 17 Aug 2023 00:00:00 +0000</pubDate>
      
      <guid>https://kuzaxak.dev/posts/2023/08/17/pg-wal-pilling-up/</guid>
      <description>Dive into a deep tech exploration of dealing with rapidly growing Write-Ahead Logging (WAL) in 
PostgreSQL PG15 within a Kubernetes environment. Learn about the challenges, mistakes made, and 
the recovery process, complete with practical SQL queries and commands. 
Essential reading for database administrators and engineers.
</description>
      <content:encoded><![CDATA[<h2 id="intro">Intro</h2>
<p>Our setup consists of two clusters, each with two nodes interconnected through logical replication managed by the <a href="https://github.com/zalando/postgres-operator">Patroni PostgreSQL operator</a>.</p>
<div class="highlight"><pre tabindex="0" style="color:#f8f8f2;background-color:#272822;-moz-tab-size:4;-o-tab-size:4;tab-size:4;-webkit-text-size-adjust:none;"><code class="language-fallback" data-lang="fallback"><span style="display:flex;"><span> Prod                   
</span></span><span style="display:flex;"><span>+---------+    logical replication     +----------+
</span></span><span style="display:flex;"><span>| Primary | &lt;----------------------&gt;   | Replica  |
</span></span><span style="display:flex;"><span>+---------+   (managed by patroni)     +----------+
</span></span><span style="display:flex;"><span>    | |
</span></span><span style="display:flex;"><span>    | | logical replication (pglogical)
</span></span><span style="display:flex;"><span>    | |
</span></span><span style="display:flex;"><span>    v v
</span></span><span style="display:flex;"><span>+---------+    logical replication     +----------+
</span></span><span style="display:flex;"><span>| Primary | &lt;----------------------&gt;   | Replica  |
</span></span><span style="display:flex;"><span>+---------+   (managed by patroni)     +----------+
</span></span><span style="display:flex;"><span> SIT (Beta)
</span></span></code></pre></div><p>After upgrading to PostgreSQL 15, we received an alert about rapidly growing WAL files. The root cause was not immediately apparent, prompting an investigation.
What follows is an account of our endeavors to mitigate this issue and the pivotal
lessons we learned along the way.</p>
<h2 id="investigation-and-missteps">Investigation and Missteps</h2>
<p>In an attempt to resolve the issue, we came across a <a href="https://www.postgresql.fastware.com/blog/how-to-solve-the-problem-if-pg-wal-is-full">recommendation</a> to inspect the latest checkpoint file using the pg_controldata command.</p>
<div class="highlight"><pre tabindex="0" style="color:#f8f8f2;background-color:#272822;-moz-tab-size:4;-o-tab-size:4;tab-size:4;-webkit-text-size-adjust:none;"><code class="language-fallback" data-lang="fallback"><span style="display:flex;"><span>root@main-db-1:/home/postgres# pg_controldata -D /home/postgres/pgdata/pgroot/data
</span></span><span style="display:flex;"><span>...
</span></span><span style="display:flex;"><span>Latest checkpoint&#39;s REDO WAL file: 00000001000012A5000000E2
</span></span><span style="display:flex;"><span>...
</span></span></code></pre></div><p>Noticing that files in <code>data/pg_wal/archive_status/</code> had been renamed with <code>.done</code> extensions, we proceeded to manually remove them using the <code>pg_archivecleanup</code> command to reclaim space.</p>
<div class="highlight"><pre tabindex="0" style="color:#f8f8f2;background-color:#272822;-moz-tab-size:4;-o-tab-size:4;tab-size:4;-webkit-text-size-adjust:none;"><code class="language-fallback" data-lang="fallback"><span style="display:flex;"><span>pg_archivecleanup -d /home/postgres/pgdata/pgroot/data/pg_wal 00000001000012A5000000E0
</span></span><span style="display:flex;"><span>pg_archivecleanup: keeping WAL file &#34;/home/postgres/pgdata/pgroot/data/pg_wal/00000001000012A5000000E0&#34; and later
</span></span><span style="display:flex;"><span>pg_archivecleanup: removing file &#34;/home/postgres/pgdata/pgroot/data/pg_wal/00000001000012A4000000FE&#34;
</span></span><span style="display:flex;"><span>pg_archivecleanup: removing file &#34;/home/postgres/pgdata/pgroot/data/pg_wal/00000001000012A20000007D&#34;
</span></span><span style="display:flex;"><span>...
</span></span></code></pre></div><p>However, this action inadvertently disrupted the replication on the secondary node. To verify the replication status, we ran the following query on the receiver side (beta):</p>
<div class="highlight"><pre tabindex="0" style="color:#f8f8f2;background-color:#272822;-moz-tab-size:4;-o-tab-size:4;tab-size:4;-webkit-text-size-adjust:none;"><code class="language-sql" data-lang="sql"><span style="display:flex;"><span><span style="color:#960050;background-color:#1e0010">\</span><span style="color:#66d9ef">c</span> database_name
</span></span><span style="display:flex;"><span><span style="color:#66d9ef">select</span> subscription_name, status <span style="color:#66d9ef">FROM</span> pglogical.show_subscription_status();
</span></span></code></pre></div><p>The output confirmed that the replication was down:</p>
<div class="highlight"><pre tabindex="0" style="color:#f8f8f2;background-color:#272822;-moz-tab-size:4;-o-tab-size:4;tab-size:4;-webkit-text-size-adjust:none;"><code class="language-fallback" data-lang="fallback"><span style="display:flex;"><span>    subscription_name    | status
</span></span><span style="display:flex;"><span>-------------------------+--------
</span></span><span style="display:flex;"><span> sit_main_db             | down
</span></span><span style="display:flex;"><span>(1 row)
</span></span></code></pre></div><p>To gain further insights, we used another <a href="https://stackoverflow.com/questions/49539938/postgres-wal-file-not-getting-deleted">query</a> to check the replication slots and their associated WAL files on the sender side (prod):</p>
<div class="highlight"><pre tabindex="0" style="color:#f8f8f2;background-color:#272822;-moz-tab-size:4;-o-tab-size:4;tab-size:4;-webkit-text-size-adjust:none;"><code class="language-sql" data-lang="sql"><span style="display:flex;"><span><span style="color:#66d9ef">SELECT</span> slot_name,
</span></span><span style="display:flex;"><span>       lpad((pg_control_checkpoint()).timeline_id::text, <span style="color:#ae81ff">8</span>, <span style="color:#e6db74">&#39;0&#39;</span>) <span style="color:#f92672">||</span>
</span></span><span style="display:flex;"><span>       lpad(split_part(restart_lsn::text, <span style="color:#e6db74">&#39;/&#39;</span>, <span style="color:#ae81ff">1</span>), <span style="color:#ae81ff">8</span>, <span style="color:#e6db74">&#39;0&#39;</span>) <span style="color:#f92672">||</span>
</span></span><span style="display:flex;"><span>       lpad(substr(split_part(restart_lsn::text, <span style="color:#e6db74">&#39;/&#39;</span>, <span style="color:#ae81ff">2</span>), <span style="color:#ae81ff">1</span>, <span style="color:#ae81ff">2</span>), <span style="color:#ae81ff">8</span>, <span style="color:#e6db74">&#39;0&#39;</span>)
</span></span><span style="display:flex;"><span>       <span style="color:#66d9ef">AS</span> wal_file
</span></span><span style="display:flex;"><span><span style="color:#66d9ef">FROM</span> pg_replication_slots;
</span></span></code></pre></div><p>The results highlighted the WAL files that were inadvertently deleted:</p>
<div class="highlight"><pre tabindex="0" style="color:#f8f8f2;background-color:#272822;-moz-tab-size:4;-o-tab-size:4;tab-size:4;-webkit-text-size-adjust:none;"><code class="language-fallback" data-lang="fallback"><span style="display:flex;"><span> main_db_0                               | 00000001000012A5000000E5
</span></span><span style="display:flex;"><span> pgl_bonds_prod_a869e54_sit_mfd26307     | 00000001000012A10000007C
</span></span><span style="display:flex;"><span> pgl_import_prod_f3ab9f4_sit_mc9d8484    | 00000001000012A10000007C
</span></span><span style="display:flex;"><span> pgl_prod_828fc61_sit_meced8c6           | 00000001000012A10000007C
</span></span><span style="display:flex;"><span> pgl_data_prod_0c0d721_sit_m6d604ee      | 00000001000012A10000007C
</span></span><span style="display:flex;"><span> pgl_sentiment_prod_3b3f93a_sit_m5397d6e | 00000001000012A10000007C
</span></span></code></pre></div><h2 id="recovery-process">Recovery Process</h2>
<p>Fortunately, we had WAL files stored in S3 using wal-g for point-in-time recovery. Instead of manually restoring with wal-g, we decided to initialize a fresh instance in the Patroni cluster:</p>
<div class="highlight"><pre tabindex="0" style="color:#f8f8f2;background-color:#272822;-moz-tab-size:4;-o-tab-size:4;tab-size:4;-webkit-text-size-adjust:none;"><code class="language-shell" data-lang="shell"><span style="display:flex;"><span>patronictl reinit main-db main-db-0
</span></span><span style="display:flex;"><span>...
</span></span></code></pre></div><p>After restoring the WAL to its original state, we initiated a failover:</p>
<div class="highlight"><pre tabindex="0" style="color:#f8f8f2;background-color:#272822;-moz-tab-size:4;-o-tab-size:4;tab-size:4;-webkit-text-size-adjust:none;"><code class="language-shell" data-lang="shell"><span style="display:flex;"><span>patronictl failover
</span></span><span style="display:flex;"><span>...
</span></span></code></pre></div><p>We then restarted the replication and used the following query on the source database to monitor the replication lag for our most data-intensive database:</p>
<div class="highlight"><pre tabindex="0" style="color:#f8f8f2;background-color:#272822;-moz-tab-size:4;-o-tab-size:4;tab-size:4;-webkit-text-size-adjust:none;"><code class="language-sql" data-lang="sql"><span style="display:flex;"><span><span style="color:#66d9ef">select</span>   pid, client_addr, application_name, <span style="color:#66d9ef">state</span>, sync_state,
</span></span><span style="display:flex;"><span>         pg_wal_lsn_diff(pg_current_wal_lsn(),sent_lsn) <span style="color:#66d9ef">AS</span> sent_lag,
</span></span><span style="display:flex;"><span>         pg_wal_lsn_diff(sent_lsn,flush_lsn) <span style="color:#66d9ef">AS</span> receiving_lag,
</span></span><span style="display:flex;"><span>         pg_wal_lsn_diff(flush_lsn,replay_lsn) <span style="color:#66d9ef">AS</span> replay_lag,
</span></span><span style="display:flex;"><span>         pg_wal_lsn_diff(pg_current_wal_lsn(),replay_lsn) <span style="color:#66d9ef">AS</span> total_lag,
</span></span><span style="display:flex;"><span>         now()<span style="color:#f92672">-</span>reply_time <span style="color:#66d9ef">AS</span> reply_delay
</span></span><span style="display:flex;"><span><span style="color:#66d9ef">from</span> pg_stat_replication;
</span></span></code></pre></div><p>The output showed the replication lag for each subscriber:</p>
<div class="highlight"><pre tabindex="0" style="color:#f8f8f2;background-color:#272822;-moz-tab-size:4;-o-tab-size:4;tab-size:4;-webkit-text-size-adjust:none;"><code class="language-fallback" data-lang="fallback"><span style="display:flex;"><span>  pid  | client_addr |        application_name         |   state   | sync_state |  write_lag  |  flush_lag  | replay_lag
</span></span><span style="display:flex;"><span>-------+-------------+---------------------------------+-----------+------------+-------------+-------------+-------------
</span></span><span style="display:flex;"><span> 75525 | 10.23.8.19  | sit_main_db                     | catchup   | async      | -9811896992 | -9811896992 | -9811896992
</span></span><span style="display:flex;"><span> 65825 | 10.23.8.24  | sit_main_db_import              | streaming | async      |           0 |           0 |           0
</span></span><span style="display:flex;"><span> 65827 | 10.23.8.57  | sit_main_db_market_sentiment    | streaming | async      |           0 |           0 |           0
</span></span><span style="display:flex;"><span> 65828 | 10.23.8.40  | sit_main_db_market_data         | streaming | async      |           0 |           0 |           0
</span></span><span style="display:flex;"><span> 65826 | 10.23.8.57  | sit_main_db_bonds               | streaming | async      |           0 |           0 |           0
</span></span><span style="display:flex;"><span> 79684 | 10.24.134.7 | main-db-1                       | streaming | async      |           0 |           0 |           0
</span></span></code></pre></div><p>Eventually, the replication caught up, and WAL files were properly managed and removed from the disk.</p>
<div class="highlight"><pre tabindex="0" style="color:#f8f8f2;background-color:#272822;-moz-tab-size:4;-o-tab-size:4;tab-size:4;-webkit-text-size-adjust:none;"><code class="language-fallback" data-lang="fallback"><span style="display:flex;"><span>postgres=# SELECT slot_name,
</span></span><span style="display:flex;"><span>       lpad((pg_control_checkpoint()).timeline_id::text, 8, &#39;0&#39;) ||
</span></span><span style="display:flex;"><span>       lpad(split_part(restart_lsn::text, &#39;/&#39;, 1), 8, &#39;0&#39;) ||
</span></span><span style="display:flex;"><span>       lpad(substr(split_part(restart_lsn::text, &#39;/&#39;, 2), 1, 2), 8, &#39;0&#39;)
</span></span><span style="display:flex;"><span>       AS wal_file
</span></span><span style="display:flex;"><span>FROM pg_replication_slots;
</span></span><span style="display:flex;"><span>                      slot_name                |         wal_file
</span></span><span style="display:flex;"><span>-----------------------------------------------+-------------------------
</span></span><span style="display:flex;"><span> main_db_1                                     | 00000002000012A6000000BE
</span></span><span style="display:flex;"><span> pgl_bonds_prod_a869e54_sit_mfd26307           | 00000002000012A6000000BE
</span></span><span style="display:flex;"><span> pgl_import_prod_f3ab9f4_sit_mc9d8484          | 00000002000012A6000000BE
</span></span><span style="display:flex;"><span> pgl_prod_828fc61_sit_meced8c6                 | 00000002000012A6000000BE
</span></span><span style="display:flex;"><span> pgl_data_prod_0c0d721_sit_m6d604ee            | 00000002000012A6000000BE
</span></span><span style="display:flex;"><span> pgl_sentiment_prod_3b3f93a_sit_m5397d6e       | 00000002000012A6000000BE
</span></span></code></pre></div><h2 id="additional-tips-for-debugging-pglogical-replication">Additional Tips for Debugging pglogical Replication</h2>
<ol>
<li>Monitor the replication delay on the provider side:</li>
</ol>
<div class="highlight"><pre tabindex="0" style="color:#f8f8f2;background-color:#272822;-moz-tab-size:4;-o-tab-size:4;tab-size:4;-webkit-text-size-adjust:none;"><code class="language-sql" data-lang="sql"><span style="display:flex;"><span><span style="color:#66d9ef">SELECT</span> slot_name,
</span></span><span style="display:flex;"><span>       lpad((pg_control_checkpoint()).timeline_id::text, <span style="color:#ae81ff">8</span>, <span style="color:#e6db74">&#39;0&#39;</span>) <span style="color:#f92672">||</span>
</span></span><span style="display:flex;"><span>       lpad(split_part(restart_lsn::text, <span style="color:#e6db74">&#39;/&#39;</span>, <span style="color:#ae81ff">1</span>), <span style="color:#ae81ff">8</span>, <span style="color:#e6db74">&#39;0&#39;</span>) <span style="color:#f92672">||</span>
</span></span><span style="display:flex;"><span>       lpad(substr(split_part(restart_lsn::text, <span style="color:#e6db74">&#39;/&#39;</span>, <span style="color:#ae81ff">2</span>), <span style="color:#ae81ff">1</span>, <span style="color:#ae81ff">2</span>), <span style="color:#ae81ff">8</span>, <span style="color:#e6db74">&#39;0&#39;</span>)
</span></span><span style="display:flex;"><span>       <span style="color:#66d9ef">AS</span> wal_file,
</span></span><span style="display:flex;"><span>	   pg_size_pretty(pg_wal_lsn_diff(pg_current_wal_lsn(), restart_lsn)) <span style="color:#66d9ef">AS</span> replication_delay
</span></span><span style="display:flex;"><span><span style="color:#66d9ef">FROM</span> pg_replication_slots;
</span></span></code></pre></div><ol start="2">
<li>Check per table status on receiver side:</li>
</ol>
<div class="highlight"><pre tabindex="0" style="color:#f8f8f2;background-color:#272822;-moz-tab-size:4;-o-tab-size:4;tab-size:4;-webkit-text-size-adjust:none;"><code class="language-sql" data-lang="sql"><span style="display:flex;"><span><span style="color:#66d9ef">SELECT</span> <span style="color:#f92672">*</span> <span style="color:#66d9ef">FROM</span> pglogical.local_sync_status;
</span></span></code></pre></div><h2 id="takeaways">Takeaways</h2>
<ul>
<li><strong>Root Cause Analysis</strong>: Before making any amendments, always attempt to identify the root cause of the problem.</li>
<li><strong>Preemptive Backups</strong>: Prior to deleting any files, especially critical ones, ensure you have a backup in place.</li>
<li><strong>Educate Yourself</strong>: Dive deeper into how systems like WAL function. One article is seldom enough; broadening your knowledge base can help in averting potential crises.</li>
</ul>
]]></content:encoded>
    </item>
    
    <item>
      <title>Enabling IRSA for ServiceAccounts in Kops</title>
      <link>https://kuzaxak.dev/posts/2023/04/24/kops-irsa/</link>
      <pubDate>Mon, 24 Apr 2023 00:00:00 +0000</pubDate>
      
      <guid>https://kuzaxak.dev/posts/2023/04/24/kops-irsa/</guid>
      <description>Learn from my experience on how to enable IRSA (IAM Roles for ServiceAccounts)
support in Kops managed clusters, and leverage AWS authentication and authorization
for your applications.
</description>
      <content:encoded><![CDATA[<p><p>
  <img src="https://kuzaxak.dev/posts/2023/04/24/kops-irsa/art-cover.webp" alt="image"  />
</p>
</p>
<p>When I first started looking into adding IRSA (IAM Roles for ServiceAccounts) support to my
Kops managed Kubernetes clusters, I found <a href="https://kops.sigs.k8s.io/cluster_spec/#service-account-issuer-discovery-and-aws-iam-roles-for-service-accounts-irsa">the documentation</a> to be extremely confusing.
I wanted to share my journey and the steps I took to successfully enable IRSA support,
configure my cluster, and test the setup.</p>
<p>I decided to do that because we faced the <a href="https://github.com/uswitch/kiam/issues/398">same issue</a> as described in Kiam repo
with random <code>502</code> errors for <code>/latest/api/token</code> endpoint which broke our pipeline.</p>
<h3 id="enable-oidc-and-face-the-dots-dilemma">Enable OIDC and face the dots dilemma</h3>
<p>The first step I took was to configure the <code>serviceAccountIssuerDiscovery</code> section in
Kops cluster configuration.</p>
<p>Dots in S3 bucket names are widely used in our terraform modules, to add Java style prefixes
like <code>ee.hdo.beta.</code>. I wanted to use the same approach for OIDC public bucket, which was
a bit puzzling at first.
The reason is that IRSA relies on the OpenID Connect (OIDC) discovery endpoint, and dots
in the hostname can cause potential issues with SSL certificate validation.</p>
<div class="highlight"><pre tabindex="0" style="color:#f8f8f2;background-color:#272822;-moz-tab-size:4;-o-tab-size:4;tab-size:4;-webkit-text-size-adjust:none;"><code class="language-yaml" data-lang="yaml"><span style="display:flex;"><span>  <span style="color:#f92672">serviceAccountIssuerDiscovery</span>:
</span></span><span style="display:flex;"><span>    <span style="color:#f92672">discoveryStore</span>: <span style="color:#ae81ff">s3://ee.hdo.beta.irsa.providers</span>
</span></span><span style="display:flex;"><span>    <span style="color:#f92672">enableAWSOIDCProvider</span>: <span style="color:#66d9ef">true</span>
</span></span></code></pre></div><p>error was displayed in yaml file header:</p>
<div class="highlight"><pre tabindex="0" style="color:#f8f8f2;background-color:#272822;-moz-tab-size:4;-o-tab-size:4;tab-size:4;-webkit-text-size-adjust:none;"><code class="language-yaml" data-lang="yaml"><span style="display:flex;"><span><span style="color:#75715e"># error populating cluster spec: spec.serviceAccountIssuerDiscovery.serviceAccountIssuerDiscovery.discoveryStore: Invalid value: &#34;s3://ee.hdo.beta.irsa.providers&#34;: Bucket name cannot contain dots</span>
</span></span></code></pre></div><p>I ended up with the next configuration:</p>
<div class="highlight"><pre tabindex="0" style="color:#f8f8f2;background-color:#272822;-moz-tab-size:4;-o-tab-size:4;tab-size:4;-webkit-text-size-adjust:none;"><code class="language-yaml" data-lang="yaml"><span style="display:flex;"><span>  <span style="color:#f92672">serviceAccountIssuerDiscovery</span>:
</span></span><span style="display:flex;"><span>    <span style="color:#f92672">discoveryStore</span>: <span style="color:#ae81ff">s3://hdo-beta-irsa-providers</span>
</span></span><span style="display:flex;"><span>    <span style="color:#f92672">enableAWSOIDCProvider</span>: <span style="color:#66d9ef">true</span>
</span></span></code></pre></div><p>It is important to note that the bucket shouldn't block public access.</p>
<p>The <code>serviceAccountIssuerDiscovery</code> configuration enabled a trust relation between AWS IAM
and the certificate issued by Kops.
For that, Kops required having a publicly available store.
The previously issued keys used by Kops were replaced with a new one.
The public part of it was stored in S3.</p>
<p>This way tokens that are stored in SA secrets can be validated by external agent, AWS in our case
using the asymmetrical JWT key pair.</p>
<p>Keeping in mind that enabling <code>serviceAccountIssuerDiscovery</code> in runtime will cause all SA to regenerate
<a href="https://kops.sigs.k8s.io/cluster_spec/#service-account-issuer-discovery-and-aws-iam-roles-for-service-accounts-irsa">their secrets</a> I handled it after rolling cluster update.</p>
<blockquote>
<p>Warning: Enabling the following configuration on an existing cluster can be disruptive due to the
control plane provisioning tokens with different issuers. The symptom is that Pods are unable to
authenticate to the Kubernetes API. To resolve this, delete Service Account token secrets that exists
in the cluster and kill all pods unable to authenticate.</p>
</blockquote>
<div class="highlight"><pre tabindex="0" style="color:#f8f8f2;background-color:#272822;-moz-tab-size:4;-o-tab-size:4;tab-size:4;-webkit-text-size-adjust:none;"><code class="language-yaml" data-lang="yaml"><span style="display:flex;"><span>      -     <span style="color:#f92672">serviceAccountIssuer</span>: <span style="color:#ae81ff">https://api.internal.k8s.beta.hdo.ee</span>
</span></span><span style="display:flex;"><span>      -     <span style="color:#f92672">serviceAccountJWKSURI</span>: <span style="color:#ae81ff">https://api.internal.k8s.beta.hdo.ee/openid/v1/jwks</span>
</span></span><span style="display:flex;"><span>      <span style="color:#f92672">+     serviceAccountIssuer</span>: <span style="color:#ae81ff">https://hdo-beta-irsa-providers.s3.eu-west-1.amazonaws.com</span>
</span></span><span style="display:flex;"><span>      <span style="color:#f92672">+     serviceAccountJWKSURI</span>: <span style="color:#ae81ff">https://hdo-beta-irsa-providers.s3.eu-west-1.amazonaws.com/openid/v1/jwks</span>
</span></span></code></pre></div><h3 id="dealing-with-cert-manager">Dealing with cert-manager</h3>
<p>Next, I needed to enable <code>certManager</code> in my Kops cluster configuration. However, I was already
managing <code>cert-manager</code> using Helm, so I set the <code>managed</code> field to <code>false</code>.</p>
<div class="highlight"><pre tabindex="0" style="color:#f8f8f2;background-color:#272822;-moz-tab-size:4;-o-tab-size:4;tab-size:4;-webkit-text-size-adjust:none;"><code class="language-yaml" data-lang="yaml"><span style="display:flex;"><span>  <span style="color:#f92672">serviceAccountIssuerDiscovery</span>:
</span></span><span style="display:flex;"><span>    <span style="color:#f92672">discoveryStore</span>: <span style="color:#ae81ff">s3://hdo-beta-irsa-providers</span>
</span></span><span style="display:flex;"><span>    <span style="color:#f92672">enableAWSOIDCProvider</span>: <span style="color:#66d9ef">true</span>
</span></span><span style="display:flex;"><span>  <span style="color:#f92672">certManager</span>:
</span></span><span style="display:flex;"><span>    <span style="color:#f92672">enabled</span>: <span style="color:#66d9ef">true</span>
</span></span><span style="display:flex;"><span>    <span style="color:#f92672">managed</span>: <span style="color:#66d9ef">false</span>
</span></span></code></pre></div><h3 id="the-podidentitywebhook-magic">The <code>podIdentityWebhook</code> magic</h3>
<p>I then enabled the <code>podIdentityWebhook</code> in my Kops cluster configuration.
That service registers a webhook to <a href="https://github.com/aws/amazon-eks-pod-identity-webhook">mutate</a> pods, by adding environment variables necessary
for assuming IAM roles.</p>
<div class="highlight"><pre tabindex="0" style="color:#f8f8f2;background-color:#272822;-moz-tab-size:4;-o-tab-size:4;tab-size:4;-webkit-text-size-adjust:none;"><code class="language-yaml" data-lang="yaml"><span style="display:flex;"><span>  <span style="color:#f92672">serviceAccountIssuerDiscovery</span>:
</span></span><span style="display:flex;"><span>    <span style="color:#f92672">discoveryStore</span>: <span style="color:#ae81ff">s3://hdo-beta-irsa-providers</span>
</span></span><span style="display:flex;"><span>    <span style="color:#f92672">enableAWSOIDCProvider</span>: <span style="color:#66d9ef">true</span>
</span></span><span style="display:flex;"><span>  <span style="color:#f92672">certManager</span>:
</span></span><span style="display:flex;"><span>    <span style="color:#f92672">enabled</span>: <span style="color:#66d9ef">true</span>
</span></span><span style="display:flex;"><span>    <span style="color:#f92672">managed</span>: <span style="color:#66d9ef">false</span>
</span></span><span style="display:flex;"><span>  <span style="color:#f92672">podIdentityWebhook</span>:
</span></span><span style="display:flex;"><span>    <span style="color:#f92672">enabled</span>: <span style="color:#66d9ef">true</span>
</span></span></code></pre></div><h3 id="enabling-irsa-for-managed-addons">Enabling IRSA for managed addons</h3>
<p>By default, kops adds additional permissions to the node roles; instead it can use
dedicated roles attached to SA.</p>
<div class="highlight"><pre tabindex="0" style="color:#f8f8f2;background-color:#272822;-moz-tab-size:4;-o-tab-size:4;tab-size:4;-webkit-text-size-adjust:none;"><code class="language-yaml" data-lang="yaml"><span style="display:flex;"><span>  <span style="color:#f92672">iam</span>:
</span></span><span style="display:flex;"><span>    <span style="color:#f92672">useServiceAccountExternalPermissions</span>: <span style="color:#66d9ef">true</span>
</span></span><span style="display:flex;"><span>  <span style="color:#f92672">serviceAccountIssuerDiscovery</span>:
</span></span><span style="display:flex;"><span>    <span style="color:#f92672">discoveryStore</span>: <span style="color:#ae81ff">s3://hdo-beta-irsa-providers</span>
</span></span><span style="display:flex;"><span>    <span style="color:#f92672">enableAWSOIDCProvider</span>: <span style="color:#66d9ef">true</span>
</span></span><span style="display:flex;"><span>  <span style="color:#f92672">certManager</span>:
</span></span><span style="display:flex;"><span>    <span style="color:#f92672">enabled</span>: <span style="color:#66d9ef">true</span>
</span></span><span style="display:flex;"><span>    <span style="color:#f92672">managed</span>: <span style="color:#66d9ef">false</span>
</span></span><span style="display:flex;"><span>  <span style="color:#f92672">podIdentityWebhook</span>:
</span></span><span style="display:flex;"><span>    <span style="color:#f92672">enabled</span>: <span style="color:#66d9ef">true</span>
</span></span></code></pre></div><h3 id="updating-the-cluster-and-applying-changes">Updating the cluster and applying changes</h3>
<p>With my Kops configuration complete, I updated the terraform code:</p>
<div class="highlight"><pre tabindex="0" style="color:#f8f8f2;background-color:#272822;-moz-tab-size:4;-o-tab-size:4;tab-size:4;-webkit-text-size-adjust:none;"><code class="language-bash" data-lang="bash"><span style="display:flex;"><span>kops update cluster --target<span style="color:#f92672">=</span>terraform --out<span style="color:#f92672">=</span>update --yes
</span></span></code></pre></div><p>I carefully checked the diff between the update folder and my Terraform
content before applying the changes using <code>terraform apply</code>.</p>
<h3 id="rotating-control-plane-nodes">Rotating control plane nodes</h3>
<p>After applying the Terraform state to the target cluster, I rotated the
control plane nodes by draining them. This was crucial because enabling
<code>serviceAccountIssuerDiscovery</code> caused all <code>ServiceAccounts</code> to regenerate their tokens.</p>
<p>Since the SA is invalid, <code>aws-iam-authenticator</code> isn't working anymore.
This is because the tokens generated by the new service account issuer aren't compatible
with the previous <code>aws-iam-authenticator</code> service account token and as a result pod can't
issue new auth token to K8s API.</p>
<p>I needed to issue static credentials to access the cluster:</p>
<div class="highlight"><pre tabindex="0" style="color:#f8f8f2;background-color:#272822;-moz-tab-size:4;-o-tab-size:4;tab-size:4;-webkit-text-size-adjust:none;"><code class="language-bash" data-lang="bash"><span style="display:flex;"><span>kops export kubeconfig --auth-plugin --admin<span style="color:#f92672">=</span>5h
</span></span></code></pre></div><p>After that, I needed to delete all Secrets with a type <code>service-account</code> and then delete pods.</p>
<div class="highlight"><pre tabindex="0" style="color:#f8f8f2;background-color:#272822;-moz-tab-size:4;-o-tab-size:4;tab-size:4;-webkit-text-size-adjust:none;"><code class="language-bash" data-lang="bash"><span style="display:flex;"><span>kubectl get secrets -A -o json <span style="color:#ae81ff">\
</span></span></span><span style="display:flex;"><span>| jq <span style="color:#e6db74">&#39;.items[] | select(.type==&#34;kubernetes.io/service-account-token&#34;) | &#34;kubectl delete secret \(.metadata.name) -n \(.metadata.namespace)&#34;&#39;</span> <span style="color:#ae81ff">\
</span></span></span><span style="display:flex;"><span>| xargs -n <span style="color:#ae81ff">1</span> bash -c
</span></span></code></pre></div><p>Next, I deleted all the pods to force their recreation with the new service account tokens:</p>
<div class="highlight"><pre tabindex="0" style="color:#f8f8f2;background-color:#272822;-moz-tab-size:4;-o-tab-size:4;tab-size:4;-webkit-text-size-adjust:none;"><code class="language-bash" data-lang="bash"><span style="display:flex;"><span>kubectl get pods --all-namespaces <span style="color:#ae81ff">\
</span></span></span><span style="display:flex;"><span>| grep -v <span style="color:#e6db74">&#39;NAMESPACE&#39;</span> <span style="color:#ae81ff">\
</span></span></span><span style="display:flex;"><span>| awk <span style="color:#e6db74">&#39;{print $1&#34; &#34;$2}&#39;</span> <span style="color:#ae81ff">\
</span></span></span><span style="display:flex;"><span>| xargs -n <span style="color:#ae81ff">2</span> bash -c <span style="color:#e6db74">&#39;kubectl delete pod -n $0 $1&#39;</span>
</span></span></code></pre></div><p>You may delete only selected pods if deleting all of them is too disruptive for your cluster.</p>
<p>After running these commands, my cluster began recreating the pods with the new service account tokens,
and they were able to authenticate using IRSA.</p>
<h3 id="configuring-iam-trust">Configuring IAM trust</h3>
<p>All activities with the cluster were done, and I started adopting IAM roles to trust the new OIDC provider.</p>
<p>To make sure that roles trust created OIDC, I added additional trust policy to it:</p>
<div class="highlight"><pre tabindex="0" style="color:#f8f8f2;background-color:#272822;-moz-tab-size:4;-o-tab-size:4;tab-size:4;-webkit-text-size-adjust:none;"><code class="language-terraform" data-lang="terraform"><span style="display:flex;"><span>  <span style="color:#a6e22e">statement</span> {
</span></span><span style="display:flex;"><span>    <span style="color:#a6e22e">actions</span> <span style="color:#f92672">=</span> [<span style="color:#e6db74">&#34;sts:AssumeRoleWithWebIdentity&#34;</span>]
</span></span><span style="display:flex;"><span>    <span style="color:#a6e22e">principals</span> {
</span></span><span style="display:flex;"><span>      <span style="color:#a6e22e">type</span>        <span style="color:#f92672">=</span> <span style="color:#e6db74">&#34;Federated&#34;</span>
</span></span><span style="display:flex;"><span>      <span style="color:#a6e22e">identifiers</span> <span style="color:#f92672">=</span> [var.<span style="color:#a6e22e">oidc_provider_arn</span>]
</span></span><span style="display:flex;"><span>    }
</span></span><span style="display:flex;"><span>    <span style="color:#a6e22e">condition</span> {
</span></span><span style="display:flex;"><span>      <span style="color:#a6e22e">variable</span> <span style="color:#f92672">=</span> <span style="color:#e6db74">&#34;</span><span style="color:#e6db74">${</span>replace(var.<span style="color:#a6e22e">oidc_provider_arn</span>, <span style="color:#e6db74">&#34;/^(.*provider/)/&#34;</span>, <span style="color:#e6db74">&#34;&#34;</span>)<span style="color:#e6db74">}</span><span style="color:#e6db74">:sub&#34;</span>
</span></span><span style="display:flex;"><span>      <span style="color:#a6e22e">values</span>   <span style="color:#f92672">=</span> [<span style="color:#66d9ef">for</span> <span style="color:#a6e22e">sa</span> <span style="color:#66d9ef">in</span> module.<span style="color:#a6e22e">configuration</span>.<span style="color:#a6e22e">trusted_sa</span> <span style="color:#f92672">:</span> <span style="color:#e6db74">&#34;system:serviceaccount:</span><span style="color:#e6db74">${</span><span style="color:#a6e22e">sa</span><span style="color:#e6db74">}</span><span style="color:#e6db74">&#34;</span>]
</span></span><span style="display:flex;"><span>      <span style="color:#a6e22e">test</span>     <span style="color:#f92672">=</span> <span style="color:#e6db74">&#34;StringEquals&#34;</span>
</span></span><span style="display:flex;"><span>    }<span style="color:#75715e">
</span></span></span><span style="display:flex;"><span><span style="color:#75715e">    # :aud is not added because it works only with EKS
</span></span></span><span style="display:flex;"><span>  }
</span></span></code></pre></div><p><code>trusted_sa</code> is an array of strings with concatenated <code>namespace:sa_name</code>.</p>
<details>
  <summary>Full code of the Terraform role creation</summary>
  <div class="highlight"><pre tabindex="0" style="color:#f8f8f2;background-color:#272822;-moz-tab-size:4;-o-tab-size:4;tab-size:4;-webkit-text-size-adjust:none;"><code class="language-terraform" data-lang="terraform"><span style="display:flex;"><span><span style="color:#66d9ef">resource</span> <span style="color:#e6db74">&#34;aws_iam_role&#34;</span> <span style="color:#e6db74">&#34;this&#34;</span> {
</span></span><span style="display:flex;"><span>  <span style="color:#a6e22e">name</span>               <span style="color:#f92672">=</span> var.<span style="color:#a6e22e">name</span>
</span></span><span style="display:flex;"><span>  <span style="color:#a6e22e">assume_role_policy</span> <span style="color:#f92672">=</span> data.<span style="color:#a6e22e">aws_iam_policy_document</span>.<span style="color:#a6e22e">assume_role_policy</span>.<span style="color:#a6e22e">json</span>
</span></span><span style="display:flex;"><span>
</span></span><span style="display:flex;"><span>  <span style="color:#a6e22e">tags</span> <span style="color:#f92672">=</span> {
</span></span><span style="display:flex;"><span>    <span style="color:#a6e22e">Terraform</span> <span style="color:#f92672">=</span> <span style="color:#66d9ef">true</span>
</span></span><span style="display:flex;"><span>  }
</span></span><span style="display:flex;"><span>}
</span></span><span style="display:flex;"><span>
</span></span><span style="display:flex;"><span><span style="color:#66d9ef">data</span> <span style="color:#e6db74">&#34;aws_iam_policy_document&#34;</span> <span style="color:#e6db74">&#34;assume_role_policy&#34;</span> {<span style="color:#75715e">
</span></span></span><span style="display:flex;"><span><span style="color:#75715e">  # Preserve legacy trust policy for the migration period
</span></span></span><span style="display:flex;"><span>  <span style="color:#a6e22e">statement</span> {
</span></span><span style="display:flex;"><span>    <span style="color:#a6e22e">actions</span> <span style="color:#f92672">=</span> [<span style="color:#e6db74">&#34;sts:AssumeRole&#34;</span>]
</span></span><span style="display:flex;"><span>
</span></span><span style="display:flex;"><span>    <span style="color:#a6e22e">dynamic</span> <span style="color:#e6db74">&#34;principals&#34;</span> {
</span></span><span style="display:flex;"><span>      <span style="color:#a6e22e">for_each</span> <span style="color:#f92672">=</span> module.<span style="color:#a6e22e">configuration</span>.<span style="color:#a6e22e">identifiers</span>
</span></span><span style="display:flex;"><span>
</span></span><span style="display:flex;"><span>      <span style="color:#a6e22e">content</span> {
</span></span><span style="display:flex;"><span>        <span style="color:#a6e22e">type</span>        <span style="color:#f92672">=</span> <span style="color:#a6e22e">principals</span>.<span style="color:#a6e22e">key</span>
</span></span><span style="display:flex;"><span>        <span style="color:#a6e22e">identifiers</span> <span style="color:#f92672">=</span> <span style="color:#a6e22e">principals</span>.<span style="color:#a6e22e">value</span>
</span></span><span style="display:flex;"><span>      }
</span></span><span style="display:flex;"><span>    }
</span></span><span style="display:flex;"><span>  }<span style="color:#75715e">
</span></span></span><span style="display:flex;"><span><span style="color:#75715e">
</span></span></span><span style="display:flex;"><span><span style="color:#75715e">  # Trust to the SA if they are configured
</span></span></span><span style="display:flex;"><span>  <span style="color:#a6e22e">statement</span> {
</span></span><span style="display:flex;"><span>    <span style="color:#a6e22e">actions</span> <span style="color:#f92672">=</span> [<span style="color:#e6db74">&#34;sts:AssumeRoleWithWebIdentity&#34;</span>]
</span></span><span style="display:flex;"><span>
</span></span><span style="display:flex;"><span>    <span style="color:#a6e22e">principals</span> {
</span></span><span style="display:flex;"><span>      <span style="color:#a6e22e">type</span>        <span style="color:#f92672">=</span> <span style="color:#e6db74">&#34;Federated&#34;</span>
</span></span><span style="display:flex;"><span>      <span style="color:#a6e22e">identifiers</span> <span style="color:#f92672">=</span> [var.<span style="color:#a6e22e">oidc_provider_arn</span>]
</span></span><span style="display:flex;"><span>    }
</span></span><span style="display:flex;"><span>
</span></span><span style="display:flex;"><span>    <span style="color:#a6e22e">condition</span> {
</span></span><span style="display:flex;"><span>      <span style="color:#a6e22e">variable</span> <span style="color:#f92672">=</span> <span style="color:#e6db74">&#34;</span><span style="color:#e6db74">${</span>replace(var.<span style="color:#a6e22e">oidc_provider_arn</span>, <span style="color:#e6db74">&#34;/^(.*provider/)/&#34;</span>, <span style="color:#e6db74">&#34;&#34;</span>)<span style="color:#e6db74">}</span><span style="color:#e6db74">:sub&#34;</span>
</span></span><span style="display:flex;"><span>      <span style="color:#a6e22e">values</span>   <span style="color:#f92672">=</span> [<span style="color:#66d9ef">for</span> <span style="color:#a6e22e">sa</span> <span style="color:#66d9ef">in</span> module.<span style="color:#a6e22e">configuration</span>.<span style="color:#a6e22e">trusted_sa</span> <span style="color:#f92672">:</span> <span style="color:#e6db74">&#34;system:serviceaccount:</span><span style="color:#e6db74">${</span><span style="color:#a6e22e">sa</span><span style="color:#e6db74">}</span><span style="color:#e6db74">&#34;</span>]
</span></span><span style="display:flex;"><span>      <span style="color:#a6e22e">test</span>     <span style="color:#f92672">=</span> <span style="color:#e6db74">&#34;StringEquals&#34;</span>
</span></span><span style="display:flex;"><span>    }<span style="color:#75715e">
</span></span></span><span style="display:flex;"><span><span style="color:#75715e">
</span></span></span><span style="display:flex;"><span><span style="color:#75715e">    # :aud is not added because it works only with EKS
</span></span></span><span style="display:flex;"><span><span style="color:#75715e">    #    # https://aws.amazon.com/premiumsupport/knowledge-center/eks-troubleshoot-oidc-and-irsa/?nc1=h_ls
</span></span></span><span style="display:flex;"><span><span style="color:#75715e">    #    condition {
</span></span></span><span style="display:flex;"><span><span style="color:#75715e">    #      variable = &#34;${replace(var.oidc_provider_arn, &#34;/^(.*provider/)/&#34;, &#34;&#34;)}:aud&#34;
</span></span></span><span style="display:flex;"><span><span style="color:#75715e">    #      values   = [&#34;sts.amazonaws.com&#34;]
</span></span></span><span style="display:flex;"><span><span style="color:#75715e">    #      test     = &#34;StringEquals&#34;
</span></span></span><span style="display:flex;"><span><span style="color:#75715e">    #    }
</span></span></span><span style="display:flex;"><span>  }
</span></span><span style="display:flex;"><span>}
</span></span></code></pre></div>
</details>

<h3 id="testing-the-setup">Testing the setup</h3>
<p>To test that my SA with the assigned role via <code>eks.amazonaws.com/role-arn</code> annotation worked as expected,
I ran a debug pod with the mounted SA,
found in <a href="https://gist.github.com/aramse/6eb4a5b11269bb19d9ad35bb21ace6f6">gist</a>.</p>
<div class="highlight"><pre tabindex="0" style="color:#f8f8f2;background-color:#272822;-moz-tab-size:4;-o-tab-size:4;tab-size:4;-webkit-text-size-adjust:none;"><code class="language-bash" data-lang="bash"><span style="display:flex;"><span>kubectl run --rm -it debug --image amazon/aws-cli --command bash --overrides<span style="color:#f92672">=</span><span style="color:#e6db74">&#39;{ &#34;spec&#34;: { &#34;serviceAccount&#34;: &#34;my-cool-service-account&#34; }  }&#39;</span>
</span></span></code></pre></div><p>Then, I called the default sts method to verify the assumed role:</p>
<div class="highlight"><pre tabindex="0" style="color:#f8f8f2;background-color:#272822;-moz-tab-size:4;-o-tab-size:4;tab-size:4;-webkit-text-size-adjust:none;"><code class="language-bash" data-lang="bash"><span style="display:flex;"><span>aws sts get-caller-identity
</span></span></code></pre></div><h3 id="additional-information">Additional information</h3>
<p>To get a better understanding of how IRSA works in general, you may check incredible <a href="https://medium.com/@ankit.wal/the-how-of-iam-roles-for-service-accounts-irsa-on-aws-eks-3d76badb8942">article</a>.
It explains in an easy graphical form how trust relations works and why this way of granting permissions
is better than <code>Kiam</code> or <code>Kube2iam</code></p>
<h3 id="post-migration">Post migration</h3>
<p>I planned to migrate my workloads to use IRSA via mounted ServiceAccounts.
After that, I would enforce IMDSv2 with a hop limit set to <code>1</code>.
That would help me protect instance attached roles from being used by pods deployed to the same machine.</p>
<p>Pods with <code>hostNetwork</code> will be denied by gatekeeper to enforce the security policy.</p>
]]></content:encoded>
    </item>
    
    <item>
      <title>Hello World!</title>
      <link>https://kuzaxak.dev/posts/2023/04/23/hello-world/</link>
      <pubDate>Sun, 23 Apr 2023 00:00:00 +0000</pubDate>
      
      <guid>https://kuzaxak.dev/posts/2023/04/23/hello-world/</guid>
      <description>First blog post here. Hugo initial setup and simple hosting
on Digital Ocean static apps.
</description>
      <content:encoded><![CDATA[<p>I decided to start writing posts and was curious about what type of blog to start.
As an engineer, I wanted the flexibility of Markdown, version control traceability,
and automatic rollouts.</p>
<p>I chose Hugo for static website generation and, for now, Digital Ocean apps for serving pages.
Later on, I'll migrate to AWS CloudFront.</p>
<h3 id="theming">Theming</h3>
<p>Initially, I considered an interactive <a href="https://themes.gohugo.io/themes/hugo-theme-shell/">shell-like</a> theme, but it would be too nerdy.
So, I opted for <a href="https://themes.gohugo.io/themes/archie/">Archie</a> as the main theme, which provided a minimalistic, engineering-like blog style.</p>
<p>If you cloned your theme to a sub-path with <code>git clone</code>, don't forget to create a <code>.gitmodules</code> file;
otherwise, the DO builder won't pull the theme from the target repository.</p>
<div class="highlight"><pre tabindex="0" style="color:#f8f8f2;background-color:#272822;-moz-tab-size:4;-o-tab-size:4;tab-size:4;-webkit-text-size-adjust:none;"><code class="language-fallback" data-lang="fallback"><span style="display:flex;"><span>[submodule &#34;themes/archie&#34;]
</span></span><span style="display:flex;"><span>  path = themes/archie
</span></span><span style="display:flex;"><span>  url = https://github.com/athul/archie.git
</span></span></code></pre></div><h3 id="deploy">Deploy</h3>
<p>DO apps offer an easy deployment process. A simple YAML file placed in <code>.do/app.yaml</code>
provides everything needed to deploy a static app:</p>
<div class="highlight"><pre tabindex="0" style="color:#f8f8f2;background-color:#272822;-moz-tab-size:4;-o-tab-size:4;tab-size:4;-webkit-text-size-adjust:none;"><code class="language-yaml" data-lang="yaml"><span style="display:flex;"><span><span style="color:#f92672">name</span>: <span style="color:#ae81ff">static-hugo</span>
</span></span><span style="display:flex;"><span><span style="color:#f92672">static_sites</span>:
</span></span><span style="display:flex;"><span>- <span style="color:#f92672">build_command</span>: <span style="color:#ae81ff">rm -r ./public; hugo --destination ./public</span>
</span></span><span style="display:flex;"><span>  <span style="color:#f92672">environment_slug</span>: <span style="color:#ae81ff">hugo</span>
</span></span><span style="display:flex;"><span>  <span style="color:#f92672">github</span>:
</span></span><span style="display:flex;"><span>    <span style="color:#f92672">branch</span>: <span style="color:#ae81ff">master</span>
</span></span><span style="display:flex;"><span>    <span style="color:#f92672">deploy_on_push</span>: <span style="color:#66d9ef">true</span>
</span></span><span style="display:flex;"><span>    <span style="color:#f92672">repo</span>: <span style="color:#ae81ff">kuzaxak/blog</span>
</span></span><span style="display:flex;"><span>  <span style="color:#f92672">name</span>: <span style="color:#ae81ff">static-hugo</span>
</span></span></code></pre></div><p>I bound the DNS to DO nameservers on the Namecheap main page. Remember to check that they're
propagated, and you can continue with:</p>
<div class="highlight"><pre tabindex="0" style="color:#f8f8f2;background-color:#272822;-moz-tab-size:4;-o-tab-size:4;tab-size:4;-webkit-text-size-adjust:none;"><code class="language-bash" data-lang="bash"><span style="display:flex;"><span>dig NS kuzaxak.dev
</span></span></code></pre></div><p>After creating a DO App, follow the wizard – it's self-explanatory. You'll get a DO subdomain.
Attach your domain on the Settings page, and you'll automatically receive <code>A</code> records
in your DNS zone.</p>
<h3 id="fine-tuning">Fine tuning</h3>
<p>DO apps require default pages to be present in the final filesystem folder:</p>
<ul>
<li><code>index.html</code></li>
<li><code>404.html</code></li>
</ul>
<p>Hugo creates <code>index.html</code> without any additional configuration.</p>
<p><code>404.html</code> might be created by the theme, but not in my case. The theme's <code>404</code> was empty.
Easily fix this by creating a <code>layouts/404.html</code> <a href="https://gohugo.io/templates/404/">file</a>. Copy your theme's <code>index.html</code>
and add a helpful error message to guide people back or report a broken link.</p>
<p>Check that DO detects your content in the build log. It should look like this:</p>
<div class="highlight"><pre tabindex="0" style="color:#f8f8f2;background-color:#272822;-moz-tab-size:4;-o-tab-size:4;tab-size:4;-webkit-text-size-adjust:none;"><code class="language-gdscript3" data-lang="gdscript3"><span style="display:flex;"><span><span style="color:#960050;background-color:#1e0010">›</span> <span style="color:#66d9ef">static</span> site output directory is <span style="color:#f92672">not</span> explicitly set, attempting to look <span style="color:#66d9ef">for</span> <span style="color:#66d9ef">static</span> files
</span></span><span style="display:flex;"><span>[<span style="color:#ae81ff">2023</span><span style="color:#f92672">-</span><span style="color:#ae81ff">04</span><span style="color:#f92672">-</span><span style="color:#ae81ff">23</span> <span style="color:#ae81ff">11</span>:<span style="color:#ae81ff">47</span>:<span style="color:#ae81ff">52</span>] <span style="color:#960050;background-color:#1e0010">│</span>      checking <span style="color:#f92672">/</span>workspace<span style="color:#f92672">/</span>_static: <span style="color:#f92672">not</span> found
</span></span><span style="display:flex;"><span>[<span style="color:#ae81ff">2023</span><span style="color:#f92672">-</span><span style="color:#ae81ff">04</span><span style="color:#f92672">-</span><span style="color:#ae81ff">23</span> <span style="color:#ae81ff">11</span>:<span style="color:#ae81ff">47</span>:<span style="color:#ae81ff">52</span>] <span style="color:#960050;background-color:#1e0010">│</span>      checking <span style="color:#f92672">/</span>workspace<span style="color:#f92672">/</span>dist: <span style="color:#f92672">not</span> found
</span></span><span style="display:flex;"><span>[<span style="color:#ae81ff">2023</span><span style="color:#f92672">-</span><span style="color:#ae81ff">04</span><span style="color:#f92672">-</span><span style="color:#ae81ff">23</span> <span style="color:#ae81ff">11</span>:<span style="color:#ae81ff">47</span>:<span style="color:#ae81ff">52</span>] <span style="color:#960050;background-color:#1e0010">│</span>      checking <span style="color:#f92672">/</span>workspace<span style="color:#f92672">/</span>public: found
</span></span><span style="display:flex;"><span>[<span style="color:#ae81ff">2023</span><span style="color:#f92672">-</span><span style="color:#ae81ff">04</span><span style="color:#f92672">-</span><span style="color:#ae81ff">23</span> <span style="color:#ae81ff">11</span>:<span style="color:#ae81ff">47</span>:<span style="color:#ae81ff">52</span>] <span style="color:#960050;background-color:#1e0010">│</span>  <span style="color:#960050;background-color:#1e0010">✔</span> using document root <span style="color:#f92672">/</span>workspace<span style="color:#f92672">/</span>public
</span></span><span style="display:flex;"><span>[<span style="color:#ae81ff">2023</span><span style="color:#f92672">-</span><span style="color:#ae81ff">04</span><span style="color:#f92672">-</span><span style="color:#ae81ff">23</span> <span style="color:#ae81ff">11</span>:<span style="color:#ae81ff">47</span>:<span style="color:#ae81ff">52</span>] <span style="color:#960050;background-color:#1e0010">│</span> 
</span></span><span style="display:flex;"><span>[<span style="color:#ae81ff">2023</span><span style="color:#f92672">-</span><span style="color:#ae81ff">04</span><span style="color:#f92672">-</span><span style="color:#ae81ff">23</span> <span style="color:#ae81ff">11</span>:<span style="color:#ae81ff">47</span>:<span style="color:#ae81ff">52</span>] <span style="color:#960050;background-color:#1e0010">│</span> <span style="color:#f92672">=&gt;</span> Uploading files to Spaces
</span></span><span style="display:flex;"><span>[<span style="color:#ae81ff">2023</span><span style="color:#f92672">-</span><span style="color:#ae81ff">04</span><span style="color:#f92672">-</span><span style="color:#ae81ff">23</span> <span style="color:#ae81ff">11</span>:<span style="color:#ae81ff">47</span>:<span style="color:#ae81ff">52</span>] <span style="color:#960050;background-color:#1e0010">│</span>     categories<span style="color:#f92672">/</span>index<span style="color:#f92672">.</span>html
</span></span><span style="display:flex;"><span>[<span style="color:#ae81ff">2023</span><span style="color:#f92672">-</span><span style="color:#ae81ff">04</span><span style="color:#f92672">-</span><span style="color:#ae81ff">23</span> <span style="color:#ae81ff">11</span>:<span style="color:#ae81ff">47</span>:<span style="color:#ae81ff">52</span>] <span style="color:#960050;background-color:#1e0010">│</span>     contact<span style="color:#f92672">/</span>index<span style="color:#f92672">.</span>html
</span></span><span style="display:flex;"><span>[<span style="color:#ae81ff">2023</span><span style="color:#f92672">-</span><span style="color:#ae81ff">04</span><span style="color:#f92672">-</span><span style="color:#ae81ff">23</span> <span style="color:#ae81ff">11</span>:<span style="color:#ae81ff">47</span>:<span style="color:#ae81ff">52</span>] <span style="color:#960050;background-color:#1e0010">│</span>     about<span style="color:#f92672">/</span>index<span style="color:#f92672">.</span>html
</span></span><span style="display:flex;"><span>[<span style="color:#ae81ff">2023</span><span style="color:#f92672">-</span><span style="color:#ae81ff">04</span><span style="color:#f92672">-</span><span style="color:#ae81ff">23</span> <span style="color:#ae81ff">11</span>:<span style="color:#ae81ff">47</span>:<span style="color:#ae81ff">52</span>] <span style="color:#960050;background-color:#1e0010">│</span>     <span style="color:#ae81ff">404.</span>html
</span></span><span style="display:flex;"><span>[<span style="color:#ae81ff">2023</span><span style="color:#f92672">-</span><span style="color:#ae81ff">04</span><span style="color:#f92672">-</span><span style="color:#ae81ff">23</span> <span style="color:#ae81ff">11</span>:<span style="color:#ae81ff">47</span>:<span style="color:#ae81ff">52</span>] <span style="color:#960050;background-color:#1e0010">│</span>     categories<span style="color:#f92672">/</span>feed<span style="color:#f92672">.</span>xml
</span></span><span style="display:flex;"><span>[<span style="color:#ae81ff">2023</span><span style="color:#f92672">-</span><span style="color:#ae81ff">04</span><span style="color:#f92672">-</span><span style="color:#ae81ff">23</span> <span style="color:#ae81ff">11</span>:<span style="color:#ae81ff">47</span>:<span style="color:#ae81ff">52</span>] <span style="color:#960050;background-color:#1e0010">│</span>     feed<span style="color:#f92672">.</span>xml
</span></span><span style="display:flex;"><span>[<span style="color:#ae81ff">2023</span><span style="color:#f92672">-</span><span style="color:#ae81ff">04</span><span style="color:#f92672">-</span><span style="color:#ae81ff">23</span> <span style="color:#ae81ff">11</span>:<span style="color:#ae81ff">47</span>:<span style="color:#ae81ff">52</span>] <span style="color:#960050;background-color:#1e0010">│</span>     index<span style="color:#f92672">.</span>html
</span></span><span style="display:flex;"><span>[<span style="color:#ae81ff">2023</span><span style="color:#f92672">-</span><span style="color:#ae81ff">04</span><span style="color:#f92672">-</span><span style="color:#ae81ff">23</span> <span style="color:#ae81ff">11</span>:<span style="color:#ae81ff">47</span>:<span style="color:#ae81ff">52</span>] <span style="color:#960050;background-color:#1e0010">│</span>     js<span style="color:#f92672">/</span>feather<span style="color:#f92672">.</span>min<span style="color:#f92672">.</span>js
</span></span><span style="display:flex;"><span>[<span style="color:#ae81ff">2023</span><span style="color:#f92672">-</span><span style="color:#ae81ff">04</span><span style="color:#f92672">-</span><span style="color:#ae81ff">23</span> <span style="color:#ae81ff">11</span>:<span style="color:#ae81ff">47</span>:<span style="color:#ae81ff">52</span>] <span style="color:#960050;background-color:#1e0010">│</span>     js<span style="color:#f92672">/</span>main<span style="color:#f92672">.</span>js
</span></span><span style="display:flex;"><span>[<span style="color:#ae81ff">2023</span><span style="color:#f92672">-</span><span style="color:#ae81ff">04</span><span style="color:#f92672">-</span><span style="color:#ae81ff">23</span> <span style="color:#ae81ff">11</span>:<span style="color:#ae81ff">47</span>:<span style="color:#ae81ff">52</span>] <span style="color:#960050;background-color:#1e0010">│</span>     js<span style="color:#f92672">/</span>themetoggle<span style="color:#f92672">.</span>js
</span></span><span style="display:flex;"><span>[<span style="color:#ae81ff">2023</span><span style="color:#f92672">-</span><span style="color:#ae81ff">04</span><span style="color:#f92672">-</span><span style="color:#ae81ff">23</span> <span style="color:#ae81ff">11</span>:<span style="color:#ae81ff">47</span>:<span style="color:#ae81ff">52</span>] <span style="color:#960050;background-color:#1e0010">│</span>     page<span style="color:#f92672">/</span><span style="color:#ae81ff">1</span><span style="color:#f92672">/</span>index<span style="color:#f92672">.</span>html
</span></span><span style="display:flex;"><span>[<span style="color:#ae81ff">2023</span><span style="color:#f92672">-</span><span style="color:#ae81ff">04</span><span style="color:#f92672">-</span><span style="color:#ae81ff">23</span> <span style="color:#ae81ff">11</span>:<span style="color:#ae81ff">47</span>:<span style="color:#ae81ff">52</span>] <span style="color:#960050;background-color:#1e0010">│</span>     robots<span style="color:#f92672">.</span>txt
</span></span><span style="display:flex;"><span>[<span style="color:#ae81ff">2023</span><span style="color:#f92672">-</span><span style="color:#ae81ff">04</span><span style="color:#f92672">-</span><span style="color:#ae81ff">23</span> <span style="color:#ae81ff">11</span>:<span style="color:#ae81ff">47</span>:<span style="color:#ae81ff">52</span>] <span style="color:#960050;background-color:#1e0010">│</span>     sitemap<span style="color:#f92672">.</span>xml
</span></span><span style="display:flex;"><span>[<span style="color:#ae81ff">2023</span><span style="color:#f92672">-</span><span style="color:#ae81ff">04</span><span style="color:#f92672">-</span><span style="color:#ae81ff">23</span> <span style="color:#ae81ff">11</span>:<span style="color:#ae81ff">47</span>:<span style="color:#ae81ff">52</span>] <span style="color:#960050;background-color:#1e0010">│</span>     tags<span style="color:#f92672">/</span>feed<span style="color:#f92672">.</span>xml
</span></span><span style="display:flex;"><span>[<span style="color:#ae81ff">2023</span><span style="color:#f92672">-</span><span style="color:#ae81ff">04</span><span style="color:#f92672">-</span><span style="color:#ae81ff">23</span> <span style="color:#ae81ff">11</span>:<span style="color:#ae81ff">47</span>:<span style="color:#ae81ff">52</span>] <span style="color:#960050;background-color:#1e0010">│</span>     tags<span style="color:#f92672">/</span>index<span style="color:#f92672">.</span>html
</span></span><span style="display:flex;"><span>[<span style="color:#ae81ff">2023</span><span style="color:#f92672">-</span><span style="color:#ae81ff">04</span><span style="color:#f92672">-</span><span style="color:#ae81ff">23</span> <span style="color:#ae81ff">11</span>:<span style="color:#ae81ff">47</span>:<span style="color:#ae81ff">52</span>] <span style="color:#960050;background-color:#1e0010">│</span> 
</span></span><span style="display:flex;"><span>[<span style="color:#ae81ff">2023</span><span style="color:#f92672">-</span><span style="color:#ae81ff">04</span><span style="color:#f92672">-</span><span style="color:#ae81ff">23</span> <span style="color:#ae81ff">11</span>:<span style="color:#ae81ff">47</span>:<span style="color:#ae81ff">52</span>] <span style="color:#960050;background-color:#1e0010">│</span> <span style="color:#f92672">=&gt;</span> Configuring index document: index<span style="color:#f92672">.</span>html
</span></span><span style="display:flex;"><span>[<span style="color:#ae81ff">2023</span><span style="color:#f92672">-</span><span style="color:#ae81ff">04</span><span style="color:#f92672">-</span><span style="color:#ae81ff">23</span> <span style="color:#ae81ff">11</span>:<span style="color:#ae81ff">47</span>:<span style="color:#ae81ff">52</span>] <span style="color:#960050;background-color:#1e0010">│</span> <span style="color:#f92672">=&gt;</span> Configuring error document: <span style="color:#ae81ff">404.</span>html
</span></span><span style="display:flex;"><span>[<span style="color:#ae81ff">2023</span><span style="color:#f92672">-</span><span style="color:#ae81ff">04</span><span style="color:#f92672">-</span><span style="color:#ae81ff">23</span> <span style="color:#ae81ff">11</span>:<span style="color:#ae81ff">47</span>:<span style="color:#ae81ff">52</span>] <span style="color:#960050;background-color:#1e0010">│</span> <span style="color:#f92672">=&gt;</span> Uploaded <span style="color:#ae81ff">34</span> files
</span></span></code></pre></div><h3 id="content">Content</h3>
<p>Adding content is pretty easy. I recommend using date prefixes for the posts.
This way, you can easily get a sorted list without introducing subfolders.</p>
<p>To automate date parsing from a new file name, create <code>archetypes/posts.md</code> with a parser regex:</p>
<div class="highlight"><pre tabindex="0" style="color:#f8f8f2;background-color:#272822;-moz-tab-size:4;-o-tab-size:4;tab-size:4;-webkit-text-size-adjust:none;"><code class="language-yaml" data-lang="yaml"><span style="display:flex;"><span>---
</span></span><span style="display:flex;"><span><span style="color:#f92672">title</span>: <span style="color:#e6db74">&#34;{{ replace (.Name | replaceRE &#34;</span><span style="color:#ae81ff">^[0-9]{4}-[0-9]{2}-[0-9]{2}-&#34; &#34;&#34;) &#34;-&#34; &#34; &#34; | title }}&#34;</span>
</span></span><span style="display:flex;"><span><span style="color:#f92672">description</span>: |<span style="color:#e6db74">
</span></span></span><span style="display:flex;"><span><span style="color:#e6db74">  Description</span>
</span></span><span style="display:flex;"><span><span style="color:#f92672">tags</span>: []
</span></span><span style="display:flex;"><span><span style="color:#f92672">date</span>: {{ <span style="color:#ae81ff">.Name | replaceRE &#34;^([0-9]{4}-[0-9]{2}-[0-9]{2}).*&#34; &#34;$1&#34; }}</span>
</span></span><span style="display:flex;"><span><span style="color:#f92672">slug</span>: <span style="color:#e6db74">&#34;{{ .Name | replaceRE &#34;</span><span style="color:#ae81ff">^[0-9]{4}-[0-9]{2}-[0-9]{2}-&#34; &#34;&#34; }}&#34;</span>
</span></span><span style="display:flex;"><span>---
</span></span></code></pre></div><p>Now, you'll be able to create your first page. I started with this article to practice
writing.</p>
<div class="highlight"><pre tabindex="0" style="color:#f8f8f2;background-color:#272822;-moz-tab-size:4;-o-tab-size:4;tab-size:4;-webkit-text-size-adjust:none;"><code class="language-bash" data-lang="bash"><span style="display:flex;"><span>hugo new posts/2023-04-23-hello-world.md
</span></span></code></pre></div><h3 id="stats">Stats</h3>
<p>You're probably curious if anyone is actually reading your blog. To find out, you can add Google Analytics,
which is supported by Hugo.</p>
<p>Register your website and prove ownership via DNS; after that, you'll receive a G-tag code.</p>
<p>Add it to your <code>config.toml</code>; mine looks like this:</p>
<div class="highlight"><pre tabindex="0" style="color:#f8f8f2;background-color:#272822;-moz-tab-size:4;-o-tab-size:4;tab-size:4;-webkit-text-size-adjust:none;"><code class="language-toml" data-lang="toml"><span style="display:flex;"><span><span style="color:#a6e22e">relativeURLs</span> = <span style="color:#66d9ef">true</span>
</span></span><span style="display:flex;"><span><span style="color:#a6e22e">baseURL</span> = <span style="color:#e6db74">&#34;/&#34;</span>
</span></span><span style="display:flex;"><span><span style="color:#a6e22e">languageCode</span> = <span style="color:#e6db74">&#34;en-us&#34;</span>
</span></span><span style="display:flex;"><span><span style="color:#a6e22e">title</span> = <span style="color:#e6db74">&#34;Vladimir Kuznichenkov | Engineer&#34;</span>
</span></span><span style="display:flex;"><span><span style="color:#a6e22e">author</span> = <span style="color:#e6db74">&#34;Vladimir Kuznichenkov&#34;</span>
</span></span><span style="display:flex;"><span><span style="color:#a6e22e">theme</span> = <span style="color:#e6db74">&#34;archie&#34;</span>
</span></span><span style="display:flex;"><span><span style="color:#a6e22e">copyright</span> = <span style="color:#e6db74">&#34;© Vladimir Kuznichenkov&#34;</span>
</span></span><span style="display:flex;"><span>
</span></span><span style="display:flex;"><span><span style="color:#75715e"># get information about the Git repo, such as last commit to a file</span>
</span></span><span style="display:flex;"><span><span style="color:#a6e22e">enableGitInfo</span> = <span style="color:#66d9ef">true</span>
</span></span><span style="display:flex;"><span>
</span></span><span style="display:flex;"><span><span style="color:#75715e"># Code Highlight</span>
</span></span><span style="display:flex;"><span><span style="color:#a6e22e">pygmentsstyle</span> = <span style="color:#e6db74">&#34;monokai&#34;</span>
</span></span><span style="display:flex;"><span><span style="color:#a6e22e">pygmentscodefences</span> = <span style="color:#66d9ef">true</span>
</span></span><span style="display:flex;"><span><span style="color:#a6e22e">pygmentscodefencesguesssyntax</span> = <span style="color:#66d9ef">true</span>
</span></span><span style="display:flex;"><span>
</span></span><span style="display:flex;"><span><span style="color:#a6e22e">paginate</span>=<span style="color:#ae81ff">5</span>
</span></span><span style="display:flex;"><span>
</span></span><span style="display:flex;"><span><span style="color:#a6e22e">googleAnalytics</span> = <span style="color:#e6db74">&#39;G-MEASUREMENT_ID&#39;</span>
</span></span><span style="display:flex;"><span><span style="color:#a6e22e">enableRobotsTXT</span> = <span style="color:#66d9ef">true</span>
</span></span><span style="display:flex;"><span>
</span></span><span style="display:flex;"><span>[<span style="color:#a6e22e">params</span>]
</span></span><span style="display:flex;"><span>  <span style="color:#a6e22e">mode</span> = <span style="color:#e6db74">&#34;toggle&#34;</span>
</span></span><span style="display:flex;"><span>  <span style="color:#a6e22e">author</span> = <span style="color:#e6db74">&#34;Vladimir Kuznichenkov&#34;</span>
</span></span><span style="display:flex;"><span>  <span style="color:#a6e22e">copyright</span> = <span style="color:#e6db74">&#34;@ Vladimir Kuznichenkov&#34;</span>
</span></span><span style="display:flex;"><span>  <span style="color:#a6e22e">WebMentionDomain</span> = <span style="color:#e6db74">&#34;kuzaxak.dev&#34;</span>
</span></span><span style="display:flex;"><span>  <span style="color:#a6e22e">mathjax</span> = <span style="color:#66d9ef">true</span> <span style="color:#75715e"># enable MathJax support</span>
</span></span><span style="display:flex;"><span>  <span style="color:#a6e22e">katex</span> = <span style="color:#66d9ef">true</span> <span style="color:#75715e"># enable KaTeX support</span>
</span></span><span style="display:flex;"><span>
</span></span><span style="display:flex;"><span><span style="color:#75715e"># Social Tags</span>
</span></span><span style="display:flex;"><span>[[<span style="color:#a6e22e">params</span>.<span style="color:#a6e22e">social</span>]]
</span></span><span style="display:flex;"><span>  <span style="color:#a6e22e">name</span> = <span style="color:#e6db74">&#34;GitHub&#34;</span>
</span></span><span style="display:flex;"><span>  <span style="color:#a6e22e">icon</span> = <span style="color:#e6db74">&#34;github&#34;</span>
</span></span><span style="display:flex;"><span>  <span style="color:#a6e22e">url</span> = <span style="color:#e6db74">&#34;https://github.com/kuzaxak/blog&#34;</span>
</span></span><span style="display:flex;"><span>
</span></span><span style="display:flex;"><span>[[<span style="color:#a6e22e">params</span>.<span style="color:#a6e22e">social</span>]]
</span></span><span style="display:flex;"><span>  <span style="color:#a6e22e">name</span> = <span style="color:#e6db74">&#34;Twitter&#34;</span>
</span></span><span style="display:flex;"><span>  <span style="color:#a6e22e">icon</span> = <span style="color:#e6db74">&#34;twitter&#34;</span>
</span></span><span style="display:flex;"><span>  <span style="color:#a6e22e">url</span> = <span style="color:#e6db74">&#34;https://twitter.com/kuzaxak/&#34;</span>
</span></span><span style="display:flex;"><span>
</span></span><span style="display:flex;"><span>[<span style="color:#a6e22e">permalinks</span>]
</span></span><span style="display:flex;"><span>  <span style="color:#a6e22e">posts</span> = <span style="color:#e6db74">&#34;/posts/:year/:month/:day/:slug/&#34;</span>
</span></span><span style="display:flex;"><span>  <span style="color:#a6e22e">tags</span> = <span style="color:#e6db74">&#34;/tags/:title/&#34;</span>
</span></span><span style="display:flex;"><span>
</span></span><span style="display:flex;"><span>[<span style="color:#a6e22e">outputFormats</span>]
</span></span><span style="display:flex;"><span>[<span style="color:#a6e22e">outputFormats</span>.<span style="color:#a6e22e">rss</span>]
</span></span><span style="display:flex;"><span>  <span style="color:#a6e22e">mediatype</span> = <span style="color:#e6db74">&#34;application/rss&#34;</span>
</span></span><span style="display:flex;"><span>  <span style="color:#a6e22e">baseName</span> = <span style="color:#e6db74">&#34;feed&#34;</span>
</span></span><span style="display:flex;"><span>
</span></span><span style="display:flex;"><span>[<span style="color:#a6e22e">markup</span>]
</span></span><span style="display:flex;"><span>  [<span style="color:#a6e22e">markup</span>.<span style="color:#a6e22e">goldmark</span>]
</span></span><span style="display:flex;"><span>    [<span style="color:#a6e22e">markup</span>.<span style="color:#a6e22e">goldmark</span>.<span style="color:#a6e22e">extensions</span>]
</span></span><span style="display:flex;"><span>    <span style="color:#a6e22e">typographer</span> = <span style="color:#66d9ef">false</span>
</span></span><span style="display:flex;"><span>    [<span style="color:#a6e22e">markup</span>.<span style="color:#a6e22e">goldmark</span>.<span style="color:#a6e22e">renderer</span>]
</span></span><span style="display:flex;"><span>    <span style="color:#75715e"># allow inline HTML, such as spoiler warnings</span>
</span></span><span style="display:flex;"><span>    <span style="color:#a6e22e">unsafe</span> = <span style="color:#66d9ef">true</span>
</span></span><span style="display:flex;"><span>  [<span style="color:#a6e22e">markup</span>.<span style="color:#a6e22e">tableOfContents</span>]
</span></span><span style="display:flex;"><span>  <span style="color:#a6e22e">startLevel</span> = <span style="color:#ae81ff">1</span>
</span></span><span style="display:flex;"><span>  <span style="color:#a6e22e">endLevel</span> = <span style="color:#ae81ff">6</span>
</span></span><span style="display:flex;"><span>  <span style="color:#a6e22e">ordered</span> = <span style="color:#66d9ef">false</span>
</span></span><span style="display:flex;"><span>
</span></span><span style="display:flex;"><span>[<span style="color:#a6e22e">menu</span>]
</span></span><span style="display:flex;"><span>  [[<span style="color:#a6e22e">menu</span>.<span style="color:#a6e22e">main</span>]]
</span></span><span style="display:flex;"><span>    <span style="color:#a6e22e">identifier</span> = <span style="color:#e6db74">&#34;blog&#34;</span>
</span></span><span style="display:flex;"><span>    <span style="color:#a6e22e">name</span> = <span style="color:#e6db74">&#34;./blog&#34;</span>
</span></span><span style="display:flex;"><span>    <span style="color:#a6e22e">url</span> = <span style="color:#e6db74">&#34;/posts&#34;</span>
</span></span><span style="display:flex;"><span>    <span style="color:#a6e22e">weight</span> = <span style="color:#ae81ff">2</span>
</span></span><span style="display:flex;"><span>  [[<span style="color:#a6e22e">menu</span>.<span style="color:#a6e22e">main</span>]]
</span></span><span style="display:flex;"><span>    <span style="color:#a6e22e">identifier</span> = <span style="color:#e6db74">&#34;about&#34;</span>
</span></span><span style="display:flex;"><span>    <span style="color:#a6e22e">name</span> = <span style="color:#e6db74">&#34;./about&#34;</span>
</span></span><span style="display:flex;"><span>    <span style="color:#a6e22e">url</span> = <span style="color:#e6db74">&#34;/about&#34;</span>
</span></span><span style="display:flex;"><span>    <span style="color:#a6e22e">weight</span> = <span style="color:#ae81ff">3</span>
</span></span><span style="display:flex;"><span>
</span></span><span style="display:flex;"><span>  [[<span style="color:#a6e22e">menu</span>.<span style="color:#a6e22e">footer</span>]]
</span></span><span style="display:flex;"><span>    <span style="color:#a6e22e">name</span> = <span style="color:#e6db74">&#34;Tags&#34;</span>
</span></span><span style="display:flex;"><span>    <span style="color:#a6e22e">url</span> = <span style="color:#e6db74">&#34;/tags&#34;</span>
</span></span><span style="display:flex;"><span>    <span style="color:#a6e22e">weight</span> = <span style="color:#ae81ff">4</span>
</span></span><span style="display:flex;"><span>
</span></span><span style="display:flex;"><span>[<span style="color:#a6e22e">sitemap</span>]
</span></span><span style="display:flex;"><span>  <span style="color:#a6e22e">changefreq</span> = <span style="color:#e6db74">&#39;monthly&#39;</span>
</span></span><span style="display:flex;"><span>  <span style="color:#a6e22e">filename</span> = <span style="color:#e6db74">&#39;sitemap.xml&#39;</span>
</span></span><span style="display:flex;"><span>  <span style="color:#a6e22e">priority</span> = <span style="color:#ae81ff">0.5</span>
</span></span></code></pre></div>]]></content:encoded>
    </item>
    
  </channel>
</rss>
