Jekyll2023-10-30T17:17:53+00:00/Alex Wheelerđ”
Library Configuration2019-05-15T13:36:14+00:002019-05-15T13:36:14+00:00/ruby/2019/05/15/library-configuration<p>At some point in your open source projectâs lifetime users are going to want the ability to configure your software to suit their needs. Theyâll want to add their own text to a dashboard. Theyâll need to disable the ability to take some action. Theyâll want to rip out your <a href="https://github.com/jnunemaker/flipper/pull/384">Taylor Swift video</a> from the UI. FWIW I actually find the Taylor Swift embed pretty funny.</p>
<p>What if I told you there was a way we could all be happy? And, no, the answer doesnât involve monkey patching. The easiest pattern Iâve come across involves introducing a <code class="highlighter-rouge">Configuration</code> class:</p>
<div class="language-ruby highlighter-rouge"><pre class="highlight"><code><span class="k">class</span> <span class="nc">Configuration</span>
<span class="kp">attr_accessor</span> <span class="ss">:title</span><span class="p">,</span> <span class="ss">:text</span>
<span class="k">def</span> <span class="nf">initialize</span>
<span class="vi">@title</span> <span class="o">=</span> <span class="s2">"Title Default"</span>
<span class="vi">@text</span> <span class="o">=</span> <span class="s2">"Text Default"</span>
<span class="k">end</span>
<span class="k">end</span>
</code></pre>
</div>
<p>Adding a getter to your libraryâs base class to retrieve a <code class="highlighter-rouge">Configuration</code> instance:</p>
<div class="language-ruby highlighter-rouge"><pre class="highlight"><code><span class="k">class</span> <span class="nc">Library</span>
<span class="k">def</span> <span class="nc">self</span><span class="o">.</span><span class="nf">configuration</span>
<span class="vi">@configuration</span> <span class="o">||=</span> <span class="no">Configuration</span><span class="p">.</span><span class="nf">new</span>
<span class="k">end</span>
<span class="k">end</span>
</code></pre>
</div>
<p>Note <code class="highlighter-rouge">@configuration</code> is memoized so the first time <code class="highlighter-rouge">configuration</code> is called the instance variable is set and any following calls will return this reference instead of initializing a new instance each time.</p>
<p>Finally providing a <code class="highlighter-rouge">configure</code> method that yields the configuration - making our instance <em>configurable</em>:</p>
<div class="language-ruby highlighter-rouge"><pre class="highlight"><code><span class="k">class</span> <span class="nc">Library</span>
<span class="k">def</span> <span class="nc">self</span><span class="o">.</span><span class="nf">configure</span>
<span class="k">yield</span><span class="p">(</span><span class="n">configuration</span><span class="p">)</span>
<span class="k">end</span>
<span class="k">def</span> <span class="nc">self</span><span class="o">.</span><span class="nf">configuration</span>
<span class="vi">@configuration</span> <span class="o">||=</span> <span class="no">Configuration</span><span class="p">.</span><span class="nf">new</span>
<span class="k">end</span>
<span class="k">end</span>
</code></pre>
</div>
<p>Now consumers of the library can:</p>
<div class="language-ruby highlighter-rouge"><pre class="highlight"><code><span class="no">Library</span><span class="p">.</span><span class="nf">configure</span> <span class="k">do</span> <span class="o">|</span><span class="n">config</span><span class="o">|</span>
<span class="n">config</span><span class="p">.</span><span class="nf">title</span> <span class="o">=</span> <span class="s2">"My Custom Title"</span>
<span class="n">config</span><span class="p">.</span><span class="nf">text</span> <span class="o">=</span> <span class="s2">"My Custom Text"</span>
<span class="k">end</span>
</code></pre>
</div>
<p>Any place we reference <code class="highlighter-rouge">Library.configuration.title</code> and <code class="highlighter-rouge">Library.configuration.text</code> will return the configured values. If none have been configured then our defaults in <code class="highlighter-rouge">Configuration#initialize</code> will be returned:</p>
<div class="language-erb highlighter-rouge"><pre class="highlight"><code> <span class="nt"><h1></span><span class="cp"><%=</span> <span class="no">Library</span><span class="p">.</span><span class="nf">configuration</span><span class="p">.</span><span class="nf">title</span> <span class="cp">%></span><span class="nt"></h1></span>
<span class="nt"><p></span><span class="cp"><%=</span> <span class="no">Library</span><span class="p">.</span><span class="nf">configuration</span><span class="p">.</span><span class="nf">body</span> <span class="cp">%></span><span class="nt"></p></span>
</code></pre>
</div>
<p>This all works for two reasons:</p>
<ol>
<li>
<p>We memozied the configuration with <code class="highlighter-rouge">@configuration ||= Configuration.new</code> so any configured changes applied to <code class="highlighter-rouge">@configuration</code> will be returned.</p>
</li>
<li>
<p>Classes in Ruby are just instances of <code class="highlighter-rouge">Class</code> referenced by a constant, which canât be changed (Thatâs not entirely true, but weâll save that explanation for another time). This means that no matter where we reference <code class="highlighter-rouge">Library.configuration</code> we know weâre getting the configured configuration.</p>
</li>
</ol>
<p><em>Fun Fact: These two lines do the same thing. Maybe weâll explore this in a future blog post:</em></p>
<div class="language-ruby highlighter-rouge"><pre class="highlight"><code><span class="k">class</span> <span class="nc">Library</span><span class="p">;</span> <span class="k">end</span>
<span class="no">Library</span> <span class="o">=</span> <span class="no">Class</span><span class="p">.</span><span class="nf">new</span>
</code></pre>
</div>
<p>This is the exact pattern I used to implement the <a href="https://github.com/jnunemaker/flipper/commit/3ced5fb32021a428b369cdabb83afc7e2c64c22c">Flipper::UI` configuration</a> and users seem find it pretty cool.</p>At some point in your open source projectâs lifetime users are going to want the ability to configure your software to suit their needs. Theyâll want to add their own text to a dashboard. Theyâll need to disable the ability to take some action. Theyâll want to rip out your Taylor Swift video from the UI. FWIW I actually find the Taylor Swift embed pretty funny.Auditing with Rack Middleware2019-04-25T13:36:14+00:002019-04-25T13:36:14+00:00/ruby/2019/04/25/auditing-with-rack-middleware<h1 id="rack">Rack</h1>
<p><a href="https://github.com/rack/rack">Rack</a> describes itself as a âminimal, modular, and adaptable interface for
developing web applications in Rubyâ. In laymanâs terms its just a set of rules that, when followed,
allow web servers and web applications to talk to each other.</p>
<p>The typical flow of a web app is as follows:</p>
<ul>
<li>Web server receives request over the wire as plain-text</li>
<li>Web server hands this request off to web application to do something</li>
<li>Rails application takes some action, for example, updates a user record in the database</li>
<li>Rails hands the response back to the web server</li>
<li>Web server responds to the client as plain-text</li>
</ul>
<p>Plain-text in, plain-text out.</p>
<p>There are lots of web servers: WEBrick, Thin, Puma, etc. There are even more Ruby web frameworks: Rails, Sinatra, Rum, Hanami, Merb, just to name a few.</p>
<p>One day web server authors and web framework authors got wise and held a big meeting. The web framework authors were growing really tired of having
to write tons of code to support all the different ways web servers were handing them representations of these plain-text HTTP requests.
Some passed a request as a raw String, others as an Array, and some gave it as a Hash - in Spanish! The web server authors were also getting pretty fed up with
framework authors. Every framework was handing their responses to the web servers in a different format. Complete chaos. So what happened?</p>
<p>The web server peeps came up with a set of rules. They said your frameworks need to provide an <em>app</em> object that responds to the <code class="highlighter-rouge">call</code> method, taking a <code class="highlighter-rouge">Hash</code> representing the request.
This object needs to return an <code class="highlighter-rouge">Array</code> with three elements:</p>
<ul>
<li>HTTP response code</li>
<li>Hash of headers</li>
<li>the response body, which must respond to <code class="highlighter-rouge">each</code></li>
</ul>
<p>And with that the seas parted and Rack was born.</p>
<p>Hereâs a bare-bones rack app built from a Proc. Notice it follows all of the above rules.</p>
<div class="language-ruby highlighter-rouge"><pre class="highlight"><code><span class="nb">require</span> <span class="s1">'rack'</span>
<span class="n">app</span> <span class="o">=</span> <span class="no">Proc</span><span class="p">.</span><span class="nf">new</span> <span class="k">do</span> <span class="o">|</span><span class="n">env</span><span class="o">|</span>
<span class="p">[</span><span class="s1">'200'</span><span class="p">,</span> <span class="p">{</span><span class="s1">'Content-Type'</span> <span class="o">=></span> <span class="s1">'text/html'</span><span class="p">},</span> <span class="p">[</span><span class="s1">'A barebones rack app.'</span><span class="p">]]</span>
<span class="k">end</span>
<span class="no">Rack</span><span class="o">::</span><span class="no">Handler</span><span class="o">::</span><span class="no">WEBrick</span><span class="p">.</span><span class="nf">run</span> <span class="n">app</span>
</code></pre>
</div>
<p>Not a week after Rack was released Medium and HackerNews exploded with posts about the <em>Death of Rack</em>. They
thought Rack was great, but there were three problems:</p>
<ol>
<li>Developers needed the ability to manipulate requests after their web server had received them, but before
they reached their web framework.</li>
<li>They needed the option to manipulate responses after their web framework had responded, but before they
reached their web server to be sent to the client.</li>
<li>No keyboard warrior on either site could decide which color the logo should be.</li>
</ol>
<p>Problem #3 was punted for a further date, but the Rack authors quickly came up with a solution to problems #1
and #2. They figured their solution would sit in the middle of the web server and the application and it
needed to sound somewhat smart so they tacked on <em>ware</em>. They following day Rack Middleware was introduced.</p>
<h1 id="rack-middleware">Rack Middleware</h1>
<p>Middlware sits between a web server and an app and allows developers to manipulate the requests and responses
the two exchange. Middlewares can be chained, each one altering the request/response and passing it along to
the next one in the chain. Rack provides a small <code class="highlighter-rouge">Rack::Builder</code> DSL for constructing Rack apps that use
middleware.</p>
<div class="language-ruby highlighter-rouge"><pre class="highlight"><code><span class="n">app</span> <span class="o">=</span> <span class="no">Proc</span><span class="p">.</span><span class="nf">new</span> <span class="p">{</span> <span class="o">|</span><span class="n">env</span><span class="o">|</span> <span class="p">[</span><span class="s1">'200'</span><span class="p">,</span> <span class="p">{</span><span class="s1">'Content-Type'</span> <span class="o">=></span> <span class="s1">'text/html'</span><span class="p">},</span> <span class="p">[</span><span class="s1">'A barebones rack app.'</span><span class="p">]]</span> <span class="p">}</span>
<span class="n">builder</span> <span class="o">=</span> <span class="no">Rack</span><span class="o">::</span><span class="no">Builder</span><span class="p">.</span><span class="nf">new</span>
<span class="n">builder</span><span class="p">.</span><span class="nf">use</span> <span class="no">FooMiddleware</span>
<span class="n">builder</span><span class="p">.</span><span class="nf">use</span> <span class="no">BarMiddleware</span>
<span class="n">builder</span><span class="p">.</span><span class="nf">run</span> <span class="n">app</span>
<span class="no">Rack</span><span class="o">::</span><span class="no">Handler</span><span class="o">::</span><span class="no">WEBrick</span><span class="p">.</span><span class="nf">run</span> <span class="n">builder</span>
</code></pre>
</div>
<p><code class="highlighter-rouge">use</code> adds middleware to the stack. <code class="highlighter-rouge">run</code> dispatches to an application.</p>
<p>Rack Middleware must follow the same Rack guidelines with one extra rule - it must have an initialize method
that takes 1 argument, the next thing in the stack built by Rack::Builder. When a request comes in the <code class="highlighter-rouge">call</code>
method will be invoked with the request named <code class="highlighter-rouge">env</code>. The middleware can then decide to manipulate the request and
pass it down the stack towards the app. It can also manipulate it when the response is passed back up the
stack to it.</p>
<p>Letâs take a look at a basic middleware:</p>
<div class="language-ruby highlighter-rouge"><pre class="highlight"><code><span class="k">class</span> <span class="nc">FooMiddleware</span>
<span class="k">def</span> <span class="nf">initialize</span><span class="p">(</span><span class="n">app</span><span class="p">)</span>
<span class="vi">@app</span> <span class="o">=</span> <span class="n">app</span>
<span class="k">end</span>
<span class="k">def</span> <span class="nf">call</span><span class="p">(</span><span class="n">env</span><span class="p">)</span>
<span class="c1"># env represents the request</span>
<span class="c1"># @app.call passes the request down the stack to the app</span>
<span class="n">status</span><span class="p">,</span> <span class="n">headers</span><span class="p">,</span> <span class="n">body</span> <span class="o">=</span> <span class="vi">@app</span><span class="p">.</span><span class="nf">call</span><span class="p">(</span><span class="n">env</span><span class="p">)</span>
<span class="c1"># At this point the response can be manipulated</span>
<span class="p">[</span><span class="n">status</span><span class="p">,</span> <span class="n">headers</span><span class="p">,</span> <span class="n">body</span><span class="p">]</span>
<span class="k">end</span>
<span class="k">end</span>
</code></pre>
</div>
<p>Thatâs really all there is to know about Rack. This little bit of knowledge will prove super useful when
working with most Ruby web applications as weâll see in the next sections.</p>
<h1 id="flipperui">Flipper::UI</h1>
<p>Flipper provides a really slick UI for managing feature flags that can be mounted at any endpoint.</p>
<div class="language-ruby highlighter-rouge"><pre class="highlight"><code><span class="no">Rails</span><span class="p">.</span><span class="nf">application</span><span class="p">.</span><span class="nf">routes</span><span class="p">.</span><span class="nf">draw</span> <span class="k">do</span>
<span class="n">mount</span> <span class="no">Flipper</span><span class="o">::</span><span class="no">UI</span><span class="p">.</span><span class="nf">app</span><span class="p">(</span><span class="no">Flipper</span><span class="p">)</span> <span class="o">=></span> <span class="s1">'/flipper'</span>
<span class="k">end</span>
</code></pre>
</div>
<p>Navigating to <code class="highlighter-rouge">/flipper</code> will now display a list of features:</p>
<p><img src="/assets/middleware-audit/flipper-ui-features.png" alt="flipper-ui-features" /></p>
<p>Clicking into one of those features will take users to a page for managing that feature:</p>
<p><img src="/assets/middleware-audit/flipper-ui-feature.png" alt="flipper-ui-features" /></p>
<p>With one line of code you can provide users of your library with a really nice web interface thanks to Rack. <code class="highlighter-rouge">mount</code> expects a Rack app, which is exactly what <code class="highlighter-rouge">Flipper::UI.app</code> returns. Whenever a request hits <code class="highlighter-rouge">/flipper</code> we can guarantee that
our appâs call method will be invoked with a Hash representing the request.</p>
<p>Since we all now know how Rack works you will have no problem seeing that <code class="highlighter-rouge">Flipper::UI</code> is just a Rack
application built on top of a plain old Ruby lambda:
<em><a href="https://github.com/jnunemaker/flipper/blob/master/lib/flipper/ui.rb#L42">source</a></em></p>
<div class="language-ruby highlighter-rouge"><pre class="highlight"><code><span class="k">module</span> <span class="nn">Flipper</span>
<span class="k">class</span> <span class="nc">UI</span>
<span class="k">def</span> <span class="nc">self</span><span class="o">.</span><span class="nf">app</span><span class="p">(</span><span class="n">flipper</span> <span class="o">=</span> <span class="kp">nil</span><span class="p">,</span> <span class="n">options</span> <span class="o">=</span> <span class="p">{})</span>
<span class="n">env_key</span> <span class="o">=</span> <span class="n">options</span><span class="p">.</span><span class="nf">fetch</span><span class="p">(</span><span class="ss">:env_key</span><span class="p">,</span> <span class="s1">'flipper'</span><span class="p">)</span>
<span class="n">app</span> <span class="o">=</span> <span class="o">-></span><span class="p">()</span> <span class="p">{</span> <span class="p">[</span><span class="mi">200</span><span class="p">,</span> <span class="p">{</span> <span class="s1">'Content-Type'</span> <span class="o">=></span> <span class="s1">'text/html'</span> <span class="p">},</span> <span class="p">[</span><span class="s1">''</span><span class="p">]]</span> <span class="p">}</span>
<span class="n">builder</span> <span class="o">=</span> <span class="no">Rack</span><span class="o">::</span><span class="no">Builder</span><span class="p">.</span><span class="nf">new</span>
<span class="k">yield</span> <span class="n">builder</span> <span class="k">if</span> <span class="nb">block_given?</span>
<span class="n">builder</span><span class="p">.</span><span class="nf">use</span> <span class="no">Rack</span><span class="o">::</span><span class="no">Protection</span>
<span class="n">builder</span><span class="p">.</span><span class="nf">use</span> <span class="no">Rack</span><span class="o">::</span><span class="no">Protection</span><span class="o">::</span><span class="no">AuthenticityToken</span>
<span class="n">builder</span><span class="p">.</span><span class="nf">use</span> <span class="no">Rack</span><span class="o">::</span><span class="no">MethodOverride</span>
<span class="n">builder</span><span class="p">.</span><span class="nf">use</span> <span class="no">Flipper</span><span class="o">::</span><span class="no">Middleware</span><span class="o">::</span><span class="no">SetupEnv</span><span class="p">,</span> <span class="n">flipper</span><span class="p">,</span> <span class="ss">env_key: </span><span class="n">env_key</span>
<span class="n">builder</span><span class="p">.</span><span class="nf">use</span> <span class="no">Flipper</span><span class="o">::</span><span class="no">Middleware</span><span class="o">::</span><span class="no">Memoizer</span><span class="p">,</span> <span class="ss">env_key: </span><span class="n">env_key</span>
<span class="n">builder</span><span class="p">.</span><span class="nf">use</span> <span class="no">Middleware</span><span class="p">,</span> <span class="ss">env_key: </span><span class="n">env_key</span>
<span class="n">builder</span><span class="p">.</span><span class="nf">run</span> <span class="n">app</span>
<span class="n">klass</span> <span class="o">=</span> <span class="nb">self</span>
<span class="n">builder</span><span class="p">.</span><span class="nf">define_singleton_method</span><span class="p">(</span><span class="ss">:inspect</span><span class="p">)</span> <span class="p">{</span> <span class="n">klass</span><span class="p">.</span><span class="nf">inspect</span> <span class="p">}</span> <span class="c1"># pretty rake routes output</span>
<span class="n">builder</span>
<span class="k">end</span>
<span class="k">end</span>
<span class="k">end</span>
</code></pre>
</div>
<p>If you look closely youâll notice one line that opens some pretty cool doors.</p>
<div class="language-ruby highlighter-rouge"><pre class="highlight"><code><span class="k">def</span> <span class="nc">self</span><span class="o">.</span><span class="nf">app</span><span class="p">(</span><span class="n">flipper</span> <span class="o">=</span> <span class="kp">nil</span><span class="p">,</span> <span class="n">options</span> <span class="o">=</span> <span class="p">{})</span>
<span class="c1"># env_key = options.fetch(:env_key, 'flipper')</span>
<span class="c1"># app = ->() { [200, { 'Content-Type' => 'text/html' }, ['']] }</span>
<span class="c1"># builder = Rack::Builder.new</span>
<span class="k">yield</span> <span class="n">builder</span> <span class="k">if</span> <span class="nb">block_given?</span>
<span class="c1"># builder.use Rack::Protection</span>
<span class="c1"># builder.use Rack::Protection::AuthenticityToken</span>
<span class="c1"># builder.use Rack::MethodOverride</span>
<span class="c1"># builder.use Flipper::Middleware::SetupEnv, flipper, env_key: env_key</span>
<span class="c1"># builder.use Flipper::Middleware::Memoizer, env_key: env_key</span>
<span class="c1"># builder.use Middleware, env_key: env_key</span>
<span class="c1"># builder.run app</span>
<span class="c1"># klass = self</span>
<span class="c1"># builder.define_singleton_method(:inspect) { klass.inspect } # pretty rake routes output</span>
<span class="c1"># builder</span>
<span class="k">end</span>
</code></pre>
</div>
<p>When provided a block, <code class="highlighter-rouge">Flipper::UI.app</code> invokes the block, yielding the <code class="highlighter-rouge">Rack::Builder</code> instance. This
exposes the internal Rack::Builder instance allowing developers to chain any additional middleware theyâd
like:</p>
<div class="language-ruby highlighter-rouge"><pre class="highlight"><code><span class="no">Rails</span><span class="p">.</span><span class="nf">application</span><span class="p">.</span><span class="nf">routes</span><span class="p">.</span><span class="nf">draw</span> <span class="k">do</span>
<span class="n">mount</span> <span class="no">Flipper</span><span class="o">::</span><span class="no">UI</span><span class="p">.</span><span class="nf">app</span><span class="p">(</span><span class="no">Flipper</span><span class="p">)</span> <span class="p">{</span> <span class="o">|</span><span class="n">builder</span><span class="o">|</span>
<span class="n">builder</span><span class="p">.</span><span class="nf">use</span><span class="p">(</span><span class="no">Middleware</span><span class="p">)</span>
<span class="p">}</span> <span class="o">=></span> <span class="s1">'/flipper'</span>
<span class="k">end</span>
</code></pre>
</div>
<h1 id="use-case-flipper-auditing">Use Case: Flipper Auditing</h1>
<p>At work we needed an audit trail of any changes made through the Flipper UI.</p>
<ul>
<li>
<p>Who made the change?</p>
</li>
<li>
<p>Which feature was enabled?</p>
</li>
<li>
<p>What date were they enabled?</p>
</li>
</ul>
<p>Clicking the <em>Enable</em> button enables a feature for all users:</p>
<p><img src="/assets/middleware-audit/enable-button.png" alt="enable-button" /></p>
<p>In Chromeâs network inspector we see this button makes a:</p>
<ul>
<li>POST request to /flipper/features/<em>feature_name</em>/boolean</li>
<li>Body includes some form data <code class="highlighter-rouge"><span class="p">{</span><span class="w"> </span><span class="err">action:</span><span class="w"> </span><span class="err">enable,</span><span class="w"> </span><span class="err">authenticity_token:</span><span class="w"> </span><span class="err">*****</span><span class="w"> </span><span class="p">}</span></code></li>
<li>Responds with a 302 HTTP status code (redirect)</li>
</ul>
<p><img src="/assets/middleware-audit/enable-zoom.png" alt="enable" /></p>
<p>We also know we have an authenticated user interacting with the UI so can get the user making these requests from the server.
All of this information combined tells us:</p>
<p><strong>Who made the change?</strong> - currently signed in user</p>
<p><strong>Which feature was enabled?</strong> - <em>/features/<strong>optimized_search</strong>/boolean</em></p>
<p><strong>Who was enabled?</strong> - <em>/features/optimized_search/<strong>boolean</strong></em>; <code class="highlighter-rouge"><span class="p">{</span><span class="w"> </span><span class="err">action:</span><span class="w"> </span><span class="err">'Enable'</span><span class="w"> </span><span class="p">}</span></code>, in other words
everyone</p>
<p><strong>302</strong> - tells us this request was succesful</p>
<h1 id="audit-middleware">Audit Middleware</h1>
<p>Letâs implement a simple middleware that simply logs this info.</p>
<div class="language-ruby highlighter-rouge"><pre class="highlight"><code><span class="k">class</span> <span class="nc">FlipperUIMiddleware</span>
<span class="k">def</span> <span class="nf">initialize</span><span class="p">(</span><span class="n">app</span><span class="p">)</span>
<span class="vi">@app</span> <span class="o">=</span> <span class="n">app</span>
<span class="k">end</span>
<span class="k">def</span> <span class="nf">call</span><span class="p">(</span><span class="n">env</span><span class="p">)</span>
<span class="n">status</span><span class="p">,</span> <span class="n">headers</span><span class="p">,</span> <span class="n">body</span> <span class="o">=</span> <span class="vi">@app</span><span class="p">.</span><span class="nf">call</span><span class="p">(</span><span class="n">env</span><span class="p">)</span>
<span class="n">request</span> <span class="o">=</span> <span class="no">Rack</span><span class="o">::</span><span class="no">Request</span><span class="p">.</span><span class="nf">new</span><span class="p">(</span><span class="n">env</span><span class="p">)</span>
<span class="n">user_email</span> <span class="o">=</span> <span class="n">env</span><span class="p">[</span><span class="s1">'warden'</span><span class="p">].</span><span class="nf">user</span><span class="p">.</span><span class="nf">email</span>
<span class="n">message</span> <span class="o">=</span> <span class="nb">format</span><span class="p">(</span><span class="n">user_email</span><span class="p">,</span> <span class="n">request</span><span class="p">,</span> <span class="n">status</span><span class="p">)</span>
<span class="no">Rails</span><span class="p">.</span><span class="nf">logger</span><span class="p">.</span><span class="nf">info</span><span class="p">(</span><span class="n">message</span><span class="p">)</span> <span class="k">unless</span> <span class="n">asset_request?</span><span class="p">(</span><span class="n">request</span><span class="p">.</span><span class="nf">url</span><span class="p">)</span>
<span class="p">[</span><span class="n">status</span><span class="p">,</span> <span class="n">headers</span><span class="p">,</span> <span class="n">body</span><span class="p">]</span>
<span class="k">end</span>
<span class="kp">private</span>
<span class="k">def</span> <span class="nf">asset_request?</span><span class="p">(</span><span class="n">url</span><span class="p">)</span>
<span class="sr">/\.\w+$/</span><span class="p">.</span><span class="nf">match</span><span class="p">(</span><span class="n">url</span><span class="p">)</span>
<span class="k">end</span>
<span class="k">def</span> <span class="nf">format</span><span class="p">(</span><span class="n">user_email</span><span class="p">,</span> <span class="n">request</span><span class="p">,</span> <span class="n">status</span><span class="p">)</span>
<span class="n">msg</span> <span class="o">=</span> <span class="p">{</span>
<span class="ss">email: </span><span class="n">user_email</span><span class="p">,</span>
<span class="ss">method: </span><span class="n">request</span><span class="p">.</span><span class="nf">request_method</span><span class="p">,</span>
<span class="ss">url: </span><span class="n">request</span><span class="p">.</span><span class="nf">url</span><span class="p">,</span>
<span class="ss">params: </span><span class="n">request</span><span class="p">.</span><span class="nf">params</span><span class="p">.</span><span class="nf">except</span><span class="p">(</span><span class="s1">'authenticity_token'</span><span class="p">),</span>
<span class="ss">status: </span><span class="n">status</span>
<span class="p">}</span>
<span class="no">JSON</span><span class="p">.</span><span class="nf">generate</span><span class="p">(</span><span class="n">msg</span><span class="p">)</span>
<span class="k">end</span>
<span class="k">end</span>
</code></pre>
</div>
<p>And make sure <code class="highlighter-rouge">Flipper::UI</code> is using it</p>
<div class="language-ruby highlighter-rouge"><pre class="highlight"><code><span class="no">Rails</span><span class="p">.</span><span class="nf">application</span><span class="p">.</span><span class="nf">routes</span><span class="p">.</span><span class="nf">draw</span> <span class="k">do</span>
<span class="n">mount</span> <span class="no">Flipper</span><span class="o">::</span><span class="no">UI</span><span class="p">.</span><span class="nf">app</span><span class="p">(</span><span class="no">Flipper</span><span class="p">)</span> <span class="p">{</span> <span class="o">|</span><span class="n">builder</span><span class="o">|</span>
<span class="n">builder</span><span class="p">.</span><span class="nf">use</span><span class="p">(</span><span class="no">FlipperUIMiddleware</span><span class="p">)</span>
<span class="p">}</span> <span class="o">=></span> <span class="s1">'/flipper'</span>
<span class="k">end</span>
</code></pre>
</div>
<p>Now all requests made to <em>flipper/</em> will pass their responses through our middleware.</p>
<p>Some notes on the middleware:</p>
<ul>
<li>Weâre using a warden-based authentication framework, Devise, which uses its own middleware to add the current user to the request</li>
<li><code class="highlighter-rouge">Rack::Request</code> just provides a simple interface for interacting with the request.</li>
<li>We only log requests triggered by the user. Since all requests to <em>flipper/</em> will pass through this middleware we need to filter out any requests for asset files (files ending in .css, .js, .png, etc.). /.\w+$/.match(url) takes care of this.</li>
<li>We filter our the authenticity_token, which is there to protect against CSRF attacks.</li>
</ul>
<p>Now in our logs we can easily query for the these messages and get an understanding of whoâs done what within
the Flipper UI.</p>
<p><img src="/assets/middleware-audit/logentries.png" alt="flipper-ui-features" />
<em>Logentriesâ <a href="https://docs.logentries.com/docs/search">advanced LEQL</a></em></p>
<p>I always find it useful connecting the theory with real-world examples. Hopefully this gives you just a bit
more understanding of your Rack applications.</p>RackProcs, Blocks, Lambdas, and Lies2019-04-22T13:36:14+00:002019-04-22T13:36:14+00:00/ruby/2019/04/22/procs-blocks-lambdas-and-lies<p>Github user supersam654 recently opened a great <a href="https://github.com/jnunemaker/flipper/issues/405">issue</a> on the Flipper repo after running into trouble trying to
register a group using Rubyâs <code class="highlighter-rouge">&:method</code> shorthand.</p>
<p><img src="/assets/ruby-arity-symbol-proc/issue.png" alt="issue" /></p>
<p>Groups allow enabling features based on the return value of a block, which is passed the actor
when checking for enabledness.</p>
<h1 id="admin">&:admin?</h1>
<p>When Rubyists want to call a method on every member of an Enumerable we tend to avoid the extra keystrokes
needed to type out the whole block:</p>
<div class="language-ruby highlighter-rouge"><pre class="highlight"><code><span class="p">[</span><span class="s2">"a"</span><span class="p">,</span> <span class="s2">"b"</span><span class="p">,</span> <span class="s2">"c"</span><span class="p">].</span><span class="nf">map</span> <span class="p">{</span> <span class="o">|</span><span class="n">letter</span><span class="o">|</span> <span class="n">letter</span><span class="p">.</span><span class="nf">upcase</span> <span class="p">}</span>
<span class="c1">#=> ["A", "B", "C"]</span>
</code></pre>
</div>
<p>in favor of the more concise:</p>
<div class="language-ruby highlighter-rouge"><pre class="highlight"><code><span class="p">[</span><span class="s2">"a"</span><span class="p">,</span> <span class="s2">"b"</span><span class="p">,</span> <span class="s2">"c"</span><span class="p">].</span><span class="nf">map</span><span class="p">(</span><span class="o">&</span><span class="ss">:upcase</span><span class="p">)</span>
<span class="c1">#=> ["A", "B", "C"]</span>
</code></pre>
</div>
<p>This works with any Enumerable method that yields each member of the collection to the provided block. If
this works with <code class="highlighter-rouge">map</code>, <code class="highlighter-rouge">each</code>, <code class="highlighter-rouge">select</code>, etc. why is supersam654 having trouble using it with
<code class="highlighter-rouge">Flipper.register</code>? To answer this weâre going to have to take a tour of some of the more advanced features
of Ruby. Letâs get started.</p>
<h1 id="registering-groups">Registering Groups</h1>
<p>When a Flipper group is registered it stores the provided block in an instance variable <code class="highlighter-rouge">@block</code>.</p>
<p>Given a simplified version of <code class="highlighter-rouge">register</code>:</p>
<p><em><a href="https://github.com/jnunemaker/flipper/blob/master/lib/flipper.rb#L84">source</a></em></p>
<div class="language-ruby highlighter-rouge"><pre class="highlight"><code><span class="k">def</span> <span class="nf">register</span><span class="p">(</span><span class="nb">name</span><span class="p">,</span> <span class="o">&</span><span class="n">block</span><span class="p">)</span>
<span class="vi">@block</span> <span class="o">=</span> <span class="n">block</span>
<span class="k">end</span>
</code></pre>
</div>
<p>When this code is executed:</p>
<div class="language-ruby highlighter-rouge"><pre class="highlight"><code><span class="no">Flipper</span><span class="p">.</span><span class="nf">register</span><span class="p">(</span><span class="ss">:working</span><span class="p">)</span> <span class="k">do</span> <span class="o">|</span><span class="n">actor</span><span class="o">|</span>
<span class="n">actor</span><span class="p">.</span><span class="nf">admin?</span>
<span class="k">end</span>
</code></pre>
</div>
<p>We end up with a gate instance that has a <code class="highlighter-rouge">match?</code> method locked and loaded, ready to check if a feature is enabled for an actor:</p>
<div class="language-ruby highlighter-rouge"><pre class="highlight"><code> <span class="k">def</span> <span class="nf">register</span><span class="p">(</span><span class="nb">name</span><span class="p">,</span> <span class="o">&</span><span class="n">block</span><span class="p">)</span>
<span class="vi">@block</span> <span class="o">=</span> <span class="n">block</span> <span class="c1"># { |actor| actor.admin? }</span>
<span class="k">end</span>
<span class="k">def</span> <span class="nf">match?</span><span class="p">(</span><span class="n">thing</span><span class="p">,</span> <span class="n">context</span><span class="p">)</span>
<span class="k">if</span> <span class="vi">@block</span><span class="p">.</span><span class="nf">arity</span> <span class="o">==</span> <span class="mi">1</span> <span class="c1"># @block = { |actor| actor.admin? }</span>
<span class="vi">@block</span><span class="p">.</span><span class="nf">call</span><span class="p">(</span><span class="n">thing</span><span class="p">)</span>
<span class="k">else</span>
<span class="vi">@block</span><span class="p">.</span><span class="nf">call</span><span class="p">(</span><span class="n">thing</span><span class="p">,</span> <span class="n">context</span><span class="p">)</span>
<span class="k">end</span>
<span class="k">end</span>
</code></pre>
</div>
<p>When checking if a feature is enabled for a given user, <code class="highlighter-rouge">match?</code> eventually gets called with the user and optionally the context if your provided block takes two arguments . The feature
is enabled for the user if the result of the block is true.</p>
<h1 id="arity">Arity</h1>
<div class="language-ruby highlighter-rouge"><pre class="highlight"><code><span class="k">if</span> <span class="vi">@block</span><span class="p">.</span><span class="nf">arity</span> <span class="o">==</span> <span class="mi">1</span> <span class="c1"># @block = { |actor| actor.admin? }</span>
<span class="vi">@block</span><span class="p">.</span><span class="nf">call</span><span class="p">(</span><span class="n">thing</span><span class="p">)</span>
<span class="k">else</span>
<span class="vi">@block</span><span class="p">.</span><span class="nf">call</span><span class="p">(</span><span class="n">thing</span><span class="p">,</span> <span class="n">context</span><span class="p">)</span>
<span class="k">end</span>
</code></pre>
</div>
<p>Arity is the number of arguments a function takes (<em>Thanks <a href="https://en.wikipedia.org/wiki/Arity">Wikipedia!</a>,
and screw you high school teacher who said Wikipedia was not a credible source for my essays!</em>).</p>
<p>Ruby, with
its powerful metaprogramming abilities lets us introspect any object weâd like <em>including</em> method objects.
Who said Ruby methods arenât first class citizens? To figure out a methodâs arity we donât need
any special wizardry we just need to politely ask it.</p>
<div class="language-ruby highlighter-rouge"><pre class="highlight"><code><span class="k">def</span> <span class="nf">greet</span><span class="p">(</span><span class="n">person</span><span class="p">)</span>
<span class="s2">"Nice to meet ya </span><span class="si">#{</span><span class="n">person</span><span class="p">.</span><span class="nf">name</span><span class="si">}</span><span class="s2">"</span>
<span class="k">end</span>
<span class="nb">method</span><span class="p">(</span><span class="ss">:greet</span><span class="p">).</span><span class="nf">arity</span>
<span class="c1"># => 1</span>
<span class="k">def</span> <span class="nf">introduce</span><span class="p">(</span><span class="n">person1</span><span class="p">,</span> <span class="n">person2</span><span class="p">)</span>
<span class="s2">"</span><span class="si">#{</span><span class="n">person1</span><span class="p">.</span><span class="nf">name</span><span class="si">}</span><span class="s2">, I'd like to introduce you to </span><span class="si">#{</span><span class="n">person2</span><span class="p">.</span><span class="nf">name</span><span class="si">}</span><span class="s2">"</span>
<span class="k">end</span>
<span class="nb">method</span><span class="p">(</span><span class="ss">:introduce</span><span class="p">).</span><span class="nf">arity</span>
<span class="c1"># => 2</span>
</code></pre>
</div>
<p>When a method takes a variable number of arguments <code class="highlighter-rouge">arity</code> returns <em>-n-1</em>, where <em>n</em> is the number of required arguments.</p>
<div class="language-ruby highlighter-rouge"><pre class="highlight"><code><span class="k">def</span> <span class="nf">call</span><span class="p">(</span><span class="o">*</span><span class="n">people</span><span class="p">)</span>
<span class="n">people</span><span class="p">.</span><span class="nf">each</span><span class="p">(</span><span class="o">&</span><span class="ss">:call</span><span class="p">)</span>
<span class="k">end</span>
<span class="nb">method</span><span class="p">(</span><span class="ss">:call</span><span class="p">).</span><span class="nf">arity</span>
<span class="c1"># => -1</span>
</code></pre>
</div>
<p>Just like methods, procs and lambdas can also be asked their arity.</p>
<div class="language-ruby highlighter-rouge"><pre class="highlight"><code><span class="nb">proc</span> <span class="p">{</span> <span class="o">|</span><span class="n">arg</span><span class="o">|</span> <span class="p">}.</span><span class="nf">arity</span>
<span class="c1"># => 1</span>
<span class="o">-></span><span class="p">(</span><span class="n">arg</span><span class="p">)</span> <span class="p">{</span> <span class="p">}.</span><span class="nf">arity</span>
<span class="c1"># => 1</span>
<span class="nb">lambda</span> <span class="p">{</span><span class="o">|*</span><span class="n">args</span><span class="o">|</span> <span class="p">}.</span><span class="nf">arity</span>
<span class="c1"># => -1</span>
</code></pre>
</div>
<h1 id="lies-lies-and-more-lies">Lies, Lies, and More Lies</h1>
<p>I have to apologize for telling a slight lie earlier. The register method doesnât just store the block in an
instance variable <code class="highlighter-rouge">@block</code>. When <code class="highlighter-rouge">&</code> is prepended to a parameter like <code class="highlighter-rouge">&block</code> Ruby converts the provided block
to a proc, which allows us to reference it by dropping the ampersand. You can name this whatever you want e.g. <code class="highlighter-rouge">&foo</code>. If passed an argument
prepended with <code class="highlighter-rouge">&</code> instead of a block, it converts the argument to a proc by calling <code class="highlighter-rouge">to_proc</code> on it.</p>
<div class="language-ruby highlighter-rouge"><pre class="highlight"><code><span class="k">def</span> <span class="nf">register</span><span class="p">(</span><span class="nb">name</span><span class="p">,</span> <span class="o">&</span><span class="n">block</span><span class="p">)</span>
<span class="c1"># Before giving us a refernce to block</span>
<span class="c1"># Ruby implicitly calls: block = block.to_proc</span>
<span class="vi">@block</span> <span class="o">=</span> <span class="n">block</span>
<span class="k">end</span>
</code></pre>
</div>
<p>When <code class="highlighter-rouge">register</code> is passed:</p>
<div class="language-ruby highlighter-rouge"><pre class="highlight"><code><span class="n">register</span><span class="p">(</span><span class="ss">:admin</span><span class="p">,</span> <span class="o">&</span><span class="ss">:admin?</span><span class="p">)</span>
</code></pre>
</div>
<p>The following happens:</p>
<div class="language-ruby highlighter-rouge"><pre class="highlight"><code><span class="k">def</span> <span class="nf">register</span><span class="p">(</span><span class="nb">name</span><span class="p">,</span> <span class="o">&</span><span class="ss">:admin?</span><span class="p">)</span>
<span class="c1"># block = :admin.to_proc - Ruby calls this under the hood for us</span>
<span class="vi">@block</span> <span class="o">=</span> <span class="n">block</span>
<span class="k">end</span>
</code></pre>
</div>
<p>And what would happen if we checked this <em>procâs</em> arity?</p>
<div class="language-ruby highlighter-rouge"><pre class="highlight"><code><span class="k">def</span> <span class="nf">register</span><span class="p">(</span><span class="nb">name</span><span class="p">,</span> <span class="o">&</span><span class="n">block</span><span class="p">)</span>
<span class="c1"># block = :admin.to_proc - Ruby calls this under the hood for us</span>
<span class="vi">@block</span> <span class="o">=</span> <span class="n">block</span>
<span class="nb">puts</span> <span class="vi">@block</span><span class="p">.</span><span class="nf">arity</span>
<span class="k">end</span>
</code></pre>
</div>
<div class="language-ruby highlighter-rouge"><pre class="highlight"><code><span class="n">register</span><span class="p">(</span><span class="ss">:admin</span><span class="p">,</span> <span class="o">&</span><span class="ss">:admin?</span><span class="p">)</span>
<span class="c1"># => -1</span>
<span class="n">register</span><span class="p">(</span><span class="ss">:admin</span><span class="p">)</span> <span class="p">{</span> <span class="o">|</span><span class="n">actor</span><span class="o">|</span> <span class="n">actor</span><span class="p">.</span><span class="nf">admin?</span> <span class="p">}</span>
<span class="c1"># => 1</span>
</code></pre>
</div>
<p>Hold up. With the block the correct arity is returned, but with <code class="highlighter-rouge">&:admin?</code> -1 is returned! This explains why
<code class="highlighter-rouge">Flipper.register</code> is working with blocks, but not the <code class="highlighter-rouge">&:admin?</code> shortcut.</p>
<div class="language-ruby highlighter-rouge"><pre class="highlight"><code><span class="k">def</span> <span class="nf">match?</span><span class="p">(</span><span class="n">thing</span><span class="p">,</span> <span class="n">context</span><span class="p">)</span>
<span class="k">if</span> <span class="vi">@block</span><span class="p">.</span><span class="nf">arity</span> <span class="o">==</span> <span class="mi">1</span>
<span class="vi">@block</span><span class="p">.</span><span class="nf">call</span><span class="p">(</span><span class="n">thing</span><span class="p">)</span> <span class="c1"># This path gets called with block</span>
<span class="k">else</span>
<span class="vi">@block</span><span class="p">.</span><span class="nf">call</span><span class="p">(</span><span class="n">thing</span><span class="p">,</span> <span class="n">context</span><span class="p">)</span> <span class="c1"># This path gets called with &:admin and raises an exception</span>
<span class="k">end</span>
<span class="k">end</span>
</code></pre>
</div>
<p>To figure out what exactly is happening here weâll need to keep on digging.</p>
<h1 id="i-thought-the-following-did-the-exact-same-thing">I thought the following did the exact same thingâŠ</h1>
<div class="language-ruby highlighter-rouge"><pre class="highlight"><code><span class="p">[</span><span class="s2">"a"</span><span class="p">,</span> <span class="s2">"b"</span><span class="p">,</span> <span class="s2">"c"</span><span class="p">].</span><span class="nf">map</span> <span class="p">{</span> <span class="o">|</span><span class="n">letter</span><span class="o">|</span> <span class="n">letter</span><span class="p">.</span><span class="nf">upcase</span> <span class="p">}</span>
</code></pre>
</div>
<div class="language-ruby highlighter-rouge"><pre class="highlight"><code><span class="p">[</span><span class="s2">"a"</span><span class="p">,</span> <span class="s2">"b"</span><span class="p">,</span> <span class="s2">"c"</span><span class="p">].</span><span class="nf">map</span><span class="p">(</span><span class="o">&</span><span class="ss">:upcase</span><span class="p">)</span>
</code></pre>
</div>
<p>Almost, but not quite. This is because <code class="highlighter-rouge">Symbol</code> implements its own version of <a href="https://ruby-doc.org/core-2.5.0/Symbol.html#method-i-to_proc">to_proc</a>.</p>
<p><em>simplified version of <code class="highlighter-rouge">Symbol#to_proc</code>. The real implementation is now written in C</em></p>
<div class="language-ruby highlighter-rouge"><pre class="highlight"><code><span class="k">class</span> <span class="nc">Symbol</span>
<span class="k">def</span> <span class="nf">to_proc</span>
<span class="nb">proc</span> <span class="p">{</span> <span class="o">|</span><span class="n">arg</span><span class="o">|</span> <span class="n">arg</span><span class="p">.</span><span class="nf">send</span><span class="p">(</span><span class="nb">self</span><span class="p">)</span> <span class="p">}</span>
<span class="k">end</span>
<span class="k">end</span>
</code></pre>
</div>
<p>If we had a symbol such as <code class="highlighter-rouge">:upcase</code>,
when invoked:</p>
<div class="language-ruby highlighter-rouge"><pre class="highlight"><code><span class="p">[</span><span class="s2">"a"</span><span class="p">,</span> <span class="s2">"b"</span><span class="p">,</span> <span class="s2">"c"</span><span class="p">].</span><span class="nf">map</span> <span class="p">{</span> <span class="o">|</span><span class="n">letter</span><span class="o">|</span> <span class="n">letter</span><span class="p">.</span><span class="nf">upcase</span> <span class="p">}</span>
<span class="p">[</span><span class="s2">"a"</span><span class="p">,</span> <span class="s2">"b"</span><span class="p">,</span> <span class="s2">"c"</span><span class="p">].</span><span class="nf">map</span><span class="p">(</span><span class="o">&</span><span class="ss">:upcase</span><span class="p">)</span> <span class="c1"># same as the shorthand</span>
</code></pre>
</div>
<p>we could replace these variables to see what it would actually look like:</p>
<div class="language-ruby highlighter-rouge"><pre class="highlight"><code><span class="k">def</span> <span class="nf">to_proc</span>
<span class="nb">proc</span> <span class="p">{</span> <span class="o">|</span><span class="n">arg</span><span class="o">|</span> <span class="n">arg</span><span class="p">.</span><span class="nf">send</span><span class="p">(</span><span class="nb">self</span><span class="p">)</span>
<span class="c1"># proc { |letter| letter.send(:upcase) }</span>
<span class="k">end</span>
</code></pre>
</div>
<p>Now we understand what this <code class="highlighter-rouge">&:upcase</code> business is all about, but that still doesnât explain why the two
examples above arenât actually the same. Thatâs beacuse Iâve lied done it again! Another lie! The real definition of
<code class="highlighter-rouge">Symbol#to_proc</code> looks more like this:</p>
<div class="language-ruby highlighter-rouge"><pre class="highlight"><code><span class="k">class</span> <span class="nc">Symbol</span>
<span class="k">def</span> <span class="nf">to_proc</span>
<span class="nb">proc</span> <span class="k">do</span> <span class="o">|*</span><span class="n">args</span><span class="o">|</span>
<span class="n">receiver</span><span class="p">,</span> <span class="o">*</span><span class="n">rest</span> <span class="o">=</span> <span class="n">args</span>
<span class="k">if</span> <span class="n">rest</span><span class="p">.</span><span class="nf">nil?</span>
<span class="n">receiver</span><span class="p">.</span><span class="nf">send</span><span class="p">(</span><span class="nb">self</span><span class="p">)</span>
<span class="k">else</span>
<span class="n">receiver</span><span class="p">.</span><span class="nf">send</span><span class="p">(</span><span class="nb">self</span><span class="p">,</span> <span class="o">*</span><span class="n">rest</span><span class="p">)</span>
<span class="k">end</span>
<span class="k">end</span>
<span class="k">end</span>
<span class="k">end</span>
</code></pre>
</div>
<p>As seen in our simplified example above <code class="highlighter-rouge">Symbol#to_proc</code> <strong>always</strong> returns a proc that takes a variable number of
arguments, while blocks converted to procs via <code class="highlighter-rouge">&block</code> return procs that take the exact same number of
arguments as the block:</p>
<div class="language-ruby highlighter-rouge"><pre class="highlight"><code><span class="k">def</span> <span class="nf">arity</span><span class="p">(</span><span class="o">&</span><span class="n">block</span><span class="p">)</span>
<span class="n">block</span><span class="p">.</span><span class="nf">arity</span>
<span class="k">end</span>
<span class="n">arity</span><span class="p">()</span> <span class="p">{</span> <span class="o">|</span><span class="n">user</span><span class="o">|</span> <span class="n">user</span><span class="p">.</span><span class="nf">admin?</span> <span class="p">}</span>
<span class="c1"># => 1</span>
<span class="n">arity</span><span class="p">(</span><span class="o">&</span><span class="ss">:to_s</span><span class="p">)</span>
<span class="c1"># => -1</span>
</code></pre>
</div>
<p>This means that in the <code class="highlighter-rouge">&:admin?</code> case we end up in the else condition and pass arguments to
<code class="highlighter-rouge">admin?</code>, which expects 0. As always the computer does exactly what we tell it to do and not what we want it
to. The computer wins and an exception is raised.</p>
<div class="language-ruby highlighter-rouge"><pre class="highlight"><code><span class="k">def</span> <span class="nf">match?</span><span class="p">(</span><span class="n">thing</span><span class="p">,</span> <span class="n">context</span><span class="p">)</span>
<span class="k">if</span> <span class="vi">@block</span><span class="p">.</span><span class="nf">arity</span> <span class="o">==</span> <span class="mi">1</span> <span class="c1"># @block = { |actor| actor.admin? }</span>
<span class="vi">@block</span><span class="p">.</span><span class="nf">call</span><span class="p">(</span><span class="n">thing</span><span class="p">)</span>
<span class="k">else</span>
<span class="vi">@block</span><span class="p">.</span><span class="nf">call</span><span class="p">(</span><span class="n">thing</span><span class="p">,</span> <span class="n">context</span><span class="p">)</span> <span class="c1"># this path gets called</span>
<span class="k">end</span>
<span class="k">end</span>
</code></pre>
</div>
<h1 id="ruby-friends-to-the-rescue">Ruby friends to the rescue</h1>
<p>Tim, being the legend that he is opened a PR using Samâs (also total legend - both of whom Iâve never actually met, but Iâm sure theyâre legends) proposed idea to take the absolute value of the arity in match.</p>
<p><img src="/assets/ruby-arity-symbol-proc/abs-arity.png" alt="arity" /></p>
<p><img src="/assets/ruby-arity-symbol-proc/pr.png" alt="pr" /></p>
<p>Since this has been merged it should be going out in the next release after 0.16.2. As always check out the
<a href="https://github.com/jnunemaker/flipper/blob/master/Changelog.md">Changelog</a> to be sure.</p>
<h1 id="follow-up">Follow Up</h1>
<p>If you thought this was a fun dive into some parts of Ruby you may not have explored I encourage you to
experiment and implement your own to_proc method. Think of some ways you can make it more powerful. Iâll get
you started:</p>
<div class="language-ruby highlighter-rouge"><pre class="highlight"><code><span class="k">class</span> <span class="nc">String</span>
<span class="k">def</span> <span class="nf">to_proc</span>
<span class="nb">proc</span> <span class="p">{</span> <span class="o">|</span><span class="n">arg</span><span class="o">|</span> <span class="n">arg</span><span class="p">.</span><span class="nf">send</span><span class="p">(</span><span class="nb">self</span><span class="p">)</span> <span class="p">}</span>
<span class="k">end</span>
<span class="k">end</span>
<span class="p">[</span><span class="s2">"a"</span><span class="p">,</span> <span class="s2">"b"</span><span class="p">,</span> <span class="s2">"c"</span><span class="p">].</span><span class="nf">map</span><span class="p">(</span><span class="o">&</span><span class="s2">"upcase"</span><span class="p">)</span>
</code></pre>
</div>Github user supersam654 recently opened a great issue on the Flipper repo after running into trouble trying to register a group using Rubyâs &:method shorthand.Advanced Ag2018-08-02T13:36:14+00:002018-08-02T13:36:14+00:00/command-line/2018/08/02/ag<p>I want to talk about one of the more powerful tools in my toolbox - <a href="https://github.com/ggreer/the_silver_searcher">Ag</a>. As a developer Iâll bet you spend a fair amount of time writing code, but Iâd argue you spend even more time reading code. Having a solid understanding of your languageâs standard library, or a frameworkâs core abstractions will no doubt come in handy, but if youâre working on anything larger than a <em>Hello World</em> program chances are you spend a lot of time digging through code - lots of code. This is where a code searching tool like Ag proves indespensible. Yes, Ag can do your basic keyword searches, but so can your favorite editor. I want to share some of the more advanced options I find myself using day-to-day to shine a light on just how powerful of a tool it can be.</p>
<p><em>All examples will use <a href="https://github.com/jnunemaker/flipper">Flipper</a> as an example codebase we can search through. Feel free to <code class="highlighter-rouge">git clone https://github.com/jnunemaker/flipper.git && cd flipper</code> to follow along!</em></p>
<h2 id="why-ag">Why Ag?</h2>
<ul>
<li>fastâŠreally fast (orders of magnitude faster than grep/ack)</li>
<li>version control system awareness - ignores file patterns in your .gitignore</li>
<li>ignores file patterns added to .ignore (eg .min.js)</li>
</ul>
<h2 id="install">Install</h2>
<p><code class="highlighter-rouge">brew install ag</code></p>
<h1 id="basic-searching">Basic Searching</h1>
<p><code class="highlighter-rouge">ag [file-type] [options] PATTERN [PATH]</code></p>
<p>With one argument Ag recursively searches the current directory for the given pattern.</p>
<p><code class="highlighter-rouge">ag Gate</code></p>
<p><img src="/assets/searching/ag.png" alt="ag" /></p>
<p>Pass it a second argument to recursively search a directory. This becomes especially useful when working in an opinionated framework like Rails where I know all of my controller code lives under <em>app/controllers</em>, specs under <em>spec/controllers</em>, etc.</p>
<p><code class="highlighter-rouge">ag Gate spec</code></p>
<p><img src="/assets/searching/ag-directory.png" alt="ag" /></p>
<h1 id="whole-word">Whole Word</h1>
<p>Often weâre searching for a pattern that appears in many different contexts within our app. Usually this is a core domain concept like User, Property, or Gate. We only want to hit on cases where this pattern is a whole word - meaning not a subset of a larger word.</p>
<p><code class="highlighter-rouge">ag Gate</code></p>
<p><em>Note how <code class="highlighter-rouge">Gate</code> is matched, but so is <code class="highlighter-rouge">Flipper::GateNotFound</code> since it contains <code class="highlighter-rouge">Gate</code>.</em></p>
<p><img src="/assets/searching/ag.png" alt="ag" /></p>
<p><code class="highlighter-rouge">ag -w Gate</code></p>
<p><img src="/assets/searching/whole-word.png" alt="ag" /></p>
<h1 id="file-type">File Type</h1>
<p><code class="highlighter-rouge">ag --ruby User</code></p>
<p><code class="highlighter-rouge">ag --js console</code></p>
<p>Restrict search to specific file-type. This is really useful if you have, for example, Ruby and JavaScript
services that share the same name.</p>
<p>To see all supported file-types: <code class="highlighter-rouge">ag --list-file-types</code>.</p>
<h1 id="list">List</h1>
<p>Ag returns the filename and line matching PATTERN. Sometimes we just want to list the filenames.</p>
<p><code class="highlighter-rouge">ag Gate -l</code></p>
<p><img src="/assets/searching/list.png" alt="ag" /></p>
<p><code class="highlighter-rouge">ag Gate -l | sort</code> - <em>pro tip: who doesnât prefer a sorted list?</em></p>
<p><img src="/assets/searching/ag-list-sort.png" alt="ag" /></p>
<h1 id="file-pattern">File Pattern</h1>
<p>Limit search to file names (including path) that contain a pattern:</p>
<p><code class="highlighter-rouge">ag User -G controllers</code></p>
<p><strong>Usecase:</strong></p>
<p>Your Rails and JavaScript admin panel code are both namespaced under an <em>admin</em> directory and you want to search both:</p>
<p><code class="highlighter-rouge">ag User -G /admin/</code></p>
<h1 id="ignore-directory">Ignore Directory</h1>
<p><code class="highlighter-rouge">ag UserCreator --ignore-dir spec/</code></p>
<p><code class="highlighter-rouge">--ignore-dir</code> can be used to search everywhere, <em>except</em> a given directory. Multiple directories can be ignored by repeating the option.</p>
<p><code class="highlighter-rouge">ag User --ignore-dir spec/ --ignore-dir config/</code>.</p>
<h1 id="piping">Piping</h1>
<p>Ag reads from Standard Input, allowing us to pipe results into the filter.</p>
<p><strong>Usecase 1:</strong> searching rails routes</p>
<p>In rails we can get an overview of all defined routes by running the <code class="highlighter-rouge">rake routes</code> task. With thousands of routes and not enough coffee left in my mug this would take way too long to search through by hand.</p>
<p><code class="highlighter-rouge">rake routes</code></p>
<p><img src="/assets/searching/rake-routes.png" alt="ag" /></p>
<p><code class="highlighter-rouge">rake routes | ag sign_in</code></p>
<p><img src="/assets/searching/rake-routes-ag.png" alt="ag" /></p>
<p><strong>Usecase 2:</strong> filtering logs</p>
<p>Have you ever tried to figure out which controllers actions are handling requests on page load? Rails logs every request a controller processes. With Ag we can easily filter for these messages:</p>
<p><code class="highlighter-rouge">tail -f log/development.log</code></p>
<p><img src="/assets/searching/tail.png" alt="ag" /></p>
<p><code class="highlighter-rouge">tail -f log/development.log | ag "Processing by"</code></p>
<p><img src="/assets/searching/tail-ag.png" alt="ag" /></p>
<p><strong>Usecase 3:</strong> killing processes</p>
<p>Every once in a while Iâll try to shut down an irb session by sending a SIGINT via <em>ctrl-c</em>, only to have the process fail to terminate. When a process wants to play these kind of games thereâs only one option - SIGTERM baby!</p>
<p>Get the PID by listing running processes with <code class="highlighter-rouge">ps -ax</code></p>
<p><img src="/assets/searching/ps.png" alt="ag" /></p>
<p>Filter the results for any irb processes <code class="highlighter-rouge">ps -ax | ag irb</code></p>
<p><img src="/assets/searching/irb.png" alt="ag" /></p>
<p><code class="highlighter-rouge">kill 78154</code></p>
<h1 id="regex">Regex</h1>
<p><code class="highlighter-rouge">ag [file-type] [options] PATTERN [PATH]</code></p>
<p>Weâve mostly seen basic words as the PATTERN parameter. It turns out PATTERN supports Perl-style regular expressions making Agâs searching capablities limitless.</p>
<p><strong>Usecase:</strong> very specific searches</p>
<p>Letâs search for every reference to the <code class="highlighter-rouge">Feature</code> class in files ending in <em>adapter.rb</em> or <em>store.rb</em>.</p>
<p><code class="highlighter-rouge">ag -w -G "(adapter|store).rb$"</code></p>
<p><img src="/assets/searching/regex.png" alt="ag" /></p>
<h1 id="context">Context</h1>
<p><code class="highlighter-rouge">man ag | ag -C case</code></p>
<p>Ag will display the line matching PATTERN, but sometimes we need some more context. For this we can pass the context (-C) flag to print the 2 lines above and below each match. Note the difference between the above command and the result of <code class="highlighter-rouge">man ag | ag case</code>.</p>
<h1 id="eof">EOF</h1>
<p>This really just scratches the surface of the powers of Ag. To learn more be sure to take a read through the man pages (<code class="highlighter-rouge">man ag</code>) in your shell of choice.</p>
<p>Happy <del>hacking</del> Ag-ing!</p>I want to talk about one of the more powerful tools in my toolbox - Ag. As a developer Iâll bet you spend a fair amount of time writing code, but Iâd argue you spend even more time reading code. Having a solid understanding of your languageâs standard library, or a frameworkâs core abstractions will no doubt come in handy, but if youâre working on anything larger than a Hello World program chances are you spend a lot of time digging through code - lots of code. This is where a code searching tool like Ag proves indespensible. Yes, Ag can do your basic keyword searches, but so can your favorite editor. I want to share some of the more advanced options I find myself using day-to-day to shine a light on just how powerful of a tool it can be.Default Scope2018-07-22T13:36:14+00:002018-07-22T13:36:14+00:00/rails/2018/07/22/default-scope<p>ActiveRecordâs default_scope can be a handy feature for the lazy programmer (A flag I proudly wave). Rails developers can leverage it
to add a default condition that will apply to all queries accessing a given model. Letâs say youâre
building a news site. Every time you query for an article you only want to return published articles.
Instead of explicitly filtering by published articles with every query:</p>
<div class="language-ruby highlighter-rouge"><pre class="highlight"><code> <span class="no">Article</span><span class="p">.</span><span class="nf">where</span><span class="p">(</span><span class="ss">published: </span><span class="kp">true</span><span class="p">).</span><span class="nf">find</span><span class="p">(</span><span class="mi">1</span><span class="p">)</span>
</code></pre>
</div>
<p>You can define a default_scope on the <code class="highlighter-rouge">Article</code> model to always filter by published articles:</p>
<div class="language-ruby highlighter-rouge"><pre class="highlight"><code><span class="k">class</span> <span class="nc">Article</span> <span class="o"><</span> <span class="no">ActiveRecord</span><span class="o">::</span><span class="no">Base</span>
<span class="n">default_scope</span> <span class="n">where</span><span class="p">(</span><span class="ss">:published</span> <span class="o">=></span> <span class="kp">true</span><span class="p">)</span>
<span class="k">end</span>
</code></pre>
</div>
<p>Now any query accessing Article will have the filter applied:</p>
<div class="language-ruby highlighter-rouge"><pre class="highlight"><code> <span class="no">Article</span><span class="p">.</span><span class="nf">find</span><span class="p">(</span><span class="mi">1</span><span class="p">)</span>
</code></pre>
</div>
<div class="language-sql highlighter-rouge"><pre class="highlight"><code><span class="k">SELECT</span> <span class="n">articles</span><span class="p">.</span><span class="o">*</span> <span class="k">FROM</span> <span class="n">articles</span> <span class="k">WHERE</span> <span class="n">articles</span><span class="p">.</span><span class="n">published</span> <span class="o">=</span> <span class="k">true</span>
</code></pre>
</div>
<p>While Iâm not a huge fan of default_scope, it is a useful ActiveRecord feature that youâre likely to come
across if you spend your days in Rails projects.</p>
<p>Myself and the rest of the App Platform team spent the last few months working on some major schema
changes to the VTS database to maximize our ability to represent the many types of relationships that can
exist between an owner and an asset. As part of this work we updated the relationship between Users
and Accounts. Instead of a many-to-many relationship which allows a user to exist on multiple accounts, the
new data model actually simplifies this relationship by ensuring every user is only on one account. This
meant getting rid of the idea of an <code class="highlighter-rouge">AccountUser</code> in the application by:</p>
<ol>
<li>
<p>removing the account_users junction table in favor of a one-to-many relationship now described with a
users.account_id foreign key</p>
</li>
<li>
<p>removing any references to account_users</p>
</li>
</ol>
<p>A schema + data migration solves the first issue, and the second issue comes down to deleting code and who
doesnât enjoy that?</p>
<p>As I said before default scopes can help the lazy programmer, but at VTS they have also served another purpose - security. With some of the largest financial firms in the world over managing over 7 billion sqft on the platform you can bet <a href="https://www.vts.com/blog/uncategorized/vts-security-update-soc-2-eu-us-privacy-shield">we take security seriously</a>, both
at the operations and application layers of the business. We have a dedicated security team to handle things
like our annual SOC2 audit, EU-U.S. Privacy Shield, etc, but at the end of the day security is everyoneâs
responsibility. Many in the industry describe security best practices as an onion, with many layers of
protection stacked on top of each other. The reality is that thereâs no such thing as a 100% secure
application so when one layer inevitably fails you want to be sure you have another right behind it that has
your back. In our case this extra layer was implemented with default_scope.</p>
<p>We have come to really enjoy using [https://github.com/varvet/pundit] as our authorization system to define
and check authorization of various resources. This means that before a resource is served we pass it through
Pundit to make sure the given user or destination has access to the object. For example before showing
the currently signed in user a given Property, we have to make sure this user does indeed have access to this
Property</p>ActiveRecordâs default_scope can be a handy feature for the lazy programmer (A flag I proudly wave). Rails developers can leverage it to add a default condition that will apply to all queries accessing a given model. Letâs say youâre building a news site. Every time you query for an article you only want to return published articles. Instead of explicitly filtering by published articles with every query:Monorepos2017-09-16T13:36:14+00:002017-09-16T13:36:14+00:00/ruby/git/2017/09/16/monorepos<p>Like many technology companies we happily use Git to decentralize our
development workflow. Browsing through the VTS organization on Github youâll
notice a similar trend. Each of our repositories tracks one logical project.
This works really well for us and countless other companies around the world.
However, just because this is the optimal way to organize our projects, doesnât
mean its the best approach for every project. The alternative to this approach
is the monorepo.</p>
<p>Flipper is a Ruby gem we use extensively at VTS to turn features on or off for
a set of users. Like many of our projects, Flipper chose to use Git as a key
part of its development workflow, but unlike VTS, its repository is organized
as a monorepo. The rest this post weâll walk through a real-world issue opened
on the Flipper repo to answer two important questionsâââWhat is a monorepo, and
when would I use one?</p>
<h2 id="two-weeks-earlier">Two Weeks EarlierâŠ</h2>
<p><em>Thursday, 09-14-2017, 9am</em></p>
<p>With a warm cup of coffee and visions of glory, Orbyt is ready to use Flipper
for the very first time. Following a provided example, he gem installs Flipper
and smiles to himself as the all too familiar characters stream down the
monitor in front of him.</p>
<p><code class="highlighter-rouge">1 gem installed</code></p>
<p>Its time. Orbyt initializes a new Flipper instance using the ActiveRecord
adapter:</p>
<div class="language-ruby highlighter-rouge"><pre class="highlight"><code><span class="nb">require</span> <span class="s1">'flipper/adapters/active_record'</span>
<span class="n">adapter</span> <span class="o">=</span> <span class="no">Flipper</span><span class="o">::</span><span class="no">Adapters</span><span class="o">::</span><span class="no">ActiveRecord</span><span class="p">.</span><span class="nf">new</span>
<span class="n">flipper</span> <span class="o">=</span> <span class="no">Flipper</span><span class="p">.</span><span class="nf">new</span><span class="p">(</span><span class="n">adapter</span><span class="p">)</span>
</code></pre>
</div>
<p><code class="highlighter-rouge">uninitialized constant Flipper::Adapters::ActiveRecord</code></p>
<p>The dreaded uninitialized constant. As much as weâd like to believe the <code class="highlighter-rouge">Flipper::Adapters::ActiveRecord</code>
class is defined, the Ruby interpreter insists otherwise - and Ruby is always right.</p>
<p>At this point in the story Orbyt opens a really great issue on the Flipper
repo, which inspired me to write this post. How can it be that this adapter
class is undefined?</p>
<p>Browsing through the <a href="https://github.com/jnunemaker/flipper">Flipper</a> repo you see it right there,
<a href="https://github.com/jnunemaker/flipper/blob/master/lib/flipper/adapters/active_record.rb">Flipper::Adapters::ActiveRecord</a>.</p>
<p><img src="/assets/monorepos/active_record_full.png" alt="active_record" /></p>
<p>Weâve all been here. Youâre installing unfamiliar software that you expect to
work, but for whatever reason it isnât. Is this library broken? Is my computer broken? What day is it?</p>
<h2 id="down-the-rabbit-hole">Down the Rabbit Hole</h2>
<p>A great first step in this case is to locate and open the installed library to see the actual source thatâs
running. Ruby developers can do this with <code class="highlighter-rouge">bundle show gemname</code>, where <em>gemname</em> is the name of the gem
youâd like to locate.</p>
<p><em>note: Iâm using RVM for managing my ruby versions, your path might look a little different</em></p>
<p><code class="highlighter-rouge">bundle show flipper</code></p>
<p><code class="highlighter-rouge">=> /Users/Alex/.rvm/gems/ruby-2.3.3/gems/flipper-0.10.2</code></p>
<p>Change into the flipper directory</p>
<p><code class="highlighter-rouge">cd /Users/Alex/.rvm/gems/ruby-2.3.3/gems/flipper-0.10.2</code></p>
<p>and list the adapters:</p>
<p><img src="/assets/monorepos/local_adapters.png" alt="local_adapters" /></p>
<p>We see 7 ruby files corresponding to 7 adapters. Letâs compare this with the adapters in the Flipper
repo on GitHub:</p>
<p><img src="/assets/monorepos/remote_adapters.png" alt="remote_adapters" /></p>
<p>The Flipper we downloaded only has 7 adapters, none of which define <code class="highlighter-rouge">Flipper::Adapters::ActiveRecord</code>, while
the Github repo lists 14.</p>
<h2 id="enter-monorepo">Enter Monorepo</h2>
<p>Flipper uses whatâs known as a monorepo - a single git repository to track multiple projects. This
approach works really well in Flipperâs case. Flipper has a core API that calls into
<a href="https://github.com/jnunemaker/flipper/blob/master/docs/Adapters.md">adapters</a> when it needs to interact with
a datastore. Each adapter implements the same interface so they are completely interchangeable.</p>
<ul>
<li>
<p><code class="highlighter-rouge">features</code> - Get the set of known features.</p>
</li>
<li>
<p><code class="highlighter-rouge">add(feature)</code> - Add a feature to the set of known features.</p>
</li>
<li>
<p><code class="highlighter-rouge">remove(feature)</code> - Remove a feature from the set of known features.</p>
</li>
<li>
<p><code class="highlighter-rouge">clear(feature)</code> - Clear all gate values for a feature.</p>
</li>
<li>
<p><code class="highlighter-rouge">get(feature)</code> - Get all gate values for a feature.</p>
</li>
<li>
<p><code class="highlighter-rouge">enable(feature,</code> gate, thing) - Enable a gate for a thing.</p>
</li>
<li>
<p><code class="highlighter-rouge">disable(feature,</code> gate, thing) - Disable a gate for a thing.</p>
</li>
<li>
<p><code class="highlighter-rouge">get_multi(features)</code> - Get all gate values for several features at once.</p>
</li>
</ul>
<p>Letâs see an example:</p>
<p>Here weâre using the ActiveRecord adapter, which knows how to query a PostgreSQL database using
ActiveRecord. We enable a feature for a given user and the adapter takes care of storing this in the
database.</p>
<div class="language-ruby highlighter-rouge"><pre class="highlight"><code><span class="n">adapter</span> <span class="o">=</span> <span class="no">Flipper</span><span class="o">::</span><span class="no">Adapters</span><span class="o">::</span><span class="no">ActiveRecord</span>
<span class="n">flipper</span> <span class="o">=</span> <span class="no">Flipper</span><span class="p">.</span><span class="nf">new</span><span class="p">(</span><span class="n">adapter</span><span class="p">)</span>
<span class="n">flipper</span><span class="p">[</span><span class="ss">:cool_feature</span><span class="p">].</span><span class="nf">enable</span><span class="p">(</span><span class="n">user</span><span class="p">)</span>
</code></pre>
</div>
<p>Tomorrow the team decides document databases are much cooler and switches to MongoDB. Instead of having to rewrite all
of the flipper code we simply swap out the ActiveRecord adapter for the Mongo adapter and everything continues
to function as normal - the only difference being where Flipper puts the data, this time in a MongoDB
instance.</p>
<div class="language-ruby highlighter-rouge"><pre class="highlight"><code><span class="n">collection</span> <span class="o">=</span> <span class="no">Mongo</span><span class="o">::</span><span class="no">Client</span><span class="p">.</span><span class="nf">new</span><span class="p">([</span><span class="s2">"127.0.0.1:27017"</span><span class="p">],</span> <span class="ss">database: </span><span class="s1">'testing'</span><span class="p">)[</span><span class="s1">'flipper'</span><span class="p">]</span>
<span class="n">adapter</span> <span class="o">=</span> <span class="no">Flipper</span><span class="o">::</span><span class="no">Adapters</span><span class="o">::</span><span class="no">Mongo</span><span class="p">.</span><span class="nf">new</span><span class="p">(</span><span class="n">collection</span><span class="p">)</span>
<span class="n">flipper</span> <span class="o">=</span> <span class="no">Flipper</span><span class="p">.</span><span class="nf">new</span><span class="p">(</span><span class="n">adapter</span><span class="p">)</span>
<span class="n">flipper</span><span class="p">[</span><span class="ss">:cool_feature</span><span class="p">].</span><span class="nf">enable</span><span class="p">(</span><span class="n">user</span><span class="p">)</span>
</code></pre>
</div>
<p>The <code class="highlighter-rouge">Flipper</code> class provides a simple public interface, which is used in favor of interacting with adapters
directly. This means youâll always develop a Flipper adapter alongside the Flipper core API code, which brings
us to the first benefit of monorepos:</p>
<p><strong>1. Dependency Management</strong></p>
<p>Instead of requiring developers to figure out how to include the necessary Flipper code in their own adapter
repo, they can develop the adapter in the Flipper repo with full access to the code base. This reminds me of
the second benefit to monorepos:</p>
<p><strong>2. Easy Navigation</strong></p>
<p>While building an adapter youâll surely need to refer to various sections of the code. Navigation much easier
when everything is in one project. Iâd prefer to <code class="highlighter-rouge">ctrl-p</code> for something over searching through multiple repos
any day.</p>
<p><strong>3. Simple Setup</strong></p>
<p>If you wanted to build a Flipper adapter right now (and you should!) all youâd need to do is:</p>
<ul>
<li>
<p><code class="highlighter-rouge">git clone https://github.com/jnunemaker/flipper.git</code></p>
</li>
<li>
<p><code class="highlighter-rouge">cd flipper</code></p>
</li>
<li>
<p><code class="highlighter-rouge">vim spec/flipper/adapters/your_awesome_adapter_spec.rb</code></p>
</li>
</ul>
<div class="language-ruby highlighter-rouge"><pre class="highlight"><code><span class="c1"># your_awesome_adapter_spec.rb</span>
<span class="nb">require</span> <span class="s1">'flipper/spec/shared_adapter_specs'</span>
<span class="n">it_should_behave_like</span> <span class="s1">'a flipper adapter'</span>
</code></pre>
</div>
<p><code class="highlighter-rouge">rspec spec/flipper/adapters/your_awesome_adapter_spec.rb</code></p>
<p>The Flipper repo already has an RSpec shared adapter spec to code against to make sure your adapter works as
expected. Red, green, refactor your way to an open PR.</p>
<p>A few days from now, once your awesome PR is merged, letâs imagine <a href="http://github.com/jnunemaker">John</a>
decides to add a new method to the adapter interface. With this change comes the work of implementing this
method for all existing adapters, which brings us to the fourth benefit of monorepos:</p>
<p><strong>4. Cross-repo Changes</strong></p>
<p>Being an open source project, Flipper leans on the global community of developers to move it forward.
Multiple people from around the world will be working together to update the many adapters. Want to see when
Sarah updated the <a href="https://github.com/jnunemaker/flipper/tree/master/docs/redis">Redis adpater</a>? No problem.
Just check the git history. We have a monorepo, remember, so all changes are tracked within the same git
repository. No need to switch between projects to get the whole picture of the projectâs history. This
brings us to the fifth, and our final, benefit to monorepos:</p>
<p><strong>5. Common Versioning</strong></p>
<p>We check git and see that all adapters have been updated. Now comes the time to release a new version into
the wild. As a Flipper user Iâd love to use these updated adatpers, but which adapter versions support this
new method? Mongo adpater 2.1.5? ActiveRecord adapter 8.2.8? Memcached 1.6.2? Letâs keep it simple! All
projects in the monorepo share a common version number, statically defined in a single place for reference!</p>
<div class="language-ruby highlighter-rouge"><pre class="highlight"><code><span class="c1"># lib/flipper/version.rb</span>
<span class="k">module</span> <span class="nn">Flipper</span>
<span class="no">VERSION</span> <span class="o">=</span> <span class="s1">'0.11.0.beta7'</span><span class="p">.</span><span class="nf">freeze</span>
<span class="k">end</span>
</code></pre>
</div>
<p>Flipper 10.2 works with flipper-active_record 10.2, flipper-mongo 10.2, flipper-redis-10.2, etc.</p>
<p>By now we should have a good idea why Flipper uses a monorepo, but we havenât answered <code class="highlighter-rouge">Orbyt's</code> question.</p>
<p>Why is <code class="highlighter-rouge">Flipper::Adatpers::ActiveRecord</code> <code class="highlighter-rouge">undefined</code>?</p>
<p>This is where the monorepo really shines. You see, flipper-active_record depends on
<a href="https://github.com/rails/rails/tree/master/activerecord">ActiveRecord</a>, flipper-mongo depends on a <a href="https://github.com/mongodb/mongo-ruby-driver">mongo
ruby driver</a>, flipper-redis requires the <a href="https://github.com/redis/redis-rb">ruby redis
client</a>âŠyou get the point. Just because we develop these together
doesnât mean we have to distribute them together. Flipper users install the core Flipper gem with <code class="highlighter-rouge">bundle install
flipper</code> and then individually download any adapters theyâll need.</p>
<p><em>e.g.</em>
<code class="highlighter-rouge">bundle install flipper-active_record</code></p>
<p>The key here is that <code class="highlighter-rouge">bundle install flipper</code> <strong>does not</strong> download any
adapter dependencies it doesnât need. Letâs see how this works.</p>
<h2 id="anatomy-of-a-gem">Anatomy of A Gem</h2>
<p>Gems are ruby libraries - packaged code that can be shared with the world. A gem has three components:</p>
<ol>
<li>
<p>code</p>
</li>
<li>
<p>documentation (ideally)</p>
</li>
<li>
<p>gemspec</p>
</li>
</ol>
<h3 id="gemspec">gemspec</h3>
<p>The gemspec is a file stored in the root directory of a gem that specifies information about the gem, such as
its name, version, description, authors, etc. But this isnât all that a gemspec can specifiy. A gemspec can
configure many options, most importantly in our case, <code class="highlighter-rouge">files</code>- which files to include in the packaged gem.</p>
<p><em>For a full list of options check out RubyGemâs
<a href="http://guides.rubygems.org/specification-reference/">reference</a></em>.</p>
<p>Youâll find Flipperâs gemspec, like all gemspecs, in the root directory. Youâll also notice some other
gemspecs that have names resembling many of the adapters mentioned earlier. You might see where weâre
headed with thisâŠ</p>
<p><img src="/assets/monorepos/gemspecs.png" alt="gemspecs" /></p>
<p>Opening flipper.gemspec and removing unnecessary code for this conversation weâll focus on how Flipper is able
to ignore unnecessary code when packaging the gem.</p>
<div class="language-ruby highlighter-rouge"><pre class="highlight"><code><span class="n">plugin_files</span> <span class="o">=</span> <span class="p">[]</span>
<span class="no">Dir</span><span class="p">[</span><span class="s1">'flipper-*.gemspec'</span><span class="p">].</span><span class="nf">map</span> <span class="k">do</span> <span class="o">|</span><span class="n">gemspec</span><span class="o">|</span>
<span class="n">spec</span> <span class="o">=</span> <span class="nb">eval</span><span class="p">(</span><span class="no">File</span><span class="p">.</span><span class="nf">read</span><span class="p">(</span><span class="n">gemspec</span><span class="p">))</span>
<span class="n">plugin_files</span> <span class="o"><<</span> <span class="n">spec</span><span class="p">.</span><span class="nf">files</span>
<span class="k">end</span>
<span class="n">ignored_files</span> <span class="o">=</span> <span class="n">plugin_files</span>
<span class="n">ignored_files</span> <span class="o"><<</span> <span class="no">Dir</span><span class="p">[</span><span class="s1">'script/*'</span><span class="p">]</span>
<span class="n">ignored_files</span> <span class="o"><<</span> <span class="s1">'.travis.yml'</span>
<span class="n">ignored_files</span> <span class="o"><<</span> <span class="s1">'.gitignore'</span>
<span class="n">ignored_files</span> <span class="o"><<</span> <span class="s1">'Guardfile'</span>
<span class="n">ignored_files</span><span class="p">.</span><span class="nf">flatten!</span><span class="p">.</span><span class="nf">uniq!</span>
<span class="n">ignored_test_files</span> <span class="o">=</span> <span class="n">plugin_test_files</span>
<span class="n">ignored_test_files</span><span class="p">.</span><span class="nf">flatten!</span><span class="p">.</span><span class="nf">uniq!</span>
<span class="no">Gem</span><span class="o">::</span><span class="no">Specification</span><span class="p">.</span><span class="nf">new</span> <span class="k">do</span> <span class="o">|</span><span class="n">gem</span><span class="o">|</span>
<span class="n">gem</span><span class="p">.</span><span class="nf">files</span> <span class="o">=</span> <span class="sb">`git ls-files`</span><span class="p">.</span><span class="nf">split</span><span class="p">(</span><span class="s2">"</span><span class="se">\n</span><span class="s2">"</span><span class="p">)</span> <span class="o">-</span> <span class="n">ignored_files</span> <span class="o">+</span> <span class="p">[</span><span class="s1">'lib/flipper/version.rb'</span><span class="p">]</span>
<span class="k">end</span>
</code></pre>
</div>
<p>We start with an empty array <code class="highlighter-rouge">plugin_files</code>. <code class="highlighter-rouge">Dir['flipper-*.gemspec']</code> returns an array of all file names
that start with <code class="highlighter-rouge">flipper-</code> followed by any number of characters, <code class="highlighter-rouge">*</code>, and end with <code class="highlighter-rouge">.gemspec</code>. To better
demonstrate whatâs happening we could evaluate that line and replace it with its result.</p>
<div class="language-ruby highlighter-rouge"><pre class="highlight"><code><span class="p">[</span><span class="s1">'flipper-active_record.gemspec'</span><span class="p">,</span> <span class="s1">'flipper-api.gemspec'</span><span class="p">,</span> <span class="p">.</span><span class="nf">.</span><span class="o">.</span><span class="p">].</span><span class="nf">map</span> <span class="k">do</span> <span class="o">|</span><span class="n">gemspec</span><span class="o">|</span>
<span class="n">spec</span> <span class="o">=</span> <span class="nb">eval</span><span class="p">(</span><span class="no">File</span><span class="p">.</span><span class="nf">read</span><span class="p">(</span><span class="n">gemspec</span><span class="p">))</span>
<span class="n">plugin_files</span> <span class="o"><<</span> <span class="n">spec</span><span class="p">.</span><span class="nf">files</span>
<span class="k">end</span>
</code></pre>
</div>
<p>Continuing down the file we come across one of the coolest uses of Rubyâs <code class="highlighter-rouge">Kernel#eval</code> Iâve come across in
the wild. <code class="highlighter-rouge">eval(str)</code> evaluates the ruby expressions in <em>str</em>. Gemspecs might have the <em>.gemspec</em> file
extension, but donât let that fool you, theyâre plain old ruby files that can be read into a string and
evaluated.</p>
<div class="language-ruby highlighter-rouge"><pre class="highlight"><code><span class="n">spec</span> <span class="o">=</span> <span class="nb">eval</span><span class="p">(</span><span class="no">File</span><span class="p">.</span><span class="nf">read</span><span class="p">(</span><span class="n">gemspec</span><span class="p">))</span>
</code></pre>
</div>
<p>By evaling the gemspec we get a reference, <code class="highlighter-rouge">spec</code>, to the last ruby expression executed in the file <code class="highlighter-rouge">Gem::Specification.new do ... end</code>.</p>
<p>We push all of the yielded gemspecâs files into <code class="highlighter-rouge">plugin_files</code>.</p>
<div class="language-ruby highlighter-rouge"><pre class="highlight"><code><span class="n">plugin_files</span> <span class="o"><<</span> <span class="n">spec</span><span class="p">.</span><span class="nf">files</span>
</code></pre>
</div>
<p>The rest of the file is pretty straightforward. Ignore travis (CI) configuration, .gitignore, anything under
scripts/, etc.</p>
<div class="language-ruby highlighter-rouge"><pre class="highlight"><code><span class="n">ignored_files</span> <span class="o">=</span> <span class="n">plugin_files</span>
<span class="n">ignored_files</span> <span class="o"><<</span> <span class="no">Dir</span><span class="p">[</span><span class="s1">'script/*'</span><span class="p">]</span>
<span class="n">ignored_files</span> <span class="o"><<</span> <span class="s1">'.travis.yml'</span>
<span class="n">ignored_files</span> <span class="o"><<</span> <span class="s1">'.gitignore'</span>
<span class="n">ignored_files</span> <span class="o"><<</span> <span class="s1">'Guardfile'</span>
<span class="n">ignored_files</span><span class="p">.</span><span class="nf">flatten!</span><span class="p">.</span><span class="nf">uniq!</span>
<span class="n">ignored_test_files</span> <span class="o">=</span> <span class="n">plugin_test_files</span>
<span class="n">ignored_test_files</span><span class="p">.</span><span class="nf">flatten!</span><span class="p">.</span><span class="nf">uniq!</span>
</code></pre>
</div>
<p>Take all of these files weâd like to ignore and remove them from the the total files tracked in this git
repository. The result is a list of all the files that will be bundled with the published Flipper gem on
RubyGems.</p>
<div class="language-ruby highlighter-rouge"><pre class="highlight"><code><span class="n">gem</span><span class="p">.</span><span class="nf">files</span> <span class="o">=</span> <span class="sb">`git ls-files`</span><span class="p">.</span><span class="nf">split</span><span class="p">(</span><span class="s2">"</span><span class="se">\n</span><span class="s2">"</span><span class="p">)</span> <span class="o">-</span> <span class="n">ignored_files</span> <span class="o">+</span> <span class="p">[</span><span class="s1">'lib/flipper/version.rb'</span><span class="p">]</span>
</code></pre>
</div>
<p>And there we have it. Flipper::Adapters::ActiveRecord is undefined because
although the source exists in the monorepo, the bundled flipper gem does not
include it when published. Thanks monorepo.</p>
<p>Big shoutout to <code class="highlighter-rouge">Orbyt</code> for opening an awesome issue and submitting a
<a href="https://github.com/jnunemaker/flipper/commit/5761282a4b47acba7faee4c706be7227d31467d5">PR</a> that improves
documentation for all flipper users.</p>Like many technology companies we happily use Git to decentralize our development workflow. Browsing through the VTS organization on Github youâll notice a similar trend. Each of our repositories tracks one logical project. This works really well for us and countless other companies around the world. However, just because this is the optimal way to organize our projects, doesnât mean its the best approach for every project. The alternative to this approach is the monorepo.On Feature Flipping, Deli Coffee, and Hashing Algorithms2017-07-03T13:36:14+00:002017-07-03T13:36:14+00:00/ruby/2017/07/03/feature-flipping-deli-coffee-hashing-algorithms<p>Lifeâs full of goods and bads.</p>
<p>Surfing. Good. Warm beer. Bad</p>
<p>In-N-Out animal style fries. Really good.</p>
<p>Rush hour L train traffic. Really bad.</p>
<p>Modern digital computers might be binary, but the world isnât. Thereâs a whole
list of things that fall somewhere in the middle. Theyâre not really good, and
not totally bad. Theyâre good enough.</p>
<ul>
<li>
<p>Deli coffee - its no Blue Bottle, but it gets the job done.</p>
</li>
<li>
<p>Dollar slice pizza - It will never approach Prince Streetâs level, but at 4am
who cares?</p>
</li>
<li>
<p>pseudorandomness - âŠ</p>
</li>
</ul>
<p>Each of the above examples is a worthy candidate for todayâs post, but this is
a programming blog and you bet weâll be talking about programming. Letâs look
at pseudorandomness and use it to prove why sometimes in software good enough
can be, wellâŠgood enough.</p>
<p>What would we do without Google? (read: StackOverflow). A quick search and
200ms later Merriam-Webster dictionary is breaking this fancy word down for us:</p>
<p><em>pseudo</em> - âbeing apparently rather than actually as statedâ synonym: sham</p>
<p><em>random</em> - âwithout definite aim, direction, rule, or methodâ</p>
<p>Pseudorandomness is when something seems like it was generated without any
rhyme or reason, when it actually did follow a series of rules. Not truly random, but random enough.</p>
<p>Even if this is the first time youâve seen this word, it surely isnât the first
time youâve experienced it. Think about the last time you played music on
shuffle. Wasnât it rad how your computer randomly flipped through songs,
avoiding you the pain of having to listen to that playlist in the same order as
yesterdayâs morning commute. The truth is the song picks werenât actually
random, but that didnât matter to you, did it? They were chosen pseudorandomly - not truly random, but random enough.</p>
<p>Our computers are great at lots of things. Performing complex calculations,
predicting the outcome of events, even mining cryptocurrencies, but if thereâs
one thing theyâre horrible at, its randomness. The problem is your computer is
actually really dumb. It not only needs to be told what to do, but also exactly
how to do it. This includes generating randomness.</p>
<p><em>side note: You havenât experienced how dumb these machines really are until
youâve had a compiler scream at you for forgetting a semicolon.</em></p>
<p><img src="/assets/feature-flipping-hashing-algorithms/java-error.png" alt="java error" /></p>
<p>Luckily computer scientists have been working on this problem of randomness for
a long time and theyâve gotten pretty good at it as demonstrated by the shuffle
example. The other day, while reviewing a pull request on a popular open source
project I collaborate on, I came across a few lines of code that show a great
use case for pseudorandomness. I figure if I find them cool then you might
too. Letâs walk through this code with the intention that by the end of this
post weâll be equipped with concrete examples as to how us software developers
can use pseudorandomness to build software that is <em>good enough</em>.</p>
<p><a href="https://github.com/jnunemaker/flipper">Flipper</a> is an open source Ruby gem that makes feature flipping easy, while
having minimal impact on your application no matter your data store,
throughput, or experience. Letâs say our engineering team at <a href="https://www.vts.com/">VTS</a> just finished
a brand new Net Effective Rent (NER) calculator. Before we roll it out to all
users weâd like to test it with a subset. This is what Flipper lets us do. We
can enable this calculator for:</p>
<ul>
<li>
<p>specific user</p>
</li>
<li>
<p>group of users</p>
</li>
<li>
<p>percentage of users</p>
</li>
<li>
<p>percentage of time</p>
</li>
<li>
<p>all actors</p>
</li>
</ul>
<p>Flipper calls each of these ways a feature can be enabled a <a href="https://github.com/jnunemaker/flipper/blob/master/docs/Gates.md">gate</a>. As for all
actors: An actor is anything that responds to flipper_id. For todayâs
conversation weâll keep it simple and say an actor is a user, but its good to
keep in mind this could be anything.</p>
<p>Flipper definitely lives up to its goal of being easy to use. You create
features and enable them for one of these five options. As users access a page
with said feature, your code asks Flipper âis this feature enabled for this
user?â. Flipper responds true or false.</p>
<p><em>e.g. enabling a feature for specific users</em></p>
<div class="language-ruby highlighter-rouge"><pre class="highlight"><code><span class="n">user</span> <span class="o">=</span> <span class="no">User</span><span class="p">.</span><span class="nf">find</span><span class="p">(</span><span class="mi">1</span><span class="p">)</span>
<span class="n">user2</span> <span class="o">=</span> <span class="no">User</span><span class="p">.</span><span class="nf">find</span><span class="p">(</span><span class="mi">2</span><span class="p">)</span>
<span class="n">flipper</span><span class="p">[</span><span class="ss">:ner_calculator</span><span class="p">].</span><span class="nf">enable</span><span class="p">(</span><span class="n">user</span><span class="p">)</span>
</code></pre>
</div>
<div class="language-ruby highlighter-rouge"><pre class="highlight"><code><span class="n">flipper</span><span class="p">[</span><span class="ss">:ner_calculator</span><span class="p">].</span><span class="nf">enabled?</span><span class="p">(</span><span class="n">user</span><span class="p">)</span>
</code></pre>
</div>
<p><code class="highlighter-rouge">=> true</code></p>
<div class="language-ruby highlighter-rouge"><pre class="highlight"><code><span class="n">flipper</span><span class="p">[</span><span class="ss">:ner_calculator</span><span class="p">].</span><span class="nf">enabled?</span><span class="p">(</span><span class="n">user2</span><span class="p">)</span>
</code></pre>
</div>
<p><code class="highlighter-rouge">=> false</code></p>
<p>While Iâd love to spend time talking about all of these gates in depth, the
<a href="https://github.com/jnunemaker/flipper/blob/master/README.md">README</a> does a great job. Instead, letâs focus on two of the more interesting
gates.</p>
<ul>
<li>
<p>percentage of time</p>
</li>
<li>
<p>percentage of actors</p>
</li>
</ul>
<p><strong>Percentage of Time</strong></p>
<p>The percentage of time gate allows you to enable a feature for a given
percentage of time. A feature that is enabled for 20% of the time means that
20% of the time a user navigates to a page with that feature it will be on. The
other 80% it will be off. Put another way if we call</p>
<div class="language-ruby highlighter-rouge"><pre class="highlight"><code><span class="n">flipper</span><span class="p">.</span><span class="nf">enabled?</span><span class="p">(</span><span class="n">user</span><span class="p">)</span>
</code></pre>
</div>
<p>one hundred times it should return true 20 out of the 100 invocations. While
there arenât tons of use cases for this, there are a few. A great use case for
the percentage of time gate is load testing. Weâd like to see how our
application responds to different numbers of users interacting with it. We
start with a small percentage, say 5%. This means that 5% of users visiting the
page will see the feature. Over the next few weeks we can gradually increase
this percentage.</p>
<p><strong>Show me the code!</strong></p>
<div class="language-ruby highlighter-rouge"><pre class="highlight"><code><span class="n">flipper</span><span class="p">[</span><span class="ss">:ner_calculator</span><span class="p">].</span><span class="nf">enabled?</span><span class="p">(</span><span class="n">user</span><span class="p">)</span>
</code></pre>
</div>
<p>What happens when the above line is evaluated? At the end of the day, or
really the end of the call-stack, Flipper invokes a function, open? that
returns true if a given user can see a feature, and false otherwise. Every one
of the fives gates implements open?.</p>
<p>Can we write an implementation of <code class="highlighter-rouge">open?</code> that returns true n% of the time it is
invoked?</p>
<div class="language-ruby highlighter-rouge"><pre class="highlight"><code><span class="k">def</span> <span class="nf">open?</span><span class="p">(</span><span class="n">percentage</span><span class="p">)</span>
<span class="k">end</span>
</code></pre>
</div>
<p>This is software so of course we can and weâre going to use pseudorandomness to
accomplish it. It might not be very obvious at first, but the solution is
actually quite simple:</p>
<div class="language-ruby highlighter-rouge"><pre class="highlight"><code><span class="k">def</span> <span class="nf">open?</span><span class="p">(</span><span class="n">percentage</span><span class="p">)</span>
<span class="nb">rand</span> <span class="o"><</span> <span class="n">percentage</span>
<span class="k">end</span>
</code></pre>
</div>
<p>If youâre not familiar with Rubyâs <code class="highlighter-rouge">Kernel#rand</code>, no problem, its a core function
that when âcalled without an argumentâŠreturns a pseudo-random floating point
number between 0.0 and 1.0, including 0.0 and excluding 1.0âł. Some possible
outcomes of our function given a percentage of 0.5 are:</p>
<p>0.2 < 0.5</p>
<p>0.1 < 0.5</p>
<p>0.6 < 0.5</p>
<p>Assuming <code class="highlighter-rouge">rand</code> generates a random enough number weâre expecting that ~50% of the
time the generated number is going to be less than 0.5, which is true.</p>
<p>Just in case you donât believe me letâs test it out by invoking our function
100 times and counting the number of times it returns true.</p>
<div class="language-ruby highlighter-rouge"><pre class="highlight"><code><span class="p">(</span><span class="mi">0</span><span class="p">.</span><span class="nf">.</span><span class="mi">100</span><span class="p">).</span><span class="nf">count</span> <span class="p">{</span> <span class="nb">open</span><span class="p">?(</span><span class="mi">0</span><span class="o">.</span><span class="mi">5</span><span class="p">)</span> <span class="p">}</span>
</code></pre>
</div>
<p><code class="highlighter-rouge">=> 53</code></p>
<p>Great that looks about right, but to avoid disappointing statistics professors
worldwide (you never know who reads these things), letâs use a more appropriate
sample size. This time weâll call <code class="highlighter-rouge">open?(0.5)</code> a total of 10,000 times, recording
the number of times per 100 calls it returns true.</p>
<p><em>note: every element in the array is a sample of 100 calls to open?(0.50) and
shows the number of times true was returned.</em></p>
<p><img src="/assets/feature-flipping-hashing-algorithms/result.png" alt="result" /></p>
<p>It looks like the majority of the results are ~50, and with a <a href="https://gist.github.com/AlexWheeler/2733dbbb1670754f3de41873788d9bfb">little more code</a>
we get a nice visualization using Rubyâs awesome <a href="https://github.com/topfunky/gruff">Gruff</a> library.</p>
<p><img src="/assets/feature-flipping-hashing-algorithms/chart.png" alt="chart" /></p>
<p>Letâs also take the average while weâre at it:</p>
<div class="language-ruby highlighter-rouge"><pre class="highlight"><code><span class="p">(</span><span class="mi">0</span><span class="p">.</span><span class="nf">.</span><span class="mi">100</span><span class="p">).</span><span class="nf">map</span> <span class="p">{</span> <span class="p">(</span><span class="mi">1</span><span class="p">.</span><span class="nf">.</span><span class="mi">100</span><span class="p">).</span><span class="nf">count</span> <span class="p">{</span> <span class="nb">open</span><span class="p">?(</span><span class="mi">0</span><span class="o">.</span><span class="mi">50</span><span class="p">)</span> <span class="p">}</span> <span class="p">}.</span><span class="nf">reduce</span><span class="p">(:</span><span class="o">+</span><span class="p">)</span> <span class="o">/</span> <span class="mi">100</span><span class="p">.</span><span class="nf">to_f</span>
</code></pre>
</div>
<p>On average our <code class="highlighter-rouge">open?</code> implementation returns true ~50.44 percent of the time.
It isnât exactly 50%, but Iâd say its damn good enough. You now know how
<a href="https://github.com/jnunemaker/flipper/blob/master/lib/flipper/gates/percentage_of_time.rb#L25">Flipperâs percentage of time gate</a> implements open?</p>
<p>Hopefully, by now its clear how Flipper is able to turn a feature on for a
percentage of time by taking advantage of pseudorandomness to produce a
solution that isnât perfect but undoubtedly works good enough. That leaves us
with our next and most interesting question.</p>
<p><strong>Percentage of Actors</strong></p>
<p>How can Flipper turn a feature on for a percentage of users, while consistently
returning the same answer when asked whether a given user is enabled without
computing and persisting the list of users in some datastore? Taking our
current formula, adding another form of pseudorandomness, and topping it off
with a bit of creative thinking is all weâll need to produce a really clever
solution.</p>
<p>Given we want to check if a feature is turned on for a user, letâs start with
our inputs:</p>
<ul>
<li>
<p>User: { id: 1 }</p>
</li>
<li>
<p>Feature: { name: âner_calculatorâ }</p>
</li>
<li>
<p>Percentage of time this feature should be on: 30 (<em>note: weâll use whole number representations of
percentages to keep things simple</em>)</p>
</li>
</ul>
<p>If we could get a deterministic pseudo-random number based on the given user and feature then we should be able
to use the same approach as above. Something along the lines of:</p>
<div class="language-ruby highlighter-rouge"><pre class="highlight"><code><span class="k">def</span> <span class="nf">open?</span><span class="p">(</span><span class="n">user</span><span class="p">,</span> <span class="n">feature</span><span class="p">,</span> <span class="n">percentage</span><span class="p">)</span>
<span class="nb">rand</span><span class="p">(</span><span class="n">user</span><span class="p">,</span> <span class="n">feature</span><span class="p">)</span> <span class="o"><</span> <span class="n">percentage</span>
<span class="k">end</span>
</code></pre>
</div>
<p>We know we canât use <code class="highlighter-rouge">rand</code> as it only accepts one integer argument, not a user
or feature. This is where Flipper gets very clever. Iâll show a simplified
example (as to avoid needing to understand any of the implementation details
such as how Flipper stores values in your datastore) of how Flipper
accomplishes the percent of actors gate, and then explain step-by-step whatâs
going on.</p>
<div class="language-ruby highlighter-rouge"><pre class="highlight"><code><span class="k">def</span> <span class="nf">open?</span><span class="p">(</span><span class="n">user</span><span class="p">,</span> <span class="n">feature</span><span class="p">,</span> <span class="n">percentage</span><span class="p">)</span>
<span class="nb">id</span> <span class="o">=</span> <span class="s2">"</span><span class="si">#{</span><span class="n">user</span><span class="p">.</span><span class="nf">id</span><span class="si">}#{</span><span class="n">feature</span><span class="p">.</span><span class="nf">name</span><span class="si">}</span><span class="s2">"</span>
<span class="no">Zlibcrc32</span><span class="p">(</span><span class="nb">id</span><span class="p">)</span> <span class="o">%</span> <span class="mi">100</span> <span class="o"><</span> <span class="n">percentage</span>
<span class="k">end</span>
</code></pre>
</div>
<p>Genius. The interesting idea here is figuring out how to generate a random
number given a user and a feature. The key point is that this random number
needs to always be the same number given the same inputs. Otherwise this gate
would act more like a percentage of time gate and sometimes say yes this
feature is enabled for this user and other times say no. Flipper accomplishes
this via the crc32 checksum, a commonly used algorithm to detect accidental
changes to data sent over a network, as a hashing function. Hashing algorithms
take some input and (ideally) produce a unique output. Crc32 serves as a great
hashing algorithm for our case because its efficient and only produces integers
unlike some other hashing algorithms such as SHA-1 and MD5 that produce
alphanumerics. The important thing to note here is that crc32 isnât a perfect
solution for our use case, but its good enough. The outputs arenât random or
else weâd end up with non-deterministic values. Theyâre just seemingly random!</p>
<p>e.g. <em>crc32 checksum of two similar strings. Notice there is seemingly no
similarity between the two produced values despite the similarity of inputs.
Given the same inputs the output will always be the same.</em></p>
<div class="language-ruby highlighter-rouge"><pre class="highlight"><code><span class="no">Zlib</span><span class="p">.</span><span class="nf">crc32</span><span class="p">(</span><span class="s2">"alex"</span><span class="p">)</span>
</code></pre>
</div>
<p><code class="highlighter-rouge">=> 2772900818</code></p>
<div class="language-ruby highlighter-rouge"><pre class="highlight"><code><span class="no">Zlib</span><span class="p">.</span><span class="nf">crc32</span><span class="p">(</span><span class="s2">"Rlex"</span><span class="p">)</span>
</code></pre>
</div>
<p><code class="highlighter-rouge">=> 1205423261</code></p>
<div class="language-ruby highlighter-rouge"><pre class="highlight"><code><span class="no">Zlib</span><span class="p">.</span><span class="nf">crc32</span><span class="p">(</span><span class="s2">"Rlex"</span><span class="p">)</span>
</code></pre>
</div>
<p><code class="highlighter-rouge">=> 1205423261</code></p>
<p>One more hashing function to make sure we all understand the high-level idea
because it is important for this conversation. Say we have a hash function
that given a string input returns the sum of its bytes.</p>
<div class="language-ruby highlighter-rouge"><pre class="highlight"><code><span class="k">def</span> <span class="nf">hash_str</span><span class="p">(</span><span class="n">str</span><span class="p">)</span>
<span class="n">str</span><span class="p">.</span><span class="nf">bytes</span><span class="p">.</span><span class="nf">reduce</span><span class="p">(:</span><span class="o">+</span><span class="p">)</span>
<span class="k">end</span>
</code></pre>
</div>
<p>Hashing a few strings:</p>
<div class="language-ruby highlighter-rouge"><pre class="highlight"><code><span class="n">hash_str</span><span class="p">(</span><span class="s2">"ruby"</span><span class="p">)</span>
</code></pre>
</div>
<p><code class="highlighter-rouge">=> 450</code></p>
<div class="language-ruby highlighter-rouge"><pre class="highlight"><code><span class="n">hash_str</span><span class="p">(</span><span class="s2">"Alex"</span><span class="p">)</span>
</code></pre>
</div>
<p><code class="highlighter-rouge">=> 394</code></p>
<p>At first this might seem like a decent hash function, but its actually
horrible. Without even getting into the issues with passing in strings with
varying encodings (Definitely check out my post on character encodings if
youâre not up to speed on how computers handle characters. hint: âAlexâ might
not always sum to 394), its a bad hash function because its capable of
producing many collisions. A collision is when two or more inputs hash to the
same output. By simply rearranging the characters weâre not changing the total
number of bytes so:</p>
<div class="language-ruby highlighter-rouge"><pre class="highlight"><code><span class="n">hash_str</span><span class="p">(</span><span class="s2">"xlAe"</span><span class="p">)</span>
</code></pre>
</div>
<div class="language-ruby highlighter-rouge"><pre class="highlight"><code><span class="n">hash_str</span><span class="p">(</span><span class="s2">"eAlx"</span><span class="p">)</span>
</code></pre>
</div>
<div class="language-ruby highlighter-rouge"><pre class="highlight"><code><span class="n">hash_str</span><span class="p">(</span><span class="s2">"lxeA"</span><span class="p">)</span>
</code></pre>
</div>
<p>all hash to 394! - assuming weâre working with ASCII or UTF-8 encoded strings.</p>
<p>Back to Flipper.</p>
<p>open? takes the userâs id and concatenates it with the feature name.</p>
<div class="language-ruby highlighter-rouge"><pre class="highlight"><code><span class="k">def</span> <span class="nf">open?</span><span class="p">(</span><span class="n">user</span><span class="p">,</span> <span class="n">feature</span><span class="p">,</span> <span class="n">percentage</span><span class="p">)</span>
<span class="nb">id</span> <span class="o">=</span> <span class="s2">"</span><span class="si">#{</span><span class="n">user</span><span class="p">.</span><span class="nf">id</span><span class="si">}#{</span><span class="n">feature</span><span class="p">.</span><span class="nf">name</span><span class="si">}</span><span class="s2">"</span>
<span class="no">Zlibcrc32</span><span class="p">(</span><span class="nb">id</span><span class="p">)</span> <span class="o">%</span> <span class="mi">100</span> <span class="o"><</span> <span class="n">percentage</span>
<span class="k">end</span>
</code></pre>
</div>
<p>Substituting id with its value for a user with id: 22 and feature name:
:ner_calculator. We get:</p>
<div class="language-ruby highlighter-rouge"><pre class="highlight"><code><span class="no">Zlib</span><span class="p">.</span><span class="nf">crc32</span><span class="p">(</span><span class="s2">"22ner_calculator"</span><span class="p">)</span> <span class="o">%</span> <span class="mi">100</span> <span class="o"><</span> <span class="n">percentage</span>
</code></pre>
</div>
<p>Futher reducing the lefthand side of the equation we calculate the crc32
checksum of â22ner_calculatorâ to get:</p>
<div class="language-ruby highlighter-rouge"><pre class="highlight"><code><span class="mi">1316767308</span> <span class="o">%</span> <span class="mi">100</span> <span class="o"><</span> <span class="n">percentage</span>
</code></pre>
</div>
<p>Since crc32 checksums are large integers we mod the result by 100 to get a
value between 0 and 100:</p>
<div class="language-ruby highlighter-rouge"><pre class="highlight"><code><span class="mi">8</span> <span class="o"><</span> <span class="n">percentage</span>
</code></pre>
</div>
<p>Finally we stated weâre checking this user and feature with a percentage of 30:</p>
<div class="language-ruby highlighter-rouge"><pre class="highlight"><code><span class="mi">8</span> <span class="o"><</span> <span class="mi">30</span>
</code></pre>
</div>
<p>At last Flipper tells us, âYes, Alex, this user is enabled for this feature
based on this percentageâ:</p>
<p><code class="highlighter-rouge">=> true</code></p>
<p>Hopefully youâve learned something and have a new-found appreciation for
randomness! Be sure to check out the <a href="https://github.com/jnunemaker/flipper/blob/master/lib/flipper/gates/percentage_of_actors.rb#L27">actual source code</a> as youâre armed with
enough knowledge to know exactly how these features are implemented.</p>Lifeâs full of goods and bads.Callstack Profiling with Stackprof2016-09-29T13:36:14+00:002016-09-29T13:36:14+00:00/performance/ruby/2016/09/29/callstack-profiling<p>We recently had <a href="https://twitter.com/brynary">Bryan Helmkamp</a>, Code Climate founder and CEO, drop by the
office and give a talk on what makes code good code - a subject heâs pretty
knowledgeable on having spent the last few years building a very popular
static code analysis tool that your team might even be using. One of his
coolest slides was a graph of what exactly defines technical debt. Bryanâs
definition of this term we hear all too often is summed up in his slide found
below.</p>
<p><img src="/assets/callstack-profiling/technical-drift.png" alt="technical drift" /></p>
<p>The x-axis is the lifetime of some piece of code. The y-axis is the current
business domain. You can see when the code was written the business domain
and the code aligned perfectly. Over time business goals shift, while
existing code remains unchanged. The difference is technical debt. The only
way to decrease technical debt is to better align code with the current
business domain.</p>
<p>As our users have been busy adding data points, weâve been busy adding code -
lots of code. Going off of Brianâs interpretation of technical debt its
pretty clear that, like any growing application, weâve racked up some debt
ourselves. In this post Iâd like to walk through a recent adventure of mine
descending down the deep rabbit hole of performance tuning. Along the way
weâll go over some of the tools Iâve picked up along the path to the
automated synchronization of VTS clientsâ 3 billion square feet, but whoâs
counting? đ</p>
<p><img src="/assets/callstack-profiling/diff.png" alt="diff" /></p>
<p>One particular space my team, integrations, spends a majority of its time is
building out our portfolio import infrastructure. This covers the onboarding
of new clientâs portfolios, updating existing clientsâ portfolio
acquisitions, as well as an ongoing technical effort to ensure VTS data
reflects actions taken in any of the industryâs leading ERP systems, namely
Yardi, MRI, JD Edwards, etc. It also means a constant tug-of-war between
integrations, product, and sales around what data points our imports support.
As Bryan reassured us, business domains change over time. As weâre heads down
cranking on the integration infrastructure other engineering squads are
tackling their respective features, consistently adding more data and metrics
our users require access to. What started as a basic rent roll:</p>
<p><em>What is the RBA, or total rentable area of my asset?</em></p>
<p><em>How much of that is vacant?</em></p>
<p>has grown into one hell of a datastore:</p>
<p>Rights / encumbrances</p>
<ul>
<li>
<p>Rent escalations (OpExs, Other Revenues, Real Estate Taxes, etc)</p>
</li>
<li>
<p>Future/historical leases</p>
</li>
<li>
<p>TIMS (Tenants In The Market)</p>
</li>
<li>
<p>Comparables</p>
</li>
</ul>
<p>and this is only a fraction of whatâs coming down the pipeline.</p>
<p>The code that efficiently imported one or two propertysâ basic rent rolls
might not be suited to ingest the sheer volume of data our landlords, agency
brokers, and tenant rep brokers work with on a daily basis.</p>
<p>At some point in the past few weeks we noticed our larger clientsâ
integrations taking a bit longer than usual. Nothing dramatic, but enough to
warrant discussion. During our monthly 1:1 my CTO Karl mentioned heâs had
lots of success pinpointing performance bottlenecks with a tool called
<a href="https://github.com/tmm1/stackprof">stackprof</a> and suggested I give it a look.</p>
<p><strong>Profiling</strong></p>
<p>The first step in any performance tuning epic is narrowing down exactly where
the performance problem lives. Just looking at the logs proved one code
segment in particular was taking significantly longer than usual. This path
involves the parsing of flat files (<em>.csv</em>, <em>.xls</em>, <em>.xlsx</em>, <em>.tsv</em>, <em>.psv</em>) before
importing the extracted data into a local datastore. The main entry point is `load_files_into_db()*.</p>
<div class="language-ruby highlighter-rouge"><pre class="highlight"><code><span class="k">def</span> <span class="nf">load_files_into_db</span><span class="p">(</span><span class="o">*</span><span class="n">files</span><span class="p">)</span>
</code></pre>
</div>
<p>As you can guess takes some file objects and loads them into a
database. Like any other application, this method kicks off a chain of method
calls that end in a full database and a successfully terminated program. What
would be cool is if we could locate exactly which function our application is
spending the majority of its time executing. Enter stackprof.</p>
<p>Performance tuning is comparable to a trip to the doctorâs office for some
persistent pain. Your doctor likely begins by taking vitals and asking basic
questions in an attempt to form a hypothesis about what might be causing your
pain. After arriving at some idea of what it could be she can employ more
advanced techniques to truly find the issue. Dietary problem? Might be time
for a blood test. Recently fell skating? An MRI should provide a better
picture of whatâs going on.</p>
<p>While Iâm no doctor, my business card says software engineer and it turns out
us software engineers have some pretty cool tools for diagnosing performance
issues.</p>
<p>Taking an unusually long time to parse a file? Check.</p>
<p>You began noticing the issue when file sizes approached some memory
threshold?</p>
<p>1MB? 4GB? 1TB? Gotcha.</p>
<p>Memory pressure remains relatively constant? Great to know.</p>
<p>Given my general software knowledge, my CTOâs recommendation, and a genuine
interest in using your application as a guinea pig to learn more about
performance optimization Iâll prescribe stackprof, a call-stack profiler to
be taken at 1000ms intervals.</p>
<p>A great tool that stands out for having the biggest bang for its buck is a
trusty sampling call-stack profiler. These profilers take a snapshot of an
applicationâs call-stack at some given interval and record a variety of
metrics that can be analyzed to draw a better picture of where an application
may be spending too much of its time. Many, including stackprof, can be
deployed to a production environment and silently collect data over time for
later analysis, but Iâve found they can be just as useful on your local
machine. For Ruby applications I prefer stackprof for a couple of reasons:</p>
<ol>
<li>
<p>freely available on RubyGems</p>
</li>
<li>
<p>supports Ruby 2.1+</p>
</li>
<li>
<p>simple to use</p>
</li>
</ol>
<p>The README does a great job explaining basic setup and usage so I wonât go into too much
detail but weâll cover the basics.</p>
<p>Add âstackprofâ to your Gemfile and <code class="highlighter-rouge">bundle install</code>.</p>
<p>The gem provides a class method <code class="highlighter-rouge">run</code>, taking some arguments and a block,
which weâll use to wrap any code weâd like to profile. Wrapping
<code class="highlighter-rouge">load_files_into_db</code> we pass the following arguments:</p>
<ul>
<li>
<p>mode: :wall - use wall clock time. This is just another way of saying profile this code using the overall amount of time it takes for this method (I/O included) to complete, not just the amount of time itâs hogging CPU.</p>
</li>
<li>
<p>out: âtmp/load_db.dumpâ - write the results to a file <em>tmp/load_db.dump</em></p>
</li>
<li>
<p>interval: 1000 - sample the call stack every 1000ms or 1 second.</p>
</li>
</ul>
<p>With the profiler in place we can run our program as normal and analyze the
results written to <em>tmp/load_db.dump</em> using the stackprof executable plus a
couple of arguments instructing it to print the results in text mode and
limit them to just the top 5 sampled methods.</p>
<p><code class="highlighter-rouge">stackprof tmp/load_db.dump --text --limit 5</code></p>
<p><img src="/assets/callstack-profiling/output.png" alt="output" /></p>
<p><strong>Refactoring</strong></p>
<p>The first two columns show the total percentage and total number of samples
that the given method was on the call stack at all. The third and fourth
columns are a bit more interesting and are where weâll focus our attention.
They show the number and percentage of samples that this method was on the
very top of the stack. An interesting point to note is that even though
<code class="highlighter-rouge">CSV#shift</code> was sampled more frequently than <code class="highlighter-rouge">Amalgalite::Statement#initialize</code>,
(637 v. 365) the latter appears higher in the list since it was on top of the
call-stack more frequently (365 v. 234). The higher number of total samples
with a lower percentage of time on top of the stack could mean that <code class="highlighter-rouge">CSV#shift</code>
might have been simply calling child helper methods and therefore was hanging
out on the stack until these functions returned. Or maybe we could have
picked a better sampling interval. For now letâs assume our interval was good
enough and take a look at the method stackprof most frequently found at the
top of the stack. These tend to be the most inefficient methods and some of
the lowest hanging fruit for serious performance improvements.</p>
<p>The method coming in at number one with stackprof finding it at the top
49.7%of the time is <code class="highlighter-rouge">Statement#next_row</code>. This is something weâd expect to see
seeing that it handles inserting a record into our database, an operation
that is hard to avoid given the whole job of this code path is to load
records into a database. Although we could probably optimize this letâs
assume itâs doing the best it can and look at the next method in the list
create_date_if_date. Now this is interesting. A method a little further
removed from the act of interacting directly with the database. What could
this method possibly be doing thatâs taking so long? Looking at a similar,
yet overly simplified, example of our production code letâs see what we can
find out.</p>
<div class="language-ruby highlighter-rouge"><pre class="highlight"><code><span class="k">def</span> <span class="nf">create_date_if_date</span><span class="p">(</span><span class="n">field</span><span class="p">)</span>
<span class="no">Date</span><span class="p">.</span><span class="nf">parse</span><span class="p">(</span><span class="n">field</span><span class="p">)</span> <span class="k">rescue</span> <span class="kp">nil</span>
<span class="k">end</span>
</code></pre>
</div>
<p>The first thing that stands out is the rescue clause swallowing errors and
returning <code class="highlighter-rouge">nil</code>. As we know exceptions should be <em>exceptional</em> and for good
reason. Exceptions not only make our programs more difficult to reason
about, theyâre also extremely slow to handle. <code class="highlighter-rouge">create_date_if_date()</code> takes one
argument <code class="highlighter-rouge">field</code>, which may or may not be a valid date string and passes it to
<code class="highlighter-rouge">Date.parse</code>. Date.parse in turn tries to parse it into a date object. Donât
worry if youâre not familiar with the implementation of Date.parse, just know
it will try its best to parse a given string, but if passed something that
looks nothing like a date (i.e. <code class="highlighter-rouge">Date.parse("Alex")</code> or <code class="highlighter-rouge">Date.parse("")</code>) it
throws its hands up and raises an <code class="highlighter-rouge">ArgumentError</code>. Rubyâs rescue defaults to
handling <code class="highlighter-rouge">StandardError</code> (a superclass of <code class="highlighter-rouge">ArgumentError</code>), and eventually
catches the exception, returning nil. Great, if <code class="highlighter-rouge">create_date_if_date()</code> is
passed a date-like string itâs successfully parsed and when passed anything
else it simply returns nil.</p>
<p>This method works great if the majority of the time our method is receiving
date-like strings to parse, but the reality is data comes in all shapes and
sizes and while landlords do track lots of dates, they also track dollar
amounts, square footages, street addresses, numbers, letters, and everything
in between. This means that a good amount of the time weâre paying this
method (in computational resources) to rescue tons of exceptions only to
return nil - not a very efficient use of its time or ours. Just how much
time is it wasting though? Before we can answer this and close out our
discussion on performance tuning we need to jump down one more level and get
a better understanding of just how poorly exception handling code performs.
For this weâll use Rubyâs Benchmark module. The Benchmark module allows us
to to time two methods and print their results next to each other for easy
comparison Letâs use two implementations of our function that weâd like to
benchmark against each other.</p>
<p><code class="highlighter-rouge">with_rescue</code> is an improved implementation that when passed an empty string
immediately returns nil avoiding the costly exception that would be raised
should an empty string be passed to <code class="highlighter-rouge">Date.parse</code>. <code class="highlighter-rouge">with_rescue</code> is our original
poorly performing function.</p>
<div class="language-ruby highlighter-rouge"><pre class="highlight"><code><span class="nb">require</span> <span class="s2">"benchmark"</span>
<span class="nb">require</span> <span class="s2">"date"</span>
<span class="k">def</span> <span class="nf">without_rescue</span><span class="p">(</span><span class="n">value</span><span class="p">)</span>
<span class="k">return</span> <span class="k">if</span> <span class="n">value</span><span class="p">.</span><span class="nf">empty?</span>
<span class="no">Date</span><span class="p">.</span><span class="nf">parse</span><span class="p">(</span><span class="n">value</span><span class="p">)</span> <span class="k">rescue</span> <span class="kp">nil</span>
<span class="k">end</span>
<span class="k">def</span> <span class="nf">with_rescue</span><span class="p">(</span><span class="n">value</span><span class="p">)</span>
<span class="no">Date</span><span class="p">.</span><span class="nf">parse</span><span class="p">(</span><span class="n">value</span><span class="p">)</span> <span class="k">rescue</span> <span class="kp">nil</span>
<span class="k">end</span>
<span class="no">Benchmark</span><span class="p">.</span><span class="nf">bm</span> <span class="k">do</span> <span class="o">|</span><span class="n">x</span><span class="o">|</span>
<span class="n">x</span><span class="p">.</span><span class="nf">report</span><span class="p">(</span><span class="s2">"without rescue"</span><span class="p">)</span> <span class="k">do</span>
<span class="mi">100</span><span class="p">.</span><span class="nf">times</span> <span class="p">{</span> <span class="n">without_rescue</span><span class="p">(</span><span class="s2">""</span><span class="p">)</span> <span class="p">}</span>
<span class="k">end</span>
<span class="n">x</span><span class="p">.</span><span class="nf">report</span><span class="p">(</span><span class="s2">"with rescue"</span><span class="p">)</span> <span class="k">do</span>
<span class="mi">100</span><span class="p">.</span><span class="nf">times</span> <span class="p">{</span> <span class="n">with_rescue</span><span class="p">(</span><span class="s2">""</span><span class="p">)</span> <span class="p">}</span>
<span class="k">end</span>
<span class="k">end</span>
</code></pre>
</div>
<p>Calling each function 100 times with an empty string produces the following
results.</p>
<p><img src="/assets/callstack-profiling/output2.png" alt="output 2" /></p>
<p>This tell us <code class="highlighter-rouge">without_rescue</code> took a total of .000022 seconds (.022ms) to run
100 times, while <code class="highlighter-rouge">with_rescue</code> ran in .000791 seconds (0.791ms) . Clearly,
avoiding the exception using <code class="highlighter-rouge">without_rescue</code> is faster, but only by 0.766ms
faster, so who cares? Does it really matter that we can save ~half a ms with
our new implementation? Well, actually we do. Just as every data point in
practice doesnât happen to be a date, our client portfolios in practice donât
happen to pass only 100 values through this little function. Think of your
basic spreadsheet. You might easily have 20,000 rows and 100 columns
totaling 2,000,000 cells - all of which will be passed through this little
function. Letâs rerun this import using a more realistic number, say
2,000,000, still well below production loads, but enough to give us a result
that makes more sense to us humans who have an easier time thinking in terms
of seconds not thousandths of a second.</p>
<p>This time we run the same code but call each function 2,000,000 times and
check out the results.</p>
<div class="language-ruby highlighter-rouge"><pre class="highlight"><code><span class="no">Benchmark</span><span class="p">.</span><span class="nf">bm</span> <span class="k">do</span> <span class="o">|</span><span class="n">x</span><span class="o">|</span>
<span class="n">x</span><span class="p">.</span><span class="nf">report</span><span class="p">(</span><span class="s2">"without rescue"</span><span class="p">)</span> <span class="k">do</span>
<span class="mi">2_000_000</span><span class="p">.</span><span class="nf">times</span> <span class="p">{</span> <span class="n">without_rescue</span><span class="p">(</span><span class="s2">""</span><span class="p">)</span> <span class="p">}</span>
<span class="k">end</span>
<span class="n">x</span><span class="p">.</span><span class="nf">report</span><span class="p">(</span><span class="s2">"with rescue"</span><span class="p">)</span> <span class="k">do</span>
<span class="mi">2_000_000</span><span class="p">.</span><span class="nf">times</span> <span class="p">{</span> <span class="n">with_rescue</span><span class="p">(</span><span class="s2">""</span><span class="p">)</span> <span class="p">}</span>
<span class="k">end</span>
<span class="k">end</span>
</code></pre>
</div>
<p><img src="/assets/callstack-profiling/output3.png" alt="output 3" /></p>
<p>Our improved function <code class="highlighter-rouge">without_rescue</code> took a mere .33 seconds (330ms) versus a
whopping 13.26seconds (13260ms) for <code class="highlighter-rouge">with_rescue</code> to finish! Now this was only
2 million data points run through a simplified version of our code, but you
can imagine the performance hit weâd take with 5 million or 20 million or 1
billion values passing through this guy.</p>
<p>We certainly havenât finished refactoring this function, but Iâd say weâre
off to a good start. Rerunning stackprof we no longer see our function in
the top 5 most profiled methods and for this Iâm stoked.</p>
<p>Now the real point of this post isnât to teach you the proper way to handle
Ruby exceptions, the ancestry chain of the Ruby Exception api, or even how to
use a call stack-profiler like stackprof, even though those are all cool
topics. I think the whole reason I wanted to share this short story was to
paint a better picture of how business needs change over time and because of
this we need to make sure that as software developers our code keeps up with
ever shifting product requirements. This wasnât poorly performing code at
the time it was committed and deployed. It did its job and it did it very
well. For our requirement at the time 1ms v. 50ms didnât make much of a
difference when we were importing 1 unit and the whole program completed in
under 1 second. Over time our needs changed, portfolios grew, and we made
time to narrow the gap between our x and y axes. Hope you enjoyed.</p>
<p>Find this cool?</p>
<p>Tweet me @askwheeler</p>
<p>https://github.com/alexwheeler</p>We recently had Bryan Helmkamp, Code Climate founder and CEO, drop by the office and give a talk on what makes code good code - a subject heâs pretty knowledgeable on having spent the last few years building a very popular static code analysis tool that your team might even be using. One of his coolest slides was a graph of what exactly defines technical debt. Bryanâs definition of this term we hear all too often is summed up in his slide found below.Netfïżœix aïżœd Chïżœll?2016-01-24T13:36:14+00:002016-01-24T13:36:14+00:00/golang/encoding/2016/01/24/netflix-and-chill<p><em>Note: No prior knowledge of Golang or online dating required for this post</em></p>
<p>Youâre having so much fun exploring the ins and outs of the Go programming
language that you decide its time to finally start working on that social
network dating app for meeting surfer chicks youâve always wanted to build.
While youâre super excited about the seemingly endless list of features you
have planned, youâre quickly growing tired of meeting girls on Tinder just to
find out that they boogie board. A decision is made to focus on the most
important feature and ship an alpha version two weeks from today. The target
audience is young, single millennials - a generation completely comfortable
with experiencing the world through a 2.5x4.5 inch digital screen. Although our
parents had to pick up the phone and call a potential date, weâre cool with
sending a quick text and ending it with that cute cactus emoji. Because who
doesnât like that thing?</p>
<p>đ”đ”đ”đ”đ”đ”đ”đ”đ”đ”đ”đ”đ”đ”đ”đ”đ”đ”đ”đ”đ”đ”đ”đ”đ”đ”đ”</p>
<p>Emojis are dope and even though technology continues to completely change the
dating experience, its clear communication is still key. You start with feature
number 1 - messaging. Messaging has two requirements:</p>
<ul>
<li>Messages have to be 10 characters or less</li>
<li>Messages MUST support emojis</li>
</ul>
<p>V1 ships and everythingâs going great. Your inbox is flooded with messages from
all types of surfer girls, and youâve spent the last few days really hitting it
off with Lauren from Brooklyn. Sheâs smart, funny, and well travelled - a point
made very clear by the Chinese symbol on her forearm from <del>Spring Break 2012</del> her
trip through Southern China . Hey, even though it might not even actually mean
wave, it beats the <em>pura vida</em> tattoo Jackie picked up on her two-day layover in
Costa Rica. You wait 19.5 minutes, because you obviously canât risk seeming too
eager, and reply with an idea you had for your own Chinese symbol tattoo.</p>
<p>characters: 11/10</p>
<p>message: âcool æ”·æŽ?â</p>
<p>Wait. What? As much as you try to type âcool æ”·æŽ?â, the app says youâre over
the character limit. Somethingâs wrong. Thereâs only 8 characters with the
space, clearly under the 10 character limit, yet the app is saying thereâs 11
and you canât click send.</p>
<p>Go is an awesome language, but there are for sure a few gotchas for new
Gophers. Letâs take a look at the code that validates a given message is under
the character limit.</p>
<div class="language-golang highlighter-rouge"><pre class="highlight"><code><span class="k">const</span><span class="x"> </span><span class="n">charLimit</span><span class="x"> </span><span class="o">=</span><span class="x"> </span><span class="m">10</span><span class="x">
</span><span class="k">func</span><span class="x"> </span><span class="n">isUnderMsgLimit</span><span class="p">(</span><span class="n">msg</span><span class="x"> </span><span class="kt">string</span><span class="p">)</span><span class="x"> </span><span class="kt">bool</span><span class="x"> </span><span class="p">{</span><span class="x">
</span><span class="k">return</span><span class="x"> </span><span class="nb">len</span><span class="p">(</span><span class="n">msg</span><span class="p">)</span><span class="x"> </span><span class="o"><=</span><span class="x"> </span><span class="n">charLimit</span><span class="x">
</span><span class="p">}</span><span class="x">
</span></code></pre>
</div>
<p>A constant, <code class="highlighter-rouge">charLimit</code>, is initialized with a value of 10. A function,
<code class="highlighter-rouge">isUnderLimit</code>, is declared which takes 1 string argument, <code class="highlighter-rouge">msg</code>, and returns a
boolean - true if the length of <code class="highlighter-rouge">msg</code> is less than or equal to <code class="highlighter-rouge">charLimit</code>, and
false otherwise.</p>
<p>You scratch your head, and conclude the language is wrong and canât do basic
arithmetic - Thanks Rob Pike. Until you finally do the reasonable thing and
look up the documentation for the <a href="https://golang.org/pkg/builtin/">builtin</a> packageâs len function.</p>
<p><img src="/assets/netflix-and-chill/len-go.png" alt="len-go" /></p>
<p>Aha! When <code class="highlighter-rouge">len</code> is passed a string it returns the number of bytes in the string
and NOT the number of characters - a little different than some other languages
you might have experience with, like Rubyâs definition of length defined on the
String class.</p>
<p><img src="/assets/netflix-and-chill/len-ruby.png" alt="len-ruby" /></p>
<p>So why does this happen? Its actually pretty simple. In Go, all strings are
just slices of bytes. So when you take the length of âæ”·æŽâ its actually taking
the length of the slice containing the bytes representing those characters. We
can convert the string to its byte slice representation with T(v), which
converts the value of v to type T.</p>
<div class="language-golang highlighter-rouge"><pre class="highlight"><code><span class="p">[]</span><span class="kt">byte</span><span class="p">(</span><span class="err">â</span><span class="n">æ”·æŽ</span><span class="err">â</span><span class="p">)</span><span class="x">
</span></code></pre>
</div>
<p><code class="highlighter-rouge">=> [230, 181, 183, 230, 180, 139]</code></p>
<div class="language-golang highlighter-rouge"><pre class="highlight"><code><span class="nb">len</span><span class="p">([</span><span class="m">230</span><span class="p">,</span><span class="x"> </span><span class="m">181</span><span class="p">,</span><span class="x"> </span><span class="m">183</span><span class="p">,</span><span class="x"> </span><span class="m">230</span><span class="p">,</span><span class="x"> </span><span class="m">180</span><span class="p">,</span><span class="x"> </span><span class="m">139</span><span class="p">])</span><span class="x">
</span></code></pre>
</div>
<p><code class="highlighter-rouge">=> 6</code></p>
<div class="language-golang highlighter-rouge"><pre class="highlight"><code><span class="nb">len</span><span class="p">(</span><span class="s">"æ”·æŽ"</span><span class="p">)</span><span class="x">
</span></code></pre>
</div>
<p><code class="highlighter-rouge">=> 6</code></p>
<p>These are the same thing, just one, the byte array, is mutable, while the
string is not (strings in Go are immutable). But this leads us to the next
question. Why do some strings seem to have one byte per character while others
use more?</p>
<div class="language-golang highlighter-rouge"><pre class="highlight"><code><span class="nb">len</span><span class="p">(</span><span class="s">"foo"</span><span class="p">)</span><span class="x">
</span></code></pre>
</div>
<p><code class="highlighter-rouge">=> 3</code></p>
<div class="language-golang highlighter-rouge"><pre class="highlight"><code><span class="p">[]</span><span class="kt">byte</span><span class="p">(</span><span class="s">"foo"</span><span class="p">)</span><span class="x">
</span></code></pre>
</div>
<p><code class="highlighter-rouge">=> [102, 111, 111]</code></p>
<div class="language-golang highlighter-rouge"><pre class="highlight"><code><span class="nb">len</span><span class="p">(</span><span class="s">"æ”·æŽ"</span><span class="p">)</span><span class="x">
</span></code></pre>
</div>
<p><code class="highlighter-rouge">=> 6</code></p>
<div class="language-golang highlighter-rouge"><pre class="highlight"><code><span class="p">[]</span><span class="kt">byte</span><span class="p">(</span><span class="s">"æ”·æŽ"</span><span class="p">)</span><span class="x">
</span></code></pre>
</div>
<p><code class="highlighter-rouge">=> [230, 181, 183, 230, 180, 139]</code></p>
<p>To answer this weâre going to have to take a bit of a detour and learn what a
character really is to a computer.</p>
<p><strong>Character Encoding</strong></p>
<p>Letâs count from 0 - 9.</p>
<p>0, 1, 2, 3, 4, 5, 6, 7, 8, 9</p>
<p>Okay, great! Now how about 0 - 10?</p>
<p>0, 1, 2, 3, 4, 5, 6, 7, 8, 9, 10</p>
<p>Wait! What did you just do there? I understand that 1 comes after 0 and 3 comes
after 2. I even remember learning why 6 is afraid of 7, but how did you get
from 9 to 10? You probably didnât even think twice about what came after 9.
Obviously its 10. And what about 99? What comes next? 100! And after 100? 101!</p>
<p>Do you remember all those hours spent learning to count? To tell you the truth
my memory is pretty fuzzy about those early years, but how about learning your
times tables? I clearly remember the flashcards, worksheets, and quizzes we
spent an entire year going over in the 1st grade. We take it for granted now,
but your teachers, family, and friends put a considerable amount of effort into
teaching you one of the most widely used counting systems in the world - the
decimal system.</p>
<p>But, before we go there letâs back up a bit and quickly go over how we got to
where we are today.</p>
<p><img src="/assets/netflix-and-chill/caveman.png" alt="caveman" /></p>
<p>Unary</p>
<p>It all started with the number 1. Well, not actually the number 1, but the
idea of <em>one</em> being a single unit of something. 1 could be represented with a
single tick:</p>
<p>I</p>
<p>One more than one could be represented with two ticks.</p>
<p>II</p>
<p>Two more than one could be represented with three ticks.</p>
<p>III</p>
<p>To keep this short, letâs just say that in unary the amount <em>n</em> can be
represented with <em>n</em> ticks. Hereâs twenty represented in unary:</p>
<p>IIIIIIIIIIIIIIIIIIII</p>
<p><strong>The Romans</strong></p>
<p><img src="/assets/netflix-and-chill/roman.png" alt="roman" /></p>
<p>Soon enough the world saw civilizations expanding, mathematics and languages
developing, and the Roman Empire hell-bent on world domination. The Romans were
pretty smart, and one day a whole bunch of them got together and hashed out a
wild idea: âWouldnât it be rad if we could turn these ticks into unique symbols
that better represent their valueâ.</p>
<p>One could be represented with:</p>
<p><img src="/assets/netflix-and-chill/one.png" alt="one" /></p>
<p>Two could be represented with:</p>
<p><img src="/assets/netflix-and-chill/two.png" alt="two" /></p>
<p>And five could be:</p>
<p><img src="/assets/netflix-and-chill/five.png" alt="five" /></p>
<p>Clearly much easier to work with than their unary alternatives:</p>
<p>IIIIIIIIIIIIIIIIIIIIIIIIIIIIIIIIIIIIIIIIIIII</p>
<p>Yes, these symbols look a bit cooler and make for some awesome tattoos, but the
real genius is exposed when you understand how the Romans took advantage of
position to change the meaning of a given symbol.</p>
<p>These Roman nerds understood that four is really just one less than five and
can be represented by placing the I to the left of V:</p>
<p><img src="/assets/netflix-and-chill/four.png" alt="four" /></p>
<p>Six is really just one more than five and can be represented by placing the I
to the right of five:</p>
<p><img src="/assets/netflix-and-chill/six.png" alt="six" /></p>
<p><strong>Decimal</strong></p>
<p>Fast forward a couple of years and we find another group of smart peeps
thinking about this same concept of a symbolâs position having a huge
significance on its value. These smart peeps were the inventors of the decimal
system and the rest of this post can be attributed to their findings (as well
as this quadruple IPA I happened to have stumbled upon a few hours ago).</p>
<p>Taking influence from the Romanâs numeral system, the decimal system uses
something called <a href="https://en.wikipedia.org/wiki/Positional_notation">positional notation</a>, which is simply a way of encoding, or
representing, numbers - Positional notation uses the same symbols, also
called glyphs, combined with placeholders signifying different orders of
magnitude to represent any number value in the world. The order of magnitude
applied is also known as a radix - which in the case of the decimal system is 10. For this reason decimal is referred to as base 10 and takes its name from
the Latin <em>decimus</em> meaning tenth. Take for example the number 6,523:</p>
<p>6,523</p>
<p><img src="/assets/netflix-and-chill/6523.png" alt="6523" /></p>
<p>Pictured above is each number with its corresponding order of magnitude based
on its position. Reading this right-to-left using positional notation we could
say.</p>
<p>There are:</p>
<p>3 ones</p>
<p>2 tens</p>
<p>5 one-hundreds</p>
<p>6 one-thousands</p>
<p>for a total of 6, 523</p>
<p>We can do this for any number in the universe. To represent 23 we use the same
notation.</p>
<p>There are:</p>
<p>3 ones</p>
<p>2 tens</p>
<p>for a total of 23</p>
<p><strong>Binary</strong></p>
<p>Alright, so that was a pretty fun trip down memory lane, but youâre probably
wondering how the hell this possibly relates to programming? Well computers,
it turns out, use a similar counting system. While the decimal system uses ten
distinct glyphs to represent every number in the entire world, a very
convenient fact for those of us living with ten fingers, what happens when you
donât have the blessing of ten glyphs, or fingers? Is there a way to represent
every number in the world with only two symbols, or in our case two signals -
off and on? Sounds wild, but there is. Its called binary and everything from
your phone, to your MacBook, to your hover board uses this counting system
every single day.</p>
<p><em>Bi</em> comes from Latin <em>binarius</em>, meaning consisting of two. Binary has two
glyphs, 0 and 1, and like the decimal system, uses the power of positional
notation. In this case its based on multiples of 2, which gives it the name
base 2. Letâs follow the same rules we used before and write some binary.
Below we have the number 38 represented in binary.</p>
<p><img src="/assets/netflix-and-chill/binary-38.png" alt="binary-38" /></p>
<p>Using positional notation to read right-to-left there are:</p>
<p>0 ones</p>
<p>1 two</p>
<p>1 four</p>
<p>0 eights</p>
<p>0 sixteens</p>
<p>1 thirty-two</p>
<p>for a total 38!</p>
<p>What is the maximum number that can be represented in base 2 with 8 positions?</p>
<p><img src="/assets/netflix-and-chill/255.png" alt="255" /></p>
<p>1 one</p>
<p>1 two</p>
<p>1 four</p>
<p>1 eight</p>
<p>1 sixteen</p>
<p>1 thirty-two</p>
<p>1 sixty-four</p>
<p>1 one-hundred-twenty-eight</p>
<p>for a total of 255.</p>
<p>So, with 8 placeholders we can represent 0- 255 or 256 unique values. And if
we substitute the word placeholder with bit, then we can say that with 8 bits
we can represent 256 different values. Further, understanding that 8 bits
combine to make 1 byte, we can conclude that with 1 byte, we can represent 256
values.</p>
<p><em>note: A bit is the most basic unit of information a computer can store. 0 or 1,
off or on. Since bits donât tell us much on their own, we address them in
groups of 8, and call this 1 byte.</em></p>
<p>So, using a combination of just two signals off or on, a computer is capable of
representing any number in the world. While this is awesome, youâre still
probably wondering how this relates to our initial problem of potentially
losing a date because our application thinks that our message is too long.
Letâs take a very brief look at our last counting system before moving onto how
this translates to computers representing characters.</p>
<p><strong>Hexadecimal</strong></p>
<p>It turns out that after a while developers got really tired of constantly
writing 1 and 0 to describe their programsâ memory usage, and set out to think
of a more concise way to describe the values these 0s and 1s represent. Some
smart, fed-up, 0-and-1-hating-programmer came along and decided that instead of
using base 2 or base 10, what if they used a radix of 16, also known as
hexadecimal? The Greek <em>hex</em> means six and we already know that decimal means
tenth. Combing the two we get hexadecimal or 16. We still have 0, 1, 2, 3, 4,
5, 6, 7, 8, 9. But we now have 6 additional glyphs to work with for a total of</p>
<ol>
<li>For these, the inventors of hexadecimal decided on A - F. The full range
of hexadecimal symbols is:</li>
</ol>
<p>0, 1, 2, 3, 4, 5, 6, 7, 8, 9, A, B, C, D, E, F</p>
<p>If youâd like you can go through the same process as earlier, breaking
placeholders into columns based on their order of magnitude and doing basic
algebra to deserialize, or decode, the decimal value of a hex number. To make
things short, programmers love hexadecimal because a byte can be encoded using
at most 2 symbols and many times just 1.</p>
<p>1111 1111 (255) in binary becomes FF in hex, and 0000 1010 (ten) is A.</p>
<p><strong>Unicode</strong></p>
<p>Weâve now learned what a counting system is, the differences between decimal,
binary, and hexadecimal, as well as how a computer is able to represent
different numbers using the only two signals it knows, off or on. But up until
now we havenât touched on memory and how computers store this information that
ultimately gets rendered to an applicationâs UI leading to a possible date with
some random person you met on your phone.</p>
<p><strong>ASCII</strong></p>
<p>One of the earliest attempts at character encoding was ASCII, American Standard
Code for Information Interchange. ASCII accomplishes two goals:</p>
<ol>
<li>
<p>assigns 0 - 127 for a total of 128 numbers to English alphabet letters along with some control commands such as Shift, Backspace.</p>
</li>
<li>
<p>Stores each character in memory as its corresponding value from the chart below.</p>
</li>
</ol>
<p><img src="/assets/netflix-and-chill/ascii.png" alt="ascii" /></p>
<p>Since 128 values can be stored in 7 bits this means that any ASCII character
can be stored in 1 byte (with a whole bit to spare since only 7 of the 8 are
used).</p>
<p>Letâs look at an example. Using the chart for reference, lowercase latin a is
assigned a value of 97, so a computer encoding text in ASCII will store this
character in memory as 01100001 (decimal 97). My name, âAlexâ could be stored
as:</p>
<p><img src="/assets/netflix-and-chill/alex.png" alt="alex" /></p>
<p>ASCII works great until you remember that there are over 7 billion people on
Earth speaking thousands of languages. Since only 7 bits are required to store
every ASCII character people realized thereâs one leftover bit to do something
with. This extra bit provides 128 more possible characters (2^8). Thatâs great
news, until the Germans decided to use the remaining 128 values to map their
own characters, the Russians wanted their own, and the Chinese needed their own
as well. You would have a Spanish program that decided 190 meant ñ, while a
Chinese program decided that 190 is obviously æłą. It would be similar to two
persons knowing the same language, English for example, but interpreting the
meaning of words in completely different ways. The word <em>happy</em> to Drew means
<em>firetruck</em>, while to Willy it means <em>dog</em>. Nonsense. Someone needed to come up
with a common language that all computer programs, no matter what language they
were trying to store could correctly translate to the right character given
some value. This is exactly what Unicode has been solving for the past few
decades.</p>
<p><strong>Unicode</strong></p>
<p>Unicode assigns every character in every language in the entire world a number,
which we call a code point. A list of these code points can be found on the
<a href="http://unicode.org/">Unicode website</a> and are represented using âU+nâ where U+ says this is a Unicode
code point and n is the value represented in hexadecimal. Unicode code points
0 - 127 are 1:1 the same with ASCII values, which weâll see in a minute can be
very useful.</p>
<p>For example Latin capital A is U+41 (decimal 65), a is U+61 (decimal 97), and
đ©is U+1F4A9 (decimal 128169)</p>
<p>Weâve gone over how Unicode maps characters to some numerical value and denotes
them in hex, but we havenât talked about how these characters are actually
stored in memory. ASCII was able to store every one of its 128 values in 1
byte, but many Unicode values are much larger than 255, like the pile-of-poop
emoji seen above. This is because Unicode says nothing about how a value is
stored in memory. It simply says Iâll map a character to a number and if you
can give me that number, Iâll give you the character. To actually store a
characterâs value Unicode needs an encoding scheme to translate the number into
bytes. To better understand this weâll look at one of the most common encoding
schemes, UTF-8.</p>
<p><strong>UTF-8</strong></p>
<p>Since Unicode just maps characters to values, encoding schemes are used to
actually translate these values into bytes. UTF-8 is one of many
variable-width encodings. This means that If a Unicode value is less than 255
UTF-8 will store this character in 1 byte, otherwise it will use as many bytes
as it needs to store the character. Because of this, characters with Unicode
values < 255 encoded with ASCII (represented by 0 -127) or UTF-8 will look the
exact same in memory. This means any ASCII encoded document can be read
without any problems using UTF-8! Some other encoding schemes youâve probably
heard of are UTF-16, and UTF-32, with UTF-16 using at minimum 2 bytes (16 bits)
to store a character and UTF-32 using at minimum 4 bytes, (32 bits), causing
them to be incompatible with ASCII. Therefore the most important rule whenever
working with text is to <strong>ALWAYS KNOW THE ENCODING!</strong> Due to the vast number of
encodings that all look very similar at the byte level this is near impossible
and is why HTTP Requests declare the character set used, HTML has a meta tag to
declare a character set, and databases are explicitly configured to use a
specific charset or encoding.</p>
<p><strong>Solution</strong></p>
<p>That was a lot to go over, but by now you should have a clear answer to our
original question - why <em>æ”·æŽ</em> is represented using 3 bytes and <em>foo</em> only one,
assuming these characters were encoded with UTF-8, which Go source code is.</p>
<p>I almost forgot! Lauren is still waiting for a response, and we have a bug to
fix! The fix is pretty simple. In Go, as previously stated, strings are just
arrays of bytes. We now know that characters are just combinations of bytes.
Go provides the Rune type to describe a character as a single unit (regardless
of number of bytes) So instead of checking the length of the array of total
bytes, we can covert the string to any array of its individual characters, or
Runes as Go calls them, and take the length of this array.</p>
<div class="language-golang highlighter-rouge"><pre class="highlight"><code><span class="n">fmt</span><span class="o">.</span><span class="n">Println</span><span class="p">([</span><span class="x"> </span><span class="p">]</span><span class="kt">rune</span><span class="p">(</span><span class="s">"æ”·æŽ"</span><span class="p">))</span><span class="x">
</span></code></pre>
</div>
<p><code class="highlighter-rouge">=> [28023, 27915]</code></p>
<div class="language-golang highlighter-rouge"><pre class="highlight"><code><span class="n">fmt</span><span class="o">.</span><span class="n">Println</span><span class="p">(</span><span class="nb">len</span><span class="p">([]</span><span class="x"> </span><span class="kt">rune</span><span class="p">(</span><span class="s">"æ”·æŽ"</span><span class="p">)))</span><span class="x">
</span></code></pre>
</div>
<p><code class="highlighter-rouge">=> 2</code></p>
<p>Update: Despite the bug-fix, Lauren didnât find character encodings, or cacti emojis (emoji
plural?) as rad as I do, and it never quite worked out. Questions, comments?
Reach out on twitter @askwheeler or find me on <a href="github.com/alexwheeler">GitHub</a>!</p>Note: No prior knowledge of Golang or online dating required for this postHacking for Good Waves2015-10-21T13:36:14+00:002015-10-21T13:36:14+00:00/javascript/security/2015/10/21/hacking-for-good-waves<p><em>Note: Surfline has been contacted and is working on patching the issue. They
gave permission to post this and hooked up a year premium membership. Very
metal Surfline \m/.</em></p>
<p>Every surfer knows a Rex. Heâs the guy who always seems to be in the right
spot, at the right time, every time. Rex actually understands all these terms
us surfers love to throw around while sizing up a 5-7ft@21 seconds WNW swell
cranking down from the Gulf of Alaska at 280 degrees. While you and the rest of
the boys froth over how hard youâre going to score at the local beach break in
the morning, Rex is already running the numbers in his head, knowing this time
tomorrow heâll be scoring head-high, empty barrels, while you duke it out with
the rest of the weekend warriors in two foot slop. He was right, you were
wrong. God damnet.</p>
<p><em>Rex - Baja Norte, Mexico 2014 (yes we scored)</em></p>
<p><img src="/assets/hacking-for-good-waves/rex.png" alt="rex" /></p>
<p>While you may know a Rex, you may or may not know anything about the world wide
web or how we ended up where we are today.</p>
<p>In the early days of the web there really wasnât a way to programmatically make
changes to a web document after a request succeeded. Any modifications to a
document had to be made as it dynamically rendered on the server. Then during
the mid 1990âs, in the midst of an era that would later come to be known as the
browser wars, emerged the amazing language that we all know as JavaScript.
JavaScript opened the doors to client-side scripting, giving application
developers the ability to run code in the context of a web document within the
browser after it returned from the server! It also opened the door to countless
security vulnerabilities and gave people like me an entire playground to
explore to our hearts content.</p>
<p>While I certainly believe I have one of the coolest jobs in the world, spending
most of my days trying to find more efficient ways to tell computers what to
do, I have also had the amazing opportunity of pissing away the majority of my
youth enjoying one of lifeâs greatest gifts - surfing. Even since moving to NYC
a few months ago Iâve made it a point to get in the water nearly every weekend
there are waves. At first glance these may seem like two opposing worlds, but
long gone are the days of Bruce Brownâs <a href="https://en.wikipedia.org/wiki/The_Endless_Summer">Endless Summer</a>. Weâre in the 21st
century, meaning surf forecasting has not only entered the digital age, its
thriving. While our friend Rex can easily decipher the complex global swell
models and forecasting tools built on top of publicly available data from the
likes of the National Data Buoy Center, most of us just want to know how the
waves are looking right now. From there we can make educated guesses as to how
theyâll look in an hour or so.</p>
<p><em>Long Beach, NY summer 2015</em></p>
<p><img src="/assets/hacking-for-good-waves/long_beach.png" alt="rex" /></p>
<p>Any surfer whoâs actively surfed in the past decade is no doubt familiar with
<a href="www.surfline.com">Surflineâs</a> camera services that allow anyone with an internet connection to
check the waves at thousands of spots around the world in real time. As with
all awesome things in life thereâs a catch. Unless youâve shelled out the money
for a premium account the stream pauses, displays an advertisement, and forces
a refresh every 30 seconds. Its an accepted fact by now that this will
continually happen just before the set youâve been waiting for is seen rolling
in from the horizon. By the time the stream refreshes said set is long gone. If
only we had a few more minutes without the refresh we could make a sound
decision whether to head for the beach or try our luck tomorrow. Turns out
today I donât even need to check the cams to see what the swell is doing.
Looking out the window tells me the weather is crap and winds are picking up.
There couldnât be a better day to grab some wine, get comfy on the couch, and
<del>watch netflix</del> read source code!</p>
<p>We already know two very important pieces of information.</p>
<ol>
<li>
<p>JavaScript allows developers to manipulate client-side content</p>
</li>
<li>
<p>Surflineâs cameras pause after 30 seconds to show an ad</p>
</li>
</ol>
<p>Its not too difficult to put the two together and conclude that Surfline is most likely using JavaScript to
trigger an event that pauses he video stream and displays the ad. Since your
web browser downloads all of the requested resources from the server we can go
ahead and dive into some of Surflineâs code.</p>
<p>The first step is to locate the scripts. This is simple in most modern,
evergreen browsers - my preference being Chrome. Simply open the developer
console with <em>Cmd + Shift + i</em> and navigate to the sources tab. Here we can find
all of the downloaded JavaScript files.</p>
<p><img src="/assets/hacking-for-good-waves/sources1.png" alt="sources1" /></p>
<p>Most of them will be from third party applications theyâre running such as ad
clients, analytics tools, etc., but at the top level of the directories to the
left we quickly find the code weâre looking for in a ColdFusion Markup File
<em>report-c.cfm</em>, nested under surfline.com/surfdata.</p>
<p>Now the fun begins. We want to find any events trigged on a camera. So how
about we start by searching for any code containing the word <em>camera</em>?</p>
<p><img src="/assets/hacking-for-good-waves/sources2.png" alt="sources2" /></p>
<p>Toggling through the 134 results we soon come across our first interesting
statement.</p>
<p>Clearly this is initializing a variable called <code class="highlighter-rouge">cameraCurrentTime</code> to 30000
milliseconds - a very common time unit used in JavaScript programming. Some
elementary math tells us 30000 milliseconds is equal to 30 seconds - the same
time frame the camera runs for before requiring a refresh!</p>
<p><img src="/assets/hacking-for-good-waves/sources3.png" alt="sources3" /></p>
<p>Digging a bit deeper we find the culprit - <code class="highlighter-rouge">updateTimer()</code>. Every time this
function is called it first checks if <code class="highlighter-rouge">cameraCurrentTime</code> is equal to 0. If it is
equal to 0 it ends the camera stream, otherwise it decrements <code class="highlighter-rouge">cameraCurrentTime</code>
by 1000 milliseconds or 1 second.</p>
<p>Heading another level deeper we discover a magical line of code on line 5268
that informs us of two things.</p>
<p><img src="/assets/hacking-for-good-waves/sources4.png" alt="sources4" /></p>
<p>There is a variable <code class="highlighter-rouge">userType</code> defined somwhere in this file that can be one of many different values, some of which are:</p>
<ul>
<li>
<p>premium</p>
</li>
<li>
<p>vip</p>
</li>
<li>
<p>vip_adfree</p>
</li>
<li>
<p>vip-adfree</p>
</li>
</ul>
<p>Iâm just going to take a wild guess here and say that these values are probably pretty important.</p>
<p>Whoever fulfilled Gavinâs request really didnât think this one through.</p>
<p>The next step is to find out where <code class="highlighter-rouge">userInfo</code> is initalized. This could take a
long time given this is a 10,666 line file, but since weâre technologists weâll
let our machines do the work. A quick cmd-f search for âuserType = â (notice
whitespace to avoid matching userType == boolean checks) reveals the keys to
scoring better waves for free. Dissecting these eleven lines of JavaScript
should tell us exactly how to set a userâs privilege level.</p>
<p><img src="/assets/hacking-for-good-waves/sources5.png" alt="sources5" /></p>
<p>Line 4935 informs us user information is stored in a cookie <code class="highlighter-rouge">'USERINFO'</code>. Cookies
are just small text files stored on disk that websites often use to keep
information about a user.</p>
<p><img src="/assets/hacking-for-good-waves/sources6.png" alt="sources6" /></p>
<p>Line 4938 parses <code class="highlighter-rouge">userInfo</code> as a JSON string, transforming it from a string to a
basic JavaScript object, an interface for data storage thatâs much easier to
work with than a standard string.</p>
<p><img src="/assets/hacking-for-good-waves/sources7.png" alt="sources7" /></p>
<p>Line 4939 accesses the value we care about. It begins by asking <code class="highlighter-rouge">userInfoParsed</code>
for its user attribute, which returns an array. This array stores an object at
its first index (arrays in JavaScript are 0 indexed). Asking this object for
its <code class="highlighter-rouge">usType</code> returns the current userâs privilege level.</p>
<p>As a free member we wonât have a USERINFO cookie stored, however that doesnât
mean we canât store one before Surfline can check for it. As far as the
JavaScript is concerned, Surfline set this cookie and because of this we can
trick the browser by simply reversing the operations, setting the cookie
ourselves, and reloading the page, in turn re-executing the script - this time
with an escalated privilege level.</p>
<p><strong>reversing the steps</strong></p>
<ol>
<li>
<p>Build data structure</p>
<p><img src="/assets/hacking-for-good-waves/sources8.png" alt="sources8" /></p>
</li>
<li>
<p>Transform JSON object to JSON string via <code class="highlighter-rouge">JSON.stringify()</code> (opposite of JSON.parse)</p>
<p><img src="/assets/hacking-for-good-waves/sources9.png" alt="sources9" /></p>
</li>
<li>
<p>Set cookie USERINFO to this value</p>
<p><img src="/assets/hacking-for-good-waves/sources10.png" alt="sources10" /></p>
</li>
<li>
<p>Confirm it works :)</p>
<p><img src="/assets/hacking-for-good-waves/sources11.png" alt="sources11" /></p>
</li>
</ol>
<p>Now when the browser executes this script it will find the <code class="highlighter-rouge">'USERINFO'</code> cookie, parse it, and grant premium access.</p>
<p><img src="/assets/hacking-for-good-waves/still-watching.png" alt="still-watching" /></p>
<p>To this day Surfline remains, hands down, one of my favorite technology
companies. Theyâve no doubt played an enormous role in helping my friends, as
well as the rest of the surf world, score epic surf sessions year after year.
The ethical action for any software professional to take upon discovering a
security issue is to follow the <a href="https://en.wikipedia.org/wiki/Responsible_disclosure">responsible disclosure</a> process by notifying the
vendor and allowing them a reasonable amount of time to patch it before
notifying the public. (Unless its Fox News - then please feel free to burn it
to the ground). But really, if you find an issue with a vendorâs site that you
respect report it! After disclosing the vulnerabilty and providing a proof of
concept the epic team at Surfline has addressed the issue, upgraded my account
to premium for the next year, as well as given permission to publish the post.
Rock on \m/</p>
<p><img src="/assets/hacking-for-good-waves/email.png" alt="email" /></p>Note: Surfline has been contacted and is working on patching the issue. They gave permission to post this and hooked up a year premium membership. Very metal Surfline \m/.