Jekyll2024-03-04T14:47:20+00:00https://eviltrout.com/feed.xmlEvil Trout’s BlogA place for my thoughts on software development.The Secrets of Skellig2023-07-17T00:00:00+00:002023-07-17T00:00:00+00:00https://eviltrout.com/2023/07/17/the-secrets-of-skellig<p>I’ve wanted to dip my feet back into game development for a while, and last year I decided to put my money where my
mouth is. I resigned from my full time job at Discourse and pulled up a bunch of YouTube tutorials on Blender and Godot.</p>
<p>It’s been a ridiculous amount of fun, and I’ve learned so much about how modern 3D games are made. I’m finally ready to
announce what I’ve been working on. It’s a puzzle game, inspired by crosswords, called “The Secrets of Skellig.”</p>
<p><img alt="The Secrets of Skellig Logo" src="/images/skellig-banner.png" /></p>
<p>The road ahead is long, and I have a lot more work to do. I’m making it primarily myself at this point, so it’ll be slow
going but I’m making excellent progress.</p>
<p>You can see more here: <a href="https://secretsofskellig.com">The Secrets of Skellig</a>.</p>I’ve wanted to dip my feet back into game development for a while, and last year I decided to put my money where my mouth is. I resigned from my full time job at Discourse and pulled up a bunch of YouTube tutorials on Blender and Godot.The Roottrees are (still) Dead2023-07-17T00:00:00+00:002023-07-17T00:00:00+00:00https://eviltrout.com/2023/07/17/the-roottrees-are-dead<p>Last year I broke my elbow. This meant that I had to take a break from programming for about six weeks, and <a href="https://secretsofskellig.com">The Secrets of Skellig</a>, my indie game, was put on hold.</p>
<p>After a couple of weeks, I recovered enough to do some basic keyboard and mouse movements, and I stumbled on a free game called <a href="https://jjohnstongames.itch.io/the-roottrees-are-dead">The Roottrees are Dead</a> by Jeremy Johnston. It was <em>exactly</em> my kind of thing: a thinky game where you have to use your brain to fill in a family tree, based on results from a search engine. It was <a href="https://www.herstorygame.com">Her Story</a> meets <a href="https://en.wikipedia.org/wiki/Return_of_the_Obra_Dinn">Return of the Obra Dinn</a> meets <a href="https://en.wikipedia.org/wiki/The_Case_of_the_Golden_Idol">The Case of the Golden Idol</a>, and it consumed my free time for several days in a row.</p>
<p>When I was finished, I noticed it was not released on Steam, presumably due to the generative AI art it used. I thought: <em>This game deserves to be much bigger than it is.</em></p>
<p>I reached out to Jeremy on the game’s Discord and said, I know you don’t know me, but hear me out. <strong>What if I worked with you to bring this to Steam?</strong></p>
<h3 id="shelving-my-passion-project">Shelving my passion project</h3>
<p>Putting my main project on hold was not an easy decision. I still love The Secrets of Skellig and I have about an hour of gameplay finished. I recently had some people playtest it, and while feedback was not super positive, it identified a bunch of areas that I thought could improve and I remain excited about the project. But it’s a long-term effort, there are literally years of work ahead of me.</p>
<p>Still, I could not shake the idea of how great The Roottrees are Dead could be. It was <em>already</em> a fantastic game. The puzzles and writing were clever, the worldbuilding was fun and interesting, and I knew it was within my abilities to spruce up the user interface. I have also been fortunate enough to have had some financial success from co-founding <a href="https://discourse.org">Discourse</a>, so I could help with funding professional illustrations.</p>
<p>Jeremy and I kept talking, and I kept prototyping, and recently we announced the Steam version of The Roottrees are Dead!</p>
<iframe src="https://store.steampowered.com/widget/2754380/" frameborder="0" width="646" height="190"></iframe>
<h3 id="what-have-we-enhanced">What have we enhanced?</h3>
<ul>
<li>
<p>All the AI art will be replaced with professional illustrations.</p>
</li>
<li>
<p>There’s a totally new hybrid 3D/2D interface. You zip around your living room in 3D, then explore the Internet circa 1998 on your retro computer in 2D.</p>
</li>
<li>
<p>We’ve remastered the audio with new sound effects and voice acting.</p>
</li>
<li>
<p>Every aspect of the user interface has been rethought to make solving the mystery smoother and faster. If we did our job properly it’ll just get out of the way and let your brain do the work.</p>
</li>
<li>
<p>The game has been ported from Unity to Godot. We’ll be releasing on Windows, Mac, and Linux.</p>
</li>
<li>
<p>Care has been taken to allow for localization, although we haven’t committed to any other languages right now. If the game does well I hope we can release it to non-English speakers.</p>
</li>
</ul>
<p>The original free game isn’t going anywhere. I’ve always been a fan of preserving web content for as long as possible. I’m hopeful though, that our new version will be more than good enough to entice people to throw us a few bucks. But hey, if you disagree, that’s cool! The free version is right there.</p>
<p>If this sounds like something you’d like to play, I’d love it if you could <a href="https://store.steampowered.com/app/2754380/The_Roottrees_are_Dead/">wishlist it on steam</a>. Also, if you know anyone who would like this kind of game please spread the word.</p>Last year I broke my elbow. This meant that I had to take a break from programming for about six weeks, and The Secrets of Skellig, my indie game, was put on hold.My Retro PC Gaming Nostalgia Kick2022-07-18T00:00:00+00:002022-07-18T00:00:00+00:00https://eviltrout.com/2022/07/18/my-retro-pc-gaming-nostalgia-kick<p>Over the last couple of months, I’ve been posting a series of videos about early PC gaming and programming on my <a href="https://www.youtube.com/channel/UCBfKBvnIbB93AQgLNo_K8Bg">YouTube Channel</a>. It’s been quite a fun journey and I thought I might write a few words about it.</p>
<p>I have a lot of nostalgia for early games, which makes sense since it was how I spent most of my free time in the 80’s and 90’s. Times were boring before the Internet :)</p>
<p>Early PC architecture is so simple compared to modern computers. Any program, regardless of language, can write to any part of memory or port. The memory and processing power of the systems at the time are limited, but your ability to harness them is not!</p>
<p>Don’t get me wrong: I’d never want to go back to the time when you could only run one program at a time, and that program could crash your computer by doing something it shouldn’t. Modern systems and tools are <em>so</em> much better.</p>
<hr />
<p>The first video I posted in the series is about how Sierra game graphics worked, and my attempts to upscale them. It’s the most popular thing I’ve posted so far and I’ve been delighted with the reception.</p>
<iframe width="560" height="315" src="https://www.youtube.com/embed/sclZDCjUVvI" title="YouTube video player" frameborder="0" allow="accelerometer; autoplay; clipboard-write; encrypted-media; gyroscope; picture-in-picture" allowfullscreen=""></iframe>
<p>Sierra games were amongst my favorites growing up, and as an adult I was very curious about how they worked, so after much googling I found myself on the <a href="https://wiki.scummvm.org/index.php?title=AGI/Specifications">ScummVM AGI Specifications</a> page. Once I discovered that the formats were vector based, I had to make an attempt to upscale them. It was a lot of fun to work on and I’m glad people enjoyed it.</p>
<hr />
<p>My next video was much more personal. I decided to revisit a childhood game of mine and upgrade its graphics. My first attempts were within the limits of the PC architecture of the time and I explored strategies like palette cycling and 8086 assembly language code for performance.</p>
<iframe width="560" height="315" src="https://www.youtube.com/embed/08RveywBXl4" title="YouTube video player" frameborder="0" allow="accelerometer; autoplay; clipboard-write; encrypted-media; gyroscope; picture-in-picture" allowfullscreen=""></iframe>
<p>The video culminates with me throwing away the old techniques and rebuilding it in the <a href="https://godotengine.org/">Godot</a> engine.</p>
<p>This was my longest video yet, and I experimented with more animations and lengthy transitions to give it more of a documentary feel. At first very few watched it, but it’s picked up quite a few more views over time. I’m proud of it.</p>
<hr />
<p>Godot has been such fun for me to work with. I wondered if I could use it to bridge backwards and emulate old games within the engine. That exploration led to me learning GDNative and using C to embed old AGI games within it.</p>
<iframe width="560" height="315" src="https://www.youtube.com/embed/JMWnAJXnrW4" title="YouTube video player" frameborder="0" allow="accelerometer; autoplay; clipboard-write; encrypted-media; gyroscope; picture-in-picture" allowfullscreen=""></iframe>
<p>Again, this video hasn’t done the rounds as the previous two, but I love the result anyway. It’s so cool to boot up the 3D computer, flip the switch and hear those PC sounds as the Sierra game loads up and plays perfectly.</p>
<p>Even though it’s not popular, I am quite tempted to keep working on it. I have a few ideas of other retro projects that could load within it.</p>
<p>As for the future of my channel - I’m not quite sure! I’ve found the algorithm and what people will be interested in watching quite unpredictable. I am currently working on a puzzle game and I know that in the future I will be featuring that. Otherwise, I’ll see what random moods and thoughts inspire me.</p>Over the last couple of months, I’ve been posting a series of videos about early PC gaming and programming on my YouTube Channel. It’s been quite a fun journey and I thought I might write a few words about it.Home streaming my games in 4K2019-12-23T00:00:00+00:002019-12-23T00:00:00+00:00https://eviltrout.com/2019/12/23/home-streaming-in-4k<p>I have two 4K TVs at home, one in the bedroom and one in my living room. Both have consoles attached. One has a
<a href="https://amzn.to/374Rd6V">PS4 Pro</a> and one has a <a href="https://amzn.to/2ET1VkF">Nintendo Switch</a> and both are great.</p>
<p>However, I also have a gaming PC in my office, which is quite powerful compared to any home console
Mine is equipped with a <a href="https://amzn.to/2EPnTW2">RTX 2080 TI</a> GPU, and a <a href="https://amzn.to/2sTFjOq">9900K</a> processor, so it can play pretty much any new game at 4K with a solid 60 fps on high settings.</p>
<p>For any cross platform AAA game, my PC is going to provide a signficantly better experience than the consoles.
It can do 4K without upscaling techniques like <a href="https://en.wikipedia.org/wiki/Checkerboard_rendering">checkerboard rendering</a>. Also, thanks to a <a href="https://amzn.to/35RjnSH">NVMe M.2 SSD</a>,
it loads games and levels significantly faster than any home console.</p>
<p>PC Gaming is a little more janky than console gaming, as there can be things to mess with like device drivers,
but given the chance I always prefer it. Having said that, I also work from
home, and sometimes I want to get out of my office and play games on the couch instead of an office chair.</p>
<p>I briefly considered building a second gaming PC to attach to one of my TVs, but that’s a considerable cost
just for the ability to game on my couch or bed. Instead, I decided to try out streaming from my home PC to
my other TVs.</p>
<p>I buy most of my PC games on Steam, and it has a product called <a href="https://store.steampowered.com/steamlink/about/">Steam Link</a> which allows you to stream games from your PC to other devices.</p>
<h4 id="apple-tv-4k-setup">Apple TV 4K: Setup</h4>
<p>I already owned an <a href="https://www.apple.com/ca/apple-tv-4k">Apple TV 4K</a>, which <a href="https://support.apple.com/en-us/HT210414">as of tvOS 13</a> allows you to pair
game controllers and supports the Steam Link app. Installing the Steam Link app is quite easy as it’s available
in the App Store, but I did have a little more trouble getting my <a href="https://amzn.to/2PO3Op8">Xbox One controller</a> to work. tvOS found it
and I was quickly able to navigate the menus, but the Steam Link app insited on being taught the buttons on the
controller. In theory that should have been easy and simply involved pressing each button one at a time, but
several of the buttons would quit out of the Steam Link app and back to the home screen! Fortunately I found a <a href="https://www.youtube.com/watch?v=NGGnC5JvVLk">workaround</a> and was up and running.
Afer that, Steam launched quickly and my games worked perfectly.</p>
<h4 id="apple-tv-4k-performance">Apple TV 4K: Performance</h4>
<p>The Apple TV initially showed a lot of promise. My games were up and running in seconds in 4K and as I moved
the camera around they sure felt like they were running at 60 frames per second.</p>
<p>Unfortunately, as I started to actually play the games, I got the feeling that something was wrong. I was playing
<a href="https://en.wikipedia.org/wiki/Prey_(2017_video_game)">Prey</a>, a first person shooter, and the reticule
just felt <em>off</em>. It was so subtle that I was wondering if I was making it up or if it was something I was
really experiencing. I lowered the resolution to 1440p and it felt much better and playable.</p>
<p>I tried another game, <a href="https://en.wikipedia.org/wiki/Resident_Evil_2_(2019_video_game)">Resident Evil 2</a>, which
is a 3rd person shooter, and felt similar problems aiming at zombies.</p>
<p>To help debug, I was very happy to see that Steam Link has an option to display a visual overlay with lots of
extra information. While playing Resident Evil 2 in 4K, it reported that it was playing at 60 frames per second, with 1.5ms of input latency and 80ms of display latency.</p>
<p>At 60 frames per second, there should be one frame rendered to screen every 16ms or so. The display latency being 80ms meant I was about 5 frames behind which is not ideal, but the way steam split off display and input latency
implied to me that it prioritized my input over the display. 1.5ms delay is such a small number that it was
hard to believe I was noticing anything at all, as it should have been receving my input quite frequently.</p>
<h4 id="improving-my-network-the-overhead-of-moca">Improving my Network: The Overhead of MoCA</h4>
<p>At this point I decided to see if there was any way I could lower those numbers. Like most people, I have a
wireless network, but I have hard wired several points. My Apple TV and computer were connected via Gb Ethernet,
which is quite fast.</p>
<p>However, something came to mind about my setup: My TVs were near Coax outlets and had no ethernet jacks.
To wire them, I’d set up <a href="https://amzn.to/2t0qs4A">MoCA</a> adapters.</p>
<center><img src="/images/moca-network.png" alt="network diagram" class="diagram" /></center>
<p>This meant that for my PC to reach my Apple TV, it had to go from my PC to a Gb switch, then to my router,
then into a MoCA adapter in the closet, then through my Coax cables, out into another MoCA adapter, finally
to a switch and then into the Apple TV.</p>
<p>That’s quite a lot of steps! As a test, I pinged my NAS near my router that did not need to go through the MoCA
adapters and received a response time of <1ms. Pinging my Apple TV was taking around 3-4ms for the round
trip, which lines up with the 1.5ms that Steam was reporting. It seems that performing the conversion from
ethernet into Coax and back took about 1.5ms each way!</p>
<p>Now I should report for most networks, that’s a perfectly acceptable ping. However, I wanted my
streaming experience to be better, so I thought of ways I might remove them.</p>
<p>I’ve not had a cable subscription for over a decade, so I wondered how challenging it would be to replace
my Coax outlets with ethernet jacks. To do this, I unscrewed the face plates for the Coax outlets, and disconnected
the cables from the plates. I then went into my closet where they connect to a splitter and pulled
slightly. Surprisingly, they moved!</p>
<p>I bought a box of <a href="https://amzn.to/2ZhqYHE">bulk Cat5e</a>, and duct taped one end of the bulk cable to the Coax cable, then pulled
from the other side. I got a little worried at one point when it jammed, but I pulled harder and it jogged
loose and came through. The next part was more challenging for me: I had to attach jacks to each cable.
This task is quite challenging as each twisted pair in the ethernet cable has to be weaved into an impossibly
small area before you punch it down, but it was even harder for me because I’m <a href="https://en.wikipedia.org/wiki/Color_blindness">colourblind</a>. I ended up getting a somewhat crappy app on my phone that I could point at a cable
and it would give me a rough idea what colour it was.</p>
<p>After hours of frustration that involved rebuilding and testing the cables over and over, things worked
perfectly. I powered everything on, and Steam was reporting less than 1ms of input delay, and my display
latency had dropped to 65ms.</p>
<p>Another thing I wondered was whether the switches were adding latency. I experimented with 1 switch instead of
2 and even a direct connection and noticed no measuable difference. Any overhead added by a switch did not
seem to matter in my experiments.</p>
<h4 id="apple-tv-gigabit-performance">Apple TV: Gigabit Performance</h4>
<p>I did immediately notice an improvement with the better input and display latency. Things were now much more
playable than before. However, I’d already made some progress at improving those numbers and I wasn’t out of
ideas! The display latency was still quite high, and I couldn’t help but think the AppleTV was not the ideal
device for doing this.</p>
<p>To test my theory, I connected my laptop to the same network port. My laptop does not have ethernet, so I
had to use a USB-C ethernet adapter. When I started up Steam I had better results! Input latency was the
same but the display latency dropped from 65ms to 55ms. This was surprising indeed because I have always
heard that USB ethernet cards are quite bad. I decided to investigate alternatives to the Apple TV.</p>
<h4 id="nvidia-shield-tv-pro-setup">nVidia Shield TV Pro: Setup</h4>
<p>On paper, the <a href="https://amzn.to/34M8QGV">nVidia Shield TV Pro</a> is not much better than an AppleTV. Both have 3GB of RAM, and in this <a href="https://www.notebookcheck.net/X1-vs-A10X-Fusion_6612_9162.247596.0.html">CPU benchmark I found</a> the AppleTV processor seems superior. Still, I decided to give it a go.</p>
<center><img src="/images/nvidia-shield.jpg" alt="nVidia Shield TV Pro" /></center>
<p>The setup was quite good. It’s an Android TV device and I find the overall UX not as polished as tvOS, but
definitely not unpleasant. It paired with my Xbox One controller right away and the Steam Link app
had no issue using it without the annoying button configuration I had to do on the Apple TV.</p>
<p>Without any changes to the base configuration, I was streaming at 4K with sub-1ms input latency and 45ms
display latency - the best results I’d seen to date.</p>
<p>I was able to improve things even further after experimenting with the settings. I’d turned on <a href="https://en.wikipedia.org/wiki/High_Efficiency_Video_Coding">HEVC</a> streaming
because on the AppleTV I noticed a minor improvement. However, on the shield I found that by disabling
HEVC, my display latency dropped into the 35-40ms range.</p>
<p>With these numbers, not only were all my games playable at 40k@60, but I couldn’t notice any lag
whatsoever. I’d found my perfect setup. Since then I’ve put in over a dozen hours of streaming various
kinds of games and it works like a dream.</p>
<h4 id="moonlight">Moonlight</h4>
<p>Update Nov 26, 2020: I’ve since found that <a href="https://play.google.com/store/apps/details?id=com.limelight&hl=en_CA&gl=US">Moonlight Game Streaming</a>
works better than Steam’s streaming, and has the benefit of being able to stream games that come from
other stores than Steam.</p>
<p>I now recommend running nVidia’s streaming with Moonlight as a client. It can do 4k@60Hz for any game
with no noticeable lag.</p>
<h4 id="tldr">TL;DR</h4>
<ul>
<li>The best setup for in-home 4K, 60fps game streaming is a gigabit wired ethernet with an nVidia Shield TV</li>
<li>Do whatever you can to minimize the ping time between your PC and streaming device</li>
<li>Use nVidia’s streaming to Moonlight.</li>
</ul>I have two 4K TVs at home, one in the bedroom and one in my living room. Both have consoles attached. One has a PS4 Pro and one has a Nintendo Switch and both are great.Building a Home Media Server with Docker Compose2019-07-03T00:00:00+00:002019-07-03T00:00:00+00:00https://eviltrout.com/2019/07/03/building-home-media-download-server<p>I recently moved into a bigger condo which had a wall mount pre-installed in the bedroom for a TV, so I decided
to take advantage of it and bought a <a href="https://www.amazon.ca/gp/product/B07DY5152H/ref=as_li_tl?ie=UTF8&camp=15121&creative=330641&creativeASIN=B07DY5152H&linkCode=as2&tag=eviltrout-20&linkId=08b5818587b23a3ca62b39e33a847386">second TV</a>.</p>
<p>Previously, I was using an <a href="https://www.amazon.ca/gp/product/B01N2UMKZ5/ref=as_li_tl?ie=UTF8&camp=15121&creative=330641&creativeASIN=B01N2UMKZ5&linkCode=as2&tag=eviltrout-20&linkId=b50817b8e8ea4b8eccc3dea40a8fd530">Intel NUC</a> attached to my TV and running <a href="https://libreelec.tv/">LibreElec</a>. If you’ve not heard of LibreElec, it’s a very cool minimal Linux OS that sets itself up to run <a href="https://kodi.tv/">Kodi</a>. It is remarkably easy to set up, and even found and setup my obscure USB remote control automatically.</p>
<p>However, with my second TV that setup wasn’t going to cut it, so I decided to repurpose the NUC as a home media server.</p>
<h3 id="the-plan">The Plan</h3>
<ol>
<li>
<p>Move the Intel NUC into my closet with the router and run it as a headless server.</p>
</li>
<li>
<p>Stream content to each TV, as well as any phone or tablet via WiFi.</p>
</li>
<li>
<p>Have the ability to download additional content via torrents and newsgroups.</p>
</li>
<li>
<p>The content should be accessible via a Samba share so computers on my network can read/write it.</p>
</li>
</ol>
<h3 id="customizing-the-nuc">Customizing the NUC</h3>
<p>The Intel NUC I purchased has a M.2 slot for a SSD, which I highly recommend as a boot drive. If you buy a NUC, be careful <strong>NOT</strong> to get the one that has 16GB of Optane “Memory.” That is not actual RAM like you might expect and it eats up your M.2 SSD slot. I recommend a <a href="https://www.amazon.ca/gp/product/B0781Z7Y3S/ref=as_li_tl?ie=UTF8&camp=15121&creative=330641&creativeASIN=B0781Z7Y3S&linkCode=as2&tag=eviltrout-20&linkId=47e0432030da7cc0aa37aaffdebc1e2e">Samsung SSD</a>.</p>
<p>It does not come with RAM. You probably want 8GB but <a href="https://www.amazon.ca/gp/product/B01HI14AZ4/ref=as_li_tl?ie=UTF8&camp=15121&creative=330641&creativeASIN=B01HI14AZ4&linkCode=as2&tag=eviltrout-20&linkId=4de6aca646d55011846b09fc74ff78f4">16GB would be nicer</a>.</p>
<p>You will want a second hard drive for your media in the NUC’s 2.5” slot. My current build has a 2TB drive but if I was building one today I’d throw in a <a href="https://www.amazon.ca/gp/product/B01M0AADIX/ref=as_li_tl?ie=UTF8&camp=15121&creative=330641&creativeASIN=B01M0AADIX&linkCode=as2&tag=eviltrout-20&linkId=b745f96fce63aedb721290672bfa4991">5TB 2.5” Drive</a>. I am not a huge data hoarder so 2TB has lasted me a while but your mileage may vary.</p>
<h3 id="installing-the-os-and-docker">Installing the OS and Docker</h3>
<p>For my build, I used <a href="https://ubuntu.com/download/desktop">Ubuntu 64-Bit</a> but honestly in this setup we’re going to use docker for everything so as long as it can install docker and docker compose easily, you’ll be good to go.</p>
<p>You’ll want to attach a keyboard, mouse and monitor until it’s all working.</p>
<p>Create a <a href="https://tutorials.ubuntu.com/tutorial/tutorial-create-a-usb-stick-on-windows#0">USB stick</a> and boot into it on your NUC and do the basic install. You can pretty much use the defaults for everything.</p>
<p>Once everything is installed, we can <a href="https://docs.docker.com/install/linux/docker-ce/ubuntu/">install docker</a> following the standard instructions.</p>
<p>The last thing you’ll want to do is format and mount the second hard drive. If you installed a Desktop linux this is <a href="https://askubuntu.com/a/125277">pretty straightforward</a>. In the examples below I’m going to assume you mounted the drive at <code class="language-plaintext highlighter-rouge">/storage</code></p>
<h3 id="permissions">Permissions</h3>
<p>One thing I’d recommend doing to make your life a lot easier is to set really global permissions on the <code class="language-plaintext highlighter-rouge">/storage</code> hard drive:</p>
<div class="language-plaintext highlighter-rouge"><div class="highlight"><pre class="highlight"><code>sudo chown nobody:nogroup /storage
sudo chmod 0777 /storage
</code></pre></div></div>
<p>To be clear, this means anyone can read/write the files there. This is a huge security compromise, and I am comfortable with it because I am only storing media on the drive and not anything of importance. I can trust anyone with access to my local network to read and write files in that drive because it’ll just be movie files. Make sure this is safely behind your router and not visible to the external world.</p>
<h3 id="the-magic-of-docker-compose">The Magic of Docker Compose</h3>
<p>Here’s where things get really fun. In the past, I’ve found setting up and installing server applications somewhat difficult. For example, <a href="https://sonarr.tv/">sonarr</a> uses Mono. I’m a developer but I’ve never used .NET and have no idea how to configure and set up a server in that framework. Usually you can follow the instructions in the README, but debugging snags can be difficult when you don’t know what you’re doing.</p>
<p>Additionally, you have to familiarize yourself with the init/startup scripts for every OS to make sure each server application boots up properly and will restart in the event of a crash.</p>
<p>Finally, updating each server application has a slightly different process. Some are one click via the web app, but others involve package managers or even recompiling code depending on how you installed them.</p>
<p><a href="https://docs.docker.com/compose/">Docker Compose</a> solves all these problems, and it does it with a remarkably small amount of configuration.</p>
<p>Here’s a simple example which installs sonarr and <a href="https://nzbget.net/">nzbget</a>:</p>
<div class="language-yaml highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="nn">---</span>
<span class="na">version</span><span class="pi">:</span> <span class="s2">"</span><span class="s">3"</span>
<span class="na">services</span><span class="pi">:</span>
<span class="na">sonarr</span><span class="pi">:</span>
<span class="na">image</span><span class="pi">:</span> <span class="s">linuxserver/sonarr</span>
<span class="na">container_name</span><span class="pi">:</span> <span class="s">sonarr</span>
<span class="na">depends_on</span><span class="pi">:</span>
<span class="pi">-</span> <span class="s">nzbget</span>
<span class="na">environment</span><span class="pi">:</span>
<span class="pi">-</span> <span class="s">PUID=1000</span>
<span class="pi">-</span> <span class="s">PGID=1001</span>
<span class="pi">-</span> <span class="s">TZ=America/Toronto</span>
<span class="pi">-</span> <span class="s">UMASK_SET=022</span> <span class="c1">#optional</span>
<span class="na">volumes</span><span class="pi">:</span>
<span class="pi">-</span> <span class="s">/docker/sonarr/config:/config</span>
<span class="pi">-</span> <span class="s">/storage/tv:/tv</span>
<span class="pi">-</span> <span class="s">/docker/nzbget/downloads:/downloads</span>
<span class="na">ports</span><span class="pi">:</span>
<span class="pi">-</span> <span class="s">8989:8989</span>
<span class="na">restart</span><span class="pi">:</span> <span class="s">always</span>
<span class="na">nzbget</span><span class="pi">:</span>
<span class="na">image</span><span class="pi">:</span> <span class="s">linuxserver/nzbget</span>
<span class="na">container_name</span><span class="pi">:</span> <span class="s">nzbget</span>
<span class="na">environment</span><span class="pi">:</span>
<span class="pi">-</span> <span class="s">PUID=1000</span>
<span class="pi">-</span> <span class="s">PGID=1001</span>
<span class="pi">-</span> <span class="s">TZ=America/Toronto</span>
<span class="na">volumes</span><span class="pi">:</span>
<span class="pi">-</span> <span class="s">/docker/nzbget/config:/config</span>
<span class="pi">-</span> <span class="s">/docker/nzbget/downloads:/downloads</span>
<span class="na">ports</span><span class="pi">:</span>
<span class="pi">-</span> <span class="s">6789:6789</span>
<span class="na">restart</span><span class="pi">:</span> <span class="s">always</span>
</code></pre></div></div>
<p>A few notes:</p>
<ul>
<li>
<p>I installed all my docker stuff in <code class="language-plaintext highlighter-rouge">/docker</code> but you can put it anywhere you like. Make sure to create the <code class="language-plaintext highlighter-rouge">/docker/sonarr</code>, <code class="language-plaintext highlighter-rouge">docker/nzbget</code> etc directories before you start it up so that the applications have a place to write their files. Note when setting up <code class="language-plaintext highlighter-rouge">volumes</code> that the paths on the left hand side of the colon are on your local linux box and can be changed. The right hand side is where they’ll be visible to the server application.</p>
</li>
<li>
<p><code class="language-plaintext highlighter-rouge">sonarr</code> has an attribute, <code class="language-plaintext highlighter-rouge">depends_on</code> for <code class="language-plaintext highlighter-rouge">nzbget</code>. This means that the nzbget application will start up before sonarr, which is nice because sonarr needs a program to download from newsgroups.</p>
</li>
<li>
<p><code class="language-plaintext highlighter-rouge">restart: always</code> means that the server applications will be restarted if they crash. They’ll also restart automatically if you reboot your computer.</p>
</li>
</ul>
<p>The images in this docker compose configuration come from <a href="https://www.linuxserver.io/">linuxserver.io</a>, which I’ve found to be reliable and trustworthy. You should find images that suit your purposes and that you trust.</p>
<p>To install everything, go into the same folder as your <code class="language-plaintext highlighter-rouge">docker-compose.yml</code> and type <code class="language-plaintext highlighter-rouge">docker compose up -d</code>. It’ll download the images and start everything. You can then visit <code class="language-plaintext highlighter-rouge">http://your-server-name:6789</code> and <code class="language-plaintext highlighter-rouge">http://your-server-name:8989</code> and start setting things up to your preferences.</p>
<p>How easy is that?</p>
<h3 id="adding-more-applications">Adding more applications</h3>
<p>In order to stream content to my TVs, I use <a href="https://www.plex.tv">Plex</a>, which can be installed just as easily in the same configuration:</p>
<div class="language-yaml highlighter-rouge"><div class="highlight"><pre class="highlight"><code> <span class="na">plex</span><span class="pi">:</span>
<span class="na">image</span><span class="pi">:</span> <span class="s">linuxserver/plex</span>
<span class="na">container_name</span><span class="pi">:</span> <span class="s">plex</span>
<span class="na">network_mode</span><span class="pi">:</span> <span class="s">host</span>
<span class="na">environment</span><span class="pi">:</span>
<span class="pi">-</span> <span class="s">PUID=1000</span>
<span class="pi">-</span> <span class="s">PGID=1001</span>
<span class="pi">-</span> <span class="s">VERSION=docker</span>
<span class="na">volumes</span><span class="pi">:</span>
<span class="pi">-</span> <span class="s">/docker/plex/config:/config</span>
<span class="pi">-</span> <span class="s">/storage/tv:/data/tvshows</span>
<span class="pi">-</span> <span class="s">/storage/movies:/data/movies</span>
<span class="pi">-</span> <span class="s">/docker/plex/transcode:/transcode</span>
<span class="na">restart</span><span class="pi">:</span> <span class="s">always</span>
</code></pre></div></div>
<p>Plex is well supported by many streaming sticks, set top boxes and even smart TVs. Personally I am using it via <a href="https://firecore.com/infuse">Infuse</a> on <a href="https://www.amazon.ca/gp/product/B078865VC3/ref=as_li_tl?ie=UTF8&camp=15121&creative=330641&creativeASIN=B078865VC3&linkCode=as2&tag=eviltrout-20&linkId=c8657876e9e9c98601657509faae1d98">Apple TV</a> but there are many options depending on your device of choice.</p>
<p>Another nice application is <a href="https://www.qbittorrent.org/">qBittorrent</a>:</p>
<div class="language-yaml highlighter-rouge"><div class="highlight"><pre class="highlight"><code> <span class="na">qbittorrent</span><span class="pi">:</span>
<span class="na">image</span><span class="pi">:</span> <span class="s">linuxserver/qbittorrent</span>
<span class="na">container_name</span><span class="pi">:</span> <span class="s">qbittorrent</span>
<span class="na">environment</span><span class="pi">:</span>
<span class="pi">-</span> <span class="s">PUID=1000</span>
<span class="pi">-</span> <span class="s">PGID=1001</span>
<span class="pi">-</span> <span class="s">TZ=America/Toronto</span>
<span class="pi">-</span> <span class="s">UMASK_SET=022</span>
<span class="pi">-</span> <span class="s">WEBUI_PORT=8080</span>
<span class="na">volumes</span><span class="pi">:</span>
<span class="pi">-</span> <span class="s">/docker/qbittorrent/config:/config</span>
<span class="pi">-</span> <span class="s">/docker/qbittorrent/downloads:/downloads</span>
<span class="pi">-</span> <span class="s">/storage/tv:/tv</span>
<span class="pi">-</span> <span class="s">/storage/movies:/movies</span>
<span class="na">ports</span><span class="pi">:</span>
<span class="pi">-</span> <span class="s">31321:31321</span>
<span class="pi">-</span> <span class="s">31321:31321/udp</span>
<span class="pi">-</span> <span class="s">8080:8080</span>
<span class="na">restart</span><span class="pi">:</span> <span class="s">always</span>
</code></pre></div></div>
<p>In this case, I have set up port 31321 with port forwarding on my router to make certain torrent services
happy. You can choose any port you want there and set it up in qBittorent accordingly.</p>
<h3 id="setting-up-a-samba-share">Setting up a Samba Share</h3>
<p>If you want to set up a public samba share, I found a great little image for that too! Add the following to your <code class="language-plaintext highlighter-rouge">docker-compose.yml</code>:</p>
<div class="language-yaml highlighter-rouge"><div class="highlight"><pre class="highlight"><code> <span class="na">samba</span><span class="pi">:</span>
<span class="na">image</span><span class="pi">:</span> <span class="s">jenserat/samba-publicshare</span>
<span class="na">tty</span><span class="pi">:</span> <span class="no">true</span>
<span class="na">environment</span><span class="pi">:</span>
<span class="pi">-</span> <span class="s">PUID=1000</span>
<span class="pi">-</span> <span class="s">PGID=1001</span>
<span class="na">ports</span><span class="pi">:</span>
<span class="pi">-</span> <span class="s">445:445</span>
<span class="pi">-</span> <span class="s">137:137</span>
<span class="pi">-</span> <span class="s">138:138</span>
<span class="pi">-</span> <span class="s">139:139</span>
<span class="na">volumes</span><span class="pi">:</span>
<span class="pi">-</span> <span class="s">/storage:/srv</span>
</code></pre></div></div>
<p>Now you’re off to the races, any computer on the network can access storage to add/remove files as they like.</p>
<h3 id="remembering-all-those-ports">Remembering all those ports</h3>
<p>If you’re like me, you’ll have trouble remembering all the custom port numbers for your web applications.
I decided to install <a href="https://www.nginx.com/">nginx</a> on port 80 so that it would be the default site when I accessed the default port of my media server. On that site I added a simple HTML index page that links to the various services:</p>
<div class="language-yaml highlighter-rouge"><div class="highlight"><pre class="highlight"><code> <span class="na">nginx</span><span class="pi">:</span>
<span class="na">image</span><span class="pi">:</span> <span class="s">linuxserver/nginx</span>
<span class="na">container_name</span><span class="pi">:</span> <span class="s">nginx</span>
<span class="na">environment</span><span class="pi">:</span>
<span class="pi">-</span> <span class="s">PUID=1000</span>
<span class="pi">-</span> <span class="s">PGID=1001</span>
<span class="pi">-</span> <span class="s">TZ=America/Toronto</span>
<span class="na">volumes</span><span class="pi">:</span>
<span class="pi">-</span> <span class="s">/docker/nginx/config:/config</span>
<span class="na">ports</span><span class="pi">:</span>
<span class="pi">-</span> <span class="s">80:80</span>
<span class="pi">-</span> <span class="s">443:443</span>
<span class="na">restart</span><span class="pi">:</span> <span class="s">always</span>
</code></pre></div></div>
<p>My index.html looks like this (my server is named <code class="language-plaintext highlighter-rouge">datacube</code>):</p>
<div class="language-html highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="nt"><html></span>
<span class="nt"><head></span>
<span class="nt"><title></span>datacube<span class="nt"></title></span>
<span class="nt"></head></span>
<span class="nt"><body></span>
<span class="nt"><div</span> <span class="na">class=</span><span class="s">"message"</span><span class="nt">></span>
<span class="nt"><h1></span>datacube<span class="nt"></h1></span>
<span class="nt"><ul></span>
<span class="nt"><li><a</span> <span class="na">href=</span><span class="s">"http://datacube.local:8080/"</span><span class="nt">></span>qBittorrent<span class="nt"></a></li></span>
<span class="nt"><li><a</span> <span class="na">href=</span><span class="s">"http://datacube.local:6789/"</span><span class="nt">></span>NZBGet<span class="nt"></a></li></span>
<span class="nt"><li><a</span> <span class="na">href=</span><span class="s">"http://datacube.local:8989/"</span><span class="nt">></span>Sonarr<span class="nt"></a></li></span>
<span class="nt"><li><a</span> <span class="na">href=</span><span class="s">"http://datacube.local:32400/"</span><span class="nt">></span>Plex<span class="nt"></a></li></span>
<span class="nt"></ul></span>
<span class="nt"></div></span>
<span class="nt"></body></span>
<span class="nt"></html></span>
</code></pre></div></div>
<h3 id="conclusion">Conclusion</h3>
<p>In the end I was blown away about how easy Docker Compose makes setting up all these various services, and keeping them running despite crashes and reboots. If you’re looking to custom build your own media server or download box I highly recommend this approach.</p>I recently moved into a bigger condo which had a wall mount pre-installed in the bedroom for a TV, so I decided to take advantage of it and bought a second TV.Candles, Programming and Archetypes2018-05-22T00:00:00+00:002018-05-22T00:00:00+00:00https://eviltrout.com/2018/05/22/candles-programming-archetypes<p>Once in a while, I daydream about being thrown back in time. I’d have no Wikipedia, no books, or any
access to information except what’s already in my head.</p>
<p>If I were thrown into Victorian London, what could I do? What could I teach them?</p>
<p>Well, the first thing I would do is tell them to wash their hands. With soap. Frequently. That would probably
be the most significant contribution I could make.</p>
<p>Beyond that, I’m a computer programmer. I like to think I know my way around a web browser pretty well,
but that’s pretty useless knowledge in a world without computers.</p>
<p>It can be a fun exercise to start with something you know, say, TypeScript, and work your way backwards
as far as you can:</p>
<ul>
<li>
<p>TypeScript is a high level languge and is transpiled into Javascript</p>
</li>
<li>
<p>Javascript is a language that is interpreted by a Web Browser</p>
</li>
<li>
<p>A Web Browser is a native program, probably written in C++</p>
</li>
<li>
<p>C++ is a programming language, which is compiled into machine language</p>
</li>
<li>
<p>The machine language is run and prioritized by an operating system</p>
</li>
<li>
<p>The operating system passes streams of machine language to the CPU</p>
</li>
<li>
<p>The CPU runs the streams out of order, interfacing with other devices on the computer</p>
</li>
<li>
<p>The computer is a collection of standardized components, powered by electricity</p>
</li>
<li>
<p>Electricity is… etc (gotta stop somewhere!)</p>
</li>
</ul>
<p>I wrote the above steps off the top of my head, and I’m fully aware there are huge gaps in my knowledge,
and probably many inaccuracies in the way I’ve written it out. This is normal in our specialized modern world.
There will always be far more things at play than you’ll ever have time to understand.</p>
<p>Having said that, there is no reason you can’t spend <em>some</em> time looking back and trying to fill in
your own personal gaps. I’ve become a huge fan of <a href="http://www.the8bitguy.com/">The 8-Bit Guy</a> on Youtube
for this kind of thing. Here’s one of his best videos, about how graphics worked on “oldschool” systems:</p>
<iframe width="560" height="315" src="https://www.youtube.com/embed/Tfh0ytz8S0k" frameborder="0" allow="autoplay; encrypted-media" allowfullscreen=""></iframe>
<hr />
<p>Over the last year or so I’ve made it somewhat of a hobby of mine to find old computer manuals and briefly
tour their programming guides.</p>
<p>You hear a lot about the challenges of programming old systems. It’s true that they had limited memory and
processing power, and programs also loaded slowly from disk. But there’s also a <strong>simplicity</strong> to them
that is beautiful.</p>
<p>A professor of mine once said something interesting about <a href="https://en.wikipedia.org/wiki/Archetype">Archetypes</a> (paraphrased):</p>
<blockquote>
<p>A candle is an archetype of the light bulb. You use it to create <em>darkness</em>, not light.</p>
</blockquote>
<hr />
<p>On an old computer such as a <a href="https://en.wikipedia.org/wiki/Commodore_64">Commodore 64</a>,
there’s nothing preventing you from accessing any piece of memory or IO device.</p>
<p>Booting a Commodore 64 (which you can now do in <a href="https://virtualconsoles.com/online-emulators/c64/">a browser</a>!)
will present you with something like this:</p>
<p><img src="/images/c64-boot.png" /></p>
<p>This is <a href="https://en.wikipedia.org/wiki/BASIC">BASIC</a>, an early high level programming language. But its
simplicity doesn’t mean it can’t access the computer directly! If you type in <code class="language-plaintext highlighter-rouge">POKE 53280, 2</code>, that will write
the value <code class="language-plaintext highlighter-rouge">2</code> into the Commodore 64’s memory at address <code class="language-plaintext highlighter-rouge">53280</code>, which happens to be the video memory that
determines the border color of the screen:</p>
<p><img src="/images/c64-poke.png" /></p>
<p>To a modern programmer, this should be alarming. What’s to stop a bad program from reading memory it shouldn’t,
or accessing devices it shouldn’t?</p>
<hr />
<p>Candles are literally open flame. I remember during the
<a href="https://en.wikipedia.org/wiki/Northeast_blackout_of_2003">Northeast Blackout of 2003</a> the mayor coming on the
radio and asking people not to use candles out of fear of starting fires all over the city.</p>
<p>Used responsibly, though, a candle creates a beautiful darkness. I recently toured
<a href="https://en.wikipedia.org/wiki/Dennis_Severs%27_House">Denis Severs’ House</a> in London. It’s a visceral
arrangement of life in London through the centuries, and it’s almost entirely candlelit, which allows it to
alternate between warmth and spookiness.</p>
<p>I probably wouldn’t object if someone told me learning to program a Commodore 64 in 2018 is a nerdy
pursuit. I <em>would</em> object if they said it’s useless though; Programming is all about trade offs, and
our predecessors made some excellent ones given the technolgical constraints of the time.</p>
<p>If you have the time and the interest, I’d highly recommend spending some time learning about how obsolete
computers worked. I can’t promise it’ll make you a better programmer, but it will give you a better
perspective, and that’s something I think all of us could use these days.</p>Once in a while, I daydream about being thrown back in time. I’d have no Wikipedia, no books, or any access to information except what’s already in my head.The value of a thousand little features2017-04-13T00:00:00+00:002017-04-13T00:00:00+00:00https://eviltrout.com/2017/04/13/a-thousand-little-features<p>It’s been over a year since I <a href="https://eviltrout.com/2016/02/25/fixing-android-performance.html">wrote a blog entry</a>! And while of course the universal excuse of “I’ve been busy” applies, I think we reached a point in <a href="http://www.discourse.org/">Discourse</a>’s development where we just were able to focus on the product without a lot of stuff getting in our way.</p>
<p>I’ve now been working on Discourse full time for 5 years. In the beginning we had a lot more uncertainty about technical decisions. Some of the things we debated interally include:</p>
<ul>
<li>Whether to use Rails or Node.js (We chose Rails)</li>
<li>Whether to use an ORM or direct SQL (We use ActiveRecord)</li>
<li>Whether we could get away with a NOSQL database instead of Postgres</li>
<li>Whether to do server side rendering or embrace a front end MVC framework (We use Ember.js)</li>
<li>Whether to use CoffeeScript/Javascript (Javascript won, now ES2015)</li>
</ul>
<p> </p>
<p>Beyond those major discussions, we also went back and forth a lot about the details of our codebase:</p>
<ul>
<li>How much test coverage is enough?</li>
<li>How and when do we introduce internationalization?</li>
<li>How should the code base look (how much space, when to add new lines, etc)</li>
<li>How do we handle code reviews from within our team?</li>
</ul>
<p> </p>
<p>If you’re reading this as a developer, I’ll bet some of the above bullet points triggered some immediate opinions :)</p>
<p>The truth is that <strong>over time our team found its mojo</strong>. Issues we had with our major decisions eventually found workarounds or solutions. Members of our team whose priorities were different (performance focused vs code quality for example) eventually found a happy ground and we just became productive.</p>
<p>We also saw some excellent performance gains in Ember.js and V8, and both of those teams deserve some serious credit. Android performance still isn’t where I’d like it to be, but it’s improved significantly since the project began.</p>
<p>To be clear: we aren’t done changing our codebase. For example <a href="https://meta.discourse.org/t/revisiting-moving-to-typescript/60519/3?u=eviltrout">there’s interest in moving to TypeScript</a>, and we are pretty tired with Sprockets so we’ll be ditching that, plus we want to upgrade Ruby and Rails versions. Web technology continues to advance and we’re along for the ride!</p>
<h3 id="a-thousand-little-features">A thousand little features</h3>
<p>Once in a while, someone will ask me (in a polite way) something like “hey, after 5 years, what is left to do for forum software?”</p>
<p>From a 10,000ft view, it seems that Discourse already does what you’d expect forum software to do. We let users log in, read content and post replies. So what the heck is our team still working on?!?</p>
<p>There is a tendency in software engineering to oversimplify products you’re not familiar with. I remember reading a Hacker News article about Disourse a few years ago where someone posted something to the effect of:</p>
<blockquote>
<p>Why is the Discourse codebase so big? Aren’t they just taking text and showing it on a web page?</p>
</blockquote>
<p>Well, yes. But isn’t <em>every</em> web application ultimately just taking text and showing it on a web page? If it were that simple I think we would have packed it up a long time ago.</p>
<p>If you zoomed in closer on Discourse, say to the 5000ft view, you’d see that we have many big features we had to implement but might not seem obvious at first including: Sub-categories, User Groups, Theming, Moderator Tools, a Setup Wizard, etc). I think a lot of experienced developers would recognize that you’d need all that stuff as your product gets used by more people.</p>
<p>The thing that has been a great learning experience for me though, is the <em>thousands</em> of little features that most people don’t see.</p>
<p>When a user posts on Discourse, they’re presented with what is essentially a HTML <code class="language-plaintext highlighter-rouge"><textarea></code>. They fill it in, hit post and their content shows up in the topic.</p>
<p>What is not obvious, unless you’ve <em>actively used the software</em> is how many little features are layered on top of that to encourage what we think is good discussion.</p>
<p>For example (and this is nowhere near an exhaustive list):</p>
<ul>
<li>If your reply is too short, we’ll encourage you to say more. Short replies offer little value.</li>
<li>If your post links to something recently mentioned, we’ll say “Hey that was already posted x replies above you.”</li>
<li>If your new topic is similar to one that already exists, we link to it and tell you to check it out first before perhaps creating a duplicate.</li>
<li>If you start dominating the topic by replying too much, we ask you to slow down and give other people a chance to participate.</li>
<li>If you are only replying to the same person over and over, we suggest that you might be derailing the topic and perhaps should take it elsewhere like a private message.</li>
</ul>
<p> </p>
<p>Those thousands of little features really add up over a while. We’ve added them because we use Discourse every day and have seen patterns of usage in our own forums and the forums of our customers that we think we can improve.</p>
<p>Of course, a truly toxic user will find a way to be toxic regardless of how many times to discourage their bad behavior, but the vast majority of users are <em>not</em> toxic and will happily follow hints that lead them towards a better discussion. Many of our users will never see this work, but the ones who do will hopefully be pointed in the right direction.</p>
<p>I love that we can add these little things quicky to our codebase, and because we’re a web application we can deploy them to users right away to see their effects and tweak them over time. It has really changed a lot of my thinking of software development, where there traditionally seems to be a huge focus on BIG features to create value.</p>
<p>Maybe instead of spending months creating a big feature, spend a day or two knocking out a small one? Those little ones really add up, and can add up to make a big difference before you know it!</p>It’s been over a year since I wrote a blog entry! And while of course the universal excuse of “I’ve been busy” applies, I think we reached a point in Discourse’s development where we just were able to focus on the product without a lot of stuff getting in our way.We finally did something about Android Performance2016-02-25T00:00:00+00:002016-02-25T00:00:00+00:00https://eviltrout.com/2016/02/25/fixing-android-performance<p>Back in September, Codinghorror wrote a popular post on <a href="https://meta.discourse.org/t/the-state-of-javascript-on-android-in-2015-is-poor/33889">the state of android Javascript performance</a>
on Discourse’s Meta forum. It drew a lot of attention, and led to some fascinating discussions on our forum
and behind the scenes with browser engineers.</p>
<p><img src="/images/android-sad.png" align="right" /></p>
<p>The poor performance of Discourse on Android was already old news to us at that point: we started paying attention
several years ago, and have spent some time contributing to the Ember.js community
tools to help profile application performance, in particular the “Render Performance” tab on the
<a href="https://chrome.google.com/webstore/detail/ember-inspector/bmdblncegkenkacieihfhpjfppoconhi?hl=en">Ember Inspector</a>,
and the <a href="http://emberperf.eviltrout.com/">ember performance suite</a>.</p>
<p>Over time, we’ve seen some modest performance improvements in newer Ember.js releases. In particular,
the <a href="http://talks.erikbryn.com/htmlbars-emberconf/">HTMLBars</a> upgrade resulted in a roughly 25% boost. However,
rendering topics was still approximately 6 times slower on the top of the line Android device versus the equivalent
iPhone.</p>
<p>After years of waiting for a breakthrough in Android or Ember performance, we decided it was time to to take the
nuclear option: we replaced the Ember rendering engine for our most common view with a custom virtual DOM based
renderer.</p>
<p>The results have been <strong>fantastic</strong>:</p>
<h3 id="large-topic-rendering-speed-initial-visit">Large Topic Rendering Speed (Initial Visit)</h3>
<ul>
<li>Desktop
<ul>
<li>633ms avg (before)</li>
<li>120ms avg (after)</li>
</ul>
</li>
<li>Nexus 6p (high-end Android)
<ul>
<li>1248ms avg (before)</li>
<li>248ms avg (after)</li>
</ul>
</li>
<li>Nexus 7 (2013 mid-range Android)
<ul>
<li>4078ms avg (before)</li>
<li>636ms avg (after)</li>
</ul>
</li>
</ul>
<h3 id="large-topic-rendering-speed-subsequent-visits">Large Topic Rendering Speed (Subsequent Visits)</h3>
<ul>
<li>Desktop
<ul>
<li>429ms avg (before)</li>
<li>69ms avg (after)</li>
</ul>
</li>
<li>Nexus 6p (high-end Android)
<ul>
<li>710ms avg (before)</li>
<li>152ms avg (after)</li>
</ul>
</li>
<li>Nexus 7 (2013 mid-range Android)
<ul>
<li>2757ms avg (before)</li>
<li>350ms avg (after)</li>
</ul>
</li>
</ul>
<h3 id="summary">Summary</h3>
<p><img src="/images/android-happy.png" align="right" width="150" /></p>
<ul>
<li>
<p>Across all platforms, viewing a topic in Discourse averages <strong>5x</strong> faster.</p>
</li>
<li>
<p>On older Android devices, viewing a topic is between <strong>6-8x</strong> faster.</p>
</li>
<li>
<p>Our worst improvement was on the Nexus 6p on subsequent renders which was <strong>4.6x</strong> faster.</p>
</li>
</ul>
<p>On a Desktop PC, the speedup is a nice touch, mainly if you are an avid Discourse user.</p>
<p>On Android, it’s a <strong>huge quality of life improvement</strong>. Going from over 4 seconds to around
half a second completely changes the experience you have on a forum.</p>
<h2 id="diagnosing-performance-problems">Diagnosing Performance Problems</h2>
<p>Ember’s rendering system is made up of a hierarchy of <a href="https://guides.emberjs.com/v1.10.0/components/">components</a>.
Every template that you render is backed by an instance of a <code class="language-plaintext highlighter-rouge">Component</code> class, and it manages the lifecycle of the
template as well as delegating events to actions. Components can contain other components, and can use helpers to
control flow. Here’s what a typical template looks like:</p>
<h3 id="posthbs-for-rendering-a-single-post">post.hbs (for rendering a single post)</h3>
<div class="language-handlebars highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="k">{{</span><span class="nv">avatar-link</span> <span class="nv">user</span><span class="o">=</span><span class="nv">post</span><span class="p">.</span><span class="nv">creator</span><span class="k">}}</span>
<span class="k">{{</span><span class="nv">poster-name</span> <span class="nv">user</span><span class="o">=</span><span class="nv">post</span><span class="p">.</span><span class="nv">creator</span><span class="k">}}</span>
<span class="nt"><span</span> <span class="na">class=</span><span class="s">'date'</span><span class="nt">></span><span class="k">{{</span><span class="nv">post</span><span class="p">.</span><span class="nv">date</span><span class="k">}}</span><span class="nt"></span></span>
<span class="nt"><div</span> <span class="na">class=</span><span class="s">'post-body'</span><span class="nt">></span>
<span class="k">{{</span><span class="nv">post</span><span class="p">.</span><span class="nv">body</span><span class="k">}}</span>
<span class="k">{{</span><span class="nv">post-controls</span> <span class="nv">post</span><span class="o">=</span><span class="nv">post</span><span class="k">}}</span>
<span class="nt"></div></span>
<span class="nt"><div</span> <span class="na">class=</span><span class="s">'gutter'</span><span class="nt">></span>
<span class="k">{{</span><span class="nv">post-links</span> <span class="nv">links</span><span class="o">=</span><span class="nv">post</span><span class="p">.</span><span class="nv">links</span><span class="k">}}</span>
<span class="nt"></div></span>
</code></pre></div></div>
<p>In the above example, <code class="language-plaintext highlighter-rouge">avatar-link</code>, <code class="language-plaintext highlighter-rouge">poster-name</code>, <code class="language-plaintext highlighter-rouge">post-controls</code> and <code class="language-plaintext highlighter-rouge">post-links</code> are all custom components
that are displayed on every post. Each one would have its own handlebars template similar to <code class="language-plaintext highlighter-rouge">post.hbs</code>,
and probably also a Javascript file to handle clicks and other interactions with it.</p>
<p>Ember also handles data binding for you. In the above example, if the <code class="language-plaintext highlighter-rouge">post.author</code> or <code class="language-plaintext highlighter-rouge">post.body</code> changed,
the template would automatically update. During development, this means you spend a lot less time thinking about
how to update your user interface. If you want to change the body of a post, just call <code class="language-plaintext highlighter-rouge">post.set('body', 'new body')</code>
and you’re done!</p>
<p>Behind the scenes, Ember does a lot of work to wire all this up. It keeps tabs on all the properties that can
change, which components they belong to and so forth. If you’re only rendering a few posts, the overhead
involved in this is small, but as you render more things with more details, it can add up.</p>
<p>It can’t be understated that <strong>all performance issues are about tradeoffs</strong>. I think Ember does the right
thing here and focuses on apps that are organized and scale up as you add more features. For the vast
majority of views in your application, the framework overhead will not concern you. However, if you
are rendering many nested components with many bindings, the situation can become pathological.</p>
<p>While Discourse might look simple on the surface, each post is rendered with quite a few components.
We have many dynamic buttons, links and effects applied every time a post is rendered with
over 150 attributes involved.</p>
<h2 id="hindsight-is-2020">Hindsight is 20/20</h2>
<p>Every time you see a blog entry that says “by switching our codebase from language X to language Y,
we improved our performance by a factor of Z!” you should realize that there’s an important
detail omitted: When you re-write code, you have the full domain knowledge of the problem it solved
the first time around. This knowledge goes a long way towards making it faster, because you know exactly
what you need to do to get it to work.</p>
<p>For example: every time an Ember template encounters a ``, it sets up a binding to make sure the
template updates when the underlying property changes. This is a good default to have, but what
if you know that the property will only change following an AJAX request? You can use this
knowledge to render significantly faster.</p>
<p>This is the general approach we took to rewriting our topic rendering. We thought about the bare
minimum amount of work the browser needed to do to implement the interface we’d already built.
How maintainable would that code be? Was there some middle ground, where we could gain
significant performance but also keep the code reasonably clean and integrated with the rest
of our Ember app?</p>
<h2 id="wrapping-code-in-ember">Wrapping code in Ember</h2>
<p>Ember’s Components are also great tools to <a href="https://www.youtube.com/watch?v=S_l_DL8ysQQ">wrap third party Javascript</a>,
as they give you lifecycle hooks for when an element is inserted into the DOM and when it is removed.
You can use these hooks to render pretty much whatever you want as long as you’re using Javascript.</p>
<p>Our approach was to create a single component, instead of a series of nested components, and
when it was inserted in the DOM, to perform the rendering ourselves. We’d only re-render the
DOM if the user performed an action on it or if our <a href="https://github.com/SamSaffron/message_bus">message bus</a>
told us to do so.</p>
<p>We looked around at various options for rendering quickly on the client side, and found
<a href="https://github.com/Matt-Esch/virtual-dom">virtual-dom</a> to be very promising. It is only 8k
when minified and gzipped, which seemed like a reasonable amount of code in exchange for a
performance boost.</p>
<p>We didn’t want to sacrifice <em>too much</em> code quality, so we implemented a lightweight version
of Ember Components called Widgets. (note: yes, I know Widget is very similar to Component,
but a distinction was required and hey… naming is hard!)</p>
<p>Widget are classes that, given a series of attributes and optionally some state, would emit
a series of virtual DOM nodes to be rendered. If you wanted to render a button that
increased a counter on click, the widget would look like this:</p>
<div class="language-javascript highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="k">import</span> <span class="p">{</span> <span class="nx">createWidget</span> <span class="p">}</span> <span class="k">from</span> <span class="dl">'</span><span class="s1">discourse/widgets/widget</span><span class="dl">'</span><span class="p">;</span>
<span class="k">import</span> <span class="p">{</span> <span class="nx">h</span> <span class="p">}</span> <span class="k">from</span> <span class="dl">'</span><span class="s1">virtual-dom</span><span class="dl">'</span><span class="p">;</span>
<span class="k">export</span> <span class="k">default</span> <span class="nx">createWidget</span><span class="p">(</span><span class="dl">'</span><span class="s1">counter</span><span class="dl">'</span><span class="p">,</span> <span class="p">{</span>
<span class="na">tagName</span><span class="p">:</span> <span class="dl">'</span><span class="s1">button.counter-button</span><span class="dl">'</span><span class="p">,</span>
<span class="nx">defaultState</span><span class="p">()</span> <span class="p">{</span>
<span class="k">return</span> <span class="p">{</span> <span class="na">count</span><span class="p">:</span> <span class="mi">0</span> <span class="p">};</span>
<span class="p">},</span>
<span class="nx">html</span><span class="p">(</span><span class="nx">attrs</span><span class="p">,</span> <span class="nx">state</span><span class="p">)</span> <span class="p">{</span>
<span class="k">return</span> <span class="s2">`Clicked </span><span class="p">${</span><span class="nx">state</span><span class="p">.</span><span class="nx">count</span><span class="p">}</span><span class="s2"> times`</span><span class="p">;</span>
<span class="p">},</span>
<span class="nx">click</span><span class="p">()</span> <span class="p">{</span>
<span class="k">this</span><span class="p">.</span><span class="nx">state</span><span class="p">.</span><span class="nx">count</span><span class="o">++</span><span class="p">;</span>
<span class="p">}</span>
<span class="p">});</span>
</code></pre></div></div>
<p>Widgets can render other widgets if they want too:</p>
<div class="language-javascript highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="k">export</span> <span class="k">default</span> <span class="nx">createWidget</span><span class="p">(</span><span class="dl">'</span><span class="s1">controls</span><span class="dl">'</span><span class="p">,</span> <span class="p">{</span>
<span class="na">tagName</span><span class="p">:</span> <span class="dl">'</span><span class="s1">div.controls</span><span class="dl">'</span><span class="p">,</span>
<span class="nx">html</span><span class="p">(</span><span class="nx">attrs</span><span class="p">,</span> <span class="nx">state</span><span class="p">)</span> <span class="p">{</span>
<span class="k">return</span> <span class="nx">h</span><span class="p">(</span><span class="dl">'</span><span class="s1">div.controls</span><span class="dl">'</span><span class="p">,</span> <span class="k">this</span><span class="p">.</span><span class="nx">attach</span><span class="p">(</span><span class="dl">'</span><span class="s1">counter</span><span class="dl">'</span><span class="p">,</span> <span class="nx">attrs</span><span class="p">,</span> <span class="nx">state</span><span class="p">));</span>
<span class="p">}</span>
<span class="p">});</span>
</code></pre></div></div>
<p>And you can mount a widget in any template template in our Ember application:</p>
<div class="language-handlebars highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="k">{{</span><span class="nv">mount-widget</span> <span class="nv">widget</span><span class="o">=</span><span class="s2">"controls"</span><span class="k">}}</span>
</code></pre></div></div>
<p>The existing actions in your Ember application can be triggered by templates:</p>
<div class="language-handlebars highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="k">{{</span><span class="nv">mount-widget</span> <span class="nv">widget</span><span class="o">=</span><span class="s2">"button"</span> <span class="nv">doSomething</span><span class="o">=</span><span class="s2">"doSomething"</span><span class="k">}}</span>
</code></pre></div></div>
<p>You can send it up from a click event easily:</p>
<div class="language-javascript highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="c1">// in your widgets/button.js code</span>
<span class="nx">click</span><span class="p">()</span> <span class="p">{</span>
<span class="k">return</span> <span class="k">this</span><span class="p">.</span><span class="nx">sendComponentAction</span><span class="p">(</span><span class="dl">'</span><span class="s1">doSomething</span><span class="dl">'</span><span class="p">);</span>
<span class="p">}</span>
</code></pre></div></div>
<p>If your action returns a promise, once it is resolved the widget will re-render itself.
Most of the time the rendering is handled automatically like this for you, but you
can also trigger a re-render yourself by calling <code class="language-plaintext highlighter-rouge">this.queueRerender()</code>.
Renders are queued up and coalesced nicely by leveraging Ember’s event loop.</p>
<h2 id="downsides">Downsides</h2>
<p>After converting our old Handlebars templates over to be virtual DOM widgets, I have to admit
I find creating HTML this way significantly uglier than just creating a template. It’s not
so bad if the widget only has a little markup, but if there is a lot it gets a little unwieldy.</p>
<p>On the other hand, there are some major advantages to emitting HTML by hand. The fine-tuned
control over the exact DOM I was creating came in very handy when rendering the gaps between
posts in a topic which was clumsier in the ember implementation.</p>
<p>It also took me quite a while to convert the entire topic renderer over, because we have
so much functionality hidden in there! The project took me about a month of full time
work to knock out an alpha, and then another 3 weeks of fixing bugs and to develop an upgrade
path for our plugin authors.</p>
<p>It did feel good to refactor some of the oldest code we had, though. I also added a couple of
hundred new acceptance tests. Traditionally the topic stream was not tested as well as the other
parts of the site because its code was the oldest and weirdest. I straightened all that out
while I was in there.</p>
<h2 id="the-road-ahead">The Road Ahead</h2>
<p>Before setting out on a similar project, I’d ask yourself: are you okay spending a lot of
time without adding new features, and in the process likely breaking a bunch of working
stuff before you fix it again? Would you sacrifice some code quality for a speed
improvement?</p>
<p>Those are the major trade offs we made, and I believe it’s worth it because Discourse is
quite a popular project used by many people. The 5-7x speed improvement will
add up across all the forums in the world, especially with all of the Android users
who browse it.</p>
<p>Going forward, we are likely to improve the widget system even more. I’d love to hook
into the HTMLBars complier AST so we could write some templates using Handlebars, so
we are not supporting two very different ways of creating templates. I spent about
a day looking into this now but couldn’t find sufficient documentation or public
APIs, but I’m told that in the future it’s likely we will be able to do this.</p>
<p>Until then, full speed ahead!</p>
<p><strong>Update:</strong> If you want to learn more about how the Widget framework works,
<a href="https://meta.discourse.org/t/a-tour-of-how-the-widget-virtual-dom-code-in-discourse-works/40347">I’ve written more here</a>.</p>Back in September, Codinghorror wrote a popular post on the state of android Javascript performance on Discourse’s Meta forum. It drew a lot of attention, and led to some fascinating discussions on our forum and behind the scenes with browser engineers.The 5 Layers of Depth in The Witness2016-02-19T00:00:00+00:002016-02-19T00:00:00+00:00https://eviltrout.com/2016/02/19/about-the-witness.html<p>Recently I became enamored with <a href="https://en.wikipedia.org/wiki/The_Witness_(2016_video_game)">The Witness</a>
and after thinking about the game for a long time decided to sit down and record a little essay
about it.</p>
<p>Check it out below!</p>
<iframe width="640" height="360" src="https://www.youtube.com/embed/h_HuWDbWsnU" frameborder="0" allowfullscreen=""></iframe>Recently I became enamored with The Witness and after thinking about the game for a long time decided to sit down and record a little essay about it.TIS-100: My emulator for a CPU that doesn’t exist2015-06-29T00:00:00+00:002015-06-29T00:00:00+00:00https://eviltrout.com/2015/06/29/an-emulator-for-tis-100.html<p>Recently I became infatuated with <a href="http://www.zachtronics.com/tis-100/">TIS-100</a>, a game which aptly
describes itself as “the assembly language programming game you never asked for!”</p>
<p>The point of the game is to program the (imaginary) TIS-100 CPU to solve problems. For example,
you might need to take input from two ports and swap them, then write the outputs to two other
ports.</p>
<p>The game flies in the face of all modern game design: The first thing you need to do is sit and
read a 14 page PDF that outlines the TIS-100 instruction set. And when I say “read”, I mean “learn”,
because a quick skim is not going to cut it! There are no tutorial levels or handholding.
You <strong>must read the manual.</strong></p>
<p>After solving the first few problems and feeling good about myself, I approached some of my
programmer friends and tried to get them to buy the game so I could compare my solutions
to theirs. I swear I tried bringing this up with 3 people and had the exact same conversation:</p>
<p>Them: “So, it’s a game about programming…”</p>
<p>Me: “Yes, it’s so much fun!”</p>
<p>Them: “But I program all day.”</p>
<p>Me: “Me too!”</p>
<p>Them: “The last thing I want to do when I come home is program again”</p>
<p><em>awkward silence</em></p>
<p>Them: “You’re nuts.”</p>
<h2 id="the-rabbit-hole-goes-deeper">The rabbit hole goes deeper</h2>
<p>Despite not having any close friends to play with, I plowed through the puzzles in the game. One in
particular was quite devious; The TIS-100 is, as I mentioned, an imaginary CPU. And it is clearly
designed to be puzzling rather than practical. It has only two registers, and one is a backup that
cannot be addressed directly. This afformentioned puzzle involved taking the input of two numbers
and dividing one by the other. You then output the resulting quotient to one port and the
remainder to another.</p>
<p>It was quite fun to work through, but to my dismay my solution was quite inefficient. If I
clicked the regular “Play” button to execute it it would take several minutes to finish.
Even if I ran it in “Fast” mode it would take about 5 seconds to complete successfully.</p>
<p>This was obviously unacceptable.</p>
<p>A typical person might call it a day and say, “well, the real
victory is solving the puzzle!”. Another, more eccentric person might spend the time
figuring out how to optimize their solution so it executes in less time. And then there’s me.</p>
<h2 id="introducing-my-tis-100-emulator">Introducing my TIS-100 emulator</h2>
<p>I decided the most logical thing to do was to implement the TIS-100 CPU myself in pure C.
This seemed like a good idea to me despite having not used C in about 15 years.</p>
<p>Amazingly, most of the concepts came back fairly quickly. Maybe C is like riding a bike?
Maybe using so much Javascript (and its C syntax) kept me on the ball? I’m not sure.</p>
<p>I first wrote a parser to input the TIS-100 assembly language as defined in game. It writes
it to memory in byte code, which is then interpreted. The resulting performance is really
impressive!</p>
<p>The Unity version of TIS-100 that runs on my Mac executes my division program in about
5 seconds, which is an eternity as far as programs go! My C emulator runs the code in a
sleek 0.005s, or roughly 1000x faster!</p>
<p>The full <a href="https://github.com/eviltrout/tis-100">source code</a> is on GitHub, so feel free to
download it and check it out. I’ll even accept pull requests as I’m sure there’s a lot of
room for improvement.</p>
<h2 id="why-did-i-spend-my-time-on-this-pointless-project">Why did I spend my time on this pointless project?</h2>
<p>I try to not use the word “crazy” often because I don’t want to trivialize mental illness,
but let’s be honest: I have to be a least a <em>little</em> off base to attempt a project like this.
Programming is a legitimate hobby of mine. I make a living at it but I also do it in my
spare time. TIS-100 was a perfect storm of programming and fun, and I didn’t
want it to end.</p>
<p>Obivously I’m not <a href="http://gamasutra.com/view/news/244969/Things_we_create_tell_people_who_we_are_Designing_Zachtronics_TIS100.php">the only one</a>
who enjoyed the game, so there is a market for this kind of thing. Maybe this is
the long tail of games?</p>
<p>All I know is I had a lot of fun doing it, and I hope someone has fun with my
emulator. Let me know if you do!</p>Recently I became infatuated with TIS-100, a game which aptly describes itself as “the assembly language programming game you never asked for!”