<?xml version="1.0" encoding="UTF-8"?>
<rss  xmlns:atom="http://www.w3.org/2005/Atom" 
      xmlns:media="http://search.yahoo.com/mrss/" 
      xmlns:content="http://purl.org/rss/1.0/modules/content/" 
      xmlns:dc="http://purl.org/dc/elements/1.1/" 
      version="2.0">
<channel>
<title>David Mawdsley</title>
<link>https://www.mawds.co.uk/</link>
<atom:link href="https://www.mawds.co.uk/index.xml" rel="self" type="application/rss+xml"/>
<description>David Mawdsley&#39;s blog</description>
<generator>quarto-1.6.42</generator>
<lastBuildDate>Wed, 22 Jul 2026 23:00:00 GMT</lastBuildDate>
<item>
  <title>EMF installation - ‘Press space or fire, commander’</title>
  <link>https://www.mawds.co.uk/posts/elite_ships/elite_ships.html</link>
  <description><![CDATA[ 




<p>I really enjoyed <a href="https://www.emfcamp.org">Electromagnetic Field 2024</a>, and wanted to contribute something to the 2026 edition. I was really inspired by <a href="https://www.emfcamp.org/schedule/2024/113-exploring-and-extending-the-elite-game-engine">Martin Ling’s talk on the Elite Game Engine</a> at the last emfcamp, and decided to make some models of some of the ships from the game.</p>
<p>Mark Moxon has <a href="https://elite.bbcelite.com/">decompiled Elite</a>, and <a href="https://elite.bbcelite.com/">has blueprints for each of the ships</a>, which I used (with his permission) for the build</p>
<p>I wanted to recreate the wire-frame polygon aesthetic of the original. I wasn’t that sure how best to achieve this; eventually I settled on 3d printed brackets for each vertex, and joining these with rods. At this stage, I wasn’t really sure where I was going next…</p>
<section id="the-vertices" class="level2">
<h2 class="anchored" data-anchor-id="the-vertices">The vertices</h2>
<p>I ended up using AI to write a <a href="https://github.com/mawds/elite_ships">Python script</a> to convert a ship blueprint into a Freecad file containing the brackets. This was the first time I’d really tried any vibe coding (coming rather late to the “party”, I know, but this seemed a low stakes self contained project to try it with). I’m not sure it saved any time compared to doing things conventionally and figuring out the FreeCAD Python library. The process was quite frustrating, and I don’t really trust the script. That said, it does seem to (mostly) work.</p>
<p>Having made a script for the vertices, these could be 3D printed.</p>
</section>
<section id="the-rods" class="level2">
<h2 class="anchored" data-anchor-id="the-rods">The rods</h2>
<p>I wasn’t too sure what to use for the rods. Originally I was thinking of having these on show, but I decided it would make more sense to cover the whole structure. This meant that it didn’t really matter what they looked like.</p>
<p>I settled on using garden canes, as they’re cheap and (I assume) sustainable. It was a little fiddly finding canes that were straight and not too fat for the brackets I’d already printed.</p>
<p>The script was persuaded to output a csv of edge lengths, so I could cut the canes down to size (trying to avoid having any lumps near the ends, since these wouldn’t fit into the brackets)</p>
<div class="quarto-figure quarto-figure-center">
<figure class="figure">
<p><img src="https://www.mawds.co.uk/posts/elite_ships/images/PXL_20260514_175807206.jpg" class="img-fluid figure-img"></p>
<figcaption>ship frame</figcaption>
</figure>
</div>
</section>
<section id="the-cover" class="level2">
<h2 class="anchored" data-anchor-id="the-cover">The cover</h2>
<p>Having decided to cover the structure, I settled on using Lycra (spandex), as it’s stretchy, and would be taught over the frame. My original plan was to use white Lycra, and illuminate the ships internally, to give a glowing ship shaped polygon.</p>
<p>I got the script to output an SVG containing polygons for each face of the ship. These needed to be moved and rotated in Inkscape to get something I could turn into a pattern I could sew.</p>
<p>I used <a href="https://pdfposter.readthedocs.io/en/stable/">pdfposter</a> to print a life-size pattern. I manually added seam allowances when cutting the fabric. I sewed the seams with a zigzag stitch (so they’d stretch).</p>
<p>I eventually settled on using a zipper to close the skin. If you try this yourself:</p>
<ul>
<li>Remember to add the zipper to a long enough edge you can get the frame inside the ship</li>
<li>Remember that Lycra stretches and zippers don’t - it’s really fiddly sewing the zipper to the fabric stretched the same amount as it will be when it’s on the frame</li>
</ul>
<p>I tried a few things to get a uniform glow from LEDs inside the ship, but didn’t have a lot of luck with this; the fabric was too thin to diffuse the light effectively.</p>
<div class="quarto-figure quarto-figure-center">
<figure class="figure">
<p><img src="https://www.mawds.co.uk/posts/elite_ships/images/PXL_20260405_163505959.jpg" class="img-fluid figure-img"></p>
<figcaption>Elite ship in white</figcaption>
</figure>
</div>
<p>It was at this stage I (tentatively) pitched the idea to EMF and started considering just lighting the edges. Jonathan at EMF was really helpful, and steered me towards this, and using COB LED strips (which I’d not heard of before).</p>
<p>This meant making the skins from black Lycra (which hides my sewing mistakes better).</p>
<section id="lighting-and-wiring" class="level3">
<h3 class="anchored" data-anchor-id="lighting-and-wiring">Lighting and wiring</h3>
<p>I tried to minimise the number of pieces of LED strips I needed for each ship, as soldering wires on the ends is very fiddly. The <a href="https://www.aliexpress.com/item/1005012237834195.html">COB strips</a> were self adhesive, which should’ve made my life easily Unfortunately, they tended to come unpeeled after a little while. This meant that I ended up blanket stitching the strips to the fabric, using “invisible” thread (which was like thin fishing line). This was very fiddly and time consuming.<br>
The LED ends were wired together and attached to an <a href="https://www.aliexpress.com/item/1005008649510584.html">LED dimmer</a>.</p>
<div class="quarto-figure quarto-figure-center">
<figure class="figure">
<p><img src="https://www.mawds.co.uk/posts/elite_ships/images/PXL_20260606_101226117.jpg" class="img-fluid figure-img"></p>
<figcaption>Ship insides</figcaption>
</figure>
</div>
<div class="quarto-figure quarto-figure-center">
<figure class="figure">
<p><img src="https://www.mawds.co.uk/posts/elite_ships/images/PXL_20260607_095902353.MP.jpg" class="img-fluid figure-img"></p>
<figcaption>Illuminated ship</figcaption>
</figure>
</div>
</section>
<section id="problems" class="level3">
<h3 class="anchored" data-anchor-id="problems">Problems</h3>
<p>Unfortunately the ships got a bit melty in transit, and had slumped under their own weight and the tension in the fabric (note to self… don’t use PLA next time). Fortunately, the helpful people in the design tent were able to cut some lengths of wood down to size so I could wedge the ships open into roughly their correct shape.</p>
<p>If I did this again, I’d definitely find a better way of attaching the COB strip to the fabric; sewing it on was very tedious.</p>
</section>
</section>
<section id="the-finished-thing" class="level2">
<h2 class="anchored" data-anchor-id="the-finished-thing">The finished thing</h2>
<div class="quarto-figure quarto-figure-center">
<figure class="figure">
<p><img src="https://www.mawds.co.uk/posts/elite_ships/images/PXL_20260716_175854399.jpg" class="img-fluid figure-img"></p>
<figcaption>Installing</figcaption>
</figure>
</div>
<div class="quarto-figure quarto-figure-center">
<figure class="figure">
<p><img src="https://www.mawds.co.uk/posts/elite_ships/images/PXL_20260719_163831899.MP.jpg" class="img-fluid figure-img"></p>
<figcaption>Done</figcaption>
</figure>
</div>
<section id="note" class="level3">
<h3 class="anchored" data-anchor-id="note">Note</h3>
<p>I wasn’t very good a documenting my progress, so the pictures show the different stages of the build for different ships. I started with the white Mamba, then built the Cobra. I then reused the Mamba brackets to make a smaller model, and finally built a Viper.</p>


</section>
</section>

 ]]></description>
  <guid>https://www.mawds.co.uk/posts/elite_ships/elite_ships.html</guid>
  <pubDate>Wed, 22 Jul 2026 23:00:00 GMT</pubDate>
</item>
<item>
  <title>oggcamp</title>
  <link>https://www.mawds.co.uk/posts/oggcamp2026/oggcamp.html</link>
  <description><![CDATA[ 




<p>I was passing through Manchester on the way back from Scotland, so decided to go to my second <a href="https://www.oggcamp.org/">Oggcamp</a>. This is my belated writeup of it (and an attempt to remember some of the talks I saw)</p>
<p>There was a scheduled track, as well as two unconference tracks. Some of the talks for these went up very late, and I missed (most of) a couple I’d have liked to have seen.</p>
<p>Some highlights, in no particular order:</p>
<p>I caught the tail end of <a href="https://williammorland.com/">William Morland’s</a> talk on Home Assistant, and discussion of wiring up boilers to it. Apparently Dutch gas boilers <em>have</em> to have an Operntherm interface, so something to consider when mine needs replacing.</p>
<p>There were only a couple of lightning talks unfortunately. There was an interesting one about an LED score-board for unicycle hockey(!), and using DNS exfiltration to get save the scores (and get the time) over a locked down wireless network.</p>
<p>Next up was a great talk on <a href="https://media.ccc.de/v/39c3-battling-obsolescence-keeping-an-80s-laser-tag-sys">keeping an 80s laser tag system alive</a></p>
<p><a href="https://mastodon.me.uk/@megjlow@fosstodon.org">Margaret Low</a> gave a good talk on <a href="https://www.turtlestitch.org/">Turtlestitch</a> (essentially Logo for embroidery). She spoke on this at the last Oggcamp, which had inspired me to go down a rabbit hole of making <a href="https://mastodon.me.uk/@megjlow@fosstodon.org/116470237751772163">this</a>, so it was nice to be able to say thanks for the inspiration.</p>
<p>There was an incredibly depressing talk about whistleblowing and content moderation in Big Tech; really bleak.</p>
<p>The retro computing show and tell was good - mostly Psions and Z88s</p>
<p>There was a great talk on repairing Dyson battery packs (which involved <a href="https://codeberg.org/davidmpye/V10_Dyson_BMS">writing custom firmware</a> to unbrick them).</p>
<p>One of the highlights was <a href="http://awfulwoman.com/">Charlie O’Hara’s</a> on infrastructure as code. It was a great demo of Ansible, and concluded with a spontaneous (and somewhat nerve-wracking) “shall I delete my website live on stage and restore it”? (answers: yes and yes)</p>
<p>The whole event was great; really friendly and interesting people. (Maybe ~150 in total?)</p>
<p>This year the <a href="https://www.oggcamp.org/schedule/">main track was recorded</a>.</p>
<p>A great warm up for EMFCamp :-)</p>



 ]]></description>
  <guid>https://www.mawds.co.uk/posts/oggcamp2026/oggcamp.html</guid>
  <pubDate>Thu, 07 May 2026 23:00:00 GMT</pubDate>
</item>
<item>
  <title>Reverse engineering Janome embroidery cards (part 3)</title>
  <link>https://www.mawds.co.uk/posts/janome/janome_3.html</link>
  <description><![CDATA[ 




<p>My attempts to reverse engineer an old sewing machine’s embroidery function continue. <a href="../../janome/janome_3.html">In (a belated) part 3 of this saga, some success, but not how I was expecting</a>.</p>



 ]]></description>
  <guid>https://www.mawds.co.uk/posts/janome/janome_3.html</guid>
  <pubDate>Fri, 02 Jan 2026 00:00:00 GMT</pubDate>
</item>
<item>
  <title>Reverse engineering Janome embroidery cards (part 2)</title>
  <link>https://www.mawds.co.uk/posts/janome/janome_2.html</link>
  <description><![CDATA[ 




<p>My attempts to reverse engineer an old sewing machine’s embroidery function continue. The original post was getting a bit long (as is the project…). <a href="../../janome/janome_2.html">This is an update further progress, which I’ve put on a separate page</a>.</p>



 ]]></description>
  <guid>https://www.mawds.co.uk/posts/janome/janome_2.html</guid>
  <pubDate>Wed, 28 May 2025 23:00:00 GMT</pubDate>
</item>
<item>
  <title>Reverse engineering Janome embroidery cards</title>
  <link>https://www.mawds.co.uk/posts/janome/janome.html</link>
  <description><![CDATA[ 




<p>I recently went down a bit of a rabbit hole trying to reverse engineer the embroidery cards for a 1990s Janome Memory Craft 8000 sewing machine I bought.</p>
<p>I’m hoping to be able to write my own designs to a card, but haven’t got that far yet. As it’s rather a long write up, and still a work in progress <a href="../../janome/janome.html">I’ve put the write up on a separate page</a>.</p>



 ]]></description>
  <guid>https://www.mawds.co.uk/posts/janome/janome.html</guid>
  <pubDate>Mon, 17 Mar 2025 00:00:00 GMT</pubDate>
</item>
<item>
  <title>Ubuntu 24.04 upgrade</title>
  <link>https://www.mawds.co.uk/posts/ubuntu_reinstall/ubuntu_reinstall.html</link>
  <description><![CDATA[ 




<p>I recently had to reinstall Ubuntu on my ageing Thinkpad X260, after the upgrade from 22.04 to 24.04 went horribly wrong.</p>
<p>I’m still not quite sure what went wrong. It <em>think</em> I’d somehow ended up with (only) the 22.04 kernel installed, which led to lots of “Unknown symbol” errors. My system was quite heavily customised, so it was perhaps a bit optimistic to expect the upgrade to work.</p>
<p>This post is mostly a note to self about how I reconfigured the dual boot setup in case I need to do it again (which I hope I don’t).</p>
<p>I initially thought it was <em>only</em> the kernel that was the problem. <a href="https://askubuntu.com/a/528295">This stackexchange post</a> was really useful for mounting the encrypted drive, setting up a chroot environment and reinstalling the kernel. That got me booting to the command prompt.</p>
<p>It turned out I was missing most packages, at which point I decided to wipe and restore from backups.</p>
<p>(I could actually access my encrypted drive through the Ubuntu Live<del>CD</del>USB, so took a fresh backup of <code>/home</code> and <code>/etc</code> from this)</p>
<p>I’d got an (essentially unused and tiny) Windows 10 partition dual booting with Ubuntu. I was keen to keep Windows 10 on the machine, as it’s occasionally useful. I’d previously set this up to dual boot with full disk encryption for both OSs roughly following the guide <a href="https://www.mikekasberg.com/blog/2020/04/08/dual-boot-ubuntu-and-windows-with-encryption.html">here</a>, but using the legacy BIOS, as that’s what Windows 10 had come preinstalled with.</p>
<p>I was pleased to see that the guide had been <a href="https://www.mikekasberg.com/blog/2024/05/20/dual-boot-ubuntu-24-04-and-windows-with-encryption.html">updated for Ubuntu 24.04</a> and is now <em>much</em> easier. I took this opportunity to wipe the Windows partition and switch to UEFI too.</p>
<p>The basic process was:</p>
<ul>
<li>Use Ubuntu 24.04 USB image to check backups are OK and work</li>
<li>Use the Ubuntu USB image and gparted to remove all partitions</li>
<li>Reboot and set the BIOS to UEFI mode (only) (Startup menu in BIOS), and check the security chip is set to “Intel PTT”.</li>
<li>Install Windows 10 Pro from installation USB (avoiding creating a Microsoft account by selecting “Work or school account” then “Domain join instead” <a href="https://helpdeskgeek.com/windows-10/how-to-setup-windows-10-without-a-microsoft-account/">via</a></li>
<li>Install interminable updates and reboot lots of times (I kept force checking for updates until there were no more). (Bitlocker is not enabled at this point)</li>
<li>Reboot with the Ubuntu USB image</li>
<li>Follow the instructions at https://www.mikekasberg.com/blog/2024/05/20/dual-boot-ubuntu-24-04-and-windows-with-encryption.html to setup the dual boot with an encrypted partition setup.</li>
<li>Check both still boot</li>
<li>Enable bitlocker on the Windows partition <strong>You need to boot Windows via the BIOS’s boot menu (F12 on my machine), not via Grub, otherwise it will prompt you for the bitlocker recovery key on boot</strong> (details of how I removed the Windows option from Grub below)</li>
</ul>
<p>Once everything was working, I rebooted using the USB image to mount my nfs share containing the backups and restored my data.</p>
<p>A few post install tweaks:</p>
<p>Hibernate doesn’t work out of the box. I enabled this following the guide on <a href="https://elbrarc.at/blog/2024/05/30/ubuntu-fde-hibernate-tpm-secureboot.html">Elbrar’s corner</a>. This requires booting with the USB image again, to resize volumes to make space for the encrypted swap partition (with hindsight, the resizing would probably have been quicker if I did this <em>before</em> restoring my data; even so, it was only a few minutes). I didn’t follow the “Install Clevis” section, as I prefer to enter the decryption key before the machine boots. I used the <a href="https://extensions.gnome.org/extension/755/hibernate-status-button/">Hibernate Status Button extension</a> to enable the hibernate menu option on the desktop.</p>
<p>Suspend didn’t work properly. The system would suspend, but not wake up. I eventually solved this via <a href="https://askubuntu.com/questions/1523542/thinkpad-t460-shutdown-suspend-in-ubuntu-24-04">this Stackexchange post</a>. You need to add <code>intel_iommu=off</code> to the <code>GRUB_CMDLINE_LINUX_DEFAULT</code> options in <code>/etc/default/grub</code> and then <code>update-grub</code>.</p>
<p>I also took this opportunity to add <code>DISABLE_OS_PROBER=true</code> to the file, so we don’t have the option to boot to Windows from grub, which causes issues with Bitlocker, as mentioned above.</p>
<p>I also had to force an update of the installed Snaps so that Firefox was at the same version as on the 20.04 system, so that it would load my profile data. Presumably this would’ve happend automatically at some point anyway.</p>



 ]]></description>
  <guid>https://www.mawds.co.uk/posts/ubuntu_reinstall/ubuntu_reinstall.html</guid>
  <pubDate>Fri, 15 Nov 2024 00:00:00 GMT</pubDate>
</item>
<item>
  <title>Local fuel prices</title>
  <link>https://www.mawds.co.uk/posts/fuelprice/fuelprice.html</link>
  <description><![CDATA[ 




<p>I’ve recently launched a new site <a href="https://localfuelprices.co.uk">localfuelprices.co.uk</a> to display the cost of <a href="https://localfuelprices.co.uk/map/E10/">petrol</a> and <a href="https://localfuelprices.co.uk/map/B7/">diesel</a> nearby petrol stations.</p>
<p>It’s built using <a href="www.djangoproject.com">django</a> and <a href="https://docs.djangoproject.com/en/5.1/ref/contrib/gis/">geodjango</a>, with <a href="https://leafletjs.com/">leaflet</a> for the mapping.</p>



 ]]></description>
  <guid>https://www.mawds.co.uk/posts/fuelprice/fuelprice.html</guid>
  <pubDate>Thu, 19 Sep 2024 23:00:00 GMT</pubDate>
</item>
<item>
  <title>Travels</title>
  <link>https://www.mawds.co.uk/posts/travels/travels.html</link>
  <description><![CDATA[ 




<p>I’m taking a few months career break to go travelling, which I plan to blog (hopefully more regularly than I manage here!) at <a href="https://travel.mawds.co.uk/index.php/europe-2023/">https://travel.mawds.co.uk/index.php/europe-2023/</a>. This is running on Wordpress, which is a lot easier to update on a phone than this Quarto based site.</p>



 ]]></description>
  <guid>https://www.mawds.co.uk/posts/travels/travels.html</guid>
  <pubDate>Sun, 11 Jun 2023 23:00:00 GMT</pubDate>
</item>
<item>
  <title>Adding epubs to Kobo from my phone</title>
  <link>https://www.mawds.co.uk/posts/kobo/kobo.html</link>
  <description><![CDATA[ 




<p>I’ve got a Kobo e-reader, which is very fussy about the USB cable used to copy books to it (and even the USB port the cable is connected to). I want to be able to sideload books from, e.g.&nbsp;<a href="https://www.gutenberg.org">Project Gutenberg</a> to it from my phone. The (beta) web browser that’s included with the Kobo is <em>very</em> basic, and I’ve struggled to get it to work with most web sites, so the obvious solution doesn’t really work.</p>
<p>With thanks to <a href="https://medium.com/techtofreedom/this-is-how-i-set-up-an-http-server-on-my-android-phone-1fed20e4f0c8">this post</a>, I figured out a way to do this.</p>
<p>Essentially you download your books using Chrome on the phone, and then use <a href="https://termux.com/">termux</a>, which give a Linux-like environment on the phone, to run a minimal web-server whose output works with the Kobo’s browser.</p>
<section id="setup" class="level2">
<h2 class="anchored" data-anchor-id="setup">Setup</h2>
<p>At the termux prompt: (this only needs to be done once)</p>
<pre class="{bash}"><code>termux-setup-storage # Gives access to the Download folder on android 
pkg install python3 # Install Python</code></pre>
</section>
<section id="to-use" class="level2">
<h2 class="anchored" data-anchor-id="to-use">To use</h2>
<p>Enable tethering on the phone (I also found I had to disable the VPN running on the phone)</p>
<p>At the termux prompt:</p>
<pre class="{bash}"><code>ifconfig # Note the ip associated with wlan0
cd ~/storage/downloads # Assuming this is where the book is
python -m http.server 8080</code></pre>
<p>On the Kobo, connect to the phone’s hotspot, open the browser (more, beta features), and navigate to http://&lt;phone’s ip&gt;:8080 This gives a very basic directory listing, which works OK on the Kobo</p>


</section>

 ]]></description>
  <guid>https://www.mawds.co.uk/posts/kobo/kobo.html</guid>
  <pubDate>Sun, 11 Jun 2023 23:00:00 GMT</pubDate>
</item>
<item>
  <title>Not so smart metering</title>
  <link>https://www.mawds.co.uk/posts/powermon/powermon.html</link>
  <description><![CDATA[ 




<section id="introduction" class="level1">
<h1>Introduction</h1>
<p>This project has been a while in the making, and I’ve finally got round to finishing it off (well as much as anything is finished). A big part of the delay was, somewhat ireonically, not having a power socket near the meters.</p>
<p>I’m not a huge fan of Smart Meters - being able to be cut off remotely doesn’t seem a selling point, but I was curious how much energy I was using. I built a circuit with the Raspberry Pi to keep track of much gas and electricity I was using - this uses the GPIO pins to count the flashes on the electricity meter, using an LDR, and to count the rotations of the gas meter using a Hall probe (it turns out the 0 has a magnet in it, which can be detected as it rotates).</p>
<p>The measurements are much more precise for the electic meter - there are 1000 flashes per kWh, compared to ~9 per kWh for the gas meter.</p>
<p>I was originally simply dumping the timestamps (and time difference between successive ones for each power source) into a CSV file, and looking at this in R.</p>
<p>I wanted to make something where I could see at a glance how much energy I’d been using. I had an old <a href="https://www.johnlewis.com/nook-simple-touch-ereader/p231759496">Nook E-Reader</a>, which I planed to use as a display. <a href="https://shkspr.mobi/blog/2020/02/turn-an-old-ereader-into-an-information-screen-nook-str/">This blog post</a> was very useful in figuring out how to do this.</p>
<p>tldr; it runs Android 2.x, the apks linked to on that page were sideloaded via adb, so no need to set up a Google Account. The plan was to use Electic Sign to display a web page containing the current power usage.</p>
<section id="sqllite-and-fastapi" class="level2">
<h2 class="anchored" data-anchor-id="sqllite-and-fastapi">Sqllite and FastAPI</h2>
<p>I’ve now switched to dumping the timestamps to a sqllite database, and am using FastAPI to generate an endpoint that I can display on the Nook:</p>
<p><img src="https://www.mawds.co.uk/posts/powermon/api.png" class="img-fluid" alt="Example screenshot from API"></p>
</section>
<section id="futher-work" class="level2">
<h2 class="anchored" data-anchor-id="futher-work">Futher work</h2>
<p>I plan to move the logging over to a “proper” database. Handling timestamps in Sqllite is a little fiddly. Doing some visualisation of the data is also on the list.</p>
<p>Code is <a href="https://www.github.com/mawds/powermon">here</a></p>


</section>
</section>

 ]]></description>
  <category>Python</category>
  <category>Raspberry Pi</category>
  <guid>https://www.mawds.co.uk/posts/powermon/powermon.html</guid>
  <pubDate>Tue, 21 Mar 2023 00:00:00 GMT</pubDate>
</item>
<item>
  <title>Easily deploying Shiny apps</title>
  <link>https://www.mawds.co.uk/posts/shinysender/shinysender.html</link>
  <description><![CDATA[ 




<section id="introduction" class="level1">
<h1>Introduction</h1>
<p>I’m currently running a pilot Shiny web hosting service at the University of Manchester. One issue with the standalone (free) <a href="https://shiny.rstudio.com/articles/shiny-server.html">Shiny Server</a> is that it’s not particularly easy to deploy apps to it. Essentially you have to copy them to the server (e.g.&nbsp;using <code>scp</code>), and sort out any library dependency issues. <del>RStudio’s</del> Posit’s commercial offerings, like <del>RStudio</del> Posit Connect make this process much easier, but are more expensive<sup>1</sup>.</p>
<p>To fill this gap, I wrote an R Package to simplify the deployment of apps to a standalone, open source Shiny Server: <a href="https://github.com/UoMResearchIT/r-shinysender">Shinysender</a></p>
<section id="usage---server-setup" class="level2">
<h2 class="anchored" data-anchor-id="usage---server-setup">Usage - server setup</h2>
<p>I designed the package to be as simple to use as possible, to minimise the amount of support required.</p>
<p>The server being deployed to is set up in the usual way. A few R packages (shiny, rmarkdown, packrat and devtools) need to be installed for all users, and the server needs to be configured to serve apps from users’ home directories. It’s also sensible to pre-install the “dev” versions of of system libraries that are required by common R packages (e.g.&nbsp;<code>libssl-dev</code>, <code>libxml2-dev</code> etc.; the full list of the ones I’ve used is in the README)</p>
<p>Full details are provided in the <a href="https://github.com/UoMResearchIT/r-shinysender/blob/master/README.md#server-setup">package’s readme</a>. Users need an account on the server to be able to deploy to it.</p>
</section>
<section id="usage---app-deployment" class="level2">
<h2 class="anchored" data-anchor-id="usage---app-deployment">Usage - app deployment</h2>
<p>The package isn’t (yet) on CRAN, so it needs to be installed from github:</p>
<div class="cell">
<div class="sourceCode cell-code" id="cb1" style="background: #f1f3f5;"><pre class="sourceCode r code-with-copy"><code class="sourceCode r"><span id="cb1-1"><span class="fu" style="color: #4758AB;
background-color: null;
font-style: inherit;">install.packages</span>(<span class="st" style="color: #20794D;
background-color: null;
font-style: inherit;">"devtools"</span>)  <span class="co" style="color: #5E5E5E;
background-color: null;
font-style: inherit;"># If you don't already have devtools installed</span></span>
<span id="cb1-2">devtools<span class="sc" style="color: #5E5E5E;
background-color: null;
font-style: inherit;">::</span><span class="fu" style="color: #4758AB;
background-color: null;
font-style: inherit;">install_github</span>(<span class="st" style="color: #20794D;
background-color: null;
font-style: inherit;">"UoMResearchIT/r-shinysender"</span>)</span></code></pre></div>
</div>
<p>The server and username are determined by the <code>SHINYSENDER_SERVER</code> and <code>SHINYSENDER_USER</code> environment variables. These can be set from the R console, or by adding them to <code>~/.Rprofile</code>/<code>~/.Renviron</code>:</p>
<div class="cell">
<div class="sourceCode cell-code" id="cb2" style="background: #f1f3f5;"><pre class="sourceCode r code-with-copy"><code class="sourceCode r"><span id="cb2-1"><span class="fu" style="color: #4758AB;
background-color: null;
font-style: inherit;">Sys.setenv</span>(<span class="at" style="color: #657422;
background-color: null;
font-style: inherit;">SHINYSENDER_SERVER=</span><span class="st" style="color: #20794D;
background-color: null;
font-style: inherit;">"myshinyserver.mawds.co.uk"</span>)  </span>
<span id="cb2-2"><span class="fu" style="color: #4758AB;
background-color: null;
font-style: inherit;">Sys.setenv</span>(<span class="at" style="color: #657422;
background-color: null;
font-style: inherit;">SHINYSENDER_USER=</span><span class="st" style="color: #20794D;
background-color: null;
font-style: inherit;">"david"</span>)</span></code></pre></div>
</div>
<p>The “upload app” addin will appear in RStudio, and will deploy the current app to http://myshinyserver.mawds.co.uk/username/appname The user is prompted for their password on the server (an ssh key will be used if available).</p>
<p>The user’s environment is replicated as closely as possible using Packrat.</p>
<p>There are various options to allow the user to set the name of the app on the server, etc. These are detailed in the <a href="https://github.com/UoMResearchIT/r-shinysender/blob/master/README.md">readme</a>.</p>
</section>
<section id="how-it-works" class="level2">
<h2 class="anchored" data-anchor-id="how-it-works">How it works</h2>
<p>We use the <a href="https://cran.r-project.org/package=ssh">ssh</a> library to drive the remote session.</p>
<p>The app deployment process checks that the users <code>ShinyApps</code> directory exists and creates it if required. We also create as <code>ShinyApps_staging</code> directory.</p>
<p>We then use the <a href="https://cran.r-project.org/package=rsconnect">rsconnect</a> library to prepare a “bundle” containing the app’s code and dependencies. The bundle is then uploaded over scp to the staging directory and decompressed</p>
<p>We then upload a custom <code>.Rprofile</code> which sets up the cache directory for Packrat (so that a user can share R packages between their apps). This doesn’t actually <em>activate</em> Packrat (see below). We then restore the libraries into the Packrat environment, before swapping the <code>.Rprofile</code> for one that actually activates Packrat when the app is run.</p>
<p>Assuming everything has worked, we then move the app directory from <code>ShinyApps_staging</code> to <code>ShinyApps</code>, to make it live.</p>
<section id="swapping-.rprofiles" class="level3">
<h3 class="anchored" data-anchor-id="swapping-.rprofiles">Swapping .Rprofiles</h3>
<p>One feature I was keen on implementing was being able to install custom libraries from Github. These might be stored in a private repository. To handle this, we set any local <code>GITHUB_PAT</code> on the remote before we restore the R packages (We do this as a single command, with history turned off, so the PAT is never actually stored on the server<sup>2</sup>).</p>
<p>If we use the <em>same</em> <code>.Rprofile</code> for deployment and staging, we won’t have access to the (system-wide) <code>devtools</code> library, which is required to install packages from github. Hence the need for a staging <code>.Rprofile</code> and a live one.</p>
<p>I should add that our <code>.Rprofile</code> is appended to any existing <code>.Rprofile</code>, rather than being overwritten, so any settings the user has used will be preserved.</p>
</section>
</section>
<section id="limitations" class="level2">
<h2 class="anchored" data-anchor-id="limitations">Limitations</h2>
<p>Shiny server doesn’t let you use more than one version of R. Although we attempt to replicate the user’s environment as closely as possible, this does mean that they’re stuck with the version of R on the server. In practice, this hasn’t proved an issue so far, but is likely to become problematic when R’s major version increments.</p>
<p>The usual limitations of hosting apps on the free version of Shiny server (i.e.&nbsp;an app only has a single process, which is shared by all users applies). On my aspirational todo list is to deploy apps using <a href="https://www.shinyproxy.io/">Shinyproxy</a> - this would mean each app was hosted in its own Docker container. In principle that should be fairly straightforward, since the packrat steps are generic. From a security perspective it’s more difficult, since the shinyproxy config would need to be modified by a privileged user to pick up the new app. Essentially I’d need to write an API for the server to do this, rather than just “driving” a remote ssh session as the user.</p>
<p>It’d be better to run the whole thing behind an API, but since the Shiny apps are run as the user, and are deployed using the user account the security risks are much lower. The server is locked down so users cannot see others’ home directories etc.</p>


</section>
</section>


<div id="quarto-appendix" class="default"><section id="footnotes" class="footnotes footnotes-end-of-document"><h2 class="anchored quarto-appendix-heading">Footnotes</h2>

<ol>
<li id="fn1"><p>They also offer other features, like making the app more scalable if it has many concurrent users.↩︎</p></li>
<li id="fn2"><p>This was the most secure way I could think of doing this..if there are better ways I’d be grateful to hear of them.↩︎</p></li>
</ol>
</section></div> ]]></description>
  <category>Shiny</category>
  <category>R</category>
  <guid>https://www.mawds.co.uk/posts/shinysender/shinysender.html</guid>
  <pubDate>Thu, 24 Nov 2022 00:00:00 GMT</pubDate>
</item>
</channel>
</rss>
