Running RStudio Shiny Server & Apps

These are descriptions and examples that demonstrate how to build and run RStudio Shiny applications on DNAnexus.

Shiny – an RStudio package – is becoming more and more popular among R language developers for ability to rapidly create web apps from R scripts. So, if you created an R script that users find useful but they prefer to click buttons rather than work in a terminal window, you may convert it to a Shiny app – a web application with graphical user interface (GUI).

If you now want to run your new and shiny app on the DNAnexus platform, you can easily wrap it as a special DNAnexus web app, run it, and access it using a web browser. Authentication, permission control, encrypted communication (via HTTPS), ability to share objects (like files and applets) are integral parts of the DNAnexus platform and therefore come as a bonus. To read more about DNAnexus web apps, please visit our documentation page .

In the following chapters, we will work on three three different examples with increasing complexity and varied features. Let's get started!

Prerequisites

To be able to successfully build apps described below, you will need to have the following already set up and working.

DNAnexus account

If you do not have a DNAnexus account, create it by navigating to https://platform.dnanexus.com/register

DNAnexus Platform SDK

Our SDK (aka. dx-toolkit) allows you to develop software (primarily in Python) that interacts with the DNAnexus platform. It includes the command-line interface (CLI) client, tools for building and debugging apps, etc. You can download it from https://documentation.dnanexus.com/downloads#dnanexus-platform-sdk

Docker

Docker is an incredibly useful software that allows you to package your code/software and data into containers that can then be run as a virtual Linux machine. To get the software, visit https://www.docker.com/ . Largest container repositories with lots of different container images are Docker Hub and Quay. Many bioinformatics tools have already been containerized. Their Dockerfiles and/or container images can be found at Biocontainers (website link, Github link) and Dockstore.

Source code

Full source code of the apps described in the following chapters is deposited in the Github repository in this folder:

Method 1. Pull everything on-the-fly

This is the simplest method and is useful if you want to create the app with the least effort. Here, I will demonstrate how to create a new DNAnexus web app based on the K-means R Shiny app example, deploy it to DNAnexus, and interact with it via a web browser. It will pull everything it needs from the internet on-the-fly: an example Shiny app's R source code and Shiny Server's docker image.

Let's create a DNAnexus applet

We can create a DNAnexus app using a special DX app wizard that comes with the SDK. To create it:

1. Log in

Log into the platform and select your project using the CLI client. (See Prerequisites chapter above.) For this, open your favorite terminal app (e.g. Terminal or iTerm on Macs or whatever on Windows) and execute the command:

dx login
# Enter your username, password
# Select the project where you want to deploy your app.

2. Create new app

Now, to create a new app called flying_kmeans, start the app wizard by executing the command:

dx-app-wizard
# The wizard will ask you questions to finalize the app's configuration.
# Here are my answers:
App Name: flying_kmeans
Title []: Flying K-Means
Summary []: K-means Shiny app on DNAnexus
Version [0.0.1]: # click <ENTER>
1st input name (<ENTER> to finish): # click <ENTER>
1st output name (<ENTER> to finish): # click <ENTER>
Timeout policy [48h]: # click <ENTER>
Programming language: bash
Will this app need access to the Internet? [y/N]: y
Will this app need access to the parent project? [y/N]: y
Choose an instance type for your app [mem1_ssd1_v2_x4]: # click <ENTER>

If you are familiar with this wizard, you can change some of the other parameters. For example, if your app requires more resources than the default server instance (mem1_ssd1_v2_x4) provides, you may elect a larger instance. (Table of all instances is printed before this choice.)

The wizard will create a folder flying_kmeans on your computer with this hierarchy:

flying_kmeans
├── dxapp.json
├── src
│ └── flying_kmeans.sh
├── Readme.developer.md
├── Readme.md
├── resources
└── test

3. Create web app

To convert this app into a web app we must modify the dxapp.json in the app's root folder. This is a JSON-formatted file containing the app's configuration as key-value pairs. (Docs are here.) In the file, find the place where it says:

"version": "0.0.1",
"inputSpec": [],

And modify it to:

"version": "0.0.1",
"httpsApp": {
"ports": [443],
"shared_access": "VIEW"
},
"inputSpec": [],

By adding the new httpsApp key and its values, we've enabled communication via port 443 (HTTPS). The shared_access is a string and its values can be: VIEW, CONTRIBUTE, or NONE. To access your running app, a user must have at least VIEW permission to your project. If the web app must only be accessed by whoever launched it then set the value of this key to NONE.

Keys in the httpsApp describe how and who can access the web app. The "ports": [443] means that we've enabled communication via port 443 (HTTPS; one can add more ports). With "shared_access": "VIEW" we defined that to access this web app while it is running, another user must have at least VIEW permission to your project. If the web app must only be accessed by whoever launched it then change this value to NONE. Read more about web applications at https://documentation.dnanexus.com/developer/apps/https-applications

4. Create app code

Now it is time to create the app's code. The idea is very simple: we want to get the code of the K-means example app from Github, get a docker image of the R Shiny Server, and connect them together. As the docker image, we can use one created by Rocker Project. Open file src/flying_kmeans.sh. This is the Bash script that contains the function where our app starts from - the main() function. The full content (half being comments) of my script is just 12 lines:

#!/bin/bash
# flying_kmeans 0.0.1
set -eux
main() {
# get K-means app code
mkdir kmeans_app
url=https://raw.githubusercontent.com/rstudio/shiny-examples/master/050-kmeans-example
wget -P kmeans_app/ $url/DESCRIPTION $url/server.R $url/ui.R
# pull and run Shiny Server docker image
# attach our K-means app's folder as a volume
docker run --rm -p 443:3838 -v $PWD/kmeans_app:/srv/shiny-server/ rocker/shiny
}

Copy this code, paste into your file, and save it.

5. Build and deploy

Now let's build and deploy our app. In your terminal, navigate to the folder above the flying_kmeans and execute command:

dx build -f flying_kmeans

If everything goes well, the app will be built, deployed to the DNAnexus platform, and you will get its ID in response, like this:

{"id": "applet-Fkxz86j0ffy5jfXgJXGq9732"}

I added the -f command line argument to overwrite the old version of this applet. If you want to archive old versions of built applets, add -a argument. For help about the build process, run dx build -hor go to our Documentation page.

The new app will appear in your DNAnexus project:

6. Start the app

To start the app, click on it. In the app's window, click on Run as Analysis… button.

The app will start launching that can be seen in the platform's Monitor tab:

In a few minutes, the app's status will change to Running. After a moment, a web URL will appear in the last column of the table - Worker URL.

This means that the app is ready to accept connections.

If you don't see the URL for a while, even though the job is running, just reload the page.

7. Use the app

Click on the app's link. A new web browser tab will open and display the interface of our K-means Clustering app:

If you got "502 Bad Gateway" browser error when you loaded the page, please wait about a minute or so and reload the page. This happens when the web server part is ready and the URL is available but your Shiny app or some internal process is not yet ready to listen to connections yet.

Conclusions

We created the R Shiny web app with the least effort, just a few lines of code. It is deployed and is working. With this method, you can quickly build apps from existing code. The caveats are that (1) your Shiny app's code must be accessible on the internet to be pulled; (2) every time you run the app, it will pull the 1.43 GB rocker/shiny docker image anew, which takes platform's bandwidth and your time, and (3) the app does not have any input files that are in your DNAnexus project.

Method 2. Multi-purpose Shiny Server

What if you have a multitude of Shiny apps and you don't want to build and deploy separate applets for every single one of them, or every time you change them? Also, perhaps you don't want to spend that additional minute it takes to dynamically pull your app's code and rocker/shiny image. Internet connection to other web sites may be unreliable; sites go down, and this should not affect your work. In such a case, it would be more convenient to build a multi-purpose Shiny Server. All of its parts should be cached in DNAnexus. Also, it should be able to run any Shiny app that you have cached in DNAnexus. Let's go ahead and build such an applet.

Let's create the applet

1. Create a new applet using the wizard

Just like in the Method 1, invoke the dx-app-wizard and provide the following parameters:

App Name: dx_shiny
Title []: Shiny Server
Summary []: Shiny Server on DNAnexus
Version [0.0.1]: # click <ENTER>

Unlike the previous case though, configure the app to have an input:

1st input name (<ENTER> to finish): app_gz
Label (optional human-readable name) []: Gzip archive of my Shiny app
Choose a class (<TAB> twice for choices): file
This is an optional parameter [y/n]: n

Your new and shiny applet will be created in the dx_shiny folder.

2. Convert it to a web app

As before, to convert it to a web app, open its dxapp.json and add

"httpsApp": { "ports": [443], "shared_access": "VIEW" },

3. Enable input file validation

To validate that the input files have .gz extension, change the value of the "pattern" list from "*" to "*.gz" under the "inputSpec" key so that it looks like this:

"inputSpec": [
{
"name": "app_gz",
"label": "Gzip archive of my Shiny app",
"class": "file",
"optional": false,
"patterns": [
"*.gz"
],
"help": ""
}
],

Save the dxapp.json file.

4. Create main app script

Now modify the src/dxshiny.sh script. Here is the content:

#!/bin/bash
# dxshiny 0.0.1
set -eux
main() {
# Download the input: your Shiny app's source code archive from the project
echo "Value of app_gz: '$app_gz'"
dx download "$app_gz" -o app.gz
# Unpack the app code. Here, `--strip 1` allows to skip the top folder
# of the archive and unpack the code directly underneath ./app/
# This allows you to go to the job's URL and launch your Shiny app directly.
# Otherwise, the code would be one folder level deeper and one would have to
# click on the app folder's link in the browser.
mkdir app
tar -zxvf app.gz -C app --strip 1
# Set up the Shiny Server
# Assuming that the archive of the docker image of the Shiny Server is
# in your project at /rshiny/rshiny.docker.gz
dx download $DX_PROJECT_CONTEXT_ID:/rshiny/rshiny.docker.gz
# ($DX_PROJECT_CONTEXT_ID contains your project ID)
# Now load the image from the archive, create docker image
docker load -i rshiny.docker.gz
# Run the docker image. Attach your app's folder as a volume.
docker run --rm -p 443:3838 -v $PWD/app:/srv/shiny-server/ rocker/shiny
}

Essentially, we are downloading our Shiny app's source code archive - a single input that this app will require; we are also downloading the archive of the Shiny Server's docker image, loading and running it. Importantly, we are attaching our Shiny app's code as a volume to the Shiny Server's docker container.

5. Build the applet

Navigate to the parent of the dx_shiny folder and run:

dx build -f dx_shiny

The new applet dx_shiny should appear in the current directory of your DNAnexus project.

Archive and upload the file archives

Our app requires two file archives: (1) our K-means Shiny App, and (2) Shine Server's docker image.

Let's create and upload the K-means app's archive.

1. In your terminal, run the following commands:

mkdir kmeans_app url=https://raw.githubusercontent.com/rstudio/shiny-examples/master/050-kmeans-example
wget -P kmeans_app/ $url/DESCRIPTION $url/server.R $url/ui.R
tar -zcvf kmeans_app.gz kmeans_app

This will download all three files of the app to your local kmeans_app folder and archive it. The new archive will be called kmeans_app.gz.

2. Upload the file to the platform:

dx upload kmeans_app.gz

Now pull the Shiny Server's docker image and upload it to the platform.

1. Pull the docker image from Docker Hub:

docker pull rocker/shiny

2. Save it to an archive file:

docker save -o rshiny.docker.gz rocker/shiny

3. Now upload it to the platform:

dx upload rshiny.docker.gz

Let's assume your current directory in the DNAnexus platform is /shiny. After these operations, you should have three objects in the folder: archive files kmeans_app.gz, rshiny.docker.gz, and the dx_shiny applet.

Run

To test if everything worked, run the applet. You can run it from the UI by clicking on it, providing the kmeans_app.gz as an input. (Click on the input rectangle with an orange border on the left. (I am assuming you are familiar with DNAnexus apps and applets and how to run them.)

It should look like this:

Click on the Run as Analysis… button.

Alternatively, start your app with this command:

dx run dx_shiny -y -i app_gz=kmeans_app.gz

Often, I also add --ssh --debug-on All to my command. This way, if the job fails, it will not be terminated for 72 hours; I can connect to it via SSH, look around, find the problem, fix and re-run it on the spot.

As before in Method 1, the job will soon start and provide the URL to access it. Open it to see the familiar interface of the K-means Shiny app.

Conclusions

We created the DNAnexus applet for Shiny Server. It can take an archive of a Shiny app code and run it. You can run different Shiny apps on this server, which makes it a versatile tool. There are caveats, however, the most obvious and important ones being that this applet does not take any input files from its DNAnexus project, neither can it store results back to the project. These can be addressed by adding additional input and output slots during the creation of the app, just like we did it for the app_gz input. There is an alternative to this: take a look at the next method.

Method 3. Mount your project as a folder

Many Shiny apps out there will require one or multiple input files. They may also need to save some results as files. Shuttling files back and forth from your project could be done by adding inputs and outputs to your app. (See Method 2, Conclusions chapter.) However, it may not be flexible enough for your case. Much more flexible way is to mount our project to our web app as yet another folder. The Shiny app would effortlessly open project files as if they were local ones, and would save outputs the same way, being completely agnostic that these files are physically stored elsewhere, maybe even on the other side of the globe.

Enter dxFUSE ( https://github.com/dnanexus/dxfuse/ ). This is our new open-source project, still in Beta mode. It allows us to mount one or more DNAnexus projects as local folders. We can use it for our scenario.

Let's create a new applet

1. Create new applet

Invoke the dx-app-wizard and provide the the same answers except these:

App Name: fused_shiny
Title []: dxFUSE Shiny Server
Summary []: Shiny Server with mounted DNAnexus project

2. Modify the main app script

Now modify the fused_shiny.sh; add code to download the dxfuse tool, set it up, mount your DNAnexus project as a folder, and then mount this folder to Shiny Server's docker image. The code is a modification of the script from dx_shiny app and a bit too long to quote here, but here is the key piece:

# Mount the parent project using dxFUSE
wget https://github.com/dnanexus/dxfuse/releases/download/v0.21/dxfuse-linux
...
FUSE_MOUNT=$HOME/projects
mkdir -p $FUSE_MOUNT
sudo -E ./dxfuse-linux -uid $(id -u) -gid $(id -g) -verbose 2 $FUSE_MOUNT $DX_PROJECT_CONTEXT_ID
...
...
docker run --rm -p 443:3838 -v $PWD/app:/srv/shiny-server/ -v $PROJ_PATH:/srv/project/ rocker/shiny

Basically, we download the dxfuse tool and set it up. We also dxfuse-mounted our DNAnexus project as a folder, and then mounted this folder to Shiny Server's docker image.

Conclusions

Thus, we created the new Shiny Server applet with the DNAnexus project mounted as a folder. This way, our Shiny apps can read and write to the project directly, which in many cases will be more convenient than the previous methods. To do this, we used the new open-source project dxFUSE. Please keep in mind that it is still in Beta mode and may have some bugs.

Final Thoughts

In these examples, I described how easy it is to wrap the Shiny Server into a DNAnexus web application. I tried to keep the scenarios simple yet satisfy popular use cases. However, there are lots of different Shiny apps and lots of different use cases; the methods described above may not fully fit yours. This should provide you a good starting point to build on. To further improve your apps and knowledge, you can, for instance, create monolithic docker images containing both Shiny Server and your app, as well as configuration files and other data, you can start from a Dockerfile of the Shiny Server and modify it. Examples of such Dockerfiles can be found on the internet. Creating an Rstudio Pro server is also possible using the same methods. The code and example Docker files can be found on their Github site. (Remember: you will need a license.)