Some notes on packaging

Overall, providing an AUR package is a simple process. Using any existing PKGBUILD can serve as a template. However, I had to make changes to few PKGBUILDS because not all releases on GitHub follow the same directory structure. Also sometimes files were named differently then expected, for example having LICENSE.md instead of just LICENSE would promt changes to the packaging files.

Additionally, it seems the authors of these tools were not always too mindful of other tools that exist with the same name. This is the case with air (a Go package), ark (the KDE archiving tool), and rig (A random identity generator) where binaries for other purposes alreary exist in the Linux distribution, so renaming those to r-air, r-ark, and r-rig was necessary to avoid conflicts.

Anyway, hopefully people find these useful. And if you try them out please report any issues.

Setting up a Ducklake

Assuming you have opened a new Rstudio project with git enabled, and the duckdb package is installed on your system setting up Ducklake is as follows:

library("duckdb")

con <- dbConnect(duckdb(), dbdir=":memory:")

dbExecute(con, "INSTALL ducklake;")

dbExecute(con, "ATTACH 'ducklake:metadata.ducklake' AS r_ducklake;")

dbExecute(con, "USE r_ducklake;")

This creates a in memory duckdb database, installs the ducklake extension and then creates the ducklake on disk. In your project you will see two files: metadata.ducklake and metadata.ducklake.wal. The first one is a duckdb database with a different extension and the second is the write-ahead log that duckdb uses to manage the metadata.

If we close this connection (but if you do this remember to run the above again before continuing below:

dbDisconnect(con)

And then open the ducklake file with a database browser, for example the duckdb cli:

$ duckdb -ui metadata.ducklake 

you will see that it is a database with the tables as described on the specification overview for the table format.

Writing data to the ducklake

Next I will pretend that the mtcars dataset needs to be cleaned and aggregated. There may be a better datasets to give this example, but it is simple to see the steps like this. I am also using SQL but same can be achieved with dplyr or anything handy in R:

# Bronze: raw data
dbWriteTable(con, "bronze_mtcars", mtcars)

# Silver: filtered / cleaned
dbExecute(con, "
  CREATE TABLE silver_mtcars AS
  SELECT *, (mpg/mean(mpg) OVER()) AS mpg_norm
  FROM bronze_mtcars;
")

# Gold: aggregated summary
dbExecute(con, "
  CREATE TABLE gold_mtcars AS
  SELECT cyl, AVG(mpg) AS avg_mpg, AVG(mpg_norm) AS avg_mpg_norm
  FROM silver_mtcars
  GROUP BY cyl;
")

In the medallion architecture the first layer, the raw data, is called bronze, and in this case we just write the mtcars as is.

Then the second layer, the silver layer, we sort of clean these data. I have no idea if normalizing miles per gallon makes sense, but hey why not.

The final layer is about an aggregate summary, a gold layer, so group by and summarize is what is happening there.

If you run the above in your demo project you will notice new files created:

[metadata.ducklake.files]$ tree
.
└── main
    ├── bronze_mtcars
    │   └── ducklake-01996884-c393-7134-bee0-cbffb31e19c7.parquet
    ├── gold_mtcars
    │   └── ducklake-01996884-c3cd-7ae6-bcaa-0b69196040cb.parquet
    └── silver_mtcars
        └── ducklake-01996884-c3b5-7d9d-b9f8-05e3c8377bee.parquet

This is ducklake keeping track of your data in parquet format.

Next we need to make something to track the relationships between the tables. For now i think this is not supported in ducklake(v.0.0.3), but probably will be in the future.

Ideally the following should be executed as each table is written, but for sake of example it is below:

dbExecute(con, "
CREATE TABLE IF NOT EXISTS ducklake_lineage (
  parent_table VARCHAR,
  child_table  VARCHAR,
  created_at   TIMESTAMP,
  description  VARCHAR
);
")

# After creating a silver table from bronze
dbExecute(con, "
    INSERT INTO ducklake_lineage (parent_table, child_table, created_at, description)
    VALUES ('bronze_mtcars', 'silver_mtcars', NOW(), 'normalization / filtering');
")

# After creating gold table from silver
dbExecute(con, "
    INSERT INTO ducklake_lineage (parent_table, child_table, created_at, description)
    VALUES ('silver_mtcars', 'gold_mtcars', NOW(), 'aggregation');
")

Let’s imagine that we have done some more transformation and maybe created 2 more silver tables and one more gold table. Of course, the decription should be updated to make more sense:

dbExecute(con, "
    INSERT INTO ducklake_lineage (parent_table, child_table, created_at, description)
    VALUES ('bronze_mtcars', 'silver_mtcars_2', NOW(), 'normalization / filtering');
")

dbExecute(con, "
    INSERT INTO ducklake_lineage (parent_table, child_table, created_at, description)
    VALUES ('bronze_mtcars', 'silver_mtcars_3', NOW(), 'normalization / filtering');
")

dbExecute(con, "
    INSERT INTO ducklake_lineage (parent_table, child_table, created_at, description)
    VALUES ('silver_mtcars', 'gold_mtcars_2', NOW(), 'aggregation');
")

Visualize relationships among tables

Since digging trough the parquet files to see what the relationships between the tables are does not make much sense, we can use some of good old ggplot2 and friend magic to visualize that.

library(igraph)
library(ggraph)

edges <- dbGetQuery(con, "SELECT parent_table AS from_table, child_table AS to_table FROM ducklake_lineage")
g <- graph_from_data_frame(edges, directed = TRUE)

ggraph(g, layout = "tree") +
  geom_edge_diagonal(arrow = arrow(length = unit(4, 'mm')), end_cap = circle(3, 'mm')) +
  geom_node_point(shape = 15, size = 8, color = "steelblue", fill = "lightblue") + 
  geom_node_text(aes(label = name), vjust = -1.2, size = 4) +               
  coord_flip() +
  scale_y_reverse() +  
  theme_void() +       
  ggtitle("DuckLake Medallion Lineage")

This is the sort of a thing you get to see under the lineage in Databricks’ Unity Catalog, although not as elaborate. However, a proper ggplot2 wizard may succeed in making duplicating that view.

You can imagine having this plot as part of a Quarto report or a Shiny app with some interactivity to keep track of the transformations that happen to the data you are managing.

Summary

As we set out in the beginning we set up a Ducklake with R, wrote some data to it and visualized relationships among tables. While the example is trivial, I think it is obvious that this could be a useful approach for managing local pipelines and for prototyping things that could run on the cloud.

For local pipelines in particular, I can see this approach been used with something like {maestro} to orchestrate a data pipeline.

The code above is available as a gist.

Why is it named r-air?

Because there is already a binary named Air: a Go library Air. In order not to overwite the user’s system that may have the Go binary, the name of the binary for the formatter is r-air.

See more at the github discussion with Posit here.

Plumber API for filter and aggregate

The {plumber} API is organized in the form of an R package.

├── DESCRIPTION
├── inst
│   └── plumber.r
├── man
│   ├── dot-aggregate.Rd
│   ├── dot-filter.Rd
│   └── run_api.Rd
├── NAMESPACE
├── plumb.ex.Rproj
├── R
│   ├── api.r
│   └── funs.r
└── test.r

The R directory contains the script funs.r where I define the .filter and .aggregate functions I covered in the previous two posts. The .api.r script contains a single function that starts the API on our preferred host and port (defaults to localhost and 5000).

#' start plumber api
#' @param host the host address
#' @param port the port to use
#' @importFrom plumber pr pr_run pr_get
#' @export
run_api <- function(port = 5000, host = "127.0.0.1") {
  path <- system.file("plumber.r", package = "plumb.ex")
  path |>
    pr() |>
    pr_run(port = port, host = host)
}

This is similar to {golem}’s run_app() function and it streamlines the development cycle. Whenever I make updates to the underlying code in funs.r or the {plumber} endpoints in inst/plumber.r, I can run:

devtools::document() # re-documents and loads all functions 
run_api()

and that would start the API making it accessible at http://127.0.0.1:5000, with the {swagger} interface for manual testing at http://127.0.0.1:5000/__docs__/. It is then ready for queries.

Endpoints

The package exports the .filter and .aggregate functions. Then the API uses these when defining the endpoints. In this example API we have three endpoints defined in inst/plumber.r:

# plumber.R.

#* @plumber
#* @apiTitle Plumber filter or aggregate
#* @apiDescription Plumber example for dynamic filtering and aggregation
#* @apiVersion 1.0.0
function(pr) {
  pr |>
    plumber::pr_set_serializer(plumber::serializer_unboxed_json())
}

#* Hello
#* @get /hello
function() {
  "Hello, there!"
}

#* Filter
#* @param data the name of the data frame to filter, `"iris"` or `"mtcars"`
#* @param instructions_list a list of instructions for aggregation.
#* @post /filter
function(data, instructions_list) {
  d <- get(data)
  l <- jsonlite::fromJSON(instructions_list, simplifyVector = FALSE)
  .filter(d, l)
}

#* Aggregate
#* @param data the name of the data frame to aggregate, `"iris"` or `"mtcars"`
#* @param instructions_list a list of instructions for aggregation.
#* @post /aggregate
function(data, instructions_list) {
  d <- get(data)
  l <- jsonlite::fromJSON(instructions_list, simplifyVector = FALSE)
  .aggregate(d, l)
}

The first is /hello and is simply a health check endpoint. Using httr2 we can make requests to it as follows:

r$> library(jsonlite)
    library(httr2)
 
    request("http://127.0.0.1:5000/hello") |>
      req_perform() |>
      resp_body_json()
[1] "Hello, there!"

The other two endpoints are simple wrappers around .filter and .aggregate to do two steps:

1. get the dataset requested by the user. For simplicity in this example, the dataset argument is passed on as a string, and then we user get to get that dataset, assuming its one of the ones prepackaged with R. But its easy to modify that and simply send JSON data to the endpoint.

2. convert the JSON received by the API into a list as required in R. This is needed because the instructions list for filtering or aggregation is sent with the request as JSON, and the validation within our functions expects a list. One could relax these requirement as well by allowing a JSON string in the instructions_list argument and parsing this internally.

Interactive demo

devtools::document() # re-documents and loads all functions 
run_api()

library(plumb.ex) # re-documents and loads all functions 
plumb.ex::run_api(port = 6000)

library(jsonlite)
library(httr2)

filter_data <- list(
  list(col = "Sepal.Length", fun = "between", min = 4.9, max = 5),
  list(col = "Species", fun = "in", val = c("setosa", "versicolor"))
)

filter_json <- jsonlite::toJSON(filter_data, auto_unbox = TRUE)
jsonlite::prettify(filter_json)

[
    {
        "col": "Sepal.Length",
        "fun": "between",
        "min": 4.9,
        "max": 5
    },
    {
        "col": "Species",
        "fun": "in",
        "val": [
            "setosa",
            "versicolor"
        ]
    }
]
 

request("http://127.0.0.1:6000") |>
  req_url_path("filter") |>
  req_body_json(
    data = list(data = "iris", instructions_list = filter_json),
    auto_unbox = TRUE
  ) |>
  req_perform() |>
  resp_body_json(simplifyVector = TRUE)

   Sepal.Length Sepal.Width Petal.Length Petal.Width    Species
1           4.9         3.0          1.4         0.2     setosa
2           5.0         3.6          1.4         0.2     setosa
3           5.0         3.4          1.5         0.2     setosa
4           4.9         3.1          1.5         0.1     setosa
5           5.0         3.0          1.6         0.2     setosa
6           5.0         3.4          1.6         0.4     setosa
7           4.9         3.1          1.5         0.2     setosa
8           5.0         3.2          1.2         0.2     setosa
9           4.9         3.6          1.4         0.1     setosa
10          5.0         3.5          1.3         0.3     setosa
11          5.0         3.5          1.6         0.6     setosa
12          5.0         3.3          1.4         0.2     setosa
13          4.9         2.4          3.3         1.0 versicolor
14          5.0         2.0          3.5         1.0 versicolor
15          5.0         2.3          3.3         1.0 versicolor

library(jsonlite)
library(httr2)

aggregate_instructions <- list(
  groups = list(col = "Species"),
  aggregates = list(
    list(col = "Sepal.Length", fun = "mean"),
    list(col = "Sepal.Width", fun = "mean")
  )
)

aggregate_json <- jsonlite::toJSON(aggregate_instructions, auto_unbox = TRUE)
jsonlite::prettify(aggregate_json)

{
    "groups": {
        "col": "Species"
    },
    "aggregates": [
        {
            "col": "Sepal.Length",
            "fun": "mean"
        },
        {
            "col": "Sepal.Width",
            "fun": "mean"
        }
    ]
}
 

request("http://127.0.0.1:5000") |>
  req_url_path("aggregate") |>
  req_body_json(
    data = list(data = "iris", instructions_list = aggregate_json),
    auto_unbox = TRUE
  ) |>
  req_perform() |>
  resp_body_json(simplifyVector = TRUE)

     Species Sepal.Length_mean Sepal.Width_mean
1     setosa              5.01             3.43
2 versicolor              5.94             2.77
3  virginica              6.59             2.97

How is this useful?

I struggle with this sometimes my self. Isn’t this why we have databases to begin with? Of course, having an API that queries a database would be preferable in most applications requiring large amounts of data. However, sometimes we don’t have the expertise, bandwidth, or budget for a database setup, or we could be working with local files, in arrow, parquet, qs or some other format where it might make sense or is justifiable to run the queries in R it self.

Summary

In this post, we demonstrated how to create a {plumber} API that provides endpoints for dynamic filtering and aggregation of datasets in R. We structured the API as an R package, defined the necessary functions, and set up the endpoints to handle JSON requests. We also provided examples of how to interact with the API using httr2 from R. This approach can be useful for scenarios where a database setup is not feasible, and we need to work with data files directly in R.

Dynamic aggregation

As with the dynamic filter, we want something of the form:

.aggregate(my_data, my_list_of_instructions)

For aggregation to work, we need to specify at least one grouping variable, and then the names and aggregation functions for the columns we would like to summarize. So the instructions list should have the form:

aggr_list <- list(
    groups = list(col = "group_var"),
    aggregates = list(
        list(col = "column 1", fun = "mean"),
        list(col = "column 2", fun = "sd"),
        ...
    )
)

To show an example, we could get the mean, median, and standard deviation of Sepal Length in species from the iris dataset:

d <- iris
l <- list(
  groups = list(col = "Species"),
  aggregates = list(
    list(col = "Sepal.Length", fun = "mean"),
    list(col = "Sepal.Length", fun = "median"),
    list(col = "Sepal.Length", fun = "sd")
  )
)
.aggregate(d, l)

Joining with `by = join_by(Species)`
Joining with `by = join_by(Species)`

     Species Sepal.Length_mean Sepal.Length_median Sepal.Length_sd
1     setosa              5.01                 5.0            0.35
2 versicolor              5.94                 5.9            0.52
3  virginica              6.59                 6.5            0.64

As we can tell, we in the aggregates list, we can specify the same column multiple times, and the resulting table will only contain the columns for which we’ve specified an aggregation.

A more complex example would be to group on more than variable and summarize multiple columns. For example, group by cylinder, gear and carburetor and find the mean of miles per gallon, the total horse power and the median weight of cars in the mtcars dataset:

d <- mtcars
l <- list(
  groups = list(col = c("cyl", "gear", "carb")),
  aggregates = list(
    list(col = "mpg", fun = "mean"),
    list(col = "hp", fun = "sum"),
    list(col = "wt", fun = "median")
  )
)
.aggregate(d, l)

Joining with `by = join_by(cyl, gear, carb)`
Joining with `by = join_by(cyl, gear, carb)`

   cyl gear carb mpg_mean hp_sum wt_median
1    6    4    4    19.75    466      3.16
2    4    4    1    29.10    290      2.07
3    6    3    1    19.75    215      3.34
4    8    3    2    17.15    650      3.48
5    8    3    4    12.62   1140      5.25
6    4    4    2    24.75    318      2.96
7    8    3    3    16.30    540      3.78
8    4    3    1    21.50     97      2.46
9    4    5    2    28.20    204      1.83
10   8    5    4    15.80    264      3.17
11   6    5    6    19.70    175      2.77
12   8    5    8    15.00    335      3.57

Neat. Again, the motivation here is not to replace the sweet {dplyr} syntax, rather to come up with a mechanism to do complex aggregation in one step, by creating an instructions list (or JSON) which, again, in {shiny} or {plumber} context would most likely be constructed programmatically.

How does it work?

Similar to our filtering function, we have two sections. First is input validation, again using {checkmate} because it’s awesome. We can make sure that the inputs are of correct type, that they are named, and that the columns specified as grouping and aggregation variables are present in the dataset. Likewise, we ensure that the requested aggregation functions are supported.

In the second section, we first convert the string passed to fun to a function, so it can be used downstream in dplyr::summarize. We do this with a helper function that can be called at the time summarize executes.

Then, we loop over the contents of the aggregates sub-list and apply each aggregation independently resulting in a list of aggregated datasets. For example, if we specified the mean, median, and sd for Sepal Length by Species, at this stage we would have a list of three data frames with the grouping column and another column representing the aggregate.

Finally, to collate the tables we use reduce with left_join, essentially joining the three data frames by species.

Note the nice argument .by for dplyr::summarize. In this case we just send the string from the groups$col slot in our instructions list. i.e we don’t need to use dplyr::group_by and the NSE construct it would require (converting it to symbol, and !! it).

#' Aggregate a dataset
#' @param .data the data frame
#' @param .arglist a list of column names to group by
#' @examples
#' d <- iris
#' l <- list(
#'   groups = list(col = "Species"),
#'   aggregates = list(
#'     list(col = "Sepal.Length", fun = "mean"),
#'     list(col = "Sepal.Width", fun = "mean")
#'   )
#' )
#' .aggregate(d, l)
#' @return a data frame with the columns aggregated
#' @export
.aggregate <- function(.data, .arglist) {
  checkmate::assert_data_frame(.data)
  checkmate::assert_list(.arglist, types = "list")
  checkmate::assert_named(.arglist)
  checkmate::assert_subset(
    names(.arglist),
    choices = c("groups", "aggregates")
  )
  checkmate::assert_list(.arglist$groups, len = 1)
  checkmate::assert_subset(
    purrr::pluck(.arglist, "groups", "col"),
    choices = names(.data)
  )
  checkmate::assert_subset(
    purrr::pluck(.arglist, "aggregates", "col"),
    choices = names(.data)
  )
  checkmate::assert_subset(
    purrr::pluck(.arglist, "aggregates", "fun"),
    choices = c(
      "mean", "median", "sum", "min", "max", "sd", "var", "count"
    )
  )

  .get_aggr_fun <- function(x) {
    switch(x,
      "mean" = mean,
      "median" = stats::median,
      "sum" = sum,
      "min" = min,
      "max" = max,
      "sd" = sd,
      "var" = var,
      "count" = length
    )
  }

  aggr_dataset <- purrr::map(
    .arglist$aggregates,
    .f = ~ dplyr::summarise(
      .data,
      !!paste(.x$col, .x$fun, sep = "_") :=
        .get_aggr_fun(.x$fun)(!!rlang::sym(.x$col)),
      .by = .arglist$groups$col
    )
  ) |>
    purrr::reduce(dplyr::left_join) |>
    dplyr::mutate_if(is.numeric, round, 2)

  aggr_dataset
}

In a {shiny} application, where we want to enable the user to select any column(s) to aggregate with some choices for aggregating function, it would require quite a bit of if/else logic to capture all the cases manually. Using this alternative, we simply harvest the user’s selections from the input and construct the list of instructions before plugging it into .aggregate to obtain our results.

Summary

In this post, we explored a method for dynamically aggregating datasets in R using purrr and dplyr. We created a function .aggregate that takes a data frame and a list of instructions specifying the grouping variables and the aggregation functions to apply. This approach is particularly useful in non-interactive or automated workflows, such as in {shiny} applications or {plumber} APIS, where the user might specify different aggregation requirements.

We demonstrated the usage of the .aggregate function with examples using the iris and mtcars datasets, showing how to group by one or more variables and apply various aggregation functions. The function ensures input validation using {checkmate} and dynamically applies the specified aggregation functions, collating the results into a single data frame.

This method provides a flexible and efficient way to perform complex aggregations programmatically, reducing the need for extensive if/else logic and making it easier to handle user-defined aggregation instructions.

Interactive and dynamic filtering

The {tidyverse}, and {dplyr} in particular, have made data manipulation in R a breeze. In an interactive setting, the pattern:

my_data |> 
    dplyr::filter(my_col == my_value) |>
    ...

gives a lot of power and joy to everyone, including beginner programmers.

However, we often run into hurdles when trying to translate our script written for interactive analysis into code that can be run automatically. Where we won’t know ahead of time what column to filter on, or what values we should retain.

In such cases, we often write if statements, along the lines of:

if (x == "my_col1") {
    dplyr::filter(my_col1 == my_val1)
} else if (x == my_col2) {
    dplyr::filter(my_col2 == my_val2)
} ...

Its easy to see how such approaches become long-winded, error prone, slow, and result in untidy, less extensible and maintainable code.

This is common, for example if you have a {plumber} API and an arbitrary user attempts to filter an arbitrary table in a data base. Or if you have a {shiny} application where the user uploads a table, with columns unknown to the code and then select one or more of them to filter on.

I have faced this many times, and have always wanted a solution where I can send instructions for filtering in the form of a named list, and get the filtered table. For this to work, the named list would need specify the column, filtering function, and values to retain (min, max, val). Then a function would take each element of the list, filter the dataset based on it, and finally return a table that combines all the filters. Or, more accurately, return the intersect of the filtered datasets.

Below is a function that solves this problem that I developed recently to add flexible dynamic filtering to a {shiny} application where the user can upload an arbitrary dataset, then select arbitrary filter columns and values and obtain the joint filtered result.

To find the iris flowers with sepal length between 4.9 and 5 and from species setosa and versicolor, we would write the following:

suppressPackageStartupMessages({
    library(checkmate)
    library(dplyr)
    library(purrr)
})
l <- list(
  list(col = "Sepal.Length", fun = "between", min = 4.9, max = 5),
  list(col = "Species", fun = "in", val = c("setosa", "versicolor"))
)
.filter(iris, l)

   Sepal.Length Sepal.Width Petal.Length Petal.Width    Species
1           4.9         3.0          1.4         0.2     setosa
2           5.0         3.6          1.4         0.2     setosa
3           5.0         3.4          1.5         0.2     setosa
4           4.9         3.1          1.5         0.1     setosa
5           5.0         3.0          1.6         0.2     setosa
6           5.0         3.4          1.6         0.4     setosa
7           4.9         3.1          1.5         0.2     setosa
8           5.0         3.2          1.2         0.2     setosa
9           4.9         3.6          1.4         0.1     setosa
10          5.0         3.5          1.3         0.3     setosa
11          5.0         3.5          1.6         0.6     setosa
12          5.0         3.3          1.4         0.2     setosa
13          4.9         2.4          3.3         1.0 versicolor
14          5.0         2.0          3.5         1.0 versicolor
15          5.0         2.3          3.3         1.0 versicolor

How does it work

The function below has two sections. First we do input validation using {checkmate}. In a non-interactive setting, we can’t rely on the inputs being always correct, so validating, and returning meaningful errors is essential. For this job, I prefer checkmate because it has a nice and consistent interface with many built-in checking, asserting, and testing functions, but any other approach, including base R’s stopifnot or assertthat would work.

The second part is the filtering bit, where we have purrr::map cycle over each element of the filter list, grab the required filter function and go into one of two paths. If the column is numeric (date, datetime, integer, double), we use dplyr::between to filter. If, the column is character (string), factor or logical, we go with %in%.

After cycling over all elements of the filter list, the output would be a list of filtered datasets. We must now combine these into a single table, i.e. the table that we would get if we applied all these filters in one call, or sequentially. To do this, we use a bit more functional programming magic with purrr::reduce with the intersect function from dplyr. This final step, reduces the list of data frames to a single table containing only the rows that are shared between the data frames.

#' Filter a dataset
#' @param .data the data frame
#' @param .arglist a list of filter specifications
#' @examples
#' d <- iris
#' l <- list(
#'   list(col = "Sepal.Length", fun = "between", min = 4, max = 5),
#'   list(col = "Species", fun = "in", val = c("setosa", "versicolor"))
#' )
#' .filter(d, l)
#' @import checkmate
#' @import purrr
#' @import dplyr
#' @export
.filter <- function(.data, .arglist) {
  checkmate::assert_data_frame(.data)
  checkmate::assert_list(.arglist, types = "list")
  checkmate::assert_subset(
    purrr::map_chr(.arglist, purrr::pluck, "col"),
    choices = names(.data)
  )
  checkmate::assert_subset(
    purrr::map_chr(.arglist, purrr::pluck, "fun"),
    choices = c("in", "between")
  )

  purrr::map(.arglist, function(x) {
    col <- x$col
    fun <- x$fun
    if (fun == "between") {
      min <- x$min
      max <- x$max
      if (is.null(min) || is.null(max)) {
        return(.data)
      }
      return(
        dplyr::filter(.data, dplyr::between(
          !!rlang::sym(col), min, max
        ))
      )
    }
    if (fun == "in") {
      val <- x$val
      if (is.null(val)) {
        return(.data)
      }
      return(dplyr::filter(.data, `%in%`(
        as.character(!!rlang::sym(col)), val
      )))
    }
  }) |> purrr::reduce(dplyr::intersect)
}

To show another example, filter cars with mpg between 15 and 20 and cyl in 4 or 6

l <- list(
    list(col = "mpg", fun = "between", min = 15, max = 20),
    list(col = "cyl", fun = "in", val = c(4, 6))
)
.filter(mtcars, l)

              mpg cyl  disp  hp drat   wt  qsec vs am gear carb
Valiant      18.1   6 225.0 105 2.76 3.46 20.22  1  0    3    1
Merc 280     19.2   6 167.6 123 3.92 3.44 18.30  1  0    4    4
Merc 280C    17.8   6 167.6 123 3.92 3.44 18.90  1  0    4    4
Ferrari Dino 19.7   6 145.0 175 3.62 2.77 15.50  0  1    5    6

Pretty neat. In future posts, I’ll show how I use this in {shiny} context, where the construction of the filter list should also be automated.

Summary

In this post, we explored a method to dynamically filter datasets in R using purrr and dplyr. We discussed the limitations of traditional filtering approaches and introduced a flexible function that accepts a list of filter specifications. This function validates inputs using checkmate and applies the filters using purrr and dplyr. We demonstrated its usage with examples on the iris and mtcars datasets. This approach is particularly useful in interactive applications like shiny, where users can upload arbitrary datasets and apply custom filters dynamically.

Prep code

options(browser = "/usr/bin/firefox")
options(launch.browser = TRUE)

library(shiny)
library(brochure)
library(cachem)

unlink("brochure_cache", recursive = TRUE)
brochure_cache <- cachem::cache_disk("brochure_cache")

Variables are not shared among {brochure} pages

One of the features of {brochure} is that each page runs in its own session and is ignorant of variables, reactives, inputs, etc. from other pages. We can show this easily with this basic example. The variable some_var is created in the home page (/) and when we try to renderPrint it in page2 we get an error that the object is not found.

brochureApp(
  # First page
  page(
    href = "/",
    ui = fluidPage(
      h1("This is my first page")
    ),
    server = function(input, output, session) {
      some_var <- 10
    }
  ),
  # Second page
  page(
    href = "/page2",
    ui = fluidPage(
      h1("This is my second page"),
      verbatimTextOutput("some_var")
    ),
    server = function(input, output, session) {
      output$some_var <- renderPrint(some_var)
    }
  )
)

Sharing variables with caching to disk

As we said, we have to store some_var somewhere outside the session of the page where it was created so we can retrieve it in the server of another page. The chunk below does that, in / we set the cache value some_var and in page2 we get it, and can print it in the UI.

brochureApp(
  # First page
  page(
    href = "/",
    ui = fluidPage(
      h1("This is my first page")
    ),
    server = function(input, output, session) {
      some_var <- 10
      brochure_cache$set("some_var", some_var)
    }
  ),
  # Second page
  page(
    href = "/page2",
    ui = fluidPage(
      h1("This is my second page"),
      verbatimTextOutput("some_var")
    ),
    server = function(input, output, session) {
      some_var <- brochure_cache$get("some_var")
      output$some_var <- renderPrint(some_var)
    }
  )
)

Now, this is all great, except if we have dozens of variables that we need to share between pages. In this case we’d have to individually cache each of them or update a list and then re-cache the list every time its updated. Or if we need to cache an input, we’d need an observer to monitor that input and cache it on each change, potentially adding unnecessary workload to the session. It becomes cumbersome quickly, and the code gets long and potentially complex, for a relatively basic operation.

brochureApp(
  # First page
  page(
    href = "/",
    ui = fluidPage(
      h1("This is my first page")
    ),
    server = function(input, output, session) {
      var1 <- 10
      brochure_cache$set("var1", var1)
      var2 <- 11
      brochure_cache$set("var2", var2)
      var3 <- 12
      brochure_cache$set("var3", var3)
      var4 <- 13
      brochure_cache$set("var4", var4)
      var5 <- 14
      brochure_cache$set("var5", var5)
      var6 <- 15
      brochure_cache$set("var6", var6)
    }
  ),
  # Second page
  page(
    href = "/page2",
    ui = fluidPage(
      h1("This is my second page"),
      verbatimTextOutput("some_var")
    ),
    server = function(input, output, session) {
      vars <- sapply(1:6, \(i) brochure_cache$get(paste0("var", i)))
      output$some_var <- renderPrint(sum(vars))
    }
  )
)

Simplifying the caching of several variables

What I was hoping for at the time is for a single-step caching of the state of the page session right before the user navigates to another page. It did not occur to me that a mechanism for this was already available in {shiny}. Simply use onSessionEnded to harvest and cache the variables you need to share with another page. So simple.

brochureApp(
  # First page
  page(
    href = "/",
    ui = fluidPage(
      h1("This is my first page")
    ),
    server = function(input, output, session) {
      var1 <- 1
      var2 <- 2
      var3 <- 3
      var4 <- 4
      var5 <- 5
      var6 <- 6

      onSessionEnded(function(x) {
        lapply(1:6, \(i) {
          brochure_cache$set(paste0("var", i), get(paste0("var", i)))
        })
      })
    }
  ),
  # Second page
  page(
    href = "/page2",
    ui = fluidPage(
      h1("This is my second page"),
      verbatimTextOutput("some_var")
    ),
    server = function(input, output, session) {
      vars <- sapply(1:6, \(i) brochure_cache$get(paste0("var", i)))
      output$some_var <- renderPrint(sum(vars))
    }
  )
)

Regardless of how many times var1 is updated during the session, we don’t have to worry about caching it. It will get done when the session is closed, i.e. when the user navigates to another page or closes the web page.

Cleaner approach. A list to store variables to be cached

Thinking about this some more, it certainly makes sense to have a page-level list of variables that need to be cached. Then, when we cache on session end, we can cycle over the contents of that list, not dozens of individual variables. Along the lines of:

brochureApp(
  # First page
  page(
    href = "/",
    ui = fluidPage(
      h1("This is my first page")
    ),
    server = function(input, output, session) {
      var1 <- 1
      var2 <- 2
      var3 <- 3
      var4 <- 4
      var5 <- 5
      var6 <- 6

      to_cache_list <- list()
      to_cache_list$var1 <- var1
      to_cache_list$var3 <- var3
      to_cache_list$var5 <- var5

      onSessionEnded(function() {
        lapply(names(to_cache_list), \(x) {
          brochure_cache$set(x, to_cache_list[[x]])
        })
      })
    }
  ),
  # Second page
  page(
    href = "/page2",
    ui = fluidPage(
      h1("This is my second page"),
      verbatimTextOutput("some_var")
    ),
    server = function(input, output, session) {
      var1 <- brochure_cache$get("var1")
      var3 <- brochure_cache$get("var3")
      var5 <- brochure_cache$get("var5")
      output$some_var <- renderPrint(sum(var1, var3, var5))
    }
  )
)

Still to cumbersome? Cache the kitchen sink

Finally, we can take this a step further, perhaps unwisely, and store the entire environment. Within onSessionEnded, we define a callback function whose environment is a child of the server’s environment. So we can collect the contents of the parent environment, convert to a list for easier handling, and simply cache all of it as pageX_data. Then, when we need to retrieve a value from pageX, we load the cache and index it with the benefit of knowing that the cache’s structure is going to reflect the structure of the pageX environment.

brochureApp(
  # First page
  page(
    href = "/",
    ui = fluidPage(
      h1("This is my first page"),
      sliderInput("slide", "Shared input", 1, 10, 1)
    ),
    server = function(input, output, session) {
      observe({
        slide_value <- brochure_cache$get("page_1_data")[["input"]][["slide"]]
        if (!is.null(slide_value)) {
          updateSliderInput(
            inputId = "slide",
            label = "Shared input",
            value = slide_value
          )
        }
      })
      var1 <- 6
      var2 <- 5
      var3 <- 4
      var4 <- 3
      var5 <- 2
      var6 <- 1

      onSessionEnded(function() {
        env <- environment()
        page_env <- as.list(parent.env(env))
        page_env$output <- NULL
        page_env$session <- NULL
        brochure_cache$set("page_1_data", page_env)
      })
    }
  ),
  # Second page
  page(
    href = "/page2",
    ui = fluidPage(
      h1("This is my second page"),
      h5("Sum of some variables"),
      verbatimTextOutput("some_var"),
      h5("Input from previous page"),
      verbatimTextOutput("slide_input")
    ),
    server = function(input, output, session) {
      var1 <- brochure_cache$get("page_1_data")[["var1"]]
      var3 <- brochure_cache$get("page_1_data")[["var3"]]
      var5 <- brochure_cache$get("page_1_data")[["var5"]]
      output$some_var <- renderPrint(sum(var1, var3, var5))
      slide <- reactive(brochure_cache$get("page_1_data")[["input"]][["slide"]])
      output$slide_input <- renderPrint(slide())
    }
  )
)

The example above also covers caching the input R6 from the {shiny} session and removes the output and session object, although if needed those could be cached as well. Apart from some input value being needed in another page, caching the input makes sense so we can update the input widget when we return to pageX. Again, as each page is its own session, unlike base {shiny}, when we return to pageX the input will be re-initialized with the value set in the code, not with the value previously set by the user, so updating it is necessary.

This last approach comes with a caveat. Its almost never a good idea to cache everything. The input object, for example, is a large R6 object. There could be data frames, lists, model results, etc within the session that have a large footprint and are not used in other pages. This could inflate the size of the cache or cause slowdowns when navigating between pages if saving takes a bit.

Personally, I prefer the approach of a page-level list of objects to be cached. It is more explicit and can be curated to minimize the size of the cache and potentially save time.

Summary

In this post, we explored various methods for sharing data between pages in a multi-page {brochure} {shiny} application. We started with a basic example demonstrating the isolation of sessions in {brochure} and the need for caching to share data. We then showed how to use cachem to store and retrieve variables between pages. To simplify the process, we introduced the use of onSessionEnded to cache variables at the end of a session. Finally, we discussed caching the entire environment and the potential pitfalls of this approach. These methods can help streamline data sharing in multi-page {brochure} applications, reducing repetitive code and improving maintainability.

Config JSON

Nothing particularly novel here. We just create a JSON file with a simple structure. The “meta” section has general info about the application and the client (logo, homepage, etc.). The “general” section contains info on the title of the particular app instance, and path to the data object. Depending on the needs this can be a folder (bucket) with several files, or a single file (like in the simple example here). Finally, we have the “pages” section where add information specific to the different pages/tabs the application has.

Some of the configurability comes in here. We can have separate pages for different types of data or subscription levels and enable/disable pages from this section accordingly.

{
  "meta" : {
    "name" : "Shiny golem config",
    "version" : "0.0.1",
    "description" : "An example of an app configured by a config file in the cloud",
    "author" : "Discindo",
    "email" : "hello@discindodata.com",
    "logo" : "www/logo.png",
    "favicon" : "favicon.ico",
    "theme" : "default",
    "language" : "en",
    "homepage" : "discindo.org"
  },
  "general" : {
    "title" : "Test Title",
    "subtitle" : "A little longer subtitle just to show it (disable with `null`)",
    "data_path" : "https://raw.githubusercontent.com/discindo/shiny-golem-json-config/refs/heads/main/iris.csv",
    "data_format" : "csv"
  },
  "pages" : {
   "home" : {
        "title" : "Home",
        "icon" : "home",
        "url" : "/",
        "template" : "inst/html/home.html",
        "ui" : null,
        "server" : null
    },
    "data" : {
        "title" : "Data",
        "icon" : "table",
        "url" : "/data",
        "template" : null,
        "ui" : "mod_data_ui",
        "server" : "mod_data_server"
    },
    "about" : {
        "title" : "About",
        "icon" : "info",
        "url" : "/about",
        "template" : "inst/html/about.html",
        "ui" : null,
        "server" : null
    }
  }
}

The config can be placed in the inst folder of the R package (if {golem}) or some other folder that is in the resource path of the Shiny app. But it can also live in the remote cloud location. Of course the Shiny app should have the required permissions to access these files.

In the example here, to keep things simple, the ‘remote-data’ folder is a public folder in the GitHub repository for the code used in this post. No additional files or variables are needed to access the config JSON and the data files. But if the data are in a private bucket at a cloud provider, the app would need to have read permissions for that bucket. Either through AWS IAM credentials, or Google cloud token, etc..

Shiny code

The Shiny app will need to load the config JSON from the cloud or local location, and use the configurations set by the user to generate the right content. In the example, I am using the {golem} setup, so there are separate app_ui.r and app_server.r scripts. Each one loads the JSON config using jsonlite and accesses the slots in the resulting list for the values it needs. If there are some expensive or repeated steps performed in the ui and server functions, we could use golem options or some other mechanism to parse the config JSON only once and pass the list as an argument or option to the main ui and server functions. But it this, admittedly simplified example, we parse the config twice, as its quite small.

In the {rhino} framework, something similar can be achieved by loading the config in app.R, and then using it in functions from /logic/. And in a “base” Shiny app, a straightforward approach would be to parse the config in global.r which would make the variable available in the scope of the ui and server functions.

app_ui <- function(request) {
  config <- jsonlite::read_json("https://raw.githubusercontent.com/discindo/shiny-golem-json-config/refs/heads/main/remote-data/app-config.json")
  pages <- purrr::map(
    config$pages, function(x) {
      if (!is.null(x$template)) {
        return(bslib::nav_panel(
          title = x$title,
          shiny::htmlTemplate(x$template)
        ))
      } else if (!is.null(x$ui)) {
        return(bslib::nav_panel(
          title = x$title,
          getFunction(x$ui)(id = "data_1")
        ))
      } else {
        return(bslib::nav_panel(
          title = x$title,
          x$title
        ))
      }
    }
  )
  shiny::tagList(
    # Leave this function for adding external resources
    golem_add_external_resources(),
    # Your application UI logic
    shiny::div(
      class = "container",
      bslib::page_navbar(
        theme = bslib::bs_theme(
          bootswatch = "litera",
          base_font = bslib::font_google("Lato")
        ),
        inverse = FALSE,
        underline = FALSE,
        title = shiny::a(
          href = config$meta$homepage,
          target = "_blank",
          shiny::img(src = config$meta$logo, height = "70px")
        ),
        header = shiny::div(
          class = "p-5 text-center",
          shiny::h4(config$general$title),
          shiny::h6(config$general$subtitle)
        ),
        bslib::nav_spacer(),
        rlang::splice(unname(pages))
      )
    )
  )
}

app_server <- function(input, output, session) {
  # Your application server logic
  config <- jsonlite::read_json("inst/app-config.json")
  data <- read.csv(config$general$data_path)
  purrr::map(config$pages, function(x) {
    if (!is.null(x$server)) {
      getFunction(x$server)(id = "data_1", data = data)
    }
  })
}

remote-data/
├── app-config.json
├── iris.csv
└── mtcars.csv

Summary

In this post, we explored how to configure a Shiny application using a remote JSON file. This approach allows for flexible and dynamic updates to the app’s content and appearance without major code refactoring. We discussed the structure of the JSON configuration file, how to load it in the Shiny app, and how to use it to generate the UI and server logic dynamically. This method can be particularly useful for applications that need to display different data or configurations for different clients, enabling easy updates by simply modifying the JSON file in a cloud storage location.

Step 1: Use renv for your projects.

I will not go into details here. I presume everyone is doing this? :)

When using renv you will notice the renv.lock that are being generated.

Step 2: Create a Github repository with folder for each renv.lock file

Assuming you have multiple projects/environments you need to check and build have a directory structure where each folder has a renv.lock file. For example:

my-renv-files/
├── my-first-comlicated-env/
    └── renv.lock
└── my-second-complicated-env/
    └── renv.lock

Maybe you would want to add a README.md there as well.

Step 3: Add a requirements.txt file to the repository

This will hold any system libraries that are needed - one per line. You probably know at least one at this point so just add it on the first line.

Step 4: Add a GitHub Action yaml file to the repository.

Create a .github/workflows folder in the repository and add the following as build_renv.yaml:

name: Build renv environments

on:
  pull_request:
    paths:
      - '**/renv.lock'
  workflow_dispatch: 

jobs:
  build-renv:
    runs-on: ubuntu-latest

    steps:
      # 1. Checkout the repository
      - name: Checkout repository
        uses: actions/checkout@v4
        with:
          fetch-depth: 0

      # 2. Set up R environment
      - name: Set up R
        uses: r-lib/actions/setup-r@v2
        with:
          r-version: '4.4.1' # Specify your desired R version

      # 3. Install system dependencies
      - name: Install system dependencies
        run: |
          sudo apt-get update
          sudo apt-get install -y $(cat requirements.txt)

      # 4. Install renv package in R
      - name: Install renv
        run: |
          Rscript -e "install.packages('renv')"

      # 5. Rebuild the R environment for the changed renv.lock file
      - name: Rebuild R environment
        run: |
          # Extract the changed renv.lock file
          file=$(git diff --name-only ${{ github.event.pull_request.base.sha }} ${{ github.event.pull_request.head.sha }} | grep 'renv.lock')
          dir=$(dirname "$file")

          echo "Restoring renv environment for $dir"
          cd "$dir"
          Rscript -e "renv::restore(prompt = FALSE)"
        env:
          RENV_PATHS_CACHE: ~/.cache/R/renv

This will run automatically on any pull requests that have renv.lock files, and also can be manually triggered (good for testing!).

Then it will setup R, install the dependencies, and set up renv.

Then in the final step 5 it will try to build the environment for each folder and fail if system dependencies are missing. So then you can update the requirements.txt and have it run again until you have a green pass and a complete requiremtenst.txt that you can share with the administrator. :)

Now, this runs on Ubuntu and the server that runs the RStudio Server may be on a different Linux distribution. And while not ideal, you will still know what are the names of the missing libraries, though their names for the other distribution maybe slightly different.

Still, a nice way to keep track of renv environments, and especially convenient if you have a very specific one that you need to share with coworkers or with the public.

Deploy a project with multiple R scripts and {renv}-managed environment to AWS Lambda

It has been a while since I’ve had the chance to work on my {r2lambda} project. In particular, there were a couple of good points made by a user on GitHub about functionality that is missing from the package. The option to deploy multiple files, e.g., one runtime function that depends on helpers in the same project organized in different files. And another, to enable {renv} management of the R environment within the AWS Lambda docker image. Both excellent points that I wished I addressed earlier. But better late than never.

Both of these features required minor adjustments to the codebase. Copying additional supports scripts and restoring the {renv} environment should both happen when the AWS Lambda docker image is built, so the logic to create the Dockerfile needed to be updated. Accordinly, the r2lambda::build_lambda function now has two additional arguments:

#' @param support_path path to the support files (if any). Either NULL 
#' (the default) if all needed code is in the same `runtime_path` script, or a 
#' character vector of paths to additional files needed by the runtime script.
#' @param renvlock_path path to the renv.lock file (if any). Default is NULL.
#' 
#' @details Use either `renvlock_path` or `dependencies` to install required
#' packages, not both. By default, both are `NULL`, so the Docker image will
#' have no additional packages installed.

To include any support scripts, provide a character vector script paths to the support_path argument when building the Lambda docker image locally with build_lamdba.

[Note that, multi-file project was supported previously as well, although perhaps not explicitly. An approach that I like is to create an ‘R’ package that exports the runtime function needed for the Lambda. Then one just needs to make that custom R package a dependency of the project and either install in the AWS Lambda docker image it through dependencies or renvlock_path.]

To use an existing renv.lock for installation of dependencies, provide its path to the renvlock_path argument to build_lambda. This instructs the code to copy the renv.lock file to the image and run renv::restore() which will reconstruct the R environment inside the docker image. I really like this feature, as it minimizes the size of the Dockefile and removes some potential headaches with R package dependencies from different repositories (CRAN, BioConductor, GitHub, etc).

Demo code

Assuming we have a folder with the following structure:

~/Desktop$ ls -1 iris-lambda/
renv/
renv.lock
runtime.r
support.r
test-code.r

Where, support.r defines some function that runtime.r uses for the Lambda:

get_iris_summary_by_species <- function(species) {
    iris |>
    dplyr::filter(Species == species) |>
    dplyr::summarise(
      mean = mean(Sepal.Length),
      sd = sd(Sepal.Length)
    )
}

Then runtime.r, sources the support script, and calls the function defined there:

source("support.r")

iris_summary <- function(species) {
  get_iris_summary_by_species(species)
}

lambdr::start_lambda()

Then the following should work, passing the support script and renv.lock to r2lambda::build_lambda:

dir("~/Desktop/iris-lambda")
runtime_function <- "iris_summary"
runtime_path <- "~/Desktop/iris-lambda/runtime.r"
support_path <- "~/Desktop/iris-lambda/support.r"
renvlock_path <- "~/Desktop/iris-lambda/renv.lock"
dependencies <- NULL

# Might take a while, its building a docker image
build_lambda(
  tag = "my_iris_lambda",
  runtime_function = runtime_function,
  runtime_path = runtime_path,
  support_path = support_path,
  renvlock_path = renvlock_path,
  dependencies = dependencies
)

# test
payload <- list(species = "setosa")
tag <- "my_iris_lambda"
test_lambda(tag = tag, payload)


# deploy

# Might take a while, its pushing it to a remote repository
deploy_lambda(
  tag = "my_iris_lambda",
  Timeout = 30
)

invoke_lambda(
  function_name = "my_iris_lambda",
  invocation_type = "RequestResponse",
  payload = list(species = "versicolor"),
  include_logs = FALSE
)

invoke_lambda(
  function_name = "my_iris_lambda",
  invocation_type = "RequestResponse",
  payload = list(species = "setosa"),
  include_logs = FALSE
)

The Python deployment

Initially I was looking to publish a Python Shiny app, and indeed the rsconnect-python package seems to be able only to deploy new content even when there are no changes to the manifest.json file.

Below you can see that there are two pyshinywikidata deployments, but it is the same app. It is just what happens when I run the pipeline twice.

In other words, it seems that the command:

uvx --from rsconnect-python rsconnect deploy manifest --server=$CONNECT_SERVER --api-key=$CONNECT_API_KEY  manifest.json

does not recognize that there is already an existing content with the same manifest file on the server.

The R deployment

However, this is not the case with deploying an R Shiny app. In particular, this seems not to be the case when deploying an R Shiny app using the RStudio actions provided by Posit. This one is focused on R and doesn’t show in Google searches when looking to deploy a Python app.

Using this action in a workflow makes it possible to update existing deployments, which is great.

However, the R way of doing things has some other strange quirks, most notably that the renv.lock file should be kept updated in the repository as well. This complicates things a bit when the app that needs to be deployed is written with {golem}.

In this case, renv.init() does not know that the app needs to be installed in the deployment too, in order to be loaded. And, that additional packages are required as well. Therefore, adding these dependencies has to happen manually with renv::record(). Besides the packaged app, four more packages need to be added with renv.record(): pkgload, pkgbuild, desc, and procesx.

This complicates matters a bit, because it is another manual step that can be easily forgotten, and in that case the deployment will fail.

This is because, as the renv FAQ explains:

For a package to be recorded in the lockfile, it must be both: 1. Installed your project library, and 2. Used by the project, as determined by renv::dependencies().

Altering the renv settings may be the best approach as described in the FAQ too.

The Python deployment, again

But, if the action exists, can it be used for Python apps as well?

The answer to that is yes.

Digging through the repository for RStudio actions, and the TypeScript client library for the Posit Connect API I found out that the good people at Posit have written tests for deploying a Flask app (which is Python!). So if it works for Flask it will work for Shiny, and indeed it does.

It seems to me that rsconnect-ts offers a bit more flexibility than rsconnect-python. As far as I can tell, I cannot use rsconnect-python to check if content already exists. At the same time, it seems the rsconnect-ts, still being a beta-quality package, is not really being advertised, or maybe not even been work on actively.

Summary

In this article I tried to clarify some of my confusion regarding using GitHub Actions to deploy R and Python Shiny apps, and probably I added a bit of new confusion as well. I will try to keep this series updated as I learn new things or better ways to achieve gh-actions deployments. I would also appreciate suggestions for other relevant content if anyone has found something I have missed.

It all starts with a notification

A couple of weeks ago, I got a notification from LinkedIn. Unlike the usual notifications, this was not from an anonymous recruiter viewing my profile. It was a post by Russ Hyde who was looking for examples on how to organize the code in a Python Shiny app. He bumped into my repository on the topic. For the curious, there is also an accompanying blog post where I describe a simple approach to package a Python Shiny app.

It was a great reminder that I should look into my previous work, as I was anyhow trying to figure out different ways to deploy Python Shiny apps to Posit Connect.

At the same time the relatively new Python packager and project manager called uv caught my attention with many internet resources about its features popping up, more or less, at the same time (for example: demo project by Damjan, Unified Python packaging with uv).

So I thought why not test how Shiny for Python would work with uv and whether this package manager can be used to in a setup for deployments to Posit Connect.

Step 1: Start a new uv project and add your code

Setting up a new uv project is pretty straightforward, and their projects guide make is even simpler.

I just ran:

$ uv init py-shiny-uv

and I get a new folder with the following contents:

.
├── hello.py
├── pyproject.toml
└── README.md

No surprises here.

Then, since I already had all the code organized in the repo above, I just copied everything (including the .git folder and .gitignore file) to the uv project folder, and removed the hello.py. Now the Shiny Python application is part of the uv directory.

Step 2: Project-specific python version

Next, I needed a specific version of Python (my Posit Connect instance runs on 3.11.5).

First, I updated my pyproject.toml to have the required Python version.

Then, adding a specific Python version to the project is also simple. After navigating to the project folder, I just ran:

$ uv venv --python 3.11.5

And it gets installed in seconds. I can activate the environment in the usual way:

$ source .venv/bin/activate

And check the version:

(py-shiny-uv) $ python --version

3.11.5

This is the environment in which further development of the app could happen.

At this point I checked the git status and I have one new untracked file .python-version, which, as expected, has the Python version written inside.

Step 3: Install Shiny and other dependencies for the app

The example application depends on three packages: shiny, SPARQLWrapper, and pandas, and I added those. Adding this to the uv environment can be done with uv add:

(py-shiny-uv) $ uv add pandas shiny SPARQLWrapper

Then I just verified that the packages load by running python and importing.

Step 4: Create the requirements.txt and manifest.json needed for deploying to Posit Connect

As you might know if you are working with Shiny app in Python, the rsconnect-python package is needed to generate the manifest.json file. The manifest is used by Posit Connect when publishing from a git repository (which is something that I want to do).

Usually the way to do it is to run in the app folder:

$ rsconnect write-manifest shiny .

But since the idea is to use uv I had to try, and fail multiple times, with it.

First, the problem with rsconnect is that it generates the files in the app directory, instead of the top level python project. Moving the files is a possibility, of course, but it seems it is an unnecessary complication.

The default way to get the requirements with uv is:

$ uv export -o requirements.txt

This, however, generates the dependencies with hashes, which then is a problem with the package environment not having a hash. To quote the error log:

The editable requirement pyshinywikidata cannot be installed when requiring hashes, because there is no single file to hash.

Omitting the package with:

$ uv export --no-emit-project -o requirements.txt

Fails because now the package containing the app is no longer in requirements.txt, and Posit Connect can’t find the module to run.

Finally, after a few more trial and errors, I found the solution in the --no-hashes option of the uv export command.

Then, I needed to use uv to generate the manifest too. And uv has this nice feature where a tool can be invoked without installing it, which is handy for the rsconnect-python package. Here, the --entrypoint needs to be set up so that Posit Connect knows that the app is in the installed package.

The full uv set of commands is:

# update the project environment
$ uv sync 

# generate the the requirements.txt file
$ uv export --no-hashes -o requirements.txt 

# generate the manifest.json file. note the entrypoint.
$ uvx --from rsconnect-python --python .venv/bin/python rsconnect write-manifest shiny .  --entrypoint pyshinywikidata.app:app 

At this point git status said I have new files in the repository, as expected. So then I just added them to the repository.

Step 5: Generate requirements.txt and manifest.json with GitHub actions

Whenever changes to the code are made, requirements.txt and manifest.json may need to be regenerated and committed to the repository so that Posit Connect knows how to update the app. But, forgetting to do this would not be strange. So why not automate it?

Posit Connect can only listen to branches, so the idea is to have a deploy branch which Connect publishes, but which is managed by GitHub actions.

With a little help from existing yaml files, I stitched together a workflow script that creates the needed files on the deploy branch and then successfully deployed it to Posit Connect.

Amazing!

Additionally, I needed to allow workflow permission in my repository settings to be able to read and write. That’s under Settings -> Select Actions → General -> Workflow -> Read and write permissions.

All the code and my trials and errors are under the repo at: https://github.com/novica/pyshinywikidata/.

Summary

In this article, I reviewed the procedure of setting up a uv project manager environment for a Python Shiny application and integrating the project with GitHub Actions to enable automated deployment to Posit Connect.

Connecting to MotherDuck from the Shiny server

The main part of the Shiny app (well, at least the demo Shiny, scroll down for the full code) is establishing a connection to MotherDuck. And since the R version of duckdb doesn’t automatically load the motherduck extension, we have to do that step by step:

con <- DBI::dbConnect(duckdb::duckdb(), ":memory:")
  
# Install and load the MotherDuck extenstion
DBI::dbExecute(con, "INSTALL 'motherduck';")
DBI::dbExecute(con, "LOAD 'motherduck';")
  
# Define the query to authenticate
auth_query <- glue::glue_sql("SET motherduck_token= {`Sys.getenv('MD_TOKEN')`};", .con = con)
  
DBI::dbExecute(con, auth_query)
  
# Connect to MotherDuck
DBI::dbExecute(con, "PRAGMA MD_CONNECT")

Here we create an in-memory duckdb and use that to install and load the extension. Then we authenticate with the token that is stored in an environment variable.

Note, the PRAGMA statement here is duckdb’s way of “changing the behavior of the system” which is what we are doing with loading the extension.

If you run the above code in a normal R script you will still connect to MotherDuck, which is of course expected.

Then is just about adding the other pats of the shiny app together:

And the whole code below:

library(shiny)
library(duckdb)

ui <- fluidPage(
  titlePanel("DuckDB and Shiny Integration"),
  sidebarLayout(sidebarPanel(
    helpText(
      "This app connects to MotherDuck and queries the sample WHO dataset."
    ),
    uiOutput("cities")
  ), mainPanel(tableOutput("data_table")))
)


server <- function(input, output, session) {
  # Connect to an in-memory DuckDB database
  con <- DBI::dbConnect(duckdb::duckdb(), ":memory:")
  
  # Install and load the MotherDuck extenstion
  DBI::dbExecute(con, "INSTALL 'motherduck';")
  DBI::dbExecute(con, "LOAD 'motherduck';")
  
  # Define the query to authenticate
  auth_query <- glue::glue_sql("SET motherduck_token= {`Sys.getenv('MD_TOKEN')`};", .con = con)
  
  DBI::dbExecute(con, auth_query)
  
  # Connect to MotherDuck
  DBI::dbExecute(con, "PRAGMA MD_CONNECT")
  
  cities <- DBI::dbGetQuery(con,
                            "SELECT DISTINCT(city) FROM sample_data.who.ambient_air_quality LIMIT 25;")
  
  output$cities <- renderUI({
    tagList(selectInput(
      inputId = "city",
      label = "City",
      choices = cities
    ))
  })
  
  query_rct <- reactive({
    req(input$city)
    city_name <- input$city
    glue::glue_sql(
      "SELECT country_name, city, \"year\", pm10_concentration, pm25_concentration, no2_concentration, FROM sample_data.who.ambient_air_quality WHERE  city = '{`city_name`}';",
      .con = con
    )
  })
  
  
  data_rct <- reactive({
    req(query_rct())
    message(query_rct())
    DBI::dbGetQuery(con, query_rct())
  })
  
  
  # Render the table output
  output$data_table <- renderTable({
    data_rct()
  })
  
  onSessionEnded({function() {DBI::dbDisconnect(con)} })  
}

shinyApp(ui = ui, server = server)

Closing notes

Since this is just a demo app I am limiting the cities output to 25 to avoid Shiny complaining about the long vector of city names (8000+). You could obviously modify that to get cities in Europe or something else.

The SQL statements can be rewritten in duckplyr, but for me it was convenient to test a query in MotherDuck and just paste it in the R code like this.

Of course, it would be better to have dashboard-ready tables (aggregations, summaries) that can be used in the Shiny app directly, and something like that can be achieved with dbt or SQLMesh, but maybe I will do that in another post.

Finally, I don’t recommend deploying this app on shinyapps.io. I tried and it takes too much time to compile and install the duckdb package, that I got a timeout. But it works nicely when running it from RStudio.

Happy quacking!

Update 2024-06-23, 16:44 CEST

Many thanks to Chuck Powell for sending a message that all of this can be achieved with one simple command from the package pak:

pak::pkg_list()

Awesome. :)

My setup

I am running R version 4.4.1 on Linux, with duckdb version 1.0.0.

I have python version 3.12, and I am installing duckb 1.0.0 in a virtual environment.

I installed the duckdb version 1.0.0 binary as well, and installed the motherduck extension.

And, of course, I also have created an account on MotherDuck.

Running duckdb in R

This has been probably covered many times so far. Nevertheless, for completeness, running duckdb in R is fairly straight forward:

# Connect to an in-memory DuckDB database
con <- dbConnect(duckdb::duckdb(), ":memory:")

# Write the Iris dataset to the DuckDB in-memory database
dbWriteTable(con, "iris", iris)

# Query data from the DuckDB database
data <- dbGetQuery(con, "SELECT * FROM iris")

Of course, there is the possibility of doing things with duckplyr, but we are not going to go into that.

Connecting to MotherDuck

MotherDuck documentation has details about connecting to MotherDuck using python, but not for connecting using R.

After asking a few questions on the discord, I learned that the process should be similar.

Let’s see how that looks.

import duckdb

# connect to MotherDuck using 'md:' or 'motherduck:'
con = duckdb.connect('md:')

Attempting to automatically open the SSO authorization page in your default browser.
1. Please open this link to login into your account: https://auth.motherduck.com/activate
2. Enter the following code: XXXX-XXXX

con <- DBI::dbConnect(duckdb::duckdb(), "md:")

ls -lh md\: 
-rw-r--r-- 1 novica novica 12K jun 21 10:32 md:

# Create a local database
con <- DBI::dbConnect(duckdb::duckdb(), "local.duckdb")


# Note: Installing the extension is not nececsary here since 
# I already have it installed on my system. However, if you don't want
# to go to the trouble of installing the duckdb binary and then installing
# extensions, then it is possible to do the installation here before
# loading with:
# DBI::dbExecute(con, "INSTALL 'motherduck';")

# Load the Mother Duck extension
DBI::dbExecute(con, "LOAD 'motherduck';")


# Verify that the extension is loaded
DBI::dbGetQuery(
  con,
  "SELECT extension_name, loaded, installed FROM duckdb_extensions() WHERE
  extension_name = 'motherduck'"
)

# Connect to MotherDuck
DBI::dbExecute(con, "PRAGMA MD_CONNECT")

# Query the sample data about air quality that is avaiable on MotherDuck
DBI::dbGetQuery(
  con,
  "SELECT country_name, city, \"year\", pm10_concentration, pm25_concentration, no2_concentration, FROM sample_data.who.ambient_air_quality WHERE  city = 'Skopje' OR city = 'Oslo';"
)