How to Use Yjs in Rails With Importmap

Feb 21, 2024

Yjs is a CRDT (Conflict-free Replicated Data Types) implementation in JavaScript.

What it means is when you want many users work on the same document, permanently adding, updating, deleting parts of it, CRDT is what you need to synchronize document content between all of the users, automatically resolving conflicts between their editings.

Yjs provides a Ruby port and even an ActionCable-based adapter so users of your Rails application can enjoy collaborative editing using an ActionCable websocket connection.

Unfortunately at the moment of writing this Yjs is not so well suited “out of the box” to be used with importmap, which become a standard since Rails 7. Below is my (kinda hacky) recipe how to make it work.

First, fortunately there’s already a ESM-based implementation of the lib0 library which is the primary dependency of Yjs.

So would it be just enought to importmap pin yjs? Nope :( Yes, it does some job and adds many entries into config/importmap.rb but unfortunately the results are unusable.

$ cat vendor/javascript/lib0--array.js 

import"./_/9E9p6XoE.js";export{d as appendTo,b as copy,c as create,e as equalFlat,g as every,h as flatten,j as fold,f as from,i as isArray,l as last,m as map,s as some,u as unfold,k as unique,n as uniqueBy}from"./_/xs_xZIei.js";

These cryptic imports are not usable for us.

Also one will soon find that many dependencies are not included.

importmap uses jspm by default, so I tried jsdelivr and got better luck with it.

importmap pin yjs -f jsdelivr

$ cat vendor/javascript/lib0--array.js 

/**
 * Utility module to work with Arrays.
 *
 * @module array
 */

import * as set from './set.js'

...

Imports are referring to existing files. Trying to make it work fails with errors though:

As imports are relative, they refer to /vendor/javascript/<module>.js instead of lib0/<module>.js which makes it impossible for importmap to include them properly.
Still many missing dependencies are reported in the browser dev tools.

My next idea was to grab lib0 directly from the repo and copy it into vendor/javascript.

$ git clone https://github.com/dmonad/lib0

Cloning into 'lib0'...
remote: Enumerating objects: 2461, done.
remote: Counting objects: 100% (1129/1129), done.
remote: Compressing objects: 100% (221/221), done.
remote: Total 2461 (delta 938), reused 1054 (delta 908), pack-reused 1332
Receiving objects: 100% (2461/2461), 862.11 KiB | 4.87 MiB/s, done.
Resolving deltas: 100% (1634/1634), done.

$ ls lib0

LICENSE diff.test.js logging.js performance.node.js symbol.js
...

We don’t need to copy all files, there are tests and configs which we skip, so we copy only modules.

But there’s same problem with relative imports.

$ cat lib0/array.js

/**
 * Utility module to work with Arrays.
 *
 * @module array
 */

import * as set from './set.js'

...

So we can’t just copy them as they are, we have to make these imports refer to lib0 as it’s root.

To making the whole process automated in case a new version is released, I wrote a simple bash script.

I put it into my Rails app as bin/importmap-pin-lib0.

#!/usr/bin/env /bin/bash

LIB0_DIR=$([ -n "$1" ] && realpath "$1" || echo "$HOME/lib/lib0")
RAILS_ROOT=$(realpath "$(dirname "$0")/..")
IMPORTMAP_RB="$RAILS_ROOT/config/importmap.rb"
VENDOR_JAVASCRIPT="$RAILS_ROOT/vendor/javascript"

all_paths_exist=1
echo "lib0 Git repository path: $LIB0_DIR"
[ -d "$LIB0_DIR" ] || { echo "  !does not exist (did you git clone it from https://github.com/dmonad/lib0 ?)" && all_paths_exist=0; }
echo "Rails root: $RAILS_ROOT"
[ -d "$RAILS_ROOT" ] || { echo "  !does not exist" && all_paths_exist=0; }
echo "Importmap file: $IMPORTMAP_RB"
[ -f "$IMPORTMAP_RB" ] || { echo "  !does not exist" && all_paths_exist=0; }
echo "Vendor JavaScript directory: $VENDOR_JAVASCRIPT"
[ -d "$VENDOR_JAVASCRIPT" ] || { echo "  !does not exist" && all_paths_exist=0; }
[ $all_paths_exist -eq 0 ] && { echo "Error: one or more paths do not exist" && exit 1; }

read -p "All paths exist. Continue? [Y/n] " -n 1 -r
[[ ! $REPLY =~ ^[Nn]$ ]] || { echo "Aborted" && exit 1; }

echo "Entering lib0 directory"
cd "$LIB0_DIR" || { echo "Error: could not enter lib0 directory" && exit 1; }
version=$(git describe --tags --abbrev=0) || { echo "Error: could not get tags" && exit 1; }
# remove the leading 'v' from the version
version=$(echo "$version" | cut -c 2-)
echo "Detected lib0 version: $version"

module_files=$(ls -1 *.js | grep -v '.test.js' | grep -v '.node.js' | grep -v '.deno.js' | grep -v 'rollup.config.js')
for module_file in $module_files; do
  module=$(basename "$module_file" .js)
  echo "Found module: $module"

  copy_to="$VENDOR_JAVASCRIPT/lib0--$module_file"
  echo "  Copy to $copy_to"
  cp "$module_file" "$copy_to" || { echo "Error: could not copy module" && exit 1; }

  exists_in_importmap=$(grep -c "lib0--$module" "$IMPORTMAP_RB" | grep -v "^\s*#")
  if [ "$exists_in_importmap" -eq 0 ]; then
    echo "  Add to importmap"
    echo "pin 'lib0/$module', to: 'lib0--$module_file' # @$version" >> "$IMPORTMAP_RB"
    continue
  fi

  same_version=$(grep -c "lib0/$module.*@$version" "$IMPORTMAP_RB" | grep -v "^\s*#")
  if [ "$same_version" -eq 0 ]; then
    echo "  Update version in importmap"
    sed -i "s/lib0\/$module.*$/lib0\/$module', to: 'lib0--$module_file' # @$version/" "$IMPORTMAP_RB" \
      || { echo "Error: could not update version in importmap" && exit 1; }
    continue
  fi

  echo "  Already up to date in importmap"
done

# original files contain relative imports, like `import * as set from './set.js'`
# https://github.com/dmonad/lib0/blob/v0.2.88/array.js#L7
# which becomes trouble for importmap, so we need to replace them with absolute imports
# like `import * as set from 'lib0/set'` (note the lack of file extension)
sed -i -E "s/import \* as (.+) from '\.\/(.+)\.js'/import * as \1 from 'lib0\/\2'/g" "$VENDOR_JAVASCRIPT"/lib0--*.js
# and like that's not enough :) they also contain relative exports, like
# export { ... } from './logging.common.js'
sed -i -E "s/export \{(.+)\} from '\.\/(.+)\.js'/export \{ \1 \} from 'lib0\/\2'/g" "$VENDOR_JAVASCRIPT"/lib0--*.js

So in the end:

Delete from your config/importmap.rb and vendor/javascript all remnants of previous attempts of pinning anything related to lib0 / Yjs / yrb-actioncable.
- Yes, Yjs / yrb-actioncable must be unpinned / removed too if you pinned them earlier, because of most likely they brought or refer to an incorrect / incomplete set of dependencies.
Run the script above from your Rails app dir. Now you should have correct and complete set of lib0 modules in your config/importmap.rb and vendor/javascript.
importmap pin yjs -f jsdelivr this will now find already added dependencies and won’t fetch them from CDN, only download Yjs itself.
And only now do importmap pin @y-rb/actioncable -f jsdelivr which again will find already installed dependencies and download a few files it needs, which is @y-rb/actioncable itself plus some modules implementing the Yjs protocols.

There’s the corresponding part of my config/importmap.rb after applying the steps above.

# https://docs.yjs.dev/
# "Yjs is a high-performance CRDT for building collaborative applications that sync automatically."
# ME:
#   this thing should help us with collaborative pattern editing
#   installation is not without pitfalls
#   lib0 downloaded from jspm contain cryptic imports like `import "./_/9E9p6XoE.js";`
#   lib0 downloaded from jsdelivr is missing a lot of dependencies
#   see bin/importmap-pin-lib0
pin 'lib0/array', to: 'lib0--array.js' # @0.2.88
pin 'lib0/binary', to: 'lib0--binary.js' # @0.2.88
pin 'lib0/broadcastchannel', to: 'lib0--broadcastchannel.js' # @0.2.88
pin 'lib0/buffer', to: 'lib0--buffer.js' # @0.2.88
pin 'lib0/cache', to: 'lib0--cache.js' # @0.2.88
pin 'lib0/component', to: 'lib0--component.js' # @0.2.88
pin 'lib0/conditions', to: 'lib0--conditions.js' # @0.2.88
pin 'lib0/decoding', to: 'lib0--decoding.js' # @0.2.88
pin 'lib0/diff', to: 'lib0--diff.js' # @0.2.88
pin 'lib0/dom', to: 'lib0--dom.js' # @0.2.88
pin 'lib0/encoding', to: 'lib0--encoding.js' # @0.2.88
pin 'lib0/environment', to: 'lib0--environment.js' # @0.2.88
pin 'lib0/error', to: 'lib0--error.js' # @0.2.88
pin 'lib0/eventloop', to: 'lib0--eventloop.js' # @0.2.88
pin 'lib0/function', to: 'lib0--function.js' # @0.2.88
pin 'lib0/index', to: 'lib0--index.js' # @0.2.88
pin 'lib0/indexeddb', to: 'lib0--indexeddb.js' # @0.2.88
pin 'lib0/isomorphic', to: 'lib0--isomorphic.js' # @0.2.88
pin 'lib0/iterator', to: 'lib0--iterator.js' # @0.2.88
pin 'lib0/json', to: 'lib0--json.js' # @0.2.88
pin 'lib0/list', to: 'lib0--list.js' # @0.2.88
pin 'lib0/logging', to: 'lib0--logging.js' # @0.2.88
pin 'lib0/logging.common', to: 'lib0--logging.common.js' # @0.2.88
pin 'lib0/map', to: 'lib0--map.js' # @0.2.88
pin 'lib0/math', to: 'lib0--math.js' # @0.2.88
pin 'lib0/metric', to: 'lib0--metric.js' # @0.2.88
pin 'lib0/mutex', to: 'lib0--mutex.js' # @0.2.88
pin 'lib0/number', to: 'lib0--number.js' # @0.2.88
pin 'lib0/object', to: 'lib0--object.js' # @0.2.88
pin 'lib0/observable', to: 'lib0--observable.js' # @0.2.88
pin 'lib0/pair', to: 'lib0--pair.js' # @0.2.88
pin 'lib0/performance', to: 'lib0--performance.js' # @0.2.88
pin 'lib0/prng', to: 'lib0--prng.js' # @0.2.88
pin 'lib0/promise', to: 'lib0--promise.js' # @0.2.88
pin 'lib0/queue', to: 'lib0--queue.js' # @0.2.88
pin 'lib0/random', to: 'lib0--random.js' # @0.2.88
pin 'lib0/set', to: 'lib0--set.js' # @0.2.88
pin 'lib0/sort', to: 'lib0--sort.js' # @0.2.88
pin 'lib0/statistics', to: 'lib0--statistics.js' # @0.2.88
pin 'lib0/storage', to: 'lib0--storage.js' # @0.2.88
pin 'lib0/string', to: 'lib0--string.js' # @0.2.88
pin 'lib0/symbol', to: 'lib0--symbol.js' # @0.2.88
pin 'lib0/test', to: 'lib0--test.js' # @0.2.88
pin 'lib0/testing', to: 'lib0--testing.js' # @0.2.88
pin 'lib0/time', to: 'lib0--time.js' # @0.2.88
pin 'lib0/tree', to: 'lib0--tree.js' # @0.2.88
pin 'lib0/url', to: 'lib0--url.js' # @0.2.88
pin 'lib0/webcrypto', to: 'lib0--webcrypto.js' # @0.2.88
pin 'lib0/websocket', to: 'lib0--websocket.js' # @0.2.88
# ME: remember to pin it from jsdelivr, not jspm
pin "yjs" # @13.6.12

# https://github.com/y-crdt/yrb-actioncable
# "An ActionCable companion for Y.js clients"
# ME: remember to pin it from jsdelivr, not jspm
pin "@y-rb/actioncable", to: "@y-rb--actioncable.js" # @0.2.1
# ME: will I have one day to import them same way as lib0? https://cdn.jsdelivr.net/npm/y-protocols/
pin "y-protocols/auth", to: "y-protocols--auth.js" # @1.0.6
pin "y-protocols/awareness", to: "y-protocols--awareness.js" # @1.0.6
pin "y-protocols/sync", to: "y-protocols--sync.js" # @1.0.6

Phaengris.Art hand-drawn arts, city photos, linux / dev stories

How to Use Yjs in Rails With Importmap