Go to content Go to menu

Samvera Connect 2018 Recap

Mar 15, 10:04 AM

Samvera Connect 2018 was hosted by the University of Utah’s J. Willard Marriott Library. For me, the theme of this conference was testing. I registered for two different workshops on testing. The first was run by Carolyn Cole, and based on the excellent Rails Bridge training. [Carolyn’s workshop materials] The second was run by Tom Johnson, and was a slightly more Samvera-specific workshop on testing topics. [Tom’s workshop materials] I have several pages of notes from both workshops. But I’ll try to mention a few bullet point takaways from them:

From Carolyn’s workshop:

  • try to write tests that make sense to you
  • tests are really only useful if you understand how they work
  • generated code doesn’t always have tests, or what tests it does have are incomplete
  • to quote Carolyn: “Yes there are smart people writing this stuff, but it’s important to understand what you’re using.”
  • use Coveralls to find places where you are testing too much (ooh, interesting, but good tip)
  • name your tests in such a way that, when you see them fail, the failure message makes sense… you’ll see your tests fail right away, if you’re doing TDD correctly, and it helps a lot if you can understand what the failure message is telling you
  • a thing to read about later: Shameless Green from Sandi Metz.
  • Feature specs are expensive, but that’s because they are complete… since big tests require a long time investment, ensure there is a big payoff for running them, especially if you’re running them all the time as part of some automated process

From Tom’s workshop:

  • Are your tests using too-specific dependencies?
  • Do your tests “know too much”?
  • Don’t just “rebuild the implementation” in the test
  • Complicated tests should worry you, they are a warning sign
  • this snippet is golden: it (require 'pry' binding pry) see this article for more details

Tom also had lots to say about mock objects, and other such things. For sure check out both Carolyn and Tom’s workshop materials, and if you’re doing any sort of development work with Samvera, you should actually go through both of those workshops and follow along with the exercises. It’s good experience, you’ll learn a lot.

OK, workshops done, on to the main conference. Here are some highlights from my notes: The Code Stability Working Group’s Recommendations is something I should read. Glancing at it again, I see there are a few interesting links from that page… I need to set aside some time to explore all of that info more thoroughly. Along those same lines, the Hyrax Roadmap is something anyone who works with Hyrax should be aware of, and read. Also, Hyrax doesn’t just travel down that road all on its own, there is likely lots of work to be done, and if you’re in a position to help, there will be Hyrax pull requests waiting for review. Have some spare time? Make yourself useful and pitch in!

Another theme to this conference was the community coming to terms with what bringing in Valkyrie will actually mean, and how that work will proceed. And while that might have been interesting many months ago, proceed they have! :-) Valkyrie is now up to version 1.5.1 with version 2.0 expected in coming weeks. The project has been promoted to the core Samvera repository, and has been added as a dependency of Hyrax as of February 2.

During the Hyrax Working Group, Valkyrie discussion dominated. At some point, someone mentioned Martin Fowler’s Branch by Abstraction article … I need to read that.

Now I’ll just list off the talks I got the most out of, in the order they appear in my notes (likely chronological order).

David Schober from Northwestern presented a talk on DevOps [slides]
…really useful from the trenches information on running containers in AWS, and using Terraform to do it. This is an area my team at UCLA Library is trying to develop expertise in, these slides will be useful for our team. I particularly like their DevStack tool for setting up dev environments for the various services they build and maintain.

Kate Lynch from University of Pennsylvania Libraries presented a talk about a workflow they’ve written to manage backup of content to Amazon Glacier, which they call “Guardian Workflow”. [slides]
… really cool stuff, and worth a look if you want to do something similar. Even if you’re not interested in backing up to Glacier, it’s worth a look to see how they’ve managed to tackle it, as the approach might be useful for other work you have to do.

Justin Coyne from Stanford University Library spoke about deploying with AWS Elastic Container Service [slides] From my notes, they’re using CloudWatch to pull all their log files together, and auto-scaling will save you money, you can turn off the service when you don’t need it.

James Griffin from Princeton University Library presented on Synchronizing Samvera [slides] Lots of helpful information in this talk about how to get Samvera to talk to other web services. This is information that will help anyone who is building microservices alongside a Samvera repository, or anyone who needs to integrate a Samvera repository in a larger IT ecosystem.

There was a session about IIIF, and my notes from this session are pretty sparse, but I did jot down that Simeon Warner from Cornell mentioned they use something called “Art Store” to handle moving files, and I think it’s actually Archival Storage Ingest … which actually looks pretty cool, and useful for my team.

There was a Batch Ingest Working Group meeting which I attended, but the facilitator couldn’t make it. My colleague Lisa McAulay jumped in to lead the conversation, and we ended up walking through the working group’s docs on the wiki, and added new information from those present at this meeting, so hopefully we ended up helping. Those notes have moved on the wiki since this meeting, I think this is their current location

I, along with my colleague from UCSD, Jon Robinson, facilitated an unconference session we called the “DevOps Sandbox Swap Meet”, here are the [notes from the session]

My colleague from UCLA Library, Stephen Gurnick, and I presented a poster on things we’ve learned about making DevOps work at UCLA Library. [our poster]

That’s about it. Sorry these notes are so late.


Over the past few months I have been looking into getting Hyrax (aka a Samvera reference implementation) set up on my work notebook. I think I’m close, and I wanted to share my working notes here, in case you want to follow along…

You must first love Ruby

OK, maybe “love” is too strong of a word, but you’ll at least need to install it, if you haven’t already.

Install Ruby

You’ll probably want to use a dedicated tool to manage Ruby versions, because that’s part of the fun of Ruby—you’ll need to use a different version of Ruby some day. And that day is going to be hard enough without trying to un-do or work around some other way of installing Ruby. Trust me.

My favorite way to install Ruby is with rbenv however uru is supposedly easier and cross-platform. And there are many other methods.

Gem Install Rails and Railties

You’re not going to be running all of Rails on your workstation (though you could, but there are a bunch of dependencies for Hyrax, and you’re going to use Docker to manage that mess). However, you’ll need the Rails gem installed so you can use CLI-based Rails generator to build a new Hyrax application. And you’ll need Railties version 5.0.6 so you can run the preferred Rails generator (Railties is the gem that manages the generator stuff). So, as soon as you have Ruby installed, run these commands:

gem install rails
gem install railties:5.0.6

An early milestone of success: let’s generate a Hyrax application!

We have all the pieces we need, so let’s get this out of the way now. Run this:

cd path/to/your/project/or/workspace/folder
rails _5.0.6_ new awesomenameforyournewapp -m 
https://raw.githubusercontent.com/samvera/hyrax/v2.1.0.beta2/template.rb

Blam, that felt good.

Gem Install Stack Car

We’re going to use Notch8’s Stack Car gem to help us manage Docker and Docker-Compose competently.

gem install stack_car

OK, don’t get too excited, but you’re almost ready to seriously hack on Hyrax. But first, you do have Docker and Docker-Compose installed, right?

Install Docker and Docker Compose

Sorry, that will probably be an epic journey of discovery. Docker seems to work better on Linux than any other OS… I’ve heard good things about OSX. But, this is a terse guide, and those links will get you started. Come back when you have Docker and Docker Compose installed. Good luck!

Right, back to Hyrax and Stack Car

cd path/to/your/project/or/workspace/folder/awesomenameforyournewapp
sc dockerize .

Oooh, shiny. Now, as awesome as this is, you’ll need to make some adjustments.

Add the following to the default .env file:

REGISTRY_HOST=hub.docker.com
REGISTRY_URI=/phusion/passenger-ruby23
TAG=0.9.28

Remove the last 4 lines from the docker-compose.yml file

   depends_on:

volumes:

^^^ that may or may not be necessary, but on the version of Docker and Docker Compose I was using, it was.

Change the ports line in the docker-compose.yml file to read

    ports:
      - "3000:80"

Let’s start this baby up

Now we can crank up our Dockerized Hyrax app:

sc up

Have fun

You should be in a good place to further explore what Hyrax is and what you can make it do. Need a place to start? Start here:

This is a work in progress!

So, I don’t yet have access to any documentation on how Stack Car works or what you should do next. I wish I did. If you’re following along, you can muddle through with me, or you can wait patiently for me to update this blog post with links to more documentation.

UPDATE: so, you’ll have a instance of Hyrax running on http://127.0.0.1:3000/ but you’ll need to do a couple of things before it’s actually usable.

1. You’ll need to run db:migrate:

sc be rails db:migrate

2. You'll need to chown and chmod your tmp folder
sc exec bash
chown root tmp
chmod 777 tmp
# note this is really naughty, but you know, it's a dev environment, so get over it

Now check http://127.0.0.1:3000/ and you should see your new Hyrax site waiting for you to hack on. Get to it, buddy. Oh, for the db:migrate command, you might need to send a slightly different one. Rails will tell you what to run. You’re almost there.

UPDATE2: If you’re running Docker on a Linux notebook (I am) all sorts of things will be slightly off while you’re working with Stack Car. I’m not sure of the cause, I’m researching, but it’s something to do with the way the named mounts are loaded with the Docker Compose file created by Stack Car. The main problem is that Docker wants to run as root, which means files created by root in the containers are owned by root (hence the janky chown and “rootme” permissions up above). There are workarounds for most of the issues, but the one that I can’t seem to fix is that whenever a rails command is run on the container, the files that rails command creates will be owned by the root user. That ownership translates over to the host. Which means I won’t be able to edit those files. Which is a real downer as far as developer experience is concerned (the entire point is to be able to work with these files… not being able to work with them makes me very table-flippy). I suspect a bit more cleverness with file permissions might be enough to hobble along… but… I also suspect there may be a simple thing I can change in the Docker Compose file to have these mounts work correctly without any monkey business. I suppose the real question is: is it worth my time to invest any further effort to get this working environment to work for me or should I move back to Vagrant—a tool I trust to deliver a usable (albeit rather slow) working environment.

UPDATE3: I think this is the source of the magic on OSX… apparently things just work over there? More research required.

UPDATE4: Docker uses something called a storage driver to handle how containers talk to storage on the host computer. it looks like my default storage driver was set to aufs, which isn’t quite the same as overlay2, which is the recommended storage driver. So, I’ve followed this suggestion and now hoave overlay2 set as my Docker storage driver. I then did a bunch more research because that config change was not enough to handle the permissions issues I am encountering. And I found this on StackOverflow, so I added a :z at the end of my volume lines in docker-compose.yml, like so:

solr:
      image: solr:latest
      env_file:
       - .env
       - .env.development
      ports:
       - "8983:8983"
      volumes:
       - './solr:/opt/solr/server/solr/mycores:z'
      entrypoint:
       - docker-entrypoint.sh
       - solr-precreate
       - samvera
       - /opt/solr/server/solr/mycores/config

the :z flag tells the storage driver to pass through the permissions from the host folder and files. With this flag in place, you can run

chmod -R 777 solr

in your host working directory, and you’ll have proper permissions set up, so you can edit those solr configs on your host and then have Docker load the Solr container correctly. The same strategy applies to all the other mounts you might want to work with (like the one for the web container, where all the app files and configs go). After you run a rails task to generate new MVC files, you’ll probably need to run the following in a terminal on your host:

find -type f -user root | sudo xargs chmod 777

You can then edit the files the rails tasks leave for you, via an editor or IDE (Atom or RubyMine), on your host. Is this annoying that you have to tinker with permissions all the time? Yes… but it does work, and you’ll only have to do it once for each file you create. Probably you could get creative with a sticky bit and setting the GID on the container… that will be fun for another day.

Speaking of fun, it turns out you may need to run another task before you can create the default admin_set. The docs may be out of date, so I made a ticket for that issue

TLDR:, here’s what you need to type in order to get your Hyrax app running in Stack Car:

sc be bundle install
sc be rails hyrax:default_collection_types:create #may not be necessary, but there's no harm in re-running it
sc be rails hyrax:default_admin_set:create

And you need to add an admin role to the role_map.yml file.

Basically you need to follow along with the getting started guide ENJOY!

UPDATE5: Here are a few other helpful links for getting started: