Here’s a fun story from today:

For context, I was working on initializing our staging database in Cloud66.

Trying to initialize the tables for the POC kept failing. psql complained a relation defined in our dump already existed in the db. I list the schema, 17 tables exist (nowhere near enough), all in alphabetical order (as defined in the dump file).

A google search lead me down a goose chase believing the issue lied with the pg search path. That didn’t seem likely so I decided to just drop the schema from psql, list the schema again to see that it’s empty, and then run the rake task from SSH to initialize the tables.

Failed again. Same error. ???

This time I want to make sure nothing is connected to the db. I drop the schema again, ensure it’s empty, and then kill all connections and disconnect from psql. Run the rake task again from SSH.

Failed. Error is the same.

This time I don’t trust psql. I drop the schema, I ensure it’s empty, I kill the connections, but I wait…. I wait a minute and I check the schema again. Lo and behold, 2 tables are listed. One of them is the same one listed in the error when the rake task explodes.

Rails implicitly maintains two tables, one for schema migrations and one for environment metadata. I knew about the schema migrations table of course, but the other one was new to me (ar_internal_metadata). Once I saw these tables listed, I knew what was going on. Either invoking the rake tasks / rails console was causing Rails to recreate the tables upon initialization (most likely IMO) or puma was being restarted (I wasn’t checking logs) and was causing the tables to get re-created before I ran the rake task to initialize the tables. So I quickly wiped the tables and then ran the rake task to initialize the database and that worked as expected.

PHEW

Note: Ultimately I was bouncing back and forth from SSH to psql and attempting to run db:setup, db:schema:load, db:create + db:schema:load, among other things, so while it’s not exactly a definitive answer, I believe this bouncing around clouded my view of which action was causing these tables to keep cropping up.