Why do I get an Unknown Primary Key exception for a join table in Rails 4?

The id: false in your original schema indicates that there is no id field in the table. Rails 4 added a create_join_table helper method which creates tables with no id field and uses this for any migration with JoinTable in the name.

The only way I can imagine that you're getting difference results with Rails 4 than with Rails 3 is if you regenerated your migrations and had a migration with JoinTable in the name. Do you still have your schema from Rails 3 around? It would be interesting to note it has id: false for the join table.

As for the primary_key, the reason you could set the primary key to an array but it subsequently didn't work is because the primary_key= method blindly converts its argument to a string per line 115 of https://github.com/rails/rails/blob/a0dfd84440f28d2862b7eb7ea340ca28d98fb23f/activerecord/lib/active_record/attribute_methods/primary_key.rb#L115

See also https://stackoverflow.com/a/20016034/1008891 and its links.

In model add the following:

self.primary_key = [:order_id, :product_id]

and I think it would be wise to ensure that there's an index on those columns.

You may create one with following migration:

add_index :line_items, [:order_id, :product_id]