How to prevent UNABLE_TO_LOCK_ROW via the REST API?

You can really only effectively use FOR UPDATE for record locking within Apex code (which you could expose via a REST resource and call from your external system). It will lock the records returned by the query for the remainder of the transaction, and the transaction is free to make updates to the records in question while it holds the lock. Other simultaneous transactions that attempt to lock the record will have to wait.

You'll be unable to use FOR UPDATE when simply querying a record via the standard REST API because the transaction is over as soon as the data is returned to you. If you're then making an update to the record via the standard REST API after the initial query, this is a separate transaction and Salesforce does not allow any mechanism to lock records between transactions like this. In order to avoid the misconception that a record is locked after your query transaction takes place, Salesforce returns the error messages you've been getting.