I'd say the effect you are seeing is due to the API being stateless - all it sees is the parameters in your latest call, there's no 'memory' of the previous call - i.e. at step 3 the API does not 'know' it is what you call page 1, it is simply a page.
So if you want to follow 'prev' links for this call, I would check for empty response and stop there.
Pagination details do vary a bit depending on the API call involved, I've learned that different calls may need different handling to detect when to stop in order to prevent long/endless loops.