HeatherBob. Migrants or refugees travelling into other EU countries do not have any right to travel on to the UK, they are not EU citizens. Refugees are covered by international laws and are not migrants, the words are not interchangeable.
Let us asume a person migrates to Spain from a country in Africa and legally resides there. They will need to establish 10 years residency and pass language and cultural test before being granted Spanish citizenship, which would take another couple of years and thereby have the same rights as all EU citizens on freedom of movement. If, which is unlikely, they then decided to move to the UK after being settled in Spain for 12 to 15 years, then they would be doing so as an EU citizen, one that you say you have no objection to. They would also be doing so within the FOM restrictions.
As I said, refugees are governed by separate international laws and agreements. Those who arrived in mainland Europe three summers ago have no automatic right to travel to the UK.
I am therefore not sure your reason stated for voting leave has any foundation. I am genuinely interested in why you think leaving the EU would reduce immigration from countries outside the EU, when it will in practice probably result (as current stats show) in increases from non EU countries as we make those from our closest neighbours less welcome.