I will give you two references and if you or others who read this have further references let me know and I will add them.
Stanislas Dehaene, The Number Sense: How the Mind Creates Mathematics, Oxford University Press, 1997
George Lakoff and Rafael Núñez, Where Mathematics Comes From: How the Embodied Mind Brings Mathematics into Being, Basic Books, 2000.
The first of these is lighter reading than the second but I found both engrossing.
And here is a third
Barry Mazur, Imagining Numbers (particularly the square root of minus fifteen), Farrar Straus Giroux, 2003.