Here is brief idea of what is going on:
"a deprived childhood may affect the physical development of the brain and render its owner less intellectually capable." That comes from The London Times article It’s just as you might think: being poor can damage your brain. Another article, Study shows stress affects brain growth, says "CHILDREN who suffer deprivation in early life show altered patterns of brain growth by the time they are teenagers." This article also said "the change in brain development might be a consequence of high blood levels of stress" and that "children who lived in poverty were likely to suffer more adverse consequences of stress." Another article, from the Financial Times, Poverty mars formation of infant brains, also discusses the issue.
But here is a quote from the Chronicle of Higher Education which tells a different story:
"While many of the researchers at the session supported the hypothesis that socioeconomic status plays a strong role in affecting brain development in children, Mabel L. Rice, director of the doctoral program in child language at the University of Kansas, described a new study that goes against the hypothesis, at least in the case of early verbal abilities. In tests of 1,766 children in Australia, Ms. Rice and her colleagues found no correlation between a child's verbal abilities at 24 months old and the parents' socioeconomic status or their education levels.
"The conclusion is that we don't want to assume too strongly that children of poverty are unable to acquire early vocabulary," she told The Chronicle."
That is not available online unless you subscribe to the Chronicle.