This is what they meant by ”dark skinned” in 19th century.

In the book characters speculate that he is indian, gypsy etc which more so reflects the hysteria surrounding the characters physical appearance and background (being adopted).