First it is important to note that the Bible does not say that these "brothers and sisters" of Jesus were children of Mary.
Second, the word for brother (or sister), adelphos (adelpha) in Greek, denotes a brother or sister, or near kinsman. Aramaic and other Semitic languages could not distinguish between a blood brother or sister and a cousin, for example. Hence, John the Baptist, a cousin of Jesus (the son of Elizabeth, cousin of Mary) would be called "a brother (adelphos) of Jesus." In the plural, the word means a community based on identity of origin or life.