About 3,400 inmates were already confined in places like halfway houses.
Another 1,700 of the inmates are not US citizens and will possibly face deportation.
The changes come as part of an effort to reverse long jail terms for non-violent offenders, who make up about half of the federal prison population.
Many of these prisoners were released from federal custody on Friday. The rest will be released by Tuesday.