Below taken from paper linked by @USS ALASKA :
Relations between the two disintegrated. Pressure grew from civilians throughout the north, especially relatives of Union prisoners suffering in below-standard prisons such as Andersonville, Georgia, and Libby in Richmond, Virginia. In response, Stanton replaced Ludlow with Brigadier General Sullivan A. Meredith in the hopes of resuming exchanges. Pressure also came from other groups in the North. The Democrats cried for an end to the war, which meant more men were needed with which to win by superior numbers. Pressure also came from abolitionists wanting to see negroes in the army fighting for their own freedom. President Lincoln, among others, realized the existence of a large as yet untapped resource in the growing black population available to serve in the Union army. Large numbers of free blacks from both the north and the south, and former slaves were prepared to fight for a cause which had by now come to the forefront of the war--freedom. The cessation of exchanges and the need for replacements in the North led the Federal Government, specifically President Lincoln, to employ black soldiers on a large scale.
Though Meredith and Ould met on many occasions to discuss accounting of paroles and exchanges, little was accomplished. One of the major points of contention was the refusal on the part of the Confederates to consider captured black soldiers as prisoners of war. Though not many had been captured, those that had were returned to previous owners or sold as property. The Union demanded black Federal soldiers not be considered any different than white soldiers. Accusations, it seemed, were exchanged more often than prisoners. On November 15, 1863, Confederate Secretary of War James Seddon declared, "All exchanges have now ceased with little apparent prospect of renewal. The stalemate continued.
On November 18, General Benjamin Butler, Federal Commander of the Department of Virginia and North Carolina, and newly appointed Agent of Exchange, wrote to Secretary Stanton with a request to resume negotiations for exchange. His information from various sources convinced him that the Confederates would agree to an exchange. The Union held approximately 26,000 prisoners, while the Confederacy had 13,000. General Butler's recommendation to Secretary Stanton was to propose to Robert Ould an exchange, man for man and officer for officer, until all Union soldiers suffering in Confederate prisons were returned. The excess of some 10,000 prisoners in the hands of the Union would then give them something substantial with which to bargain for the return of any colored soldiers and their officers still within Confederate hands. Butler's proposal was approved by the Secretary of War, and after notifying Mr. Ould, several small exchanges occurred during the next two months.
Despite limited success, Butler and Ould did not reach agreement to any great extent on behalf of their respective governments. On April 1, 1864, shortly after General Grant was appointed as commanding general of all Union forces, he visited General Butler at Fort Monroe. Butler advised him of the difficulties thus far experienced in the exchange negotiations, and of the large number of Confederate prisoners still in Union prisons. On April 17, General Grant ordered all exchanges to cease. Meanwhile, the public in the north, and Union prisoners themselves, were increasing pressure on the government to get all Union prisoners released and sent home. Prisoners in Andersonville, Georgia, submitted a petition to the Union government to "effect our speedy release, either on parole or by exchange."
As they explained:
No one can know the horrors of imprisonment in crowded and filthy quarters but him who has endured it . . . . But hunger, filth, nakedness, squalor, and disease are as nothing compared with the heartsickness that wears prisoners down . . . .
Letters from the public were also addressed to Lincoln. One example, from a concerned father whose son was in Andersonville wrote that his son:
has a family here consisting of a wife and two children in indigent circumstances . . . my said son and 30,000more brave soldiers must perish unless the Government should relieve them by bringing about an exchange.
The purpose for the cessation of exchanges is determinably linked to its effects. While Grant openly stated that he refused to exchange any more prisoners until the South agreed to include captured black soldiers, he more privately insisted that it would end the war more quickly. He keenly recognized the lack of replacements available to the South, and strongly believed that released Confederate prisoners would quickly rearm and reenter the fight. The continued internment of tens of thousands of Confederate prisoners denied the South many badly needed soldiers. The population of deployable white males in the South was significantly smaller that in the North. In addition, by 1864, desertion, casualties, and the inability to properly supply the soldiers had taken its toll on the Confederate armies.
Although the Union maintained an advantage in numbers there was an ever increasing need for replacements in the Union army as well. As the North expanded its stranglehold on the South, manning of ports and harbors, control of railroads and depots, and the greater numbers required to take the offensive demanded a larger force than ever before. As with the Confederacy, desertion and casualties took their toll on Federal strength. Gross abuses of the bounty and substitution laws among draftees aided in keeping the army below needed strength. Many Union prisoners had already died in Southern prisons, and many were still interned. The large number still in prison, should they be exchanged, would have provided many badly needed replacements. The effects of this cessation on the manning of both the Union and Confederate armies, and the significant increase in negro regiments in combat, are the central point of this paper.