Here are some examples of how the men under him felt about him from some of my research of the 14h Iowa in the Red River Expedition:
The day of the battle at Mansfield April 8, 1864, wagon trains got in the way of the troops. The veteran regiments in the back couldn't move forward to fight, and the retreating soldiers couldn't retreat because the road was blocked. One soldier wrote about Banks to the Wisconsin State Journal afterward, “The idea of fighting a large battle with wagon trains, I don’t think ever originated with any other man.” The Union lost that battle.
The next day, at Pleasant Hill, Banks began a retreat. He sent the survivors from the cavalry and the 13th Corps, several batteries, and at least part of the Corps D’Afrique back to Grand Ecore with the wagon trains. His reasoning was there was little to no food or water at Pleasant Hill. He left about seven brigades there but told people there was not going to be a battle. The rebels did attack at 5 p.m, but the Union troops fought until the rebels retreated. They planned to go out onto the battlefield that night to take care of the dead and wounded, but instead Banks ordered them all to join the retreat to Grand Ecore. The dead were left on the battlefield, and the 400 wounded all became prisoners (and half of those died). The brigades who had just fought for several hours were ordered to move out silently, leaving their fallen friends, and to defend the rear of the retreating army. They learned later that the rebels had retreated at least 6 miles at the end of the day.
Colonel John Scott of the 32d Iowa wrote that Banks "gave to a brilliant victory all the moral results of a defeat. ...[T]he Thirty-second Iowa Infantry blushes to place upon its banner the name of a field where its dead and wounded were cruelly abandoned to an enemy, who, many hours afterward, humbly asked leave to care for his own!"
An anonymous letter dated May 19, 1864, and published in the Burlington (Iowa) Weekly Hawk-Eye said—
Gen. Banks, as a military genius, is not very highly appreciated by the officers and men of this department, and especially the men of this expedition, who say they believe that if he had had twice as large an army as he had, lie would have gotten them all “gobbled” [captured]. And their eyes fairly flash with indignation when they tell of being compelled by orders to retreat from a retreating foe, which they say was the case at Pleasant Hill, and which was corroborated by some of our men who were taken prisoner in the fight and afterwards escaped, who said that the rebels retreated eighteen miles before they halted their columns."
At the completion of Banks's retreat from the Red River in May, Colonel William T. Shaw, commander of the Second Brigade of the Third Division of the 16th Army Corps wrote in a letter to his hometown paper, "I am candid when I say that I think I could capture [Banks's] whole army with 500 men, so demoralized are they by his Union sentiments and policy." (Among grievances already described, the men resented being used to confiscate cotton.)
Shaw was dismissed from the army in October for expressing his opinions publicly, and many were sure that Banks forced the dismissal.