From the new Charles Calhoun history of the Grant administration:
Certainly the current occupant of the White House harbored no illusions about his successor’s genius. To Andrew Johnson, Grant would always be “a liar, guilty of duplicity, false to his duty and his trust.” Gideon Welles labored to convince Johnson and his cabinet colleagues to boycott the inauguration “of this ignorant, vulgar man.” Grant may have heard of Welles’s efforts, for he let it be known that he would not ride to the Capitol with Johnson. The inaugural committee suggested separate carriages, but in the end, Johnson refused to go at all. On the morning of March 4 the retiring president busied himself signing bills, and at noon he and his allies departed the White House. Later that day Welles confided to his diary that Grant had ridden up Capitol Hill followed by “a long procession, mostly of negroes,—at least two thirds, I should judge.”16 Upon arrival at the Capitol, the president-elect walked to the Senate chamber for the swearing in of Vice President Schuyler Colfax. During that ceremony, according to one observer, Grant exhibited “his imperturbable expression as unchanged as ever. . . . There he sits, quiet, ‘calm,’ self-possessed, as though he were at his own fireside, instead of being in the presence of the assembled wisdom, wealth, intelligence and power of the country, whose destinies have been confided to him.”
Calhoun, Charles W.. The Presidency of Ulysses S. Grant (American Presidency Series) (p. 66). University Press of Kansas. Kindle Edition.