Pickett was out of commission recovering from a shoulder wound received at Gaines Mill while Malvern Hill was being fought. Perhaps it was Gettysburg or any of the battles that his division participated in after that. Pickett's Division was lightly engaged at Fredericksburg and at Suffolk with Longstreet when Chancellorsville was being fought. At Gettysburg, the Irish Brigade was engaged on the second day of fighting and Pickett's Division made its charge on the third day so it probably wasn't that brigade he faced when that comment
was made. Pickett's Division did run up against the Pennsylvania Irish Brigade on July 3rd at Gettysburg and if they flew a green flag
it may be that brigade he was speaking of.