I dont know for sure since there are a lot of papers and I have not gone through them.
An early biographer (Harrington) was convinced of it, but IIRC never cited any direct evidence.
Others who have looked at his papers didnt find anything but did find contrarian statements such as a letter to his wife in which he stated "I am astonished to find that people—pretty active and intelligent from other states than my own, think I have something to do in the future...I wish I was more ambitious and thought better of myself."
There were definitely a lot of second-hand rumors and accusations, sort of like the accusations that Rosecrans was an opium eater.