I think that Red John was at that 4th of July party that Jane's Happy Memory was set at. That was also the time where they both shook hands, when they were both kids, probably about the same age, and randomly met in the park with other families milling around.

Red John was probably some self absorbed child genius who thought that he was the smartest person in the world. But when they first met, Jane might have tricked him somehow and made him look like a fool. Jane's probably forgotten this, dismissed it as just a bit of fun, but Red John remembers and he's been stalking Jane ever since.

So while the 4th of July party is the scene of one of Jane's happier memories, it's also the scene of Red John's unhappiest one.