It seems admirable and sneaky of Jess and Leslie to write a fake love letter to embarrass Janice when she's mean to May Belle, since they can't beat her up in a physical fight. It's a clever way of dealing with a problem that has to be dealt with.
But they feel badly for Janice afterwards. We learn later in the movie that Janice's bullying comes from a darker place, and is really an example of someone perpetuating a cycle of abuse. In other words, because her father beats her up so badly, the only thing she knows how to do is be mean to other, weaker people. Leslie explains to Jess that Janice's father "really beats her. The kind of beatings they take people to jail for in Arlington. What gets Janice is not the abuse as if that's not bad enough but the fact that her friend blabbed about it and everybody at school knows. That embarrasses her at a totally different level than the silly letter Jess and Leslie wrote.
After Leslie's death, Janice watches in sympathy when Gary Fulcher pushes Jess on the floor and is last seen when she sits next to Jess on the bus and says "Hi" in a sweet voice, much to Jess's surprise.