In a medium bowl, whisk together Cool Whip, 1/2 cup milk, and 1/4 cup Nutella until smooth. In a separate small bowl, mix 2 tablespoons of Nutella and 1 teaspoon of milk until combined. (This is the Nutella swirl.)
Spread a generous teaspoon of Nutella in the bottom of each of 6 (3-ounce) Dixie cups, creating an even layer. Divide the Cool Whip mixture evenly among the cups.
Cover each cup with a small square of aluminum foil. Using a paring knife, make a small slit in the center of each cup and insert a wooden popsicle stick. Freeze for 3 hours or until firm.
When you're ready to enjoy them, snip off a bit of the cup and peel off the popsicle.