Bring a large pot of salted water to a boil. Cut potatoes into even 1 1/2-inch chunks. Boil 10 minutes, or until potatoes are tender but have a little bite. Drain potatoes, set in a large bowl, and set aside.
In a small bowl, combine the onion, celery, parsley, zest and juice, mayo, and mustard. Mix to incorporate. Fold mixture in with the warm potatoes. Add the capers. Check salt, mayo, and acid levels, and adjust as needed.
I love my potato salad just after it has been made, when the flavors still stand alone, however, if left to mingle in the fridge overnight, you may find the flavor more integrated. Potato salad can be very personal, so please adjust according to your own tastes.