In a small sauce pot - combine the oil, vinegar, garlic, bay leaf, paprika and pepper flakes over medium heat. When it is almost at a simmer, turn off and remove from heat.
Place olives, rosemary sprigs, lemon and orange rinds in a glass jar and pour olive oil mixture over the olives.
Let cool, then store in refrigerator. Let olives marinate at least 1 hour before serving.
Notes
I prefer to wait at least 24 hours before serving, but if you're in a pinch, wait at least an hour.