I took the following picture of a paper wasp situation at Oakmont sub in Nampa, Idaho: 
Notice these paper wasps congregating, and not around a nest? In my experience, It usually means one of two things.

1. The nest has grown too large for a small space–like inside vinyl siding. Or:

2. Their nest has been destroyed and the remaining wasps are considering their next move.

Either way, it is always important to try and detect paper wasp problems before they become too large to manage easily.
If a paper wasp nest is inside vinyl siding, short of taking down your siding to kill the nest, you may have to either wait it out, or plug up the entrance/exit holes to starve them out. A licensed pest control professional should be able to do so with a protected suit while he does this.
If you’ve recently killed a wasp nest, and the wasps are congregating like in the photo above, make sure to observe them carefully over the next week or so. If they begin another nest, make sure to spray it and remove it as quickly as possible.