Yep, just create a new layer in your scene using an appropriate capture mode (Window Capture is preferable... Monitor Capture is super, super slow and should only be used as a last-ditch measure). You can set the sub-region to cut out only the webcam window.
Five PIP boxes might prove... problematic. Better to just capture the whole Hangout in one. Capturing the same window multiple times can just outright fail, and each Monitor Capture you use is going to slow down your preprocessing rate (again, monitor capture is suuuuuuuuuuper slow compared to any of the other capture modes!) and should be avoided if you can help it.
(edit) Actually just did a quick test (with a Notepad session); if you just make multiple new Window Captures all tied to the same window, you can create a sub-region for each. So yes, if your Google Hangout was in your browser window, you'd be able to Add->Window Capture, set a name, click 'use sub-region', 'select sub-region', and repeat for each user in the hangout. Should work, anyway... didn't try it with a live webcam session.
DO BE AWARE!! If your webcam is in use by a Google Hangout, OBS will NOT be able grab it locally as a 'video capture source' input, and vice-versa, the Hangout won't be able to grab it if OBS is started first.
