How do I setup and use the NVENC encoder settings?

As of XSplit Broadcaster 3.7, the NVENC H.264 codec has been updated to allow users to enable advanced features and settings based on Nvidia's recommendations.

On your broadcast output properties, select NVENC H.264 under Video Encoding properties then click the gear icon on the right.
The available options for you to use will be different depending on the type of broadcast output your are using.

For most broadcast outputs such as Facebook Live, YouTube Live and Twitch - the available settings will be like this:

For Custom RTMP and Local Streaming outputs, these options will be available:

Enable Look-ahead
Can improve your stream's visual quality by buffering a specified number of frames in memory to optimize the encoding of the current frame. Note that enabling this will add a bit of latency to your final output and it can also be detrimental to high-motion games such as first-person shooters.
Use 2-Pass Encoding
Improves stream's visual quality by estimating the complexity of the frame to be encoded and determines bit distribution first, then encodes the frame after in a second pass using the distribution determined in the first pass. Disabled, estimation and encoding happens in one pass. Note that enabling this would add a bit of latency to the final output of the stream.
Use Adaptive Quantization
Improves subjective visual quality by automatically adjusting the how the encoder compresses different portions of a frame based on the type on content in them or beside them. This should have no impact on performance but if users experience any issues, they may disable the option.