The Gatekeepers (2012)

This is a documentary from Israel which interviews six former heads of Shin Bet.  Here is a fair warning.  It is pretty terrifying, even if you have very neutral feelings about the Israeli-Palestinian situation, or the Middle East in general.  These guys talk in a very straight forward manner about just exactly what their job entailed and how they accomplished things.
Its first lesson is that in tracking and targeting terrorists, politicians want black-and-white answers, while these intelligence officers usually inhabit in the grey of doubt. If they strike, they might kill civilians. If they do not strike, they might leave Israel vulnerable to an assault. But, if they give their heads of state the incorrect answers, they risk widespread humiliation. Thus, public perception might be valued more than Palestinian lives. And the film offers a second lesson, which is that this godlike power to kill is unnatural. It corrupts all involved. 
However, the most chilling aspect, however, is the blunt commentary about the work itself. These men accept that national security has no morality.