The Prime Minister is understood to be concerned about the spread of the Islamic State of Iraq and the Levant (Isil) in the African country since the fall of the dictator, Muammar Gaddafi.
Security chiefs fear the Islamists are using Libya as a base to stir up violence in neighbouring Tunisia, where 30 British holidaymakers were massacred last month.
The new focus on Libya is part of a growing determination by Mr Cameron to confront Isil.
He is expected to trigger a Commons vote on military strikes against jihadist targets in Syria when Parliament returns this autumn.
A senior Ministry of Defence source said yesterday British strikes in Libya were not being discussed.
Instead, officials were looking at preparing to offer military assistance to authorities in Tripoli.
At present, there is no central government in Libya but efforts are being made to establish a national unity administration.
Mr Cameron was beginning a tour of South-East Asia last night, where he is expected to speak to foreign leaders about cooperating against extremist terror groups.
He said: “Isil is one of the biggest threats our world has faced. We will only defeat these brutal terrorists if we take action at home, overseas and online and if we unite with countries around the world against this common enemy.”
Possible action in Libya could involve sending military specialists to help and train local forces.
Foreign Office minister Baroness Anelay said: “We are working with Libya’s neighbours to enhance their capacity to tackle the threat [of Isil]. A lasting and inclusive political settlement and the establishment of a Government of National Accord is the best way to tackle the threat in the long term.
“We stand ready to support the Government of National Accord on security issues, not least tackling Isil, as soon as it is formed.”
Officials insist the focus of the security effort in Libya is to tackle traffickers behind the migrants ferried across the Mediterranean.